diff options
Diffstat (limited to 'doc/administration')
31 files changed, 1456 insertions, 1310 deletions
diff --git a/doc/administration/auth/google_secure_ldap.md b/doc/administration/auth/google_secure_ldap.md index 760af0cfd1a..c668f19ca7d 100644 --- a/doc/administration/auth/google_secure_ldap.md +++ b/doc/administration/auth/google_secure_ldap.md @@ -13,7 +13,7 @@ The steps below cover: ## Configuring Google LDAP client -1. Navigate to https://admin.google.com and sign in as a GSuite domain administrator. +1. Navigate to <https://admin.google.com> and sign in as a GSuite domain administrator. 1. Go to **Apps > LDAP > Add Client**. diff --git a/doc/administration/auth/ldap-ee.md b/doc/administration/auth/ldap-ee.md index 15f093bb62d..b45966fa920 100644 --- a/doc/administration/auth/ldap-ee.md +++ b/doc/administration/auth/ldap-ee.md @@ -185,7 +185,7 @@ group, as opposed to the full DN. ## Global group memberships lock -"Lock memberships to LDAP synchronization" setting allows instance administrators +"Lock memberships to LDAP synchronization" setting allows instance administrators to lock down user abilities to invite new members to a group. When enabled following happens: 1. Only administrator can manage memberships of any group including access levels. @@ -198,14 +198,14 @@ to lock down user abilities to invite new members to a group. When enabled follo NOTE: **Note:** These are cron formatted values. You can use a crontab generator to create -these values, for example http://www.crontabgenerator.com/. +these values, for example <http://www.crontabgenerator.com/>. By default, GitLab will run a worker once per day at 01:30 a.m. server time to check and update GitLab users against LDAP. You can manually configure LDAP user sync times by setting the following configuration values. The example below shows how to set LDAP user -sync to run once every 12 hours at the top of the hour. +sync to run once every 12 hours at the top of the hour. **Omnibus installations** @@ -233,7 +233,7 @@ sync to run once every 12 hours at the top of the hour. NOTE: **Note:** These are cron formatted values. You can use a crontab generator to create -these values, for example http://www.crontabgenerator.com/. +these values, for example <http://www.crontabgenerator.com/>. By default, GitLab will run a group sync process every hour, on the hour. @@ -245,8 +245,8 @@ for installations with a large number of LDAP users. Please review the your installation compares before proceeding. You can manually configure LDAP group sync times by setting the -following configuration values. The example below shows how to set group -sync to run once every 2 hours at the top of the hour. +following configuration values. The example below shows how to set group +sync to run once every 2 hours at the top of the hour. **Omnibus installations** diff --git a/doc/administration/auth/ldap.md b/doc/administration/auth/ldap.md index 54279897e04..79ac7fe0352 100644 --- a/doc/administration/auth/ldap.md +++ b/doc/administration/auth/ldap.md @@ -1,6 +1,4 @@ -[//]: # (Do *NOT* modify this file in EE documentation. All changes in this) -[//]: # (file should happen in CE, too. If the change is EE-specific, put) -[//]: # (it in `ldap-ee.md`.) +<!-- If the change is EE-specific, put it in `ldap-ee.md`, NOT here. --> # LDAP @@ -396,21 +394,34 @@ omniauth-ldap. ### Escaping special characters -If the `user_filter` DN contains special characters. For example, a comma: +The `user_filter` DN can contain special characters. For example: -``` -OU=GitLab, Inc,DC=gitlab,DC=com -``` +- A comma: -This character needs to be escaped as documented in [RFC 4515](https://tools.ietf.org/search/rfc4515). + ``` + OU=GitLab, Inc,DC=gitlab,DC=com + ``` -Due to the way the string is parsed, the special character needs to be converted -to hex and `\\5C\\` (`5C` = `\` in hex) added before it. -As an example the above DN would look like +- Open and close brackets: -``` -OU=GitLab\\5C\\2C Inc,DC=gitlab,DC=com -``` + ``` + OU=Gitlab (Inc),DC=gitlab,DC=com + ``` + + These characters must be escaped as documented in + [RFC 4515](https://tools.ietf.org/search/rfc4515). + +- Escape commas with `\2C`. For example: + + ``` + OU=GitLab\2C Inc,DC=gitlab,DC=com + ``` + +- Escape open and close brackets with `\28` and `\29`, respectively. For example: + + ``` + OU=Gitlab \28Inc\29,DC=gitlab,DC=com + ``` ## Enabling LDAP sign-in for existing GitLab users diff --git a/doc/administration/auth/oidc.md b/doc/administration/auth/oidc.md index 00422ec347c..6e48add6930 100644 --- a/doc/administration/auth/oidc.md +++ b/doc/administration/auth/oidc.md @@ -144,20 +144,20 @@ for more details: If you're having trouble, here are some tips: 1. Ensure `discovery` is set to `true`. Setting it to `false` requires -specifying all the URLs and keys required to make OpenID work. + specifying all the URLs and keys required to make OpenID work. 1. Check your system clock to ensure the time is synchronized properly. 1. As mentioned in [the -documentation](https://github.com/m0n9oose/omniauth_openid_connect), -make sure `issuer` corresponds to the base URL of the Discovery URL. For -example, `https://accounts.google.com` is used for the URL -`https://accounts.google.com/.well-known/openid-configuration`. + documentation](https://github.com/m0n9oose/omniauth_openid_connect), + make sure `issuer` corresponds to the base URL of the Discovery URL. For + example, `https://accounts.google.com` is used for the URL + `https://accounts.google.com/.well-known/openid-configuration`. 1. The OpenID Connect client uses HTTP Basic Authentication to send the -OAuth2 access token. For example, if you are seeing 401 errors upon -retrieving the `userinfo` endpoint, you may want to check your OpenID -Web server configuration. For example, for -[oauth2-server-php](https://github.com/bshaffer/oauth2-server-php), you -may need to [add a configuration parameter to -Apache](https://github.com/bshaffer/oauth2-server-php/issues/926#issuecomment-387502778). + OAuth2 access token. For example, if you are seeing 401 errors upon + retrieving the `userinfo` endpoint, you may want to check your OpenID + Web server configuration. For example, for + [oauth2-server-php](https://github.com/bshaffer/oauth2-server-php), you + may need to [add a configuration parameter to + Apache](https://github.com/bshaffer/oauth2-server-php/issues/926#issuecomment-387502778). diff --git a/doc/administration/container_registry.md b/doc/administration/container_registry.md index 4d55f2357c1..2e4b4efa0ac 100644 --- a/doc/administration/container_registry.md +++ b/doc/administration/container_registry.md @@ -689,6 +689,20 @@ You can add a configuration option for backwards compatibility. 1. Restart the registry for the changes to take affect. +### Docker connection error + +A Docker connection error can occur when there are special characters in either the group, +project or branch name. Special characters can include: + +* Leading underscore +* Trailing hyphen/dash +* Double hyphen/dash + +To get around this, you can [change the group path](../user/group/index.md#changing-a-groups-path), +[change the project path](../user/project/settings/index.md#renaming-a-repository) or change the +branch name. Another option is to create a [push rule](../push_rules/push_rules.html) to prevent +this at the instance level. + [ce-18239]: https://gitlab.com/gitlab-org/gitlab-ce/issues/18239 [docker-insecure-self-signed]: https://docs.docker.com/registry/insecure/#use-self-signed-certificates diff --git a/doc/administration/database_load_balancing.md b/doc/administration/database_load_balancing.md index 7f3be402b84..98404ff2a10 100644 --- a/doc/administration/database_load_balancing.md +++ b/doc/administration/database_load_balancing.md @@ -40,16 +40,16 @@ For example, say you have a primary (`db1.gitlab.com`) and two secondaries, `db2.gitlab.com` and `db3.gitlab.com`. For this setup you will need to have 3 load balancers, one for every host. For example: -* `primary.gitlab.com` forwards to `db1.gitlab.com` -* `secondary1.gitlab.com` forwards to `db2.gitlab.com` -* `secondary2.gitlab.com` forwards to `db3.gitlab.com` +- `primary.gitlab.com` forwards to `db1.gitlab.com` +- `secondary1.gitlab.com` forwards to `db2.gitlab.com` +- `secondary2.gitlab.com` forwards to `db3.gitlab.com` Now let's say that a failover happens and db2 becomes the new primary. This means forwarding should now happen as follows: -* `primary.gitlab.com` forwards to `db2.gitlab.com` -* `secondary1.gitlab.com` forwards to `db1.gitlab.com` -* `secondary2.gitlab.com` forwards to `db3.gitlab.com` +- `primary.gitlab.com` forwards to `db2.gitlab.com` +- `secondary1.gitlab.com` forwards to `db1.gitlab.com` +- `secondary2.gitlab.com` forwards to `db3.gitlab.com` GitLab does not take care of this for you, so you will need to do so yourself. @@ -209,9 +209,9 @@ without it immediately leading to errors being presented to the users. The load balancer logs various messages, such as: -* When a host is marked as offline -* When a host comes back online -* When all secondaries are offline +- When a host is marked as offline +- When a host comes back online +- When all secondaries are offline Each log message contains the tag `[DB-LB]` to make searching/filtering of such log entries easier. For example: diff --git a/doc/administration/geo/disaster_recovery/background_verification.md b/doc/administration/geo/disaster_recovery/background_verification.md index d4c8c2d3624..e19cd9bbfec 100644 --- a/doc/administration/geo/disaster_recovery/background_verification.md +++ b/doc/administration/geo/disaster_recovery/background_verification.md @@ -143,11 +143,13 @@ If the **primary** and **secondary** nodes have a checksum verification mismatch 1. On the project admin page get the **Gitaly storage name**, and **Gitaly relative path**:  -1. Navigate to the project's repository directory on both **primary** and **secondary** nodes (the path is usually `/var/opt/gitlab/git-data/repositories`). Note that if `git_data_dirs` is customized, check the directory layout on your server to be sure. +1. Navigate to the project's repository directory on both **primary** and **secondary** nodes + (the path is usually `/var/opt/gitlab/git-data/repositories`). Note that if `git_data_dirs` + is customized, check the directory layout on your server to be sure. - ```sh - cd /var/opt/gitlab/git-data/repositories - ``` + ```sh + cd /var/opt/gitlab/git-data/repositories + ``` 1. Run the following command on the **primary** node, redirecting the output to a file: diff --git a/doc/administration/geo/disaster_recovery/bring_primary_back.md b/doc/administration/geo/disaster_recovery/bring_primary_back.md index f4d31a98080..9a981b49349 100644 --- a/doc/administration/geo/disaster_recovery/bring_primary_back.md +++ b/doc/administration/geo/disaster_recovery/bring_primary_back.md @@ -21,20 +21,20 @@ To bring the former **primary** node up to date: 1. SSH into the former **primary** node that has fallen behind. 1. Make sure all the services are up: - ```sh - sudo gitlab-ctl start - ``` - - > **Note 1:** If you [disabled the **primary** node permanently][disaster-recovery-disable-primary], - > you need to undo those steps now. For Debian/Ubuntu you just need to run - > `sudo systemctl enable gitlab-runsvdir`. For CentOS 6, you need to install - > the GitLab instance from scratch and set it up as a **secondary** node by - > following [Setup instructions][setup-geo]. In this case, you don't need to follow the next step. - > - > **Note 2:** If you [changed the DNS records](index.md#step-4-optional-updating-the-primary-domain-dns-record) - > for this node during disaster recovery procedure you may need to [block - > all the writes to this node](https://gitlab.com/gitlab-org/gitlab-ee/blob/master/doc/gitlab-geo/planned-failover.md#block-primary-traffic) - > during this procedure. + ```sh + sudo gitlab-ctl start + ``` + + NOTE: **Note:** If you [disabled the **primary** node permanently][disaster-recovery-disable-primary], + you need to undo those steps now. For Debian/Ubuntu you just need to run + `sudo systemctl enable gitlab-runsvdir`. For CentOS 6, you need to install + the GitLab instance from scratch and set it up as a **secondary** node by + following [Setup instructions][setup-geo]. In this case, you don't need to follow the next step. + + NOTE: **Note:** If you [changed the DNS records](index.md#step-4-optional-updating-the-primary-domain-dns-record) + for this node during disaster recovery procedure you may need to [block + all the writes to this node](planned_failover.md#prevent-updates-to-the-primary-node) + during this procedure. 1. [Setup database replication][database-replication]. Note that in this case, **primary** node refers to the current **primary** node, and **secondary** node refers to the diff --git a/doc/administration/geo/disaster_recovery/index.md b/doc/administration/geo/disaster_recovery/index.md index 71dc797f281..86182b84062 100644 --- a/doc/administration/geo/disaster_recovery/index.md +++ b/doc/administration/geo/disaster_recovery/index.md @@ -39,48 +39,50 @@ must disable the **primary** node. 1. SSH into the **primary** node to stop and disable GitLab, if possible: - ```sh - sudo gitlab-ctl stop - ``` + ```sh + sudo gitlab-ctl stop + ``` - Prevent GitLab from starting up again if the server unexpectedly reboots: + Prevent GitLab from starting up again if the server unexpectedly reboots: - ```sh - sudo systemctl disable gitlab-runsvdir - ``` + ```sh + sudo systemctl disable gitlab-runsvdir + ``` - > **CentOS only**: In CentOS 6 or older, there is no easy way to prevent GitLab from being - > started if the machine reboots isn't available (see [gitlab-org/omnibus-gitlab#3058]). - > It may be safest to uninstall the GitLab package completely: + NOTE: **Note:** + (**CentOS only**) In CentOS 6 or older, there is no easy way to prevent GitLab from being + started if the machine reboots isn't available (see [gitlab-org/omnibus-gitlab#3058]). + It may be safest to uninstall the GitLab package completely: - ```sh - yum remove gitlab-ee - ``` + ```sh + yum remove gitlab-ee + ``` - > **Ubuntu 14.04 LTS**: If you are using an older version of Ubuntu - > or any other distro based on the Upstart init system, you can prevent GitLab - > from starting if the machine reboots by doing the following: + NOTE: **Note:** + (**Ubuntu 14.04 LTS**) If you are using an older version of Ubuntu + or any other distro based on the Upstart init system, you can prevent GitLab + from starting if the machine reboots by doing the following: - ```sh - initctl stop gitlab-runsvvdir - echo 'manual' > /etc/init/gitlab-runsvdir.override - initctl reload-configuration - ``` + ```sh + initctl stop gitlab-runsvvdir + echo 'manual' > /etc/init/gitlab-runsvdir.override + initctl reload-configuration + ``` 1. If you do not have SSH access to the **primary** node, take the machine offline and - prevent it from rebooting by any means at your disposal. - Since there are many ways you may prefer to accomplish this, we will avoid a - single recommendation. You may need to: - - Reconfigure the load balancers. - - Change DNS records (e.g., point the primary DNS record to the **secondary** - node in order to stop usage of the **primary** node). - - Stop the virtual servers. - - Block traffic through a firewall. - - Revoke object storage permissions from the **primary** node. - - Physically disconnect a machine. - -1. If you plan to - [update the primary domain DNS record](#step-4-optional-updating-the-primary-domain-dns-record), + prevent it from rebooting by any means at your disposal. + Since there are many ways you may prefer to accomplish this, we will avoid a + single recommendation. You may need to: + + - Reconfigure the load balancers. + - Change DNS records (e.g., point the primary DNS record to the **secondary** + node in order to stop usage of the **primary** node). + - Stop the virtual servers. + - Block traffic through a firewall. + - Revoke object storage permissions from the **primary** node. + - Physically disconnect a machine. + +1. If you plan to [update the primary domain DNS record](#step-4-optional-updating-the-primary-domain-dns-record), you may wish to lower the TTL now to speed up propagation. ### Step 3. Promoting a **secondary** node @@ -94,26 +96,26 @@ the **secondary** to the **primary**. 1. SSH in to your **secondary** node and login as root: - ```sh - sudo -i - ``` + ```sh + sudo -i + ``` 1. Edit `/etc/gitlab/gitlab.rb` to reflect its new status as **primary** by removing any lines that enabled the `geo_secondary_role`: - ```ruby - ## In pre-11.5 documentation, the role was enabled as follows. Remove this line. - geo_secondary_role['enable'] = true + ```ruby + ## In pre-11.5 documentation, the role was enabled as follows. Remove this line. + geo_secondary_role['enable'] = true - ## In 11.5+ documentation, the role was enabled as follows. Remove this line. - roles ['geo_secondary_role'] - ``` + ## In 11.5+ documentation, the role was enabled as follows. Remove this line. + roles ['geo_secondary_role'] + ``` 1. Promote the **secondary** node to the **primary** node. Execute: - ```sh - gitlab-ctl promote-to-primary-node - ``` + ```sh + gitlab-ctl promote-to-primary-node + ``` 1. Verify you can connect to the newly promoted **primary** node using the URL used previously for the **secondary** node. @@ -129,31 +131,31 @@ do this manually. 1. SSH in to the database node in the **secondary** and trigger PostgreSQL to promote to read-write: - ```bash - sudo gitlab-pg-ctl promote - ``` + ```bash + sudo gitlab-pg-ctl promote + ``` 1. Edit `/etc/gitlab/gitlab.rb` on every machine in the **secondary** to reflect its new status as **primary** by removing any lines that enabled the `geo_secondary_role`: - ```ruby - ## In pre-11.5 documentation, the role was enabled as follows. Remove this line. - geo_secondary_role['enable'] = true + ```ruby + ## In pre-11.5 documentation, the role was enabled as follows. Remove this line. + geo_secondary_role['enable'] = true - ## In 11.5+ documentation, the role was enabled as follows. Remove this line. - roles ['geo_secondary_role'] - ``` + ## In 11.5+ documentation, the role was enabled as follows. Remove this line. + roles ['geo_secondary_role'] + ``` - After making these changes [Reconfigure GitLab](../../restart_gitlab.md#omnibus-gitlab-reconfigure) each - machine so the changes take effect. + After making these changes [Reconfigure GitLab](../../restart_gitlab.md#omnibus-gitlab-reconfigure) each + machine so the changes take effect. 1. Promote the **secondary** to **primary**. SSH into a single application server and execute: - ```bash - sudo gitlab-rake geo:set_secondary_as_primary - ``` + ```bash + sudo gitlab-rake geo:set_secondary_as_primary + ``` 1. Verify you can connect to the newly promoted **primary** using the URL used previously for the **secondary**. @@ -167,37 +169,37 @@ secondary domain, like changing Git remotes and API URLs. 1. SSH into the **secondary** node and login as root: - ```sh - sudo -i - ``` + ```sh + sudo -i + ``` 1. Update the primary domain's DNS record. After updating the primary domain's DNS records to point to the **secondary** node, edit `/etc/gitlab/gitlab.rb` on the **secondary** node to reflect the new URL: - ```ruby - # Change the existing external_url configuration - external_url 'https://<new_external_url>' - ``` + ```ruby + # Change the existing external_url configuration + external_url 'https://<new_external_url>' + ``` - NOTE: **Note** - Changing `external_url` won't prevent access via the old secondary URL, as - long as the secondary DNS records are still intact. + NOTE: **Note** + Changing `external_url` won't prevent access via the old secondary URL, as + long as the secondary DNS records are still intact. 1. Reconfigure the **secondary** node for the change to take effect: - ```sh - gitlab-ctl reconfigure - ``` + ```sh + gitlab-ctl reconfigure + ``` 1. Execute the command below to update the newly promoted **primary** node URL: - ```sh - gitlab-rake geo:update_primary_node_url - ``` + ```sh + gitlab-rake geo:update_primary_node_url + ``` - This command will use the changed `external_url` configuration defined - in `/etc/gitlab/gitlab.rb`. + This command will use the changed `external_url` configuration defined + in `/etc/gitlab/gitlab.rb`. 1. Verify you can connect to the newly promoted **primary** using its URL. If you updated the DNS records for the primary domain, these changes may @@ -231,62 +233,61 @@ and after that you also need two extra steps. 1. SSH into the new **primary** node and login as root: - ```sh - sudo -i - ``` + ```sh + sudo -i + ``` 1. Edit `/etc/gitlab/gitlab.rb` - ```ruby - ## Enable a Geo Primary role (if you haven't yet) - roles ['geo_primary_role'] + ```ruby + ## Enable a Geo Primary role (if you haven't yet) + roles ['geo_primary_role'] - ## - # Allow PostgreSQL client authentication from the primary and secondary IPs. These IPs may be - # public or VPC addresses in CIDR format, for example ['198.51.100.1/32', '198.51.100.2/32'] - ## - postgresql['md5_auth_cidr_addresses'] = ['<primary_node_ip>/32', '<secondary_node_ip>/32'] + ## + # Allow PostgreSQL client authentication from the primary and secondary IPs. These IPs may be + # public or VPC addresses in CIDR format, for example ['198.51.100.1/32', '198.51.100.2/32'] + ## + postgresql['md5_auth_cidr_addresses'] = ['<primary_node_ip>/32', '<secondary_node_ip>/32'] - # Every secondary server needs to have its own slot so specify the number of secondary nodes you're going to have - postgresql['max_replication_slots'] = 1 + # Every secondary server needs to have its own slot so specify the number of secondary nodes you're going to have + postgresql['max_replication_slots'] = 1 - ## - ## Disable automatic database migrations temporarily - ## (until PostgreSQL is restarted and listening on the private address). - ## - gitlab_rails['auto_migrate'] = false + ## + ## Disable automatic database migrations temporarily + ## (until PostgreSQL is restarted and listening on the private address). + ## + gitlab_rails['auto_migrate'] = false + ``` - ``` - - (For more details about these settings you can read [Configure the primary server][configure-the-primary-server]) + (For more details about these settings you can read [Configure the primary server][configure-the-primary-server]) 1. Save the file and reconfigure GitLab for the database listen changes and the replication slot changes to be applied. - ```sh - gitlab-ctl reconfigure - ``` + ```sh + gitlab-ctl reconfigure + ``` - Restart PostgreSQL for its changes to take effect: + Restart PostgreSQL for its changes to take effect: - ```sh - gitlab-ctl restart postgresql - ``` + ```sh + gitlab-ctl restart postgresql + ``` 1. Re-enable migrations now that PostgreSQL is restarted and listening on the private address. - Edit `/etc/gitlab/gitlab.rb` and **change** the configuration to `true`: + Edit `/etc/gitlab/gitlab.rb` and **change** the configuration to `true`: - ```ruby - gitlab_rails['auto_migrate'] = true - ``` + ```ruby + gitlab_rails['auto_migrate'] = true + ``` - Save the file and reconfigure GitLab: + Save the file and reconfigure GitLab: - ```sh - gitlab-ctl reconfigure - ``` + ```sh + gitlab-ctl reconfigure + ``` ### Step 2. Initiate the replication process diff --git a/doc/administration/geo/disaster_recovery/planned_failover.md b/doc/administration/geo/disaster_recovery/planned_failover.md index b8071b5993f..c1a95157f8d 100644 --- a/doc/administration/geo/disaster_recovery/planned_failover.md +++ b/doc/administration/geo/disaster_recovery/planned_failover.md @@ -143,26 +143,26 @@ access to the **primary** node during the maintenance window. all HTTP, HTTPS and SSH traffic to/from the **primary** node, **except** for your IP and the **secondary** node's IP. - For instance, you might run the following commands on the server(s) making up your **primary** node: + For instance, you might run the following commands on the server(s) making up your **primary** node: - ```sh - sudo iptables -A INPUT -p tcp -s <secondary_node_ip> --destination-port 22 -j ACCEPT - sudo iptables -A INPUT -p tcp -s <your_ip> --destination-port 22 -j ACCEPT - sudo iptables -A INPUT --destination-port 22 -j REJECT + ```sh + sudo iptables -A INPUT -p tcp -s <secondary_node_ip> --destination-port 22 -j ACCEPT + sudo iptables -A INPUT -p tcp -s <your_ip> --destination-port 22 -j ACCEPT + sudo iptables -A INPUT --destination-port 22 -j REJECT - sudo iptables -A INPUT -p tcp -s <secondary_node_ip> --destination-port 80 -j ACCEPT - sudo iptables -A INPUT -p tcp -s <your_ip> --destination-port 80 -j ACCEPT - sudo iptables -A INPUT --tcp-dport 80 -j REJECT + sudo iptables -A INPUT -p tcp -s <secondary_node_ip> --destination-port 80 -j ACCEPT + sudo iptables -A INPUT -p tcp -s <your_ip> --destination-port 80 -j ACCEPT + sudo iptables -A INPUT --tcp-dport 80 -j REJECT - sudo iptables -A INPUT -p tcp -s <secondary_node_ip> --destination-port 443 -j ACCEPT - sudo iptables -A INPUT -p tcp -s <your_ip> --destination-port 443 -j ACCEPT - sudo iptables -A INPUT --tcp-dport 443 -j REJECT - ``` + sudo iptables -A INPUT -p tcp -s <secondary_node_ip> --destination-port 443 -j ACCEPT + sudo iptables -A INPUT -p tcp -s <your_ip> --destination-port 443 -j ACCEPT + sudo iptables -A INPUT --tcp-dport 443 -j REJECT + ``` - From this point, users will be unable to view their data or make changes on the - **primary** node. They will also be unable to log in to the **secondary** node. - However, existing sessions will work for the remainder of the maintenance period, and - public data will be accessible throughout. + From this point, users will be unable to view their data or make changes on the + **primary** node. They will also be unable to log in to the **secondary** node. + However, existing sessions will work for the remainder of the maintenance period, and + public data will be accessible throughout. 1. Verify the **primary** node is blocked to HTTP traffic by visiting it in browser via another IP. The server should refuse connection. @@ -187,10 +187,11 @@ access to the **primary** node during the maintenance window. before it is completed will cause the work to be lost. 1. On the **primary** node, navigate to **Admin Area > Geo** and wait for the following conditions to be true of the **secondary** node you are failing over to: - - All replication meters to each 100% replicated, 0% failures. - - All verification meters reach 100% verified, 0% failures. - - Database replication lag is 0ms. - - The Geo log cursor is up to date (0 events behind). + + - All replication meters to each 100% replicated, 0% failures. + - All verification meters reach 100% verified, 0% failures. + - Database replication lag is 0ms. + - The Geo log cursor is up to date (0 events behind). 1. On the **secondary** node, navigate to **Admin Area > Monitoring > Background Jobs > Queues** and wait for all the `geo` queues to drop to 0 queued and 0 running jobs. diff --git a/doc/administration/geo/replication/configuration.md b/doc/administration/geo/replication/configuration.md index 3d4f69d3abe..dd5e09c0dd7 100644 --- a/doc/administration/geo/replication/configuration.md +++ b/doc/administration/geo/replication/configuration.md @@ -16,11 +16,10 @@ The basic steps of configuring a **secondary** node are to: You are encouraged to first read through all the steps before executing them in your testing/production environment. -> **Notes:** -> - **Do not** setup any custom authentication for the **secondary** nodes. This will be - handled by the **primary** node. -> - Any change that requires access to the **Admin Area** needs to be done in the - **primary** node because the **secondary** node is a read-only replica. +NOTE: **Note:** +**Do not** set up any custom authentication for the **secondary** nodes. This will be handled by the **primary** node. +Any change that requires access to the **Admin Area** needs to be done in the +**primary** node because the **secondary** node is a read-only replica. ### Step 1. Manually replicate secret GitLab values @@ -31,47 +30,47 @@ they must be manually replicated to the **secondary** node. 1. SSH into the **primary** node, and execute the command below: - ```sh - sudo cat /etc/gitlab/gitlab-secrets.json - ``` + ```sh + sudo cat /etc/gitlab/gitlab-secrets.json + ``` - This will display the secrets that need to be replicated, in JSON format. + This will display the secrets that need to be replicated, in JSON format. 1. SSH into the **secondary** node and login as the `root` user: - ```sh - sudo -i - ``` + ```sh + sudo -i + ``` 1. Make a backup of any existing secrets: - ```sh - mv /etc/gitlab/gitlab-secrets.json /etc/gitlab/gitlab-secrets.json.`date +%F` - ``` + ```sh + mv /etc/gitlab/gitlab-secrets.json /etc/gitlab/gitlab-secrets.json.`date +%F` + ``` 1. Copy `/etc/gitlab/gitlab-secrets.json` from the **primary** node to the **secondary** node, or copy-and-paste the file contents between nodes: - ```sh - sudo editor /etc/gitlab/gitlab-secrets.json + ```sh + sudo editor /etc/gitlab/gitlab-secrets.json - # paste the output of the `cat` command you ran on the primary - # save and exit - ``` + # paste the output of the `cat` command you ran on the primary + # save and exit + ``` 1. Ensure the file permissions are correct: - ```sh - chown root:root /etc/gitlab/gitlab-secrets.json - chmod 0600 /etc/gitlab/gitlab-secrets.json - ``` + ```sh + chown root:root /etc/gitlab/gitlab-secrets.json + chmod 0600 /etc/gitlab/gitlab-secrets.json + ``` 1. Reconfigure the **secondary** node for the change to take effect: - ```sh - gitlab-ctl reconfigure - gitlab-ctl restart - ``` + ```sh + gitlab-ctl reconfigure + gitlab-ctl restart + ``` ### Step 2. Manually replicate the **primary** node's SSH host keys @@ -89,80 +88,80 @@ keys must be manually replicated to the **secondary** node. 1. SSH into the **secondary** node and login as the `root` user: - ```sh - sudo -i - ``` + ```sh + sudo -i + ``` 1. Make a backup of any existing SSH host keys: - ```sh - find /etc/ssh -iname ssh_host_* -exec cp {} {}.backup.`date +%F` \; - ``` + ```sh + find /etc/ssh -iname ssh_host_* -exec cp {} {}.backup.`date +%F` \; + ``` 1. Copy OpenSSH host keys from the **primary** node: - If you can access your **primary** node using the **root** user: + If you can access your **primary** node using the **root** user: - ```sh - # Run this from the secondary node, change `<primary_node_fqdn>` for the IP or FQDN of the server - scp root@<primary_node_fqdn>:/etc/ssh/ssh_host_*_key* /etc/ssh - ``` + ```sh + # Run this from the secondary node, change `<primary_node_fqdn>` for the IP or FQDN of the server + scp root@<primary_node_fqdn>:/etc/ssh/ssh_host_*_key* /etc/ssh + ``` - If you only have access through a user with **sudo** privileges: + If you only have access through a user with **sudo** privileges: - ```sh - # Run this from your primary node: - sudo tar --transform 's/.*\///g' -zcvf ~/geo-host-key.tar.gz /etc/ssh/ssh_host_*_key* + ```sh + # Run this from your primary node: + sudo tar --transform 's/.*\///g' -zcvf ~/geo-host-key.tar.gz /etc/ssh/ssh_host_*_key* - # Run this from your secondary node: - scp <user_with_sudo>@<primary_node_fqdn>:geo-host-key.tar.gz . - tar zxvf ~/geo-host-key.tar.gz -C /etc/ssh - ``` + # Run this from your secondary node: + scp <user_with_sudo>@<primary_node_fqdn>:geo-host-key.tar.gz . + tar zxvf ~/geo-host-key.tar.gz -C /etc/ssh + ``` 1. On your **secondary** node, ensure the file permissions are correct: - ```sh - chown root:root /etc/ssh/ssh_host_*_key* - chmod 0600 /etc/ssh/ssh_host_*_key* - ``` + ```sh + chown root:root /etc/ssh/ssh_host_*_key* + chmod 0600 /etc/ssh/ssh_host_*_key* + ``` 1. To verify key fingerprint matches, execute the following command on both nodes: - ```sh - for file in /etc/ssh/ssh_host_*_key; do ssh-keygen -lf $file; done - ``` + ```sh + for file in /etc/ssh/ssh_host_*_key; do ssh-keygen -lf $file; done + ``` - You should get an output similar to this one and they should be identical on both nodes: + You should get an output similar to this one and they should be identical on both nodes: - ```sh - 1024 SHA256:FEZX2jQa2bcsd/fn/uxBzxhKdx4Imc4raXrHwsbtP0M root@serverhostname (DSA) - 256 SHA256:uw98R35Uf+fYEQ/UnJD9Br4NXUFPv7JAUln5uHlgSeY root@serverhostname (ECDSA) - 256 SHA256:sqOUWcraZQKd89y/QQv/iynPTOGQxcOTIXU/LsoPmnM root@serverhostname (ED25519) - 2048 SHA256:qwa+rgir2Oy86QI+PZi/QVR+MSmrdrpsuH7YyKknC+s root@serverhostname (RSA) - ``` + ```sh + 1024 SHA256:FEZX2jQa2bcsd/fn/uxBzxhKdx4Imc4raXrHwsbtP0M root@serverhostname (DSA) + 256 SHA256:uw98R35Uf+fYEQ/UnJD9Br4NXUFPv7JAUln5uHlgSeY root@serverhostname (ECDSA) + 256 SHA256:sqOUWcraZQKd89y/QQv/iynPTOGQxcOTIXU/LsoPmnM root@serverhostname (ED25519) + 2048 SHA256:qwa+rgir2Oy86QI+PZi/QVR+MSmrdrpsuH7YyKknC+s root@serverhostname (RSA) + ``` 1. Verify that you have the correct public keys for the existing private keys: - ```sh - # This will print the fingerprint for private keys: - for file in /etc/ssh/ssh_host_*_key; do ssh-keygen -lf $file; done + ```sh + # This will print the fingerprint for private keys: + for file in /etc/ssh/ssh_host_*_key; do ssh-keygen -lf $file; done - # This will print the fingerprint for public keys: - for file in /etc/ssh/ssh_host_*_key.pub; do ssh-keygen -lf $file; done - ``` + # This will print the fingerprint for public keys: + for file in /etc/ssh/ssh_host_*_key.pub; do ssh-keygen -lf $file; done + ``` - NOTE: **Note**: - The output for private keys and public keys command should generate the same fingerprint. + NOTE: **Note:** + The output for private keys and public keys command should generate the same fingerprint. 1. Restart sshd on your **secondary** node: - ```sh - # Debian or Ubuntu installations - sudo service ssh reload + ```sh + # Debian or Ubuntu installations + sudo service ssh reload - # CentOS installations - sudo service sshd reload - ``` + # CentOS installations + sudo service sshd reload + ``` ### Step 3. Add the **secondary** node @@ -176,22 +175,22 @@ keys must be manually replicated to the **secondary** node. 1. Click the **Add node** button. 1. SSH into your GitLab **secondary** server and restart the services: - ```sh - gitlab-ctl restart - ``` + ```sh + gitlab-ctl restart + ``` - Check if there are any common issue with your Geo setup by running: + Check if there are any common issue with your Geo setup by running: - ```sh - gitlab-rake gitlab:geo:check - ``` + ```sh + gitlab-rake gitlab:geo:check + ``` 1. SSH into your **primary** server and login as root to verify the **secondary** node is reachable or there are any common issue with your Geo setup: - ```sh - gitlab-rake gitlab:geo:check - ``` + ```sh + gitlab-rake gitlab:geo:check + ``` Once added to the admin panel and restarted, the **secondary** node will automatically start replicating missing data from the **primary** node in a process known as **backfill**. @@ -250,9 +249,8 @@ The two most obvious issues that can become apparent in the dashboard are: 1. Database replication not working well. 1. Instance to instance notification not working. In that case, it can be something of the following: - - You are using a custom certificate or custom CA (see the - [troubleshooting document]). - - The instance is firewalled (check your firewall rules). + - You are using a custom certificate or custom CA (see the [troubleshooting document](troubleshooting.md)). + - The instance is firewalled (check your firewall rules). Please note that disabling a **secondary** node will stop the synchronization process. @@ -304,5 +302,4 @@ See the [troubleshooting document](troubleshooting.md). [gitlab-org/gitlab-ee#3789]: https://gitlab.com/gitlab-org/gitlab-ee/issues/3789 [gitlab-com/infrastructure#2821]: https://gitlab.com/gitlab-com/infrastructure/issues/2821 [omnibus-ssl]: https://docs.gitlab.com/omnibus/settings/ssl.html -[troubleshooting document]: troubleshooting.md [using-geo]: using_a_geo_server.md diff --git a/doc/administration/geo/replication/database.md b/doc/administration/geo/replication/database.md index 6658c37752a..021ed2d9f3c 100644 --- a/doc/administration/geo/replication/database.md +++ b/doc/administration/geo/replication/database.md @@ -52,186 +52,188 @@ There is an [issue where support is being discussed](https://gitlab.com/gitlab-o 1. SSH into your GitLab **primary** server and login as root: - ```sh - sudo -i - ``` + ```sh + sudo -i + ``` 1. Execute the command below to define the node as **primary** node: - ```sh - gitlab-ctl set-geo-primary-node - ``` + ```sh + gitlab-ctl set-geo-primary-node + ``` - This command will use your defined `external_url` in `/etc/gitlab/gitlab.rb`. + This command will use your defined `external_url` in `/etc/gitlab/gitlab.rb`. 1. GitLab 10.4 and up only: Do the following to make sure the `gitlab` database user has a password defined: - Generate a MD5 hash of the desired password: + Generate a MD5 hash of the desired password: - ```sh - gitlab-ctl pg-password-md5 gitlab - # Enter password: <your_password_here> - # Confirm password: <your_password_here> - # fca0b89a972d69f00eb3ec98a5838484 - ``` + ```sh + gitlab-ctl pg-password-md5 gitlab + # Enter password: <your_password_here> + # Confirm password: <your_password_here> + # fca0b89a972d69f00eb3ec98a5838484 + ``` - Edit `/etc/gitlab/gitlab.rb`: + Edit `/etc/gitlab/gitlab.rb`: - ```ruby - # Fill with the hash generated by `gitlab-ctl pg-password-md5 gitlab` - postgresql['sql_user_password'] = '<md5_hash_of_your_password>' + ```ruby + # Fill with the hash generated by `gitlab-ctl pg-password-md5 gitlab` + postgresql['sql_user_password'] = '<md5_hash_of_your_password>' - # Every node that runs Unicorn or Sidekiq needs to have the database - # password specified as below. If you have a high-availability setup, this - # must be present in all application nodes. - gitlab_rails['db_password'] = '<your_password_here>' - ``` + # Every node that runs Unicorn or Sidekiq needs to have the database + # password specified as below. If you have a high-availability setup, this + # must be present in all application nodes. + gitlab_rails['db_password'] = '<your_password_here>' + ``` 1. Omnibus GitLab already has a [replication user] called `gitlab_replicator`. You must set the password for this user manually. You will be prompted to enter a password: - ```sh - gitlab-ctl set-replication-password - ``` + ```sh + gitlab-ctl set-replication-password + ``` - This command will also read the `postgresql['sql_replication_user']` Omnibus - setting in case you have changed `gitlab_replicator` username to something - else. + This command will also read the `postgresql['sql_replication_user']` Omnibus + setting in case you have changed `gitlab_replicator` username to something + else. - If you are using an external database not managed by Omnibus GitLab, you need - to create the replicator user and define a password to it manually: + If you are using an external database not managed by Omnibus GitLab, you need + to create the replicator user and define a password to it manually: - ```sql - --- Create a new user 'replicator' - CREATE USER gitlab_replicator; + ```sql + --- Create a new user 'replicator' + CREATE USER gitlab_replicator; - --- Set/change a password and grants replication privilege - ALTER USER gitlab_replicator WITH REPLICATION ENCRYPTED PASSWORD '<replication_password>'; - ``` + --- Set/change a password and grants replication privilege + ALTER USER gitlab_replicator WITH REPLICATION ENCRYPTED PASSWORD '<replication_password>'; + ``` 1. Configure PostgreSQL to listen on network interfaces: - For security reasons, PostgreSQL does not listen on any network interfaces - by default. However, Geo requires the **secondary** node to be able to - connect to the **primary** node's database. For this reason, we need the address of - each node. Note: For external PostgreSQL instances, see [additional instructions](external_database.md). - - If you are using a cloud provider, you can lookup the addresses for each - Geo node through your cloud provider's management console. - - To lookup the address of a Geo node, SSH in to the Geo node and execute: - - ```sh - ## - ## Private address - ## - ip route get 255.255.255.255 | awk '{print "Private address:", $NF; exit}' - - ## - ## Public address - ## - echo "External address: $(curl --silent ipinfo.io/ip)" - ``` - - In most cases, the following addresses will be used to configure GitLab - Geo: - - | Configuration | Address | - |:----------------------------------------|:------------------------------------------------------| - | `postgresql['listen_address']` | **Primary** node's public or VPC private address. | - | `postgresql['md5_auth_cidr_addresses']` | **Secondary** node's public or VPC private addresses. | - - If you are using Google Cloud Platform, SoftLayer, or any other vendor that - provides a virtual private cloud (VPC) you can use the **secondary** node's private - address (corresponds to "internal address" for Google Cloud Platform) for - `postgresql['md5_auth_cidr_addresses']` and `postgresql['listen_address']`. - - The `listen_address` option opens PostgreSQL up to network connections - with the interface corresponding to the given address. See [the PostgreSQL - documentation][pg-docs-runtime-conn] for more details. - - Depending on your network configuration, the suggested addresses may not - be correct. If your **primary** node and **secondary** nodes connect over a local - area network, or a virtual network connecting availability zones like - [Amazon's VPC](https://aws.amazon.com/vpc/) or [Google's VPC](https://cloud.google.com/vpc/) - you should use the **secondary** node's private address for `postgresql['md5_auth_cidr_addresses']`. - - Edit `/etc/gitlab/gitlab.rb` and add the following, replacing the IP - addresses with addresses appropriate to your network configuration: - - ```ruby - ## - ## Geo Primary role - ## - configure dependent flags automatically to enable Geo - ## - roles ['geo_primary_role'] - - ## - ## Primary address - ## - replace '<primary_node_ip>' with the public or VPC address of your Geo primary node - ## - postgresql['listen_address'] = '<primary_node_ip>' - - ## - # Allow PostgreSQL client authentication from the primary and secondary IPs. These IPs may be - # public or VPC addresses in CIDR format, for example ['198.51.100.1/32', '198.51.100.2/32'] - ## - postgresql['md5_auth_cidr_addresses'] = ['<primary_node_ip>/32', '<secondary_node_ip>/32'] - - ## - ## Replication settings - ## - set this to be the number of Geo secondary nodes you have - ## - postgresql['max_replication_slots'] = 1 - # postgresql['max_wal_senders'] = 10 - # postgresql['wal_keep_segments'] = 10 - - ## - ## Disable automatic database migrations temporarily - ## (until PostgreSQL is restarted and listening on the private address). - ## - gitlab_rails['auto_migrate'] = false - ``` + For security reasons, PostgreSQL does not listen on any network interfaces + by default. However, Geo requires the **secondary** node to be able to + connect to the **primary** node's database. For this reason, we need the address of + each node. + + NOTE: **Note:** For external PostgreSQL instances, see [additional instructions](external_database.md). + + If you are using a cloud provider, you can lookup the addresses for each + Geo node through your cloud provider's management console. + + To lookup the address of a Geo node, SSH in to the Geo node and execute: + + ```sh + ## + ## Private address + ## + ip route get 255.255.255.255 | awk '{print "Private address:", $NF; exit}' + + ## + ## Public address + ## + echo "External address: $(curl --silent ipinfo.io/ip)" + ``` + + In most cases, the following addresses will be used to configure GitLab + Geo: + + | Configuration | Address | + |:----------------------------------------|:------------------------------------------------------| + | `postgresql['listen_address']` | **Primary** node's public or VPC private address. | + | `postgresql['md5_auth_cidr_addresses']` | **Secondary** node's public or VPC private addresses. | + + If you are using Google Cloud Platform, SoftLayer, or any other vendor that + provides a virtual private cloud (VPC) you can use the **secondary** node's private + address (corresponds to "internal address" for Google Cloud Platform) for + `postgresql['md5_auth_cidr_addresses']` and `postgresql['listen_address']`. + + The `listen_address` option opens PostgreSQL up to network connections + with the interface corresponding to the given address. See [the PostgreSQL + documentation][pg-docs-runtime-conn] for more details. + + Depending on your network configuration, the suggested addresses may not + be correct. If your **primary** node and **secondary** nodes connect over a local + area network, or a virtual network connecting availability zones like + [Amazon's VPC](https://aws.amazon.com/vpc/) or [Google's VPC](https://cloud.google.com/vpc/) + you should use the **secondary** node's private address for `postgresql['md5_auth_cidr_addresses']`. + + Edit `/etc/gitlab/gitlab.rb` and add the following, replacing the IP + addresses with addresses appropriate to your network configuration: + + ```ruby + ## + ## Geo Primary role + ## - configure dependent flags automatically to enable Geo + ## + roles ['geo_primary_role'] + + ## + ## Primary address + ## - replace '<primary_node_ip>' with the public or VPC address of your Geo primary node + ## + postgresql['listen_address'] = '<primary_node_ip>' + + ## + # Allow PostgreSQL client authentication from the primary and secondary IPs. These IPs may be + # public or VPC addresses in CIDR format, for example ['198.51.100.1/32', '198.51.100.2/32'] + ## + postgresql['md5_auth_cidr_addresses'] = ['<primary_node_ip>/32', '<secondary_node_ip>/32'] + + ## + ## Replication settings + ## - set this to be the number of Geo secondary nodes you have + ## + postgresql['max_replication_slots'] = 1 + # postgresql['max_wal_senders'] = 10 + # postgresql['wal_keep_segments'] = 10 + + ## + ## Disable automatic database migrations temporarily + ## (until PostgreSQL is restarted and listening on the private address). + ## + gitlab_rails['auto_migrate'] = false + ``` 1. Optional: If you want to add another **secondary** node, the relevant setting would look like: - ```ruby - postgresql['md5_auth_cidr_addresses'] = ['<primary_node_ip>/32', '<secondary_node_ip>/32', '<another_secondary_node_ip>/32'] - ``` + ```ruby + postgresql['md5_auth_cidr_addresses'] = ['<primary_node_ip>/32', '<secondary_node_ip>/32', '<another_secondary_node_ip>/32'] + ``` - You may also want to edit the `wal_keep_segments` and `max_wal_senders` to - match your database replication requirements. Consult the [PostgreSQL - - Replication documentation][pg-docs-runtime-replication] - for more information. + You may also want to edit the `wal_keep_segments` and `max_wal_senders` to + match your database replication requirements. Consult the [PostgreSQL - + Replication documentation][pg-docs-runtime-replication] + for more information. 1. Save the file and reconfigure GitLab for the database listen changes and the replication slot changes to be applied: - ```sh - gitlab-ctl reconfigure - ``` + ```sh + gitlab-ctl reconfigure + ``` - Restart PostgreSQL for its changes to take effect: + Restart PostgreSQL for its changes to take effect: - ```sh - gitlab-ctl restart postgresql - ``` + ```sh + gitlab-ctl restart postgresql + ``` 1. Re-enable migrations now that PostgreSQL is restarted and listening on the private address. - Edit `/etc/gitlab/gitlab.rb` and **change** the configuration to `true`: + Edit `/etc/gitlab/gitlab.rb` and **change** the configuration to `true`: - ```ruby - gitlab_rails['auto_migrate'] = true - ``` + ```ruby + gitlab_rails['auto_migrate'] = true + ``` - Save the file and reconfigure GitLab: + Save the file and reconfigure GitLab: - ```sh - gitlab-ctl reconfigure - ``` + ```sh + gitlab-ctl reconfigure + ``` 1. Now that the PostgreSQL server is set up to accept remote connections, run `netstat -plnt | grep 5432` to make sure that PostgreSQL is listening on port @@ -241,143 +243,143 @@ There is an [issue where support is being discussed](https://gitlab.com/gitlab-o will be used automatically to protect your PostgreSQL traffic from eavesdroppers, but to protect against active ("man-in-the-middle") attackers, the **secondary** node needs a copy of the certificate. Make a copy of the PostgreSQL - `server.crt` file on the **primary** node by running this command: + `server.crt` file on the **primary** node by running this command: - ```sh - cat ~gitlab-psql/data/server.crt - ``` + ```sh + cat ~gitlab-psql/data/server.crt + ``` - Copy the output into a clipboard or into a local file. You - will need it when setting up the **secondary** node! The certificate is not sensitive - data. + Copy the output into a clipboard or into a local file. You + will need it when setting up the **secondary** node! The certificate is not sensitive + data. ### Step 2. Configure the **secondary** server 1. SSH into your GitLab **secondary** server and login as root: - ``` - sudo -i - ``` + ``` + sudo -i + ``` 1. Stop application server and Sidekiq - ``` - gitlab-ctl stop unicorn - gitlab-ctl stop sidekiq - ``` + ``` + gitlab-ctl stop unicorn + gitlab-ctl stop sidekiq + ``` - NOTE: **Note**: - This step is important so we don't try to execute anything before the node is fully configured. + NOTE: **Note:** + This step is important so we don't try to execute anything before the node is fully configured. 1. [Check TCP connectivity][rake-maintenance] to the **primary** node's PostgreSQL server: - ```sh - gitlab-rake gitlab:tcp_check[<primary_node_ip>,5432] - ``` + ```sh + gitlab-rake gitlab:tcp_check[<primary_node_ip>,5432] + ``` - NOTE: **Note**: - If this step fails, you may be using the wrong IP address, or a firewall may - be preventing access to the server. Check the IP address, paying close - attention to the difference between public and private addresses and ensure - that, if a firewall is present, the **secondary** node is permitted to connect to the - **primary** node on port 5432. + NOTE: **Note:** + If this step fails, you may be using the wrong IP address, or a firewall may + be preventing access to the server. Check the IP address, paying close + attention to the difference between public and private addresses and ensure + that, if a firewall is present, the **secondary** node is permitted to connect to the + **primary** node on port 5432. 1. Create a file `server.crt` in the **secondary** server, with the content you got on the last step of the **primary** node's setup: - ``` - editor server.crt - ``` + ``` + editor server.crt + ``` 1. Set up PostgreSQL TLS verification on the **secondary** node: - Install the `server.crt` file: + Install the `server.crt` file: - ```sh - install \ - -D \ - -o gitlab-psql \ - -g gitlab-psql \ - -m 0400 \ - -T server.crt ~gitlab-psql/.postgresql/root.crt - ``` + ```sh + install \ + -D \ + -o gitlab-psql \ + -g gitlab-psql \ + -m 0400 \ + -T server.crt ~gitlab-psql/.postgresql/root.crt + ``` - PostgreSQL will now only recognize that exact certificate when verifying TLS - connections. The certificate can only be replicated by someone with access - to the private key, which is **only** present on the **primary** node. + PostgreSQL will now only recognize that exact certificate when verifying TLS + connections. The certificate can only be replicated by someone with access + to the private key, which is **only** present on the **primary** node. 1. Test that the `gitlab-psql` user can connect to the **primary** node's database (the default Omnibus database name is gitlabhq_production): - ```sh - sudo \ - -u gitlab-psql /opt/gitlab/embedded/bin/psql \ - --list \ - -U gitlab_replicator \ - -d "dbname=gitlabhq_production sslmode=verify-ca" \ - -W \ - -h <primary_node_ip> - ``` + ```sh + sudo \ + -u gitlab-psql /opt/gitlab/embedded/bin/psql \ + --list \ + -U gitlab_replicator \ + -d "dbname=gitlabhq_production sslmode=verify-ca" \ + -W \ + -h <primary_node_ip> + ``` - When prompted enter the password you set in the first step for the - `gitlab_replicator` user. If all worked correctly, you should see - the list of **primary** node's databases. + When prompted enter the password you set in the first step for the + `gitlab_replicator` user. If all worked correctly, you should see + the list of **primary** node's databases. - A failure to connect here indicates that the TLS configuration is incorrect. - Ensure that the contents of `~gitlab-psql/data/server.crt` on the **primary** node - match the contents of `~gitlab-psql/.postgresql/root.crt` on the **secondary** node. + A failure to connect here indicates that the TLS configuration is incorrect. + Ensure that the contents of `~gitlab-psql/data/server.crt` on the **primary** node + match the contents of `~gitlab-psql/.postgresql/root.crt` on the **secondary** node. 1. Configure PostgreSQL to enable FDW support: - This step is similar to how we configured the **primary** instance. - We need to enable this, to enable FDW support, even if using a single node. - - Edit `/etc/gitlab/gitlab.rb` and add the following, replacing the IP - addresses with addresses appropriate to your network configuration: - - ```ruby - ## - ## Geo Secondary role - ## - configure dependent flags automatically to enable Geo - ## - roles ['geo_secondary_role'] - - ## - ## Secondary address - ## - replace '<secondary_node_ip>' with the public or VPC address of your Geo secondary node - ## - postgresql['listen_address'] = '<secondary_node_ip>' - postgresql['md5_auth_cidr_addresses'] = ['<secondary_node_ip>/32'] - - ## - ## Database credentials password (defined previously in primary node) - ## - replicate same values here as defined in primary node - ## - postgresql['sql_user_password'] = '<md5_hash_of_your_password>' - gitlab_rails['db_password'] = '<your_password_here>' - - ## - ## Enable FDW support for the Geo Tracking Database (improves performance) - ## - geo_secondary['db_fdw'] = true - ``` - - For external PostgreSQL instances, see [additional instructions](external_database.md). - If you bring a former **primary** node back online to serve as a **secondary** node, then you also need to remove `roles ['geo_primary_role']` or `geo_primary_role['enable'] = true`. + This step is similar to how we configured the **primary** instance. + We need to enable this, to enable FDW support, even if using a single node. + + Edit `/etc/gitlab/gitlab.rb` and add the following, replacing the IP + addresses with addresses appropriate to your network configuration: + + ```ruby + ## + ## Geo Secondary role + ## - configure dependent flags automatically to enable Geo + ## + roles ['geo_secondary_role'] + + ## + ## Secondary address + ## - replace '<secondary_node_ip>' with the public or VPC address of your Geo secondary node + ## + postgresql['listen_address'] = '<secondary_node_ip>' + postgresql['md5_auth_cidr_addresses'] = ['<secondary_node_ip>/32'] + + ## + ## Database credentials password (defined previously in primary node) + ## - replicate same values here as defined in primary node + ## + postgresql['sql_user_password'] = '<md5_hash_of_your_password>' + gitlab_rails['db_password'] = '<your_password_here>' + + ## + ## Enable FDW support for the Geo Tracking Database (improves performance) + ## + geo_secondary['db_fdw'] = true + ``` + + For external PostgreSQL instances, see [additional instructions](external_database.md). + If you bring a former **primary** node back online to serve as a **secondary** node, then you also need to remove `roles ['geo_primary_role']` or `geo_primary_role['enable'] = true`. 1. Reconfigure GitLab for the changes to take effect: - ```sh - gitlab-ctl reconfigure - ``` + ```sh + gitlab-ctl reconfigure + ``` 1. Restart PostgreSQL for the IP change to take effect and reconfigure again: - ```sh - gitlab-ctl restart postgresql - gitlab-ctl reconfigure - ``` + ```sh + gitlab-ctl restart postgresql + gitlab-ctl reconfigure + ``` - This last reconfigure will provision the FDW configuration and enable it. + This last reconfigure will provision the FDW configuration and enable it. ### Step 3. Initiate the replication process @@ -394,9 +396,9 @@ data before running `pg_basebackup`. 1. SSH into your GitLab **secondary** server and login as root: - ```sh - sudo -i - ``` + ```sh + sudo -i + ``` 1. Choose a database-friendly name to use for your **secondary** node to use as the replication slot name. For example, if your domain is @@ -404,38 +406,40 @@ data before running `pg_basebackup`. name as shown in the commands below. 1. Execute the command below to start a backup/restore and begin the replication - CAUTION: **Warning:** Each Geo **secondary** node must have its own unique replication slot name. - Using the same slot name between two secondaries will break PostgreSQL replication. - - ```sh - gitlab-ctl replicate-geo-database \ - --slot-name=<secondary_node_name> \ - --host=<primary_node_ip> - ``` - - When prompted, enter the _plaintext_ password you set up for the `gitlab_replicator` - user in the first step. - - This command also takes a number of additional options. You can use `--help` - to list them all, but here are a couple of tips: - - If PostgreSQL is listening on a non-standard port, add `--port=` as well. - - If your database is too large to be transferred in 30 minutes, you will need - to increase the timeout, e.g., `--backup-timeout=3600` if you expect the - initial replication to take under an hour. - - Pass `--sslmode=disable` to skip PostgreSQL TLS authentication altogether - (e.g., you know the network path is secure, or you are using a site-to-site - VPN). This is **not** safe over the public Internet! - - You can read more details about each `sslmode` in the - [PostgreSQL documentation][pg-docs-ssl]; - the instructions above are carefully written to ensure protection against - both passive eavesdroppers and active "man-in-the-middle" attackers. - - Change the `--slot-name` to the name of the replication slot - to be used on the **primary** database. The script will attempt to create the - replication slot automatically if it does not exist. - - If you're repurposing an old server into a Geo **secondary** node, you'll need to - add `--force` to the command line. - - When not in a production machine you can disable backup step if you - really sure this is what you want by adding `--skip-backup` + + CAUTION: **Warning:** Each Geo **secondary** node must have its own unique replication slot name. + Using the same slot name between two secondaries will break PostgreSQL replication. + + ```sh + gitlab-ctl replicate-geo-database \ + --slot-name=<secondary_node_name> \ + --host=<primary_node_ip> + ``` + + When prompted, enter the _plaintext_ password you set up for the `gitlab_replicator` + user in the first step. + + This command also takes a number of additional options. You can use `--help` + to list them all, but here are a couple of tips: + + - If PostgreSQL is listening on a non-standard port, add `--port=` as well. + - If your database is too large to be transferred in 30 minutes, you will need + to increase the timeout, e.g., `--backup-timeout=3600` if you expect the + initial replication to take under an hour. + - Pass `--sslmode=disable` to skip PostgreSQL TLS authentication altogether + (e.g., you know the network path is secure, or you are using a site-to-site + VPN). This is **not** safe over the public Internet! + - You can read more details about each `sslmode` in the + [PostgreSQL documentation][pg-docs-ssl]; + the instructions above are carefully written to ensure protection against + both passive eavesdroppers and active "man-in-the-middle" attackers. + - Change the `--slot-name` to the name of the replication slot + to be used on the **primary** database. The script will attempt to create the + replication slot automatically if it does not exist. + - If you're repurposing an old server into a Geo **secondary** node, you'll need to + add `--force` to the command line. + - When not in a production machine you can disable backup step if you + really sure this is what you want by adding `--skip-backup` The replication process is now complete. @@ -452,42 +456,42 @@ it will need a separate read-only user to make [PostgreSQL FDW queries][FDW] work: 1. On the **primary** Geo database, enter the PostgreSQL on the console as an - admin user. If you are using an Omnibus-managed database, log onto the **primary** - node that is running the PostgreSQL database (the default Omnibus database name is gitlabhq_production): + admin user. If you are using an Omnibus-managed database, log onto the **primary** + node that is running the PostgreSQL database (the default Omnibus database name is gitlabhq_production): - ```sh - sudo \ - -u gitlab-psql /opt/gitlab/embedded/bin/psql \ - -h /var/opt/gitlab/postgresql gitlabhq_production - ``` + ```sh + sudo \ + -u gitlab-psql /opt/gitlab/embedded/bin/psql \ + -h /var/opt/gitlab/postgresql gitlabhq_production + ``` 1. Then create the read-only user: - ```sql - -- NOTE: Use the password defined earlier - CREATE USER gitlab_geo_fdw WITH password 'mypassword'; - GRANT CONNECT ON DATABASE gitlabhq_production to gitlab_geo_fdw; - GRANT USAGE ON SCHEMA public TO gitlab_geo_fdw; - GRANT SELECT ON ALL TABLES IN SCHEMA public TO gitlab_geo_fdw; - GRANT SELECT ON ALL SEQUENCES IN SCHEMA public TO gitlab_geo_fdw; + ```sql + -- NOTE: Use the password defined earlier + CREATE USER gitlab_geo_fdw WITH password 'mypassword'; + GRANT CONNECT ON DATABASE gitlabhq_production to gitlab_geo_fdw; + GRANT USAGE ON SCHEMA public TO gitlab_geo_fdw; + GRANT SELECT ON ALL TABLES IN SCHEMA public TO gitlab_geo_fdw; + GRANT SELECT ON ALL SEQUENCES IN SCHEMA public TO gitlab_geo_fdw; - -- Tables created by "gitlab" should be made read-only for "gitlab_geo_fdw" - -- automatically. - ALTER DEFAULT PRIVILEGES FOR USER gitlab IN SCHEMA public GRANT SELECT ON TABLES TO gitlab_geo_fdw; - ALTER DEFAULT PRIVILEGES FOR USER gitlab IN SCHEMA public GRANT SELECT ON SEQUENCES TO gitlab_geo_fdw; - ``` + -- Tables created by "gitlab" should be made read-only for "gitlab_geo_fdw" + -- automatically. + ALTER DEFAULT PRIVILEGES FOR USER gitlab IN SCHEMA public GRANT SELECT ON TABLES TO gitlab_geo_fdw; + ALTER DEFAULT PRIVILEGES FOR USER gitlab IN SCHEMA public GRANT SELECT ON SEQUENCES TO gitlab_geo_fdw; + ``` 1. On the **secondary** nodes, change `/etc/gitlab/gitlab.rb`: - ``` - geo_postgresql['fdw_external_user'] = 'gitlab_geo_fdw' - ``` + ``` + geo_postgresql['fdw_external_user'] = 'gitlab_geo_fdw' + ``` 1. Save the file and reconfigure GitLab for the changes to be applied: - ```sh - gitlab-ctl reconfigure - ``` + ```sh + gitlab-ctl reconfigure + ``` ## Troubleshooting diff --git a/doc/administration/geo/replication/external_database.md b/doc/administration/geo/replication/external_database.md index 177ca68613e..452e4f490a6 100644 --- a/doc/administration/geo/replication/external_database.md +++ b/doc/administration/geo/replication/external_database.md @@ -4,7 +4,7 @@ This document is relevant if you are using a PostgreSQL instance that is *not managed by Omnibus*. This includes cloud-managed instances like AWS RDS, or manually installed and configured PostgreSQL instances. -NOTE: **Note**: +NOTE: **Note:** We strongly recommend running Omnibus-managed instances as they are actively developed and tested. We aim to be compatible with most external (not managed by Omnibus) databases but we do not guarantee compatibility. @@ -13,17 +13,17 @@ developed and tested. We aim to be compatible with most external 1. SSH into a GitLab **primary** application server and login as root: - ```sh - sudo -i - ``` + ```sh + sudo -i + ``` 1. Execute the command below to define the node as **primary** node: - ```sh - gitlab-ctl set-geo-primary-node - ``` + ```sh + gitlab-ctl set-geo-primary-node + ``` - This command will use your defined `external_url` in `/etc/gitlab/gitlab.rb`. + This command will use your defined `external_url` in `/etc/gitlab/gitlab.rb`. ### Configure the external database to be replicated @@ -101,26 +101,27 @@ To configure the connection to the external read-replica database and enable Log 1. SSH into a GitLab **secondary** application server and login as root: - ```bash - sudo -i - ``` + ```bash + sudo -i + ``` 1. Edit `/etc/gitlab/gitlab.rb` and add the following - ```ruby - ## - ## Geo Secondary role - ## - configure dependent flags automatically to enable Geo - ## - roles ['geo_secondary_role'] + ```ruby + ## + ## Geo Secondary role + ## - configure dependent flags automatically to enable Geo + ## + roles ['geo_secondary_role'] + + # note this is shared between both databases, + # make sure you define the same password in both + gitlab_rails['db_password'] = '<your_password_here>' - # note this is shared between both databases, - # make sure you define the same password in both - gitlab_rails['db_password'] = '<your_password_here>' + gitlab_rails['db_username'] = 'gitlab' + gitlab_rails['db_host'] = '<database_read_replica_host>' + ``` - gitlab_rails['db_username'] = 'gitlab' - gitlab_rails['db_host'] = '<database_read_replica_host>' - ``` 1. Save the file and [reconfigure GitLab](../../restart_gitlab.md#omnibus-gitlab-reconfigure) ### Configure the tracking database @@ -147,73 +148,72 @@ the tracking database on port 5432. 1. SSH into a GitLab **secondary** server and login as root: - ```bash - sudo -i - ``` + ```bash + sudo -i + ``` 1. Edit `/etc/gitlab/gitlab.rb` with the connection params and credentials for - the machine with the PostgreSQL instance: + the machine with the PostgreSQL instance: - ```ruby - geo_secondary['db_username'] = 'gitlab_geo' - geo_secondary['db_password'] = '<your_password_here>' + ```ruby + geo_secondary['db_username'] = 'gitlab_geo' + geo_secondary['db_password'] = '<your_password_here>' - geo_secondary['db_host'] = '<tracking_database_host>' - geo_secondary['db_port'] = <tracking_database_port> # change to the correct port - geo_secondary['db_fdw'] = true # enable FDW - geo_postgresql['enable'] = false # don't use internal managed instance - ``` + geo_secondary['db_host'] = '<tracking_database_host>' + geo_secondary['db_port'] = <tracking_database_port> # change to the correct port + geo_secondary['db_fdw'] = true # enable FDW + geo_postgresql['enable'] = false # don't use internal managed instance + ``` 1. Save the file and [reconfigure GitLab](../../restart_gitlab.md#omnibus-gitlab-reconfigure) 1. Run the tracking database migrations: - ```bash - gitlab-rake geo:db:create - gitlab-rake geo:db:migrate - ``` - -1. Configure the - [PostgreSQL FDW](https://www.postgresql.org/docs/9.6/static/postgres-fdw.html) - connection and credentials: - - Save the script below in a file, ex. `/tmp/geo_fdw.sh` and modify the connection - params to match your environment. Execute it to set up the FDW connection. - - ```bash - #!/bin/bash - - # Secondary Database connection params: - DB_HOST="<public_ip_or_vpc_private_ip>" - DB_NAME="gitlabhq_production" - DB_USER="gitlab" - DB_PASS="<your_password_here>" - DB_PORT="5432" - - # Tracking Database connection params: - GEO_DB_HOST="<public_ip_or_vpc_private_ip>" - GEO_DB_NAME="gitlabhq_geo_production" - GEO_DB_USER="gitlab_geo" - GEO_DB_PORT="5432" - - query_exec () { - gitlab-psql -h $GEO_DB_HOST -d $GEO_DB_NAME -p $GEO_DB_PORT -c "${1}" - } - - query_exec "CREATE EXTENSION postgres_fdw;" - query_exec "CREATE SERVER gitlab_secondary FOREIGN DATA WRAPPER postgres_fdw OPTIONS (host '${DB_HOST}', dbname '${DB_NAME}', port '${DB_PORT}');" - query_exec "CREATE USER MAPPING FOR ${GEO_DB_USER} SERVER gitlab_secondary OPTIONS (user '${DB_USER}', password '${DB_PASS}');" - query_exec "CREATE SCHEMA gitlab_secondary;" - query_exec "GRANT USAGE ON FOREIGN SERVER gitlab_secondary TO ${GEO_DB_USER};" - ``` - - NOTE: **Note:** The script template above uses `gitlab-psql` as it's intended to be executed from the Geo machine, - but you can change it to `psql` and run it from any machine that has access to the database. We also recommend using - `psql` for AWS RDS. + ```bash + gitlab-rake geo:db:create + gitlab-rake geo:db:migrate + ``` + +1. Configure the [PostgreSQL FDW](https://www.postgresql.org/docs/9.6/static/postgres-fdw.html) + connection and credentials: + + Save the script below in a file, ex. `/tmp/geo_fdw.sh` and modify the connection + params to match your environment. Execute it to set up the FDW connection. + + ```bash + #!/bin/bash + + # Secondary Database connection params: + DB_HOST="<public_ip_or_vpc_private_ip>" + DB_NAME="gitlabhq_production" + DB_USER="gitlab" + DB_PASS="<your_password_here>" + DB_PORT="5432" + + # Tracking Database connection params: + GEO_DB_HOST="<public_ip_or_vpc_private_ip>" + GEO_DB_NAME="gitlabhq_geo_production" + GEO_DB_USER="gitlab_geo" + GEO_DB_PORT="5432" + + query_exec () { + gitlab-psql -h $GEO_DB_HOST -d $GEO_DB_NAME -p $GEO_DB_PORT -c "${1}" + } + + query_exec "CREATE EXTENSION postgres_fdw;" + query_exec "CREATE SERVER gitlab_secondary FOREIGN DATA WRAPPER postgres_fdw OPTIONS (host '${DB_HOST}', dbname '${DB_NAME}', port '${DB_PORT}');" + query_exec "CREATE USER MAPPING FOR ${GEO_DB_USER} SERVER gitlab_secondary OPTIONS (user '${DB_USER}', password '${DB_PASS}');" + query_exec "CREATE SCHEMA gitlab_secondary;" + query_exec "GRANT USAGE ON FOREIGN SERVER gitlab_secondary TO ${GEO_DB_USER};" + ``` + + NOTE: **Note:** The script template above uses `gitlab-psql` as it's intended to be executed from the Geo machine, + but you can change it to `psql` and run it from any machine that has access to the database. We also recommend using + `psql` for AWS RDS. 1. Save the file and [restart GitLab](../../restart_gitlab.md#omnibus-gitlab-restart) 1. Populate the FDW tables: - ```bash - gitlab-rake geo:db:refresh_foreign_tables - ``` + ```bash + gitlab-rake geo:db:refresh_foreign_tables + ``` diff --git a/doc/administration/geo/replication/faq.md b/doc/administration/geo/replication/faq.md index dd1af0dbf9c..c527248bc72 100644 --- a/doc/administration/geo/replication/faq.md +++ b/doc/administration/geo/replication/faq.md @@ -6,8 +6,8 @@ The requirements are listed [on the index page](index.md#requirements-for-runnin ## How does Geo know which projects to sync? -On each **secondary** node, there is a read-only replicated copy of the GitLab database. -A **secondary** node also has a tracking database where it stores which projects have been synced. +On each **secondary** node, there is a read-only replicated copy of the GitLab database. +A **secondary** node also has a tracking database where it stores which projects have been synced. Geo compares the two databases to find projects that are not yet tracked. At the start, this tracking database is empty, so Geo will start trying to update from every project that it can see in the GitLab database. @@ -15,19 +15,19 @@ At the start, this tracking database is empty, so Geo will start trying to updat For each project to sync: 1. Geo will issue a `git fetch geo --mirror` to get the latest information from the **primary** node. -If there are no changes, the sync will be fast and end quickly. Otherwise, it will pull the latest commits. + If there are no changes, the sync will be fast and end quickly. Otherwise, it will pull the latest commits. 1. The **secondary** node will update the tracking database to store the fact that it has synced projects A, B, C, etc. 1. Repeat until all projects are synced. -When someone pushes a commit to the **primary** node, it generates an event in the GitLab database that the repository has changed. +When someone pushes a commit to the **primary** node, it generates an event in the GitLab database that the repository has changed. The **secondary** node sees this event, marks the project in question as dirty, and schedules the project to be resynced. To ensure that problems with pipelines (for example, syncs failing too many times or jobs being lost) don't permanently stop projects syncing, Geo also periodically checks the tracking database for projects that are marked as dirty. This check happens when -the number of concurrent syncs falls below `repos_max_capacity` and there are no new projects waiting to be synced. +the number of concurrent syncs falls below `repos_max_capacity` and there are no new projects waiting to be synced. -Geo also has a checksum feature which runs a SHA256 sum across all the Git references to the SHA values. -If the refs don't match between the **primary** node and the **secondary** node, then the **secondary** node will mark that project as dirty and try to resync it. -So even if we have an outdated tracking database, the validation should activate and find discrepancies in the repository state and resync. +Geo also has a checksum feature which runs a SHA256 sum across all the Git references to the SHA values. +If the refs don't match between the **primary** node and the **secondary** node, then the **secondary** node will mark that project as dirty and try to resync it. +So even if we have an outdated tracking database, the validation should activate and find discrepancies in the repository state and resync. ## Can I use Geo in a disaster recovery situation? diff --git a/doc/administration/geo/replication/high_availability.md b/doc/administration/geo/replication/high_availability.md index 921a3ef1c7a..61e18df2480 100644 --- a/doc/administration/geo/replication/high_availability.md +++ b/doc/administration/geo/replication/high_availability.md @@ -50,17 +50,17 @@ The following steps enable a GitLab cluster to serve as the **primary** node. 1. Edit `/etc/gitlab/gitlab.rb` and add the following: - ```ruby - ## - ## Enable the Geo primary role - ## - roles ['geo_primary_role'] - - ## - ## Disable automatic migrations - ## - gitlab_rails['auto_migrate'] = false - ``` + ```ruby + ## + ## Enable the Geo primary role + ## + roles ['geo_primary_role'] + + ## + ## Disable automatic migrations + ## + gitlab_rails['auto_migrate'] = false + ``` After making these changes, [reconfigure GitLab][gitlab-reconfigure] so the changes take effect. @@ -79,9 +79,9 @@ The **primary** database will require modification later, as part of A **secondary** cluster is similar to any other GitLab HA cluster, with two major differences: -* The main PostgreSQL database is a read-only replica of the **primary** node's +- The main PostgreSQL database is a read-only replica of the **primary** node's PostgreSQL database. -* There is also a single PostgreSQL database for the **secondary** cluster, +- There is also a single PostgreSQL database for the **secondary** cluster, called the "tracking database", which tracks the synchronization state of various resources. @@ -93,9 +93,9 @@ from the normal HA setup. Configure the following services, again using the non-Geo high availability documentation: -* [Configuring Redis for GitLab HA](../../high_availability/redis.md) for high +- [Configuring Redis for GitLab HA](../../high_availability/redis.md) for high availability. -* [NFS](../../high_availability/nfs.md) which will store data that is +- [NFS](../../high_availability/nfs.md) which will store data that is synchronized from the **primary** node. ### Step 2: Configure the main read-only replica PostgreSQL database on the **secondary** node @@ -107,36 +107,36 @@ Configure the [**secondary** database](database.md) as a read-only replica of the **primary** database. Use the following as a guide. 1. Edit `/etc/gitlab/gitlab.rb` in the replica database machine, and add the - following: - - ```ruby - ## - ## Configure the PostgreSQL role - ## - roles ['postgres_role'] - - ## - ## Secondary address - ## - replace '<secondary_node_ip>' with the public or VPC address of your Geo secondary node - ## - replace '<tracking_database_ip>' with the public or VPC address of your Geo tracking database node - ## - postgresql['listen_address'] = '<secondary_node_ip>' - postgresql['md5_auth_cidr_addresses'] = ['<secondary_node_ip>/32', '<tracking_database_ip>/32'] - - ## - ## Database credentials password (defined previously in primary node) - ## - replicate same values here as defined in primary node - ## - postgresql['sql_user_password'] = '<md5_hash_of_your_password>' - gitlab_rails['db_password'] = '<your_password_here>' - - ## - ## When running the Geo tracking database on a separate machine, disable it - ## here and allow connections from the tracking database host. And ensure - ## the tracking database IP is in postgresql['md5_auth_cidr_addresses'] above. - ## - geo_postgresql['enable'] = false - ``` + following: + + ```ruby + ## + ## Configure the PostgreSQL role + ## + roles ['postgres_role'] + + ## + ## Secondary address + ## - replace '<secondary_node_ip>' with the public or VPC address of your Geo secondary node + ## - replace '<tracking_database_ip>' with the public or VPC address of your Geo tracking database node + ## + postgresql['listen_address'] = '<secondary_node_ip>' + postgresql['md5_auth_cidr_addresses'] = ['<secondary_node_ip>/32', '<tracking_database_ip>/32'] + + ## + ## Database credentials password (defined previously in primary node) + ## - replicate same values here as defined in primary node + ## + postgresql['sql_user_password'] = '<md5_hash_of_your_password>' + gitlab_rails['db_password'] = '<your_password_here>' + + ## + ## When running the Geo tracking database on a separate machine, disable it + ## here and allow connections from the tracking database host. And ensure + ## the tracking database IP is in postgresql['md5_auth_cidr_addresses'] above. + ## + geo_postgresql['enable'] = false + ``` After making these changes, [reconfigure GitLab][gitlab-reconfigure] so the changes take effect. @@ -151,47 +151,47 @@ only a single machine, rather than as a PostgreSQL cluster. Configure the tracking database. 1. Edit `/etc/gitlab/gitlab.rb` in the tracking database machine, and add the - following: - - ```ruby - ## - ## Enable the Geo secondary tracking database - ## - geo_postgresql['enable'] = true - geo_postgresql['listen_address'] = '<ip_address_of_this_host>' - geo_postgresql['sql_user_password'] = '<tracking_database_password_md5_hash>' - - ## - ## Configure FDW connection to the replica database - ## - geo_secondary['db_fdw'] = true - geo_postgresql['fdw_external_password'] = '<replica_database_password_plaintext>' - geo_postgresql['md5_auth_cidr_addresses'] = ['<replica_database_ip>/32'] - gitlab_rails['db_host'] = '<replica_database_ip>' - - # Prevent reconfigure from attempting to run migrations on the replica DB - gitlab_rails['auto_migrate'] = false - - ## - ## Disable all other services that aren't needed, since we don't have a role - ## that does this. - ## - alertmanager['enable'] = false - consul['enable'] = false - gitaly['enable'] = false - gitlab_monitor['enable'] = false - gitlab_workhorse['enable'] = false - nginx['enable'] = false - node_exporter['enable'] = false - pgbouncer_exporter['enable'] = false - postgresql['enable'] = false - prometheus['enable'] = false - redis['enable'] = false - redis_exporter['enable'] = false - repmgr['enable'] = false - sidekiq['enable'] = false - unicorn['enable'] = false - ``` + following: + + ```ruby + ## + ## Enable the Geo secondary tracking database + ## + geo_postgresql['enable'] = true + geo_postgresql['listen_address'] = '<ip_address_of_this_host>' + geo_postgresql['sql_user_password'] = '<tracking_database_password_md5_hash>' + + ## + ## Configure FDW connection to the replica database + ## + geo_secondary['db_fdw'] = true + geo_postgresql['fdw_external_password'] = '<replica_database_password_plaintext>' + geo_postgresql['md5_auth_cidr_addresses'] = ['<replica_database_ip>/32'] + gitlab_rails['db_host'] = '<replica_database_ip>' + + # Prevent reconfigure from attempting to run migrations on the replica DB + gitlab_rails['auto_migrate'] = false + + ## + ## Disable all other services that aren't needed, since we don't have a role + ## that does this. + ## + alertmanager['enable'] = false + consul['enable'] = false + gitaly['enable'] = false + gitlab_monitor['enable'] = false + gitlab_workhorse['enable'] = false + nginx['enable'] = false + node_exporter['enable'] = false + pgbouncer_exporter['enable'] = false + postgresql['enable'] = false + prometheus['enable'] = false + redis['enable'] = false + redis_exporter['enable'] = false + repmgr['enable'] = false + sidekiq['enable'] = false + unicorn['enable'] = false + ``` After making these changes, [reconfigure GitLab][gitlab-reconfigure] so the changes take effect. @@ -211,50 +211,50 @@ following modifications: 1. Edit `/etc/gitlab/gitlab.rb` on each application server in the **secondary** cluster, and add the following: - ```ruby - ## - ## Enable the Geo secondary role - ## - roles ['geo_secondary_role', 'application_role'] - - ## - ## Disable automatic migrations - ## - gitlab_rails['auto_migrate'] = false - - ## - ## Configure the connection to the tracking DB. And disable application - ## servers from running tracking databases. - ## - geo_secondary['db_host'] = '<geo_tracking_db_host>' - geo_secondary['db_password'] = '<geo_tracking_db_password>' - geo_postgresql['enable'] = false - - ## - ## Configure connection to the streaming replica database, if you haven't - ## already - ## - gitlab_rails['db_host'] = '<replica_database_host>' - gitlab_rails['db_password'] = '<replica_database_password>' - - ## - ## Configure connection to Redis, if you haven't already - ## - gitlab_rails['redis_host'] = '<redis_host>' - gitlab_rails['redis_password'] = '<redis_password>' - - ## - ## If you are using custom users not managed by Omnibus, you need to specify - ## UIDs and GIDs like below, and ensure they match between servers in a - ## cluster to avoid permissions issues - ## - user['uid'] = 9000 - user['gid'] = 9000 - web_server['uid'] = 9001 - web_server['gid'] = 9001 - registry['uid'] = 9002 - registry['gid'] = 9002 - ``` + ```ruby + ## + ## Enable the Geo secondary role + ## + roles ['geo_secondary_role', 'application_role'] + + ## + ## Disable automatic migrations + ## + gitlab_rails['auto_migrate'] = false + + ## + ## Configure the connection to the tracking DB. And disable application + ## servers from running tracking databases. + ## + geo_secondary['db_host'] = '<geo_tracking_db_host>' + geo_secondary['db_password'] = '<geo_tracking_db_password>' + geo_postgresql['enable'] = false + + ## + ## Configure connection to the streaming replica database, if you haven't + ## already + ## + gitlab_rails['db_host'] = '<replica_database_host>' + gitlab_rails['db_password'] = '<replica_database_password>' + + ## + ## Configure connection to Redis, if you haven't already + ## + gitlab_rails['redis_host'] = '<redis_host>' + gitlab_rails['redis_password'] = '<redis_password>' + + ## + ## If you are using custom users not managed by Omnibus, you need to specify + ## UIDs and GIDs like below, and ensure they match between servers in a + ## cluster to avoid permissions issues + ## + user['uid'] = 9000 + user['gid'] = 9000 + web_server['uid'] = 9001 + web_server['gid'] = 9001 + registry['uid'] = 9002 + registry['gid'] = 9002 + ``` NOTE: **Note:** If you had set up PostgreSQL cluster using the omnibus package and you had set @@ -270,15 +270,15 @@ After making these changes [Reconfigure GitLab][gitlab-reconfigure] so the chang On the secondary the following GitLab frontend services will be enabled: -* geo-logcursor -* gitlab-pages -* gitlab-workhorse -* logrotate -* nginx -* registry -* remote-syslog -* sidekiq -* unicorn +- geo-logcursor +- gitlab-pages +- gitlab-workhorse +- logrotate +- nginx +- registry +- remote-syslog +- sidekiq +- unicorn Verify these services by running `sudo gitlab-ctl status` on the frontend application servers. diff --git a/doc/administration/geo/replication/index.md b/doc/administration/geo/replication/index.md index 54377f7ae4e..8e1d1cb46ba 100644 --- a/doc/administration/geo/replication/index.md +++ b/doc/administration/geo/replication/index.md @@ -80,8 +80,8 @@ In this diagram: - If present, the [LDAP server](#ldap) should be configured to replicate for [Disaster Recovery](../disaster_recovery/index.md) scenarios. - A **secondary** node performs different type of synchronizations against the **primary** node, using a special authorization protected by JWT: - - Repositories are cloned/updated via Git over HTTPS. - - Attachments, LFS objects, and other files are downloaded via HTTPS using a private API endpoint. + - Repositories are cloned/updated via Git over HTTPS. + - Attachments, LFS objects, and other files are downloaded via HTTPS using a private API endpoint. From the perspective of a user performing Git operations: @@ -107,8 +107,8 @@ The following are required to run Geo: - An operating system that supports OpenSSH 6.9+ (needed for [fast lookup of authorized SSH keys in the database](../../operations/fast_ssh_key_lookup.md)) The following operating systems are known to ship with a current version of OpenSSH: - - [CentOS](https://www.centos.org) 7.4+ - - [Ubuntu](https://www.ubuntu.com) 16.04+ + - [CentOS](https://www.centos.org) 7.4+ + - [Ubuntu](https://www.ubuntu.com) 16.04+ - PostgreSQL 9.6+ with [FDW](https://www.postgresql.org/docs/9.6/postgres-fdw.html) support and [Streaming Replication](https://wiki.postgresql.org/wiki/Streaming_Replication) - Git 2.9+ diff --git a/doc/administration/geo/replication/remove_geo_node.md b/doc/administration/geo/replication/remove_geo_node.md index b190fe7d42d..6bdaad8f783 100644 --- a/doc/administration/geo/replication/remove_geo_node.md +++ b/doc/administration/geo/replication/remove_geo_node.md @@ -10,41 +10,42 @@ Once removed from the Geo admin page, you must stop and uninstall the **secondar 1. On the **secondary** node, stop GitLab: - ```bash - sudo gitlab-ctl stop - ``` + ```bash + sudo gitlab-ctl stop + ``` + 1. On the **secondary** node, uninstall GitLab: - ```bash - # Stop gitlab and remove its supervision process - sudo gitlab-ctl uninstall + ```bash + # Stop gitlab and remove its supervision process + sudo gitlab-ctl uninstall - # Debian/Ubuntu - sudo dpkg --remove gitlab-ee + # Debian/Ubuntu + sudo dpkg --remove gitlab-ee - # Redhat/Centos - sudo rpm --erase gitlab-ee - ``` + # Redhat/Centos + sudo rpm --erase gitlab-ee + ``` Once GitLab has been uninstalled from the **secondary** node, the replication slot must be dropped from the **primary** node's database as follows: 1. On the **primary** node, start a PostgreSQL console session: - ```bash - sudo gitlab-psql - ``` + ```bash + sudo gitlab-psql + ``` - NOTE: **Note:** - Using `gitlab-rails dbconsole` will not work, because managing replication slots requires superuser permissions. + NOTE: **Note:** + Using `gitlab-rails dbconsole` will not work, because managing replication slots requires superuser permissions. 1. Find the name of the relevant replication slot. This is the slot that is specified with `--slot-name` when running the replicate command: `gitlab-ctl replicate-geo-database`. - ```sql - SELECT * FROM pg_replication_slots; - ``` + ```sql + SELECT * FROM pg_replication_slots; + ``` 1. Remove the replication slot for the **secondary** node: - ```sql - SELECT pg_drop_replication_slot('<name_of_slot>'); - ``` + ```sql + SELECT pg_drop_replication_slot('<name_of_slot>'); + ``` diff --git a/doc/administration/geo/replication/troubleshooting.md b/doc/administration/geo/replication/troubleshooting.md index 846afd8f5f4..c7c78407084 100644 --- a/doc/administration/geo/replication/troubleshooting.md +++ b/doc/administration/geo/replication/troubleshooting.md @@ -184,17 +184,17 @@ log data to build up in `pg_xlog`. Removing the unused slots can reduce the amou 1. Start a PostgreSQL console session: - ```sh - sudo gitlab-psql gitlabhq_production - ``` + ```sh + sudo gitlab-psql gitlabhq_production + ``` - > Note that using `gitlab-rails dbconsole` will not work, because managing replication slots requires superuser permissions. + Note: **Note:** Using `gitlab-rails dbconsole` will not work, because managing replication slots requires superuser permissions. 1. View your replication slots with: - ```sql - SELECT * FROM pg_replication_slots; - ``` + ```sql + SELECT * FROM pg_replication_slots; + ``` Slots where `active` is `f` are not active. @@ -204,9 +204,9 @@ Slots where `active` is `f` are not active. - If you are no longer using the slot (e.g. you no longer have Geo enabled), you can remove it with in the PostgreSQL console session: - ```sql - SELECT pg_drop_replication_slot('<name_of_extra_slot>'); - ``` + ```sql + SELECT pg_drop_replication_slot('<name_of_extra_slot>'); + ``` ### Very large repositories never successfully synchronize on the **secondary** node @@ -237,82 +237,82 @@ to start again from scratch, there are a few steps that can help you: 1. Stop Sidekiq and the Geo LogCursor - It's possible to make Sidekiq stop gracefully, but making it stop getting new jobs and - wait until the current jobs to finish processing. + It's possible to make Sidekiq stop gracefully, but making it stop getting new jobs and + wait until the current jobs to finish processing. - You need to send a **SIGTSTP** kill signal for the first phase and them a **SIGTERM** - when all jobs have finished. Otherwise just use the `gitlab-ctl stop` commands. + You need to send a **SIGTSTP** kill signal for the first phase and them a **SIGTERM** + when all jobs have finished. Otherwise just use the `gitlab-ctl stop` commands. - ```sh - gitlab-ctl status sidekiq - # run: sidekiq: (pid 10180) <- this is the PID you will use - kill -TSTP 10180 # change to the correct PID + ```sh + gitlab-ctl status sidekiq + # run: sidekiq: (pid 10180) <- this is the PID you will use + kill -TSTP 10180 # change to the correct PID - gitlab-ctl stop sidekiq - gitlab-ctl stop geo-logcursor - ``` + gitlab-ctl stop sidekiq + gitlab-ctl stop geo-logcursor + ``` - You can watch sidekiq logs to know when sidekiq jobs processing have finished: + You can watch sidekiq logs to know when sidekiq jobs processing have finished: - ```sh - gitlab-ctl tail sidekiq - ``` + ```sh + gitlab-ctl tail sidekiq + ``` 1. Rename repository storage folders and create new ones - ```sh - mv /var/opt/gitlab/git-data/repositories /var/opt/gitlab/git-data/repositories.old - mkdir -p /var/opt/gitlab/git-data/repositories - chown git:git /var/opt/gitlab/git-data/repositories - ``` + ```sh + mv /var/opt/gitlab/git-data/repositories /var/opt/gitlab/git-data/repositories.old + mkdir -p /var/opt/gitlab/git-data/repositories + chown git:git /var/opt/gitlab/git-data/repositories + ``` - TIP: **Tip** - You may want to remove the `/var/opt/gitlab/git-data/repositories.old` in the future - as soon as you confirmed that you don't need it anymore, to save disk space. + TIP: **Tip** + You may want to remove the `/var/opt/gitlab/git-data/repositories.old` in the future + as soon as you confirmed that you don't need it anymore, to save disk space. 1. _(Optional)_ Rename other data folders and create new ones - CAUTION: **Caution**: - You may still have files on the **secondary** node that have been removed from **primary** node but - removal have not been reflected. If you skip this step, they will never be removed - from this Geo node. + CAUTION: **Caution**: + You may still have files on the **secondary** node that have been removed from **primary** node but + removal have not been reflected. If you skip this step, they will never be removed + from this Geo node. - Any uploaded content like file attachments, avatars or LFS objects are stored in a - subfolder in one of the two paths below: + Any uploaded content like file attachments, avatars or LFS objects are stored in a + subfolder in one of the two paths below: - 1. /var/opt/gitlab/gitlab-rails/shared - 1. /var/opt/gitlab/gitlab-rails/uploads + - /var/opt/gitlab/gitlab-rails/shared + - /var/opt/gitlab/gitlab-rails/uploads - To rename all of them: + To rename all of them: - ```sh - gitlab-ctl stop + ```sh + gitlab-ctl stop - mv /var/opt/gitlab/gitlab-rails/shared /var/opt/gitlab/gitlab-rails/shared.old - mkdir -p /var/opt/gitlab/gitlab-rails/shared + mv /var/opt/gitlab/gitlab-rails/shared /var/opt/gitlab/gitlab-rails/shared.old + mkdir -p /var/opt/gitlab/gitlab-rails/shared - mv /var/opt/gitlab/gitlab-rails/uploads /var/opt/gitlab/gitlab-rails/uploads.old - mkdir -p /var/opt/gitlab/gitlab-rails/uploads - ``` + mv /var/opt/gitlab/gitlab-rails/uploads /var/opt/gitlab/gitlab-rails/uploads.old + mkdir -p /var/opt/gitlab/gitlab-rails/uploads + ``` - Reconfigure in order to recreate the folders and make sure permissions and ownership - are correctly + Reconfigure in order to recreate the folders and make sure permissions and ownership + are correctly - ```sh - gitlab-ctl reconfigure - ``` + ```sh + gitlab-ctl reconfigure + ``` 1. Reset the Tracking Database - ```sh - gitlab-rake geo:db:reset - ``` + ```sh + gitlab-rake geo:db:reset + ``` 1. Restart previously stopped services - ```sh - gitlab-ctl start - ``` + ```sh + gitlab-ctl start + ``` ## Fixing Foreign Data Wrapper errors @@ -331,7 +331,7 @@ There are a few key points to remember: 1. The FDW settings are configured on the Geo **tracking** database. 1. The configured foreign server enables a login to the Geo -**secondary**, read-only database. + **secondary**, read-only database. By default, the Geo secondary and tracking database are running on the same host on different ports. That is, 5432 and 5431 respectively. @@ -345,108 +345,106 @@ To check the configuration: 1. Enter the database console: - ```sh - gitlab-geo-psql - ``` + ```sh + gitlab-geo-psql + ``` 1. Check whether any tables are present. If everything is working, you -should see something like this: - - ```sql - gitlabhq_geo_production=# SELECT * from information_schema.foreign_tables; - foreign_table_catalog | foreign_table_schema | foreign_table_name | foreign_server_catalog | foreign_server_n - ame - -------------------------+----------------------+-------------------------------------------------+-------------------------+----------------- - ---- - gitlabhq_geo_production | gitlab_secondary | abuse_reports | gitlabhq_geo_production | gitlab_secondary - gitlabhq_geo_production | gitlab_secondary | appearances | gitlabhq_geo_production | gitlab_secondary - gitlabhq_geo_production | gitlab_secondary | application_setting_terms | gitlabhq_geo_production | gitlab_secondary - gitlabhq_geo_production | gitlab_secondary | application_settings | gitlabhq_geo_production | gitlab_secondary - <snip> - ``` - - However, if the query returns with `0 rows`, then continue onto the next steps. + should see something like this: + + ```sql + gitlabhq_geo_production=# SELECT * from information_schema.foreign_tables; + foreign_table_catalog | foreign_table_schema | foreign_table_name | foreign_server_catalog | foreign_server_name + -------------------------+----------------------+-------------------------------------------------+-------------------------+--------------------- + gitlabhq_geo_production | gitlab_secondary | abuse_reports | gitlabhq_geo_production | gitlab_secondary + gitlabhq_geo_production | gitlab_secondary | appearances | gitlabhq_geo_production | gitlab_secondary + gitlabhq_geo_production | gitlab_secondary | application_setting_terms | gitlabhq_geo_production | gitlab_secondary + gitlabhq_geo_production | gitlab_secondary | application_settings | gitlabhq_geo_production | gitlab_secondary + <snip> + ``` + + However, if the query returns with `0 rows`, then continue onto the next steps. 1. Check that the foreign server mapping is correct via `\des+`. The results should look something like this: - ```sql - gitlabhq_geo_production=# \des+ - List of foreign servers - -[ RECORD 1 ]--------+------------------------------------------------------------ - Name | gitlab_secondary - Owner | gitlab-psql - Foreign-data wrapper | postgres_fdw - Access privileges | "gitlab-psql"=U/"gitlab-psql" + - | gitlab_geo=U/"gitlab-psql" - Type | - Version | - FDW Options | (host '0.0.0.0', port '5432', dbname 'gitlabhq_production') - Description | - ``` - - NOTE: **Note:** Pay particular attention to the host and port under - FDW options. That configuration should point to the Geo secondary - database. - - If you need to experiment with changing the host or password, the - following queries demonstrate how: - - ```sql - ALTER SERVER gitlab_secondary OPTIONS (SET host '<my_new_host>'); - ALTER SERVER gitlab_secondary OPTIONS (SET port 5432); - ``` - - If you change the host and/or port, you will also have to adjust the - following settings in `/etc/gitlab/gitlab.rb` and run `gitlab-ctl - reconfigure`: - - - `gitlab_rails['db_host']` - - `gitlab_rails['db_port']` + ```sql + gitlabhq_geo_production=# \des+ + List of foreign servers + -[ RECORD 1 ]--------+------------------------------------------------------------ + Name | gitlab_secondary + Owner | gitlab-psql + Foreign-data wrapper | postgres_fdw + Access privileges | "gitlab-psql"=U/"gitlab-psql" + + | gitlab_geo=U/"gitlab-psql" + Type | + Version | + FDW Options | (host '0.0.0.0', port '5432', dbname 'gitlabhq_production') + Description | + ``` + + NOTE: **Note:** Pay particular attention to the host and port under + FDW options. That configuration should point to the Geo secondary + database. + + If you need to experiment with changing the host or password, the + following queries demonstrate how: + + ```sql + ALTER SERVER gitlab_secondary OPTIONS (SET host '<my_new_host>'); + ALTER SERVER gitlab_secondary OPTIONS (SET port 5432); + ``` + + If you change the host and/or port, you will also have to adjust the + following settings in `/etc/gitlab/gitlab.rb` and run `gitlab-ctl + reconfigure`: + + - `gitlab_rails['db_host']` + - `gitlab_rails['db_port']` 1. Check that the user mapping is configured properly via `\deu+`: - ```sql - gitlabhq_geo_production=# \deu+ - List of user mappings - Server | User name | FDW Options - ------------------+------------+-------------------------------------------------------------------------------- - gitlab_secondary | gitlab_geo | ("user" 'gitlab', password 'YOUR-PASSWORD-HERE') - (1 row) - ``` + ```sql + gitlabhq_geo_production=# \deu+ + List of user mappings + Server | User name | FDW Options + ------------------+------------+-------------------------------------------------------------------------------- + gitlab_secondary | gitlab_geo | ("user" 'gitlab', password 'YOUR-PASSWORD-HERE') + (1 row) + ``` - Make sure the password is correct. You can test that logins work by running `psql`: + Make sure the password is correct. You can test that logins work by running `psql`: - ```sh - # Connect to the tracking database as the `gitlab_geo` user - sudo \ - -u git /opt/gitlab/embedded/bin/psql \ - -h /var/opt/gitlab/geo-postgresql \ - -p 5431 \ - -U gitlab_geo \ - -W \ - -d gitlabhq_geo_production - ``` + ```sh + # Connect to the tracking database as the `gitlab_geo` user + sudo \ + -u git /opt/gitlab/embedded/bin/psql \ + -h /var/opt/gitlab/geo-postgresql \ + -p 5431 \ + -U gitlab_geo \ + -W \ + -d gitlabhq_geo_production + ``` - If you need to correct the password, the following query shows how: + If you need to correct the password, the following query shows how: - ```sql - ALTER USER MAPPING FOR gitlab_geo SERVER gitlab_secondary OPTIONS (SET password '<my_new_password>'); - ``` + ```sql + ALTER USER MAPPING FOR gitlab_geo SERVER gitlab_secondary OPTIONS (SET password '<my_new_password>'); + ``` - If you change the user or password, you will also have to adjust the - following settings in `/etc/gitlab/gitlab.rb` and run `gitlab-ctl - reconfigure`: + If you change the user or password, you will also have to adjust the + following settings in `/etc/gitlab/gitlab.rb` and run `gitlab-ctl + reconfigure`: - - `gitlab_rails['db_username']` - - `gitlab_rails['db_password']` + - `gitlab_rails['db_username']` + - `gitlab_rails['db_password']` - If you are using [PgBouncer in front of the secondary - database](database.md#pgbouncer-support-optional), be sure to update - the following settings: + If you are using [PgBouncer in front of the secondary + database](database.md#pgbouncer-support-optional), be sure to update + the following settings: - - `geo_postgresql['fdw_external_user']` - - `geo_postgresql['fdw_external_password']` + - `geo_postgresql['fdw_external_user']` + - `geo_postgresql['fdw_external_password']` #### Manual reload of FDW schema @@ -456,34 +454,34 @@ reload of the FDW schema. To manually reload the FDW schema: 1. On the node running the Geo tracking database, enter the PostgreSQL console via the `gitlab_geo` user: - ```sh - sudo \ - -u git /opt/gitlab/embedded/bin/psql \ - -h /var/opt/gitlab/geo-postgresql \ - -p 5431 \ - -U gitlab_geo \ - -W \ - -d gitlabhq_geo_production - ``` + ```sh + sudo \ + -u git /opt/gitlab/embedded/bin/psql \ + -h /var/opt/gitlab/geo-postgresql \ + -p 5431 \ + -U gitlab_geo \ + -W \ + -d gitlabhq_geo_production + ``` - Be sure to adjust the port and hostname for your configuration. You - may be asked to enter a password. + Be sure to adjust the port and hostname for your configuration. You + may be asked to enter a password. 1. Reload the schema via: - ```sql - DROP SCHEMA IF EXISTS gitlab_secondary CASCADE; - CREATE SCHEMA gitlab_secondary; - GRANT USAGE ON FOREIGN SERVER gitlab_secondary TO gitlab_geo; - IMPORT FOREIGN SCHEMA public FROM SERVER gitlab_secondary INTO gitlab_secondary; - ``` + ```sql + DROP SCHEMA IF EXISTS gitlab_secondary CASCADE; + CREATE SCHEMA gitlab_secondary; + GRANT USAGE ON FOREIGN SERVER gitlab_secondary TO gitlab_geo; + IMPORT FOREIGN SCHEMA public FROM SERVER gitlab_secondary INTO gitlab_secondary; + ``` 1. Test that queries work: - ```sql - SELECT * from information_schema.foreign_tables; - SELECT * FROM gitlab_secondary.projects limit 1; - ``` + ```sql + SELECT * from information_schema.foreign_tables; + SELECT * FROM gitlab_secondary.projects limit 1; + ``` [database-start-replication]: database.md#step-3-initiate-the-replication-process [database-pg-replication]: database.md#postgresql-replication @@ -504,6 +502,15 @@ To resolve this, run the following command: sudo gitlab-rake geo:db:refresh_foreign_tables ``` +## Expired artifacts + +If you notice for some reason there are more artifacts on the Geo +secondary node than on the Geo primary node, you can use the rake task +to [cleanup orphan artifact files](../../../raketasks/cleanup.md#remove-orphan-artifact-files). + +On a Geo **secondary** node, this command will also clean up all Geo +registry record related to the orphan files on disk. + ## Fixing common errors This section documents common errors reported in the Admin UI and how to fix them. diff --git a/doc/administration/geo/replication/updating_the_geo_nodes.md b/doc/administration/geo/replication/updating_the_geo_nodes.md index 933a75c47d8..d56a59f4967 100644 --- a/doc/administration/geo/replication/updating_the_geo_nodes.md +++ b/doc/administration/geo/replication/updating_the_geo_nodes.md @@ -30,70 +30,70 @@ We now require this change as we use this password to enable the Foreign Data Wr the Geo Tracking Database. We are also improving security by disabling the use of **trust** authentication method. -1. **[primary]** Login to your **primary** node and run: +1. **(primary)** Login to your **primary** node and run: - ```sh - gitlab-ctl pg-password-md5 gitlab - # Enter password: <your_password_here> - # Confirm password: <your_password_here> - # fca0b89a972d69f00eb3ec98a5838484 - ``` + ```sh + gitlab-ctl pg-password-md5 gitlab + # Enter password: <your_password_here> + # Confirm password: <your_password_here> + # fca0b89a972d69f00eb3ec98a5838484 + ``` - Copy the generated hash and edit `/etc/gitlab/gitlab.rb`: + Copy the generated hash and edit `/etc/gitlab/gitlab.rb`: - ```ruby - # Fill with the hash generated by `gitlab-ctl pg-password-md5 gitlab` - postgresql['sql_user_password'] = '<md5_hash_of_your_password>' + ```ruby + # Fill with the hash generated by `gitlab-ctl pg-password-md5 gitlab` + postgresql['sql_user_password'] = '<md5_hash_of_your_password>' - # Every node that runs Unicorn or Sidekiq needs to have the database - # password specified as below. If you have a high-availability setup, this - # must be present in all application nodes. - gitlab_rails['db_password'] = '<your_password_here>' - ``` + # Every node that runs Unicorn or Sidekiq needs to have the database + # password specified as below. If you have a high-availability setup, this + # must be present in all application nodes. + gitlab_rails['db_password'] = '<your_password_here>' + ``` - Still in the configuration file, locate and remove the `trust_auth_cidr_address`: + Still in the configuration file, locate and remove the `trust_auth_cidr_address`: - ```ruby - postgresql['trust_auth_cidr_addresses'] = ['127.0.0.1/32','1.2.3.4/32'] # <- Remove this - ``` + ```ruby + postgresql['trust_auth_cidr_addresses'] = ['127.0.0.1/32','1.2.3.4/32'] # <- Remove this + ``` -1. **[primary]** Reconfigure and restart: +1. **(primary)** Reconfigure and restart: - ```sh - sudo gitlab-ctl reconfigure - sudo gitlab-ctl restart - ``` + ```sh + sudo gitlab-ctl reconfigure + sudo gitlab-ctl restart + ``` -1. **[secondary]** Login to all **secondary** nodes and edit `/etc/gitlab/gitlab.rb`: +1. **(secondary)** Login to all **secondary** nodes and edit `/etc/gitlab/gitlab.rb`: - ```ruby - # Fill with the hash generated by `gitlab-ctl pg-password-md5 gitlab` - postgresql['sql_user_password'] = '<md5_hash_of_your_password>' + ```ruby + # Fill with the hash generated by `gitlab-ctl pg-password-md5 gitlab` + postgresql['sql_user_password'] = '<md5_hash_of_your_password>' - # Every node that runs Unicorn or Sidekiq needs to have the database - # password specified as below. If you have a high-availability setup, this - # must be present in all application nodes. - gitlab_rails['db_password'] = '<your_password_here>' + # Every node that runs Unicorn or Sidekiq needs to have the database + # password specified as below. If you have a high-availability setup, this + # must be present in all application nodes. + gitlab_rails['db_password'] = '<your_password_here>' - # Enable Foreign Data Wrapper - geo_secondary['db_fdw'] = true + # Enable Foreign Data Wrapper + geo_secondary['db_fdw'] = true - # Secondary address in CIDR format, for example '5.6.7.8/32' - postgresql['md5_auth_cidr_addresses'] = ['<secondary_node_ip>/32'] - ``` + # Secondary address in CIDR format, for example '5.6.7.8/32' + postgresql['md5_auth_cidr_addresses'] = ['<secondary_node_ip>/32'] + ``` - Still in the configuration file, locate and remove the `trust_auth_cidr_address`: + Still in the configuration file, locate and remove the `trust_auth_cidr_address`: - ```ruby - postgresql['trust_auth_cidr_addresses'] = ['127.0.0.1/32','5.6.7.8/32'] # <- Remove this - ``` + ```ruby + postgresql['trust_auth_cidr_addresses'] = ['127.0.0.1/32','5.6.7.8/32'] # <- Remove this + ``` -1. **[secondary]** Reconfigure and restart: +1. **(secondary)** Reconfigure and restart: - ```sh - sudo gitlab-ctl reconfigure - sudo gitlab-ctl restart - ``` + ```sh + sudo gitlab-ctl reconfigure + sudo gitlab-ctl restart + ``` ## Upgrading to GitLab 10.5 @@ -169,11 +169,11 @@ After you've verified that HTTP/HTTPS replication is working, you should remove the now-unused SSH keys from your secondaries, as they may cause problems if the **secondary** node if ever promoted to a **primary** node: -1. **[secondary]** Login to **all** your **secondary** nodes and run: +1. **(secondary)** Login to **all** your **secondary** nodes and run: - ```ruby - sudo -u git -H rm ~git/.ssh/id_rsa ~git/.ssh/id_rsa.pub - ``` + ```ruby + sudo -u git -H rm ~git/.ssh/id_rsa ~git/.ssh/id_rsa.pub + ``` ### Hashed Storage @@ -236,12 +236,12 @@ instructions below. When in doubt, it does not hurt to do a resync. The easiest way to do this in Omnibus is the following: - 1. Make sure you have Omnibus GitLab on the **primary** server. - 1. Run `gitlab-ctl reconfigure` and `gitlab-ctl restart postgresql`. This will enable replication slots on the **primary** database. - 1. Check the steps about defining `postgresql['sql_user_password']`, `gitlab_rails['db_password']`. - 1. Make sure `postgresql['max_replication_slots']` matches the number of **secondary** Geo nodes locations. - 1. Install GitLab on the **secondary** server. - 1. Re-run the [database replication process][database-replication]. +1. Make sure you have Omnibus GitLab on the **primary** server. +1. Run `gitlab-ctl reconfigure` and `gitlab-ctl restart postgresql`. This will enable replication slots on the **primary** database. +1. Check the steps about defining `postgresql['sql_user_password']`, `gitlab_rails['db_password']`. +1. Make sure `postgresql['max_replication_slots']` matches the number of **secondary** Geo nodes locations. +1. Install GitLab on the **secondary** server. +1. Re-run the [database replication process][database-replication]. ## Special update notes for 9.0.x @@ -260,157 +260,154 @@ Make sure to follow the steps in the exact order as they appear below and pay extra attention in what node (either **primary** or **secondary**) you execute them! Each step is prepended with the relevant node for better clarity: -1. **[secondary]** Login to **all** your **secondary** nodes and stop all services: +1. **(secondary)** Login to **all** your **secondary** nodes and stop all services: - ```ruby - sudo gitlab-ctl stop - ``` + ```ruby + sudo gitlab-ctl stop + ``` -1. **[secondary]** Make a backup of the `recovery.conf` file on **all** +1. **(secondary)** Make a backup of the `recovery.conf` file on **all** **secondary** nodes to preserve PostgreSQL's credentials: - ```sh - sudo cp /var/opt/gitlab/postgresql/data/recovery.conf /var/opt/gitlab/ - ``` + ```sh + sudo cp /var/opt/gitlab/postgresql/data/recovery.conf /var/opt/gitlab/ + ``` -1. **[primary]** Update the **primary** node to GitLab 9.0 following the +1. **(primary)** Update the **primary** node to GitLab 9.0 following the [regular update docs][update]. At the end of the update, the **primary** node will be running with PostgreSQL 9.6. -1. **[primary]** To prevent a de-synchronization of the repository replication, +1. **(primary)** To prevent a de-synchronization of the repository replication, stop all services except `postgresql` as we will use it to re-initialize the **secondary** node's database: - ```sh - sudo gitlab-ctl stop - sudo gitlab-ctl start postgresql - ``` + ```sh + sudo gitlab-ctl stop + sudo gitlab-ctl start postgresql + ``` -1. **[secondary]** Run the following steps on each of the **secondary** nodes: +1. **(secondary)** Run the following steps on each of the **secondary** nodes: - 1. **[secondary]** Stop all services: + 1. **(secondary)** Stop all services: - ```sh - sudo gitlab-ctl stop - ``` + ```sh + sudo gitlab-ctl stop + ``` - 1. **[secondary]** Prevent running database migrations: + 1. **(secondary)** Prevent running database migrations: - ```sh - sudo touch /etc/gitlab/skip-auto-migrations - ``` + ```sh + sudo touch /etc/gitlab/skip-auto-migrations + ``` - 1. **[secondary]** Move the old database to another directory: + 1. **(secondary)** Move the old database to another directory: - ```sh - sudo mv /var/opt/gitlab/postgresql{,.bak} - ``` + ```sh + sudo mv /var/opt/gitlab/postgresql{,.bak} + ``` - 1. **[secondary]** Update to GitLab 9.0 following the [regular update docs][update]. - At the end of the update, the node will be running with PostgreSQL 9.6. + 1. **(secondary)** Update to GitLab 9.0 following the [regular update docs][update]. + At the end of the update, the node will be running with PostgreSQL 9.6. - 1. **[secondary]** Make sure all services are up: + 1. **(secondary)** Make sure all services are up: - ```sh - sudo gitlab-ctl start - ``` + ```sh + sudo gitlab-ctl start + ``` - 1. **[secondary]** Reconfigure GitLab: + 1. **(secondary)** Reconfigure GitLab: - ```sh - sudo gitlab-ctl reconfigure - ``` + ```sh + sudo gitlab-ctl reconfigure + ``` - 1. **[secondary]** Run the PostgreSQL upgrade command: + 1. **(secondary)** Run the PostgreSQL upgrade command: - ```sh - sudo gitlab-ctl pg-upgrade - ``` + ```sh + sudo gitlab-ctl pg-upgrade + ``` - 1. **[secondary]** See the stored credentials for the database that you will - need to re-initialize the replication: + 1. **(secondary)** See the stored credentials for the database that you will + need to re-initialize the replication: - ```sh - sudo grep -s primary_conninfo /var/opt/gitlab/recovery.conf - ``` + ```sh + sudo grep -s primary_conninfo /var/opt/gitlab/recovery.conf + ``` - 1. **[secondary]** Create the `replica.sh` script as described in the - [database configuration document][database-source-replication]. + 1. **(secondary)** Save the snippet below in a file, let's say `/tmp/replica.sh`. Modify the + embedded paths if necessary: - 1. 1. **[secondary]** Save the snippet below in a file, let's say `/tmp/replica.sh`. Modify the - embedded paths if necessary: + ``` + #!/bin/bash - ``` - #!/bin/bash + PORT="5432" + USER="gitlab_replicator" + echo --------------------------------------------------------------- + echo WARNING: Make sure this script is run from the secondary server + echo --------------------------------------------------------------- + echo + echo Enter the IP or FQDN of the primary PostgreSQL server + read HOST + echo Enter the password for $USER@$HOST + read -s PASSWORD + echo Enter the required sslmode + read SSLMODE - PORT="5432" - USER="gitlab_replicator" - echo --------------------------------------------------------------- - echo WARNING: Make sure this script is run from the secondary server - echo --------------------------------------------------------------- - echo - echo Enter the IP or FQDN of the primary PostgreSQL server - read HOST - echo Enter the password for $USER@$HOST - read -s PASSWORD - echo Enter the required sslmode - read SSLMODE + echo Stopping PostgreSQL and all GitLab services + sudo service gitlab stop + sudo service postgresql stop - echo Stopping PostgreSQL and all GitLab services - sudo service gitlab stop - sudo service postgresql stop + echo Backing up postgresql.conf + sudo -u postgres mv /var/opt/gitlab/postgresql/data/postgresql.conf /var/opt/gitlab/postgresql/ - echo Backing up postgresql.conf - sudo -u postgres mv /var/opt/gitlab/postgresql/data/postgresql.conf /var/opt/gitlab/postgresql/ + echo Cleaning up old cluster directory + sudo -u postgres rm -rf /var/opt/gitlab/postgresql/data - echo Cleaning up old cluster directory - sudo -u postgres rm -rf /var/opt/gitlab/postgresql/data + echo Starting base backup as the replicator user + echo Enter the password for $USER@$HOST + sudo -u postgres /opt/gitlab/embedded/bin/pg_basebackup -h $HOST -D /var/opt/gitlab/postgresql/data -U gitlab_replicator -v -x -P - echo Starting base backup as the replicator user - echo Enter the password for $USER@$HOST - sudo -u postgres /opt/gitlab/embedded/bin/pg_basebackup -h $HOST -D /var/opt/gitlab/postgresql/data -U gitlab_replicator -v -x -P + echo Writing recovery.conf file + sudo -u postgres bash -c "cat > /var/opt/gitlab/postgresql/data/recovery.conf <<- _EOF1_ + standby_mode = 'on' + primary_conninfo = 'host=$HOST port=$PORT user=$USER password=$PASSWORD sslmode=$SSLMODE' + _EOF1_ + " - echo Writing recovery.conf file - sudo -u postgres bash -c "cat > /var/opt/gitlab/postgresql/data/recovery.conf <<- _EOF1_ - standby_mode = 'on' - primary_conninfo = 'host=$HOST port=$PORT user=$USER password=$PASSWORD sslmode=$SSLMODE' - _EOF1_ - " + echo Restoring postgresql.conf + sudo -u postgres mv /var/opt/gitlab/postgresql/postgresql.conf /var/opt/gitlab/postgresql/data/ - echo Restoring postgresql.conf - sudo -u postgres mv /var/opt/gitlab/postgresql/postgresql.conf /var/opt/gitlab/postgresql/data/ + echo Starting PostgreSQL + sudo service postgresql start + ``` - echo Starting PostgreSQL - sudo service postgresql start - ``` + 1. **(secondary)** Run the recovery script using the credentials from the + previous step: - 1. **[secondary]** Run the recovery script using the credentials from the - previous step: + ```sh + sudo bash /tmp/replica.sh + ``` - ```sh - sudo bash /tmp/replica.sh - ``` + 1. **(secondary)** Reconfigure GitLab: - 1. **[secondary]** Reconfigure GitLab: + ```sh + sudo gitlab-ctl reconfigure + ``` - ```sh - sudo gitlab-ctl reconfigure - ``` + 1. **(secondary)** Start all services: - 1. **[secondary]** Start all services: + ```sh + sudo gitlab-ctl start + ``` - ```sh - sudo gitlab-ctl start - ``` + 1. **(secondary)** Repeat the steps for the remaining **secondary** nodes. - 1. **[secondary]** Repeat the steps for the remaining **secondary** nodes. - -1. **[primary]** After all **secondary** nodes are updated, start all services in +1. **(primary)** After all **secondary** nodes are updated, start all services in **primary** node: - ```sh - sudo gitlab-ctl start - ``` + ```sh + sudo gitlab-ctl start + ``` ## Check status after updating @@ -419,9 +416,9 @@ everything is working correctly: 1. Run the Geo raketask on all nodes, everything should be green: - ```sh - sudo gitlab-rake gitlab:geo:check - ``` + ```sh + sudo gitlab-rake gitlab:geo:check + ``` 1. Check the **primary** node's Geo dashboard for any errors. 1. Test the data replication by pushing code to the **primary** node and see if it @@ -435,9 +432,9 @@ and it is required since 10.0. 1. Run database migrations on tracking database: - ```sh - sudo gitlab-rake geo:db:migrate - ``` + ```sh + sudo gitlab-rake geo:db:migrate + ``` 1. Repeat this step for each **secondary** node. diff --git a/doc/administration/gitaly/index.md b/doc/administration/gitaly/index.md index da8f1ee1529..a3cbc4272f0 100644 --- a/doc/administration/gitaly/index.md +++ b/doc/administration/gitaly/index.md @@ -48,7 +48,7 @@ used by Omnibus and the GitLab source installation guide. Starting with GitLab 11.4, Gitaly is able to serve all Git requests without needed a shared NFS mount for Git repository data. Between 11.4 and 11.8 the exception was the -[Elastic Search indexer](https://gitlab.com/gitlab-org/gitlab-elasticsearch-indexer). +[Elasticsearch indexer](https://gitlab.com/gitlab-org/gitlab-elasticsearch-indexer). But since 11.8 the indexer uses Gitaly for data access as well. NFS can still be leveraged for redudancy on block level of the Git data. But only has to be mounted on the Gitaly server. @@ -267,7 +267,7 @@ repository from your GitLab server over HTTP. Gitaly supports TLS encryption. To be able to communicate with a Gitaly instance that listens for secure connections you will need to use `tls://` url -scheme in the `gitaly_address` of the corresponding storage entry in the gitlab configuration. +scheme in the `gitaly_address` of the corresponding storage entry in the GitLab configuration. The admin needs to bring their own certificate as we do not provide that automatically. The certificate to be used needs to be installed on all Gitaly nodes and on all client nodes that communicate with it following procedures described in [GitLab custom certificate configuration](https://docs.gitlab.com/omnibus/settings/ssl.html#install-custom-public-certificates). diff --git a/doc/administration/high_availability/README.md b/doc/administration/high_availability/README.md index 0c4f926c579..e81d2741082 100644 --- a/doc/administration/high_availability/README.md +++ b/doc/administration/high_availability/README.md @@ -171,12 +171,12 @@ are a work-in-progress representation of the work so far. Quality will be certifying this environment in FY20-Q2. The specifications may be adjusted prior to certification based on performance testing. -- 3 PostgreSQL - 4 CPU, 8GB RAM +- 3 PostgreSQL - 4 CPU, 8GB RAM per node - 1 PgBouncer - 2 CPU, 4GB RAM -- 2 Redis - 2 CPU, 8GB RAM -- 3 Consul/Sentinel - 2 CPU, 2GB RAM -- 4 Sidekiq - 4 CPU, 8GB RAM -- 5 GitLab application nodes - 20 CPU, 64GB RAM +- 2 Redis - 2 CPU, 8GB RAM per node +- 3 Consul/Sentinel - 2 CPU, 2GB RAM per node +- 4 Sidekiq - 4 CPU, 8GB RAM per node +- 5 GitLab application nodes - 20 CPU, 64GB RAM per node - 1 Gitaly - 20 CPU, 64GB RAM - 1 Monitoring node - 4 CPU, 8GB RAM diff --git a/doc/administration/high_availability/database.md b/doc/administration/high_availability/database.md index 4db53353218..20bbfdb2603 100644 --- a/doc/administration/high_availability/database.md +++ b/doc/administration/high_availability/database.md @@ -83,7 +83,7 @@ deploy the bundled PostgreSQL. plain text password. These will be necessary when configuring the GitLab application servers later. 1. [Enable monitoring](#enable-monitoring) - + Advanced configuration options are supported and can be added if needed. @@ -204,9 +204,9 @@ Few notes on the service itself: - The service runs under a system account, by default `gitlab-consul`. - If you are using a different username, you will have to specify it. We -will refer to it with `CONSUL_USERNAME`, + will refer to it with `CONSUL_USERNAME`, - There will be a database user created with read only access to the repmgr -database + database - Passwords will be stored in the following locations: - `/etc/gitlab/gitlab.rb`: hashed - `/var/opt/gitlab/pgbouncer/pg_auth`: hashed diff --git a/doc/administration/high_availability/gitaly.md b/doc/administration/high_availability/gitaly.md index 90e5f71d835..b7eaa4ce105 100644 --- a/doc/administration/high_availability/gitaly.md +++ b/doc/administration/high_availability/gitaly.md @@ -2,13 +2,13 @@ Gitaly does not yet support full high availability. However, Gitaly is quite stable and is in use on GitLab.com. Scaled and highly available GitLab environments -should consider using Gitaly on a separate node. +should consider using Gitaly on a separate node. -See the [Gitaly HA Epic](https://gitlab.com/groups/gitlab-org/-/epics/289) to -track plans and progress toward high availability support. +See the [Gitaly HA Epic](https://gitlab.com/groups/gitlab-org/-/epics/289) to +track plans and progress toward high availability support. This document is relevant for [Scaled Architecture](README.md#scalable-architecture-examples) -environments and [High Availability Architecture](README.md#high-availability-architecture-examples). +environments and [High Availability Architecture](README.md#high-availability-architecture-examples). ## Running Gitaly on its own server @@ -24,23 +24,25 @@ Continue configuration of other components by going back to: > [Introduced](https://gitlab.com/gitlab-org/omnibus-gitlab/issues/3786) in GitLab 12.0. - 1. Create/edit `/etc/gitlab/gitlab.rb` and add the following configuration: +1. Make sure to collect [`CONSUL_SERVER_NODES`](database.md#consul-information), which are the IP addresses or DNS records of the Consul server nodes, for the next step. Note they are presented as `Y.Y.Y.Y consul1.gitlab.example.com Z.Z.Z.Z` - ```ruby - # Enable service discovery for Prometheus - consul['enable'] = true - consul['monitoring_service_discovery'] = true +1. Create/edit `/etc/gitlab/gitlab.rb` and add the following configuration: - # Replace placeholders - # Y.Y.Y.Y consul1.gitlab.example.com Z.Z.Z.Z - # with the addresses of the Consul server nodes - consul['configuration'] = { - retry_join: %w(Y.Y.Y.Y consul1.gitlab.example.com Z.Z.Z.Z), - } + ```ruby + # Enable service discovery for Prometheus + consul['enable'] = true + consul['monitoring_service_discovery'] = true - # Set the network addresses that the exporters will listen on - node_exporter['listen_address'] = '0.0.0.0:9100' - gitaly['prometheus_listen_addr'] = "0.0.0.0:9236" - ``` + # Replace placeholders + # Y.Y.Y.Y consul1.gitlab.example.com Z.Z.Z.Z + # with the addresses of the Consul server nodes + consul['configuration'] = { + retry_join: %w(Y.Y.Y.Y consul1.gitlab.example.com Z.Z.Z.Z), + } - 1. Run `sudo gitlab-ctl reconfigure` to compile the configuration. + # Set the network addresses that the exporters will listen on + node_exporter['listen_address'] = '0.0.0.0:9100' + gitaly['prometheus_listen_addr'] = "0.0.0.0:9236" + ``` + +1. Run `sudo gitlab-ctl reconfigure` to compile the configuration. diff --git a/doc/administration/high_availability/gitlab.md b/doc/administration/high_availability/gitlab.md index 0e655e49922..3045be616a6 100644 --- a/doc/administration/high_availability/gitlab.md +++ b/doc/administration/high_availability/gitlab.md @@ -138,6 +138,8 @@ need some extra configuration. If you enable Monitoring, it must be enabled on **all** GitLab servers. +1. Make sure to collect [`CONSUL_SERVER_NODES`](database.md#consul-information), which are the IP addresses or DNS records of the Consul server nodes, for the next step. Note they are presented as `Y.Y.Y.Y consul1.gitlab.example.com Z.Z.Z.Z` + 1. Create/edit `/etc/gitlab/gitlab.rb` and add the following configuration: ```ruby @@ -158,12 +160,11 @@ If you enable Monitoring, it must be enabled on **all** GitLab servers. sidekiq['listen_address'] = "0.0.0.0" unicorn['listen'] = '0.0.0.0' - # Add the monitoring node's IP address to the monitoring whitelist and allow it to scrape the NGINX metrics - # Replace placeholder - # monitoring.gitlab.example.com - # with the addresses gathered for the monitoring node - gitlab_rails['monitoring_whitelist'] = ['monitoring.gitlab.example.com'] - nginx['status']['options']['allow'] = ['monitoring.gitlab.example.com'] + # Add the monitoring node's IP address to the monitoring whitelist and allow it to + # scrape the NGINX metrics. Replace placeholder `monitoring.gitlab.example.com` with + # the address and/or subnets gathered from the monitoring node(s). + gitlab_rails['monitoring_whitelist'] = ['monitoring.gitlab.example.com', '127.0.0.0/8'] + nginx['status']['options']['allow'] = ['monitoring.gitlab.example.com', '127.0.0.0/8'] ``` 1. Run `sudo gitlab-ctl reconfigure` to compile the configuration. diff --git a/doc/administration/high_availability/monitoring_node.md b/doc/administration/high_availability/monitoring_node.md index d16bf7dc0f0..ef415dde10a 100644 --- a/doc/administration/high_availability/monitoring_node.md +++ b/doc/administration/high_availability/monitoring_node.md @@ -16,6 +16,8 @@ Omnibus: package you want using **steps 1 and 2** from the GitLab downloads page. - Do not complete any other steps on the download page. +1. Make sure to collect [`CONSUL_SERVER_NODES`](database.md#consul-information), which are the IP addresses or DNS records of the Consul server nodes, for the next step. Note they are presented as `Y.Y.Y.Y consul1.gitlab.example.com Z.Z.Z.Z` + 1. Edit `/etc/gitlab/gitlab.rb` and add the contents: ```ruby diff --git a/doc/administration/high_availability/pgbouncer.md b/doc/administration/high_availability/pgbouncer.md index 762179cf756..053dae25823 100644 --- a/doc/administration/high_availability/pgbouncer.md +++ b/doc/administration/high_availability/pgbouncer.md @@ -62,6 +62,33 @@ See our [HA documentation for PostgreSQL](database.md) for information on runnin 1. At this point, your instance should connect to the database through pgbouncer. If you are having issues, see the [Troubleshooting](#troubleshooting) section +## Enable Monitoring + +> [Introduced](https://gitlab.com/gitlab-org/omnibus-gitlab/issues/3786) in GitLab 12.0. + + If you enable Monitoring, it must be enabled on **all** pgbouncer servers. + + 1. Create/edit `/etc/gitlab/gitlab.rb` and add the following configuration: + + ```ruby + # Enable service discovery for Prometheus + consul['enable'] = true + consul['monitoring_service_discovery'] = true + + # Replace placeholders + # Y.Y.Y.Y consul1.gitlab.example.com Z.Z.Z.Z + # with the addresses of the Consul server nodes + consul['configuration'] = { + retry_join: %w(Y.Y.Y.Y consul1.gitlab.example.com Z.Z.Z.Z), + } + + # Set the network addresses that the exporters will listen on + node_exporter['listen_address'] = '0.0.0.0:9100' + pgbouncer_exporter['listen_address'] = '0.0.0.0:9188' + ``` + + 1. Run `sudo gitlab-ctl reconfigure` to compile the configuration. + ### Interacting with pgbouncer #### Administrative console diff --git a/doc/administration/high_availability/redis.md b/doc/administration/high_availability/redis.md index f61a8834af3..874525dd836 100644 --- a/doc/administration/high_availability/redis.md +++ b/doc/administration/high_availability/redis.md @@ -22,10 +22,10 @@ environments including [Basic Scaling](README.md#basic-scaling) and ### Provide your own Redis instance **[CORE ONLY]** -If you want to use your own deployed Redis instance(s), -see [Provide your own Redis instance](#provide-your-own-redis-instance-core-only) -for more details. However, you can use the GitLab Omnibus package to easily -deploy the bundled Redis. +If you want to use your own deployed Redis instance(s), +see [Provide your own Redis instance](#provide-your-own-redis-instance-core-only) +for more details. However, you can use the GitLab Omnibus package to easily +deploy the bundled Redis. ### Standalone Redis using GitLab Omnibus **[CORE ONLY]** @@ -62,11 +62,11 @@ Omnibus: pgbouncer_exporter['enable'] = false gitlab_monitor['enable'] = false gitaly['enable'] = false - + redis['bind'] = '0.0.0.0' redis['port'] = '6379' redis['password'] = 'SECRET_PASSWORD_HERE' - + gitlab_rails['auto_migrate'] = false ``` @@ -74,7 +74,7 @@ Omnibus: 1. Note the Redis node's IP address or hostname, port, and Redis password. These will be necessary when configuring the GitLab application servers later. -1. [Enable Monitoring](#enable-monitoring) +1. [Enable Monitoring](#enable-monitoring) Advanced configuration options are supported and can be added if needed. @@ -91,10 +91,10 @@ environments including [Horizontal](README.md#horizontal), ### Provide your own Redis instance **[CORE ONLY]** -If you want to use your own deployed Redis instance(s), -see [Provide your own Redis instance](#provide-your-own-redis-instance-core-only) -for more details. However, you can use the GitLab Omnibus package to easily -deploy the bundled Redis. +If you want to use your own deployed Redis instance(s), +see [Provide your own Redis instance](#provide-your-own-redis-instance-core-only) +for more details. However, you can use the GitLab Omnibus package to easily +deploy the bundled Redis. ### High Availability with GitLab Omnibus **[PREMIUM ONLY]** @@ -368,7 +368,7 @@ The prerequisites for a HA Redis setup are the following: ```ruby # Specify server role as 'redis_master_role' roles ['redis_master_role'] - + # IP address pointing to a local IP that the other machines can reach to. # You can also set bind to '0.0.0.0' which listen in all interfaces. # If you really need to bind to an external accessible IP, make @@ -382,7 +382,6 @@ The prerequisites for a HA Redis setup are the following: # Set up password authentication for Redis (use the same password in all nodes). redis['password'] = 'redis-password-goes-here' ``` - 1. Only the primary GitLab application server should handle migrations. To prevent database migrations from running on upgrade, add the following @@ -394,9 +393,9 @@ The prerequisites for a HA Redis setup are the following: 1. [Reconfigure Omnibus GitLab][reconfigure] for the changes to take effect. -> Note: You can specify multiple roles like sentinel and redis as: -> roles ['redis_sentinel_role', 'redis_master_role']. Read more about high -> availability roles at https://docs.gitlab.com/omnibus/roles/ +> Note: You can specify multiple roles like sentinel and redis as: +> `roles ['redis_sentinel_role', 'redis_master_role']`. Read more about high +> availability roles at <https://docs.gitlab.com/omnibus/roles/>. ### Step 2. Configuring the slave Redis instances @@ -412,7 +411,7 @@ The prerequisites for a HA Redis setup are the following: ```ruby # Specify server role as 'redis_slave_role' roles ['redis_slave_role'] - + # IP address pointing to a local IP that the other machines can reach to. # You can also set bind to '0.0.0.0' which listen in all interfaces. # If you really need to bind to an external accessible IP, make @@ -443,9 +442,9 @@ The prerequisites for a HA Redis setup are the following: 1. [Reconfigure Omnibus GitLab][reconfigure] for the changes to take effect. 1. Go through the steps again for all the other slave nodes. -> Note: You can specify multiple roles like sentinel and redis as: -> roles ['redis_sentinel_role', 'redis_slave_role']. Read more about high -> availability roles at https://docs.gitlab.com/omnibus/roles/ +> Note: You can specify multiple roles like sentinel and redis as: +> `roles ['redis_sentinel_role', 'redis_slave_role']`. Read more about high +> availability roles at <https://docs.gitlab.com/omnibus/roles/>. --- @@ -754,28 +753,30 @@ gitlab_rails['redis_sentinels'] = [ > [Introduced](https://gitlab.com/gitlab-org/omnibus-gitlab/issues/3786) in GitLab 12.0. - If you enable Monitoring, it must be enabled on **all** Redis servers. +If you enable Monitoring, it must be enabled on **all** Redis servers. + +1. Make sure to collect [`CONSUL_SERVER_NODES`](database.md#consul-information), which are the IP addresses or DNS records of the Consul server nodes, for the next step. Note they are presented as `Y.Y.Y.Y consul1.gitlab.example.com Z.Z.Z.Z` - 1. Create/edit `/etc/gitlab/gitlab.rb` and add the following configuration: +1. Create/edit `/etc/gitlab/gitlab.rb` and add the following configuration: - ```ruby - # Enable service discovery for Prometheus - consul['enable'] = true - consul['monitoring_service_discovery'] = true + ```ruby + # Enable service discovery for Prometheus + consul['enable'] = true + consul['monitoring_service_discovery'] = true - # Replace placeholders - # Y.Y.Y.Y consul1.gitlab.example.com Z.Z.Z.Z - # with the addresses of the Consul server nodes - consul['configuration'] = { - retry_join: %w(Y.Y.Y.Y consul1.gitlab.example.com Z.Z.Z.Z), - } + # Replace placeholders + # Y.Y.Y.Y consul1.gitlab.example.com Z.Z.Z.Z + # with the addresses of the Consul server nodes + consul['configuration'] = { + retry_join: %w(Y.Y.Y.Y consul1.gitlab.example.com Z.Z.Z.Z), + } - # Set the network addresses that the exporters will listen on - node_exporter['listen_address'] = '0.0.0.0:9100' - redis_exporter['listen_address'] = '0.0.0.0:9121' - ``` + # Set the network addresses that the exporters will listen on + node_exporter['listen_address'] = '0.0.0.0:9100' + redis_exporter['listen_address'] = '0.0.0.0:9121' + ``` - 1. Run `sudo gitlab-ctl reconfigure` to compile the configuration. +1. Run `sudo gitlab-ctl reconfigure` to compile the configuration. ## Advanced configuration diff --git a/doc/administration/logs.md b/doc/administration/logs.md index 022c23d02ce..9921ffd8ea0 100644 --- a/doc/administration/logs.md +++ b/doc/administration/logs.md @@ -125,7 +125,7 @@ This file lives in `/var/log/gitlab/gitlab-rails/integrations_json.log` for Omnibus GitLab packages or in `/home/git/gitlab/log/integrations_json.log` for installations from source. -It contains information about [integrations](../user/project/integrations/project_services.md) activities such as JIRA, Asana and Irker services. It uses JSON format like the example below: +It contains information about [integrations](../user/project/integrations/project_services.md) activities such as Jira, Asana and Irker services. It uses JSON format like the example below: ``` json {"severity":"ERROR","time":"2018-09-06T14:56:20.439Z","service_class":"JiraService","project_id":8,"project_path":"h5bp/html5-boilerplate","message":"Error sending message","client_url":"http://jira.gitlap.com:8080","error":"execution expired"} diff --git a/doc/administration/monitoring/performance/grafana_configuration.md b/doc/administration/monitoring/performance/grafana_configuration.md index 187fb2f73a1..51b0d78681d 100644 --- a/doc/administration/monitoring/performance/grafana_configuration.md +++ b/doc/administration/monitoring/performance/grafana_configuration.md @@ -103,6 +103,21 @@ repository for more information on this process. [grafana-dashboards]: https://gitlab.com/gitlab-org/grafana-dashboards +## Integration with GitLab UI + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/issues/61005) in GitLab 12.1. + +If you have set up Grafana, you can enable a link to access it easily from the sidebar: + +1. Go to the admin area under **Settings > Metrics and profiling** + and expand "Metrics - Grafana". +1. Check the "Enable access to Grafana" checkbox. +1. If Grafana is enabled through Omnibus GitLab and on the same server, + leave "Grafana URL" unchanged. In any other case, enter the full URL + path of the Grafana instance. +1. Click **Save changes**. +1. The new link will be available in the admin area under **Monitoring > Metrics Dashboard**. + --- Read more on: diff --git a/doc/administration/monitoring/prometheus/gitlab_metrics.md b/doc/administration/monitoring/prometheus/gitlab_metrics.md index 84b71ae6f1c..2d9e3f7f18b 100644 --- a/doc/administration/monitoring/prometheus/gitlab_metrics.md +++ b/doc/administration/monitoring/prometheus/gitlab_metrics.md @@ -108,7 +108,7 @@ Some basic Ruby runtime metrics are available: [GC.stat]: https://ruby-doc.org/core-2.3.0/GC.html#method-c-stat -## Puma Metrics **[EXPERIMENTAL]** +## Puma Metrics **(EXPERIMENTAL)** When Puma is used instead of Unicorn, following metrics are available: diff --git a/doc/administration/operations/extra_sidekiq_processes.md b/doc/administration/operations/extra_sidekiq_processes.md index 286b99aceb5..7297507f599 100644 --- a/doc/administration/operations/extra_sidekiq_processes.md +++ b/doc/administration/operations/extra_sidekiq_processes.md @@ -1,70 +1,132 @@ # Extra Sidekiq processes **[STARTER ONLY]** -GitLab Enterprise Edition allows one to start an extra set of Sidekiq processes +NOTE: **Note:** +The information in this page applies only to Omnibus GitLab. + +GitLab Starter allows one to start an extra set of Sidekiq processes besides the default one. These processes can be used to consume a dedicated set of queues. This can be used to ensure certain queues always have dedicated workers, no matter the number of jobs that need to be processed. -## Starting extra processes via Omnibus GitLab +## Available Sidekiq queues -To enable `sidekiq-cluster`, you must apply the `sidekiq_cluster['enable'] = true` -setting `/etc/gitlab/gitlab.rb`: +For a list of the existing Sidekiq queues, check the following files: -```ruby -sidekiq_cluster['enable'] = true -``` +- [Queues for both GitLab Community and Enterprise Editions](https://gitlab.com/gitlab-org/gitlab-ee/blob/master/app/workers/all_queues.yml) +- [Queues for GitLab Enterprise Editions only](https://gitlab.com/gitlab-org/gitlab-ee/blob/master/ee/app/workers/all_queues.yml) -You will then specify how many additional processes to create via `sidekiq-cluster` -as well as which queues for them to handle. This is done via the -`sidekiq_cluster['queue_groups']` setting. This is an array whose items contain -which queues to process. Each item in the array will equate to one additional -sidekiq process. +Each entry in the above files represents a queue on which extra Sidekiq processes +can be started. -As an example, to make additional sidekiq processes that process the -`elastic_indexer` and `mailers` queues, you would apply the following: +## Starting extra processes -```ruby -sidekiq_cluster['queue_groups'] = [ - "elastic_indexer", - "mailers" -] -``` +To start extra Sidekiq processes, you must enable `sidekiq-cluster`: -To have an additional sidekiq process handle multiple queues, you simply put a -comma after the first queue name and then put the next queue name: +1. Edit `/etc/gitlab/gitlab.rb` and add: -```ruby -sidekiq_cluster['queue_groups'] = [ - "elastic_indexer,elastic_commit_indexer", - "mailers" -] -``` + ```ruby + sidekiq_cluster['enable'] = true + ``` -Keep in mind, all changes must be followed by reconfiguring your GitLab -application via `sudo gitlab-ctl reconfigure`. +1. You will then need to specify how many additional processes to create via `sidekiq-cluster` + and which queue they should handle via the `sidekiq_cluster['queue_groups']` + array setting. Each item in the array equates to one additional Sidekiq + process, and values in each item determine the queues it works on. -### Monitoring + For example, the following setting adds additional Sidekiq processes to two + queues, one to `elastic_indexer` and one to `mailers`: -Once the Sidekiq processes are added, you can visit the "Background Jobs" + ```ruby + sidekiq_cluster['queue_groups'] = [ + "elastic_indexer", + "mailers" + ] + ``` + + To have an additional Sidekiq process handle multiple queues, add multiple + queue names to its item delimited by commas. For example: + + ```ruby + sidekiq_cluster['queue_groups'] = [ + "elastic_indexer, elastic_commit_indexer", + "mailers" + ] + ``` + +1. Save the file and reconfigure GitLab for the changes to take effect: + + ```sh + sudo gitlab-ctl reconfigure + ``` + +Once the extra Sidekiq processes are added, you can visit the "Background Jobs" section under the admin area in GitLab (`/admin/background_jobs`). - + -### All queues with exceptions +## Negating settings -To have the additional sidekiq processes work on every queue EXCEPT the ones +To have the additional Sidekiq processes work on every queue **except** the ones you list: +1. After you follow the steps for [starting extra processes](#starting-extra-processes), + edit `/etc/gitlab/gitlab.rb` and add: + + ```ruby + sidekiq_cluster['negate'] = true + ``` + +1. Save the file and reconfigure GitLab for the changes to take effect: + + ```sh + sudo gitlab-ctl reconfigure + ``` + +## Ignore all GitHub import queues + +When [importing from GitHub](../../user/project/import/github.md), Sidekiq might +use all of its resources to perform those operations. To set up a separate +`sidekiq-cluster` process to ignore all GitHub import-related queues: + 1. Edit `/etc/gitlab/gitlab.rb` and add: ```ruby + sidekiq_cluster['enable'] = true sidekiq_cluster['negate'] = true + sidekiq_cluster['queue_groups'] = [ + "github_import_advance_stage", + "github_importer:github_import_import_diff_note", + "github_importer:github_import_import_issue", + "github_importer:github_import_import_note", + "github_importer:github_import_import_lfs_object", + "github_importer:github_import_import_pull_request", + "github_importer:github_import_refresh_import_jid", + "github_importer:github_import_stage_finish_import", + "github_importer:github_import_stage_import_base_data", + "github_importer:github_import_stage_import_issues_and_diff_notes", + "github_importer:github_import_stage_import_notes", + "github_importer:github_import_stage_import_lfs_objects", + "github_importer:github_import_stage_import_pull_requests", + "github_importer:github_import_stage_import_repository" + ] ``` -1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. +1. Save the file and reconfigure GitLab for the changes to take effect: + ```sh + sudo gitlab-ctl reconfigure + ``` -### Limiting concurrency +## Number of threads + +Each process defined under `sidekiq_cluster` starts with a +number of threads that equals the number of queues, plus one spare thread. +For example, a process that handles the `process_commit` and `post_receive` +queues will use three threads in total. + +## Limiting concurrency + +To limit the concurrency of the Sidekiq processes: 1. Edit `/etc/gitlab/gitlab.rb` and add: @@ -72,11 +134,22 @@ you list: sidekiq_cluster['concurrency'] = 25 ``` -1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. +1. Save the file and reconfigure GitLab for the changes to take effect: -Keep in mind, this normally would not exceed the number of CPU cores available. + ```sh + sudo gitlab-ctl reconfigure + ``` -### Modifying the check interval +For each queue group, the concurrency factor will be set to `min(number of queues, N)`. +Setting the value to 0 will disable the limit. Keep in mind this normally would +not exceed the number of CPU cores available. + +Each thread requires a Redis connection, so adding threads may +increase Redis latency and potentially cause client timeouts. See the [Sidekiq +documentation about Redis](https://github.com/mperham/sidekiq/wiki/Using-Redis) +for more details. + +## Modifying the check interval To modify the check interval for the additional Sidekiq processes: @@ -90,9 +163,14 @@ To modify the check interval for the additional Sidekiq processes: This tells the additional processes how often to check for enqueued jobs. -## Starting extra processes via command line +## Troubleshooting using the CLI -Starting extra Sidekiq processes can be done using the command +CAUTION: **Warning:** +It's recommended to use `/etc/gitlab/gitlab.rb` to configure the Sidekiq processes. +If you experience a problem, you should contact GitLab support. Use the command +line at your own risk. + +For debugging purposes, you can start extra Sidekiq processes by using the command `/opt/gitlab/embedded/service/gitlab-rails/ee/bin/sidekiq-cluster`. This command takes arguments using the following syntax: @@ -111,29 +189,29 @@ see the relevant section in the [Sidekiq style guide](../../development/sidekiq_style_guide.md#queue-namespaces). For example, say you want to start 2 extra processes: one to process the -"process_commit" queue, and one to process the "post_receive" queue. This can be +`process_commit` queue, and one to process the `post_receive` queue. This can be done as follows: ```bash /opt/gitlab/embedded/service/gitlab-rails/ee/bin/sidekiq-cluster process_commit post_receive ``` -If you instead want to start one process processing both queues you'd use the +If you instead want to start one process processing both queues, you'd use the following syntax: ```bash /opt/gitlab/embedded/service/gitlab-rails/ee/bin/sidekiq-cluster process_commit,post_receive ``` -If you want to have one Sidekiq process process the "process_commit" and -"post_receive" queues, and one process to process the "gitlab_shell" queue, +If you want to have one Sidekiq process dealing with the `process_commit` and +`post_receive` queues, and one process to process the `gitlab_shell` queue, you'd use the following: ```bash /opt/gitlab/embedded/service/gitlab-rails/ee/bin/sidekiq-cluster process_commit,post_receive gitlab_shell ``` -### Monitoring +### Monitoring the `sidekiq-cluster` command The `sidekiq-cluster` command will not terminate once it has started the desired amount of Sidekiq processes. Instead, the process will continue running and @@ -172,24 +250,24 @@ command and not the PID(s) of the started Sidekiq processes. The Rails environment can be set by passing the `--environment` flag to the `sidekiq-cluster` command, or by setting `RAILS_ENV` to a non-empty value. The -default value is "development". +default value can be found in `/opt/gitlab/etc/gitlab-rails/env/RAILS_ENV`. -### All queues with exceptions +### Using negation You're able to run all queues in `sidekiq_queues.yml` file on a single or multiple processes with exceptions using the `--negate` flag. For example, say you want to run a single process for all queues, -except "process_commit" and "post_receive". You can do so by executing: +except `process_commit` and `post_receive`: ```bash -sidekiq-cluster process_commit,post_receive --negate +/opt/gitlab/embedded/service/gitlab-rails/ee/bin/sidekiq-cluster process_commit,post_receive --negate ``` -For multiple processes of all queues (except "process_commit" and "post_receive"): +For multiple processes of all queues (except `process_commit` and `post_receive`): ```bash -sidekiq-cluster process_commit,post_receive process_commit,post_receive --negate +/opt/gitlab/embedded/service/gitlab-rails/ee/bin/sidekiq-cluster process_commit,post_receive process_commit,post_receive --negate ``` ### Limiting concurrency @@ -201,18 +279,3 @@ the `-m N` option. For example, this would cap the maximum number of threads to ```bash /opt/gitlab/embedded/service/gitlab-rails/ee/bin/sidekiq-cluster process_commit,post_receive -m 1 ``` - -For each queue group, the concurrency factor will be set to min(number of -queues, N). Setting the value to 0 will disable the limit. - -Note that each thread requires a Redis connection, so adding threads may -increase Redis latency and potentially cause client timeouts. See the [Sidekiq -documentation about Redis](https://github.com/mperham/sidekiq/wiki/Using-Redis) -for more details. - -## Number of threads - -Each process started using `sidekiq-cluster` (whether it be via command line or -via the gitlab.rb file) starts with a number of threads that equals the number -of queues, plus one spare thread. For example, a process that handles the -"process_commit" and "post_receive" queues will use 3 threads in total. |