diff options
Diffstat (limited to 'doc/administration')
90 files changed, 2153 insertions, 1151 deletions
diff --git a/doc/administration/auth/README.md b/doc/administration/auth/README.md index 69220113940..ef82c556468 100644 --- a/doc/administration/auth/README.md +++ b/doc/administration/auth/README.md @@ -23,13 +23,11 @@ providers: - [GitHub](../../integration/github.md) - [GitLab.com](../../integration/gitlab.md) - [Google OAuth](../../integration/google.md) -- [Google Workspace SSO](../../integration/google_workspace_saml.md) - [JWT](jwt.md) - [Kerberos](../../integration/kerberos.md) - [LDAP](ldap/index.md): Includes Active Directory, Apple Open Directory, Open LDAP, and 389 Server. - [Google Secure LDAP](ldap/google_secure_ldap.md) -- [Okta](okta.md) - [Salesforce](../../integration/salesforce.md) - [SAML](../../integration/saml.md) - [SAML for GitLab.com groups](../../user/group/saml_sso/index.md) **(PREMIUM SAAS)** @@ -39,3 +37,16 @@ providers: NOTE: UltraAuth has removed their software which supports OmniAuth integration. We have therefore removed all references to UltraAuth integration. + +## SaaS vs Self-Managed Comparison + +The external authentication and authorization providers may support the following capabilities. +For more information, see the links shown on this page for each external provider. + +| Capability | SaaS | Self-Managed | +|-------------------------------------------------|-----------------------------------------|------------------------------------| +| **User Provisioning** | SCIM<br>JIT Provisioning | LDAP Sync | +| **User Detail Updating** (not group management) | Not Available | LDAP Sync | +| **Authentication** | SAML at top-level group (1 provider) | LDAP (multiple providers)<br>Generic OAuth2<br>SAML (only 1 permitted per unique provider)<br>Kerberos<br>JWT<br>Smartcard<br>OmniAuth Providers (only 1 permitted per unique provider) | +| **Provider-to-GitLab Role Sync** | SAML Group Sync | LDAP Group Sync | +| **User Removal** | SCIM (remove user from top-level group) | LDAP (Blocking User from Instance) | diff --git a/doc/administration/auth/ldap/ldap-troubleshooting.md b/doc/administration/auth/ldap/ldap-troubleshooting.md index f8360e331b6..9d9c01b870f 100644 --- a/doc/administration/auth/ldap/ldap-troubleshooting.md +++ b/doc/administration/auth/ldap/ldap-troubleshooting.md @@ -668,7 +668,7 @@ adfind -h ad.example.org:636 -ssl -u "CN=GitLabSRV,CN=Users,DC=GitLab,DC=org" -u You can also retrieve a single object by **specifying** the object name or full **DN**. In this example we specify the object name only `CN=Leroy Fox`. ```shell -adfind -h ad.example.org:636 -ssl -u "CN=GitLabSRV,CN=Users,DC=GitLab,DC=org" -up Password1 -b "OU=GitLab INT,DC=GitLab,DC=org" -f (&(objectcategory=person)(CN=Leroy Fox))” +adfind -h ad.example.org:636 -ssl -u "CN=GitLabSRV,CN=Users,DC=GitLab,DC=org" -up Password1 -b "OU=GitLab INT,DC=GitLab,DC=org" -f "(&(objectcategory=person)(CN=Leroy Fox))" ``` ### Rails console diff --git a/doc/administration/auth/okta.md b/doc/administration/auth/okta.md index 0f2851890e2..88e9180b103 100644 --- a/doc/administration/auth/okta.md +++ b/doc/administration/auth/okta.md @@ -1,176 +1,8 @@ --- -type: reference -stage: Manage -group: Access -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +redirect_to: '../../integration/saml.md' --- -# Okta SSO provider +This document was moved to [another location](../../integration/saml.md). -Okta is a [Single Sign-on provider](https://www.okta.com/products/single-sign-on/) that can be used to authenticate -with GitLab. - -The following documentation enables Okta as a SAML provider. - -## Configure the Okta application - -The following guidance is based on this Okta article, on adding a [SAML Application with an Okta Developer account](https://support.okta.com/help/s/article/Why-can-t-I-add-a-SAML-Application-with-an-Okta-Developer-account?language=en_US): - -1. In the Okta admin section, make sure to select Classic UI view in the top left corner. From there, choose to **Add an App**. -1. When the app screen comes up you see another button to **Create an App** and - choose SAML 2.0 on the next screen. -1. Optionally you can add a logo - (you can choose it from <https://about.gitlab.com/press/>). You'll have to - crop and resize it. -1. Next, you'll need the to fill in the SAML general configuration. Here's an example (showing the required URLs and attribute mapping): - image. - -  - -1. The last part of the configuration is the feedback section where you can - just say you're a customer and creating an app for internal use. -1. When you have your app you'll have a few tabs on the top of the app's - profile. Click on the SAML 2.0 configuration instructions button which should - look like the following: - -  - -1. On the screen that comes up take note of the - **Identity Provider Single Sign-On URL** which you'll use for the - `idp_sso_target_url` on your GitLab configuration file. - -1. **Before you leave Okta make sure you add your user and groups if any.** - ---- - -Now that the Okta app is configured, it's time to enable it in GitLab. - -## Configure GitLab - -1. See [Initial OmniAuth Configuration](../../integration/omniauth.md#initial-omniauth-configuration) - for initial settings. - -1. To allow your users to use Okta to sign up without having to manually create - an account first, don't forget to add the following values to your - configuration: - - **For Omnibus GitLab installations** - - Edit `/etc/gitlab/gitlab.rb`: - - ```ruby - gitlab_rails['omniauth_allow_single_sign_on'] = ['saml'] - gitlab_rails['omniauth_block_auto_created_users'] = false - ``` - - --- - - **For installations from source** - - Edit `config/gitlab.yml`: - - ```yaml - allow_single_sign_on: ["saml"] - block_auto_created_users: false - ``` - -1. You can also automatically link Okta users with existing GitLab users if - their email addresses match by adding the following setting: - - **For Omnibus GitLab installations** - - Edit `/etc/gitlab/gitlab.rb`: - - ```ruby - gitlab_rails['omniauth_auto_link_saml_user'] = true - ``` - - --- - - **For installations from source** - - Edit `config/gitlab.yml`: - - ```yaml - auto_link_saml_user: true - ``` - -1. Add the provider configuration. - - >**Notes:** - > - >- Change the value for `assertion_consumer_service_url` to match the HTTPS endpoint - > of GitLab (append `users/auth/saml/callback` to the HTTPS URL of your GitLab - > installation to generate the correct value). - > - >- To get the `idp_cert_fingerprint` fingerprint, first download the - > certificate from the Okta app you registered and then run: - > `openssl x509 -in okta.cert -noout -fingerprint`. Substitute `okta.cert` - > with the location of your certificate. - > - >- Change the value of `idp_sso_target_url`, with the value of the - > **Identity Provider Single Sign-On URL** from the step when you - > configured the Okta app. - > - >- Change the value of `issuer` to the value of the **Audience Restriction** from your Okta app configuration. This will identify GitLab - > to the IdP. - > - >- Leave `name_identifier_format` as-is. - - **For Omnibus GitLab installations** - - ```ruby - gitlab_rails['omniauth_providers'] = [ - { - name: 'saml', - args: { - assertion_consumer_service_url: 'https://gitlab.example.com/users/auth/saml/callback', - idp_cert_fingerprint: '43:51:43:a1:b5:fc:8b:b7:0a:3a:a9:b1:0f:66:73:a8', - idp_sso_target_url: 'https://gitlab.oktapreview.com/app/gitlabdev773716_gitlabsaml_1/exk8odl81tBrjpD4B0h7/sso/saml', - issuer: 'https://gitlab.example.com', - name_identifier_format: 'urn:oasis:names:tc:SAML:2.0:nameid-format:transient' - }, - label: 'Okta' # optional label for SAML login button, defaults to "Saml" - } - ] - ``` - - **For installations from source** - - ```yaml - - { - name: 'saml', - args: { - assertion_consumer_service_url: 'https://gitlab.example.com/users/auth/saml/callback', - idp_cert_fingerprint: '43:51:43:a1:b5:fc:8b:b7:0a:3a:a9:b1:0f:66:73:a8', - idp_sso_target_url: 'https://gitlab.oktapreview.com/app/gitlabdev773716_gitlabsaml_1/exk8odl81tBrjpD4B0h7/sso/saml', - issuer: 'https://gitlab.example.com', - name_identifier_format: 'urn:oasis:names:tc:SAML:2.0:nameid-format:transient' - }, - label: 'Okta' # optional label for SAML login button, defaults to "Saml" - } - ``` - -1. [Reconfigure](../restart_gitlab.md#omnibus-gitlab-reconfigure) or [restart](../restart_gitlab.md#installations-from-source) GitLab for Omnibus and installations - from source respectively for the changes to take effect. - -You might want to try this out on an incognito browser window. - -## Configuring groups - -NOTE: -Make sure the groups exist and are assigned to the Okta app. - -You can take a look of the [SAML documentation](../../integration/saml.md#saml-groups) on configuring groups. - -<!-- ## Troubleshooting - -Include any troubleshooting steps that you can foresee. If you know beforehand what issues -one might have when setting this up, or when something is changed, or on upgrading, it's -important to describe those, too. Think of things that may go wrong and include them here. -This is important to minimize requests for support, and to avoid doc comments with -questions that you know someone might ask. - -Each scenario can be a third-level heading, e.g. `### Getting error message X`. -If you have none to add when creating a doc, leave this section in place -but commented out to help encourage others to add to it in the future. --> +<!-- This redirect file can be deleted after 2021-06-15>. --> +<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page --> diff --git a/doc/administration/auth/smartcard.md b/doc/administration/auth/smartcard.md index 39c47c6c495..4fe69d0160e 100644 --- a/doc/administration/auth/smartcard.md +++ b/doc/administration/auth/smartcard.md @@ -123,8 +123,8 @@ attribute. As a prerequisite, you must use an LDAP server that: NOTE: **Note** Assign a value to at least one of the following variables: - gitlab_rails['smartcard_client_certificate_required_host'] or - gitlab_rails['smartcard_client_certificate_required_port']. + `gitlab_rails['smartcard_client_certificate_required_host']` or + `gitlab_rails['smartcard_client_certificate_required_port']`. 1. Save the file and [reconfigure](../restart_gitlab.md#omnibus-gitlab-reconfigure) GitLab for the changes to take effect. diff --git a/doc/administration/clusters/kas.md b/doc/administration/clusters/kas.md new file mode 100644 index 00000000000..1b9638411de --- /dev/null +++ b/doc/administration/clusters/kas.md @@ -0,0 +1,131 @@ +--- +stage: Configure +group: Configure +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Install the Kubernetes Agent Server (KAS) **(PREMIUM SELF)** + +The Kubernetes Agent Server (KAS) is a GitLab backend service dedicated to +managing [Kubernetes Agents](../../user/clusters/agent/index.md). + +The KAS is already installed and available in GitLab.com under `wss://kas.gitlab.com`. +See [how to use GitLab.com's KAS](../../user/clusters/agent/index.md#set-up-the-kubernetes-agent-server). +This document describes how to install a KAS for GitLab self-managed instances. + +## Installation options + +As a GitLab administrator of self-managed instances, you can install KAS according to your GitLab +installation method: + +- For [Omnibus installations](#install-kas-with-omnibus). +- For [GitLab Helm Chart installations](#install-kas-with-the-gitlab-helm-chart). + +You can also opt to use an [external KAS](#use-an-external-kas-installation). + +### Install KAS with Omnibus + +For [Omnibus](https://docs.gitlab.com/omnibus/) package installations: + +1. Edit `/etc/gitlab/gitlab.rb` to enable the Kubernetes Agent Server: + + ```ruby + gitlab_kas['enable'] = true + ``` + +1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). + +To configure any additional options related to your KAS, +refer to the **Enable GitLab KAS** section of the +[`gitlab.rb.template`](https://gitlab.com/gitlab-org/omnibus-gitlab/-/blob/master/files/gitlab-config-template/gitlab.rb.template). + +### Install KAS with the GitLab Helm Chart + +For GitLab [Helm Chart](https://docs.gitlab.com/charts/) +installations, you must set `global.kas.enabled` to `true`. +For example, in a shell with `helm` and `kubectl` +installed, run: + +```shell +helm repo add gitlab https://charts.gitlab.io/ +helm repo update +helm upgrade --install gitlab gitlab/gitlab \ + --timeout 600s \ + --set global.hosts.domain=<YOUR_DOMAIN> \ + --set global.hosts.externalIP=<YOUR_IP> \ + --set certmanager-issuer.email=<YOUR_EMAIL> \ + --set global.kas.enabled=true # <-- without this, KAS will not be installed +``` + +To configure KAS, use a `gitlab.kas` sub-section in your `values.yaml` file: + +```yaml +gitlab: + kas: + # put your KAS custom options here +``` + +For details, see [how to use the GitLab-KAS chart](https://docs.gitlab.com/charts/charts/gitlab/kas/). + +### Use an external KAS installation + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/299850) in GitLab 13.10. + +Besides installing KAS with GitLab, you can opt to configure GitLab to use an external KAS. + +For GitLab instances installed through the GitLab Helm Chart, see [how to configure your external KAS](https://docs.gitlab.com/charts/charts/globals.html#external-kas). + +For GitLab instances installed through Omnibus packages: + +1. Edit `/etc/gitlab/gitlab.rb` adding the paths to your external KAS: + + ```ruby + gitlab_kas['enable'] = false + gitlab_kas['api_secret_key'] = 'Your shared secret between GitLab and KAS' + + gitlab_rails['gitlab_kas_enabled'] = true + gitlab_rails['gitlab_kas_external_url'] = 'wss://kas.gitlab.example.com' # User-facing URL for the in-cluster agentk + gitlab_rails['gitlab_kas_internal_url'] = 'grpc://kas.internal.gitlab.example.com' # Internal URL for the GitLab backend + ``` + +1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). + +## Troubleshooting + +If you face any issues with KAS, you can read the service logs +with the following command: + +```shell +kubectl logs -f -l=app=kas -n <YOUR-GITLAB-NAMESPACE> +``` + +In Omnibus GitLab, find the logs in `/var/log/gitlab/gitlab-kas/`. + +See also the [user documentation](../../user/clusters/agent/index.md#troubleshooting) +for troubleshooting problems with individual agents. + +### KAS logs - GitOps: failed to get project info + +If you get the following error message: + +```json +{"level":"warn","time":"2020-10-30T08:37:26.123Z","msg":"GitOps: failed to get project info","agent_id":4,"project_id":"root/kas-manifest001","error":"error kind: 0; status: 404"} +``` + +It means that the specified manifest project `root/kas-manifest001` +doesn't exist or the manifest project is private. To fix it, make sure the project path is correct +and its visibility is [set to public](../../public_access/public_access.md). + +### KAS logs - Configuration file not found + +If you get the following error message: + +```plaintext +time="2020-10-29T04:44:14Z" level=warning msg="Config: failed to fetch" agent_id=2 error="configuration file not found: \".gitlab/agents/test-agent/config.yaml\ +``` + +It means that the path to the configuration project is incorrect, +or the path to `config.yaml` inside the project is not valid. + +To fix this, ensure that the paths to the configuration repo and to the `config.yaml` file +are correct. diff --git a/doc/administration/environment_variables.md b/doc/administration/environment_variables.md index f886e19b13d..a168584e754 100644 --- a/doc/administration/environment_variables.md +++ b/doc/administration/environment_variables.md @@ -21,6 +21,9 @@ You can use the following environment variables to override certain values: |--------------------------------------------|---------|---------------------------------------------------------------------------------------------------------| | `DATABASE_URL` | string | The database URL; is of the form: `postgresql://localhost/blog_development`. | | `ENABLE_BOOTSNAP` | string | Enables Bootsnap for speeding up initial Rails boot (`1` to enable). | +| `EXTERNAL_VALIDATION_SERVICE_TIMEOUT` | integer | Timeout, in seconds, for an [external CI/CD pipeline validation service](external_pipeline_validation.md). Default is `5`. | +| `EXTERNAL_VALIDATION_SERVICE_URL` | string | URL to an [external CI/CD pipeline validation service](external_pipeline_validation.md). | +| `EXTERNAL_VALIDATION_SERVICE_TOKEN` | string | The `X-Gitlab-Token` for authentication with an [external CI/CD pipeline validation service](external_pipeline_validation.md). | | `GITLAB_CDN_HOST` | string | Sets the base URL for a CDN to serve static assets (for example, `//mycdnsubdomain.fictional-cdn.com`). | | `GITLAB_EMAIL_DISPLAY_NAME` | string | The name used in the **From** field in emails sent by GitLab. | | `GITLAB_EMAIL_FROM` | string | The email address used in the **From** field in emails sent by GitLab. | @@ -29,8 +32,8 @@ You can use the following environment variables to override certain values: | `GITLAB_HOST` | string | The full URL of the GitLab server (including `http://` or `https://`). | | `GITLAB_ROOT_PASSWORD` | string | Sets the password for the `root` user on installation. | | `GITLAB_SHARED_RUNNERS_REGISTRATION_TOKEN` | string | Sets the initial registration token used for runners. | -| `GITLAB_UNICORN_MEMORY_MAX` | integer | The maximum memory threshold (in bytes) for the [unicorn-worker-killer](operations/unicorn.md#unicorn-worker-killer). | -| `GITLAB_UNICORN_MEMORY_MIN` | integer | The minimum memory threshold (in bytes) for the [unicorn-worker-killer](operations/unicorn.md#unicorn-worker-killer). | +| `GITLAB_UNICORN_MEMORY_MAX` | integer | The maximum memory threshold (in bytes) for the [unicorn-worker-killer](operations/unicorn.md#unicorn-worker-killer). | +| `GITLAB_UNICORN_MEMORY_MIN` | integer | The minimum memory threshold (in bytes) for the [unicorn-worker-killer](operations/unicorn.md#unicorn-worker-killer). | | `RAILS_ENV` | string | The Rails environment; can be one of `production`, `development`, `staging`, or `test`. | | `UNSTRUCTURED_RAILS_LOG` | string | Enables the unstructured log in addition to JSON logs (defaults to `true`). | diff --git a/doc/administration/external_pipeline_validation.md b/doc/administration/external_pipeline_validation.md index 64426a60ab0..f8329b24d6c 100644 --- a/doc/administration/external_pipeline_validation.md +++ b/doc/administration/external_pipeline_validation.md @@ -7,7 +7,7 @@ type: reference, howto # External Pipeline Validation -You can use an external service for validating a pipeline before it's created. +You can use an external service to validate a pipeline before it's created. WARNING: This is an experimental feature and subject to change without notice. @@ -19,15 +19,21 @@ data as payload. GitLab then invalidates the pipeline based on the response code. If there's an error or the request times out, the pipeline is not invalidated. -Response Code Legend: +Response codes: -- `200` - Accepted -- `4xx` - Not Accepted -- Other Codes - Accepted and Logged +- `200`: Accepted +- `4XX`: Not accepted +- All other codes: accepted and logged ## Configuration -Set the `EXTERNAL_VALIDATION_SERVICE_URL` to the external service URL. +To configure external pipeline validation, add the +[`EXTERNAL_VALIDATION_SERVICE_URL` environment variable](environment_variables.md) +and set it to the external service URL. + +By default, requests to the external service time out after five seconds. To override +the default, set the `EXTERNAL_VALIDATION_SERVICE_TIMEOUT` environment variable to the +required number of seconds. ## Payload Schema @@ -38,18 +44,21 @@ Set the `EXTERNAL_VALIDATION_SERVICE_URL` to the external service URL. "project", "user", "pipeline", - "builds" + "builds", + "namespace" ], "properties" : { "project": { "type": "object", "required": [ "id", - "path" + "path", + "created_at" ], "properties": { "id": { "type": "integer" }, - "path": { "type": "string" } + "path": { "type": "string" }, + "created_at": { "type": ["string", "null"], "format": "date-time" } } }, "user": { @@ -57,12 +66,14 @@ Set the `EXTERNAL_VALIDATION_SERVICE_URL` to the external service URL. "required": [ "id", "username", - "email" + "email", + "created_at" ], "properties": { "id": { "type": "integer" }, "username": { "type": "string" }, - "email": { "type": "string" } + "email": { "type": "string" }, + "created_at": { "type": ["string", "null"], "format": "date-time" } } }, "pipeline": { @@ -103,8 +114,21 @@ Set the `EXTERNAL_VALIDATION_SERVICE_URL` to the external service URL. } } } + }, + "namespace": { + "type": "object", + "required": [ + "plan", + "trial" + ], + "properties": { + "plan": { "type": "string" }, + "trial": { "type": "boolean" } + } } - }, - "additionalProperties": false + } } ``` + +The `namespace` field is only available in [GitLab Premium](https://about.gitlab.com/pricing/) +and higher. diff --git a/doc/administration/feature_flags.md b/doc/administration/feature_flags.md index 6882797064b..9ba50cfbf2e 100644 --- a/doc/administration/feature_flags.md +++ b/doc/administration/feature_flags.md @@ -13,7 +13,7 @@ to deploy features in an early stage of development so that they can be incrementally rolled out. Before making them permanently available, features can be deployed behind -flags for a [number of reasons](https://about.gitlab.com/handbook/product-development-flow/feature-flag-lifecycle#when-to-use-feature-flags), such as: +flags for a [number of reasons](https://about.gitlab.com/handbook/product-development-flow/feature-flag-lifecycle/#when-to-use-feature-flags), such as: - To test the feature. - To get feedback from users and customers while in an early stage of the development of the feature. diff --git a/doc/administration/geo/disaster_recovery/index.md b/doc/administration/geo/disaster_recovery/index.md index 43e5dc1d224..d1ea2978202 100644 --- a/doc/administration/geo/disaster_recovery/index.md +++ b/doc/administration/geo/disaster_recovery/index.md @@ -38,7 +38,7 @@ order to avoid unnecessary data loss. WARNING: If the **primary** node goes offline, there may be data saved on the **primary** node -that has not been replicated to the **secondary** node. This data should be treated +that have not been replicated to the **secondary** node. This data should be treated as lost if you proceed. If an outage on the **primary** node happens, you should do everything possible to @@ -46,62 +46,53 @@ avoid a split-brain situation where writes can occur in two different GitLab instances, complicating recovery efforts. So to prepare for the failover, we must disable the **primary** node. -1. SSH into the **primary** node to stop and disable GitLab, if possible: +- If you have SSH access: - ```shell - sudo gitlab-ctl stop - ``` - - Prevent GitLab from starting up again if the server unexpectedly reboots: + 1. SSH into the **primary** node to stop and disable GitLab: - ```shell - sudo systemctl disable gitlab-runsvdir - ``` - - NOTE: - (**CentOS only**) In CentOS 6 or older, there is no easy way to prevent GitLab from being - started if the machine reboots isn't available (see [Omnibus GitLab issue #3058](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/3058)). - It may be safest to uninstall the GitLab package completely: - - ```shell - yum remove gitlab-ee - ``` + ```shell + sudo gitlab-ctl stop + ``` - NOTE: - (**Ubuntu 14.04 LTS**) If you are using an older version of Ubuntu - or any other distribution based on the Upstart init system, you can prevent GitLab - from starting if the machine reboots by doing the following: + 1. Prevent GitLab from starting up again if the server unexpectedly reboots: - ```shell - initctl stop gitlab-runsvvdir - echo 'manual' > /etc/init/gitlab-runsvdir.override - initctl reload-configuration - ``` + ```shell + sudo systemctl disable gitlab-runsvdir + ``` -1. If you do not have SSH access to the **primary** node, take the machine offline and - prevent it from rebooting by any means at your disposal. - Since there are many ways you may prefer to accomplish this, we will avoid a - single recommendation. You may need to: +- If you do not have SSH access to the **primary** node, take the machine offline and + prevent it from rebooting by any means at your disposal. + Since there are many ways you may prefer to accomplish this, we will avoid a + single recommendation. You may need to: - - Reconfigure the load balancers. - - Change DNS records (for example, point the primary DNS record to the - **secondary** node to stop usage of the **primary** node). - - Stop the virtual servers. - - Block traffic through a firewall. - - Revoke object storage permissions from the **primary** node. - - Physically disconnect a machine. + - Reconfigure the load balancers. + - Change DNS records (for example, point the primary DNS record to the + **secondary** node to stop usage of the **primary** node). + - Stop the virtual servers. + - Block traffic through a firewall. + - Revoke object storage permissions from the **primary** node. + - Physically disconnect a machine. -1. If you plan to [update the primary domain DNS record](#step-4-optional-updating-the-primary-domain-dns-record), - you may wish to lower the TTL now to speed up propagation. + If you plan to [update the primary domain DNS record](#step-4-optional-updating-the-primary-domain-dns-record), + you may wish to lower the TTL now to speed up propagation. ### Step 3. Promoting a **secondary** node +WARNING: +In GitLab 13.2 and 13.3, promoting a secondary node to a primary while the +secondary is paused fails. Do not pause replication before promoting a +secondary. If the node is paused, be sure to resume before promoting. +This issue has been fixed in GitLab 13.4 and later. + Note the following when promoting a secondary: - If replication was paused on the secondary node (for example as a part of upgrading, while you were running a version of GitLab earlier than 13.4), you _must_ [enable the node by using the database](../replication/troubleshooting.md#message-activerecordrecordinvalid-validation-failed-enabled-geo-primary-node-cannot-be-disabled) - before proceeding. + before proceeding. If the secondary node + [has been paused](../../geo/index.md#pausing-and-resuming-replication), the promotion + performs a point-in-time recovery to the last known state. + Data that was created on the primary while the secondary was paused will be lost. - A new **secondary** should not be added at this time. If you want to add a new **secondary**, do this after you have completed the entire process of promoting the **secondary** to the **primary**. @@ -117,62 +108,48 @@ Note the following when promoting a secondary: sudo -i ``` -1. Edit `/etc/gitlab/gitlab.rb` to reflect its new status as **primary** by - removing any lines that enabled the `geo_secondary_role`: - - Users of GitLab 13.5 or later can skip this step, due to the appropriate - roles being enabled or disabled during the promotion in the following - step. +1. If you're using GitLab 13.5 and later, skip this step. If not, edit + `/etc/gitlab/gitlab.rb` and remove any of the following lines that + might be present: ```ruby - ## In pre-11.5 documentation, the role was enabled as follows. Remove this line. geo_secondary_role['enable'] = true - - ## In 11.5+ documentation, the role was enabled as follows. Remove this line. roles ['geo_secondary_role'] ``` -1. Promote the **secondary** node to the **primary** node. - - WARNING: - In GitLab 13.2 and 13.3, promoting a secondary node to a primary while the - secondary is paused fails. Do not pause replication before promoting a - secondary. If the node is paused, be sure to resume before promoting. This - issue has been fixed in GitLab 13.4 and later. - - WARNING: - If the secondary node [has been paused](../../geo/index.md#pausing-and-resuming-replication), this performs - a point-in-time recovery to the last known state. - Data that was created on the primary while the secondary was paused will be lost. +1. Promote the **secondary** node to the **primary** node: - NOTE: - In GitLab 13.7 and earlier, if you have a data type with zero items to sync, - this command reports `ERROR - Replication is not up-to-date` even if - replication is actually up-to-date. If replication and verification output - shows that it is complete, you can add `--skip-preflight-checks` to make the - command complete promotion. This bug was fixed in GitLab 13.8 and later. + - To promote the secondary node to primary along with [preflight checks](planned_failover.md#preflight-checks): - To promote the secondary node to primary along with preflight checks: + ```shell + gitlab-ctl promote-to-primary-node + ``` - ```shell - gitlab-ctl promote-to-primary-node - ``` + - If you have already run the preflight checks separately or don't want to run them, + you can skip them with: - If you have already run the [preflight checks](planned_failover.md#preflight-checks) separately or don't want to run them, you can skip preflight checks with: + ```shell + gitlab-ctl promote-to-primary-node --skip-preflight-checks + ``` - ```shell - gitlab-ctl promote-to-primary-node --skip-preflight-checks - ``` + NOTE: + In GitLab 13.7 and earlier, if you have a data type with zero items to sync + and don't skip the preflight checks, promoting the secondary reports + `ERROR - Replication is not up-to-date` even if replication is actually + up-to-date. If replication and verification output + shows that it is complete, you can skip the preflight checks to make the + command complete promotion. This bug was fixed in GitLab 13.8 and later. - You can also promote the secondary node to primary **without any further confirmation**, even when preflight checks fail: + - To promote the secondary node to primary **without any further confirmation**, + even when preflight checks fail: - ```shell - gitlab-ctl promote-to-primary-node --force - ``` + ```shell + gitlab-ctl promote-to-primary-node --force + ``` -1. Verify you can connect to the newly promoted **primary** node using the URL used +1. Verify you can connect to the newly-promoted **primary** node using the URL used previously for the **secondary** node. -1. If successful, the **secondary** node has now been promoted to the **primary** node. +1. If successful, the **secondary** node is now promoted to the **primary** node. #### Promoting a **secondary** node with multiple servers @@ -181,17 +158,6 @@ conjunction with multiple servers, as it can only perform changes on a **secondary** with only a single machine. Instead, you must do this manually. -WARNING: -In GitLab 13.2 and 13.3, promoting a secondary node to a primary while the -secondary is paused fails. Do not pause replication before promoting a -secondary. If the node is paused, be sure to resume before promoting. This -issue has been fixed in GitLab 13.4 and later. - -WARNING: -If the secondary node [has been paused](../../geo/index.md#pausing-and-resuming-replication), this performs -a point-in-time recovery to the last known state. -Data that was created on the primary while the secondary was paused will be lost. - 1. SSH in to the database node in the **secondary** and trigger PostgreSQL to promote to read-write: @@ -201,20 +167,17 @@ Data that was created on the primary while the secondary was paused will be lost In GitLab 12.8 and earlier, see [Message: `sudo: gitlab-pg-ctl: command not found`](../replication/troubleshooting.md#message-sudo-gitlab-pg-ctl-command-not-found). -1. Edit `/etc/gitlab/gitlab.rb` on every machine in the **secondary** to - reflect its new status as **primary** by removing any lines that enabled the - `geo_secondary_role`: +1. Edit `/etc/gitlab/gitlab.rb` on every node in the **secondary** site to + reflect its new status as **primary** by removing any of the following + lines that might be present: ```ruby - ## In pre-11.5 documentation, the role was enabled as follows. Remove this line. geo_secondary_role['enable'] = true - - ## In 11.5+ documentation, the role was enabled as follows. Remove this line. roles ['geo_secondary_role'] ``` - After making these changes [Reconfigure GitLab](../../restart_gitlab.md#omnibus-gitlab-reconfigure) each - machine so the changes take effect. + After making these changes, [reconfigure GitLab](../../restart_gitlab.md#omnibus-gitlab-reconfigure) + on each machine so the changes take effect. 1. Promote the **secondary** to **primary**. SSH into a single application server and execute: @@ -223,9 +186,9 @@ Data that was created on the primary while the secondary was paused will be lost sudo gitlab-rake geo:set_secondary_as_primary ``` -1. Verify you can connect to the newly promoted **primary** using the URL used +1. Verify you can connect to the newly-promoted **primary** using the URL used previously for the **secondary**. -1. Success! The **secondary** has now been promoted to **primary**. +1. If successful, the **secondary** node is now promoted to the **primary** node. #### Promoting a **secondary** node with a Patroni standby cluster @@ -234,17 +197,6 @@ conjunction with a Patroni standby cluster, as it can only perform changes on a **secondary** with only a single machine. Instead, you must do this manually. -WARNING: -In GitLab 13.2 and 13.3, promoting a secondary node to a primary while the -secondary is paused fails. Do not pause replication before promoting a -secondary. If the node is paused, be sure to resume before promoting. This -issue has been fixed in GitLab 13.4 and later. - -WARNING: -If the secondary node [has been paused](../../geo/index.md#pausing-and-resuming-replication), this performs -a point-in-time recovery to the last known state. -Data that was created on the primary while the secondary was paused will be lost. - 1. SSH in to the Standby Leader database node in the **secondary** and trigger PostgreSQL to promote to read-write: @@ -252,13 +204,10 @@ Data that was created on the primary while the secondary was paused will be lost sudo gitlab-ctl promote-db ``` -1. Edit `/etc/gitlab/gitlab.rb` on every application and Sidekiq nodes in the secondary to reflect its new status as primary by removing any lines that enabled the `geo_secondary_role`: +1. Edit `/etc/gitlab/gitlab.rb` on every application and Sidekiq nodes in the secondary to reflect its new status as primary by removing any of the following lines that might be present: ```ruby - ## In pre-11.5 documentation, the role was enabled as follows. Remove this line. geo_secondary_role['enable'] = true - - ## In 11.5+ documentation, the role was enabled as follows. Remove this line. roles ['geo_secondary_role'] ``` @@ -280,9 +229,9 @@ Data that was created on the primary while the secondary was paused will be lost sudo gitlab-rake geo:set_secondary_as_primary ``` -1. Verify you can connect to the newly promoted **primary** using the URL used previously for the **secondary**. - -1. Success! The **secondary** has now been promoted to **primary**. +1. Verify you can connect to the newly-promoted **primary** using the URL used + previously for the **secondary**. +1. If successful, the **secondary** node is now promoted to the **primary** node. #### Promoting a **secondary** node with an external PostgreSQL database @@ -292,12 +241,12 @@ node with GitLab and the database on the same machine. As a result, a manual pro required: 1. Promote the replica database associated with the **secondary** site. This will - set the database to read-write: - - Amazon RDS - [Promoting a Read Replica to Be a Standalone DB Instance](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_ReadRepl.html#USER_ReadRepl.Promote) - - Azure Database for PostgreSQL - [Stop replication](https://docs.microsoft.com/en-us/azure/postgresql/howto-read-replicas-portal#stop-replication) - - Other external PostgreSQL databases - save the script below in you secondary node, for example - `/tmp/geo_promote.sh`, and modify the connection parameters to match your - environment. Then, execute it to promote the replica: + set the database to read-write. The instructions vary depending on where your database is hosted: + - [Amazon RDS](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_ReadRepl.html#USER_ReadRepl.Promote) + - [Azure PostgreSQL](https://docs.microsoft.com/en-us/azure/postgresql/howto-read-replicas-portal#stop-replication) + - For other external PostgreSQL databases, save the following script in you + secondary node, for example `/tmp/geo_promote.sh`, and modify the connection + parameters to match your environment. Then, execute it to promote the replica: ```shell #!/bin/bash @@ -318,14 +267,11 @@ required: ``` 1. Edit `/etc/gitlab/gitlab.rb` on every node in the **secondary** site to - reflect its new status as **primary** by removing any lines that enabled the - `geo_secondary_role`: + reflect its new status as **primary** by removing any of the following + lines that might be present: ```ruby - ## In GitLab 11.4 and earlier, remove this line. geo_secondary_role['enable'] = true - - ## In GitLab 11.5 and later, remove this line. roles ['geo_secondary_role'] ``` @@ -339,10 +285,9 @@ required: sudo gitlab-rake geo:set_secondary_as_primary ``` -1. Verify you can connect to the newly promoted **primary** site using the URL used - previously for the **secondary** site. - -1. Success! The **secondary** site has now been promoted to **primary**. +1. Verify you can connect to the newly-promoted **primary** using the URL used + previously for the **secondary**. +1. If successful, the **secondary** node is now promoted to the **primary** node. ### Step 4. (Optional) Updating the primary domain DNS record @@ -443,7 +388,7 @@ and after that you also need two extra steps. sudo -i ``` -1. Edit `/etc/gitlab/gitlab.rb` +1. Edit `/etc/gitlab/gitlab.rb`: ```ruby ## Enable a Geo Primary role (if you haven't yet) @@ -468,7 +413,7 @@ and after that you also need two extra steps. (For more details about these settings you can read [Configure the primary server](../setup/database.md#step-1-configure-the-primary-server)) 1. Save the file and reconfigure GitLab for the database listen changes and - the replication slot changes to be applied. + the replication slot changes to be applied: ```shell gitlab-ctl reconfigure @@ -501,6 +446,134 @@ Now we need to make each **secondary** node listen to changes on the new **prima to [initiate the replication process](../setup/database.md#step-3-initiate-the-replication-process) again but this time for another **primary** node. All the old replication settings will be overwritten. +## Promoting a secondary Geo cluster in GitLab Cloud Native Helm Charts + +When updating a Cloud Native Geo deployment, the process for updating any node that is external to the secondary Kubernetes cluster does not differ from the non Cloud Native approach. As such, you can always defer to [Promoting a secondary Geo node in single-secondary configurations](#promoting-a-secondary-geo-node-in-single-secondary-configurations) for more information. + +The following sections assume you are using the `gitlab` namespace. If you used a different namespace when setting up your cluster, you should also replace `--namespace gitlab` with your namespace. + +WARNING: +In GitLab 13.2 and 13.3, promoting a secondary site to a primary while the +secondary is paused fails. Do not pause replication before promoting a +secondary. If the site is paused, be sure to resume before promoting. This +issue has been fixed in GitLab 13.4 and later. + +### Step 1. Permanently disable the **primary** cluster + +WARNING: +If the **primary** site goes offline, there may be data saved on the **primary** site +that has not been replicated to the **secondary** site. This data should be treated +as lost if you proceed. + +If an outage on the **primary** site happens, you should do everything possible to +avoid a split-brain situation where writes can occur in two different GitLab +instances, complicating recovery efforts. So to prepare for the failover, you +must disable the **primary** site: + +- If you have access to the **primary** Kubernetes cluster, connect to it and disable the GitLab webservice and Sidekiq pods: + + ```shell + kubectl --namespace gitlab scale deploy gitlab-geo-webservice-default --replicas=0 + kubectl --namespace gitlab scale deploy gitlab-geo-sidekiq-all-in-1-v1 --replicas=0 + ``` + +- If you do not have access to the **primary** Kubernetes cluster, take the cluster offline and + prevent it from coming back online by any means at your disposal. + Since there are many ways you may prefer to accomplish this, we will avoid a + single recommendation. You may need to: + + - Reconfigure the load balancers. + - Change DNS records (for example, point the primary DNS record to the + **secondary** site to stop usage of the **primary** site). + - Stop the virtual servers. + - Block traffic through a firewall. + - Revoke object storage permissions from the **primary** site. + - Physically disconnect a machine. + +### Step 2. Promote all **secondary** nodes external to the cluster + +WARNING: +If the secondary site [has been paused](../../geo/index.md#pausing-and-resuming-replication), this performs +a point-in-time recovery to the last known state. +Data that was created on the primary while the secondary was paused will be lost. + +1. SSH in to the database node in the **secondary** and trigger PostgreSQL to + promote to read-write: + + ```shell + sudo gitlab-ctl promote-db + ``` + + In GitLab 12.8 and earlier, see [Message: `sudo: gitlab-pg-ctl: command not found`](../replication/troubleshooting.md#message-sudo-gitlab-pg-ctl-command-not-found). + +1. Edit `/etc/gitlab/gitlab.rb` on the database node in the **secondary** site to + reflect its new status as **primary** by removing any lines that enabled the + `geo_secondary_role`: + + NOTE: + Depending on your architecture these steps will need to be run on any GitLab node that is external to the **secondary** Kubernetes cluster. + + ```ruby + ## In pre-11.5 documentation, the role was enabled as follows. Remove this line. + geo_secondary_role['enable'] = true + + ## In 11.5+ documentation, the role was enabled as follows. Remove this line. + roles ['geo_secondary_role'] + ``` + + After making these changes, [reconfigure GitLab](../../restart_gitlab.md#omnibus-gitlab-reconfigure) on the database node. + +### Step 3. Promote the **secondary** cluster + +1. Find the task runner pod: + + ```shell + kubectl --namespace gitlab get pods -lapp=task-runner + ``` + +1. Promote the secondary: + + ```shell + kubectl --namespace gitlab exec -ti gitlab-geo-task-runner-XXX -- gitlab-rake geo:set_secondary_as_primary + ``` + +1. Update the existing cluster configuration. + + You can retrieve the existing config with Helm: + + ```shell + helm --namespace gitlab get values gitlab-geo > gitlab.yaml + ``` + + The existing config will contain a section for Geo that should resemble: + + ```yaml + geo: + enabled: true + role: secondary + nodeName: secondary.example.com + psql: + host: geo-2.db.example.com + port: 5431 + password: + secret: geo + key: geo-postgresql-password + ``` + + To promote the **secondary** cluster to a **primary** cluster, update `role: secondary` to `role: primary`. + + You can remove the entire `psql` section if the cluster will remain as a primary site, this refers to the tracking database and will be ignored whilst the cluster is acting as a primary site. + + Update the cluster with the new config: + + ```shell + helm upgrade --install --version <current Chart version> gitlab-geo gitlab/gitlab --namespace gitlab -f gitlab.yaml + ``` + +1. Verify you can connect to the newly promoted primary using the URL used previously for the secondary. + +1. Success! The secondary has now been promoted to primary. + ## Troubleshooting This section was moved to [another location](../replication/troubleshooting.md#fixing-errors-during-a-failover-or-when-promoting-a-secondary-to-a-primary-node). diff --git a/doc/administration/geo/glossary.md b/doc/administration/geo/glossary.md index 98a29c27a89..1ec552326aa 100644 --- a/doc/administration/geo/glossary.md +++ b/doc/administration/geo/glossary.md @@ -6,7 +6,7 @@ type: howto --- -# Geo Glossary +# Geo Glossary **(PREMIUM SELF)** NOTE: We are updating the Geo documentation, user interface and commands to reflect these changes. Not all pages comply with diff --git a/doc/administration/geo/index.md b/doc/administration/geo/index.md index 296500e35e2..f1884f297e8 100644 --- a/doc/administration/geo/index.md +++ b/doc/administration/geo/index.md @@ -54,7 +54,7 @@ Geo provides: ### Gitaly Cluster Geo should not be confused with [Gitaly Cluster](../gitaly/praefect.md). For more information about -the difference between Geo and Gitaly Cluster, see [Gitaly Cluster compared to Geo](../gitaly/praefect.md#gitaly-cluster-compared-to-geo). +the difference between Geo and Gitaly Cluster, see [Gitaly Cluster compared to Geo](../gitaly/index.md#gitaly-cluster-compared-to-geo). ## How it works @@ -281,7 +281,6 @@ WARNING: This list of limitations only reflects the latest version of GitLab. If you are using an older version, extra limitations may be in place. - Pushing directly to a **secondary** node redirects (for HTTP) or proxies (for SSH) the request to the **primary** node instead of [handling it directly](https://gitlab.com/gitlab-org/gitlab/-/issues/1381), except when using Git over HTTP with credentials embedded within the URI. For example, `https://user:password@secondary.tld`. -- Cloning, pulling, or pushing repositories that exist on the **primary** node but not on the **secondary** nodes where [selective synchronization](replication/configuration.md#selective-synchronization) does not include the project is not supported over SSH [but support is planned](https://gitlab.com/groups/gitlab-org/-/epics/2562). HTTP(S) is supported. - The **primary** node has to be online for OAuth login to happen. Existing sessions and Git are not affected. Support for the **secondary** node to use an OAuth provider independent from the primary is [being planned](https://gitlab.com/gitlab-org/gitlab/-/issues/208465). - The installation takes multiple manual steps that together can take about an hour depending on circumstances. We are working on improving this experience. See [Omnibus GitLab issue #2978](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/2978) for details. - Real-time updates of issues/merge requests (for example, via long polling) doesn't work on the **secondary** node. @@ -289,6 +288,7 @@ This list of limitations only reflects the latest version of GitLab. If you are - Object pools for forked project deduplication work only on the **primary** node, and are duplicated on the **secondary** node. - GitLab Runners cannot register with a **secondary** node. Support for this is [planned for the future](https://gitlab.com/gitlab-org/gitlab/-/issues/3294). - Geo **secondary** nodes can not be configured to [use high-availability configurations of PostgreSQL](https://gitlab.com/groups/gitlab-org/-/epics/2536). +- [Selective synchronization](replication/configuration.md#selective-synchronization) only limits what repositories are replicated. The entire PostgreSQL data is still replicated. Selective synchronization is not built to accomodate compliance / export control use cases. ### Limitations on replication/verification diff --git a/doc/administration/geo/replication/datatypes.md b/doc/administration/geo/replication/datatypes.md index dfb460c9f46..61f99257844 100644 --- a/doc/administration/geo/replication/datatypes.md +++ b/doc/administration/geo/replication/datatypes.md @@ -33,8 +33,9 @@ verification methods: | Git | Project wiki repository | Geo with Gitaly | Gitaly Checksum | | Git | Project designs repository | Geo with Gitaly | Gitaly Checksum | | Git | Object pools for forked project deduplication | Geo with Gitaly | _Not implemented_ | -| Git | Project Snippets | Geo with Gitaly | _Not implemented_ | -| Git | Personal Snippets | Geo with Gitaly | _Not implemented_ | +| Git | Project Snippets | Geo with Gitaly | Gitaly Checksum | +| Git | Personal Snippets | Geo with Gitaly | Gitaly Checksum | +| Git | Group wiki repository | Geo with Gitaly | _Not implemented_ | | Blobs | User uploads _(file system)_ | Geo with API | _Not implemented_ | | Blobs | User uploads _(object storage)_ | Geo with API/Managed (*2*) | _Not implemented_ | | Blobs | LFS objects _(file system)_ | Geo with API | _Not implemented_ | @@ -51,8 +52,10 @@ verification methods: | Blobs | Versioned Terraform State _(object storage)_ | Geo with API/Managed (*2*) | _Not implemented_ | | Blobs | External Merge Request Diffs _(file system)_ | Geo with API | _Not implemented_ | | Blobs | External Merge Request Diffs _(object storage)_ | Geo with API/Managed (*2*) | _Not implemented_ | +| Blobs | Pipeline artifacts _(file system)_ | Geo with API | SHA256 checksum | +| Blobs | Pipeline artifacts _(object storage)_ | Geo with API/Managed (*2*) | SHA256 checksum | -- (*1*): Redis replication can be used as part of HA with Redis sentinel. It's not used between Geo nodes. +- (*1*): Redis replication can be used as part of HA with Redis sentinel. It's not used between Geo sites. - (*2*): Object storage replication can be performed by Geo or by your object storage provider/appliance native replication feature. @@ -68,7 +71,7 @@ performance limitations when using a remote file system). Communication is done via Gitaly's own gRPC API. There are three possible ways of synchronization: -- Using regular Git clone/fetch from one Geo node to another (with special authentication). +- Using regular Git clone/fetch from one Geo site to another (with special authentication). - Using repository snapshots (for when the first method fails or repository is corrupt). - Manual trigger from the Admin Area (a combination of both of the above). @@ -82,7 +85,7 @@ They all live in the same shard and share the same base name with a `-wiki` and for Wiki and Design Repository cases. Besides that, there are snippet repositories. They can be connected to a project or to some specific user. -Both types will be synced to a secondary node. +Both types will be synced to a secondary site. ### Blobs @@ -95,7 +98,7 @@ GitLab stores files and blobs such as Issue attachments or LFS objects into eith - A Storage Appliance that exposes an Object Storage-compatible API. When using the file system store instead of Object Storage, you need to use network mounted file systems -to run GitLab when using more than one server. +to run GitLab when using more than one node. With respect to replication and verification: @@ -113,10 +116,10 @@ as well as permissions and credentials. PostgreSQL can also hold some level of cached data like HTML rendered Markdown, cached merge-requests diff (this can also be configured to be offloaded to object storage). -We use PostgreSQL's own replication functionality to replicate data from the **primary** to **secondary** nodes. +We use PostgreSQL's own replication functionality to replicate data from the **primary** to **secondary** sites. We use Redis both as a cache store and to hold persistent data for our background jobs system. Because both -use-cases has data that are exclusive to the same Geo node, we don't replicate it between nodes. +use-cases have data that are exclusive to the same Geo site, we don't replicate it between sites. Elasticsearch is an optional database, that can enable advanced searching capabilities, like improved Advanced Search in both source-code level and user generated content in Issues / Merge-Requests and discussions. Currently it's not @@ -125,7 +128,7 @@ supported in Geo. ## Limitations on replication/verification The following table lists the GitLab features along with their replication -and verification status on a **secondary** node. +and verification status on a **secondary** site. You can keep track of the progress to implement the missing items in these epics/issues: @@ -165,40 +168,40 @@ Feature.enable(:geo_package_file_replication) WARNING: Features not on this list, or with **No** in the **Replicated** column, -are not replicated on the **secondary** node. Failing over without manually +are not replicated to a **secondary** site. Failing over without manually replicating data from those features will cause the data to be **lost**. -If you wish to use those features on a **secondary** node, or to execute a failover +If you wish to use those features on a **secondary** site, or to execute a failover successfully, you must replicate their data using some other means. -| Feature | Replicated (added in GitLab version) | Verified (added in GitLab version) | Object Storage replication (see [Geo with Object Storage](object_storage.md)) | Notes | -|:---------------------------------------------------------------------------------------------------------------|:-----------------------------------------------------------------------------------|:----------------------------------------------------------|:-------------------------------------------------------------------------------------|:---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| [Application data in PostgreSQL](../../postgresql/index.md) | **Yes** (10.2) | **Yes** (10.2) | No | | -| [Project repository](../../../user/project/repository/) | **Yes** (10.2) | **Yes** (10.7) | No | | -| [Project wiki repository](../../../user/project/wiki/) | **Yes** (10.2) | **Yes** (10.7) | No | -| [Group wiki repository](../../../user/group/index.md#group-wikis) | [**Yes** (13.10)](https://gitlab.com/gitlab-org/gitlab/-/issues/208147) | No | No | Behind feature flag `geo_group_wiki_repository_replication`, enabled by default | -| [Uploads](../../uploads.md) | **Yes** (10.2) | [No](https://gitlab.com/groups/gitlab-org/-/epics/1817) | No | Verified only on transfer or manually using [Integrity Check Rake Task](../../raketasks/check.md) on both nodes and comparing the output between them. | -| [LFS objects](../../lfs/index.md) | **Yes** (10.2) | [No](https://gitlab.com/gitlab-org/gitlab/-/issues/8922) | Via Object Storage provider if supported. Native Geo support (Beta). | Verified only on transfer or manually using [Integrity Check Rake Task](../../raketasks/check.md) on both nodes and comparing the output between them. GitLab versions 11.11.x and 12.0.x are affected by [a bug that prevents any new LFS objects from replicating](https://gitlab.com/gitlab-org/gitlab/-/issues/32696). | -| [Personal snippets](../../../user/snippets.md) | **Yes** (10.2) | **Yes** (10.2) | No | | -| [Project snippets](../../../user/snippets.md) | **Yes** (10.2) | **Yes** (10.2) | No | | -| [CI job artifacts (other than Job Logs)](../../../ci/pipelines/job_artifacts.md) | **Yes** (10.4) | [No](https://gitlab.com/gitlab-org/gitlab/-/issues/8923) | Via Object Storage provider if supported. Native Geo support (Beta) . | Verified only manually using [Integrity Check Rake Task](../../raketasks/check.md) on both nodes and comparing the output between them | -| [Job logs](../../job_logs.md) | **Yes** (10.4) | [No](https://gitlab.com/gitlab-org/gitlab/-/issues/8923) | Via Object Storage provider if supported. Native Geo support (Beta). | Verified only on transfer or manually using [Integrity Check Rake Task](../../raketasks/check.md) on both nodes and comparing the output between them | -| [Object pools for forked project deduplication](../../../development/git_object_deduplication.md) | **Yes** | No | No | | -| [Container Registry](../../packages/container_registry.md) | **Yes** (12.3) | No | No | Disabled by default. See [instructions](docker_registry.md) to enable. | -| [Content in object storage (beta)](object_storage.md) | **Yes** (12.4) | [No](https://gitlab.com/gitlab-org/gitlab/-/issues/13845) | No | | -| [Project designs repository](../../../user/project/issues/design_management.md) | **Yes** (12.7) | [No](https://gitlab.com/gitlab-org/gitlab/-/issues/32467) | Via Object Storage provider if supported. Native Geo support (Beta). | | -| [Package Registry for npm](../../../user/packages/npm_registry/index.md) | **Yes** (13.2) | **Yes** (13.10) | Via Object Storage provider if supported. Native Geo support (Beta). | Behind feature flag `geo_package_file_replication`, enabled by default | -| [Package Registry for Maven](../../../user/packages/maven_repository/index.md) | **Yes** (13.2) | **Yes** (13.10) | Via Object Storage provider if supported. Native Geo support (Beta). | Behind feature flag `geo_package_file_replication`, enabled by default | -| [Package Registry for Conan](../../../user/packages/conan_repository/index.md) | **Yes** (13.2) | **Yes** (13.10) | Via Object Storage provider if supported. Native Geo support (Beta). | Behind feature flag `geo_package_file_replication`, enabled by default | -| [Package Registry for NuGet](../../../user/packages/nuget_repository/index.md) | **Yes** (13.2) | **Yes** (13.10) | Via Object Storage provider if supported. Native Geo support (Beta). | Behind feature flag `geo_package_file_replication`, enabled by default | -| [Package Registry for PyPI](../../../user/packages/pypi_repository/index.md) | **Yes** (13.2) | **Yes** (13.10) | Via Object Storage provider if supported. Native Geo support (Beta). | Behind feature flag `geo_package_file_replication`, enabled by default | -| [Package Registry for Composer](../../../user/packages/composer_repository/index.md) | **Yes** (13.2) | **Yes** (13.10) | Via Object Storage provider if supported. Native Geo support (Beta). | Behind feature flag `geo_package_file_replication`, enabled by default | -| [Package Registry for generic packages](../../../user/packages/generic_packages/index.md) | **Yes** (13.5) | **Yes** (13.10) | Via Object Storage provider if supported. Native Geo support (Beta). | Behind feature flag `geo_package_file_replication`, enabled by default | -| [Versioned Terraform State](../../terraform_state.md) | **Yes** (13.5) | No | Via Object Storage provider if supported. Native Geo support (Beta). | Behind feature flag `geo_terraform_state_version_replication`, enabled by default | -| [External merge request diffs](../../merge_request_diffs.md) | **Yes** (13.5) | No | Via Object Storage provider if supported. Native Geo support (Beta). | Behind feature flag `geo_merge_request_diff_replication`, enabled by default | -| [Versioned snippets](../../../user/snippets.md#versioned-snippets) | [**Yes** (13.7)](https://gitlab.com/groups/gitlab-org/-/epics/2809) | [No](https://gitlab.com/groups/gitlab-org/-/epics/2810) | No | | -| [Server-side Git hooks](../../server_hooks.md) | [No](https://gitlab.com/groups/gitlab-org/-/epics/1867) | No | No | | -| [Elasticsearch integration](../../../integration/elasticsearch.md) | [No](https://gitlab.com/gitlab-org/gitlab/-/issues/1186) | No | No | | -| [GitLab Pages](../../pages/index.md) | [No](https://gitlab.com/groups/gitlab-org/-/epics/589) | No | No | | -| [CI Pipeline Artifacts](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/models/ci/pipeline_artifact.rb) | [No](https://gitlab.com/gitlab-org/gitlab/-/issues/238464) | No | Via Object Storage provider if supported. Native Geo support (Beta). | Persists additional artifacts after a pipeline completes | -| [Dependency proxy images](../../../user/packages/dependency_proxy/index.md) | [No](https://gitlab.com/gitlab-org/gitlab/-/issues/259694) | No | No | Blocked on [Geo: Secondary Mimicry](https://gitlab.com/groups/gitlab-org/-/epics/1528). Note that replication of this cache is not needed for Disaster Recovery purposes because it can be recreated from external sources. | -| [Vulnerability Export](../../../user/application_security/vulnerability_report/#export-vulnerability-details) | [Not planned](https://gitlab.com/groups/gitlab-org/-/epics/3111) | No | Via Object Storage provider if supported. Native Geo support (Beta). | Not planned because they are ephemeral and sensitive. They can be regenerated on demand. | +|Feature | Replicated (added in GitLab version) | Verified (added in GitLab version) | Object Storage replication (see [Geo with Object Storage](object_storage.md)) | Notes | +|:--------------------------------------------------------------------------------------------------------------|:------------------------------------------------------------------------|:------------------------------------------------------------------------|:------------------------------------------------------------------------------|:------| +|[Application data in PostgreSQL](../../postgresql/index.md) | **Yes** (10.2) | **Yes** (10.2) | No | | +|[Project repository](../../../user/project/repository/) | **Yes** (10.2) | **Yes** (10.7) | No | | +|[Project wiki repository](../../../user/project/wiki/) | **Yes** (10.2) | **Yes** (10.7) | No | | +|[Group wiki repository](../../../user/project/wiki/index.md#group-wikis) | [**Yes** (13.10)](https://gitlab.com/gitlab-org/gitlab/-/issues/208147) | No | No | Behind feature flag `geo_group_wiki_repository_replication`, enabled by default. | +|[Uploads](../../uploads.md) | **Yes** (10.2) | [No](https://gitlab.com/groups/gitlab-org/-/epics/1817) | No | Verified only on transfer or manually using [Integrity Check Rake Task](../../raketasks/check.md) on both sites and comparing the output between them. | +|[LFS objects](../../lfs/index.md) | **Yes** (10.2) | [No](https://gitlab.com/gitlab-org/gitlab/-/issues/8922) | Via Object Storage provider if supported. Native Geo support (Beta). | Verified only on transfer or manually using [Integrity Check Rake Task](../../raketasks/check.md) on both sites and comparing the output between them. GitLab versions 11.11.x and 12.0.x are affected by [a bug that prevents any new LFS objects from replicating](https://gitlab.com/gitlab-org/gitlab/-/issues/32696). | +|[Personal snippets](../../../user/snippets.md) | **Yes** (10.2) | **Yes** (10.2) | No | | +|[Project snippets](../../../user/snippets.md) | **Yes** (10.2) | **Yes** (10.2) | No | | +|[CI job artifacts (other than Job Logs)](../../../ci/pipelines/job_artifacts.md) | **Yes** (10.4) | [No](https://gitlab.com/gitlab-org/gitlab/-/issues/8923) | Via Object Storage provider if supported. Native Geo support (Beta). | Verified only manually using [Integrity Check Rake Task](../../raketasks/check.md) on both sites and comparing the output between them. | +|[CI Pipeline Artifacts](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/models/ci/pipeline_artifact.rb) | [**Yes** (13.11)](https://gitlab.com/gitlab-org/gitlab/-/issues/238464) | [**Yes** (13.11)](https://gitlab.com/gitlab-org/gitlab/-/issues/238464) | Via Object Storage provider if supported. Native Geo support (Beta). | Persists additional artifacts after a pipeline completes | +|[Job logs](../../job_logs.md) | **Yes** (10.4) | [No](https://gitlab.com/gitlab-org/gitlab/-/issues/8923) | Via Object Storage provider if supported. Native Geo support (Beta). | Verified only on transfer or manually using [Integrity Check Rake Task](../../raketasks/check.md) on both sites and comparing the output between them. | +|[Object pools for forked project deduplication](../../../development/git_object_deduplication.md) | **Yes** | No | No | | +|[Container Registry](../../packages/container_registry.md) | **Yes** (12.3) | No | No | Disabled by default. See [instructions](docker_registry.md) to enable. | +|[Content in object storage (beta)](object_storage.md) | **Yes** (12.4) | [No](https://gitlab.com/gitlab-org/gitlab/-/issues/13845) | No | | +|[Project designs repository](../../../user/project/issues/design_management.md) | **Yes** (12.7) | [No](https://gitlab.com/gitlab-org/gitlab/-/issues/32467) | Via Object Storage provider if supported. Native Geo support (Beta). | | +|[Package Registry for npm](../../../user/packages/npm_registry/index.md) | **Yes** (13.2) | **Yes** (13.10) | Via Object Storage provider if supported. Native Geo support (Beta). | Behind feature flag `geo_package_file_replication`, enabled by default. | +|[Package Registry for Maven](../../../user/packages/maven_repository/index.md) | **Yes** (13.2) | **Yes** (13.10) | Via Object Storage provider if supported. Native Geo support (Beta). | Behind feature flag `geo_package_file_replication`, enabled by default. | +|[Package Registry for Conan](../../../user/packages/conan_repository/index.md) | **Yes** (13.2) | **Yes** (13.10) | Via Object Storage provider if supported. Native Geo support (Beta). | Behind feature flag `geo_package_file_replication`, enabled by default. | +|[Package Registry for NuGet](../../../user/packages/nuget_repository/index.md) | **Yes** (13.2) | **Yes** (13.10) | Via Object Storage provider if supported. Native Geo support (Beta). | Behind feature flag `geo_package_file_replication`, enabled by default. | +|[Package Registry for PyPI](../../../user/packages/pypi_repository/index.md) | **Yes** (13.2) | **Yes** (13.10) | Via Object Storage provider if supported. Native Geo support (Beta). | Behind feature flag `geo_package_file_replication`, enabled by default. | +|[Package Registry for Composer](../../../user/packages/composer_repository/index.md) | **Yes** (13.2) | **Yes** (13.10) | Via Object Storage provider if supported. Native Geo support (Beta). | Behind feature flag `geo_package_file_replication`, enabled by default. | +|[Package Registry for generic packages](../../../user/packages/generic_packages/index.md) | **Yes** (13.5) | **Yes** (13.10) | Via Object Storage provider if supported. Native Geo support (Beta). | Behind feature flag `geo_package_file_replication`, enabled by default. | +|[Versioned Terraform State](../../terraform_state.md) | **Yes** (13.5) | No | Via Object Storage provider if supported. Native Geo support (Beta). | Behind feature flag `geo_terraform_state_version_replication`, enabled by default. | +|[External merge request diffs](../../merge_request_diffs.md) | **Yes** (13.5) | No | Via Object Storage provider if supported. Native Geo support (Beta). | Behind feature flag `geo_merge_request_diff_replication`, enabled by default. | +|[Versioned snippets](../../../user/snippets.md#versioned-snippets) | [**Yes** (13.7)](https://gitlab.com/groups/gitlab-org/-/epics/2809) | [No](https://gitlab.com/groups/gitlab-org/-/epics/2810) | No | | +|[Server-side Git hooks](../../server_hooks.md) | [No](https://gitlab.com/groups/gitlab-org/-/epics/1867) | No | No | | +|[Elasticsearch integration](../../../integration/elasticsearch.md) | [No](https://gitlab.com/gitlab-org/gitlab/-/issues/1186) | No | No | | +|[GitLab Pages](../../pages/index.md) | [No](https://gitlab.com/groups/gitlab-org/-/epics/589) | No | No | | +|[Dependency proxy images](../../../user/packages/dependency_proxy/index.md) | [No](https://gitlab.com/gitlab-org/gitlab/-/issues/259694) | No | No | Blocked on [Geo: Secondary Mimicry](https://gitlab.com/groups/gitlab-org/-/epics/1528). Note that replication of this cache is not needed for Disaster Recovery purposes because it can be recreated from external sources. | +|[Vulnerability Export](../../../user/application_security/vulnerability_report/#export-vulnerability-details) | [Not planned](https://gitlab.com/groups/gitlab-org/-/epics/3111) | No | Via Object Storage provider if supported. Native Geo support (Beta). | Not planned because they are ephemeral and sensitive. They can be regenerated on demand. | diff --git a/doc/administration/geo/replication/disable_geo.md b/doc/administration/geo/replication/disable_geo.md index a26adcefef5..c71cf80d0c1 100644 --- a/doc/administration/geo/replication/disable_geo.md +++ b/doc/administration/geo/replication/disable_geo.md @@ -12,26 +12,26 @@ situation and you want to disable Geo momentarily, you can use these instruction Geo setup. There should be no functional difference between disabling Geo and having an active Geo setup with -no secondary Geo nodes if you remove them correctly. +no secondary Geo sites if you remove them correctly. To disable Geo, follow these steps: -1. [Remove all secondary Geo nodes](#remove-all-secondary-geo-nodes). -1. [Remove the primary node from the UI](#remove-the-primary-node-from-the-ui). +1. [Remove all secondary Geo sites](#remove-all-secondary-geo-sites). +1. [Remove the primary site from the UI](#remove-the-primary-site-from-the-ui). 1. [Remove secondary replication slots](#remove-secondary-replication-slots). 1. [Remove Geo-related configuration](#remove-geo-related-configuration). 1. [(Optional) Revert PostgreSQL settings to use a password and listen on an IP](#optional-revert-postgresql-settings-to-use-a-password-and-listen-on-an-ip). -## Remove all secondary Geo nodes +## Remove all secondary Geo sites -To disable Geo, you need to first remove all your secondary Geo nodes, which means replication will not happen -anymore on these nodes. You can follow our docs to [remove your secondary Geo nodes](remove_geo_node.md). +To disable Geo, you need to first remove all your secondary Geo sites, which means replication will not happen +anymore on these sites. You can follow our docs to [remove your secondary Geo sites](remove_geo_site.md). -If the current node that you want to keep using is a secondary node, you need to first promote it to primary. -You can use our steps on [how to promote a secondary node](../disaster_recovery/#step-3-promoting-a-secondary-node) +If the current site that you want to keep using is a secondary site, you need to first promote it to primary. +You can use our steps on [how to promote a secondary site](../disaster_recovery/#step-3-promoting-a-secondary-node) to do that. -## Remove the primary node from the UI +## Remove the primary site from the UI 1. Go to **Admin Area > Geo** (`/admin/geo/nodes`). 1. Click the **Remove** button for the **primary** node. @@ -59,7 +59,7 @@ Geo node in a PostgreSQL console (`sudo gitlab-psql`): ## Remove Geo-related configuration -1. SSH into your primary Geo node and log in as root: +1. For each node on your primary Geo site, SSH into the node and log in as root: ```shell sudo -i diff --git a/doc/administration/geo/replication/docker_registry.md b/doc/administration/geo/replication/docker_registry.md index 1669abbc52a..ea73614511f 100644 --- a/doc/administration/geo/replication/docker_registry.md +++ b/doc/administration/geo/replication/docker_registry.md @@ -5,16 +5,16 @@ info: To determine the technical writer assigned to the Stage/Group associated w type: howto --- -# Docker Registry for a secondary node **(PREMIUM SELF)** +# Docker Registry for a secondary site **(PREMIUM SELF)** You can set up a [Docker Registry](https://docs.docker.com/registry/) on your -**secondary** Geo node that mirrors the one on the **primary** Geo node. +**secondary** Geo site that mirrors the one on the **primary** Geo site. ## Storage support Docker Registry currently supports a few types of storage. If you choose a distributed storage (`azure`, `gcs`, `s3`, `swift`, or `oss`) for your Docker -Registry on the **primary** node, you can use the same storage for a **secondary** +Registry on the **primary** site, you can use the same storage for a **secondary** Docker Registry as well. For more information, read the [Load balancing considerations](https://docs.docker.com/registry/deploying/#load-balancing-considerations) when deploying the Registry, and how to set up the storage driver for the GitLab @@ -24,22 +24,22 @@ integrated [Container Registry](../../packages/container_registry.md#use-object- You can enable a storage-agnostic replication so it can be used for cloud or local storage. Whenever a new image is pushed to the -**primary** node, each **secondary** node will pull it to its own container +**primary** site, each **secondary** site will pull it to its own container repository. To configure Docker Registry replication: -1. Configure the [**primary** node](#configure-primary-node). -1. Configure the [**secondary** node](#configure-secondary-node). +1. Configure the [**primary** site](#configure-primary-site). +1. Configure the [**secondary** site](#configure-secondary-site). 1. Verify Docker Registry [replication](#verify-replication). -### Configure **primary** node +### Configure **primary** site Make sure that you have Container Registry set up and working on -the **primary** node before following the next steps. +the **primary** site before following the next steps. We need to make Docker Registry send notification events to the -**primary** node. +**primary** site. 1. SSH into your GitLab **primary** server and login as root: @@ -85,27 +85,29 @@ We need to make Docker Registry send notification events to the gitlab-ctl reconfigure ``` -### Configure **secondary** node +### Configure **secondary** site Make sure you have Container Registry set up and working on -the **secondary** node before following the next steps. +the **secondary** site before following the next steps. -The following steps should be done on each **secondary** node you're +The following steps should be done on each **secondary** site you're expecting to see the Docker images replicated. -Because we need to allow the **secondary** node to communicate securely with -the **primary** node Container Registry, we need to have a single key -pair for all the nodes. The **secondary** node will use this key to +Because we need to allow the **secondary** site to communicate securely with +the **primary** site Container Registry, we need to have a single key +pair for all the sites. The **secondary** site will use this key to generate a short-lived JWT that is pull-only-capable to access the -**primary** node Container Registry. +**primary** site Container Registry. -1. SSH into the **secondary** node and login as the `root` user: +For each application and Sidekiq node on the **secondary** site: + +1. SSH into the node and login as the `root` user: ```shell sudo -i ``` -1. Copy `/var/opt/gitlab/gitlab-rails/etc/gitlab-registry.key` from the **primary** to the **secondary** node. +1. Copy `/var/opt/gitlab/gitlab-rails/etc/gitlab-registry.key` from the **primary** to the node. 1. Edit `/etc/gitlab/gitlab.rb`: @@ -114,7 +116,7 @@ generate a short-lived JWT that is pull-only-capable to access the gitlab_rails['geo_registry_replication_primary_api_url'] = 'https://primary.example.com:5050/' # Primary registry address, it will be used by the secondary node to directly communicate to primary registry ``` -1. Reconfigure the **secondary** node for the change to take effect: +1. Reconfigure the node for the change to take effect: ```shell gitlab-ctl reconfigure @@ -123,6 +125,6 @@ generate a short-lived JWT that is pull-only-capable to access the ### Verify replication To verify Container Registry replication is working, go to **Admin Area > Geo** -(`/admin/geo/nodes`) on the **secondary** node. +(`/admin/geo/nodes`) on the **secondary** site. The initial replication, or "backfill", will probably still be in progress. -You can monitor the synchronization process on each Geo node from the **primary** node's **Geo Nodes** dashboard in your browser. +You can monitor the synchronization process on each Geo site from the **primary** site's **Geo Nodes** dashboard in your browser. diff --git a/doc/administration/geo/replication/geo_validation_tests.md b/doc/administration/geo/replication/geo_validation_tests.md index f050d3e708c..06fd3cd70be 100644 --- a/doc/administration/geo/replication/geo_validation_tests.md +++ b/doc/administration/geo/replication/geo_validation_tests.md @@ -148,13 +148,13 @@ The following are PostgreSQL upgrade validation tests we performed. [PostgreSQL 11 upgrade procedure for Geo installations](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/4975): - Description: Prior to making PostgreSQL 11 the default version of PostgreSQL in GitLab 12.10, we - tested upgrading to PostgreSQL 11 in Geo deployments on GitLab 12.9. + tested upgrading to PostgreSQL 11 in Geo deployments in GitLab 12.9. - Outcome: Partially successful. Issues were discovered in multi-node configurations with a separate tracking database and concerns were raised about allowing automatic upgrades when Geo enabled. - Follow up issues: - [`replicate-geo-database` incorrectly tries to back up repositories](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/5241). - [`pg-upgrade` fails to upgrade a standalone Geo tracking database](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/5242). - - [`revert-pg-upgrade` fails to downgrade the PostgreSQL data of a Geo secondary’s standalone tracking database](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/5243). + - [`revert-pg-upgrade` fails to downgrade the PostgreSQL data of a Geo secondary's standalone tracking database](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/5243). - [Timeout error on Geo secondary read-replica near the end of `gitlab-ctl pg-upgrade`](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/5235). [Verify Geo installation with PostgreSQL 11](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/4971): diff --git a/doc/administration/geo/replication/remove_geo_node.md b/doc/administration/geo/replication/remove_geo_node.md index 06cf5375f0d..09ea84b6c4b 100644 --- a/doc/administration/geo/replication/remove_geo_node.md +++ b/doc/administration/geo/replication/remove_geo_node.md @@ -1,58 +1,8 @@ --- -stage: Enablement -group: Geo -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments -type: howto +redirect_to: '../../geo/replication/remove_geo_site.md' --- -# Removing secondary Geo nodes **(PREMIUM SELF)** +This document was moved to [another location](../../geo/replication/remove_geo_site.md). -**Secondary** nodes can be removed from the Geo cluster using the Geo administration page of the **primary** node. To remove a **secondary** node: - -1. Go to **Admin Area > Geo** (`/admin/geo/nodes`). -1. Click the **Remove** button for the **secondary** node you want to remove. -1. Confirm by clicking **Remove** when the prompt appears. - -Once removed from the Geo administration page, you must stop and uninstall the **secondary** node: - -1. On the **secondary** node, stop GitLab: - - ```shell - sudo gitlab-ctl stop - ``` - -1. On the **secondary** node, uninstall GitLab: - - ```shell - # Stop gitlab and remove its supervision process - sudo gitlab-ctl uninstall - - # Debian/Ubuntu - sudo dpkg --remove gitlab-ee - - # Redhat/Centos - sudo rpm --erase gitlab-ee - ``` - -Once GitLab has been uninstalled from the **secondary** node, the replication slot must be dropped from the **primary** node's database as follows: - -1. On the **primary** node, start a PostgreSQL console session: - - ```shell - sudo gitlab-psql - ``` - - NOTE: - Using `gitlab-rails dbconsole` will not work, because managing replication slots requires superuser permissions. - -1. Find the name of the relevant replication slot. This is the slot that is specified with `--slot-name` when running the replicate command: `gitlab-ctl replicate-geo-database`. - - ```sql - SELECT * FROM pg_replication_slots; - ``` - -1. Remove the replication slot for the **secondary** node: - - ```sql - SELECT pg_drop_replication_slot('<name_of_slot>'); - ``` +<!-- This redirect file can be deleted after 2022-04-01 --> +<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page --> diff --git a/doc/administration/geo/replication/remove_geo_site.md b/doc/administration/geo/replication/remove_geo_site.md new file mode 100644 index 00000000000..a42a4c4eb47 --- /dev/null +++ b/doc/administration/geo/replication/remove_geo_site.md @@ -0,0 +1,58 @@ +--- +stage: Enablement +group: Geo +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +type: howto +--- + +# Removing secondary Geo sites **(PREMIUM SELF)** + +**Secondary** sites can be removed from the Geo cluster using the Geo administration page of the **primary** site. To remove a **secondary** site: + +1. Go to **Admin Area > Geo** (`/admin/geo/nodes`). +1. Select the **Remove** button for the **secondary** site you want to remove. +1. Confirm by selecting **Remove** when the prompt appears. + +Once removed from the Geo administration page, you must stop and uninstall the **secondary** site. For each node on your secondary Geo site: + +1. Stop GitLab: + + ```shell + sudo gitlab-ctl stop + ``` + +1. Uninstall GitLab: + + ```shell + # Stop gitlab and remove its supervision process + sudo gitlab-ctl uninstall + + # Debian/Ubuntu + sudo dpkg --remove gitlab-ee + + # Redhat/Centos + sudo rpm --erase gitlab-ee + ``` + +Once GitLab has been uninstalled from each node on the **secondary** site, the replication slot must be dropped from the **primary** site's database as follows: + +1. On the **primary** site's database node, start a PostgreSQL console session: + + ```shell + sudo gitlab-psql + ``` + + NOTE: + Using `gitlab-rails dbconsole` will not work, because managing replication slots requires superuser permissions. + +1. Find the name of the relevant replication slot. This is the slot that is specified with `--slot-name` when running the replicate command: `gitlab-ctl replicate-geo-database`. + + ```sql + SELECT * FROM pg_replication_slots; + ``` + +1. Remove the replication slot for the **secondary** site: + + ```sql + SELECT pg_drop_replication_slot('<name_of_slot>'); + ``` diff --git a/doc/administration/geo/replication/security_review.md b/doc/administration/geo/replication/security_review.md index c31881910bc..abb84b95623 100644 --- a/doc/administration/geo/replication/security_review.md +++ b/doc/administration/geo/replication/security_review.md @@ -34,7 +34,7 @@ from [owasp.org](https://owasp.org/). ### How can the data be classified into categories according to its sensitivity? - The GitLab model of sensitivity is centered around public vs. internal vs. - private projects. Geo replicates them all indiscriminately. “Selective sync” + private projects. Geo replicates them all indiscriminately. "Selective sync" exists for files and repositories (but not database content), which would permit only less-sensitive projects to be replicated to a **secondary** node if desired. - See also: [GitLab data classification policy](https://about.gitlab.com/handbook/engineering/security/data-classification-standard.html). @@ -81,7 +81,7 @@ from [owasp.org](https://owasp.org/). considered an administrator with super-user privileges. - See also: [more granular access control](https://gitlab.com/gitlab-org/gitlab/-/issues/18242) (not Geo-specific). -- Much of Geo’s integration (database replication, for instance) must be +- Much of Geo's integration (database replication, for instance) must be configured with the application, typically by system administrators. ### What administrative capabilities does the application offer? @@ -165,7 +165,7 @@ from [owasp.org](https://owasp.org/). ### What aspects of the product may or may not be hosted via the cloud computing model? -- GitLab is “cloud native” and this applies to Geo as much as to the rest of the +- GitLab is "cloud native" and this applies to Geo as much as to the rest of the product. Deployment in clouds is a common and supported scenario. ## If applicable, what approach(es) to cloud computing will be taken (Managed Hosting versus "Pure" Cloud, a "full machine" approach such as AWS-EC2 versus a "hosted database" approach such as AWS-RDS and Azure, etc)? @@ -223,7 +223,7 @@ from [owasp.org](https://owasp.org/). ### What data input validation requirements have been defined? -- **Secondary** nodes must have a faithful replication of the **primary** node’s data. +- **Secondary** nodes must have a faithful replication of the **primary** node's data. ### What data does the application store and how? @@ -235,7 +235,7 @@ from [owasp.org](https://owasp.org/). rest. A subset of database columns are encrypted at rest using the `db_otp_key`. - A static secret shared across all hosts in a GitLab deployment. - In transit, data should be encrypted, although the application does permit - communication to proceed unencrypted. The two main transits are the **secondary** node’s + communication to proceed unencrypted. The two main transits are the **secondary** node's replication process for PostgreSQL, and for Git repositories/files. Both should be protected using TLS, with the keys for that managed via Omnibus per existing configuration for end-user access to GitLab. diff --git a/doc/administration/geo/replication/usage.md b/doc/administration/geo/replication/usage.md new file mode 100644 index 00000000000..2fcc0565567 --- /dev/null +++ b/doc/administration/geo/replication/usage.md @@ -0,0 +1,27 @@ +--- +stage: Enablement +group: Geo +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +type: howto +--- + +<!-- Please update EE::GitLab::GeoGitAccess::GEO_SERVER_DOCS_URL if this file is moved) --> + +# Using a Geo Site **(PREMIUM SELF)** + +After you set up the [database replication and configure the Geo nodes](../index.md#setup-instructions), use your closest GitLab site as you would do with the primary one. + +You can push directly to a **secondary** site (for both HTTP, SSH including Git LFS), and the request will be proxied to the primary site instead ([introduced](https://about.gitlab.com/releases/2018/09/22/gitlab-11-3-released/) in [GitLab Premium](https://about.gitlab.com/pricing/#self-managed) 11.3). + +Example of the output you will see when pushing to a **secondary** site: + +```shell +$ git push +remote: +remote: This request to a Geo secondary node will be forwarded to the +remote: Geo primary node: +remote: +remote: ssh://git@primary.geo/user/repo.git +remote: +Everything up-to-date +``` diff --git a/doc/administration/geo/replication/using_a_geo_server.md b/doc/administration/geo/replication/using_a_geo_server.md index 7578b6bf61a..e48e750f710 100644 --- a/doc/administration/geo/replication/using_a_geo_server.md +++ b/doc/administration/geo/replication/using_a_geo_server.md @@ -1,27 +1,8 @@ --- -stage: Enablement -group: Geo -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments -type: howto +redirect_to: '../../geo/replication/usage.md' --- -<!-- Please update EE::GitLab::GeoGitAccess::GEO_SERVER_DOCS_URL if this file is moved) --> +This document was moved to [another location](../../geo/replication/usage.md). -# Using a Geo Server **(PREMIUM SELF)** - -After you set up the [database replication and configure the Geo nodes](../index.md#setup-instructions), use your closest GitLab node as you would a normal standalone GitLab instance. - -Pushing directly to a **secondary** node (for both HTTP, SSH including Git LFS) was [introduced](https://about.gitlab.com/releases/2018/09/22/gitlab-11-3-released/) in [GitLab Premium](https://about.gitlab.com/pricing/#self-managed) 11.3. - -Example of the output you will see when pushing to a **secondary** node: - -```shell -$ git push -remote: -remote: You're pushing to a Geo secondary. We'll help you by proxying this -remote: request to the primary: -remote: -remote: ssh://git@primary.geo/user/repo.git -remote: -Everything up-to-date -``` +<!-- This redirect file can be deleted after 2022-04-01 --> +<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page --> diff --git a/doc/administration/geo/replication/version_specific_updates.md b/doc/administration/geo/replication/version_specific_updates.md index 883e5d44b69..4f0a4dc638c 100644 --- a/doc/administration/geo/replication/version_specific_updates.md +++ b/doc/administration/geo/replication/version_specific_updates.md @@ -5,7 +5,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w type: howto --- -# Version-specific update instructions +# Version-specific update instructions **(PREMIUM SELF)** Review this page for update instructions for your version. These steps accompany the [general steps](updating_the_geo_nodes.md#general-update-steps) diff --git a/doc/administration/geo/setup/database.md b/doc/administration/geo/setup/database.md index 7128d4283d1..9c2cc8fc62e 100644 --- a/doc/administration/geo/setup/database.md +++ b/doc/administration/geo/setup/database.md @@ -672,7 +672,7 @@ For each Patroni instance on the secondary site: ## Migrating from repmgr to Patroni 1. Before migrating, it is recommended that there is no replication lag between the primary and secondary sites and that replication is paused. In GitLab 13.2 and later, you can pause and resume replication with `gitlab-ctl geo-replication-pause` and `gitlab-ctl geo-replication-resume` on a Geo secondary database node. -1. Follow the [instructions to migrate repmgr to Patroni](../../postgresql/replication_and_failover.md#switching-from-repmgr-to-patroni). When configuring Patroni on each primary site database node, add `patroni['replicaton_slots'] = { '<slot_name>' => 'physical' }` +1. Follow the [instructions to migrate repmgr to Patroni](../../postgresql/replication_and_failover.md#switching-from-repmgr-to-patroni). When configuring Patroni on each primary site database node, add `patroni['replication_slots'] = { '<slot_name>' => 'physical' }` to `gitlab.rb` where `<slot_name>` is the name of the replication slot for your Geo secondary. This will ensure that Patroni recognizes the replication slot as permanent and will not drop it upon restarting. 1. If database replication to the secondary was paused before migration, resume replication once Patroni is confirmed working on the primary. diff --git a/doc/administration/geo/setup/index.md b/doc/administration/geo/setup/index.md index 8308cbfa82e..5ec18e29f21 100644 --- a/doc/administration/geo/setup/index.md +++ b/doc/administration/geo/setup/index.md @@ -5,7 +5,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w type: howto --- -# Setting up Geo +# Setting up Geo **(PREMIUM SELF)** These instructions assume you have a working instance of GitLab. They guide you through: diff --git a/doc/administration/gitaly/configure_gitaly.md b/doc/administration/gitaly/configure_gitaly.md index 7e3647d1e34..51ad376a625 100644 --- a/doc/administration/gitaly/configure_gitaly.md +++ b/doc/administration/gitaly/configure_gitaly.md @@ -5,7 +5,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w type: reference --- -# Configure Gitaly +# Configure Gitaly **(FREE SELF)** The Gitaly service itself is configured by using a [TOML configuration file](reference.md). @@ -941,3 +941,226 @@ result as you did at the start. For example: ``` Note that `enforced="true"` means that authentication is being enforced. + +## Pack-objects cache **(FREE SELF)** + +> - [Introduced](https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/372) in GitLab 13.11. +> - It's enabled on GitLab.com. +> - It's recommended for production use. + +[Gitaly](index.md), the service that provides storage for Git +repositories, can be configured to cache a short rolling window of Git +fetch responses. This can reduce server load when your server receives +lots of CI fetch traffic. + +### Overview + +The pack-objects cache wraps `git pack-objects`, an internal part of +Git that gets invoked indirectly via the PostUploadPack and +SSHUploadPack Gitaly RPCs. These are the RPCs that Gitaly runs when a +user does a Git fetch via HTTP or SSH, respectively. When the cache is +enabled, anything that uses PostUploadPack or SSHUploadPack can +benefit from it. It is orthogonal to: + +- The transport (HTTP or SSH). +- Git protocol version (v0 or v2). +- The type of fetch (full clones, incremental fetches, shallow clones, + partial clones, and so on). + +The strength of this cache is its ability to deduplicate concurrent +identical fetches. It: + +- Can benefit GitLab instances where your users run CI/CD pipelines with many concurrent jobs. + There should be a noticeable reduction in server CPU utilization. +- Does not benefit unique fetches at all. For example, if you run a spot check by cloning a + repository to your local computer, you are unlikely to see a benefit from this cache because + your fetch is probably unique. + +The pack-objects cache is a local cache. It: + +- Stores its metadata in the memory of the Gitaly process it is enabled in. +- Stores the actual Git data it is caching in files on local storage. + +Using local files has the benefit that the operating system may +automatically keep parts of the pack-objects cache files in RAM, +making it faster. + +Because the pack-objects cache can lead to a significant increase in +disk write IO, it is off by default. + +### Configure the cache + +These are the configuration settings for the pack-objects cache. Each +setting is discussed in greater detail below. + +|Setting|Default|Description| +|:---|:---|:---| +|`enabled`|`false`|Turns on the cache. When off, Gitaly runs a dedicated `git pack-objects` process for each request. | +|`dir`|`<PATH TO FIRST STORAGE>/+gitaly/PackObjectsCache`|Local directory where cache files get stored.| +|`max_age`|`5m` (5 minutes)|Cache entries older than this get evicted and removed from disk.| + +In `/etc/gitlab/gitlab.rb`, set: + +```ruby +gitaly['pack_objects_cache_enabled'] = true +## gitaly['pack_objects_cache_dir'] = '/var/opt/gitlab/git-data/repositories/+gitaly/PackObjectsCache' +## gitaly['pack_objects_cache_max_age'] = '5m' +``` + +#### `enabled` defaults to `false` + +The cache is disabled by default. This is because in some cases, it +can create an [extreme +increase](https://gitlab.com/gitlab-com/gl-infra/production/-/issues/4010#note_534564684) +in the number of bytes written to disk. On GitLab.com, we have verified +that our repository storage disks can handle this extra workload, but +we felt we cannot assume this is true everywhere. + +#### Cache storage directory `dir` + +The cache needs a directory to store its files in. This directory +should be: + +- In a filesystem with enough space. If the cache filesystem runs out of space, all + fetches start failing. +- On a disk with enough IO bandwidth. If the cache disk runs out of IO bandwidth, all + fetches, and probably the entire server, slows down. + +By default, the cache storage directory is set to a subdirectory of the first Gitaly storage +defined in the configuration file. + +Multiple Gitaly processes can use the same directory for cache storage. Each Gitaly process +uses a unique random string as part of the cache filenames it creates. This means: + +- They do not collide. +- They do not reuse another process's files. + +While the default directory puts the cache files in the same +filesystem as your repository data, this is not requirement. You can +put the cache files on a different filesystem if that works better for +your infrastructure. + +The amount of IO bandwidth required from the disk depends on: + +- The size and shape of the repositories on your Gitaly server. +- The kind of traffic your users generate. + +You can use the `gitaly_pack_objects_generated_bytes_total` metric as a pessimistic estimate, +pretending your cache hit ratio is 0%. + +The amount of space required depends on: + +- The bytes per second that your users pull from the cache. +- The size of the `max_age` cache eviction window. + +If your users pull 100 MB/s and you use a 5 minute window, then on average you have +`5*60*100MB = 30GB` of data in your cache directory. This is an expected average, not +a guarantee. Peak size may exceed this average. + +#### Cache eviction window `max_age` + +The `max_age` configuration setting lets you control the chance of a +cache hit and the average amount of storage used by cache files. +Entries older than `max_age` get evicted from the in-memory metadata +store, and deleted from disk. + +Note that eviction does not interfere with ongoing requests, so it is OK +for `max_age` to be less than the time it takes to do a fetch over a +slow connection. This is because Unix filesystems do not truly delete +a file until all processes that are reading the deleted file have +closed it. + +### Observe the cache + +The cache can be observed in logs and using metrics. + +#### Logs + +|Message|Fields|Description| +|:---|:---|:---| +|`generated bytes`|`bytes`, `cache_key`|Logged when an entry was added to the cache| +|`served bytes`|`bytes`, `cache_key`|Logged when an entry was read from the cache| + +In the case of a: + +- Cache miss, Gitaly logs both a `generated bytes` and a `served bytes` message. +- Cache hit, Gitaly logs only a `served bytes` message. + +Example: + +```json +{ + "bytes":26186490, + "cache_key":"1b586a2698ca93c2529962e85cda5eea8f0f2b0036592615718898368b462e19", + "correlation_id":"01F1MY8JXC3FZN14JBG1H42G9F", + "grpc.meta.deadline_type":"none", + "grpc.method":"PackObjectsHook", + "grpc.request.fullMethod":"/gitaly.HookService/PackObjectsHook", + "grpc.request.glProjectPath":"root/gitlab-workhorse", + "grpc.request.glRepository":"project-2", + "grpc.request.repoPath":"@hashed/d4/73/d4735e3a265e16eee03f59718b9b5d03019c07d8b6c51f90da3a666eec13ab35.git", + "grpc.request.repoStorage":"default", + "grpc.request.topLevelGroup":"@hashed", + "grpc.service":"gitaly.HookService", + "grpc.start_time":"2021-03-25T14:57:52.747Z", + "level":"info", + "msg":"generated bytes", + "peer.address":"@", + "pid":20961, + "span.kind":"server", + "system":"grpc", + "time":"2021-03-25T14:57:53.543Z" +} +{ + "bytes":26186490, + "cache_key":"1b586a2698ca93c2529962e85cda5eea8f0f2b0036592615718898368b462e19", + "correlation_id":"01F1MY8JXC3FZN14JBG1H42G9F", + "grpc.meta.deadline_type":"none", + "grpc.method":"PackObjectsHook", + "grpc.request.fullMethod":"/gitaly.HookService/PackObjectsHook", + "grpc.request.glProjectPath":"root/gitlab-workhorse", + "grpc.request.glRepository":"project-2", + "grpc.request.repoPath":"@hashed/d4/73/d4735e3a265e16eee03f59718b9b5d03019c07d8b6c51f90da3a666eec13ab35.git", + "grpc.request.repoStorage":"default", + "grpc.request.topLevelGroup":"@hashed", + "grpc.service":"gitaly.HookService", + "grpc.start_time":"2021-03-25T14:57:52.747Z", + "level":"info", + "msg":"served bytes", + "peer.address":"@", + "pid":20961, + "span.kind":"server", + "system":"grpc", + "time":"2021-03-25T14:57:53.543Z" +} +``` + +#### Metrics + +The following cache metrics are available. + +|Metric|Type|Labels|Description| +|:---|:---|:---|:---| +|`gitaly_pack_objects_cache_enabled`|gauge|`dir`,`max_age`|Set to `1` when the cache is enabled via the Gitaly config file| +|`gitaly_pack_objects_cache_lookups_total`|counter|`result`|Hit/miss counter for cache lookups| +|`gitaly_pack_objects_generated_bytes_total`|counter||Number of bytes written into the cache| +|`gitaly_pack_objects_served_bytes_total`|counter||Number of bytes read from the cache| +|`gitaly_streamcache_filestore_disk_usage_bytes`|gauge|`dir`|Total size of cache files| +|`gitaly_streamcache_index_entries`|gauge|`dir`|Number of entries in the cache| + +Some of these metrics start with `gitaly_streamcache` +because they are generated by the "streamcache" internal library +package in Gitaly. + +Example: + +```plaintext +gitaly_pack_objects_cache_enabled{dir="/var/opt/gitlab/git-data/repositories/+gitaly/PackObjectsCache",max_age="300"} 1 +gitaly_pack_objects_cache_lookups_total{result="hit"} 2 +gitaly_pack_objects_cache_lookups_total{result="miss"} 1 +gitaly_pack_objects_generated_bytes_total 2.618649e+07 +gitaly_pack_objects_served_bytes_total 7.855947e+07 +gitaly_streamcache_filestore_disk_usage_bytes{dir="/var/opt/gitlab/git-data/repositories/+gitaly/PackObjectsCache"} 2.6200152e+07 +gitaly_streamcache_filestore_removed_total{dir="/var/opt/gitlab/git-data/repositories/+gitaly/PackObjectsCache"} 1 +gitaly_streamcache_index_entries{dir="/var/opt/gitlab/git-data/repositories/+gitaly/PackObjectsCache"} 1 +``` diff --git a/doc/administration/gitaly/img/architecture_v12_4.png b/doc/administration/gitaly/img/architecture_v12_4.png Binary files differdeleted file mode 100644 index 6a3955a483b..00000000000 --- a/doc/administration/gitaly/img/architecture_v12_4.png +++ /dev/null diff --git a/doc/administration/gitaly/index.md b/doc/administration/gitaly/index.md index b314fa85af7..1a8e18ca2b2 100644 --- a/doc/administration/gitaly/index.md +++ b/doc/administration/gitaly/index.md @@ -5,72 +5,296 @@ info: To determine the technical writer assigned to the Stage/Group associated w type: reference --- -# Gitaly +# Gitaly and Gitaly Cluster **(FREE SELF)** -[Gitaly](https://gitlab.com/gitlab-org/gitaly) is the service that provides high-level RPC access to -Git repositories. Without it, no GitLab components can read or write Git data. +[Gitaly](https://gitlab.com/gitlab-org/gitaly) provides high-level RPC access to Git repositories. +It is used by GitLab to read and write Git data. -In the Gitaly documentation: +Gitaly implements a client-server architecture: -- **Gitaly server** refers to any node that runs Gitaly itself. -- **Gitaly client** refers to any node that runs a process that makes requests of the - Gitaly server. Processes include, but are not limited to: +- A Gitaly server is any node that runs Gitaly itself. +- A Gitaly client is any node that runs a process that makes requests of the Gitaly server. These + include, but are not limited to: - [GitLab Rails application](https://gitlab.com/gitlab-org/gitlab). - [GitLab Shell](https://gitlab.com/gitlab-org/gitlab-shell). - [GitLab Workhorse](https://gitlab.com/gitlab-org/gitlab-workhorse). -GitLab end users do not have direct access to Gitaly. Gitaly manages only Git -repository access for GitLab. Other types of GitLab data aren't accessed using Gitaly. +The following illustrates the Gitaly client-server architecture: + +```mermaid +flowchart TD + subgraph Gitaly clients + A[GitLab Rails] + B[GitLab Workhorse] + C[GitLab Shell] + D[...] + end + + subgraph Gitaly + E[Git integration] + end + +F[Local filesystem] + +A -- gRPC --> Gitaly +B -- gRPC--> Gitaly +C -- gRPC --> Gitaly +D -- gRPC --> Gitaly + +E --> F +``` + +End users do not have direct access to Gitaly. Gitaly manages only Git repository access for GitLab. +Other types of GitLab data aren't accessed using Gitaly. <!-- vale gitlab.FutureTense = NO --> WARNING: -From GitLab 13.0, Gitaly support for NFS is deprecated. As of GitLab 14.0, NFS-related issues -with Gitaly will no longer be addressed. Upgrade to [Gitaly Cluster](praefect.md) as soon as -possible. Tools to [enable bulk moves](https://gitlab.com/groups/gitlab-org/-/epics/4916) -of projects to Gitaly Cluster are planned. +From GitLab 14.0, enhancements and bug fixes for NFS for Git repositories will no longer be +considered and customer technical support will be considered out of scope. +[Read more about Gitaly and NFS](#nfs-deprecation-notice). <!-- vale gitlab.FutureTense = YES --> -## Architecture +## Configure Gitaly -The following is a high-level architecture overview of how Gitaly is used. +Gitaly comes pre-configured with Omnibus GitLab, which is a configuration +[suitable for up to 1000 users](../reference_architectures/1k_users.md). For: - +- Omnibus GitLab installations for up to 2000 users, see [specific Gitaly configuration instructions](../reference_architectures/2k_users.md#configure-gitaly). +- Source installations or custom Gitaly installations, see [Configure Gitaly](#configure-gitaly). -## Configure Gitaly +GitLab installations for more than 2000 users should use Gitaly Cluster. + +NOTE: +If not set in GitLab, feature flags are read as false from the console and Gitaly uses their +default value. The default value depends on the GitLab version. + +## Gitaly Cluster + +Gitaly, the service that provides storage for Git repositories, can +be run in a clustered configuration to scale the Gitaly service and increase +fault tolerance. In this configuration, every Git repository is stored on every +Gitaly node in the cluster. + +Using a Gitaly Cluster increases fault tolerance by: + +- Replicating write operations to warm standby Gitaly nodes. +- Detecting Gitaly node failures. +- Automatically routing Git requests to an available Gitaly node. + +NOTE: +Technical support for Gitaly clusters is limited to GitLab Premium and Ultimate +customers. + +The availability objectives for Gitaly clusters are: + +- **Recovery Point Objective (RPO):** Less than 1 minute. + + Writes are replicated asynchronously. Any writes that have not been replicated + to the newly promoted primary are lost. + + [Strong consistency](praefect.md#strong-consistency) can be used to avoid loss in some + circumstances. + +- **Recovery Time Objective (RTO):** Less than 10 seconds. + Outages are detected by a health check run by each Praefect node every + second. Failover requires ten consecutive failed health checks on each + Praefect node. + + [Faster outage detection](https://gitlab.com/gitlab-org/gitaly/-/issues/2608) + is planned to improve this to less than 1 second. + +Gitaly Cluster supports: + +- [Strong consistency](praefect.md#strong-consistency) of the secondary replicas. +- [Automatic failover](praefect.md#automatic-failover-and-leader-election) from the primary to the secondary. +- Reporting of possible data loss if replication queue is non-empty. +- Marking repositories as [read only](praefect.md#read-only-mode) if data loss is detected to prevent data inconsistencies. + +Follow the [Gitaly Cluster epic](https://gitlab.com/groups/gitlab-org/-/epics/1489) +for improvements including +[horizontally distributing reads](https://gitlab.com/groups/gitlab-org/-/epics/2013). + +### Overview + +Git storage is provided through the Gitaly service in GitLab, and is essential +to the operation of the GitLab application. When the number of +users, repositories, and activity grows, it is important to scale Gitaly +appropriately by: + +- Increasing the available CPU and memory resources available to Git before + resource exhaustion degrades Git, Gitaly, and GitLab application performance. +- Increase available storage before storage limits are reached causing write + operations to fail. +- Improve fault tolerance by removing single points of failure. Git should be + considered mission critical if a service degradation would prevent you from + deploying changes to production. + +### Moving beyond NFS + +WARNING: +From GitLab 13.0, using NFS for Git repositories is deprecated. In GitLab 14.0, +support for NFS for Git repositories is scheduled to be removed. Upgrade to +Gitaly Cluster as soon as possible. + +[Network File System (NFS)](https://en.wikipedia.org/wiki/Network_File_System) +is not well suited to Git workloads which are CPU and IOPS sensitive. +Specifically: + +- Git is sensitive to file system latency. Even simple operations require many + read operations. Operations that are fast on block storage can become an order of + magnitude slower. This significantly impacts GitLab application performance. +- NFS performance optimizations that prevent the performance gap between + block storage and NFS being even wider are vulnerable to race conditions. We have observed + [data inconsistencies](https://gitlab.com/gitlab-org/gitaly/-/issues/2589) + in production environments caused by simultaneous writes to different NFS + clients. Data corruption is not an acceptable risk. + +Gitaly Cluster is purpose built to provide reliable, high performance, fault +tolerant Git storage. + +Further reading: + +- Blog post: [The road to Gitaly v1.0 (aka, why GitLab doesn't require NFS for storing Git data anymore)](https://about.gitlab.com/blog/2018/09/12/the-road-to-gitaly-1-0/) +- Blog post: [How we spent two weeks hunting an NFS bug in the Linux kernel](https://about.gitlab.com/blog/2018/11/14/how-we-spent-two-weeks-hunting-an-nfs-bug/) + +### Where Gitaly Cluster fits + +GitLab accesses [repositories](../../user/project/repository/index.md) through the configured +[repository storages](../repository_storage_paths.md). Each new repository is stored on one of the +repository storages based on their configured weights. Each repository storage is either: + +- A Gitaly storage served directly by Gitaly. These map to a directory on the file system of a + Gitaly node. +- A [virtual storage](#virtual-storage-or-direct-gitaly-storage) served by Praefect. A virtual + storage is a cluster of Gitaly storages that appear as a single repository storage. + +Virtual storages are a feature of Gitaly Cluster. They support replicating the repositories to +multiple storages for fault tolerance. Virtual storages can improve performance by distributing +requests across Gitaly nodes. Their distributed nature makes it viable to have a single repository +storage in GitLab to simplify repository management. + +### Components of Gitaly Cluster + +Gitaly Cluster consists of multiple components: + +- [Load balancer](praefect.md#load-balancer) for distributing requests and providing fault-tolerant access to + Praefect nodes. +- [Praefect](praefect.md#praefect) nodes for managing the cluster and routing requests to Gitaly nodes. +- [PostgreSQL database](praefect.md#postgresql) for persisting cluster metadata and [PgBouncer](praefect.md#pgbouncer), + recommended for pooling Praefect's database connections. +- Gitaly nodes to provide repository storage and Git access. + + + +In this example: + +- Repositories are stored on a virtual storage called `storage-1`. +- Three Gitaly nodes provide `storage-1` access: `gitaly-1`, `gitaly-2`, and `gitaly-3`. +- The three Gitaly nodes store data on their file systems. + +### Virtual storage or direct Gitaly storage + +Gitaly supports multiple models of scaling: + +- Clustering using Gitaly Cluster, where each repository is stored on multiple Gitaly nodes in the + cluster. Read requests are distributed between repository replicas and write requests are + broadcast to repository replicas. GitLab accesses virtual storage. +- Direct access to Gitaly storage using [repository storage paths](../repository_storage_paths.md), + where each repository is stored on the assigned Gitaly node. All requests are routed to this node. -Gitaly comes pre-configured with Omnibus GitLab. For more information on customizing your -Gitaly installation, see [Configure Gitaly](configure_gitaly.md). +The following is Gitaly set up to use direct access to Gitaly instead of Gitaly Cluster: -## Direct Git access bypassing Gitaly + -GitLab doesn't advise directly accessing Gitaly repositories stored on disk with -a Git client, because Gitaly is being continuously improved and changed. These -improvements may invalidate assumptions, resulting in performance degradation, instability, and even data loss. +In this example: -Gitaly has optimizations, such as the -[`info/refs` advertisement cache](https://gitlab.com/gitlab-org/gitaly/blob/master/doc/design_diskcache.md), -that rely on Gitaly controlling and monitoring access to repositories by using the -official gRPC interface. Likewise, Praefect has optimizations, such as fault -tolerance and distributed reads, that depend on the gRPC interface and -database to determine repository state. +- Each repository is stored on one of three Gitaly storages: `storage-1`, `storage-2`, + or `storage-3`. +- Each storage is serviced by a Gitaly node. +- The three Gitaly nodes store data in three separate hashed storage locations. -For these reasons, **accessing repositories directly is done at your own risk -and is not supported**. +Generally, virtual storage with Gitaly Cluster can replace direct Gitaly storage configurations, at +the expense of additional storage needed to store each repository on multiple Gitaly nodes. The +benefit of using Gitaly Cluster over direct Gitaly storage is: + +- Improved fault tolerance, because each Gitaly node has a copy of every repository. +- Improved resource utilization, reducing the need for over-provisioning for shard-specific peak + loads, because read loads are distributed across replicas. +- Manual rebalancing for performance is not required, because read loads are distributed across + replicas. +- Simpler management, because all Gitaly nodes are identical. + +Under some workloads, CPU and memory requirements may require a large fleet of Gitaly nodes. It +can be uneconomical to have one to one replication factor. + +A hybrid approach can be used in these instances, where each shard is configured as a smaller +cluster. [Variable replication factor](https://gitlab.com/groups/gitlab-org/-/epics/3372) is planned +to provide greater flexibility for extremely large GitLab instances. + +### Gitaly Cluster compared to Geo + +Gitaly Cluster and [Geo](../geo/index.md) both provide redundancy. However the redundancy of: + +- Gitaly Cluster provides fault tolerance for data storage and is invisible to the user. Users are + not aware when Gitaly Cluster is used. +- Geo provides [replication](../geo/index.md) and [disaster recovery](../geo/disaster_recovery/index.md) for + an entire instance of GitLab. Users know when they are using Geo for + [replication](../geo/index.md). Geo [replicates multiple data types](../geo/replication/datatypes.md#limitations-on-replicationverification), + including Git data. + +The following table outlines the major differences between Gitaly Cluster and Geo: + +| Tool | Nodes | Locations | Latency tolerance | Failover | Consistency | Provides redundancy for | +|:---------------|:---------|:----------|:-------------------|:----------------------------------------------------------------|:-----------------------------------------|:------------------------| +| Gitaly Cluster | Multiple | Single | Approximately 1 ms | [Automatic](praefect.md#automatic-failover-and-leader-election) | [Strong](praefect.md#strong-consistency) | Data storage in Git | +| Geo | Multiple | Multiple | Up to one minute | [Manual](../geo/disaster_recovery/index.md) | Eventual | Entire GitLab instance | + +For more information, see: + +- Geo [use cases](../geo/index.md#use-cases). +- Geo [architecture](../geo/index.md#architecture). + +### Architecture + +Praefect is a router and transaction manager for Gitaly, and a required +component for running a Gitaly Cluster. + + + +For more information, see [Gitaly High Availability (HA) Design](https://gitlab.com/gitlab-org/gitaly/-/blob/master/doc/design_ha.md). + +### Configure Gitaly Cluster + +For more information on configuring Gitaly Cluster, see [Configure Gitaly Cluster](praefect.md). + +## Do not bypass Gitaly + +GitLab doesn't advise directly accessing Gitaly repositories stored on disk with a Git client, +because Gitaly is being continuously improved and changed. These improvements may invalidate +your assumptions, resulting in performance degradation, instability, and even data loss. For example: + +- Gitaly has optimizations such as the [`info/refs` advertisement cache](https://gitlab.com/gitlab-org/gitaly/blob/master/doc/design_diskcache.md), + that rely on Gitaly controlling and monitoring access to repositories by using the official gRPC + interface. +- [Gitaly Cluster](praefect.md) has optimizations, such as fault tolerance and + [distributed reads](praefect.md#distributed-reads), that depend on the gRPC interface and database + to determine repository state. + +WARNING: +Accessing Git repositories directly is done at your own risk and is not supported. ## Direct access to Git in GitLab Direct access to Git uses code in GitLab known as the "Rugged patches". -### History - -Before Gitaly existed, what are now Gitaly clients used to access Git repositories directly, either: +Before Gitaly existed, what are now Gitaly clients accessed Git repositories directly, either: -- On a local disk in the case of a single-machine Omnibus GitLab installation +- On a local disk in the case of a single-machine Omnibus GitLab installation. - Using NFS in the case of a horizontally-scaled GitLab installation. -Besides running plain `git` commands, GitLab used to use a Ruby library called +In addition to running plain `git` commands, GitLab used a Ruby library called [Rugged](https://github.com/libgit2/rugged). Rugged is a wrapper around [libgit2](https://libgit2.org/), a stand-alone implementation of Git in the form of a C library. @@ -81,9 +305,9 @@ not an external process, there was very little overhead between: - GitLab application code that tried to look up data in Git repositories. - The Git implementation itself. -Because the combination of Rugged and Unicorn was so efficient, the GitLab application code ended up with lots of -duplicate Git object lookups. For example, looking up the `master` commit a dozen times in one -request. We could write inefficient code without poor performance. +Because the combination of Rugged and Unicorn was so efficient, the GitLab application code ended up +with lots of duplicate Git object lookups. For example, looking up the default branch commit a dozen +times in one request. We could write inefficient code without poor performance. When we migrated these Git lookups to Gitaly calls, we suddenly had a much higher fixed cost per Git lookup. Even when Gitaly is able to re-use an already-running `git` process (for example, to look up @@ -94,8 +318,8 @@ a commit), you still have: Using GitLab.com to measure, we reduced the number of Gitaly calls per request until the loss of Rugged's efficiency was no longer felt. It also helped that we run Gitaly itself directly on the Git -file severs, rather than by using NFS mounts. This gave us a speed boost that counteracted the negative -effect of not using Rugged anymore. +file servers, rather than by using NFS mounts. This gave us a speed boost that counteracted the +negative effect of not using Rugged anymore. Unfortunately, other deployments of GitLab could not remove NFS like we did on GitLab.com, and they got the worst of both worlds: @@ -154,7 +378,29 @@ There are two facets to our efforts to remove direct Git access in GitLab: NFS. The second facet presents the only real solution. For this, we developed -[Gitaly Cluster](praefect.md). +[Gitaly Cluster](#gitaly-cluster). + +## NFS deprecation notice + +<!-- vale gitlab.FutureTense = NO --> + +From GitLab 14.0, enhancements and bug fixes for NFS for Git repositories will no longer be +considered and customer technical support will be considered out of scope. + +Additional information: + +- [Recommended NFS mount options and known issues with Gitaly and NFS](../nfs.md#upgrade-to-gitaly-cluster-or-disable-caching-if-experiencing-data-loss). +- [GitLab statement of support](https://about.gitlab.com/support/statement-of-support.html#gitaly-and-nfs). + +<!-- vale gitlab.FutureTense = YES --> + +GitLab recommends: + +- Creating a [Gitaly Cluster](#gitaly-cluster) as soon as possible. +- [Moving your repositories](praefect.md#migrate-to-gitaly-cluster) from NFS-based storage to Gitaly + Cluster. + +We welcome your feedback on this process: raise a support ticket, or [comment on the epic](https://gitlab.com/groups/gitlab-org/-/epics/4916). ## Troubleshooting Gitaly @@ -213,6 +459,21 @@ You can run a gRPC trace with: sudo GRPC_TRACE=all GRPC_VERBOSITY=DEBUG gitlab-rake gitlab:gitaly:check ``` +### Server side gRPC logs + +gRPC tracing can also be enabled in Gitaly itself with the `GODEBUG=http2debug` +environment variable. To set this in an Omnibus GitLab install: + +1. Add the following to your `gitlab.rb` file: + + ```ruby + gitaly['env'] = { + "GODEBUG=http2debug" => "2" + } + ``` + +1. [Reconfigure](../restart_gitlab.md#omnibus-gitlab-reconfigure) GitLab. + ### Correlating Git processes with RPCs Sometimes you need to find out which Gitaly RPC created a particular Git process. @@ -240,9 +501,9 @@ so, there's not that much visibility into what goes on inside If you have Prometheus set up to scrape your Gitaly process, you can see request rates and error codes for individual RPCs in `gitaly-ruby` by querying `grpc_client_handled_total`. Strictly speaking, this metric does -not differentiate between `gitaly-ruby` and other RPCs, but in practice -(as of GitLab 11.9), all gRPC calls made by Gitaly itself are internal -calls from the main Gitaly process to one of its `gitaly-ruby` sidecars. +not differentiate between `gitaly-ruby` and other RPCs. However from GitLab 11.9, +all gRPC calls made by Gitaly itself are internal calls from the main Gitaly process to one of its +`gitaly-ruby` sidecars. Assuming your `grpc_client_handled_total` counter observes only Gitaly, the following query shows you RPCs are (most likely) internally @@ -349,9 +610,10 @@ update the secrets file on the Gitaly server to match the Gitaly client, then ### Command line tools cannot connect to Gitaly -If you can't connect to a Gitaly server with command line (CLI) tools, -and certain actions result in a `14: Connect Failed` error message, -gRPC cannot reach your Gitaly server. +gRPC cannot reach your Gitaly server if: + +- You can't connect to a Gitaly server with command-line tools. +- Certain actions result in a `14: Connect Failed` error message. Verify you can reach Gitaly by using TCP: @@ -383,16 +645,30 @@ unset http_proxy unset https_proxy ``` -### Permission denied errors appearing in Gitaly logs when accessing repositories from a standalone Gitaly server +### Permission denied errors appearing in Gitaly or Praefect logs when accessing repositories -If this error occurs even though file permissions are correct, it's likely that -the Gitaly server is experiencing -[clock drift](https://en.wikipedia.org/wiki/Clock_drift). +You might see the following in Gitaly and Praefect logs: -Ensure the Gitaly clients and servers are synchronized, and use an NTP time -server to keep them synchronized, if possible. +```shell +{ + ... + "error":"rpc error: code = PermissionDenied desc = permission denied", + "grpc.code":"PermissionDenied", + "grpc.meta.client_name":"gitlab-web", + "grpc.request.fullMethod":"/gitaly.ServerService/ServerInfo", + "level":"warning", + "msg":"finished unary call with code PermissionDenied", + ... +} +``` -### Praefect +This is a GRPC call +[error response code](https://grpc.github.io/grpc/core/md_doc_statuscodes.html). -Praefect is a router and transaction manager for Gitaly, and a required -component for running a Gitaly Cluster. For more information see [Gitaly Cluster](praefect.md). +If this error occurs, even though +[the Gitaly auth tokens are correctly setup](../gitaly/praefect.md#debugging-praefect), +it's likely that the Gitaly servers are experiencing +[clock drift](https://en.wikipedia.org/wiki/Clock_drift). + +Ensure the Gitaly clients and servers are synchronized, and use an NTP time +server to keep them synchronized. diff --git a/doc/administration/gitaly/praefect.md b/doc/administration/gitaly/praefect.md index 80b5c1bb799..1a8df1cea92 100644 --- a/doc/administration/gitaly/praefect.md +++ b/doc/administration/gitaly/praefect.md @@ -5,154 +5,11 @@ info: To determine the technical writer assigned to the Stage/Group associated w type: reference --- -# Gitaly Cluster **(FREE SELF)** +# Configure Gitaly Cluster **(FREE SELF)** -[Gitaly](index.md), the service that provides storage for Git repositories, can -be run in a clustered configuration to increase fault tolerance. In this -configuration, every Git repository is stored on every Gitaly node in the -cluster. Multiple clusters (or storage shards) can be configured. - -NOTE: -Technical support for Gitaly clusters is limited to GitLab Premium and Ultimate -customers. - -Praefect is a router and transaction manager for Gitaly, and a required -component for running a Gitaly Cluster. - - - -Using a Gitaly Cluster increases fault tolerance by: - -- Replicating write operations to warm standby Gitaly nodes. -- Detecting Gitaly node failures. -- Automatically routing Git requests to an available Gitaly node. - -The availability objectives for Gitaly clusters are: - -- **Recovery Point Objective (RPO):** Less than 1 minute. - - Writes are replicated asynchronously. Any writes that have not been replicated - to the newly promoted primary are lost. - - [Strong consistency](#strong-consistency) can be used to avoid loss in some - circumstances. - -- **Recovery Time Objective (RTO):** Less than 10 seconds. - - Outages are detected by a health checks run by each Praefect node every - second. Failover requires ten consecutive failed health checks on each - Praefect node. - - [Faster outage detection](https://gitlab.com/gitlab-org/gitaly/-/issues/2608) - is planned to improve this to less than 1 second. - -Gitaly Cluster supports: - -- [Strong consistency](#strong-consistency) of the secondary replicas. -- [Automatic failover](#automatic-failover-and-leader-election) from the primary to the secondary. -- Reporting of possible data loss if replication queue is non-empty. -- Marking repositories as [read only](#read-only-mode) if data loss is detected to prevent data inconsistencies. - -Follow the [Gitaly Cluster epic](https://gitlab.com/groups/gitlab-org/-/epics/1489) -for improvements including -[horizontally distributing reads](https://gitlab.com/groups/gitlab-org/-/epics/2013). - -## Gitaly Cluster compared to Geo - -Gitaly Cluster and [Geo](../geo/index.md) both provide redundancy. However the redundancy of: - -- Gitaly Cluster provides fault tolerance for data storage and is invisible to the user. Users are - not aware when Gitaly Cluster is used. -- Geo provides [replication](../geo/index.md) and [disaster recovery](../geo/disaster_recovery/index.md) for - an entire instance of GitLab. Users know when they are using Geo for - [replication](../geo/index.md). Geo [replicates multiple data types](../geo/replication/datatypes.md#limitations-on-replicationverification), - including Git data. - -The following table outlines the major differences between Gitaly Cluster and Geo: - -| Tool | Nodes | Locations | Latency tolerance | Failover | Consistency | Provides redundancy for | -|:---------------|:---------|:----------|:-------------------|:-----------------------------------------------------|:------------------------------|:------------------------| -| Gitaly Cluster | Multiple | Single | Approximately 1 ms | [Automatic](#automatic-failover-and-leader-election) | [Strong](#strong-consistency) | Data storage in Git | -| Geo | Multiple | Multiple | Up to one minute | [Manual](../geo/disaster_recovery/index.md) | Eventual | Entire GitLab instance | - -For more information, see: - -- [Gitaly architecture](index.md#architecture). -- Geo [use cases](../geo/index.md#use-cases) and [architecture](../geo/index.md#architecture). - -## Where Gitaly Cluster fits - -GitLab accesses [repositories](../../user/project/repository/index.md) through the configured -[repository storages](../repository_storage_paths.md). Each new repository is stored on one of the -repository storages based on their configured weights. Each repository storage is either: - -- A Gitaly storage served directly by Gitaly. These map to a directory on the file system of a - Gitaly node. -- A [virtual storage](#virtual-storage-or-direct-gitaly-storage) served by Praefect. A virtual - storage is a cluster of Gitaly storages that appear as a single repository storage. - -Virtual storages are a feature of Gitaly Cluster. They support replicating the repositories to -multiple storages for fault tolerance. Virtual storages can improve performance by distributing -requests across Gitaly nodes. Their distributed nature makes it viable to have a single repository -storage in GitLab to simplify repository management. - -## Components of Gitaly Cluster - -Gitaly Cluster consists of multiple components: - -- [Load balancer](#load-balancer) for distributing requests and providing fault-tolerant access to - Praefect nodes. -- [Praefect](#praefect) nodes for managing the cluster and routing requests to Gitaly nodes. -- [PostgreSQL database](#postgresql) for persisting cluster metadata and [PgBouncer](#pgbouncer), - recommended for pooling Praefect's database connections. -- [Gitaly](index.md) nodes to provide repository storage and Git access. - - - -In this example: - -- Repositories are stored on a virtual storage called `storage-1`. -- Three Gitaly nodes provide `storage-1` access: `gitaly-1`, `gitaly-2`, and `gitaly-3`. -- The three Gitaly nodes store data on their file systems. - -### Virtual storage or direct Gitaly storage - -Gitaly supports multiple models of scaling: - -- Clustering using Gitaly Cluster, where each repository is stored on multiple Gitaly nodes in the - cluster. Read requests are distributed between repository replicas and write requests are - broadcast to repository replicas. GitLab accesses virtual storage. -- Direct access to Gitaly storage using [repository storage paths](../repository_storage_paths.md), - where each repository is stored on the assigned Gitaly node. All requests are routed to this node. - -The following is Gitaly set up to use direct access to Gitaly instead of Gitaly Cluster: - - - -In this example: - -- Each repository is stored on one of three Gitaly storages: `storage-1`, `storage-2`, - or `storage-3`. -- Each storage is serviced by a Gitaly node. -- The three Gitaly nodes store data in three separate hashed storage locations. - -Generally, virtual storage with Gitaly Cluster can replace direct Gitaly storage configurations, at -the expense of additional storage needed to store each repository on multiple Gitaly nodes. The -benefit of using Gitaly Cluster over direct Gitaly storage is: - -- Improved fault tolerance, because each Gitaly node has a copy of every repository. -- Improved resource utilization, reducing the need for over-provisioning for shard-specific peak - loads, because read loads are distributed across replicas. -- Manual rebalancing for performance is not required, because read loads are distributed across - replicas. -- Simpler management, because all Gitaly nodes are identical. - -Under some workloads, CPU and memory requirements may require a large fleet of Gitaly nodes. It -can be uneconomical to have one to one replication factor. - -A hybrid approach can be used in these instances, where each shard is configured as a smaller -cluster. [Variable replication factor](https://gitlab.com/groups/gitlab-org/-/epics/3372) is planned -to provide greater flexibility for extremely large GitLab instances. +In addition to Gitaly Cluster configuration instructions available as part of +[reference architectures](../reference_architectures/index.md) for installations for more than +2000 users, advanced configuration instructions are available below. ## Requirements for configuring a Gitaly Cluster @@ -167,6 +24,10 @@ See the [design document](https://gitlab.com/gitlab-org/gitaly/-/blob/master/doc/design_ha.md) for implementation details. +NOTE: +If not set in GitLab, feature flags are read as false from the console and Praefect uses their +default value. The default value depends on the GitLab version. + ## Setup Instructions If you [installed](https://about.gitlab.com/install/) GitLab using the Omnibus @@ -204,7 +65,7 @@ You need the IP/host address for each node. If you are using a cloud provider, you can look up the addresses for each server through your cloud provider's management console. -If you are using Google Cloud Platform, SoftLayer, or any other vendor that provides a virtual private cloud (VPC) you can use the private addresses for each cloud instance (corresponds to “internal address” for Google Cloud Platform) for `PRAEFECT_HOST`, `GITALY_HOST_*`, and `GITLAB_HOST`. +If you are using Google Cloud Platform, SoftLayer, or any other vendor that provides a virtual private cloud (VPC) you can use the private addresses for each cloud instance (corresponds to "internal address" for Google Cloud Platform) for `PRAEFECT_HOST`, `GITALY_HOST_*`, and `GITLAB_HOST`. #### Secrets @@ -227,6 +88,9 @@ with secure tokens as you complete the setup process. We note in the instructions below where these secrets are required. +NOTE: +Omnibus GitLab installations can use `gitlab-secrets.json` for `GITLAB_SHELL_SECRET_TOKEN`. + ### PostgreSQL NOTE: @@ -236,10 +100,11 @@ database on the same PostgreSQL server if using of GitLab and should not be replicated. These instructions help set up a single PostgreSQL database, which creates a single point of -failure. For greater fault tolerance, the following options are available: +failure. The following options are available: -- For non-Geo installations, use one of the fault-tolerant - [PostgreSQL setups](../postgresql/index.md). +- For non-Geo installations, either: + - Use one of the documented [PostgreSQL setups](../postgresql/index.md). + - Use your own third-party database setup, if fault tolerance is required. - For Geo instances, either: - Set up a separate [PostgreSQL instance](https://www.postgresql.org/docs/11/high-availability.html). - Use a cloud-managed PostgreSQL service. AWS @@ -458,7 +323,7 @@ application server, or a Gitaly node. WARNING: If you have data on an already existing storage called `default`, you should configure the virtual storage with another name and - [migrate the data to the Gitaly Cluster storage](#migrate-existing-repositories-to-gitaly-cluster) + [migrate the data to the Gitaly Cluster storage](#migrate-to-gitaly-cluster) afterwards. Replace `PRAEFECT_INTERNAL_TOKEN` with a strong secret, which is used by @@ -755,14 +620,26 @@ documentation](configure_gitaly.md#configure-gitaly-servers). gitaly['auth_token'] = 'PRAEFECT_INTERNAL_TOKEN' ``` -1. Configure the GitLab Shell `secret_token`, and `internal_api_url` which are - needed for `git push` operations. +1. Configure the GitLab Shell secret token, which is needed for `git push` operations. Either: - If you have already configured [Gitaly on its own server](index.md) + - Method 1: - ```ruby - gitlab_shell['secret_token'] = 'GITLAB_SHELL_SECRET_TOKEN' + 1. Copy `/etc/gitlab/gitlab-secrets.json` from the Gitaly client to same path on the Gitaly + servers and any other Gitaly clients. + 1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) on Gitaly servers. + - Method 2: + + 1. Edit `/etc/gitlab/gitlab.rb`. + 1. Replace `GITLAB_SHELL_SECRET_TOKEN` with the real secret. + + ```ruby + gitlab_shell['secret_token'] = 'GITLAB_SHELL_SECRET_TOKEN' + ``` + +1. Configure and `internal_api_url`, which is also needed for `git push` operations: + + ```ruby # Configure the gitlab-shell API callback URL. Without this, `git push` will # fail. This can be your front door GitLab URL or an internal load balancer. # Examples: 'https://gitlab.example.com', 'http://1.2.3.4' @@ -838,7 +715,7 @@ addition to the GitLab nodes. Some requests handled by process. `gitaly-ruby` uses the Gitaly address set in the GitLab server's `git_data_dirs` setting to make this connection. -We hope that if you’re managing fault-tolerant systems like GitLab, you have a load balancer +We hope that if you're managing fault-tolerant systems like GitLab, you have a load balancer of choice already. Some examples include [HAProxy](https://www.haproxy.org/) (open-source), [Google Internal Load Balancer](https://cloud.google.com/load-balancing/docs/internal/), [AWS Elastic Load Balancer](https://aws.amazon.com/elasticloadbalancing/), F5 @@ -887,7 +764,7 @@ Particular attention should be shown to: WARNING: If you have existing data stored on the default Gitaly storage, - you should [migrate the data your Gitaly Cluster storage](#migrate-existing-repositories-to-gitaly-cluster) + you should [migrate the data your Gitaly Cluster storage](#migrate-to-gitaly-cluster) first. ```ruby @@ -914,15 +791,23 @@ Particular attention should be shown to: }) ``` -1. Configure the `gitlab_shell['secret_token']` so that callbacks from Gitaly - nodes during a `git push` are properly authenticated by editing - `/etc/gitlab/gitlab.rb`: +1. Configure the GitLab Shell secret token so that callbacks from Gitaly nodes during a `git push` + are properly authenticated. Either: - You need to replace `GITLAB_SHELL_SECRET_TOKEN` with the real secret. + - Method 1: - ```ruby - gitlab_shell['secret_token'] = 'GITLAB_SHELL_SECRET_TOKEN' - ``` + 1. Copy `/etc/gitlab/gitlab-secrets.json` from the Gitaly client to same path on the Gitaly + servers and any other Gitaly clients. + 1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) on Gitaly servers. + + - Method 2: + + 1. Edit `/etc/gitlab/gitlab.rb`. + 1. Replace `GITLAB_SHELL_SECRET_TOKEN` with the real secret. + + ```ruby + gitlab_shell['secret_token'] = 'GITLAB_SHELL_SECRET_TOKEN' + ``` 1. Add Prometheus monitoring settings by editing `/etc/gitlab/gitlab.rb`. If Prometheus is enabled on a different node, make edits on that node instead. @@ -1036,6 +921,7 @@ cluster. > - [Made generally available and enabled by default](https://gitlab.com/gitlab-org/gitaly/-/issues/2951) in GitLab 13.3. > - [Disabled by default](https://gitlab.com/gitlab-org/gitaly/-/issues/3178) in GitLab 13.5. > - [Enabled by default](https://gitlab.com/gitlab-org/gitaly/-/issues/3334) in GitLab 13.8. +> - [Feature flag removed](https://gitlab.com/gitlab-org/gitaly/-/issues/3383) in GitLab 13.11. Praefect supports distribution of read operations across Gitaly nodes that are configured for the virtual node. @@ -1064,8 +950,10 @@ They reflect configuration defined for this instance of Praefect. > - Introduced in GitLab 13.1 in [alpha](https://about.gitlab.com/handbook/product/gitlab-the-product/#alpha-beta-ga), disabled by default. > - Entered [beta](https://about.gitlab.com/handbook/product/gitlab-the-product/#alpha-beta-ga) in GitLab 13.2, disabled by default. -> - From GitLab 13.3, disabled unless primary-wins reference transactions strategy is disabled. +> - In GitLab 13.3, disabled unless primary-wins voting strategy is disabled. > - From GitLab 13.4, enabled by default. +> - From GitLab 13.5, you must use Git v2.28.0 or higher on Gitaly nodes to enable strong consistency. +> - From GitLab 13.6, primary-wins voting strategy and `gitaly_reference_transactions_primary_wins` feature flag were removed from the source code. Praefect guarantees eventual consistency by replicating all writes to secondary nodes after the write to the primary Gitaly node has happened. @@ -1077,18 +965,12 @@ information, see the [strong consistency epic](https://gitlab.com/groups/gitlab- To enable strong consistency: -- In GitLab 13.5, you must use Git v2.28.0 or higher on Gitaly nodes to enable - strong consistency. -- In GitLab 13.4 and later, the strong consistency voting strategy has been - improved. Instead of requiring all nodes to agree, only the primary and half - of the secondaries need to agree. This strategy is enabled by default. To - disable it and continue using the primary-wins strategy, enable the - `:gitaly_reference_transactions_primary_wins` feature flag. -- In GitLab 13.3, reference transactions are enabled by default with a - primary-wins strategy. This strategy causes all transactions to succeed for - the primary and thus does not ensure strong consistency. To enable strong - consistency, disable the `:gitaly_reference_transactions_primary_wins` - feature flag. +- In GitLab 13.5, you must use Git v2.28.0 or higher on Gitaly nodes to enable strong consistency. +- In GitLab 13.4 and later, the strong consistency voting strategy has been improved and enabled by default. + Instead of requiring all nodes to agree, only the primary and half of the secondaries need to agree. +- In GitLab 13.3, reference transactions are enabled by default with a primary-wins strategy. + This strategy causes all transactions to succeed for the primary and thus does not ensure strong consistency. + To enable strong consistency, disable the `:gitaly_reference_transactions_primary_wins` feature flag. - In GitLab 13.2, enable the `:gitaly_reference_transactions` feature flag. - In GitLab 13.1, enable the `:gitaly_reference_transactions` and `:gitaly_hooks_rpc` feature flags. @@ -1368,8 +1250,9 @@ affected repositories. Praefect provides tools for: - [Automatic](#automatic-reconciliation) reconciliation, for GitLab 13.4 and later. - [Manual](#manual-reconciliation) reconciliation, for: - GitLab 13.3 and earlier. - - Repositories upgraded to GitLab 13.4 and later without entries in the `repositories` table. - A migration tool [is planned](https://gitlab.com/gitlab-org/gitaly/-/issues/3033). + - Repositories upgraded to GitLab 13.4 and later without entries in the `repositories` table. In + GitLab 13.6 and later, [a migration is run](https://gitlab.com/gitlab-org/gitaly/-/issues/3033) + when Praefect starts for these repositories. These tools reconcile the outdated repositories to bring them fully up to date again. @@ -1413,23 +1296,37 @@ sudo /opt/gitlab/embedded/bin/praefect -config /var/opt/gitlab/praefect/config.t - Replace the placeholder `<up-to-date-storage>` with the Gitaly storage name containing up to date repositories. - Replace the placeholder `<outdated-storage>` with the Gitaly storage name containing outdated repositories. -## Migrate existing repositories to Gitaly Cluster +## Migrate to Gitaly Cluster + +To migrate to Gitaly Cluster, existing repositories stored outside Gitaly Cluster must be +moved. There is no automatic migration but the moves can be scheduled with the GitLab API. -If your GitLab instance already has repositories on single Gitaly nodes, these aren't migrated to -Gitaly Cluster automatically. +GitLab repositories can be associated with projects, groups, and snippets. Each of these types +have a separate API to schedule the respective repositories to move. To move all repositories +on a GitLab instance, each of these types must be scheduled to move for each storage. -Project repositories may be moved from one storage location using the [Project repository storage moves API](../../api/project_repository_storage_moves.md). Note that this API cannot move all repository types. For moving other repositories types, see: +Each repository is made read only when the move is scheduled. The repository is not writable +until the move has completed. -- [Snippet repository storage moves API](../../api/snippet_repository_storage_moves.md). -- [Group repository storage moves API](../../api/group_repository_storage_moves.md). +After creating and configuring Gitaly Cluster: -To move repositories to Gitaly Cluster: +1. Ensure all storages are accessible to the GitLab instance. In this example, these are + `<original_storage_name>` and `<cluster_storage_name>`. +1. [Configure repository storage weights](../repository_storage_paths.md#configure-where-new-repositories-are-stored) + so that the Gitaly Cluster receives all new projects. This stops new projects being created + on existing Gitaly nodes while the migration is in progress. +1. Schedule repository moves for: + - [Projects](#bulk-schedule-projects). + - [Snippets](#bulk-schedule-snippets). + - [Groups](#bulk-schedule-groups). **(PREMIUM SELF)** + +### Bulk schedule projects 1. [Schedule repository storage moves for all projects on a storage shard](../../api/project_repository_storage_moves.md#schedule-repository-storage-moves-for-all-projects-on-a-storage-shard) using the API. For example: ```shell curl --request POST --header "Private-Token: <your_access_token>" --header "Content-Type: application/json" \ - --data '{"source_storage_name":"gitaly","destination_storage_name":"praefect"}' "https://gitlab.example.com/api/v4/project_repository_storage_moves" + --data '{"source_storage_name":"<original_storage_name>","destination_storage_name":"<cluster_storage_name>"}' "https://gitlab.example.com/api/v4/project_repository_storage_moves" ``` 1. [Query the most recent repository moves](../../api/project_repository_storage_moves.md#retrieve-all-project-repository-storage-moves) @@ -1442,9 +1339,69 @@ To move repositories to Gitaly Cluster: using the API to confirm that all projects have moved. No projects should be returned with `repository_storage` field set to the old storage. -In a similar way, you can move other repository types by using the -[Snippet repository storage moves API](../../api/snippet_repository_storage_moves.md) **(FREE SELF)** -or the [Groups repository storage moves API](../../api/group_repository_storage_moves.md) **(PREMIUM SELF)**. + ```shell + curl --header "Private-Token: <your_access_token>" --header "Content-Type: application/json" \ + "https://gitlab.example.com/api/v4/projects?repository_storage=<original_storage_name>" + ``` + + Alternatively use [the rails console](../operations/rails_console.md) to + confirm that all projects have moved. Run the following in the rails console: + + ```ruby + ProjectRepository.for_repository_storage('<original_storage_name>') + ``` + +1. Repeat for each storage as required. + +### Bulk schedule snippets + +1. [Schedule repository storage moves for all snippets on a storage shard](../../api/snippet_repository_storage_moves.md#schedule-repository-storage-moves-for-all-snippets-on-a-storage-shard) using the API. For example: + + ```shell + curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" --header "Content-Type: application/json" \ + --data '{"source_storage_name":"<original_storage_name>","destination_storage_name":"<cluster_storage_name>"}' "https://gitlab.example.com/api/v4/snippet_repository_storage_moves" + ``` + +1. [Query the most recent repository moves](../../api/snippet_repository_storage_moves.md#retrieve-all-snippet-repository-storage-moves) + using the API. The query indicates either: + - The moves have completed successfully. The `state` field is `finished`. + - The moves are in progress. Re-query the repository move until it completes successfully. + - The moves have failed. Most failures are temporary and are solved by rescheduling the move. + +1. After the moves are complete, use [the rails console](../operations/rails_console.md) to + confirm that all snippets have moved. No snippets should be returned for the original + storage. Run the following in the rails console: + + ```ruby + SnippetRepository.for_repository_storage('<original_storage_name>') + ``` + +1. Repeat for each storage as required. + +### Bulk schedule groups **(PREMIUM SELF)** + +1. [Schedule repository storage moves for all groups on a storage shard](../../api/group_repository_storage_moves.md#schedule-repository-storage-moves-for-all-groups-on-a-storage-shard) using the API. + + ```shell + curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" --header "Content-Type: application/json" \ + --data '{"source_storage_name":"<original_storage_name>","destination_storage_name":"<cluster_storage_name>"}' "https://gitlab.example.com/api/v4/group_repository_storage_moves" + ``` + +1. [Query the most recent repository moves](../../api/group_repository_storage_moves.md#retrieve-all-group-repository-storage-moves) + using the API. The query indicates either: + - The moves have completed successfully. The `state` field is `finished`. + - The moves are in progress. Re-query the repository move until it completes successfully. + - The moves have failed. Most failures are temporary and are solved by rescheduling the move. + +1. After the moves are complete, use [the rails console](../operations/rails_console.md) to + confirm that all groups have moved. No groups should be returned for the original + storage. Run the following in the rails console: + + ```ruby + GroupWikiRepository.for_repository_storage('<original_storage_name>') + ``` + +1. Repeat for each storage as required. ## Debugging Praefect diff --git a/doc/administration/gitaly/reference.md b/doc/administration/gitaly/reference.md index f08b03017e4..ec5a8d47ae2 100644 --- a/doc/administration/gitaly/reference.md +++ b/doc/administration/gitaly/reference.md @@ -5,7 +5,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w type: reference --- -# Gitaly reference +# Gitaly reference **(FREE SELF)** Gitaly is configured via a [TOML](https://github.com/toml-lang/toml) configuration file. Unlike installations from source, in Omnibus GitLab, you diff --git a/doc/administration/incoming_email.md b/doc/administration/incoming_email.md index 2eb5da7d9ab..22cd6ca097c 100644 --- a/doc/administration/incoming_email.md +++ b/doc/administration/incoming_email.md @@ -220,8 +220,11 @@ Example for source installs: incoming_email: enabled: true - # The email address including the `%{key}` placeholder that will be replaced to reference the item being replied to. - # The placeholder can be omitted but if present, it must appear in the "user" part of the address (before the `@`). + # The email address including the %{key} placeholder that will be replaced to reference the + # item being replied to. This %{key} should be included in its entirety within the email + # address and not replaced by another value. + # For example: emailadress+%{key}@gmail.com. + # The placeholder must appear in the "user" part of the address (before the `@`). address: "incoming+%{key}@gitlab.example.com" # Email account username @@ -612,3 +615,59 @@ incoming_email: # Whether the IMAP server uses SSL ssl: true ``` + +#### Microsoft Graph + +> Introduced in [GitLab 13.11](https://gitlab.com/gitlab-org/gitlab/-/issues/214900). + +GitLab can read incoming email using the Microsoft Graph API instead of +IMAP. Because [Microsoft is deprecating IMAP usage with Basic Authentication](https://techcommunity.microsoft.com/t5/exchange-team-blog/announcing-oauth-2-0-support-for-imap-and-smtp-auth-protocols-in/ba-p/1330432), the Microsoft Graph API will soon be required for new Microsoft Exchange Online +mailboxes. + +To configure GitLab for Microsoft Graph, you will need to register an +OAuth2 application in your Azure Active Directory that has the +`Mail.ReadWrite` permission for all mailboxes. See the [MailRoom step-by-step guide](https://github.com/tpitale/mail_room/#microsoft-graph-configuration) +and [Microsoft instructions](https://docs.microsoft.com/en-us/azure/active-directory/develop/quickstart-register-app) +for more details. + +Record the following when you configure your OAuth2 application: + +- Tenant ID for your Azure Active Directory +- Client ID for your OAuth2 application +- Client secret your OAuth2 application + +##### Restrict mailbox access + +For MailRoom to work as a service account, the application you create +in Azure Active Directory requires that you set the `Mail.ReadWrite` property +to read/write mail in *all* mailboxes. + +To mitigate security concerns, we recommend configuring an application access +policy which limits the mailbox access for all accounts, as described in +[Microsoft documentation](https://docs.microsoft.com/en-us/graph/auth-limit-mailbox-access). + +This example for Omnibus GitLab assumes you're using the following mailbox: `incoming@example.onmicrosoft.com`: + +##### Configure Microsoft Graph + +```ruby +gitlab_rails['incoming_email_enabled'] = true + +# The email address including the `%{key}` placeholder that will be replaced +# to reference the item being replied to. The placeholder can be omitted, but if +# present, it must appear in the "user" part of the address (before the `@`). +gitlab_rails['incoming_email_address'] = "incoming+%{key}@example.onmicrosoft.com" + +# Email account username +gitlab_rails['incoming_email_email'] = "incoming@example.onmicrosoft.com" + +gitlab_rails['incoming_email_inbox_method'] = 'microsoft_graph' +gitlab_rails['incoming_email_inbox_options'] = { + 'tenant_id': '<YOUR-TENANT-ID>', + 'client_id': '<YOUR-CLIENT-ID>', + 'client_secret': '<YOUR-CLIENT-SECRET>', + 'poll_interval': 60 # Optional +} +``` + +The Microsoft Graph API is not yet supported in source installations. See [this issue](https://gitlab.com/gitlab-org/gitlab/-/issues/326169) for more details. diff --git a/doc/administration/index.md b/doc/administration/index.md index 15a3bee4004..a7d81ca1543 100644 --- a/doc/administration/index.md +++ b/doc/administration/index.md @@ -7,7 +7,12 @@ description: 'Learn how to install, configure, update, and maintain your GitLab # Administrator documentation **(FREE SELF)** -Learn how to administer your self-managed GitLab instance. +If you use GitLab.com, only GitLab team members have access to administration tools and settings. +If you use a self-managed GitLab instance, learn how to administer it. + +Only administrator users can access GitLab administration tools and settings. + +## Available distributions GitLab has two product distributions available through [different subscriptions](https://about.gitlab.com/pricing/): @@ -15,17 +20,10 @@ GitLab has two product distributions available through [different subscriptions] - The open core [GitLab Enterprise Edition (EE)](https://gitlab.com/gitlab-org/gitlab). You can [install either GitLab CE or GitLab EE](https://about.gitlab.com/install/ce-or-ee/). -However, the features you have access to depend on your chosen [subscription](https://about.gitlab.com/pricing/). +However, the features you have access to depend on your chosen subscription. GitLab Community Edition installations have access only to Free features. -Non-administrator users can't access GitLab administration tools and settings. - -GitLab.com is administered by GitLab, Inc., and only GitLab team members have -access to its administration tools and settings. Users of GitLab.com should -instead refer to the [User documentation](../user/index.md) for GitLab -configuration and usage documentation. - ## Installing and maintaining GitLab Learn how to install, configure, update, and maintain your GitLab instance. @@ -70,8 +68,8 @@ Learn how to install, configure, update, and maintain your GitLab instance. - [Broadcast Messages](../user/admin_area/broadcast_messages.md): Send messages to GitLab users through the UI. - [Elasticsearch](../integration/elasticsearch.md): Enable Elasticsearch to - empower Advanced Search. Useful when you deal with a huge amount of data. -- [External Classification Policy Authorization](../user/admin_area/settings/external_authorization.md). + empower Advanced Search. Use when you deal with a huge amount of data. +- [External Classification Policy Authorization](../user/admin_area/settings/external_authorization.md) - [Upload a license](../user/admin_area/license.md): Upload a license to unlock features that are in paid tiers of GitLab. - [Admin Area](../user/admin_area/index.md): for self-managed instance-wide @@ -109,15 +107,15 @@ Learn how to install, configure, update, and maintain your GitLab instance. ### Upgrading or downgrading GitLab -- [Upgrade from GitLab CE to GitLab EE](../update/index.md#upgrading-between-editions): learn how to upgrade GitLab Community Edition to GitLab Enterprise Editions. +- [Upgrade from GitLab CE to GitLab EE](../update/index.md#upgrading-between-editions): Learn how to upgrade GitLab Community Edition to GitLab Enterprise Editions. - [Downgrade from GitLab EE to GitLab CE](../downgrade_ee_to_ce/index.md): Learn how to downgrade GitLab Enterprise Editions to Community Edition. ### GitLab platform integrations - [Mattermost](https://docs.gitlab.com/omnibus/gitlab-mattermost/): Integrate with [Mattermost](https://mattermost.com), an open source, private cloud workplace for web messaging. -- [PlantUML](integration/plantuml.md): Create simple diagrams in AsciiDoc and Markdown documents +- [PlantUML](integration/plantuml.md): Create diagrams in AsciiDoc and Markdown documents created in snippets, wikis, and repositories. -- [Web terminals](integration/terminal.md): Provide terminal access to your applications deployed to Kubernetes from within GitLab CI/CD [environments](../ci/environments/index.md#web-terminals). +- [Web terminals](integration/terminal.md): Provide terminal access to your applications deployed to Kubernetes from GitLab CI/CD [environments](../ci/environments/index.md#web-terminals). ## User settings and permissions @@ -126,12 +124,12 @@ Learn how to install, configure, update, and maintain your GitLab instance. - [Sign-up restrictions](../user/admin_area/settings/sign_up_restrictions.md): block email addresses of specific domains, or whitelist only specific domains. - [Access restrictions](../user/admin_area/settings/visibility_and_access_controls.md#enabled-git-access-protocols): Define which Git access protocols can be used to talk to GitLab (SSH, HTTP, HTTPS). - [Authentication and Authorization](auth/README.md): Configure external authentication with LDAP, SAML, CAS, and additional providers. - - [Sync LDAP](auth/ldap/index.md). - - [Kerberos authentication](../integration/kerberos.md). + - [Sync LDAP](auth/ldap/index.md) + - [Kerberos authentication](../integration/kerberos.md) - See also other [authentication](../topics/authentication/index.md#gitlab-administrators) topics (for example, enforcing 2FA). -- [Email users](../tools/email.md): Email GitLab users from within GitLab. +- [Email users](../tools/email.md): Email GitLab users from GitLab. - [User Cohorts](../user/admin_area/user_cohorts.md): Display the monthly cohorts of new users and their activities over time. -- [Audit events](audit_events.md): View the changes made within the GitLab server for: +- [Audit events](audit_events.md): View the changes made on the GitLab server for: - Groups and projects. - Instances. - [Auditor users](auditor_users.md): Users with read-only access to all projects, groups, and other resources on the GitLab instance. @@ -160,7 +158,7 @@ Learn how to install, configure, update, and maintain your GitLab instance. ### Repository settings -- [Repository checks](repository_checks.md): Periodic Git repository checks. +- [Repository checks](repository_checks.md): Check your repository for data corruption. - [Repository storage paths](repository_storage_paths.md): Manage the paths used to store repositories. - [Repository storage types](repository_storage_types.md): Information about the different repository storage types. - [Repository storage Rake tasks](raketasks/storage.md): A collection of Rake tasks to list and migrate existing projects and attachments associated with it from Legacy storage to Hashed storage. @@ -171,7 +169,7 @@ Learn how to install, configure, update, and maintain your GitLab instance. - [Enable/disable GitLab CI/CD](../ci/enable_or_disable_ci.md#site-wide-admin-setting): Enable or disable GitLab CI/CD for your instance. - [GitLab CI/CD administration settings](../user/admin_area/settings/continuous_integration.md): Enable or disable Auto DevOps site-wide and define the artifacts' max size and expiration time. -- [External Pipeline Validation](external_pipeline_validation.md): Enable, disable and configure external pipeline validation. +- [External Pipeline Validation](external_pipeline_validation.md): Enable, disable, and configure external pipeline validation. - [Job artifacts](job_artifacts.md): Enable, disable, and configure job artifacts (a set of files and directories which are outputted by a job when it completes successfully). - [Job logs](job_logs.md): Information about the job logs. - [Register runners](../ci/runners/README.md#types-of-runners): Learn how to register and configure runners. @@ -216,7 +214,7 @@ Learn how to install, configure, update, and maintain your GitLab instance. ## Troubleshooting -- [Debugging tips](troubleshooting/debug.md): Tips to debug problems when things go wrong +- [Debugging tips](troubleshooting/debug.md): Tips to debug problems when things go wrong. - [Log system](logs.md): Where to look for logs. - [Sidekiq Troubleshooting](troubleshooting/sidekiq.md): Debug when Sidekiq appears hung and is not processing jobs. - [Troubleshooting Elasticsearch](troubleshooting/elasticsearch.md) @@ -225,27 +223,27 @@ Learn how to install, configure, update, and maintain your GitLab instance. ### Support Team Docs -The GitLab Support Team has collected a lot of information about troubleshooting GitLab -instances. These documents are normally used by the Support Team itself, or by customers +The GitLab Support Team has collected a lot of information about troubleshooting GitLab. +The following documents are used by the Support Team or by customers with direct guidance from a Support Team member. GitLab administrators may find the -information useful for troubleshooting, but if you are experiencing trouble with your +information useful for troubleshooting. However, if you are experiencing trouble with your GitLab instance, you should check your [support options](https://about.gitlab.com/support/) before referring to these documents. WARNING: -Using the commands listed in the documentation below could result in data loss or -other damage to a GitLab instance, and should only be used by experienced administrators +The commands in the following documentation might result in data loss or +other damage to a GitLab instance. They should be used only by experienced administrators who are aware of the risks. -- [Useful diagnostics tools](troubleshooting/diagnostics_tools.md) -- [Useful Linux commands](troubleshooting/linux_cheat_sheet.md) +- [Diagnostics tools](troubleshooting/diagnostics_tools.md) +- [Linux commands](troubleshooting/linux_cheat_sheet.md) - [Troubleshooting Kubernetes](troubleshooting/kubernetes_cheat_sheet.md) - [Troubleshooting PostgreSQL](troubleshooting/postgresql.md) - [Guide to test environments](troubleshooting/test_environments.md) (for Support Engineers) - [GitLab Rails console commands](troubleshooting/gitlab_rails_cheat_sheet.md) (for Support Engineers) - [Troubleshooting SSL](troubleshooting/ssl.md) -- Useful links: - - [GitLab Developer Docs](../development/README.md) +- Related links: + - [GitLab Developer Documentation](../development/README.md) - [Repairing and recovering broken Git repositories](https://git.seveas.net/repairing-and-recovering-broken-git-repositories.html) - [Testing with OpenSSL](https://www.feistyduck.com/library/openssl-cookbook/online/ch-testing-with-openssl.html) - [`strace` zine](https://wizardzines.com/zines/strace/) diff --git a/doc/administration/instance_limits.md b/doc/administration/instance_limits.md index ac37ee00210..820006eeadf 100644 --- a/doc/administration/instance_limits.md +++ b/doc/administration/instance_limits.md @@ -105,7 +105,7 @@ Limit the maximum daily member invitations allowed per group hierarchy. ## Gitaly concurrency limit -Clone traffic can put a large strain on your Gitaly service. To prevent such workloads from overwhelming your Gitaly server, you can set concurrency limits in Gitaly’s configuration file. +Clone traffic can put a large strain on your Gitaly service. To prevent such workloads from overwhelming your Gitaly server, you can set concurrency limits in Gitaly's configuration file. Read more on [Gitaly concurrency limits](gitaly/configure_gitaly.md#limit-rpc-concurrency). @@ -145,13 +145,14 @@ limited to 1KiB, and descriptions (the rest of the message) will be limited to ## Number of issues in the milestone overview -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/39453) in GitLab 12.10. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/39453) in GitLab 12.10. +> - [Set](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/58168) to 500 in GitLab 13.11. -The maximum number of issues loaded on the milestone overview page is 3000. +The maximum number of issues loaded on the milestone overview page is 500. When the number exceeds the limit the page displays an alert and links to a paginated -[issue list](../user/project/issues/index.md#issues-list) of all issues in the milestone. +[issue list](../user/project/issues/managing_issues.md) of all issues in the milestone. -- **Limit:** 3000 issues +- **Limit:** 500 issues ## Number of pipelines per Git push @@ -376,7 +377,7 @@ Plan.default.actual_limits.update!(ci_instance_level_variables: 30) > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/37226) in GitLab 13.3. -Job artifacts defined with [`artifacts:reports`](../ci/pipelines/job_artifacts.md#artifactsreports) +Job artifacts defined with [`artifacts:reports`](../ci/yaml/README.md#artifactsreports) that are uploaded by the runner are rejected if the file size exceeds the maximum file size limit. The limit is determined by comparing the project's [maximum artifact size setting](../user/admin_area/settings/continuous_integration.md#maximum-artifacts-size) diff --git a/doc/administration/job_artifacts.md b/doc/administration/job_artifacts.md index 3f8aca2f1ff..bec1fa6ad05 100644 --- a/doc/administration/job_artifacts.md +++ b/doc/administration/job_artifacts.md @@ -48,7 +48,7 @@ this is done when the job succeeds, but can also be done on failure, or always, [`artifacts:when`](../ci/yaml/README.md#artifactswhen) parameter. Most artifacts are compressed by GitLab Runner before being sent to the coordinator. The exception to this is -[reports artifacts](../ci/pipelines/job_artifacts.md#artifactsreports), which are compressed after uploading. +[reports artifacts](../ci/yaml/README.md#artifactsreports), which are compressed after uploading. ### Using local storage @@ -89,7 +89,7 @@ _The artifacts are stored by default in > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/1762) in > [GitLab Premium](https://about.gitlab.com/pricing/) 9.4. -> - Since version 9.5, artifacts are [browsable](../ci/pipelines/job_artifacts.md#browsing-artifacts), +> - Since version 9.5, artifacts are [browsable](../ci/pipelines/job_artifacts.md#download-job-artifacts), > when object storage is enabled. 9.4 lacks this feature. > - Since version 10.6, available in [GitLab Free](https://about.gitlab.com/pricing/). > - Since version 11.0, we support `direct_upload` to S3. @@ -509,7 +509,7 @@ If you need to manually remove job artifacts associated with multiple jobs while NOTE: This step also erases artifacts that users have chosen to - ["keep"](../ci/pipelines/job_artifacts.md#browsing-artifacts). + ["keep"](../ci/pipelines/job_artifacts.md#download-job-artifacts). ```ruby builds_to_clear = builds_with_artifacts.where("finished_at < ?", 1.week.ago) @@ -526,6 +526,9 @@ If you need to manually remove job artifacts associated with multiple jobs while - `3.months.ago` - `1.year.ago` + `erase_erasable_artifacts!` is a synchronous method, and upon execution, the artifacts are removed immediately. + They are not scheduled via some background queue. + #### Delete job artifacts and logs from jobs completed before a specific date WARNING: @@ -580,3 +583,40 @@ If you need to manually remove **all** job artifacts associated with multiple jo - `7.days.ago` - `3.months.ago` - `1.year.ago` + +### Error `Downloading artifacts from coordinator... not found` + +When a job tries to download artifacts from an earlier job, you might receive an error similar to: + +```plaintext +Downloading artifacts from coordinator... not found id=12345678 responseStatus=404 Not Found +``` + +This might be caused by a `gitlab.rb` file with the following configuration: + +```ruby +gitlab_rails['artifacts_object_store_background_upload'] = false +gitlab_rails['artifacts_object_store_direct_upload'] = true +``` + +To prevent this, comment out or remove those lines, or switch to their [default values](https://gitlab.com/gitlab-org/omnibus-gitlab/blob/master/files/gitlab-config-template/gitlab.rb.template), +then run `sudo gitlab-ctl reconfigure`. + +### Job artifact upload fails with error 500 + +If you are using object storage for artifacts and a job artifact fails to upload, +you can check: + +- The job log for an error similar to: + + ```plaintext + WARNING: Uploading artifacts as "archive" to coordinator... failed id=12345 responseStatus=500 Internal Server Error status=500 token=abcd1234 + ``` + +- The [workhorse log](logs.md#workhorse-logs) for an error similar to: + + ```json + {"error":"MissingRegion: could not find region configuration","level":"error","msg":"error uploading S3 session","time":"2021-03-16T22:10:55-04:00"} + ``` + +In both cases, you might need to add `region` to the job artifact [object storage configuration](#connection-settings). diff --git a/doc/administration/logs.md b/doc/administration/logs.md index 1950403ce35..289e2cb5362 100644 --- a/doc/administration/logs.md +++ b/doc/administration/logs.md @@ -1,6 +1,6 @@ --- stage: Monitor -group: Health +group: Monitor info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/administration/maintenance_mode/img/maintenance_mode_error_message.png b/doc/administration/maintenance_mode/img/maintenance_mode_error_message.png Binary files differnew file mode 100644 index 00000000000..b2044f9717d --- /dev/null +++ b/doc/administration/maintenance_mode/img/maintenance_mode_error_message.png diff --git a/doc/administration/maintenance_mode/index.md b/doc/administration/maintenance_mode/index.md index 1cdbda05749..437f59247f6 100644 --- a/doc/administration/maintenance_mode/index.md +++ b/doc/administration/maintenance_mode/index.md @@ -67,7 +67,7 @@ The banner can be customized with a specific message. An error is displayed when a user tries to perform a write operation that isn't allowed. - + NOTE: In some cases, the visual feedback from an action could be misleading, for example when starring a project, the **Star** button changes to show the **Unstar** action, however, this is only the frontend update, and it doesn't take into account the failed status of the POST request. These visual bugs are to be fixed [in follow-up iterations](https://gitlab.com/gitlab-org/gitlab/-/issues/295197). diff --git a/doc/administration/maintenance_mode/maintenance_mode_error_message.png b/doc/administration/maintenance_mode/maintenance_mode_error_message.png Binary files differdeleted file mode 100644 index 4b422a719ca..00000000000 --- a/doc/administration/maintenance_mode/maintenance_mode_error_message.png +++ /dev/null diff --git a/doc/administration/monitoring/github_imports.md b/doc/administration/monitoring/github_imports.md index 736ff299a86..cd35b8b3f9e 100644 --- a/doc/administration/monitoring/github_imports.md +++ b/doc/administration/monitoring/github_imports.md @@ -1,6 +1,6 @@ --- stage: Monitor -group: Health +group: Monitor info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/administration/monitoring/gitlab_self_monitoring_project/index.md b/doc/administration/monitoring/gitlab_self_monitoring_project/index.md index f389a2d2176..8d3c8555660 100644 --- a/doc/administration/monitoring/gitlab_self_monitoring_project/index.md +++ b/doc/administration/monitoring/gitlab_self_monitoring_project/index.md @@ -1,6 +1,6 @@ --- stage: Monitor -group: Health +group: Monitor info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- @@ -90,7 +90,7 @@ You can add custom metrics in the self monitoring project by: ## Troubleshooting -### Getting error message in logs: `Could not create instance administrators group. Errors: ["You don’t have permission to create groups."]` +### Getting error message in logs: `Could not create instance administrators group. Errors: ["You don't have permission to create groups."]` There is [a bug](https://gitlab.com/gitlab-org/gitlab/-/issues/208676) which causes project creation to fail with the following error (which appears in the log file) @@ -98,7 +98,7 @@ when the first administrator user is an [external user](../../../user/permissions.md#external-users): ```plaintext -Could not create instance administrators group. Errors: ["You don’t have permission to create groups."] +Could not create instance administrators group. Errors: ["You don't have permission to create groups."] ``` Run the following in a Rails console to check if the first administrator user is an external user: diff --git a/doc/administration/monitoring/index.md b/doc/administration/monitoring/index.md index d2e5f40b171..4c49efd6bd5 100644 --- a/doc/administration/monitoring/index.md +++ b/doc/administration/monitoring/index.md @@ -1,6 +1,6 @@ --- stage: Monitor -group: Health +group: Monitor info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- @@ -25,5 +25,5 @@ Explore our features to monitor your GitLab instance: - [`nginx_status`](https://docs.gitlab.com/omnibus/settings/nginx.html#enablingdisabling-nginx_status): Monitor your NGINX server status. - [Auto Monitoring](../../topics/autodevops/stages.md#auto-monitoring): Automated - monitoring for your application’s server and response metrics, provided by + monitoring for your application's server and response metrics, provided by [Auto DevOps](../../topics/autodevops/index.md). diff --git a/doc/administration/monitoring/ip_whitelist.md b/doc/administration/monitoring/ip_whitelist.md index 91f9c5d560f..522267ce362 100644 --- a/doc/administration/monitoring/ip_whitelist.md +++ b/doc/administration/monitoring/ip_whitelist.md @@ -1,6 +1,6 @@ --- stage: Monitor -group: Health +group: Monitor info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/administration/monitoring/performance/gitlab_configuration.md b/doc/administration/monitoring/performance/gitlab_configuration.md index a0f170e93d9..6e7557854ad 100644 --- a/doc/administration/monitoring/performance/gitlab_configuration.md +++ b/doc/administration/monitoring/performance/gitlab_configuration.md @@ -1,6 +1,6 @@ --- stage: Monitor -group: Health +group: Monitor info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/administration/monitoring/performance/grafana_configuration.md b/doc/administration/monitoring/performance/grafana_configuration.md index 62d4f3c2625..ac322f3e1ef 100644 --- a/doc/administration/monitoring/performance/grafana_configuration.md +++ b/doc/administration/monitoring/performance/grafana_configuration.md @@ -1,6 +1,6 @@ --- stage: Monitor -group: Health +group: Monitor info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/administration/monitoring/performance/index.md b/doc/administration/monitoring/performance/index.md index cdf78811092..15c54a36f6c 100644 --- a/doc/administration/monitoring/performance/index.md +++ b/doc/administration/monitoring/performance/index.md @@ -1,6 +1,6 @@ --- stage: Monitor -group: Health +group: Monitor info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/administration/monitoring/performance/performance_bar.md b/doc/administration/monitoring/performance/performance_bar.md index 39e5e4f6c00..f6aa60b36a1 100644 --- a/doc/administration/monitoring/performance/performance_bar.md +++ b/doc/administration/monitoring/performance/performance_bar.md @@ -1,6 +1,6 @@ --- stage: Monitor -group: Health +group: Monitor info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/administration/monitoring/performance/request_profiling.md b/doc/administration/monitoring/performance/request_profiling.md index ee035c6ad7a..15a58456e05 100644 --- a/doc/administration/monitoring/performance/request_profiling.md +++ b/doc/administration/monitoring/performance/request_profiling.md @@ -1,6 +1,6 @@ --- stage: Monitor -group: Health +group: Monitor info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/administration/monitoring/prometheus/gitlab_exporter.md b/doc/administration/monitoring/prometheus/gitlab_exporter.md index 589fa799cca..4ba4cad9143 100644 --- a/doc/administration/monitoring/prometheus/gitlab_exporter.md +++ b/doc/administration/monitoring/prometheus/gitlab_exporter.md @@ -1,6 +1,6 @@ --- stage: Monitor -group: Health +group: Monitor info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/administration/monitoring/prometheus/gitlab_metrics.md b/doc/administration/monitoring/prometheus/gitlab_metrics.md index 3c6c984cbe3..9f87192aab0 100644 --- a/doc/administration/monitoring/prometheus/gitlab_metrics.md +++ b/doc/administration/monitoring/prometheus/gitlab_metrics.md @@ -1,6 +1,6 @@ --- stage: Monitor -group: Health +group: Monitor info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- @@ -139,6 +139,18 @@ The following metrics can be controlled by feature flags: | `gitlab_method_call_duration_seconds` | `prometheus_metrics_method_instrumentation` | | `gitlab_view_rendering_duration_seconds` | `prometheus_metrics_view_instrumentation` | +## Praefect metrics + +You can [configure Praefect to report metrics](../../gitaly/praefect.md#praefect). +These are some of the Praefect metrics served from the `/metrics` path on the [configured port](index.md#changing-the-port-and-address-prometheus-listens-on) +(9652 by default). + +| Metric | Type | Since | Description | Labels | +| :----- | :--- | ----: | :---------- | :----- | +| `gitaly_praefect_replication_latency_bucket` | Histogram | 12.10 | The amount of time it takes for replication to complete once the replication job starts. | | +| `gitaly_praefect_replication_delay_bucket` | Histogram | 12.10 | A measure of how much time passes between when the replication job is created and when it starts. | | +| `gitaly_praefect_node_latency_bucket` | Histogram | 12.10 | The latency in Gitaly returning health check information to Praefect. This indicates Praefect connection saturation. | | + ## Sidekiq metrics Sidekiq jobs may also gather metrics, and these metrics can be accessed if the @@ -238,10 +250,11 @@ configuration option in `gitlab.yml`. These metrics are served from the The following metrics are available: -| Metric | Type | Since | Description | -|:--------------------------------- |:--------- |:------------------------------------------------------------- |:-------------------------------------- | -| `db_load_balancing_hosts` | Gauge | [12.3](https://gitlab.com/gitlab-org/gitlab/-/issues/13630) | Current number of load balancing hosts | - +| Metric | Type | Since | Description | Labels | +|:--------------------------------- |:--------- |:------------------------------------------------------------- |:-------------------------------------- |:--------------------------------------------------------- | +| `db_load_balancing_hosts` | Gauge | [12.3](https://gitlab.com/gitlab-org/gitlab/-/issues/13630) | Current number of load balancing hosts | | +| `sidekiq_load_balancing_count` | Counter | 13.11 | Sidekiq jobs using load balancing with data consistency set to :sticky or :delayed | `queue`, `boundary`, `external_dependencies`, `feature_category`, `job_status`, `urgency`, `data_consistency`, `database_chosen` | + ## Database partitioning metrics **(PREMIUM SELF)** The following metrics are available: diff --git a/doc/administration/monitoring/prometheus/index.md b/doc/administration/monitoring/prometheus/index.md index 23bffae99cf..035c5b3ee7e 100644 --- a/doc/administration/monitoring/prometheus/index.md +++ b/doc/administration/monitoring/prometheus/index.md @@ -1,6 +1,6 @@ --- stage: Monitor -group: Health +group: Monitor info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/administration/monitoring/prometheus/node_exporter.md b/doc/administration/monitoring/prometheus/node_exporter.md index 6efc9d67e8d..68d997d7596 100644 --- a/doc/administration/monitoring/prometheus/node_exporter.md +++ b/doc/administration/monitoring/prometheus/node_exporter.md @@ -1,6 +1,6 @@ --- stage: Monitor -group: Health +group: Monitor info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/administration/monitoring/prometheus/pgbouncer_exporter.md b/doc/administration/monitoring/prometheus/pgbouncer_exporter.md index df2aa08efaa..d42c471ac71 100644 --- a/doc/administration/monitoring/prometheus/pgbouncer_exporter.md +++ b/doc/administration/monitoring/prometheus/pgbouncer_exporter.md @@ -1,6 +1,6 @@ --- stage: Monitor -group: Health +group: Monitor info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/administration/monitoring/prometheus/postgres_exporter.md b/doc/administration/monitoring/prometheus/postgres_exporter.md index 8b6d3d82018..783030a9220 100644 --- a/doc/administration/monitoring/prometheus/postgres_exporter.md +++ b/doc/administration/monitoring/prometheus/postgres_exporter.md @@ -1,6 +1,6 @@ --- stage: Monitor -group: Health +group: Monitor info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/administration/monitoring/prometheus/redis_exporter.md b/doc/administration/monitoring/prometheus/redis_exporter.md index e9d656a6493..6cc262842a1 100644 --- a/doc/administration/monitoring/prometheus/redis_exporter.md +++ b/doc/administration/monitoring/prometheus/redis_exporter.md @@ -1,6 +1,6 @@ --- stage: Monitor -group: Health +group: Monitor info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/administration/monitoring/prometheus/registry_exporter.md b/doc/administration/monitoring/prometheus/registry_exporter.md index d2bd86aa908..87b51aaed08 100644 --- a/doc/administration/monitoring/prometheus/registry_exporter.md +++ b/doc/administration/monitoring/prometheus/registry_exporter.md @@ -1,6 +1,6 @@ --- stage: Monitor -group: Health +group: Monitor info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/administration/nfs.md b/doc/administration/nfs.md index 59b18f7a74a..52948732766 100644 --- a/doc/administration/nfs.md +++ b/doc/administration/nfs.md @@ -14,17 +14,19 @@ Pages](https://gitlab.com/gitlab-org/gitlab-pages/-/issues/196). For data objects such as LFS, Uploads, Artifacts, etc., an [Object Storage service](object_storage.md) is recommended over NFS where possible, due to better performance. -WARNING: -From GitLab 13.0, using NFS for Git repositories is deprecated. -From GitLab 14.0, technical support for NFS for Git repositories -will no longer be provided. Upgrade to [Gitaly Cluster](gitaly/praefect.md) -as soon as possible. - File system performance can impact overall GitLab performance, especially for actions that read or write to Git repositories. For steps you can use to test file system performance, see [File system Performance Benchmarking](operations/filesystem_benchmarking.md). +## Gitaly and NFS deprecation + +WARNING: +From GitLab 14.0, enhancements and bug fixes for NFS for Git repositories will no longer be +considered and customer technical support will be considered out of scope. +[Read more about Gitaly and NFS](gitaly/index.md#nfs-deprecation-notice) and +[the correct mount options to use](#upgrade-to-gitaly-cluster-or-disable-caching-if-experiencing-data-loss). + ## Known kernel version incompatibilities RedHat Enterprise Linux (RHEL) and CentOS v7.7 and v7.8 ship with kernel @@ -169,11 +171,11 @@ apt-get install nfs-common Here is an example snippet to add to `/etc/fstab`: ```plaintext -10.1.0.1:/var/opt/gitlab/.ssh /var/opt/gitlab/.ssh nfs4 defaults,vers=4.1,hard,rsize=1048576,wsize=1048576,noatime,nofail,lookupcache=positive 0 2 -10.1.0.1:/var/opt/gitlab/gitlab-rails/uploads /var/opt/gitlab/gitlab-rails/uploads nfs4 defaults,vers=4.1,hard,rsize=1048576,wsize=1048576,noatime,nofail,lookupcache=positive 0 2 -10.1.0.1:/var/opt/gitlab/gitlab-rails/shared /var/opt/gitlab/gitlab-rails/shared nfs4 defaults,vers=4.1,hard,rsize=1048576,wsize=1048576,noatime,nofail,lookupcache=positive 0 2 -10.1.0.1:/var/opt/gitlab/gitlab-ci/builds /var/opt/gitlab/gitlab-ci/builds nfs4 defaults,vers=4.1,hard,rsize=1048576,wsize=1048576,noatime,nofail,lookupcache=positive 0 2 -10.1.0.1:/var/opt/gitlab/git-data /var/opt/gitlab/git-data nfs4 defaults,vers=4.1,hard,rsize=1048576,wsize=1048576,noatime,nofail,lookupcache=positive 0 2 +10.1.0.1:/var/opt/gitlab/.ssh /var/opt/gitlab/.ssh nfs4 defaults,vers=4.1,hard,rsize=1048576,wsize=1048576,noatime,nofail,_netdev,lookupcache=positive 0 2 +10.1.0.1:/var/opt/gitlab/gitlab-rails/uploads /var/opt/gitlab/gitlab-rails/uploads nfs4 defaults,vers=4.1,hard,rsize=1048576,wsize=1048576,noatime,nofail,_netdev,lookupcache=positive 0 2 +10.1.0.1:/var/opt/gitlab/gitlab-rails/shared /var/opt/gitlab/gitlab-rails/shared nfs4 defaults,vers=4.1,hard,rsize=1048576,wsize=1048576,noatime,nofail,_netdev,lookupcache=positive 0 2 +10.1.0.1:/var/opt/gitlab/gitlab-ci/builds /var/opt/gitlab/gitlab-ci/builds nfs4 defaults,vers=4.1,hard,rsize=1048576,wsize=1048576,noatime,nofail,_netdev,lookupcache=positive 0 2 +10.1.0.1:/var/opt/gitlab/git-data /var/opt/gitlab/git-data nfs4 defaults,vers=4.1,hard,rsize=1048576,wsize=1048576,noatime,nofail,_netdev,lookupcache=positive 0 2 ``` You can view information and options set for each of the mounted NFS file @@ -187,6 +189,8 @@ Note there are several options that you should consider using: | `nofail` | Don't halt boot process waiting for this mount to become available | `lookupcache=positive` | Tells the NFS client to honor `positive` cache results but invalidates any `negative` cache results. Negative cache results cause problems with Git. Specifically, a `git push` can fail to register uniformly across all NFS clients. The negative cache causes the clients to 'remember' that the files did not exist previously. | `hard` | Instead of `soft`. [Further details](#soft-mount-option). +| `cto` | `cto` is the default option, which you should use. Do not use `nocto`. [Further details](#nocto-mount-option). +| `_netdev` | Wait to mount filesystem until network is online. See also the [`high_availability['mountpoint']`](https://docs.gitlab.com/omnibus/settings/configuration.html#only-start-omnibus-gitlab-services-after-a-given-file-system-is-mounted) option. #### `soft` mount option @@ -223,6 +227,25 @@ the mount point. Use `SIGKILL` (`kill -9`) to deal with hung processes. The `intr` option [stopped working in the 2.6 kernel](https://access.redhat.com/solutions/157873). +#### `nocto` mount option + +Do not use `nocto`. Instead, use `cto`, which is the default. + +When using `nocto`, the dentry cache is always used, up to `acdirmax` seconds (attribute cache time) from the time it's created. + +This results in stale dentry cache issues with multiple clients, where each client can see a different (cached) +version of a directory. + +From the [Linux man page](https://linux.die.net/man/5/nfs), the important parts: + +> If the nocto option is specified, the client uses a non-standard heuristic to determine when files on the server have changed. +> +> Using the nocto option may improve performance for read-only mounts, but should be used only if the data on the server changes only occasionally. + +We have noticed this behavior in an issue about [refs not found after a push](https://gitlab.com/gitlab-org/gitlab/-/issues/326066), +where newly added loose refs can be seen as missing on a different client with a local dentry cache, as +[described in this issue](https://gitlab.com/gitlab-org/gitlab/-/issues/326066#note_539436931). + ### A single NFS mount It's recommended to nest all GitLab data directories within a mount, that allows automatic @@ -346,12 +369,18 @@ sudo ufw allow from <client_ip_address> to any port nfs ### Upgrade to Gitaly Cluster or disable caching if experiencing data loss WARNING: -From GitLab 13.0, using NFS for Git repositories is deprecated. In GitLab 14.0, -support for NFS for Git repositories is scheduled to be removed. Upgrade to -[Gitaly Cluster](gitaly/praefect.md) as soon as possible. +From GitLab 13.0, using NFS for Git repositories is deprecated. +As of GitLab 14.0, NFS-related issues with Gitaly will no longer be addressed. Read +more about [Gitaly and NFS deprecation](gitaly/index.md#nfs-deprecation-notice). Customers and users have reported data loss on high-traffic repositories when using NFS for Git repositories. -For example, we have seen [inconsistent updates after a push](https://gitlab.com/gitlab-org/gitaly/-/issues/2589). The problem may be partially mitigated by adjusting caching using the following NFS client mount options: +For example, we have seen: + +- [Inconsistent updates after a push](https://gitlab.com/gitlab-org/gitaly/-/issues/2589). +- `git ls-remote` [returning the wrong (or no branches)](https://gitlab.com/gitlab-org/gitaly/-/issues/3083) +causing Jenkins to intermittently re-run all pipelines for a repository. + +The problem may be partially mitigated by adjusting caching using the following NFS client mount options: | Setting | Description | | ------- | ----------- | @@ -362,7 +391,7 @@ For example, we have seen [inconsistent updates after a push](https://gitlab.com WARNING: The `actimeo=0` and `noac` options both result in a significant reduction in performance, possibly leading to timeouts. You may be able to avoid timeouts and data loss using `actimeo=0` and `lookupcache=positive` _without_ `noac`, however -we expect the performance reduction will still be significant. As noted above, we strongly recommend upgrading to +we expect the performance reduction will still be significant. Upgrade to [Gitaly Cluster](gitaly/praefect.md) as soon as possible. ### Avoid using cloud-based file systems diff --git a/doc/administration/operations/extra_sidekiq_processes.md b/doc/administration/operations/extra_sidekiq_processes.md index d07afb3bb14..88e3369e25e 100644 --- a/doc/administration/operations/extra_sidekiq_processes.md +++ b/doc/administration/operations/extra_sidekiq_processes.md @@ -75,7 +75,7 @@ To start multiple processes: When `sidekiq-cluster` is only running on a single node, make sure that at least one process is running on all queues using `*`. This means a process will - automatically pick up jobs in queues created in the future. + This includes queues that have dedicated processes. If `sidekiq-cluster` is running on more than one node, you can also use [`--negate`](#negate-settings) and list all the queues that are already being diff --git a/doc/administration/operations/moving_repositories.md b/doc/administration/operations/moving_repositories.md index fc39d8e2074..fba20da9aea 100644 --- a/doc/administration/operations/moving_repositories.md +++ b/doc/administration/operations/moving_repositories.md @@ -28,7 +28,7 @@ For more information, see: querying and scheduling snippet repository moves. - [The API documentation](../../api/group_repository_storage_moves.md) details the endpoints for querying and scheduling group repository moves **(PREMIUM SELF)**. -- [Migrate existing repositories to Gitaly Cluster](../gitaly/praefect.md#migrate-existing-repositories-to-gitaly-cluster). +- [Migrate to Gitaly Cluster](../gitaly/praefect.md#migrate-to-gitaly-cluster). ## Migrating to another GitLab instance diff --git a/doc/administration/packages/container_registry.md b/doc/administration/packages/container_registry.md index 2f6b6a2f629..853a1884821 100644 --- a/doc/administration/packages/container_registry.md +++ b/doc/administration/packages/container_registry.md @@ -4,11 +4,7 @@ group: Package info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# GitLab Container Registry administration - -> - [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/4040) in GitLab 8.8. -> - Container Registry manifest `v1` support was added in GitLab 8.9 to support -> Docker versions earlier than 1.10. +# GitLab Container Registry administration **(FREE SELF)** With the GitLab Container Registry, every project can have its own space to store Docker images. @@ -832,8 +828,6 @@ no longer directly accessible via the `:latest` tag. ### Recycling unused tags -> [Introduced](https://gitlab.com/gitlab-org/omnibus-gitlab/-/merge_requests/987) in Omnibus GitLab 8.12. - Before you run the built-in command, note the following: - The built-in command stops the registry before it starts the garbage collection. @@ -910,8 +904,6 @@ When the command is used without the `-m` flag, the Container Registry only remo ### Performing garbage collection without downtime -> [Introduced](https://gitlab.com/gitlab-org/omnibus-gitlab/-/merge_requests/764) in GitLab 8.8. - You can perform garbage collection without stopping the Container Registry by putting it in read-only mode and by not using the built-in command. On large instances this could require Container Registry to be in read-only mode for a while. diff --git a/doc/administration/packages/dependency_proxy.md b/doc/administration/packages/dependency_proxy.md index 720734bf344..b454728cc8b 100644 --- a/doc/administration/packages/dependency_proxy.md +++ b/doc/administration/packages/dependency_proxy.md @@ -4,7 +4,7 @@ group: Package info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# GitLab Dependency Proxy administration +# GitLab Dependency Proxy administration **(FREE SELF)** > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/7934) in [GitLab Premium](https://about.gitlab.com/pricing/) 11.11. > - [Moved](https://gitlab.com/gitlab-org/gitlab/-/issues/273655) to [GitLab Free](https://about.gitlab.com/pricing/) in GitLab 13.6. diff --git a/doc/administration/packages/index.md b/doc/administration/packages/index.md index 012afd6a419..5ddf47d9b33 100644 --- a/doc/administration/packages/index.md +++ b/doc/administration/packages/index.md @@ -4,7 +4,7 @@ group: Package info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# GitLab Package Registry administration +# GitLab Package Registry administration **(FREE SELF)** GitLab Packages allows organizations to use GitLab as a private repository for a variety of common package managers. Users are able to build and publish diff --git a/doc/administration/pages/index.md b/doc/administration/pages/index.md index 058445f00be..d04688dab7a 100644 --- a/doc/administration/pages/index.md +++ b/doc/administration/pages/index.md @@ -171,7 +171,7 @@ URL scheme: `https://<namespace>.example.io/<project_slug>` NGINX proxies all requests to the daemon. Pages daemon doesn't listen to the outside world. -1. Place the certificate and key inside `/etc/gitlab/ssl` +1. Place the `example.io` certificate and key inside `/etc/gitlab/ssl`. 1. In `/etc/gitlab/gitlab.rb` specify the following configuration: ```ruby @@ -180,7 +180,7 @@ outside world. pages_nginx['redirect_http_to_https'] = true ``` -1. If you haven’t named your certificate and key `example.io.crt` and `example.io.key` +1. If you haven't named your certificate and key `example.io.crt` and `example.io.key` then you'll need to also add the full paths as shown below: ```ruby @@ -189,6 +189,9 @@ then you'll need to also add the full paths as shown below: ``` 1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). +1. If you're using [Pages Access Control](#access-control), update the redirect URI in the GitLab Pages +[System OAuth application](../../integration/oauth_provider.md#instance-wide-applications) +to use the HTTPS protocol. WARNING: Multiple wildcards for one instance is not supported. Only one wildcard per instance can be assigned. @@ -303,7 +306,7 @@ In that case, the Pages daemon is running, NGINX still proxies requests to the daemon but the daemon is also able to receive requests from the outside world. Custom domains are supported, but no TLS. -1. Edit `/etc/gitlab/gitlab.rb`: +1. In `/etc/gitlab/gitlab.rb` specify the following configuration: ```ruby pages_external_url "http://example.io" @@ -334,23 +337,35 @@ In that case, the Pages daemon is running, NGINX still proxies requests to the daemon but the daemon is also able to receive requests from the outside world. Custom domains and TLS are supported. -1. Edit `/etc/gitlab/gitlab.rb`: +1. Place the `example.io` certificate and key inside `/etc/gitlab/ssl`. +1. In `/etc/gitlab/gitlab.rb` specify the following configuration: ```ruby pages_external_url "https://example.io" nginx['listen_addresses'] = ['192.0.2.1'] pages_nginx['enable'] = false - gitlab_pages['cert'] = "/etc/gitlab/ssl/example.io.crt" - gitlab_pages['cert_key'] = "/etc/gitlab/ssl/example.io.key" gitlab_pages['external_http'] = ['192.0.2.2:80', '[2001:db8::2]:80'] gitlab_pages['external_https'] = ['192.0.2.2:443', '[2001:db8::2]:443'] + # Redirect pages from HTTP to HTTPS + gitlab_pages['redirect_http'] = true ``` where `192.0.2.1` is the primary IP address that GitLab is listening to and `192.0.2.2` and `2001:db8::2` are the secondary IPs where the GitLab Pages daemon listens on. If you don't have IPv6, you can omit the IPv6 address. +1. If you haven't named your certificate and key `example.io.crt` and `example.io.key` respectively, +then you'll need to also add the full paths as shown below: + + ```ruby + gitlab_pages['cert'] = "/etc/gitlab/ssl/example.io.crt" + gitlab_pages['cert_key'] = "/etc/gitlab/ssl/example.io.key" + ``` + 1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). +1. If you're using [Pages Access Control](#access-control), update the redirect URI in the GitLab Pages +[System OAuth application](../../integration/oauth_provider.md#instance-wide-applications) +to use the HTTPS protocol. ### Custom domain verification @@ -472,7 +487,7 @@ internet connectivity is gated by a proxy. To use a proxy for GitLab Pages: ### Using a custom Certificate Authority (CA) When using certificates issued by a custom CA, [Access Control](../../user/project/pages/pages_access_control.md#gitlab-pages-access-control) and -the [online view of HTML job artifacts](../../ci/pipelines/job_artifacts.md#browsing-artifacts) +the [online view of HTML job artifacts](../../ci/pipelines/job_artifacts.md#download-job-artifacts) fails to work if the custom CA is not recognized. This usually results in this error: @@ -600,8 +615,9 @@ the below steps to do a no downtime transfer to a new storage location. ## Configure listener for reverse proxy requests -Follow the steps below to configure the proxy listener of GitLab Pages. [Introduced](https://gitlab.com/gitlab-org/omnibus-gitlab/-/merge_requests/2533) in -Omnibus GitLab 11.1. +> [Introduced](https://gitlab.com/gitlab-org/omnibus-gitlab/-/merge_requests/2533) in Omnibus GitLab 11.1. + +Follow the steps below to configure the proxy listener of GitLab Pages. 1. By default the listener is configured to listen for requests on `localhost:8090`. @@ -631,6 +647,9 @@ The default is 100MB. > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/16610) in GitLab 12.7. +NOTE: +Only GitLab admin users will be able to view and override the **Maximum size of Pages** setting. + To override the global maximum pages size for a specific project: 1. Go to your project's **Settings > Pages** page. @@ -752,7 +771,7 @@ The default value for Omnibus installations is `nil`. If left unchanged, GitLab Pages tries to use any available source (either `gitlab` or `disk`). The preferred source is `gitlab`, which uses [API-based configuration](#gitlab-api-based-configuration). -On large GitLab instances, using the API-based configuration will significantly improve the pages daemon startup time, as there is no need to load all custom domains configuration into memory. +On large GitLab instances, using the API-based configuration will significantly improve the pages daemon startup time, as there is no need to load all custom domains configuration into memory. For more details see this [blog post](https://about.gitlab.com/blog/2020/08/03/how-gitlab-pages-uses-the-gitlab-api-to-serve-content/). @@ -815,7 +834,7 @@ or persistent errors, or the Pages Daemon serving old content. NOTE: Expiry, interval and timeout flags use [Golang's duration formatting](https://golang.org/pkg/time/#ParseDuration). A duration string is a possibly signed sequence of decimal numbers, -each with optional fraction and a unit suffix, such as "300ms", "1.5h" or "2h45m". +each with optional fraction and a unit suffix, such as "300ms", "1.5h" or "2h45m". Valid time units are "ns", "us" (or "µs"), "ms", "s", "m", "h". Examples: @@ -919,6 +938,89 @@ In installations from source: 1. Save the file and [restart GitLab](../restart_gitlab.md#installations-from-source) for the changes to take effect. +## ZIP storage + +In GitLab 14.0 the underlaying storage format of GitLab Pages is changing from +files stored directly in disk to a single ZIP archive per project. + +These ZIP archives can be stored either locally on disk storage or on the [object storage](#using-object-storage) if it is configured. + +[Starting from GitLab 13.5](https://gitlab.com/gitlab-org/gitlab/-/issues/245308) ZIP archives are stored every time pages site is updated. + +### Migrate legacy storage to ZIP storage + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/59003) in GitLab 13.11. + +GitLab will [try to automatically migrate](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/54578) the old storage format to the new ZIP-based one when you upgrade to GitLab 13.11 or further. +However, some projects may fail to be migrated for different reasons. +To verify that all projects have been migrated successfully, you can manually run the migration: + +```shell +gitlab-rake gitlab:pages:migrate_legacy_storage +``` + +It's safe to interrupt this task and run it multiple times. + +There are two most common problems this task can report: + +- `Missing public directory` error: + + ```txt + E, [2021-04-09T13:11:52.534768 #911919] ERROR -- : project_id: 1 /home/vlad/gdk/gitlab/shared/pages/gitlab-org/gitlab-test failed to be migrated in 0.07 seconds: Archive not created. Missing public directory in /home/vlad/gdk/gitlab/shared/pages/gitlab-org/gitlab-test + ``` + + In this case, you should verify that these projects don't have pages deployed, and re-run the migration with an additional flag to mark those projects as not deployed with GitLab Pages: + + ```shell + sudo PAGES_MIGRATION_MARK_PROJECTS_AS_NOT_DEPLOYED=true gitlab-rake gitlab:pages:migrate_legacy_storage + ``` + +- File `is invalid` error: + + ```txt + E, [2021-04-09T14:43:05.821767 #923322] ERROR -- : project_id: 1 /home/vlad/gdk/gitlab/shared/pages/gitlab-org/gitlab-test failed to be migrated: /home/vlad/gdk/gitlab/shared/pages/gitlab-org/gitlab-test/public/link is invalid, input_dir: /home/vlad/gdk/gitlab/shared/pages/gitlab-org/gitlab-test + ``` + + This error indicates invalid files on disk storage, most commonly symlinks leading outside of the `public` directory. + You can manually remove these files, or just ignore them during migration: + + ```shell + sudo PAGES_MIGRATION_IGNORE_INVALID_ENTRIES=true gitlab-rake gitlab:pages:migrate_legacy_storage + ``` + +### Rolling back ZIP migration + +If you find that migrated data is invalid, you can remove all migrated data by running: + +```shell +sudo gitlab-rake gitlab:pages:clean_migrated_zip_storage +``` + +This will not remove any data from the legacy disk storage and the GitLab Pages daemon will automatically fallback +to using that. + +### Migrate Pages deployments to object storage + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/325285) in GitLab 13.11 + +Existing Pages deployments objects (which store [ZIP archives](#zip-storage)) can similarly be +migrated to [object storage](#using-object-storage), if +you've been having them stored locally. + +Migrate your existing Pages deployments from local storage to object storage: + +```shell +sudo gitlab-rails gitlab:pages:deployments:migrate_to_object_storage +``` + +### Rolling Pages deployments back to local storage + +After the migration to object storage is performed, you can choose to revert your Pages deployments back to local storage: + +```shell +sudo gitlab-rails gitlab:pages:deployments:migrate_to_local +``` + ## Backup GitLab Pages are part of the [regular backup](../../raketasks/backup_restore.md), so there is no separate backup to configure. @@ -1096,6 +1198,8 @@ If the wildcard DNS [prerequisite](#prerequisites) can't be met, you can still u 1. [Move](../../user/project/settings/index.md#transferring-an-existing-project-into-another-namespace) all projects you need to use Pages with into a single group namespace, for example `pages`. 1. Configure a [DNS entry](#dns-configuration) without the `*.`-wildcard, for example `pages.example.io`. +1. Configure `pages_external_url http://example.io/` in your `gitlab.rb` file. + Omit the group namespace here, because it will automatically be prepended by GitLab. ### Pages daemon fails with permission denied errors @@ -1114,3 +1218,8 @@ gitlab_pages['env'] = {'TMPDIR' => '<new_tmp_path>'} Once added, reconfigure with `sudo gitlab-ctl reconfigure` and restart GitLab with `sudo gitlab-ctl restart`. + +### `The redirect URI included is not valid.` when using Pages Access Control + +Verify that the **Callback URL**/Redirect URI in the GitLab Pages [System OAuth application](../../integration/oauth_provider.md#instance-wide-applications) +is using the protocol (HTTP or HTTPS) that `pages_external_url` is configured to use. diff --git a/doc/administration/pages/source.md b/doc/administration/pages/source.md index d1a8b703860..e1b54e11a1f 100644 --- a/doc/administration/pages/source.md +++ b/doc/administration/pages/source.md @@ -407,7 +407,7 @@ Pages access control is disabled by default. To enable it: ``` 1. [Restart GitLab](../restart_gitlab.md#installations-from-source). -1. Create a new [system OAuth application](../../integration/oauth_provider.md#add-an-application-through-the-profile). +1. Create a new [system OAuth application](../../integration/oauth_provider.md#user-owned-applications). This should be called `GitLab Pages` and have a `Redirect URL` of `https://projects.example.io/auth`. It does not need to be a "trusted" application, but it does need the `api` scope. diff --git a/doc/administration/postgresql/replication_and_failover.md b/doc/administration/postgresql/replication_and_failover.md index a9a38ce1beb..7e25bd69539 100644 --- a/doc/administration/postgresql/replication_and_failover.md +++ b/doc/administration/postgresql/replication_and_failover.md @@ -6,8 +6,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w # PostgreSQL replication and failover with Omnibus GitLab **(PREMIUM SELF)** -This document focuses on configuration supported with [GitLab Premium](https://about.gitlab.com/pricing/), using the Omnibus GitLab package. -If you're a Community Edition or Starter user, consider using a cloud hosted solution. +If you're a Free user of GitLab self-managed, consider using a cloud-hosted solution. This document doesn't cover installations from source. If a setup with replication and failover isn't what you were looking for, see diff --git a/doc/administration/redis/index.md b/doc/administration/redis/index.md index 30618075c8f..23d13491c75 100644 --- a/doc/administration/redis/index.md +++ b/doc/administration/redis/index.md @@ -5,7 +5,7 @@ group: Distribution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# Configuring Redis for scaling +# Configuring Redis for scaling **(FREE SELF)** Based on your infrastructure setup and how you have installed GitLab, there are multiple ways to configure Redis. diff --git a/doc/administration/redis/replication_and_failover_external.md b/doc/administration/redis/replication_and_failover_external.md index 1b2e93cccd9..2716d9bba37 100644 --- a/doc/administration/redis/replication_and_failover_external.md +++ b/doc/administration/redis/replication_and_failover_external.md @@ -7,7 +7,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Redis replication and failover providing your own instance **(FREE SELF)** -If you’re hosting GitLab on a cloud provider, you can optionally use a managed +If you're hosting GitLab on a cloud provider, you can optionally use a managed service for Redis. For example, AWS offers ElastiCache that runs Redis. Alternatively, you may opt to manage your own Redis instance separate from the diff --git a/doc/administration/redis/troubleshooting.md b/doc/administration/redis/troubleshooting.md index 144660c50ea..0c1046ca22d 100644 --- a/doc/administration/redis/troubleshooting.md +++ b/doc/administration/redis/troubleshooting.md @@ -5,7 +5,7 @@ group: Distribution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# Troubleshooting Redis +# Troubleshooting Redis **(FREE SELF)** There are a lot of moving parts that needs to be taken care carefully in order for the HA setup to work as expected. diff --git a/doc/administration/reference_architectures/10k_users.md b/doc/administration/reference_architectures/10k_users.md index a51641f661f..97af1fe8d3c 100644 --- a/doc/administration/reference_architectures/10k_users.md +++ b/doc/administration/reference_architectures/10k_users.md @@ -15,25 +15,31 @@ full list of reference architectures, see > - **High Availability:** Yes ([Praefect](#configure-praefect-postgresql) needs a third-party PostgreSQL solution for HA) > - **Test requests per second (RPS) rates:** API: 200 RPS, Web: 20 RPS, Git (Pull): 20 RPS, Git (Push): 4 RPS -| Service | Nodes | Configuration | GCP | AWS | Azure | -|--------------------------------------------|-------------|-------------------------|-----------------|-------------|----------| -| External load balancing node | 1 | 2 vCPU, 1.8 GB memory | n1-highcpu-2 | c5.large | F2s v2 | -| Consul | 3 | 2 vCPU, 1.8 GB memory | n1-highcpu-2 | c5.large | F2s v2 | -| PostgreSQL | 3 | 8 vCPU, 30 GB memory | n1-standard-8 | m5.2xlarge | D8s v3 | -| PgBouncer | 3 | 2 vCPU, 1.8 GB memory | n1-highcpu-2 | c5.large | F2s v2 | -| Internal load balancing node | 1 | 2 vCPU, 1.8 GB memory | n1-highcpu-2 | c5.large | F2s v2 | -| Redis - Cache | 3 | 4 vCPU, 15 GB memory | n1-standard-4 | m5.xlarge | D4s v3 | -| Redis - Queues / Shared State | 3 | 4 vCPU, 15 GB memory | n1-standard-4 | m5.xlarge | D4s v3 | -| Redis Sentinel - Cache | 3 | 1 vCPU, 1.7 GB memory | g1-small | t3.small | B1MS | -| Redis Sentinel - Queues / Shared State | 3 | 1 vCPU, 1.7 GB memory | g1-small | t3.small | B1MS | -| Gitaly | 3 | 16 vCPU, 60 GB memory | n1-standard-16 | m5.4xlarge | D16s v3 | -| Praefect | 3 | 2 vCPU, 1.8 GB memory | n1-highcpu-2 | c5.large | F2s v2 | -| Praefect PostgreSQL | 1+* | 2 vCPU, 1.8 GB memory | n1-highcpu-2 | c5.large | F2s v2 | -| Sidekiq | 4 | 4 vCPU, 15 GB memory | n1-standard-4 | m5.xlarge | D4s v3 | -| GitLab Rails | 3 | 32 vCPU, 28.8 GB memory | n1-highcpu-32 | c5.9xlarge | F32s v2 | -| Monitoring node | 1 | 4 vCPU, 3.6 GB memory | n1-highcpu-4 | c5.xlarge | F4s v2 | -| Object storage | n/a | n/a | n/a | n/a | n/a | -| NFS server | 1 | 4 vCPU, 3.6 GB memory | n1-highcpu-4 | `c5.xlarge` | F4s v2 | +| Service | Nodes | Configuration | GCP | AWS | Azure | +|--------------------------------------------|-------------|-------------------------|------------------|--------------|-----------| +| External load balancing node | 1 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | +| Consul* | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | +| PostgreSQL* | 3 | 8 vCPU, 30 GB memory | `n1-standard-8` | `m5.2xlarge` | `D8s v3` | +| PgBouncer* | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | +| Internal load balancing node | 1 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | +| Redis - Cache** | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | `D4s v3` | +| Redis - Queues / Shared State** | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | `D4s v3` | +| Redis Sentinel - Cache** | 3 | 1 vCPU, 1.7 GB memory | `g1-small` | `t3.small` | `B1MS` | +| Redis Sentinel - Queues / Shared State** | 3 | 1 vCPU, 1.7 GB memory | `g1-small` | `t3.small` | `B1MS` | +| Gitaly | 3 | 16 vCPU, 60 GB memory | `n1-standard-16` | `m5.4xlarge` | `D16s v3` | +| Praefect | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | +| Praefect PostgreSQL* | 1+ | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | +| Sidekiq | 4 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | `D4s v3` | +| GitLab Rails | 3 | 32 vCPU, 28.8 GB memory | `n1-highcpu-32` | `c5.9xlarge` | `F32s v2` | +| Monitoring node | 1 | 4 vCPU, 3.6 GB memory | `n1-highcpu-4` | `c5.xlarge` | `F4s v2` | +| Object storage | n/a | n/a | n/a | n/a | n/a | +| NFS server | 1 | 4 vCPU, 3.6 GB memory | `n1-highcpu-4` | `c5.xlarge` | `F4s v2` | + +NOTE: +Components marked with * can be optionally run on reputable +third party external PaaS PostgreSQL solutions. Google Cloud SQL and AWS RDS are known to work. +Components marked with ** can be optionally run on reputable +third party external PaaS Redis solutions. Google Memorystore and AWS Elasticache are known to work. ```plantuml @startuml 10k @@ -210,11 +216,12 @@ The following list includes descriptions of each server and its assigned IP: ## Configure the external load balancer -In an active/active GitLab configuration, you'll need a load balancer to route +In a multi-node GitLab configuration, you'll need a load balancer to route traffic to the application servers. The specifics on which load balancer to use -or its exact configuration is beyond the scope of GitLab documentation. We hope +or its exact configuration is beyond the scope of GitLab documentation. We assume that if you're managing multi-node systems like GitLab, you already have a load -balancer of choice. Some load balancer examples include HAProxy (open-source), +balancer of choice and that the routing methods used are distributing calls evenly +between all nodes. Some load balancer examples include HAProxy (open-source), F5 Big-IP LTM, and Citrix Net Scaler. This documentation outline the ports and protocols needed for use with GitLab. @@ -387,6 +394,8 @@ backend praefect ``` Refer to your preferred Load Balancer's documentation for further guidance. +Also ensure that the routing methods used are distributing calls evenly across +all nodes. <div align="right"> <a type="button" class="btn btn-default" href="#setup-components"> @@ -433,7 +442,7 @@ To configure Consul: # Set the network addresses that the exporters will listen on node_exporter['listen_address'] = '0.0.0.0:9100' - # Disable auto migrations + # Prevent database migrations from running on upgrade automatically gitlab_rails['auto_migrate'] = false ``` @@ -557,7 +566,7 @@ in the second step, do not supply the `EXTERNAL_URL` value. # Incoming recommended value for max connections is 500. See https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/5691. patroni['postgresql']['max_connections'] = 500 - # Disable automatic database migrations + # Prevent database migrations from running on upgrade automatically gitlab_rails['auto_migrate'] = false # Configure the Consul agent @@ -853,7 +862,7 @@ a node and change its status from primary to replica (and vice versa). node_exporter['listen_address'] = '0.0.0.0:9100' redis_exporter['listen_address'] = '0.0.0.0:9121' - # Prevent database migrations from running on upgrade + # Prevent database migrations from running on upgrade automatically gitlab_rails['auto_migrate'] = false ``` @@ -920,7 +929,7 @@ You can specify multiple roles, like sentinel and Redis, as: node_exporter['listen_address'] = '0.0.0.0:9100' redis_exporter['listen_address'] = '0.0.0.0:9121' - # Prevent database migrations from running on upgrade + # Prevent database migrations from running on upgrade automatically gitlab_rails['auto_migrate'] = false ``` @@ -1052,7 +1061,7 @@ To configure the Sentinel Cache server: node_exporter['listen_address'] = '0.0.0.0:9100' redis_exporter['listen_address'] = '0.0.0.0:9121' - # Disable auto migrations + # Prevent database migrations from running on upgrade automatically gitlab_rails['auto_migrate'] = false ``` @@ -1117,13 +1126,8 @@ a node and change its status from primary to replica (and vice versa). # Set the network addresses that the exporters will listen on node_exporter['listen_address'] = '0.0.0.0:9100' redis_exporter['listen_address'] = '0.0.0.0:9121' - ``` -1. Only the primary GitLab application server should handle migrations. To - prevent database migrations from running on upgrade, add the following - configuration to your `/etc/gitlab/gitlab.rb` file: - - ```ruby + # Prevent database migrations from running on upgrade automatically gitlab_rails['auto_migrate'] = false ``` @@ -1184,7 +1188,7 @@ You can specify multiple roles, like sentinel and Redis, as: node_exporter['listen_address'] = '0.0.0.0:9100' redis_exporter['listen_address'] = '0.0.0.0:9121' - # Disable auto migrations + # Prevent database migrations from running on upgrade automatically gitlab_rails['auto_migrate'] = false ``` @@ -1316,7 +1320,7 @@ To configure the Sentinel Queues server: node_exporter['listen_address'] = '0.0.0.0:9100' redis_exporter['listen_address'] = '0.0.0.0:9121' - # Disable auto migrations + # Prevent database migrations from running on upgrade automatically gitlab_rails['auto_migrate'] = false ``` @@ -1401,6 +1405,7 @@ in the second step, do not supply the `EXTERNAL_URL` value. postgresql['listen_address'] = '0.0.0.0' postgresql['max_connections'] = 200 + # Prevent database migrations from running on upgrade automatically gitlab_rails['auto_migrate'] = false # Configure the Consul agent @@ -1546,7 +1551,8 @@ To configure the Praefect nodes, on each one: praefect['enable'] = true praefect['listen_addr'] = '0.0.0.0:2305' - gitlab_rails['rake_cache_clear'] = false + # Prevent database migrations from running on upgrade automatically + praefect['auto_migrate'] = false gitlab_rails['auto_migrate'] = false # Configure the Consul agent @@ -1670,8 +1676,7 @@ On each node: alertmanager['enable'] = false prometheus['enable'] = false - # Prevent database connections during 'gitlab-ctl reconfigure' - gitlab_rails['rake_cache_clear'] = false + # Prevent database migrations from running on upgrade automatically gitlab_rails['auto_migrate'] = false # Configure the gitlab-shell API callback URL. Without this, `git push` will @@ -1905,6 +1910,7 @@ To configure the Sidekiq nodes, on each one: gitlab_rails['db_password'] = '<postgresql_user_password>' gitlab_rails['db_adapter'] = 'postgresql' gitlab_rails['db_encoding'] = 'unicode' + # Prevent database migrations from running on upgrade automatically gitlab_rails['auto_migrate'] = false ####################################### @@ -2015,6 +2021,7 @@ On each node perform the following: gitlab_rails['db_host'] = '10.6.0.20' # internal load balancer IP gitlab_rails['db_port'] = 6432 gitlab_rails['db_password'] = '<postgresql_user_password>' + # Prevent database migrations from running on upgrade automatically gitlab_rails['auto_migrate'] = false ## Redis connection details @@ -2210,7 +2217,6 @@ To configure the Monitoring node: external_url 'http://gitlab.example.com' # Disable all other services - gitlab_rails['auto_migrate'] = false alertmanager['enable'] = false gitaly['enable'] = false gitlab_exporter['enable'] = false @@ -2244,6 +2250,9 @@ To configure the Monitoring node: consul['configuration'] = { retry_join: %w(10.6.0.11 10.6.0.12 10.6.0.13) } + + # Prevent database migrations from running on upgrade automatically + gitlab_rails['auto_migrate'] = false ``` 1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). @@ -2338,10 +2347,10 @@ to use GitLab Pages, this currently [requires NFS](troubleshooting.md#gitlab-pag See how to [configure NFS](../nfs.md). WARNING: -From GitLab 13.0, using NFS for Git repositories is deprecated. -From GitLab 14.0, technical support for NFS for Git repositories -will no longer be provided. Upgrade to [Gitaly Cluster](../gitaly/praefect.md) -as soon as possible. +From GitLab 14.0, enhancements and bug fixes for NFS for Git repositories will no longer be +considered and customer technical support will be considered out of scope. +[Read more about Gitaly and NFS](../gitaly/index.md#nfs-deprecation-notice) and +[the correct mount options to use](../nfs.md#upgrade-to-gitaly-cluster-or-disable-caching-if-experiencing-data-loss). <div align="right"> <a type="button" class="btn btn-default" href="#setup-components"> @@ -2349,29 +2358,145 @@ as soon as possible. </a> </div> -## Cloud Native Deployment (optional) +## Cloud Native Hybrid reference architecture with Helm Charts (alternative) + +As an alternative approach, you can also run select components of GitLab as Cloud Native +in Kubernetes via our official [Helm Charts](https://docs.gitlab.com/charts/). +In this setup, we support running the equivalent of GitLab Rails and Sidekiq nodes +in a Kubernetes cluster, named Webservice and Sidekiq respectively. In addition, +the following other supporting services are supported: NGINX, Task Runner, Migrations, +Prometheus and Grafana. Hybrid installations leverage the benefits of both cloud native and traditional -deployments. We recommend shifting the Sidekiq and Webservice components into -Kubernetes to reap cloud native workload management benefits while the others -are deployed using the traditional server method already described. +Kubernetes, you can reap certain cloud native workload management benefits while +the others are deployed in compute VMs with Omnibus as described above in this +page. -The following sections detail this hybrid approach. +NOTE: +This is an **advanced** setup. Running services in Kubernetes is well known +to be complex. **This setup is only recommended** if you have strong working +knowledge and experience in Kubernetes. The rest of this +section will assume this. ### Cluster topology -The following table provides a starting point for hybrid -deployment infrastructure. The recommendations use Google Cloud's Kubernetes Engine (GKE) -and associated machine types, but the memory and CPU requirements should -translate to most other providers. +The following tables and diagram details the hybrid environment using the same formats +as the normal environment above. + +First starting with the components that run in Kubernetes. The recommendations at this +time use Google Cloud’s Kubernetes Engine (GKE) and associated machine types, but the memory +and CPU requirements should translate to most other providers. We hope to update this in the +future with further specific cloud provider details. + +| Service | Nodes | Configuration | GCP | Allocatable CPUs and Memory | +|-------------------------------------------------------|-------|-------------------------|------------------|-----------------------------| +| Webservice | 4 | 32 vCPU, 28.8 GB memory | `n1-standard-32` | 127.5 vCPU, 118 GB memory | +| Sidekiq | 4 | 4 vCPU, 15 GB memory | `n1-standard-4` | 15.5 vCPU, 50 GB memory | +| Supporting services such as NGINX, Prometheus, etc... | 2 | 4 vCPU, 15 GB memory | `n1-standard-4` | 7.75 vCPU, 25 GB memory | + +Next are the backend components that run on static compute VMs via Omnibus (or External PaaS +services where applicable): + +| Service | Nodes | Configuration | GCP | +|--------------------------------------------|-------|-------------------------|------------------| +| Consul* | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | +| PostgreSQL* | 3 | 8 vCPU, 30 GB memory | `n1-standard-8` | +| PgBouncer* | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | +| Internal load balancing node | 1 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | +| Redis - Cache** | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | +| Redis - Queues / Shared State** | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | +| Redis Sentinel - Cache** | 3 | 1 vCPU, 1.7 GB memory | `g1-small` | +| Redis Sentinel - Queues / Shared State** | 3 | 1 vCPU, 1.7 GB memory | `g1-small` | +| Gitaly | 3 | 16 vCPU, 60 GB memory | `n1-standard-16` | +| Praefect | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | +| Praefect PostgreSQL* | 1+ | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | +| Object storage | n/a | n/a | n/a | + +NOTE: +Components marked with * can be optionally run on reputable +third party external PaaS PostgreSQL solutions. Google Cloud SQL and AWS RDS are known to work. +Components marked with ** can be optionally run on reputable +third party external PaaS Redis solutions. Google Memorystore and AWS Elasticache are known to work. + +```plantuml +@startuml 10k + +card "Kubernetes via Helm Charts" as kubernetes { + card "**External Load Balancer**" as elb #6a9be7 + + together { + collections "**Webservice** x4" as gitlab #32CD32 + collections "**Sidekiq** x4" as sidekiq #ff8dd1 + } + + card "**Prometheus + Grafana**" as monitor #7FFFD4 + card "**Supporting Services**" as support +} + +card "**Internal Load Balancer**" as ilb #9370DB +collections "**Consul** x3" as consul #e76a9b + +card "Gitaly Cluster" as gitaly_cluster { + collections "**Praefect** x3" as praefect #FF8C00 + collections "**Gitaly** x3" as gitaly #FF8C00 + card "**Praefect PostgreSQL***\n//Non fault-tolerant//" as praefect_postgres #FF8C00 + + praefect -[#FF8C00]-> gitaly + praefect -[#FF8C00]> praefect_postgres +} + +card "Database" as database { + collections "**PGBouncer** x3" as pgbouncer #4EA7FF + card "**PostgreSQL** (Primary)" as postgres_primary #4EA7FF + collections "**PostgreSQL** (Secondary) x2" as postgres_secondary #4EA7FF + + pgbouncer -[#4EA7FF]-> postgres_primary + postgres_primary .[#4EA7FF]> postgres_secondary +} + +card "redis" as redis { + collections "**Redis Persistent** x3" as redis_persistent #FF6347 + collections "**Redis Cache** x3" as redis_cache #FF6347 + collections "**Redis Persistent Sentinel** x3" as redis_persistent_sentinel #FF6347 + collections "**Redis Cache Sentinel** x3"as redis_cache_sentinel #FF6347 + + redis_persistent <.[#FF6347]- redis_persistent_sentinel + redis_cache <.[#FF6347]- redis_cache_sentinel +} + +cloud "**Object Storage**" as object_storage #white + +elb -[#6a9be7]-> gitlab +elb -[#6a9be7]-> monitor +elb -[hidden]-> support + +gitlab -[#32CD32]> sidekiq +gitlab -[#32CD32]--> ilb +gitlab -[#32CD32]-> object_storage +gitlab -[#32CD32]---> redis +gitlab -[hidden]--> consul + +sidekiq -[#ff8dd1]--> ilb +sidekiq -[#ff8dd1]-> object_storage +sidekiq -[#ff8dd1]---> redis +sidekiq -[hidden]--> consul + +ilb -[#9370DB]-> gitaly_cluster +ilb -[#9370DB]-> database + +consul .[#e76a9b]-> database +consul .[#e76a9b]-> gitaly_cluster +consul .[#e76a9b,norank]--> redis -Machine count | Machine type | Allocatable vCPUs | Allocatable memory (GB) | Purpose --|-|-|-|- -2 | `n1-standard-4` | 7.75 | 25 | Non-GitLab resources, including Grafana, NGINX, and Prometheus -4 | `n1-standard-4` | 15.5 | 50 | GitLab Sidekiq pods -4 | `n1-highcpu-32` | 127.5 | 118 | GitLab Webservice pods +monitor .[#7FFFD4]> consul +monitor .[#7FFFD4]-> database +monitor .[#7FFFD4]-> gitaly_cluster +monitor .[#7FFFD4,norank]--> redis +monitor .[#7FFFD4]> ilb +monitor .[#7FFFD4,norank]u--> elb -"Allocatable" in this table refers to the amount of resources available to workloads deployed in Kubernetes _after_ accounting for the overhead of running Kubernetes itself. +@enduml +``` ### Resource usage settings @@ -2379,29 +2504,31 @@ The following formulas help when calculating how many pods may be deployed withi The [10k reference architecture example values file](https://gitlab.com/gitlab-org/charts/gitlab/-/blob/master/examples/ref/10k.yaml) documents how to apply the calculated configuration to the Helm Chart. +#### Webservice + +Webservice pods typically need about 1 vCPU and 1.25 GB of memory _per worker_. +Each Webservice pod will consume roughly 4 vCPUs and 5 GB of memory using +the [recommended topology](#cluster-topology) because four worker processes +are created by default and each pod has other small processes running. + +For 10k users we recommend a total Puma worker count of around 80. +With the [provided recommendations](#cluster-topology) this allows the deployment of up to 20 +Webservice pods with 4 workers per pod and 5 pods per node. Expand available resources using +the ratio of 1 vCPU to 1.25 GB of memory _per each worker process_ for each additional +Webservice pod. + +For further information on resource usage, see the [Webservice resources](https://docs.gitlab.com/charts/charts/gitlab/webservice/#resources). + #### Sidekiq Sidekiq pods should generally have 1 vCPU and 2 GB of memory. [The provided starting point](#cluster-topology) allows the deployment of up to -16 Sidekiq pods. Expand available resources using the 1vCPU to 2GB memory +16 Sidekiq pods. Expand available resources using the 1 vCPU to 2GB memory ratio for each additional pod. For further information on resource usage, see the [Sidekiq resources](https://docs.gitlab.com/charts/charts/gitlab/sidekiq/#resources). -#### Webservice - -Webservice pods typically need about 1 vCPU and 1.25 GB of memory _per worker_. -Each Webservice pod will consume roughly 2 vCPUs and 2.5 GB of memory using -the [recommended topology](#cluster-topology) because two worker processes -are created by default. - -The [provided recommendations](#cluster-topology) allow the deployment of up to 28 -Webservice pods. Expand available resources using the ratio of 1 vCPU to 1.25 GB of memory -_per each worker process_ for each additional Webservice pod. - -For further information on resource usage, see the [Webservice resources](https://docs.gitlab.com/charts/charts/gitlab/webservice/#resources). - <div align="right"> <a type="button" class="btn btn-default" href="#setup-components"> Back to setup components <i class="fa fa-angle-double-up" aria-hidden="true"></i> diff --git a/doc/administration/reference_architectures/25k_users.md b/doc/administration/reference_architectures/25k_users.md index 8cf9efe1d2c..7f9f284d085 100644 --- a/doc/administration/reference_architectures/25k_users.md +++ b/doc/administration/reference_architectures/25k_users.md @@ -17,23 +17,23 @@ full list of reference architectures, see | Service | Nodes | Configuration | GCP | AWS | Azure | |-----------------------------------------|-------------|-------------------------|-----------------|-------------|----------| -| External load balancing node | 1 | 4 vCPU, 3.6 GB memory | n1-highcpu-4 | c5.xlarge | F4s v2 | -| Consul | 3 | 2 vCPU, 1.8 GB memory | n1-highcpu-2 | c5.large | F2s v2 | -| PostgreSQL | 3 | 16 vCPU, 60 GB memory | n1-standard-16 | m5.4xlarge | D16s v3 | -| PgBouncer | 3 | 2 vCPU, 1.8 GB memory | n1-highcpu-2 | c5.large | F2s v2 | -| Internal load balancing node | 1 | 4 vCPU, 3.6GB memory | n1-highcpu-4 | c5.large | F2s v2 | -| Redis - Cache | 3 | 4 vCPU, 15 GB memory | n1-standard-4 | m5.xlarge | D4s v3 | -| Redis - Queues / Shared State | 3 | 4 vCPU, 15 GB memory | n1-standard-4 | m5.xlarge | D4s v3 | -| Redis Sentinel - Cache | 3 | 1 vCPU, 1.7 GB memory | g1-small | t3.small | B1MS | -| Redis Sentinel - Queues / Shared State | 3 | 1 vCPU, 1.7 GB memory | g1-small | t3.small | B1MS | -| Gitaly | 3 | 32 vCPU, 120 GB memory | n1-standard-32 | m5.8xlarge | D32s v3 | -| Praefect | 3 | 4 vCPU, 3.6 GB memory | n1-highcpu-4 | c5.xlarge | F4s v2 | -| Praefect PostgreSQL | 1+* | 2 vCPU, 1.8 GB memory | n1-highcpu-2 | c5.large | F2s v2 | -| Sidekiq | 4 | 4 vCPU, 15 GB memory | n1-standard-4 | m5.xlarge | D4s v3 | -| GitLab Rails | 5 | 32 vCPU, 28.8 GB memory | n1-highcpu-32 | c5.9xlarge | F32s v2 | -| Monitoring node | 1 | 4 vCPU, 3.6 GB memory | n1-highcpu-4 | c5.xlarge | F4s v2 | +| External load balancing node | 1 | 4 vCPU, 3.6 GB memory | n1-highcpu-4 | `c5.xlarge` | F4s v2 | +| Consul | 3 | 2 vCPU, 1.8 GB memory | n1-highcpu-2 | `c5.large` | F2s v2 | +| PostgreSQL | 3 | 16 vCPU, 60 GB memory | n1-standard-16 | `m5.4xlarge` | D16s v3 | +| PgBouncer | 3 | 2 vCPU, 1.8 GB memory | n1-highcpu-2 | `c5.large` | F2s v2 | +| Internal load balancing node | 1 | 4 vCPU, 3.6GB memory | n1-highcpu-4 | `c5.large` | F2s v2 | +| Redis - Cache | 3 | 4 vCPU, 15 GB memory | n1-standard-4 | `m5.xlarge` | D4s v3 | +| Redis - Queues / Shared State | 3 | 4 vCPU, 15 GB memory | n1-standard-4 | `m5.xlarge` | D4s v3 | +| Redis Sentinel - Cache | 3 | 1 vCPU, 1.7 GB memory | g1-small | `t3.small` | B1MS | +| Redis Sentinel - Queues / Shared State | 3 | 1 vCPU, 1.7 GB memory | g1-small | `t3.small` | B1MS | +| Gitaly | 3 | 32 vCPU, 120 GB memory | n1-standard-32 | `m5.8xlarge` | D32s v3 | +| Praefect | 3 | 4 vCPU, 3.6 GB memory | n1-highcpu-4 | `c5.xlarge` | F4s v2 | +| Praefect PostgreSQL | 1+* | 2 vCPU, 1.8 GB memory | n1-highcpu-2 | `c5.large` | F2s v2 | +| Sidekiq | 4 | 4 vCPU, 15 GB memory | n1-standard-4 | `m5.xlarge` | D4s v3 | +| GitLab Rails | 5 | 32 vCPU, 28.8 GB memory | n1-highcpu-32 | `c5.9xlarge` | F32s v2 | +| Monitoring node | 1 | 4 vCPU, 3.6 GB memory | n1-highcpu-4 | `c5.xlarge` | F4s v2 | | Object storage | n/a | n/a | n/a | n/a | n/a | -| NFS server | 1 | 4 vCPU, 3.6 GB memory | n1-highcpu-4 | c5.xlarge | F4s v2 | +| NFS server | 1 | 4 vCPU, 3.6 GB memory | n1-highcpu-4 | `c5.xlarge` | F4s v2 | ```plantuml @startuml 25k @@ -212,11 +212,12 @@ The following list includes descriptions of each server and its assigned IP: ## Configure the external load balancer -In an active/active GitLab configuration, you'll need a load balancer to route +In a multi-node GitLab configuration, you'll need a load balancer to route traffic to the application servers. The specifics on which load balancer to use -or its exact configuration is beyond the scope of GitLab documentation. We hope +or its exact configuration is beyond the scope of GitLab documentation. We assume that if you're managing multi-node systems like GitLab, you already have a load -balancer of choice. Some load balancer examples include HAProxy (open-source), +balancer of choice and that the routing methods used are distributing calls evenly +between all nodes. Some load balancer examples include HAProxy (open-source), F5 Big-IP LTM, and Citrix Net Scaler. This documentation outline the ports and protocols needed for use with GitLab. @@ -389,6 +390,8 @@ backend praefect ``` Refer to your preferred Load Balancer's documentation for further guidance. +Also ensure that the routing methods used are distributing calls evenly across +all nodes. <div align="right"> <a type="button" class="btn btn-default" href="#setup-components"> @@ -435,7 +438,7 @@ To configure Consul: # Set the network addresses that the exporters will listen on node_exporter['listen_address'] = '0.0.0.0:9100' - # Disable auto migrations + # Prevent database migrations from running on upgrade automatically gitlab_rails['auto_migrate'] = false ``` @@ -559,7 +562,7 @@ in the second step, do not supply the `EXTERNAL_URL` value. # Incoming recommended value for max connections is 500. See https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/5691. patroni['postgresql']['max_connections'] = 500 - # Disable automatic database migrations + # Prevent database migrations from running on upgrade automatically gitlab_rails['auto_migrate'] = false # Configure the Consul agent @@ -855,7 +858,7 @@ a node and change its status from primary to replica (and vice versa). node_exporter['listen_address'] = '0.0.0.0:9100' redis_exporter['listen_address'] = '0.0.0.0:9121' - # Prevent database migrations from running on upgrade + # Prevent database migrations from running on upgrade automatically gitlab_rails['auto_migrate'] = false ``` @@ -922,7 +925,7 @@ You can specify multiple roles, like sentinel and Redis, as: node_exporter['listen_address'] = '0.0.0.0:9100' redis_exporter['listen_address'] = '0.0.0.0:9121' - # Prevent database migrations from running on upgrade + # Prevent database migrations from running on upgrade automatically gitlab_rails['auto_migrate'] = false ``` @@ -1054,7 +1057,7 @@ To configure the Sentinel Cache server: node_exporter['listen_address'] = '0.0.0.0:9100' redis_exporter['listen_address'] = '0.0.0.0:9121' - # Disable auto migrations + # Prevent database migrations from running on upgrade automatically gitlab_rails['auto_migrate'] = false ``` @@ -1119,13 +1122,8 @@ a node and change its status from primary to replica (and vice versa). # Set the network addresses that the exporters will listen on node_exporter['listen_address'] = '0.0.0.0:9100' redis_exporter['listen_address'] = '0.0.0.0:9121' - ``` - -1. Only the primary GitLab application server should handle migrations. To - prevent database migrations from running on upgrade, add the following - configuration to your `/etc/gitlab/gitlab.rb` file: - ```ruby + # Prevent database migrations from running on upgrade automatically gitlab_rails['auto_migrate'] = false ``` @@ -1186,7 +1184,7 @@ You can specify multiple roles, like sentinel and Redis, as: node_exporter['listen_address'] = '0.0.0.0:9100' redis_exporter['listen_address'] = '0.0.0.0:9121' - # Disable auto migrations + # Prevent database migrations from running on upgrade automatically gitlab_rails['auto_migrate'] = false ``` @@ -1318,7 +1316,7 @@ To configure the Sentinel Queues server: node_exporter['listen_address'] = '0.0.0.0:9100' redis_exporter['listen_address'] = '0.0.0.0:9121' - # Disable auto migrations + # Prevent database migrations from running on upgrade automatically gitlab_rails['auto_migrate'] = false ``` @@ -1403,6 +1401,7 @@ in the second step, do not supply the `EXTERNAL_URL` value. postgresql['listen_address'] = '0.0.0.0' postgresql['max_connections'] = 200 + # Prevent database migrations from running on upgrade automatically gitlab_rails['auto_migrate'] = false # Configure the Consul agent @@ -1510,7 +1509,7 @@ Praefect requires several secret tokens to secure communications across the Clus Gitaly Cluster nodes are configured in Praefect via a `virtual storage`. Each storage contains the details of each Gitaly node that makes up the cluster. Each storage is also given a name -and this name is used in several areas of the config. In this guide, the name of the storage will be +and this name is used in several areas of the configuration. In this guide, the name of the storage will be `default`. Also, this guide is geared towards new installs, if upgrading an existing environment to use Gitaly Cluster, you may need to use a different name. Refer to the [Praefect documentation](../gitaly/praefect.md#praefect) for more info. @@ -1548,7 +1547,8 @@ To configure the Praefect nodes, on each one: praefect['enable'] = true praefect['listen_addr'] = '0.0.0.0:2305' - gitlab_rails['rake_cache_clear'] = false + # Prevent database migrations from running on upgrade automatically + praefect['auto_migrate'] = false gitlab_rails['auto_migrate'] = false # Configure the Consul agent @@ -1672,8 +1672,7 @@ On each node: alertmanager['enable'] = false prometheus['enable'] = false - # Prevent database connections during 'gitlab-ctl reconfigure' - gitlab_rails['rake_cache_clear'] = false + # Prevent database migrations from running on upgrade automatically gitlab_rails['auto_migrate'] = false # Configure the gitlab-shell API callback URL. Without this, `git push` will @@ -1907,6 +1906,7 @@ To configure the Sidekiq nodes, on each one: gitlab_rails['db_password'] = '<postgresql_user_password>' gitlab_rails['db_adapter'] = 'postgresql' gitlab_rails['db_encoding'] = 'unicode' + # Prevent database migrations from running on upgrade automatically gitlab_rails['auto_migrate'] = false ####################################### @@ -2019,6 +2019,7 @@ On each node perform the following: gitlab_rails['db_host'] = '10.6.0.20' # internal load balancer IP gitlab_rails['db_port'] = 6432 gitlab_rails['db_password'] = '<postgresql_user_password>' + # Prevent database migrations from running on upgrade automatically gitlab_rails['auto_migrate'] = false ## Redis connection details @@ -2214,7 +2215,6 @@ To configure the Monitoring node: external_url 'http://gitlab.example.com' # Disable all other services - gitlab_rails['auto_migrate'] = false alertmanager['enable'] = false gitaly['enable'] = false gitlab_exporter['enable'] = false @@ -2248,6 +2248,9 @@ To configure the Monitoring node: consul['configuration'] = { retry_join: %w(10.6.0.11 10.6.0.12 10.6.0.13) } + + # Prevent database migrations from running on upgrade automatically + gitlab_rails['auto_migrate'] = false ``` 1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). @@ -2342,10 +2345,10 @@ to use GitLab Pages, this currently [requires NFS](troubleshooting.md#gitlab-pag See how to [configure NFS](../nfs.md). WARNING: -From GitLab 13.0, using NFS for Git repositories is deprecated. -From GitLab 14.0, technical support for NFS for Git repositories -will no longer be provided. Upgrade to [Gitaly Cluster](../gitaly/praefect.md) -as soon as possible. +From GitLab 14.0, enhancements and bug fixes for NFS for Git repositories will no longer be +considered and customer technical support will be considered out of scope. +[Read more about Gitaly and NFS](../gitaly/index.md#nfs-deprecation-notice) and +[the correct mount options to use](../nfs.md#upgrade-to-gitaly-cluster-or-disable-caching-if-experiencing-data-loss). <div align="right"> <a type="button" class="btn btn-default" href="#setup-components"> diff --git a/doc/administration/reference_architectures/2k_users.md b/doc/administration/reference_architectures/2k_users.md index fc0afbff194..e5418e47ca2 100644 --- a/doc/administration/reference_architectures/2k_users.md +++ b/doc/administration/reference_architectures/2k_users.md @@ -93,11 +93,12 @@ To set up GitLab and its components to accommodate up to 2,000 users: ## Configure the external load balancer -In an active/active GitLab configuration, you'll need a load balancer to route +In a multi-node GitLab configuration, you'll need a load balancer to route traffic to the application servers. The specifics on which load balancer to use -or its exact configuration is beyond the scope of GitLab documentation. We hope +or its exact configuration is beyond the scope of GitLab documentation. We assume that if you're managing multi-node systems like GitLab, you already have a load -balancer of choice. Some load balancer examples include HAProxy (open-source), +balancer of choice and that the routing methods used are distributing calls evenly +between all nodes. Some load balancer examples include HAProxy (open-source), F5 Big-IP LTM, and Citrix Net Scaler. This documentation outline the ports and protocols needed for use with GitLab. @@ -284,7 +285,7 @@ further configuration steps. # Replace APPLICATION_SERVER_IP_BLOCK with the CIDR address of the application node postgresql['trust_auth_cidr_addresses'] = %w(127.0.0.1/32 APPLICATION_SERVER_IP_BLOCK) - # Disable automatic database migrations + # Prevent database migrations from running on upgrade automatically gitlab_rails['auto_migrate'] = false ``` @@ -385,13 +386,6 @@ are supported and can be added if needed. ## Configure Gitaly -NOTE: -[Gitaly Cluster](../gitaly/praefect.md) support -for the Reference Architectures is being -worked on as a [collaborative effort](https://gitlab.com/gitlab-org/quality/reference-architectures/-/issues/1) between the Quality Engineering and Gitaly teams. When this component has been verified -some Architecture specs will likely change as a result to support the new -and improved designed. - [Gitaly](../gitaly/index.md) server node requirements are dependent on data, specifically the number of projects and those projects' sizes. It's recommended that a Gitaly server node stores no more than 5TB of data. Although this @@ -466,8 +460,7 @@ To configure the Gitaly server, on the server node you want to use for Gitaly: alertmanager['enable'] = false prometheus['enable'] = false - # Prevent database connections during 'gitlab-ctl reconfigure' - gitlab_rails['rake_cache_clear'] = false + # Prevent database migrations from running on upgrade automatically gitlab_rails['auto_migrate'] = false # Configure the gitlab-shell API callback URL. Without this, `git push` will @@ -779,7 +772,6 @@ running [Prometheus](../monitoring/prometheus/index.md) and grafana['admin_password'] = 'toomanysecrets' # Disable all other services - gitlab_rails['auto_migrate'] = false alertmanager['enable'] = false gitaly['enable'] = false gitlab_exporter['enable'] = false @@ -794,6 +786,9 @@ running [Prometheus](../monitoring/prometheus/index.md) and unicorn['enable'] = false node_exporter['enable'] = false gitlab_exporter['enable'] = false + + # Prevent database migrations from running on upgrade automatically + gitlab_rails['auto_migrate'] = false ``` 1. Prometheus also needs some scrape configurations to pull all the data from the various @@ -956,10 +951,10 @@ possible. However, if you intend to use GitLab Pages, See how to [configure NFS](../nfs.md). WARNING: -From GitLab 13.0, using NFS for Git repositories is deprecated. -From GitLab 14.0, technical support for NFS for Git repositories -will no longer be provided. Upgrade to [Gitaly Cluster](../gitaly/praefect.md) -as soon as possible. +From GitLab 14.0, enhancements and bug fixes for NFS for Git repositories will no longer be +considered and customer technical support will be considered out of scope. +[Read more about Gitaly and NFS](../gitaly/index.md#nfs-deprecation-notice) and +[the correct mount options to use](../nfs.md#upgrade-to-gitaly-cluster-or-disable-caching-if-experiencing-data-loss). <div align="right"> <a type="button" class="btn btn-default" href="#setup-components"> diff --git a/doc/administration/reference_architectures/3k_users.md b/doc/administration/reference_architectures/3k_users.md index 7f7777455c2..b8d0a98a1f1 100644 --- a/doc/administration/reference_architectures/3k_users.md +++ b/doc/administration/reference_architectures/3k_users.md @@ -12,7 +12,7 @@ users, and then maintain uptime and access for those users. You can also use this architecture to provide improved GitLab uptime and availability for fewer than 3,000 users. For fewer users, reduce the stated node sizes as needed. -If maintining a high level of uptime for your GitLab environment isn't a +If maintaining a high level of uptime for your GitLab environment isn't a requirement, or if you don't have the expertise to maintain this sort of environment, we recommend using the [2,000-user reference architecture](2k_users.md) for your GitLab installation. @@ -26,20 +26,20 @@ For a full list of reference architectures, see | Service | Nodes | Configuration | GCP | AWS | Azure | |--------------------------------------------|-------------|-----------------------|----------------|-------------|---------| -| External load balancing node | 1 | 2 vCPU, 1.8 GB memory | n1-highcpu-2 | c5.large | F2s v2 | -| Redis | 3 | 2 vCPU, 7.5 GB memory | n1-standard-2 | m5.large | D2s v3 | -| Consul + Sentinel | 3 | 2 vCPU, 1.8 GB memory | n1-highcpu-2 | c5.large | F2s v2 | -| PostgreSQL | 3 | 2 vCPU, 7.5 GB memory | n1-standard-2 | m5.large | D2s v3 | -| PgBouncer | 3 | 2 vCPU, 1.8 GB memory | n1-highcpu-2 | c5.large | F2s v2 | -| Internal load balancing node | 1 | 2 vCPU, 1.8 GB memory | n1-highcpu-2 | c5.large | F2s v2 | -| Gitaly | 3 | 4 vCPU, 15 GB memory | n1-standard-4 | m5.xlarge | D4s v3 | -| Praefect | 3 | 2 vCPU, 1.8 GB memory | n1-highcpu-2 | c5.large | F2s v2 | -| Praefect PostgreSQL | 1+* | 2 vCPU, 1.8 GB memory | n1-highcpu-2 | c5.large | F2s v2 | -| Sidekiq | 4 | 2 vCPU, 7.5 GB memory | n1-standard-2 | m5.large | D2s v3 | -| GitLab Rails | 3 | 8 vCPU, 7.2 GB memory | n1-highcpu-8 | c5.2xlarge | F8s v2 | -| Monitoring node | 1 | 2 vCPU, 1.8 GB memory | n1-highcpu-2 | c5.large | F2s v2 | +| External load balancing node | 1 | 2 vCPU, 1.8 GB memory | n1-highcpu-2 | `c5.large` | F2s v2 | +| Redis | 3 | 2 vCPU, 7.5 GB memory | n1-standard-2 | `m5.large` | D2s v3 | +| Consul + Sentinel | 3 | 2 vCPU, 1.8 GB memory | n1-highcpu-2 | `c5.large` | F2s v2 | +| PostgreSQL | 3 | 2 vCPU, 7.5 GB memory | n1-standard-2 | `m5.large` | D2s v3 | +| PgBouncer | 3 | 2 vCPU, 1.8 GB memory | n1-highcpu-2 | `c5.large` | F2s v2 | +| Internal load balancing node | 1 | 2 vCPU, 1.8 GB memory | n1-highcpu-2 | `c5.large` | F2s v2 | +| Gitaly | 3 | 4 vCPU, 15 GB memory | n1-standard-4 | `m5.xlarge` | D4s v3 | +| Praefect | 3 | 2 vCPU, 1.8 GB memory | n1-highcpu-2 | `c5.large` | F2s v2 | +| Praefect PostgreSQL | 1+* | 2 vCPU, 1.8 GB memory | n1-highcpu-2 | `c5.large` | F2s v2 | +| Sidekiq | 4 | 2 vCPU, 7.5 GB memory | n1-standard-2 | `m5.large` | D2s v3 | +| GitLab Rails | 3 | 8 vCPU, 7.2 GB memory | n1-highcpu-8 | `c5.2xlarge` | F8s v2 | +| Monitoring node | 1 | 2 vCPU, 1.8 GB memory | n1-highcpu-2 | `c5.large` | F2s v2 | | Object storage | n/a | n/a | n/a | n/a | n/a | -| NFS server (optional, not recommended) | 1 | 4 vCPU, 3.6 GB memory | n1-highcpu-4 | c5.xlarge | F4s v2 | +| NFS server (optional, not recommended) | 1 | 4 vCPU, 3.6 GB memory | n1-highcpu-4 | `c5.xlarge` | F4s v2 | ```plantuml @startuml 3k @@ -212,11 +212,12 @@ The following list includes descriptions of each server and its assigned IP: ## Configure the external load balancer -In an active/active GitLab configuration, you'll need a load balancer to route +In a multi-node GitLab configuration, you'll need a load balancer to route traffic to the application servers. The specifics on which load balancer to use -or its exact configuration is beyond the scope of GitLab documentation. We hope +or its exact configuration is beyond the scope of GitLab documentation. We assume that if you're managing multi-node systems like GitLab, you already have a load -balancer of choice. Some load balancer examples include HAProxy (open-source), +balancer of choice and that the routing methods used are distributing calls evenly +between all nodes. Some load balancer examples include HAProxy (open-source), F5 Big-IP LTM, and Citrix Net Scaler. This documentation outline the ports and protocols needed for use with GitLab. @@ -389,6 +390,8 @@ backend praefect ``` Refer to your preferred Load Balancer's documentation for further guidance. +Also ensure that the routing methods used are distributing calls evenly across +all nodes. <div align="right"> <a type="button" class="btn btn-default" href="#setup-components"> @@ -494,7 +497,7 @@ a node and change its status from primary to replica (and vice versa). 'redis.password' => 'redis-password-goes-here', } - # Disable auto migrations + # Prevent database migrations from running on upgrade automatically gitlab_rails['auto_migrate'] = false ``` @@ -577,7 +580,7 @@ run: redis-exporter: (pid 30075) 76861s; run: log: (pid 29674) 76896s 'redis.password' => 'redis-password-goes-here', } - # Disable auto migrations + # Prevent database migrations from running on upgrade automatically gitlab_rails['auto_migrate'] = false ``` @@ -706,7 +709,7 @@ To configure the Sentinel: node_exporter['listen_address'] = '0.0.0.0:9100' redis_exporter['listen_address'] = '0.0.0.0:9121' - # Disable auto migrations + # Prevent database migrations from running on upgrade automatically gitlab_rails['auto_migrate'] = false ``` @@ -831,7 +834,7 @@ in the second step, do not supply the `EXTERNAL_URL` value. # Incoming recommended value for max connections is 500. See https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/5691. patroni['postgresql']['max_connections'] = 500 - # Disable automatic database migrations + # Prevent database migrations from running on upgrade automatically gitlab_rails['auto_migrate'] = false # Configure the Consul agent @@ -1095,6 +1098,7 @@ in the second step, do not supply the `EXTERNAL_URL` value. postgresql['listen_address'] = '0.0.0.0' postgresql['max_connections'] = 200 + # Prevent database migrations from running on upgrade automatically gitlab_rails['auto_migrate'] = false # Configure the Consul agent @@ -1240,7 +1244,8 @@ To configure the Praefect nodes, on each one: praefect['enable'] = true praefect['listen_addr'] = '0.0.0.0:2305' - gitlab_rails['rake_cache_clear'] = false + # Prevent database migrations from running on upgrade automatically + praefect['auto_migrate'] = false gitlab_rails['auto_migrate'] = false # Configure the Consul agent @@ -1364,8 +1369,7 @@ On each node: alertmanager['enable'] = false prometheus['enable'] = false - # Prevent database connections during 'gitlab-ctl reconfigure' - gitlab_rails['rake_cache_clear'] = false + # Prevent database migrations from running on upgrade automatically gitlab_rails['auto_migrate'] = false # Configure the gitlab-shell API callback URL. Without this, `git push` will @@ -1533,7 +1537,6 @@ To configure the Sidekiq nodes, one each one: nginx['enable'] = false grafana['enable'] = false prometheus['enable'] = false - gitlab_rails['auto_migrate'] = false alertmanager['enable'] = false gitaly['enable'] = false gitlab_workhorse['enable'] = false @@ -1584,6 +1587,7 @@ To configure the Sidekiq nodes, one each one: gitlab_rails['db_password'] = '<postgresql_user_password>' gitlab_rails['db_adapter'] = 'postgresql' gitlab_rails['db_encoding'] = 'unicode' + # Prevent database migrations from running on upgrade automatically gitlab_rails['auto_migrate'] = false ####################################### @@ -1720,6 +1724,7 @@ On each node perform the following: gitlab_rails['db_host'] = '10.6.0.20' # internal load balancer IP gitlab_rails['db_port'] = 6432 gitlab_rails['db_password'] = '<postgresql_user_password>' + # Prevent database migrations from running on upgrade automatically gitlab_rails['auto_migrate'] = false ## Redis connection details @@ -1885,7 +1890,6 @@ running [Prometheus](../monitoring/prometheus/index.md) and external_url 'http://gitlab.example.com' # Disable all other services - gitlab_rails['auto_migrate'] = false alertmanager['enable'] = false gitaly['enable'] = false gitlab_exporter['enable'] = false @@ -1919,6 +1923,9 @@ running [Prometheus](../monitoring/prometheus/index.md) and consul['configuration'] = { retry_join: %w(10.6.0.11 10.6.0.12 10.6.0.13) } + + # Prevent database migrations from running on upgrade automatically + gitlab_rails['auto_migrate'] = false ``` 1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). @@ -2028,10 +2035,10 @@ to use GitLab Pages, this currently [requires NFS](troubleshooting.md#gitlab-pag See how to [configure NFS](../nfs.md). WARNING: -From GitLab 13.0, using NFS for Git repositories is deprecated. -From GitLab 14.0, technical support for NFS for Git repositories -will no longer be provided. Upgrade to [Gitaly Cluster](../gitaly/praefect.md) -as soon as possible. +From GitLab 14.0, enhancements and bug fixes for NFS for Git repositories will no longer be +considered and customer technical support will be considered out of scope. +[Read more about Gitaly and NFS](../gitaly/index.md#nfs-deprecation-notice) and +[the correct mount options to use](../nfs.md#upgrade-to-gitaly-cluster-or-disable-caching-if-experiencing-data-loss). <div align="right"> <a type="button" class="btn btn-default" href="#setup-components"> diff --git a/doc/administration/reference_architectures/50k_users.md b/doc/administration/reference_architectures/50k_users.md index 606701a4d83..183a998e89a 100644 --- a/doc/administration/reference_architectures/50k_users.md +++ b/doc/administration/reference_architectures/50k_users.md @@ -17,23 +17,23 @@ full list of reference architectures, see | Service | Nodes | Configuration | GCP | AWS | Azure | |-----------------------------------------|-------------|-------------------------|-----------------|--------------|----------| -| External load balancing node | 1 | 8 vCPU, 7.2 GB memory | n1-highcpu-8 | c5.2xlarge | F8s v2 | -| Consul | 3 | 2 vCPU, 1.8 GB memory | n1-highcpu-2 | c5.large | F2s v2 | -| PostgreSQL | 3 | 32 vCPU, 120 GB memory | n1-standard-32 | m5.8xlarge | D32s v3 | -| PgBouncer | 3 | 2 vCPU, 1.8 GB memory | n1-highcpu-2 | c5.large | F2s v2 | -| Internal load balancing node | 1 | 8 vCPU, 7.2 GB memory | n1-highcpu-8 | c5.2xlarge | F8s v2 | -| Redis - Cache | 3 | 4 vCPU, 15 GB memory | n1-standard-4 | m5.xlarge | D4s v3 | -| Redis - Queues / Shared State | 3 | 4 vCPU, 15 GB memory | n1-standard-4 | m5.xlarge | D4s v3 | -| Redis Sentinel - Cache | 3 | 1 vCPU, 1.7 GB memory | g1-small | t3.small | B1MS | -| Redis Sentinel - Queues / Shared State | 3 | 1 vCPU, 1.7 GB memory | g1-small | t3.small | B1MS | -| Gitaly | 3 | 64 vCPU, 240 GB memory | n1-standard-64 | m5.16xlarge | D64s v3 | -| Praefect | 3 | 4 vCPU, 3.6 GB memory | n1-highcpu-4 | c5.xlarge | F4s v2 | -| Praefect PostgreSQL | 1+* | 2 vCPU, 1.8 GB memory | n1-highcpu-2 | c5.large | F2s v2 | -| Sidekiq | 4 | 4 vCPU, 15 GB memory | n1-standard-4 | m5.xlarge | D4s v3 | -| GitLab Rails | 12 | 32 vCPU, 28.8 GB memory | n1-highcpu-32 | c5.9xlarge | F32s v2 | -| Monitoring node | 1 | 4 vCPU, 3.6 GB memory | n1-highcpu-4 | c5.xlarge | F4s v2 | +| External load balancing node | 1 | 8 vCPU, 7.2 GB memory | n1-highcpu-8 | `c5.2xlarge` | F8s v2 | +| Consul | 3 | 2 vCPU, 1.8 GB memory | n1-highcpu-2 | `c5.large` | F2s v2 | +| PostgreSQL | 3 | 32 vCPU, 120 GB memory | n1-standard-32 | `m5.8xlarge` | D32s v3 | +| PgBouncer | 3 | 2 vCPU, 1.8 GB memory | n1-highcpu-2 | `c5.large` | F2s v2 | +| Internal load balancing node | 1 | 8 vCPU, 7.2 GB memory | n1-highcpu-8 | `c5.2xlarge` | F8s v2 | +| Redis - Cache | 3 | 4 vCPU, 15 GB memory | n1-standard-4 | `m5.xlarge` | D4s v3 | +| Redis - Queues / Shared State | 3 | 4 vCPU, 15 GB memory | n1-standard-4 | `m5.xlarge` | D4s v3 | +| Redis Sentinel - Cache | 3 | 1 vCPU, 1.7 GB memory | g1-small | `t3.small` | B1MS | +| Redis Sentinel - Queues / Shared State | 3 | 1 vCPU, 1.7 GB memory | g1-small | `t3.small` | B1MS | +| Gitaly | 3 | 64 vCPU, 240 GB memory | n1-standard-64 | `m5.16xlarge` | D64s v3 | +| Praefect | 3 | 4 vCPU, 3.6 GB memory | n1-highcpu-4 | `c5.xlarge` | F4s v2 | +| Praefect PostgreSQL | 1+* | 2 vCPU, 1.8 GB memory | n1-highcpu-2 | `c5.large` | F2s v2 | +| Sidekiq | 4 | 4 vCPU, 15 GB memory | n1-standard-4 | `m5.xlarge` | D4s v3 | +| GitLab Rails | 12 | 32 vCPU, 28.8 GB memory | n1-highcpu-32 | `c5.9xlarge` | F32s v2 | +| Monitoring node | 1 | 4 vCPU, 3.6 GB memory | n1-highcpu-4 | `c5.xlarge` | F4s v2 | | Object storage | n/a | n/a | n/a | n/a | n/a | -| NFS server | 1 | 4 vCPU, 3.6 GB memory | n1-highcpu-4 | c5.xlarge | F4s v2 | +| NFS server | 1 | 4 vCPU, 3.6 GB memory | n1-highcpu-4 | `c5.xlarge` | F4s v2 | ```plantuml @startuml 50k @@ -142,7 +142,7 @@ To set up GitLab and its components to accommodate up to 50,000 users: 1. [Configure the external load balancer](#configure-the-external-load-balancer) to handle the load balancing of the GitLab application services nodes. 1. [Configure the internal load balancer](#configure-the-internal-load-balancer). - to handle the loa + to handle the load 1. [Configure Consul](#configure-consul). 1. [Configure PostgreSQL](#configure-postgresql), the database for GitLab. 1. [Configure PgBouncer](#configure-pgbouncer). @@ -219,11 +219,12 @@ The following list includes descriptions of each server and its assigned IP: ## Configure the external load balancer -In an active/active GitLab configuration, you'll need a load balancer to route +In a multi-node GitLab configuration, you'll need a load balancer to route traffic to the application servers. The specifics on which load balancer to use -or its exact configuration is beyond the scope of GitLab documentation. We hope +or its exact configuration is beyond the scope of GitLab documentation. We assume that if you're managing multi-node systems like GitLab, you already have a load -balancer of choice. Some load balancer examples include HAProxy (open-source), +balancer of choice and that the routing methods used are distributing calls evenly +between all nodes. Some load balancer examples include HAProxy (open-source), F5 Big-IP LTM, and Citrix Net Scaler. This documentation outline the ports and protocols needed for use with GitLab. @@ -396,6 +397,8 @@ backend praefect ``` Refer to your preferred Load Balancer's documentation for further guidance. +Also ensure that the routing methods used are distributing calls evenly across +all nodes. <div align="right"> <a type="button" class="btn btn-default" href="#setup-components"> @@ -442,7 +445,7 @@ To configure Consul: # Set the network addresses that the exporters will listen on node_exporter['listen_address'] = '0.0.0.0:9100' - # Disable auto migrations + # Prevent database migrations from running on upgrade automatically gitlab_rails['auto_migrate'] = false ``` @@ -566,7 +569,7 @@ in the second step, do not supply the `EXTERNAL_URL` value. # Incoming recommended value for max connections is 500. See https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/5691. patroni['postgresql']['max_connections'] = 500 - # Disable automatic database migrations + # Prevent database migrations from running on upgrade automatically gitlab_rails['auto_migrate'] = false # Configure the Consul agent @@ -862,7 +865,7 @@ a node and change its status from primary to replica (and vice versa). node_exporter['listen_address'] = '0.0.0.0:9100' redis_exporter['listen_address'] = '0.0.0.0:9121' - # Prevent database migrations from running on upgrade + # Prevent database migrations from running on upgrade automatically gitlab_rails['auto_migrate'] = false ``` @@ -929,7 +932,7 @@ You can specify multiple roles, like sentinel and Redis, as: node_exporter['listen_address'] = '0.0.0.0:9100' redis_exporter['listen_address'] = '0.0.0.0:9121' - # Prevent database migrations from running on upgrade + # Prevent database migrations from running on upgrade automatically gitlab_rails['auto_migrate'] = false ``` @@ -1061,7 +1064,7 @@ To configure the Sentinel Cache server: node_exporter['listen_address'] = '0.0.0.0:9100' redis_exporter['listen_address'] = '0.0.0.0:9121' - # Disable auto migrations + # Prevent database migrations from running on upgrade automatically gitlab_rails['auto_migrate'] = false ``` @@ -1126,13 +1129,8 @@ a node and change its status from primary to replica (and vice versa). # Set the network addresses that the exporters will listen on node_exporter['listen_address'] = '0.0.0.0:9100' redis_exporter['listen_address'] = '0.0.0.0:9121' - ``` - -1. Only the primary GitLab application server should handle migrations. To - prevent database migrations from running on upgrade, add the following - configuration to your `/etc/gitlab/gitlab.rb` file: - ```ruby + # Prevent database migrations from running on upgrade automatically gitlab_rails['auto_migrate'] = false ``` @@ -1193,7 +1191,7 @@ You can specify multiple roles, like sentinel and Redis, as: node_exporter['listen_address'] = '0.0.0.0:9100' redis_exporter['listen_address'] = '0.0.0.0:9121' - # Disable auto migrations + # Prevent database migrations from running on upgrade automatically gitlab_rails['auto_migrate'] = false ``` @@ -1325,7 +1323,7 @@ To configure the Sentinel Queues server: node_exporter['listen_address'] = '0.0.0.0:9100' redis_exporter['listen_address'] = '0.0.0.0:9121' - # Disable auto migrations + # Prevent database migrations from running on upgrade automatically gitlab_rails['auto_migrate'] = false ``` @@ -1410,6 +1408,7 @@ in the second step, do not supply the `EXTERNAL_URL` value. postgresql['listen_address'] = '0.0.0.0' postgresql['max_connections'] = 200 + # Prevent database migrations from running on upgrade automatically gitlab_rails['auto_migrate'] = false # Configure the Consul agent @@ -1517,7 +1516,7 @@ Praefect requires several secret tokens to secure communications across the Clus Gitaly Cluster nodes are configured in Praefect via a `virtual storage`. Each storage contains the details of each Gitaly node that makes up the cluster. Each storage is also given a name -and this name is used in several areas of the config. In this guide, the name of the storage will be +and this name is used in several areas of the configuration. In this guide, the name of the storage will be `default`. Also, this guide is geared towards new installs, if upgrading an existing environment to use Gitaly Cluster, you may need to use a different name. Refer to the [Praefect documentation](../gitaly/praefect.md#praefect) for more info. @@ -1555,7 +1554,8 @@ To configure the Praefect nodes, on each one: praefect['enable'] = true praefect['listen_addr'] = '0.0.0.0:2305' - gitlab_rails['rake_cache_clear'] = false + # Prevent database migrations from running on upgrade automatically + praefect['auto_migrate'] = false gitlab_rails['auto_migrate'] = false # Configure the Consul agent @@ -1679,8 +1679,7 @@ On each node: alertmanager['enable'] = false prometheus['enable'] = false - # Prevent database connections during 'gitlab-ctl reconfigure' - gitlab_rails['rake_cache_clear'] = false + # Prevent database migrations from running on upgrade automatically gitlab_rails['auto_migrate'] = false # Configure the gitlab-shell API callback URL. Without this, `git push` will @@ -1914,6 +1913,7 @@ To configure the Sidekiq nodes, on each one: gitlab_rails['db_password'] = '<postgresql_user_password>' gitlab_rails['db_adapter'] = 'postgresql' gitlab_rails['db_encoding'] = 'unicode' + # Prevent database migrations from running on upgrade automatically gitlab_rails['auto_migrate'] = false ####################################### @@ -2033,6 +2033,7 @@ On each node perform the following: gitlab_rails['db_host'] = '10.6.0.20' # internal load balancer IP gitlab_rails['db_port'] = 6432 gitlab_rails['db_password'] = '<postgresql_user_password>' + # Prevent database migrations from running on upgrade automatically gitlab_rails['auto_migrate'] = false ## Redis connection details @@ -2228,7 +2229,6 @@ To configure the Monitoring node: external_url 'http://gitlab.example.com' # Disable all other services - gitlab_rails['auto_migrate'] = false alertmanager['enable'] = false gitaly['enable'] = false gitlab_exporter['enable'] = false @@ -2262,6 +2262,9 @@ To configure the Monitoring node: consul['configuration'] = { retry_join: %w(10.6.0.11 10.6.0.12 10.6.0.13) } + + # Prevent database migrations from running on upgrade automatically + gitlab_rails['auto_migrate'] = false ``` 1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). @@ -2356,10 +2359,10 @@ to use GitLab Pages, this currently [requires NFS](troubleshooting.md#gitlab-pag See how to [configure NFS](../nfs.md). WARNING: -From GitLab 13.0, using NFS for Git repositories is deprecated. -From GitLab 14.0, technical support for NFS for Git repositories -will no longer be provided. Upgrade to [Gitaly Cluster](../gitaly/praefect.md) -as soon as possible. +From GitLab 14.0, enhancements and bug fixes for NFS for Git repositories will no longer be +considered and customer technical support will be considered out of scope. +[Read more about Gitaly and NFS](../gitaly/index.md#nfs-deprecation-notice) and +[the correct mount options to use](../nfs.md#upgrade-to-gitaly-cluster-or-disable-caching-if-experiencing-data-loss). <div align="right"> <a type="button" class="btn btn-default" href="#setup-components"> diff --git a/doc/administration/reference_architectures/5k_users.md b/doc/administration/reference_architectures/5k_users.md index 0a236eac243..70d02bba855 100644 --- a/doc/administration/reference_architectures/5k_users.md +++ b/doc/administration/reference_architectures/5k_users.md @@ -24,20 +24,20 @@ costly-to-operate environment by using the | Service | Nodes | Configuration | GCP | AWS | Azure | |--------------------------------------------|-------------|-------------------------|----------------|-------------|----------| -| External load balancing node | 1 | 2 vCPU, 1.8 GB memory | n1-highcpu-2 | c5.large | F2s v2 | -| Redis | 3 | 2 vCPU, 7.5 GB memory | n1-standard-2 | m5.large | D2s v3 | -| Consul + Sentinel | 3 | 2 vCPU, 1.8 GB memory | n1-highcpu-2 | c5.large | F2s v2 | -| PostgreSQL | 3 | 4 vCPU, 15 GB memory | n1-standard-4 | m5.xlarge | D4s v3 | -| PgBouncer | 3 | 2 vCPU, 1.8 GB memory | n1-highcpu-2 | c5.large | F2s v2 | -| Internal load balancing node | 1 | 2 vCPU, 1.8 GB memory | n1-highcpu-2 | c5.large | F2s v2 | -| Gitaly | 3 | 8 vCPU, 30 GB memory | n1-standard-8 | m5.2xlarge | D8s v3 | -| Praefect | 3 | 2 vCPU, 1.8 GB memory | n1-highcpu-2 | c5.large | F2s v2 | -| Praefect PostgreSQL | 1+* | 2 vCPU, 1.8 GB memory | n1-highcpu-2 | c5.large | F2s v2 | -| Sidekiq | 4 | 2 vCPU, 7.5 GB memory | n1-standard-2 | m5.large | D2s v3 | -| GitLab Rails | 3 | 16 vCPU, 14.4 GB memory | n1-highcpu-16 | c5.4xlarge | F16s v2 | -| Monitoring node | 1 | 2 vCPU, 1.8 GB memory | n1-highcpu-2 | c5.large | F2s v2 | +| External load balancing node | 1 | 2 vCPU, 1.8 GB memory | n1-highcpu-2 | `c5.large` | F2s v2 | +| Redis | 3 | 2 vCPU, 7.5 GB memory | n1-standard-2 | `m5.large` | D2s v3 | +| Consul + Sentinel | 3 | 2 vCPU, 1.8 GB memory | n1-highcpu-2 | `c5.large` | F2s v2 | +| PostgreSQL | 3 | 4 vCPU, 15 GB memory | n1-standard-4 | `m5.xlarge` | D4s v3 | +| PgBouncer | 3 | 2 vCPU, 1.8 GB memory | n1-highcpu-2 | `c5.large` | F2s v2 | +| Internal load balancing node | 1 | 2 vCPU, 1.8 GB memory | n1-highcpu-2 | `c5.large` | F2s v2 | +| Gitaly | 3 | 8 vCPU, 30 GB memory | n1-standard-8 | `m5.2xlarge` | D8s v3 | +| Praefect | 3 | 2 vCPU, 1.8 GB memory | n1-highcpu-2 | `c5.large` | F2s v2 | +| Praefect PostgreSQL | 1+* | 2 vCPU, 1.8 GB memory | n1-highcpu-2 | `c5.large` | F2s v2 | +| Sidekiq | 4 | 2 vCPU, 7.5 GB memory | n1-standard-2 | `m5.large` | D2s v3 | +| GitLab Rails | 3 | 16 vCPU, 14.4 GB memory | n1-highcpu-16 | `c5.4xlarge` | F16s v2 | +| Monitoring node | 1 | 2 vCPU, 1.8 GB memory | n1-highcpu-2 | `c5.large` | F2s v2 | | Object storage | n/a | n/a | n/a | n/a | n/a | -| NFS server (optional, not recommended) | 1 | 4 vCPU, 3.6 GB memory | n1-highcpu-4 | c5.xlarge | F4s v2 | +| NFS server (optional, not recommended) | 1 | 4 vCPU, 3.6 GB memory | n1-highcpu-4 | `c5.xlarge` | F4s v2 | ```plantuml @startuml 5k @@ -205,11 +205,12 @@ The following list includes descriptions of each server and its assigned IP: ## Configure the external load balancer -In an active/active GitLab configuration, you'll need a load balancer to route +In a multi-node GitLab configuration, you'll need a load balancer to route traffic to the application servers. The specifics on which load balancer to use -or its exact configuration is beyond the scope of GitLab documentation. We hope +or its exact configuration is beyond the scope of GitLab documentation. We assume that if you're managing multi-node systems like GitLab, you already have a load -balancer of choice. Some load balancer examples include HAProxy (open-source), +balancer of choice and that the routing methods used are distributing calls evenly +between all nodes. Some load balancer examples include HAProxy (open-source), F5 Big-IP LTM, and Citrix Net Scaler. This documentation outline the ports and protocols needed for use with GitLab. @@ -382,6 +383,8 @@ backend praefect ``` Refer to your preferred Load Balancer's documentation for further guidance. +Also ensure that the routing methods used are distributing calls evenly across +all nodes. <div align="right"> <a type="button" class="btn btn-default" href="#setup-components"> @@ -487,7 +490,7 @@ a node and change its status from primary to replica (and vice versa). 'redis.password' => 'redis-password-goes-here', } - # Disable auto migrations + # Prevent database migrations from running on upgrade automatically gitlab_rails['auto_migrate'] = false ``` @@ -570,7 +573,7 @@ run: redis-exporter: (pid 30075) 76861s; run: log: (pid 29674) 76896s 'redis.password' => 'redis-password-goes-here', } - # Disable auto migrations + # Prevent database migrations from running on upgrade automatically gitlab_rails['auto_migrate'] = false ``` @@ -699,7 +702,7 @@ To configure the Sentinel: node_exporter['listen_address'] = '0.0.0.0:9100' redis_exporter['listen_address'] = '0.0.0.0:9121' - # Disable auto migrations + # Prevent database migrations from running on upgrade automatically gitlab_rails['auto_migrate'] = false ``` @@ -824,7 +827,7 @@ in the second step, do not supply the `EXTERNAL_URL` value. # Incoming recommended value for max connections is 500. See https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/5691. patroni['postgresql']['max_connections'] = 500 - # Disable automatic database migrations + # Prevent database migrations from running on upgrade automatically gitlab_rails['auto_migrate'] = false # Configure the Consul agent @@ -1087,6 +1090,7 @@ in the second step, do not supply the `EXTERNAL_URL` value. postgresql['listen_address'] = '0.0.0.0' postgresql['max_connections'] = 200 + # Prevent database migrations from running on upgrade automatically gitlab_rails['auto_migrate'] = false # Configure the Consul agent @@ -1194,7 +1198,7 @@ Praefect requires several secret tokens to secure communications across the Clus Gitaly Cluster nodes are configured in Praefect via a `virtual storage`. Each storage contains the details of each Gitaly node that makes up the cluster. Each storage is also given a name -and this name is used in several areas of the config. In this guide, the name of the storage will be +and this name is used in several areas of the configuration. In this guide, the name of the storage will be `default`. Also, this guide is geared towards new installs, if upgrading an existing environment to use Gitaly Cluster, you may need to use a different name. Refer to the [Praefect documentation](../gitaly/praefect.md#praefect) for more info. @@ -1232,7 +1236,8 @@ To configure the Praefect nodes, on each one: praefect['enable'] = true praefect['listen_addr'] = '0.0.0.0:2305' - gitlab_rails['rake_cache_clear'] = false + # Prevent database migrations from running on upgrade automatically + praefect['auto_migrate'] = false gitlab_rails['auto_migrate'] = false # Configure the Consul agent @@ -1356,8 +1361,7 @@ On each node: alertmanager['enable'] = false prometheus['enable'] = false - # Prevent database connections during 'gitlab-ctl reconfigure' - gitlab_rails['rake_cache_clear'] = false + # Prevent database migrations from running on upgrade automatically gitlab_rails['auto_migrate'] = false # Configure the gitlab-shell API callback URL. Without this, `git push` will @@ -1523,7 +1527,6 @@ To configure the Sidekiq nodes, one each one: nginx['enable'] = false grafana['enable'] = false prometheus['enable'] = false - gitlab_rails['auto_migrate'] = false alertmanager['enable'] = false gitaly['enable'] = false gitlab_workhorse['enable'] = false @@ -1574,6 +1577,7 @@ To configure the Sidekiq nodes, one each one: gitlab_rails['db_password'] = '<postgresql_user_password>' gitlab_rails['db_adapter'] = 'postgresql' gitlab_rails['db_encoding'] = 'unicode' + # Prevent database migrations from running on upgrade automatically gitlab_rails['auto_migrate'] = false ####################################### @@ -1709,6 +1713,7 @@ On each node perform the following: gitlab_rails['db_host'] = '10.6.0.20' # internal load balancer IP gitlab_rails['db_port'] = 6432 gitlab_rails['db_password'] = '<postgresql_user_password>' + # Prevent database migrations from running on upgrade automatically gitlab_rails['auto_migrate'] = false ## Redis connection details @@ -1874,7 +1879,6 @@ running [Prometheus](../monitoring/prometheus/index.md) and external_url 'http://gitlab.example.com' # Disable all other services - gitlab_rails['auto_migrate'] = false alertmanager['enable'] = false gitaly['enable'] = false gitlab_exporter['enable'] = false @@ -1908,6 +1912,9 @@ running [Prometheus](../monitoring/prometheus/index.md) and consul['configuration'] = { retry_join: %w(10.6.0.11 10.6.0.12 10.6.0.13) } + + # Prevent database migrations from running on upgrade automatically + gitlab_rails['auto_migrate'] = false ``` 1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). @@ -2017,10 +2024,10 @@ to use GitLab Pages, this currently [requires NFS](troubleshooting.md#gitlab-pag See how to [configure NFS](../nfs.md). WARNING: -From GitLab 13.0, using NFS for Git repositories is deprecated. -From GitLab 14.0, technical support for NFS for Git repositories -will no longer be provided. Upgrade to [Gitaly Cluster](../gitaly/praefect.md) -as soon as possible. +From GitLab 14.0, enhancements and bug fixes for NFS for Git repositories will no longer be +considered and customer technical support will be considered out of scope. +[Read more about Gitaly and NFS](../gitaly/index.md#nfs-deprecation-notice) and +[the correct mount options to use](../nfs.md#upgrade-to-gitaly-cluster-or-disable-caching-if-experiencing-data-loss). <div align="right"> <a type="button" class="btn btn-default" href="#setup-components"> diff --git a/doc/administration/reference_architectures/index.md b/doc/administration/reference_architectures/index.md index d5d4f8ac0cb..6698737af6a 100644 --- a/doc/administration/reference_architectures/index.md +++ b/doc/administration/reference_architectures/index.md @@ -12,7 +12,7 @@ This page details the recommended Reference Architectures that were built and verified by the GitLab Quality and Support teams. Below is a chart representing each architecture tier and the number of users -they can handle. As your number of users grow with time, it’s recommended that +they can handle. As your number of users grow with time, it's recommended that you scale GitLab accordingly.  diff --git a/doc/administration/repository_storage_paths.md b/doc/administration/repository_storage_paths.md index e342ef8c04b..cea2144122b 100644 --- a/doc/administration/repository_storage_paths.md +++ b/doc/administration/repository_storage_paths.md @@ -156,5 +156,5 @@ often it is chosen. That is, `(storage weight) / (sum of all weights) * 100 = ch ## Move repositories -To move a repository to a different repository path, use -the same process as [migrating existing repositories to Gitaly Cluster](gitaly/praefect.md#migrate-existing-repositories-to-gitaly-cluster). +To move a repository to a different repository path, use the same process as +[migrating to Gitaly Cluster](gitaly/praefect.md#migrate-to-gitaly-cluster). diff --git a/doc/administration/restart_gitlab.md b/doc/administration/restart_gitlab.md index 69b3ae5282f..f4cc98ca145 100644 --- a/doc/administration/restart_gitlab.md +++ b/doc/administration/restart_gitlab.md @@ -4,18 +4,11 @@ group: Distribution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# How to restart GitLab +# How to restart GitLab **(FREE SELF)** Depending on how you installed GitLab, there are different methods to restart its service(s). -If you want the TL;DR versions, jump to: - -- [Omnibus GitLab restart](#omnibus-gitlab-restart) -- [Omnibus GitLab reconfigure](#omnibus-gitlab-reconfigure) -- [Source installation restart](#installations-from-source) -- [Helm chart installation restart](#helm-chart-installations) - ## Omnibus installations If you have used the [Omnibus packages](https://about.gitlab.com/install/) to install GitLab, then diff --git a/doc/administration/terraform_state.md b/doc/administration/terraform_state.md index 0f3fdf4bb93..0c01279b04c 100644 --- a/doc/administration/terraform_state.md +++ b/doc/administration/terraform_state.md @@ -97,6 +97,39 @@ The following settings are: | `remote_directory` | The bucket name where Terraform state files are stored | | | `connection` | Various connection options described below | | +### Migrate to object storage + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/247042) in GitLab 13.9. + +To migrate Terraform state files to object storage, follow the instructions below. + +- For Omnibus package installations: + + ```shell + gitlab-rake gitlab:terraform_states:migrate + ``` + +- For source installations: + + ```shell + sudo -u git -H bundle exec rake gitlab:terraform_states:migrate RAILS_ENV=production + ``` + +For GitLab 13.8 and earlier versions, you can use a workaround for the Rake task: + +1. Open the GitLab [Rails console](operations/rails_console.md). +1. Run the following commands: + + ```ruby + Terraform::StateUploader.alias_method(:upload, :model) + + Terraform::StateVersion.where(file_store: ::ObjectStorage::Store::LOCAL). find_each(batch_size: 10) do |terraform_state_version| + puts "Migrating: #{terraform_state_version.inspect}" + + terraform_state_version.file.migrate!(::ObjectStorage::Store::REMOTE) + end + ``` + ### S3-compatible connection settings See [the available connection settings for different providers](object_storage.md#connection-settings). diff --git a/doc/administration/troubleshooting/elasticsearch.md b/doc/administration/troubleshooting/elasticsearch.md index 606697b5247..11425d464b9 100644 --- a/doc/administration/troubleshooting/elasticsearch.md +++ b/doc/administration/troubleshooting/elasticsearch.md @@ -203,7 +203,7 @@ To do this: ```rails u = User.find_by_email('email_of_user_doing_search') s = SearchService.new(u, {:search => 'search_term'}) - pp s.search_objects.class.name + pp s.search_objects.class ``` The output from the last command is the key here. If it shows: @@ -217,7 +217,9 @@ The output from the last command is the key here. If it shows: If all the settings look correct and it is still not using Elasticsearch for the search function, it is best to escalate to GitLab support. This could be a bug/issue. -Moving past that, it is best to attempt the same search using the [Elasticsearch Search API](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-search.html) and compare the results from what you see in GitLab. +Moving past that, it is best to attempt the same [search via the Rails console](../../integration/elasticsearch.md#i-indexed-all-the-repositories-but-i-cant-get-any-hits-for-my-search-term-in-the-ui) +or the [Elasticsearch Search API](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-search.html), +and compare the results from what you see in GitLab. If the results: diff --git a/doc/administration/troubleshooting/gitlab_rails_cheat_sheet.md b/doc/administration/troubleshooting/gitlab_rails_cheat_sheet.md index 69eb639a8db..6b1cf2d1194 100644 --- a/doc/administration/troubleshooting/gitlab_rails_cheat_sheet.md +++ b/doc/administration/troubleshooting/gitlab_rails_cheat_sheet.md @@ -215,8 +215,8 @@ project = Project.find_by_full_path('group-changeme/project-changeme') ### Destroy a project ```ruby -project = Project.find_by_full_path('') -user = User.find_by_username('') +project = Project.find_by_full_path('<project_path>') +user = User.find_by_username('<username>') ProjectDestroyWorker.perform_async(project.id, user.id, {}) # or ProjectDestroyWorker.new.perform(project.id, user.id, {}) # or Projects::DestroyService.new(project, user).execute @@ -225,8 +225,8 @@ ProjectDestroyWorker.perform_async(project.id, user.id, {}) ### Remove fork relationship manually ```ruby -p = Project.find_by_full_path('') -u = User.find_by_username('') +p = Project.find_by_full_path('<project_path>') +u = User.find_by_username('<username>') ::Projects::UnlinkForkService.new(p, u).execute ``` @@ -243,13 +243,13 @@ project.update!(repository_read_only: true) ### Transfer project from one namespace to another ```ruby - p= Project.find_by_full_path('') + p= Project.find_by_full_path('<project_path>') # To set the owner of the project current_user= p.creator # Namespace where you want this to be moved. -namespace = Namespace.find_by_full_path("") +namespace = Namespace.find_by_full_path("<new_namespace>") ::Projects::TransferService.new(p, current_user).execute(namespace) ``` @@ -468,7 +468,7 @@ end ### Skip reconfirmation ```ruby -user = User.find_by_username '' +user = User.find_by_username '<username>' user.skip_reconfirmation! ``` @@ -558,7 +558,7 @@ user.max_member_access_for_group group.id ```ruby user = User.find_by_username('<username>') group = Group.find_by_name("<group_name>") -parent_group = Group.find_by(id: "") # empty string amounts to root as parent +parent_group = Group.find_by(id: "<group_id>") service = ::Groups::TransferService.new(group, user) service.execute(parent_group) ``` @@ -679,7 +679,7 @@ conflicting_permanent_redirects.destroy_all ```ruby p = Project.find_by_full_path('<full/path/to/project>') m = p.merge_requests.find_by(iid: <iid>) -u = User.find_by_username('') +u = User.find_by_username('<username>') MergeRequests::PostMergeService.new(p, u).execute(m) ``` @@ -695,9 +695,9 @@ Issuable::DestroyService.new(m.project, u).execute(m) ### Rebase manually ```ruby -p = Project.find_by_full_path('') +p = Project.find_by_full_path('<project_path>') m = project.merge_requests.find_by(iid: ) -u = User.find_by_username('') +u = User.find_by_username('<username>') MergeRequests::RebaseService.new(m.target_project, u).execute(m) ``` @@ -734,7 +734,7 @@ build.dependencies.each do |d| { puts "status: #{d.status}, finished at: #{d.fin ### Try CI service ```ruby -p = Project.find_by_full_path('') +p = Project.find_by_full_path('<project_path>') m = project.merge_requests.find_by(iid: ) m.project.try(:ci_service) ``` @@ -762,6 +762,21 @@ end Gitlab::CurrentSettings.current_application_settings.runners_registration_token ``` +### Run pipeline schedules manually + +You can run pipeline schedules manually through the Rails console to reveal any errors that are usually not visible. + +```ruby +# schedule_id can be obtained from Edit Pipeline Schedule page +schedule = Ci::PipelineSchedule.find_by(id: <schedule_id>) + +# Select the user that you want to run the schedule for +user = User.find_by_username('<username>') + +# Run the schedule +ps = Ci::CreatePipelineService.new(schedule.project, user, ref: schedule.ref).execute!(:schedule, ignore_skip_ci: true, save_on_errors: false, schedule: schedule) +``` + ## License ### See current license information diff --git a/doc/administration/troubleshooting/group_saml_scim.md b/doc/administration/troubleshooting/group_saml_scim.md index f0d44a578ff..63e69589b54 100644 --- a/doc/administration/troubleshooting/group_saml_scim.md +++ b/doc/administration/troubleshooting/group_saml_scim.md @@ -7,7 +7,7 @@ type: reference # Troubleshooting Group SAML and SCIM **(PREMIUM SAAS)** -These are notes and screenshots regarding Group SAML and SCIM that the GitLab Support Team sometimes uses while troubleshooting, but which do not fit into the official documentation. GitLab is making this public, so that anyone can make use of the Support team’s collected knowledge. +These are notes and screenshots regarding Group SAML and SCIM that the GitLab Support Team sometimes uses while troubleshooting, but which do not fit into the official documentation. GitLab is making this public, so that anyone can make use of the Support team's collected knowledge. Please refer to the GitLab [Group SAML](../../user/group/saml_sso/index.md) docs for information on the feature and how to set it up. @@ -20,6 +20,7 @@ They may then set up a test configuration of the desired identity provider. We i This section includes relevant screenshots of the following example configurations of [Group SAML](../../user/group/saml_sso/index.md) and [Group SCIM](../../user/group/saml_sso/scim_setup.md): - [Azure Active Directory](#azure-active-directory) +- [Okta](#okta) - [OneLogin](#onelogin) WARNING: @@ -59,6 +60,14 @@ IdP Links and Certificate:  +Sign on settings: + + + +Self-managed instance example: + + + ## OneLogin Application details: @@ -76,23 +85,3 @@ Adding a user: SSO settings:  - -## ADFS - -Setup SAML SSO URL: - - - -Configure Assertions: - - - -Configure NameID: - - - -Determine Certificate Fingerprint: - -| Via UI | Via Shell | -|--------|-----------| -|  |  | diff --git a/doc/administration/troubleshooting/img/ADFS-configure-NameID.png b/doc/administration/troubleshooting/img/ADFS-configure-NameID.png Binary files differdeleted file mode 100644 index d45e189b3ab..00000000000 --- a/doc/administration/troubleshooting/img/ADFS-configure-NameID.png +++ /dev/null diff --git a/doc/administration/troubleshooting/img/ADFS-configure-assertions.png b/doc/administration/troubleshooting/img/ADFS-configure-assertions.png Binary files differdeleted file mode 100644 index 53c26ad0be1..00000000000 --- a/doc/administration/troubleshooting/img/ADFS-configure-assertions.png +++ /dev/null diff --git a/doc/administration/troubleshooting/img/ADFS-determine-token-signing-certificate-fingerprint.png b/doc/administration/troubleshooting/img/ADFS-determine-token-signing-certificate-fingerprint.png Binary files differdeleted file mode 100644 index e24c994e996..00000000000 --- a/doc/administration/troubleshooting/img/ADFS-determine-token-signing-certificate-fingerprint.png +++ /dev/null diff --git a/doc/administration/troubleshooting/img/ADFS-determine-token-signing-fingerprint-from-shell.png b/doc/administration/troubleshooting/img/ADFS-determine-token-signing-fingerprint-from-shell.png Binary files differdeleted file mode 100644 index 431141dd3ef..00000000000 --- a/doc/administration/troubleshooting/img/ADFS-determine-token-signing-fingerprint-from-shell.png +++ /dev/null diff --git a/doc/administration/troubleshooting/img/ADFS-saml-setup-sso-url.png b/doc/administration/troubleshooting/img/ADFS-saml-setup-sso-url.png Binary files differdeleted file mode 100644 index a837f1b19cc..00000000000 --- a/doc/administration/troubleshooting/img/ADFS-saml-setup-sso-url.png +++ /dev/null diff --git a/doc/administration/auth/img/okta_admin_panel_v13_9.png b/doc/administration/troubleshooting/img/okta_admin_panel_v13_9.png Binary files differindex 2ebb1f0112c..2ebb1f0112c 100644 --- a/doc/administration/auth/img/okta_admin_panel_v13_9.png +++ b/doc/administration/troubleshooting/img/okta_admin_panel_v13_9.png diff --git a/doc/administration/auth/img/okta_saml_settings.png b/doc/administration/troubleshooting/img/okta_saml_settings.png Binary files differindex ee275ece369..ee275ece369 100644 --- a/doc/administration/auth/img/okta_saml_settings.png +++ b/doc/administration/troubleshooting/img/okta_saml_settings.png diff --git a/doc/administration/troubleshooting/index.md b/doc/administration/troubleshooting/index.md index 63b056df8b7..1c205cc987a 100644 --- a/doc/administration/troubleshooting/index.md +++ b/doc/administration/troubleshooting/index.md @@ -6,11 +6,13 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Troubleshooting a GitLab installation -Below are some resources to help you troubleshoot a GitLab installation -in case something goes wrong: +This page documents a collection of resources to help you troubleshoot a GitLab +installation. -- [Debugging tips](debug.md) -- [Diagnostics tools](diagnostics_tools.md) +## Troubleshooting guides + +- [SSL](ssl.md) +- [Geo](../geo/replication/troubleshooting.md) - [Elasticsearch](elasticsearch.md) - [GitLab Rails console cheat sheet](gitlab_rails_cheat_sheet.md) - [Group SAML and SCIM troubleshooting](group_saml_scim.md) **(PREMIUM SAAS)** @@ -18,9 +20,9 @@ in case something goes wrong: - [Linux cheat sheet](linux_cheat_sheet.md) - [Parsing GitLab logs with `jq`](log_parsing.md) - [Navigating GitLab via Rails console](navigating_gitlab_via_rails_console.md) -- [PostgreSQL](postgresql.md) -- [Sidekiq](sidekiq.md) -- [SSL](ssl.md) +- [Diagnostics tools](diagnostics_tools.md) +- [Debugging tips](debug.md) +- [Tracing requests with correlation ID](tracing_correlation_id.md) If you need a testing environment to troubleshoot, see the [apps for a testing environment](test_environments.md). diff --git a/doc/administration/troubleshooting/postgresql.md b/doc/administration/troubleshooting/postgresql.md index 4ccae10e5b3..9565b7594d6 100644 --- a/doc/administration/troubleshooting/postgresql.md +++ b/doc/administration/troubleshooting/postgresql.md @@ -35,7 +35,7 @@ This section is for links to information elsewhere in the GitLab documentation. - Storing data in another location. - Destructively reseeding the GitLab database. - Guidance around updating packaged PostgreSQL, including how to stop it - happening automatically. + from happening automatically. - [Information about external PostgreSQL](../postgresql/external.md). @@ -87,11 +87,11 @@ This section is for links to information elsewhere in the GitLab documentation. ```plaintext ERROR: replication slots can only be used if max_replication_slots > 0 - FATAL: could not start WAL streaming: ERROR: replication slot “geo_secondary_my_domain_com” does not exist + FATAL: could not start WAL streaming: ERROR: replication slot "geo_secondary_my_domain_com" does not exist Command exceeded allowed execution time - PANIC: could not write to file ‘pg_xlog/xlogtemp.123’: No space left on device + PANIC: could not write to file 'pg_xlog/xlogtemp.123': No space left on device ``` - [Checking Geo configuration](../geo/replication/troubleshooting.md), including: diff --git a/doc/administration/troubleshooting/sidekiq.md b/doc/administration/troubleshooting/sidekiq.md index 147bf1123ad..8d19a8a163b 100644 --- a/doc/administration/troubleshooting/sidekiq.md +++ b/doc/administration/troubleshooting/sidekiq.md @@ -347,7 +347,7 @@ Gitlab::SidekiqDaemon::Monitor.cancel_job('job-id') > environment variable. To perform of the interrupt we use `Thread.raise` which -has number of drawbacks, as mentioned in [Why Ruby’s Timeout is dangerous (and Thread.raise is terrifying)](https://jvns.ca/blog/2015/11/27/why-rubys-timeout-is-dangerous-and-thread-dot-raise-is-terrifying/): +has number of drawbacks, as mentioned in [Why Ruby's Timeout is dangerous (and Thread.raise is terrifying)](https://jvns.ca/blog/2015/11/27/why-rubys-timeout-is-dangerous-and-thread-dot-raise-is-terrifying/): > This is where the implications get interesting, and terrifying. This means that an exception can get raised: > @@ -357,4 +357,4 @@ has number of drawbacks, as mentioned in [Why Ruby’s Timeout is dangerous (and > - while creating an object to save to the database afterwards > - in any of your code, regardless of whether it could have possibly raised an exception before > -> Nobody writes code to defend against an exception being raised on literally any line. That’s not even possible. So Thread.raise is basically like a sneak attack on your code that could result in almost anything. It would probably be okay if it were pure-functional code that did not modify any state. But this is Ruby, so that’s unlikely :) +> Nobody writes code to defend against an exception being raised on literally any line. That's not even possible. So Thread.raise is basically like a sneak attack on your code that could result in almost anything. It would probably be okay if it were pure-functional code that did not modify any state. But this is Ruby, so that's unlikely :) diff --git a/doc/administration/whats-new.md b/doc/administration/whats-new.md new file mode 100644 index 00000000000..4cbb0b854ae --- /dev/null +++ b/doc/administration/whats-new.md @@ -0,0 +1,29 @@ +--- +stage: Growth +group: Adoption +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# What's new **(FREE)** + +With each monthly release, GitLab includes some of the highlights from the last 10 +GitLab versions in the **What's new** feature. To access it: + +1. In the top navigation bar, select the **{question}** icon. +1. Select **What's new** from the menu. + +The **What's new** describes new features available in multiple +[GitLab tiers](https://about.gitlab.com/pricing). While all users can see the +feature list, the feature list is tailored to your subscription type: + +- Features only available to self-managed installations are not shown on GitLab.com. +- Features only available on GitLab.com are not shown to self-managed installations. + +The **What's new** feature cannot be disabled, but +[is planned](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/59011) for a future release. + +## Self-managed installations + +Due to our release post process, the content for **What's new** is not yet finalized +when a new version (`.0` release) is cut. The updated **What's new** is included +in the first patch release, such as `13.10.1`. diff --git a/doc/administration/wikis/index.md b/doc/administration/wikis/index.md index 57bbe913216..bf6ff457ad3 100644 --- a/doc/administration/wikis/index.md +++ b/doc/administration/wikis/index.md @@ -73,3 +73,9 @@ You can also use the API to [retrieve the current value](../../api/settings.md#g ```shell curl --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/application/settings" ``` + +## Related topics + +- [User documentation for wikis](../../user/project/wiki/index.md) +- [Project wikis API](../../api/wikis.md) +- [Group wikis API](../../api/group_wikis.md) |
