diff options
| author | GitLab Bot <gitlab-bot@gitlab.com> | 2022-01-20 09:16:11 +0000 |
|---|---|---|
| committer | GitLab Bot <gitlab-bot@gitlab.com> | 2022-01-20 09:16:11 +0000 |
| commit | edaa33dee2ff2f7ea3fac488d41558eb5f86d68c (patch) | |
| tree | 11f143effbfeba52329fb7afbd05e6e2a3790241 /doc/administration | |
| parent | d8a5691316400a0f7ec4f83832698f1988eb27c1 (diff) | |
| download | gitlab-ce-edaa33dee2ff2f7ea3fac488d41558eb5f86d68c.tar.gz | |
Add latest changes from gitlab-org/gitlab@14-7-stable-eev14.7.0-rc42
Diffstat (limited to 'doc/administration')
91 files changed, 1393 insertions, 714 deletions
diff --git a/doc/administration/audit_event_streaming.md b/doc/administration/audit_event_streaming.md index cee6cddbbf0..eac54416924 100644 --- a/doc/administration/audit_event_streaming.md +++ b/doc/administration/audit_event_streaming.md @@ -6,11 +6,11 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Audit event streaming **(ULTIMATE)** -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/332747) in GitLab 14.5 [with a flag](../administration/feature_flags.md) named `ff_external_audit_events_namespace`. Disabled by default. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/332747) in GitLab 14.5 [with a flag](../administration/feature_flags.md) named `ff_external_audit_events_namespace`. Disabled by default. +> - [Enabled on GitLab.com and by default on self-managed](https://gitlab.com/gitlab-org/gitlab/-/issues/338939) in GitLab 14.7. FLAG: -On self-managed GitLab, by default this feature is not available. To make it available per group, ask an administrator to [enable the feature flag](../administration/feature_flags.md) named `ff_external_audit_events_namespace`. On GitLab.com, this feature is not available. -You should not use this feature for production environments. +On self-managed GitLab, by default this feature is available. To hide the feature per group, ask an administrator to [disable the feature flag](../administration/feature_flags.md) named `ff_external_audit_events_namespace`. On GitLab.com, this feature is available. Event streaming allows owners of top-level groups to set an HTTP endpoint to receive **all** audit events about the group, and its subgroups and projects. diff --git a/doc/administration/auth/atlassian.md b/doc/administration/auth/atlassian.md index 5fa10c4c119..08d07b13e22 100644 --- a/doc/administration/auth/atlassian.md +++ b/doc/administration/auth/atlassian.md @@ -1,7 +1,7 @@ --- type: reference stage: Manage -group: Access +group: Authentication & Authorization info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/administration/auth/authentiq.md b/doc/administration/auth/authentiq.md index 4220e552196..23735b75991 100644 --- a/doc/administration/auth/authentiq.md +++ b/doc/administration/auth/authentiq.md @@ -1,7 +1,7 @@ --- type: reference stage: Manage -group: Access +group: Authentication & Authorization info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/administration/auth/cognito.md b/doc/administration/auth/cognito.md index 718a2919ed0..2509bbb73d7 100644 --- a/doc/administration/auth/cognito.md +++ b/doc/administration/auth/cognito.md @@ -1,7 +1,7 @@ --- type: concepts, howto stage: Manage -group: Access +group: Authentication & Authorization info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/administration/auth/crowd.md b/doc/administration/auth/crowd.md index 265bba8a9b1..9a5f6b0a642 100644 --- a/doc/administration/auth/crowd.md +++ b/doc/administration/auth/crowd.md @@ -1,7 +1,7 @@ --- type: reference stage: Manage -group: Access +group: Authentication & Authorization info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/administration/auth/index.md b/doc/administration/auth/index.md index 959c5218554..d469988e719 100644 --- a/doc/administration/auth/index.md +++ b/doc/administration/auth/index.md @@ -2,7 +2,7 @@ comments: false type: index stage: Manage -group: Access +group: Authentication & Authorization info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/administration/auth/jwt.md b/doc/administration/auth/jwt.md index 9298b04cbc1..ac2f699dd5d 100644 --- a/doc/administration/auth/jwt.md +++ b/doc/administration/auth/jwt.md @@ -1,7 +1,7 @@ --- type: reference stage: Manage -group: Access +group: Authentication & Authorization info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/administration/auth/ldap/google_secure_ldap.md b/doc/administration/auth/ldap/google_secure_ldap.md index 69f0bfdce4c..6b8e699e861 100644 --- a/doc/administration/auth/ldap/google_secure_ldap.md +++ b/doc/administration/auth/ldap/google_secure_ldap.md @@ -1,7 +1,7 @@ --- type: reference stage: Manage -group: Access +group: Authentication & Authorization info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/administration/auth/ldap/index.md b/doc/administration/auth/ldap/index.md index f551c362784..b773281b216 100644 --- a/doc/administration/auth/ldap/index.md +++ b/doc/administration/auth/ldap/index.md @@ -1,7 +1,7 @@ --- type: reference stage: Manage -group: Access +group: Authentication & Authorization info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- @@ -21,7 +21,7 @@ This integration works with most LDAP-compliant directory servers, including: Users added through LDAP: -- Take a [licensed seat](../../../subscriptions/self_managed/index.md#billable-users). +- Usually use a [licensed seat](../../../subscriptions/self_managed/index.md#billable-users). - Can authenticate with Git using either their GitLab username or their email and LDAP password, even if password authentication for Git [is disabled](../../../user/admin_area/settings/sign_in_restrictions.md#password-authentication-enabled). @@ -153,13 +153,22 @@ production: ### Basic configuration settings +> `hosts` configuration setting [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/139) in GitLab 14.7. + +You can configure either: + +- A single LDAP server using `host` and `port`. +- Many LDAP servers using `hosts`. This setting takes precedence over `host` and `port`. GitLab attempts to use the + LDAP servers in the order specified, and the first reachable LDAP server is used. + These configuration settings are available: | Setting | Description | Required | Examples | |--------------------|-------------|----------|----------| | `label` | A human-friendly name for your LDAP server. It is displayed on your sign-in page. | **{check-circle}** Yes | `'Paris'` or `'Acme, Ltd.'` | -| `host` | IP address or domain name of your LDAP server. | **{check-circle}** Yes | `'ldap.mydomain.com'` | -| `port` | The port to connect with on your LDAP server. Always an integer, not a string. | **{check-circle}** Yes | `389` or `636` (for SSL) | +| `host` | IP address or domain name of your LDAP server. Ignored when `hosts` is defined. | **{check-circle}** Yes | `'ldap.mydomain.com'` | +| `port` | The port to connect with on your LDAP server. Always an integer, not a string. Ignored when `hosts` is defined. | **{check-circle}** Yes | `389` or `636` (for SSL) | +| `hosts` (GitLab 14.7 and later) | An array of host and port pairs to open connections. | **{dotted-circle}** No | `[['ldap1.mydomain.com', 636], ['ldap2.mydomain.com', 636]]` | | `uid` | LDAP attribute for username. Should be the attribute, not the value that maps to the `uid`. | **{check-circle}** Yes | `'sAMAccountName'` or `'uid'` or `'userPrincipalName'` | | `bind_dn` | The full DN of the user you bind with. | **{dotted-circle}** No | `'america\momo'` or `'CN=Gitlab,OU=Users,DC=domain,DC=com'` | | `password` | The password of the bind user. | **{dotted-circle}** No | `'your_great_password'` | diff --git a/doc/administration/auth/ldap/ldap-troubleshooting.md b/doc/administration/auth/ldap/ldap-troubleshooting.md index 63e4490e332..97a626f4a40 100644 --- a/doc/administration/auth/ldap/ldap-troubleshooting.md +++ b/doc/administration/auth/ldap/ldap-troubleshooting.md @@ -1,7 +1,7 @@ --- type: reference stage: Manage -group: Access +group: Authentication & Authorization info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/administration/auth/ldap/ldap_synchronization.md b/doc/administration/auth/ldap/ldap_synchronization.md index 8ccd8fecbcf..3329ea8e9cc 100644 --- a/doc/administration/auth/ldap/ldap_synchronization.md +++ b/doc/administration/auth/ldap/ldap_synchronization.md @@ -1,6 +1,6 @@ --- stage: Manage -group: Access +group: Authentication & Authorization info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/administration/auth/oidc.md b/doc/administration/auth/oidc.md index 7ab1f2f5feb..efe4b7440ee 100644 --- a/doc/administration/auth/oidc.md +++ b/doc/administration/auth/oidc.md @@ -1,7 +1,7 @@ --- type: reference stage: Manage -group: Access +group: Authentication & Authorization info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/administration/auth/smartcard.md b/doc/administration/auth/smartcard.md index d79837776b2..f77d3a36a7b 100644 --- a/doc/administration/auth/smartcard.md +++ b/doc/administration/auth/smartcard.md @@ -1,6 +1,6 @@ --- stage: Manage -group: Access +group: Authentication & Authorization info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments type: reference --- diff --git a/doc/administration/clusters/kas.md b/doc/administration/clusters/kas.md index b5c0a6ee76a..ba0d7280b74 100644 --- a/doc/administration/clusters/kas.md +++ b/doc/administration/clusters/kas.md @@ -6,13 +6,13 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Install the GitLab Agent Server (KAS) **(FREE SELF)** +> - [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/3834) in GitLab 13.10, the GitLab Agent Server (KAS) became available on GitLab.com under `wss://kas.gitlab.com`. > [Moved](https://gitlab.com/groups/gitlab-org/-/epics/6290) from GitLab Premium to GitLab Free in 14.5. The GitLab Agent Server (KAS) is a GitLab backend service dedicated to managing the [GitLab Agent](../../user/clusters/agent/index.md). The KAS is already installed and available in GitLab.com under `wss://kas.gitlab.com`. -See [how to use GitLab.com's KAS](../../user/clusters/agent/install/index.md#set-up-the-agent-server). This document describes how to install a KAS for GitLab self-managed instances. ## Installation options diff --git a/doc/administration/compliance.md b/doc/administration/compliance.md index 7cecc0c30fd..843d5d0930a 100644 --- a/doc/administration/compliance.md +++ b/doc/administration/compliance.md @@ -6,37 +6,49 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Compliance features **(FREE)** -These GitLab features can help ensure that your GitLab instance meets common compliance standards. For more information -about compliance management, see the compliance management [solutions page](https://about.gitlab.com/solutions/compliance/). +These GitLab features can help ensure that your GitLab instance meets common +compliance standards. For more information about compliance management, see the +compliance management [solutions page](https://about.gitlab.com/solutions/compliance/). -The [security features](../security/index.md) in GitLab may also help you meet relevant compliance standards. +The [security features](../security/index.md) in GitLab may also help you meet +relevant compliance standards. ## Policy management -Organizations have unique policy requirements, either due to organizational standards or mandates from regulatory bodies. -The following features help you define rules and policies to adhere to workflow requirements, separation of duties, and -secure supply chain best practices: +Organizations have unique policy requirements, either due to organizational +standards or mandates from regulatory bodies. The following features help you +define rules and policies to adhere to workflow requirements, separation of duties, +and secure supply chain best practices: -- [**Credentials inventory**](../user/admin_area/credentials_inventory.md) (for instances): With a credentials inventory, - GitLab administrators can keep track of the credentials used by all of the users in their GitLab instance. -- [**Granular user roles and flexible permissions**](../user/permissions.md) (for instances, groups, and projects): Manage - access and permissions with five different user roles and settings for external users. Set permissions according to people's - role, rather than either read or write access to a repository. Don't share the source code with people that only need - access to the issue tracker. -- [**Merge request approvals**](../user/project/merge_requests/approvals/index.md) (for instances, groups, and projects): - Configure approvals required for merge requests. -- [**Push rules**](../push_rules/push_rules.md) (for instances, groups, and projects): Control pushes to your - repositories. +- [**Credentials inventory**](../user/admin_area/credentials_inventory.md) (for + instances): With a credentials inventory, GitLab administrators can keep track + of the credentials used by all of the users in their GitLab instance. +- [**Granular user roles and flexible permissions**](../user/permissions.md) + (for instances, groups, and projects): Manage access and permissions with five + different user roles and settings for external users. Set permissions according + to people's role, rather than either read or write access to a repository. Don't + share the source code with people that only need access to the issue tracker. +- [**Merge request approvals**](../user/project/merge_requests/approvals/index.md) + (for instances, groups, and projects): Configure approvals required for + merge requests. +- [**Push rules**](../push_rules/push_rules.md) (for instances, groups, and + projects): Control pushes to your repositories. - Separation of duties using [**protected branches**](../user/project/protected_branches.md#require-code-owner-approval-on-a-protected-branch) - and [**custom CI/CD configuration paths**](../ci/pipelines/settings.md#specify-a-custom-cicd-configuration-file) (for projects): - Users can leverage the GitLab cross-project YAML configurations to define deployers of code and developers of code. - See how to use this setup to define these roles in: + and [**custom CI/CD configuration paths**](../ci/pipelines/settings.md#specify-a-custom-cicd-configuration-file) (for projects): Users can leverage the GitLab cross-project YAML configurations + to define deployers of code and developers of code. See how to use this setup + to define these roles in: - The [Separation of Duties deploy project](https://gitlab.com/guided-explorations/separation-of-duties-deploy/blob/master/README.md). - The [Separation of Duties project](https://gitlab.com/guided-explorations/separation-of-duties/blob/master/README.md). ## Compliant workflow automation -It is important for compliance teams to be confident that their controls and requirements are set up correctly, but also that they _stay_ set up correctly. One way of doing this is manually checking settings periodically, but this is error prone and time consuming. A better approach is to use single-source-of-truth settings and automation to ensure that whatever a compliance team has configured, stays configured and working correctly. These features can help you automate compliance: +It is important for compliance teams to be confident that their controls and +requirements are set up correctly, but also that they _stay_ set up correctly. +One way of doing this is manually checking settings periodically, but this is +error prone and time consuming. A better approach is to use single-source-of-truth +settings and automation to ensure that whatever a compliance team has configured, +stays configured and working correctly. These features can help you automate +compliance: - [**Compliance frameworks**](../user/project/settings/index.md#compliance-frameworks) (for groups): Create a custom compliance framework at the group level to describe the type of compliance requirements any child project needs to follow. @@ -45,42 +57,59 @@ It is important for compliance teams to be confident that their controls and req ## Audit management -An important part of any compliance program is being able to go back and understand what happened, when -it happened, and who was responsible. This is useful in audit situations as well as for understanding -the root cause of issues when they occur. It is useful to have both low-level, raw lists of audit data -as well as high-level, summary lists of audit data. Between these two, compliance teams can quickly -identify if problems exist and then drill down into the specifics of those issues. These features can help provide visibility into GitLab and audit what is happening: +An important part of any compliance program is being able to go back and understand +what happened, when it happened, and who was responsible. This is useful in audit +situations as well as for understanding the root cause of issues when they occur. +It is useful to have both low-level, raw lists of audit data as well as high-level, +summary lists of audit data. Between these two, compliance teams can quickly +identify if problems exist and then drill down into the specifics of those issues. +These features can help provide visibility into GitLab and audit what is happening: -- [**Audit events**](audit_events.md) (for instances, groups, and projects): To maintain the integrity of your code, - audit events give administrators the ability to view any modifications made within the GitLab - server in an advanced audit events system, so you can control, analyze, and track every change. -- [**Auditor users**](auditor_users.md) (for instances): Auditor users are users who are given read-only access to all - projects, groups, and other resources on the GitLab instance. -- [**Compliance report**](../user/compliance/compliance_report/index.md) (for groups): Quickly get visibility into the - compliance posture of your organization. +- [**Audit events**](audit_events.md) (for instances, groups, and projects): To + maintain the integrity of your code, audit events give administrators the + ability to view any modifications made within the GitLab server in an advanced + audit events system, so you can control, analyze, and track every change. +- [**Audit reports**](audit_reports.md) (for instances, groups, and projects): + Create and access reports based on the audit events that have occurred. Use + pre-built GitLab reports or the API to build your own. +- [**Auditor users**](auditor_users.md) (for instances): Auditor users are users + who are given read-only access to all projects, groups, and other resources on + the GitLab instance. +- [**Compliance report**](../user/compliance/compliance_report/index.md) (for + groups): Quickly get visibility into the compliance posture of your organization. ## Other compliance features These features can also help with compliance requirements: -- [**Email all users of a project, group, or entire server**](../tools/email.md) (for instances): An administrator can - email groups of users based on project or group membership, or email everyone using the GitLab instance. These emails +- [**Email all users of a project, group, or entire server**](../tools/email.md) + (for instances): An administrator can email groups of users based on project + or group membership, or email everyone using the GitLab instance. These emails are great for scheduled maintenance or upgrades. -- [**Enforce ToS acceptance**](../user/admin_area/settings/terms.md) (for instances): Enforce your users accepting new - terms of service by blocking GitLab traffic. -- [**External Status Checks**](../user/project/merge_requests/status_checks.md) (for projects): Interface with third-party - systems you already use during development to ensure you remain compliant. -- [**Generate reports on permission levels of users**](../user/admin_area/index.md#user-permission-export) (for - instances): Administrators can generate a report listing all users' access permissions for groups and projects in the - instance. -- [**Lock project membership to group**](../user/group/index.md#prevent-members-from-being-added-to-projects-in-a-group) (for - groups): Group owners can prevent new members from being added to projects within a group. -- [**LDAP group sync**](auth/ldap/ldap_synchronization.md#group-sync) (for instances): Gives administrators the ability - to automatically sync groups and manage SSH keys, permissions, and authentication, so you can focus on building your - product, not configuring your tools. -- [**LDAP group sync filters**](auth/ldap/ldap_synchronization.md#group-sync) (for instances): Gives more flexibility to - synchronize with LDAP based on filters, meaning you can leverage LDAP attributes to map GitLab permissions. +- [**Enforce ToS acceptance**](../user/admin_area/settings/terms.md) (for + instances): Enforce your users accepting new terms of service by blocking GitLab + traffic. +- [**External Status Checks**](../user/project/merge_requests/status_checks.md) + (for projects): Interface with third-party systems you already use during + development to ensure you remain compliant. +- [**Generate reports on permission levels of users**](../user/admin_area/index.md#user-permission-export) + (for instances): Administrators can generate a report listing all users' access + permissions for groups and projects in the instance. +- [**License compliance**](../user/compliance/license_compliance/index.md) (for + projects): Search dependencies for their licenses. This lets you determine if + the licenses of your project's dependencies are compatible with your project's + license. +- [**Lock project membership to group**](../user/group/index.md#prevent-members-from-being-added-to-projects-in-a-group) + (for groups): Group owners can prevent new members from being added to projects + within a group. +- [**LDAP group sync**](auth/ldap/ldap_synchronization.md#group-sync) (for + instances): Gives administrators the ability to automatically sync groups and + manage SSH keys, permissions, and authentication, so you can focus on building + your product, not configuring your tools. +- [**LDAP group sync filters**](auth/ldap/ldap_synchronization.md#group-sync) + (for instances): Gives more flexibility to synchronize with LDAP based on + filters, meaning you can leverage LDAP attributes to map GitLab permissions. - [**Omnibus GitLab package supports log forwarding**](https://docs.gitlab.com/omnibus/settings/logs.html#udp-log-forwarding) (for instances): Forward your logs to a central system. -- [**Restrict SSH Keys**](../security/ssh_keys_restrictions.md) (for instances): Control the technology and key length - of SSH keys used to access GitLab. +- [**Restrict SSH Keys**](../security/ssh_keys_restrictions.md) (for instances): + Control the technology and key length of SSH keys used to access GitLab. diff --git a/doc/administration/configure.md b/doc/administration/configure.md index 822acc1a74e..e42eb632f14 100644 --- a/doc/administration/configure.md +++ b/doc/administration/configure.md @@ -15,7 +15,7 @@ Customize and configure your self-managed GitLab installation. Here are some qui - [Geo](geo/index.md) - [Packages](packages/index.md) -The following tables are intended to guide you to choose the right combination of capabilties based on your requirements. It is common to want the most +The following tables are intended to guide you to choose the right combination of capabilities based on your requirements. It is common to want the most available, quickly recoverable, highly performant and fully data resilient solution. However, there are tradeoffs. The tables lists features on the left and provides their capabilities to the right along with known trade-offs. diff --git a/doc/administration/docs_self_host.md b/doc/administration/docs_self_host.md new file mode 100644 index 00000000000..007055b5de7 --- /dev/null +++ b/doc/administration/docs_self_host.md @@ -0,0 +1,131 @@ +--- +stage: Enablement +group: Distribution +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# How to self-host the docs site **(FREE SELF)** + +If you have a self-managed instance of GitLab, you may not be able to access the +product documentation as hosted on `docs.gitlab.com` from your GitLab instance. + +Be aware of the following items if you self-host the product documentation: + +- You must host the product documentation site under a subdirectory that matches + your installed GitLab version (for example, `14.5/`). The + [Docker images](https://gitlab.com/gitlab-org/gitlab-docs/container_registry/631635) + hosted by the GitLab Docs team provide this by default. We use a + [script](https://gitlab.com/gitlab-org/gitlab-docs/-/blob/2995d1378175803b22fb8806ba77adf63e79f32c/scripts/normalize-links.sh#L28-82) + to normalize the links and prefix them with the respective version. +- The version dropdown will display additional versions that don't exist, selecting + those versions will display a 404 Not Found page. +- Results when using the search box will display results from `docs.gitlab.com` + and not the local documentation. +- When you use the Docker images to serve the product documentation site, by + default the landing page redirects to the respective version (for example, `/14.5/`), + which causes the landing page <https://docs.gitlab.com> to not be displayed. + +## Documentation self-hosting options + +You can self-host the GitLab product documentation locally using one of these +methods: + +- Docker +- GitLab Pages +- From your own webserver + +The examples on this page are based on GitLab 14.5. + +### Self-host the product documentation with Docker + +The Docker images use a built-in webserver listening on port `4000`, so you need +to expose that. + +In the server that you host GitLab, or any other server that your GitLab instance +can talk to, you can use Docker to pull the docs site: + +```shell +docker run -it --rm -p 4000:4000 registry.gitlab.com/gitlab-org/gitlab-docs:14.5 +``` + +If you use [Docker compose](../install/docker.md#install-gitlab-using-docker-compose) +to host your GitLab instance, add the following to `docker-compose.yaml`: + +```yaml +version: '3.6' +services: + docs: + image: registry.gitlab.com/gitlab-org/gitlab-docs:14.5 + hostname: 'https://gitlab.example.com' + ports: + - '4000:4000' +``` + +### Self-host the product documentation with GitLab Pages + +You use GitLab Pages to host the GitLab product documentation locally. + +Prerequisite: + +- The Pages site URL must not use a subfolder. Due to the nature of how the docs + site is pre-compiled, the CSS and JavaScript files are relative to the + main domain or subdomain. For example, URLs like `https://example.com/docs/` + are not supported. + +To host the product documentation site with GitLab Pages: + +1. [Create a new blank project](../user/project/working_with_projects.md#create-a-blank-project). +1. Create a new or edit your existing `.gitlab-ci.yml` file, and add the following + `pages` job, while ensuring the version is the same as your GitLab installation: + + ```yaml + image: registry.gitlab.com/gitlab-org/gitlab-docs:14.5 + pages: + script: + - mkdir public + - cp -a /usr/share/nginx/html/* public/ + artifacts: + paths: + - public + ``` + +1. Optional. Set the GitLab Pages domain name. Depending on the type of the + GitLab Pages website, you have two options: + + | Type of website | [Default domain](../user/project/pages/getting_started_part_one.md#gitlab-pages-default-domain-names) | [Custom domain](../user/project/pages/custom_domains_ssl_tls_certification/index.md) | + |-------------------------|----------------|---------------| + | [Project website](../user/project/pages/getting_started_part_one.md#project-website-examples) | Not supported | Supported | + | [User or group website](../user/project/pages/getting_started_part_one.md#user-and-group-website-examples) | Supported | Supported | + +### Self-host the product documentation on your own webserver + +Because the product documentation site is static, you can grab the directory from +the container (in `/usr/share/nginx/html`) and use your own web server to host +it wherever you want. + +Use the following commands, and replace `<destination>` with the directory where the +documentation files will be copied to: + +```shell +docker create -it --name gitlab-docs registry.gitlab.com/gitlab-org/gitlab-docs:14.5 +docker cp gitlab-docs:/usr/share/nginx/html <destination> +docker rm -f gitlab-docs +``` + +## Redirect the `/help` links to the new docs page + +After your local product documentation site is running, [redirect the help +links](../user/admin_area/settings/help_page.md#redirect-help-pages) in the GitLab +application to your local site. + +Be sure to use the fully qualified domain name as the docs URL. For example, if you +used the [Docker method](#self-host-the-product-documentation-with-docker), enter `http://0.0.0.0:4000`. + +You don't need to append the version, as GitLab will detect it and append it to +any documentation URL requests, as needed. For example, if your GitLab version is +14.5, the GitLab Docs URL becomes `http://0.0.0.0:4000/14.5/`. The link +inside GitLab displays as `<instance_url>/help/user/admin_area/settings/help_page#destination-requirements`, +but when you select it, you are redirected to +`http://0.0.0.0:4000/14.5/ee/user/admin_area/settings/help_page/#destination-requirements`. + +To test the setting, select a **Learn more** link within the GitLab application. diff --git a/doc/administration/geo/disaster_recovery/background_verification.md b/doc/administration/geo/disaster_recovery/background_verification.md index caa806c92c8..d7db48bb6cf 100644 --- a/doc/administration/geo/disaster_recovery/background_verification.md +++ b/doc/administration/geo/disaster_recovery/background_verification.md @@ -2,7 +2,6 @@ stage: Enablement group: Geo info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments -type: howto --- # Automatic background verification **(PREMIUM SELF)** @@ -89,8 +88,6 @@ in sync. ## Repository re-verification -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/8550) in GitLab 11.6. - Due to bugs or transient infrastructure failures, it is possible for Git repositories to change unexpectedly without being marked for verification. Geo constantly reverifies the repositories to ensure the integrity of the diff --git a/doc/administration/geo/disaster_recovery/index.md b/doc/administration/geo/disaster_recovery/index.md index bf28eb76ffd..f7367ef863b 100644 --- a/doc/administration/geo/disaster_recovery/index.md +++ b/doc/administration/geo/disaster_recovery/index.md @@ -2,7 +2,6 @@ stage: Enablement group: Geo info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments -type: howto --- # Disaster Recovery (Geo) **(PREMIUM SELF)** @@ -503,7 +502,7 @@ secondary domain, like changing Git remotes and API URLs. This command uses the changed `external_url` configuration defined in `/etc/gitlab/gitlab.rb`. -1. For GitLab 11.11 through 12.7 only, you may need to update the **primary** +1. For GitLab 12.0 through 12.7, you may need to update the **primary** node's name in the database. This bug has been fixed in GitLab 12.8. To determine if you need to do this, search for the @@ -672,7 +671,7 @@ Data that was created on the primary while the secondary was paused is lost. If you are running GitLab 14.5 and later: -1. SSH to every Sidekiq, PostgresSQL, and Gitaly node in the **secondary** site and run one of the following commands: +1. For each node outside of the **secondary** Kubernetes cluster using Omnibus such as PostgreSQL or Gitaly, SSH into the node and run one of the following commands: - To promote the secondary node to primary: @@ -686,19 +685,17 @@ If you are running GitLab 14.5 and later: sudo gitlab-ctl geo promote --force ``` -1. SSH into each Rails node on your **secondary** site and run one of the following commands: +1. Find the `toolbox` pod: - - To promote the secondary node to primary: - - ```shell - sudo gitlab-ctl geo promote - ``` + ```shell + kubectl --namespace gitlab get pods -lapp=toolbox + ``` - - To promote the secondary node to primary **without any further confirmation**: +1. Promote the secondary: - ```shell - sudo gitlab-ctl geo promote --force - ``` + ```shell + kubectl --namespace gitlab exec -ti gitlab-geo-toolbox-XXX -- gitlab-rake geo:set_secondary_as_primary + ``` If you are running GitLab 14.4 and earlier: @@ -709,8 +706,6 @@ If you are running GitLab 14.4 and earlier: sudo gitlab-ctl promote-db ``` - In GitLab 12.8 and earlier, see [Message: `sudo: gitlab-pg-ctl: command not found`](../replication/troubleshooting.md#message-sudo-gitlab-pg-ctl-command-not-found). - 1. Edit `/etc/gitlab/gitlab.rb` on the database node in the **secondary** site to reflect its new status as **primary** by removing any lines that enabled the `geo_secondary_role`: diff --git a/doc/administration/geo/index.md b/doc/administration/geo/index.md index 2cb1a424ce8..b5eb0ba5841 100644 --- a/doc/administration/geo/index.md +++ b/doc/administration/geo/index.md @@ -25,7 +25,8 @@ For a video introduction to Geo, see [Introduction to GitLab Geo - GitLab Featur To make sure you're using the right version of the documentation, navigate to [the Geo page on GitLab.com](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/administration/geo/index.md) and choose the appropriate release from the **Switch branch/tag** dropdown. For example, [`v13.7.6-ee`](https://gitlab.com/gitlab-org/gitlab/-/blob/v13.7.6-ee/doc/administration/geo/index.md). -Geo uses a set of defined terms that is described in the [Geo Glossary](glossary.md), please familiarize yourself with those terms. +Geo uses a set of defined terms that are described in the [Geo Glossary](glossary.md). +Be sure to familiarize yourself with those terms. ## Use cases @@ -119,7 +120,8 @@ The following are required to run Geo: The following operating systems are known to ship with a current version of OpenSSH: - [CentOS](https://www.centos.org) 7.4 or later - [Ubuntu](https://ubuntu.com) 16.04 or later -- PostgreSQL 12 or later with [Streaming Replication](https://wiki.postgresql.org/wiki/Streaming_Replication) +- PostgreSQL 12 with [Streaming Replication](https://wiki.postgresql.org/wiki/Streaming_Replication) + - PostgreSQL 13 is not supported for Geo, see [epic 3832](https://gitlab.com/groups/gitlab-org/-/epics/3832) - Git 2.9 or later - Git-lfs 2.4.2 or later on the user side when using LFS - All sites must run the same GitLab version. diff --git a/doc/administration/geo/replication/configuration.md b/doc/administration/geo/replication/configuration.md index 3cbde77903d..16b4848e6d3 100644 --- a/doc/administration/geo/replication/configuration.md +++ b/doc/administration/geo/replication/configuration.md @@ -199,20 +199,21 @@ keys must be manually replicated to the **secondary** site. gitlab-ctl reconfigure ``` -1. On the top bar, select **Menu > Admin**. -1. On the left sidebar, select **Geo > Sites**. -1. Select **New site**. +1. Navigate to the Primary Node GitLab Instance: + 1. On the top bar, select **Menu > Admin**. + 1. On the left sidebar, select **Geo > Sites**. + 1. Select **Add site**.  -1. Fill in **Name** with the `gitlab_rails['geo_node_name']` in + 1. Fill in **Name** with the `gitlab_rails['geo_node_name']` in `/etc/gitlab/gitlab.rb`. These values must always match *exactly*, character for character. -1. Fill in **URL** with the `external_url` in `/etc/gitlab/gitlab.rb`. These + 1. Fill in **URL** with the `external_url` in `/etc/gitlab/gitlab.rb`. These values must always match, but it doesn't matter if one ends with a `/` and the other doesn't. -1. Optionally, choose which groups or storage shards should be replicated by the + 1. (Optional) Choose which groups or storage shards should be replicated by the **secondary** site. Leave blank to replicate all. Read more in [selective synchronization](#selective-synchronization). -1. Select **Add site** to add the **secondary** site. + 1. Select **Save changes** to add the **secondary** site. 1. SSH into **each Rails, and Sidekiq node on your secondary** site and restart the services: ```shell diff --git a/doc/administration/geo/replication/datatypes.md b/doc/administration/geo/replication/datatypes.md index 31bc473d74b..0fa08dcc9e0 100644 --- a/doc/administration/geo/replication/datatypes.md +++ b/doc/administration/geo/replication/datatypes.md @@ -188,8 +188,8 @@ successfully, you must replicate their data using some other means. |[Project repository](../../../user/project/repository/) | **Yes** (10.2) | **Yes** (10.7) | No | | |[Project wiki repository](../../../user/project/wiki/) | **Yes** (10.2) | **Yes** (10.7) | No | | |[Group wiki repository](../../../user/project/wiki/group.md) | [**Yes** (13.10)](https://gitlab.com/gitlab-org/gitlab/-/issues/208147) | No | No | Behind feature flag `geo_group_wiki_repository_replication`, enabled by default. | -|[Uploads](../../uploads.md) | **Yes** (10.2) | **Yes** (14.6) | Via Object Storage provider if supported. Native Geo support (Beta). | Replication is behind the feature flag `geo_upload_replication`, enabled by default. Verification is behind the feature flag `geo_upload_verification` introduced and enabled by default in 14.6. | -|[LFS objects](../../lfs/index.md) | **Yes** (10.2) | **Yes** (14.6) | Via Object Storage provider if supported. Native Geo support (Beta). | GitLab versions 11.11.x and 12.0.x are affected by [a bug that prevents any new LFS objects from replicating](https://gitlab.com/gitlab-org/gitlab/-/issues/32696).<br /><br />Replication is behind the feature flag `geo_lfs_object_replication`, enabled by default. Verification is behind the feature flag `geo_lfs_object_verification` introduced and enabled by default in 14.6. | +|[Uploads](../../uploads.md) | **Yes** (10.2) | **Yes** (14.6) | Via Object Storage provider if supported. Native Geo support (Beta). | Replication is behind the feature flag `geo_upload_replication`, enabled by default. Verification was behind the feature flag `geo_upload_verification`, removed in 14.8. | +|[LFS objects](../../lfs/index.md) | **Yes** (10.2) | **Yes** (14.6) | Via Object Storage provider if supported. Native Geo support (Beta). | GitLab versions 11.11.x and 12.0.x are affected by [a bug that prevents any new LFS objects from replicating](https://gitlab.com/gitlab-org/gitlab/-/issues/32696).<br /><br />Replication is behind the feature flag `geo_lfs_object_replication`, enabled by default. Verification was behind the feature flag `geo_lfs_object_verification`, removed in 14.7. | |[Personal snippets](../../../user/snippets.md) | **Yes** (10.2) | **Yes** (10.2) | No | | |[Project snippets](../../../user/snippets.md) | **Yes** (10.2) | **Yes** (10.2) | No | | |[CI job artifacts](../../../ci/pipelines/job_artifacts.md) | **Yes** (10.4) | [No](https://gitlab.com/gitlab-org/gitlab/-/issues/8923) | Via Object Storage provider if supported. Native Geo support (Beta). | Verified only manually using [Integrity Check Rake Task](../../raketasks/check.md) on both sites and comparing the output between them. Job logs also verified on transfer. | @@ -200,9 +200,9 @@ successfully, you must replicate their data using some other means. |[Project designs repository](../../../user/project/issues/design_management.md) | **Yes** (12.7) | [No](https://gitlab.com/gitlab-org/gitlab/-/issues/32467) | No | Designs also require replication of LFS objects and Uploads. | |[Package Registry](../../../user/packages/package_registry/index.md) | **Yes** (13.2) | [**Yes**](#limitation-of-verification-for-files-in-object-storage) (13.10) | Via Object Storage provider if supported. Native Geo support (Beta). | Behind feature flag `geo_package_file_replication`, enabled by default. | |[Versioned Terraform State](../../terraform_state.md) | **Yes** (13.5) | [**Yes**](#limitation-of-verification-for-files-in-object-storage) (13.12) | Via Object Storage provider if supported. Native Geo support (Beta). | Replication is behind the feature flag `geo_terraform_state_version_replication`, enabled by default. Verification was behind the feature flag `geo_terraform_state_version_verification`, which was removed in 14.0. | -|[External merge request diffs](../../merge_request_diffs.md) | **Yes** (13.5) | **Yes** (14.6) | Via Object Storage provider if supported. Native Geo support (Beta). | Replication is behind the feature flag `geo_merge_request_diff_replication`, enabled by default. Verification is behind the feature flag `geo_merge_request_diff_verification`, enabled by default in 14.6.| +|[External merge request diffs](../../merge_request_diffs.md) | **Yes** (13.5) | **Yes** (14.6) | Via Object Storage provider if supported. Native Geo support (Beta). | Replication is behind the feature flag `geo_merge_request_diff_replication`, enabled by default. Verification was behind the feature flag `geo_merge_request_diff_verification`, removed in 14.7.| |[Versioned snippets](../../../user/snippets.md#versioned-snippets) | [**Yes** (13.7)](https://gitlab.com/groups/gitlab-org/-/epics/2809) | [**Yes** (14.2)](https://gitlab.com/groups/gitlab-org/-/epics/2810) | No | Verification was implemented behind the feature flag `geo_snippet_repository_verification` in 13.11, and the feature flag was removed in 14.2. | -|[GitLab Pages](../../pages/index.md) | [**Yes** (14.3)](https://gitlab.com/groups/gitlab-org/-/epics/589) | [**Yes**](#limitation-of-verification-for-files-in-object-storage) (14.6) | Via Object Storage provider if supported. Native Geo support (Beta). | Behind feature flag `geo_pages_deployment_replication`, enabled by default. Verification is behind the feature flag `geo_pages_deployment_verification`, enabled by default in 14.6. | +|[GitLab Pages](../../pages/index.md) | [**Yes** (14.3)](https://gitlab.com/groups/gitlab-org/-/epics/589) | [**Yes**](#limitation-of-verification-for-files-in-object-storage) (14.6) | Via Object Storage provider if supported. Native Geo support (Beta). | Behind feature flag `geo_pages_deployment_replication`, enabled by default. Verification was behind the feature flag `geo_pages_deployment_verification`, removed in 14.7. | |[Server-side Git hooks](../../server_hooks.md) | [Not planned](https://gitlab.com/groups/gitlab-org/-/epics/1867) | No | No | Not planned because of current implementation complexity, low customer interest, and availability of alternatives to hooks. | |[Elasticsearch integration](../../../integration/elasticsearch.md) | [Not planned](https://gitlab.com/gitlab-org/gitlab/-/issues/1186) | No | No | Not planned because further product discovery is required and Elasticsearch (ES) clusters can be rebuilt. Secondaries use the same ES cluster as the primary. | |[Dependency proxy images](../../../user/packages/dependency_proxy/index.md) | [Not planned](https://gitlab.com/gitlab-org/gitlab/-/issues/259694) | No | No | Blocked by [Geo: Secondary Mimicry](https://gitlab.com/groups/gitlab-org/-/epics/1528). Replication of this cache is not needed for disaster recovery purposes because it can be recreated from external sources. | diff --git a/doc/administration/geo/replication/faq.md b/doc/administration/geo/replication/faq.md index e613a9b5670..12b3b382bf7 100644 --- a/doc/administration/geo/replication/faq.md +++ b/doc/administration/geo/replication/faq.md @@ -2,7 +2,6 @@ stage: Enablement group: Geo info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments -type: howto --- # Geo Frequently Asked Questions **(PREMIUM SELF)** @@ -54,7 +53,7 @@ For more details, see the [supported Geo data types](datatypes.md). ## Can I `git push` to a **secondary** site? -Yes! Pushing directly to a **secondary** site (for both HTTP and SSH, including Git LFS) was [introduced](https://about.gitlab.com/releases/2018/09/22/gitlab-11-3-released/) in GitLab 11.3. +Pushing directly to a **secondary** site (for both HTTP and SSH, including Git LFS) is supported. ## How long does it take to have a commit replicated to a **secondary** site? diff --git a/doc/administration/geo/replication/geo_validation_tests.md b/doc/administration/geo/replication/geo_validation_tests.md index a4c2f156216..ce1bd8a9d3c 100644 --- a/doc/administration/geo/replication/geo_validation_tests.md +++ b/doc/administration/geo/replication/geo_validation_tests.md @@ -2,7 +2,6 @@ stage: Enablement group: Geo info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments -type: howto --- # Geo validation tests **(PREMIUM SELF)** @@ -175,7 +174,7 @@ The following are PostgreSQL upgrade validation tests we performed. [Test and validate PostgreSQL 10.0 upgrade for Geo](https://gitlab.com/gitlab-org/gitlab/-/issues/12092): - Description: With the 12.0 release, GitLab required an upgrade to PostgreSQL 10.0. We tested - various upgrade scenarios from GitLab 11.11.5 through to GitLab 12.1.8. + various upgrade scenarios up to GitLab 12.1.8. - Outcome: Multiple issues were found when upgrading and addressed in follow-up issues. - Follow up issues: - [`gitlab-ctl` reconfigure fails on Redis node in multi-node Geo setup](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/4706). diff --git a/doc/administration/geo/replication/troubleshooting.md b/doc/administration/geo/replication/troubleshooting.md index 673d8388af1..a6ea41171a9 100644 --- a/doc/administration/geo/replication/troubleshooting.md +++ b/doc/administration/geo/replication/troubleshooting.md @@ -2,19 +2,18 @@ stage: Enablement group: Geo info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments -type: howto --- # Troubleshooting Geo **(PREMIUM SELF)** -Setting up Geo requires careful attention to details and sometimes it's easy to +Setting up Geo requires careful attention to details, and sometimes it's easy to miss a step. Here is a list of steps you should take to attempt to fix problem: -- Perform [basic troubleshooting](#basic-troubleshooting). -- Fix any [replication errors](#fixing-replication-errors). -- Fix any [common](#fixing-common-errors) errors. +1. Perform [basic troubleshooting](#basic-troubleshooting). +1. Fix any [replication errors](#fixing-replication-errors). +1. Fix any [common](#fixing-common-errors) errors. ## Basic troubleshooting @@ -41,11 +40,11 @@ to help identify if something is wrong:  -For information on how to resolve common errors reported from the UI, see -[Fixing Common Errors](#fixing-common-errors). +For information about how to resolve common error messages reported from the user interface, +see [Fixing Common Errors](#fixing-common-errors). -If the UI is not working, or you are unable to log in, you can run the Geo -health check manually to get this information as well as a few more details. +If the user interface is not working, or you are unable to sign in, you can run the Geo +health check manually to get this information and a few more details. #### Health check Rake task @@ -173,124 +172,129 @@ HINT: Close open transactions soon to avoid wraparound problems. You might also need to commit or roll back old prepared transactions, or drop stale replication slots. ``` -To fix this, do the following: +To fix this: 1. [Connect to the primary database](https://docs.gitlab.com/omnibus/settings/database.html#connecting-to-the-bundled-postgresql-database). + 1. Run `SELECT * FROM pg_replication_slots;`. -1. Note the `slot_name` that reports `active` as `f` (false). -1. Follow [all these steps to remove that Geo site](remove_geo_site.md). + Note the `slot_name` that reports `active` as `f` (false). + +1. Follow [the steps to remove that Geo site](remove_geo_site.md). ## Fixing errors found when running the Geo check Rake task -When running this Rake task, you may see errors if the nodes are not properly configured: +When running this Rake task, you may see error messages if the nodes are not properly configured: ```shell sudo gitlab-rake gitlab:geo:check ``` -1. Rails did not provide a password when connecting to the database +- Rails did not provide a password when connecting to the database. - ```plaintext - Checking Geo ... + ```plaintext + Checking Geo ... - GitLab Geo is available ... Exception: fe_sendauth: no password supplied - GitLab Geo is enabled ... Exception: fe_sendauth: no password supplied - ... - Checking Geo ... Finished - ``` + GitLab Geo is available ... Exception: fe_sendauth: no password supplied + GitLab Geo is enabled ... Exception: fe_sendauth: no password supplied + ... + Checking Geo ... Finished + ``` - - Ensure that you have the `gitlab_rails['db_password']` set to the plain text-password used when creating the hash for `postgresql['sql_user_password']`. + Ensure you have the `gitlab_rails['db_password']` set to the plain-text + password used when creating the hash for `postgresql['sql_user_password']`. -1. Rails is unable to connect to the database +- Rails is unable to connect to the database. - ```plaintext - Checking Geo ... + ```plaintext + Checking Geo ... - GitLab Geo is available ... Exception: FATAL: no pg_hba.conf entry for host "1.1.1.1", user "gitlab", database "gitlabhq_production", SSL on - FATAL: no pg_hba.conf entry for host "1.1.1.1", user "gitlab", database "gitlabhq_production", SSL off - GitLab Geo is enabled ... Exception: FATAL: no pg_hba.conf entry for host "1.1.1.1", user "gitlab", database "gitlabhq_production", SSL on - FATAL: no pg_hba.conf entry for host "1.1.1.1", user "gitlab", database "gitlabhq_production", SSL off - ... - Checking Geo ... Finished - ``` + GitLab Geo is available ... Exception: FATAL: no pg_hba.conf entry for host "1.1.1.1", user "gitlab", database "gitlabhq_production", SSL on + FATAL: no pg_hba.conf entry for host "1.1.1.1", user "gitlab", database "gitlabhq_production", SSL off + GitLab Geo is enabled ... Exception: FATAL: no pg_hba.conf entry for host "1.1.1.1", user "gitlab", database "gitlabhq_production", SSL on + FATAL: no pg_hba.conf entry for host "1.1.1.1", user "gitlab", database "gitlabhq_production", SSL off + ... + Checking Geo ... Finished + ``` - - Ensure that you have the IP address of the rails node included in `postgresql['md5_auth_cidr_addresses']`. - - Ensure that you have included the subnet mask on the IP address: `postgresql['md5_auth_cidr_addresses'] = ['1.1.1.1/32']`. + Ensure you have the IP address of the rails node included in `postgresql['md5_auth_cidr_addresses']`. + Also, ensure you have included the subnet mask on the IP address: `postgresql['md5_auth_cidr_addresses'] = ['1.1.1.1/32']`. -1. Rails has supplied the incorrect password +- Rails has supplied the incorrect password. - ```plaintext - Checking Geo ... - GitLab Geo is available ... Exception: FATAL: password authentication failed for user "gitlab" - FATAL: password authentication failed for user "gitlab" - GitLab Geo is enabled ... Exception: FATAL: password authentication failed for user "gitlab" - FATAL: password authentication failed for user "gitlab" - ... - Checking Geo ... Finished - ``` + ```plaintext + Checking Geo ... + GitLab Geo is available ... Exception: FATAL: password authentication failed for user "gitlab" + FATAL: password authentication failed for user "gitlab" + GitLab Geo is enabled ... Exception: FATAL: password authentication failed for user "gitlab" + FATAL: password authentication failed for user "gitlab" + ... + Checking Geo ... Finished + ``` - - Verify the correct password is set for `gitlab_rails['db_password']` that was used when creating the hash in `postgresql['sql_user_password']` by running `gitlab-ctl pg-password-md5 gitlab` and entering the password. + Verify the correct password is set for `gitlab_rails['db_password']` that was + used when creating the hash in `postgresql['sql_user_password']` by running + `gitlab-ctl pg-password-md5 gitlab` and entering the password. -1. Check returns `not a secondary node` +- Check returns `not a secondary node`. - ```plaintext - Checking Geo ... + ```plaintext + Checking Geo ... - GitLab Geo is available ... yes - GitLab Geo is enabled ... yes - GitLab Geo secondary database is correctly configured ... not a secondary node - Database replication enabled? ... not a secondary node - ... - Checking Geo ... Finished - ``` + GitLab Geo is available ... yes + GitLab Geo is enabled ... yes + GitLab Geo secondary database is correctly configured ... not a secondary node + Database replication enabled? ... not a secondary node + ... + Checking Geo ... Finished + ``` - - Ensure that you have added the secondary node in the Admin Area of the **primary** node. - - Ensure that you entered the `external_url` or `gitlab_rails['geo_node_name']` when adding the secondary node in the Admin Area of the **primary** node. - - Prior to GitLab 12.4, edit the secondary node in the Admin Area of the **primary** node and ensure that there is a trailing `/` in the `Name` field. + Ensure you have added the secondary node in the Admin Area of the **primary** node. + Also ensure you entered the `external_url` or `gitlab_rails['geo_node_name']` + when adding the secondary node in the Admin Area of the **primary** node. + In GitLab 12.3 and earlier, edit the secondary node in the Admin Area of the **primary** + node and ensure that there is a trailing `/` in the `Name` field. -1. Check returns `Exception: PG::UndefinedTable: ERROR: relation "geo_nodes" does not exist` +- Check returns `Exception: PG::UndefinedTable: ERROR: relation "geo_nodes" does not exist`. - ```plaintext - Checking Geo ... + ```plaintext + Checking Geo ... - GitLab Geo is available ... no - Try fixing it: - Upload a new license that includes the GitLab Geo feature - For more information see: - https://about.gitlab.com/features/gitlab-geo/ - GitLab Geo is enabled ... Exception: PG::UndefinedTable: ERROR: relation "geo_nodes" does not exist - LINE 8: WHERE a.attrelid = '"geo_nodes"'::regclass + GitLab Geo is available ... no + Try fixing it: + Upload a new license that includes the GitLab Geo feature + For more information see: + https://about.gitlab.com/features/gitlab-geo/ + GitLab Geo is enabled ... Exception: PG::UndefinedTable: ERROR: relation "geo_nodes" does not exist + LINE 8: WHERE a.attrelid = '"geo_nodes"'::regclass ^ - : SELECT a.attname, format_type(a.atttypid, a.atttypmod), - pg_get_expr(d.adbin, d.adrelid), a.attnotnull, a.atttypid, a.atttypmod, - c.collname, col_description(a.attrelid, a.attnum) AS comment - FROM pg_attribute a - LEFT JOIN pg_attrdef d ON a.attrelid = d.adrelid AND a.attnum = d.adnum - LEFT JOIN pg_type t ON a.atttypid = t.oid - LEFT JOIN pg_collation c ON a.attcollation = c.oid AND a.attcollation <> t.typcollation - WHERE a.attrelid = '"geo_nodes"'::regclass - AND a.attnum > 0 AND NOT a.attisdropped - ORDER BY a.attnum - ... - Checking Geo ... Finished - ``` - - When performing a PostgreSQL major version (9 > 10) update this is expected. Follow: + : SELECT a.attname, format_type(a.atttypid, a.atttypmod), + pg_get_expr(d.adbin, d.adrelid), a.attnotnull, a.atttypid, a.atttypmod, + c.collname, col_description(a.attrelid, a.attnum) AS comment + FROM pg_attribute a + LEFT JOIN pg_attrdef d ON a.attrelid = d.adrelid AND a.attnum = d.adnum + LEFT JOIN pg_type t ON a.atttypid = t.oid + LEFT JOIN pg_collation c ON a.attcollation = c.oid AND a.attcollation <> t.typcollation + WHERE a.attrelid = '"geo_nodes"'::regclass + AND a.attnum > 0 AND NOT a.attisdropped + ORDER BY a.attnum + ... + Checking Geo ... Finished + ``` - - [initiate-the-replication-process](../setup/database.md#step-3-initiate-the-replication-process) + When performing a PostgreSQL major version (9 > 10) update this is expected. Follow + the [initiate-the-replication-process](../setup/database.md#step-3-initiate-the-replication-process). ## Fixing replication errors The following sections outline troubleshooting steps for fixing replication -errors (indicated by `Database replication working? ... no` in the +error messages (indicated by `Database replication working? ... no` in the [`geo:check` output](#health-check-rake-task). ### Message: `ERROR: replication slots can only be used if max_replication_slots > 0`? This means that the `max_replication_slots` PostgreSQL variable needs to -be set on the **primary** database. In GitLab 9.4, we have made this setting -default to 1. You may need to increase this value if you have more -**secondary** nodes. +be set on the **primary** database. This setting defaults to 1. You may need to +increase this value if you have more **secondary** nodes. Be sure to restart PostgreSQL for this to take effect. See the [PostgreSQL replication setup](../setup/database.md#postgresql-replication) guide for more details. @@ -306,7 +310,7 @@ process](../setup/database.md) on the **secondary** node . ### Message: "Command exceeded allowed execution time" when setting up replication? This may happen while [initiating the replication process](../setup/database.md#step-3-initiate-the-replication-process) on the **secondary** node, -and indicates that your initial dataset is too large to be replicated in the default timeout (30 minutes). +and indicates your initial dataset is too large to be replicated in the default timeout (30 minutes). Re-run `gitlab-ctl replicate-geo-database`, but include a larger value for `--backup-timeout`: @@ -320,7 +324,7 @@ sudo gitlab-ctl \ ``` This gives the initial replication up to six hours to complete, rather than -the default thirty minutes. Adjust as required for your installation. +the default 30 minutes. Adjust as required for your installation. ### Message: "PANIC: could not write to file `pg_xlog/xlogtemp.123`: No space left on device" @@ -336,7 +340,7 @@ log data to build up in `pg_xlog`. Removing the unused slots can reduce the amou NOTE: Using `gitlab-rails dbconsole` does not work, because managing replication slots requires superuser permissions. -1. View your replication slots with: +1. View your replication slots: ```sql SELECT * FROM pg_replication_slots; @@ -345,7 +349,7 @@ log data to build up in `pg_xlog`. Removing the unused slots can reduce the amou Slots where `active` is `f` are not active. - When this slot should be active, because you have a **secondary** node configured using that slot, - log in to that **secondary** node and check the [PostgreSQL logs](../../logs.md#postgresql-logs) + sign in to that **secondary** node and check the [PostgreSQL logs](../../logs.md#postgresql-logs) to view why the replication is not running. - If you are no longer using the slot (for example, you no longer have Geo enabled), you can remove it with in the @@ -357,11 +361,11 @@ Slots where `active` is `f` are not active. ### Message: "ERROR: canceling statement due to conflict with recovery" -This error occurs infrequently under normal usage, and the system is resilient +This error message occurs infrequently under normal usage, and the system is resilient enough to recover. However, under certain conditions, some database queries on secondaries may run -excessively long, which increases the frequency of this error. This can lead to a situation +excessively long, which increases the frequency of this error message. This can lead to a situation where some queries never complete due to being canceled on every replication. These long-running queries are @@ -428,7 +432,16 @@ If large repositories are affected by this problem, their resync may take a long time and cause significant load on your Geo nodes, storage and network systems. -If you get the error `Synchronization failed - Error syncing repository` along with the following log messages, this indicates that the expected `geo` remote is not present in the `.git/config` file +If you see the error message `Synchronization failed - Error syncing repository` along with `fatal: fsck error in packed object`, this indicates +a consistency check error when syncing the repository. + +One example of a consistency error is: `error: object f4a87a3be694fbbd6e50a668a31a8513caeaafe3: hasDotgit: contains '.git`. + +Removing the malformed objects causing consistency errors require rewriting the repository history, which is not always an option. However, +it's possible to override the consistency checks instead. To do that, follow +[the instructions in the Gitaly docs](../../gitaly/configure_gitaly.md#repository-consistency-checks). + +You can also get the error message `Synchronization failed - Error syncing repository` along with the following log messages, this indicates that the expected `geo` remote is not present in the `.git/config` file of a repository on the secondary Geo node's file system: ```json @@ -451,11 +464,11 @@ of a repository on the secondary Geo node's file system: To solve this: -1. Log into the secondary Geo node. +1. Sign in to the secondary Geo node. 1. Back up [the `.git` folder](../../repository_storage_types.md#translate-hashed-storage-paths). -1. Optional: [Spot-check](../../troubleshooting/log_parsing.md#find-all-projects-affected-by-a-fatal-git-problem) +1. Optional. [Spot-check](../../troubleshooting/log_parsing.md#find-all-projects-affected-by-a-fatal-git-problem) a few of those IDs whether they indeed correspond to a project with known Geo replication failures. Use `fatal: 'geo'` as the `grep` term and the following API call: @@ -490,18 +503,19 @@ GitLab places a timeout on all repository clones, including project imports and Geo synchronization operations. If a fresh `git clone` of a repository on the **primary** takes more than the default three hours, you may be affected by this. -To increase the timeout, add the following line to `/etc/gitlab/gitlab.rb` -on the **secondary** node: +To increase the timeout: -```ruby -gitlab_rails['gitlab_shell_git_timeout'] = 14400 -``` +1. On the **secondary** node, add the following line to `/etc/gitlab/gitlab.rb`: -Then reconfigure GitLab: + ```ruby + gitlab_rails['gitlab_shell_git_timeout'] = 14400 + ``` -```shell -sudo gitlab-ctl reconfigure -``` +1. Reconfigure GitLab: + + ```shell + sudo gitlab-ctl reconfigure + ``` This increases the timeout to four hours (14400 seconds). Choose a time long enough to accommodate a full clone of your largest repositories. @@ -512,7 +526,7 @@ If new LFS objects are never replicated to secondary Geo nodes, check the versio GitLab you are running. GitLab versions 11.11.x or 12.0.x are affected by [a bug that results in new LFS objects not being replicated to Geo secondary nodes](https://gitlab.com/gitlab-org/gitlab/-/issues/32696). -To resolve the issue, upgrade to GitLab 12.1 or newer. +To resolve the issue, upgrade to GitLab 12.1 or later. ### Failures during backfill @@ -524,7 +538,7 @@ of the backfill queue, therefore these failures only clear up **after** the back If you get a **secondary** node in a broken state and want to reset the replication state, to start again from scratch, there are a few steps that can help you: -1. Stop Sidekiq and the Geo LogCursor +1. Stop Sidekiq and the Geo LogCursor. It's possible to make Sidekiq stop gracefully, but making it stop getting new jobs and wait until the current jobs to finish processing. @@ -547,7 +561,7 @@ to start again from scratch, there are a few steps that can help you: gitlab-ctl tail sidekiq ``` -1. Rename repository storage folders and create new ones. If you are not concerned about possible orphaned directories and files, then you can simply skip this step. +1. Rename repository storage folders and create new ones. If you are not concerned about possible orphaned directories and files, you can skip this step. ```shell mv /var/opt/gitlab/git-data/repositories /var/opt/gitlab/git-data/repositories.old @@ -559,14 +573,14 @@ to start again from scratch, there are a few steps that can help you: You may want to remove the `/var/opt/gitlab/git-data/repositories.old` in the future as soon as you confirmed that you don't need it anymore, to save disk space. -1. Optional. Rename other data folders and create new ones +1. Optional. Rename other data folders and create new ones. WARNING: You may still have files on the **secondary** node that have been removed from the **primary** node, but this - removal has not been reflected. If you skip this step, these files are not removed at all from the Geo node. + removal has not been reflected. If you skip this step, these files are not removed from the Geo node. - Any uploaded content like file attachments, avatars or LFS objects are stored in a - subfolder in one of the two paths below: + Any uploaded content (like file attachments, avatars, or LFS objects) is stored in a + subfolder in one of these paths: - `/var/opt/gitlab/gitlab-rails/shared` - `/var/opt/gitlab/gitlab-rails/uploads` @@ -593,7 +607,7 @@ to start again from scratch, there are a few steps that can help you: gitlab-ctl reconfigure ``` -1. Reset the Tracking Database +1. Reset the Tracking Database. ```shell gitlab-rake geo:db:drop # on a secondary app node @@ -601,7 +615,7 @@ to start again from scratch, there are a few steps that can help you: gitlab-rake geo:db:setup # on a secondary app node ``` -1. Restart previously stopped services +1. Restart previously stopped services. ```shell gitlab-ctl start @@ -611,10 +625,10 @@ to start again from scratch, there are a few steps that can help you: On the top bar, under **Menu > Admin > Geo > Nodes**, if the Design repositories progress bar shows -`Synced` and `Failed` greater than 100%, and negative `Queued`, then the instance +`Synced` and `Failed` greater than 100%, and negative `Queued`, the instance is likely affected by [a bug in GitLab 13.2 and 13.3](https://gitlab.com/gitlab-org/gitlab/-/issues/241668). -It was [fixed in 13.4+](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/40643). +It was [fixed in GitLab 13.4 and later](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/40643). To determine the actual replication status of design repositories in a [Rails console](../../operations/rails_console.md): @@ -665,7 +679,7 @@ determine the actual replication status of Design repositories. `gitlab-ctl promote-to-primary-node` fails since it runs preflight checks. If the [previous snippet](#design-repository-failures-on-mirrored-projects-and-project-imports) -shows that all designs are synced, then you can use the +shows that all designs are synced, you can use the `--skip-preflight-checks` option or the `--force` option to move forward with promotion. @@ -676,15 +690,70 @@ promotion. [previous snippet](#design-repository-failures-on-mirrored-projects-and-project-imports) to determine the actual replication status of Design repositories. +### Sync failure message: "Verification failed with: Error during verification: File is not checksummable" + +In GitLab 14.5 and earlier, certain data types which were missing on the Geo primary site were marked as "synced" on Geo secondary sites. This was because from the perspective of Geo secondary sites, the state matched the primary site and nothing more could be done on secondary sites. + +Secondaries would regularly try to sync these files again by using the "verification" feature: + +- Verification fails since the file doesn't exist. +- The file is marked "sync failed". +- Sync is retried. +- The file is marked "sync succeeded". +- The file is marked "needs verification". +- Repeat until the file is available again on the primary site. + +This can be confusing to troubleshoot, since the registry entries are moved through a logical loop by various background jobs. Also, `last_sync_failure` and `verification_failure` are empty after "sync succeeded" but before verification is retried. + +If you see sync failures repeatedly and alternately increase, while successes decrease and vice versa, this is likely to be caused by missing files on the primary site. You can confirm this by searching `geo.log` on secondary sites for `File is not checksummable` affecting the same files over and over. + +After confirming this is the problem, the files on the primary site need to be fixed. Some possible causes: + +- An NFS share became unmounted. +- A disk died or became corrupted. +- Someone unintentionally deleted a file or directory. +- Bugs in GitLab application: + - A file was moved when it shouldn't have been moved. + - A file wasn't moved when it should have been moved. + - A wrong path was generated in the code. +- A non-atomic backup was restored. +- Services or servers or network infrastructure was interrupted/restarted during use. + +The appropriate action sometimes depends on the cause. For example, you can remount an NFS share. Often, a root cause may not be apparent or not useful to discover. If you have regular backups, it may be expedient to look through them and pull files from there. + +In some cases, a file may be determined to be of low value, and so it may be worth deleting the record. + +Geo itself is an excellent mitigation for files missing on the primary. If a file disappears on the primary but it was already synced to the secondary, you can grab the secondary's file. In cases like this, the `File is not checksummable` error message will not occur on Geo secondary sites, and only the primary will log this error message. + +This problem is more likely to show up in Geo secondary sites which were set up long after the original GitLab site. In this case, Geo is only surfacing an existing problem. + +This behavior affects only the following data types through GitLab 14.6: + +| Data type | From version | +| ------------------------ | ------------ | +| Package Registry | 13.10 | +| Pipeline Artifacts | 13.11 | +| Terraform State Versions | 13.12 | +| Infrastructure Registry | 14.0 | +| External MR diffs | 14.6 | +| LFS Objects | 14.6 | +| Pages Deployments | 14.6 | +| Uploads | 14.6 | +| CI Job Artifacts | 14.6 | + +[Since GitLab 14.7, files that are missing on the primary site are now treated as sync failures](https://gitlab.com/gitlab-org/gitlab/-/issues/348745) +to make Geo visibly surface data loss risks. The sync/verification loop is +therefore short-circuited. `last_sync_failure` is now set to `The file is missing on the Geo primary site`. + ## Fixing errors during a failover or when promoting a secondary to a primary node -The following are possible errors that might be encountered during failover or +The following are possible error messages that might be encountered during failover or when promoting a secondary to a primary node with strategies to resolve them. ### Message: ActiveRecord::RecordInvalid: Validation failed: Name has already been taken When [promoting a **secondary** site](../disaster_recovery/index.md#step-3-promoting-a-secondary-site), -you might encounter the following error: +you might encounter the following error message: ```plaintext Running gitlab-rake geo:set_secondary_as_primary... @@ -712,7 +781,7 @@ or `gitlab-ctl promote-to-primary-node`, either: Rake::Task['geo:set_secondary_as_primary'].invoke ``` -- Upgrade to GitLab 12.6.3 or newer if it is safe to do so. For example, +- Upgrade to GitLab 12.6.3 or later if it is safe to do so. For example, if the failover was just a test. A [caching-related bug](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/22021) was fixed. @@ -720,8 +789,8 @@ or `gitlab-ctl promote-to-primary-node`, either: ### Message: ActiveRecord::RecordInvalid: Validation failed: Enabled Geo primary node cannot be disabled If you disabled a secondary node, either with the [replication pause task](../index.md#pausing-and-resuming-replication) -(13.2) or by using the user interface (13.1 and earlier), you must first -re-enable the node before you can continue. This is fixed in 13.4. +(GitLab 13.2) or by using the user interface (GitLab 13.1 and earlier), you must first +re-enable the node before you can continue. This is fixed in GitLab 13.4. This can be fixed in the database. @@ -747,12 +816,12 @@ This can be fixed in the database. UPDATE geo_nodes SET enabled = true WHERE url = 'https://<secondary url>/' AND enabled = false;" ``` - This should update 1 row. + This should update one row. ### Message: ``NoMethodError: undefined method `secondary?' for nil:NilClass`` When [promoting a **secondary** site](../disaster_recovery/index.md#step-3-promoting-a-secondary-site), -you might encounter the following error: +you might encounter the following error message: ```plaintext sudo gitlab-rake geo:set_secondary_as_primary @@ -767,7 +836,7 @@ Tasks: TOP => geo:set_secondary_as_primary (See full trace by running task with --trace) ``` -This command is intended to be executed on a secondary site only, and this error +This command is intended to be executed on a secondary site only, and this error message is displayed if you attempt to run this command on a primary site. ### Message: `sudo: gitlab-pg-ctl: command not found` @@ -789,7 +858,7 @@ In this case, the workaround is to use the full path to the binary, for example: sudo /opt/gitlab/embedded/bin/gitlab-pg-ctl promote ``` -GitLab 12.9 and later are [unaffected by this error](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/5147). +GitLab 12.9 and later are [unaffected by this error message](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/5147). ### Message: `ERROR - Replication is not up-to-date` during `gitlab-ctl promotion-preflight-checks` @@ -807,7 +876,7 @@ shows that it is complete, you can add `--skip-preflight-checks` to make the com ### Errors when using `--skip-preflight-checks` or `--force` -Before GitLab 13.5, you could bump into one of the following errors when using +In GitLab 13.4 and earlier, you could receive one of the following error messages when using `--skip-preflight-checks` or `--force`: ```plaintext @@ -817,7 +886,7 @@ get_ctl_options': invalid option: --force (OptionParser::InvalidOption) ``` This can happen with XFS or file systems that list files in lexical order, because the -load order of the Omnibus command files can be different than expected, and a global function would get redefined. +load order of the Omnibus GitLab command files can be different than expected, and a global function would get redefined. More details can be found in [the related issue](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/6076). The workaround is to manually run the preflight checks and promote the database, by running @@ -843,8 +912,8 @@ registry record related to the orphan files on disk. ### Message: The redirect URI included is not valid -If you are able to log in to the **primary** node, but you receive this error -when attempting to log into a **secondary**, you should check that the Geo +If you are able to sign in to the **primary** node, but you receive this error message +when attempting to sign in to a **secondary**, you should verify the Geo node's URL matches its external URL. On the **primary** node: @@ -858,7 +927,7 @@ On the **primary** node: ## Fixing common errors -This section documents common errors reported in the Admin Area and how to fix them. +This section documents common error messages reported in the Admin Area, and how to fix them. ### Geo database configuration file is missing @@ -879,20 +948,28 @@ It is safest to use a fresh secondary, or reset the whole secondary by following ### Geo node has a database that is writable which is an indication it is not configured for replication with the primary node -This error refers to a problem with the database replica on a **secondary** node, +This error message refers to a problem with the database replica on a **secondary** node, which Geo expects to have access to. It usually means, either: - An unsupported replication method was used (for example, logical replication). -- The instructions to setup a [Geo database replication](../setup/database.md) were not followed correctly. +- The instructions to set up a [Geo database replication](../setup/database.md) were not followed correctly. - Your database connection details are incorrect, that is you have specified the wrong user in your `/etc/gitlab/gitlab.rb` file. -A common source of confusion with **secondary** nodes is that it requires two separate -PostgreSQL instances: +Geo **secondary** sites require two separate PostgreSQL instances: - A read-only replica of the **primary** node. - A regular, writable instance that holds replication metadata. That is, the Geo tracking database. +This error message indicates that the replica database in the **secondary** site is misconfigured and replication has stopped. + +To restore the database and resume replication, you can do one of the following: + +- [Reset the Geo secondary site replication](#resetting-geo-secondary-node-replication). +- [Set up a new secondary Geo Omnibus instance](../setup/index.md#using-omnibus-gitlab). + +If you set up a new secondary from scratch, you must also [remove the old site from the Geo cluster](remove_geo_site.md#removing-secondary-geo-sites). + ### Geo node does not appear to be replicating the database from the primary node The most common problems that prevent the database from replicating correctly are: @@ -920,7 +997,7 @@ This can be caused by orphaned records in the project registry. You can clear th ### Geo Admin Area returns 404 error for a secondary node Sometimes `sudo gitlab-rake gitlab:geo:check` indicates that the **secondary** node is -healthy, but a 404 error for the **secondary** node is returned in the Geo Admin Area on +healthy, but a 404 Not Found error message for the **secondary** node is returned in the Geo Admin Area on the **primary** node. To resolve this issue: @@ -938,7 +1015,7 @@ If using a load balancer, ensure that the load balancer's URL is set as the `ext ### Geo Admin Area shows 'Unhealthy' after enabling Maintenance Mode -In GitLab 13.9 through GitLab 14.3, when [GitLab Maintenance Mode](../../maintenance_mode/index.md) is enabled, the status of Geo secondary sites will stop getting updated. After 10 minutes, the status will become `Unhealthy`. +In GitLab 13.9 through GitLab 14.3, when [GitLab Maintenance Mode](../../maintenance_mode/index.md) is enabled, the status of Geo secondary sites will stop getting updated. After 10 minutes, the status changes to `Unhealthy`. Geo secondary sites will continue to replicate and verify data, and the secondary sites should still be usable. You can use the [Sync status Rake task](#sync-status-rake-task) to determine the actual status of a secondary site during Maintenance Mode. @@ -947,7 +1024,7 @@ This bug was [fixed in GitLab 14.4](https://gitlab.com/gitlab-org/gitlab/-/issue ### GitLab Pages return 404 errors after promoting This is due to [Pages data not being managed by Geo](datatypes.md#limitations-on-replicationverification). -Find advice to resolve those errors in the +Find advice to resolve those error messages in the [Pages administration documentation](../../../administration/pages/index.md#404-error-after-promoting-a-geo-secondary-to-a-primary-node). ## Fixing client errors @@ -958,4 +1035,4 @@ You may have problems if you're running a version of [Git LFS](https://git-lfs.g As noted in [this authentication issue](https://github.com/git-lfs/git-lfs/issues/3025), requests redirected from the secondary to the primary node do not properly send the Authorization header. This may result in either an infinite `Authorization <-> Redirect` -loop, or Authorization errors. +loop, or Authorization error messages. diff --git a/doc/administration/geo/replication/usage.md b/doc/administration/geo/replication/usage.md index f3c8f6ac759..b1183e56cd0 100644 --- a/doc/administration/geo/replication/usage.md +++ b/doc/administration/geo/replication/usage.md @@ -2,7 +2,6 @@ stage: Enablement group: Geo info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments -type: howto --- <!-- Please update EE::GitLab::GeoGitAccess::GEO_SERVER_DOCS_URL if this file is moved) --> @@ -11,7 +10,8 @@ type: howto After you set up the [database replication and configure the Geo nodes](../index.md#setup-instructions), use your closest GitLab site as you would do with the primary one. -You can push directly to a **secondary** site (for both HTTP, SSH including Git LFS), and the request will be proxied to the primary site instead ([introduced](https://about.gitlab.com/releases/2018/09/22/gitlab-11-3-released/) in GitLab 11.3). +You can push directly to a **secondary** site (for both HTTP, SSH including +Git LFS), and the request will be proxied to the primary site instead. Example of the output you will see when pushing to a **secondary** site: diff --git a/doc/administration/geo/replication/version_specific_updates.md b/doc/administration/geo/replication/version_specific_updates.md index 883e335ff94..d3a132a6666 100644 --- a/doc/administration/geo/replication/version_specific_updates.md +++ b/doc/administration/geo/replication/version_specific_updates.md @@ -2,7 +2,6 @@ stage: Enablement group: Geo info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments -type: howto --- # Version-specific update instructions **(PREMIUM SELF)** @@ -378,10 +377,3 @@ WARNING: This version is affected by a [bug that results in new LFS objects not being replicated to Geo secondary nodes](https://gitlab.com/gitlab-org/gitlab/-/issues/32696). The issue is fixed in GitLab 12.1. Be sure to upgrade to GitLab 12.1 or later. - -## Updating to GitLab 11.11 - -WARNING: -This version is affected by a [bug that results in new LFS objects not being -replicated to Geo secondary nodes](https://gitlab.com/gitlab-org/gitlab/-/issues/32696). -The issue is fixed in GitLab 12.1. Be sure to upgrade to GitLab 12.1 or later. diff --git a/doc/administration/geo/setup/external_database.md b/doc/administration/geo/setup/external_database.md index 756e73f022f..cd77647e7dc 100644 --- a/doc/administration/geo/setup/external_database.md +++ b/doc/administration/geo/setup/external_database.md @@ -2,7 +2,6 @@ stage: Enablement group: Geo info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments -type: howto --- # Geo with external PostgreSQL instances **(PREMIUM SELF)** @@ -184,7 +183,7 @@ To configure the connection to the external read-replica database and enable Log database to keep track of replication status and automatically recover from potential replication issues. Omnibus automatically configures a tracking database when `roles ['geo_secondary_role']` is set. -If you want to run this database external to Omnibus, please follow the instructions below. +If you want to run this database external to Omnibus GitLab, use the following instructions. If you are using a cloud-managed service for the tracking database, you may need to grant additional roles to your tracking database user (by default, this is diff --git a/doc/administration/get_started.md b/doc/administration/get_started.md index bff2aee1d2d..3be3c11947e 100644 --- a/doc/administration/get_started.md +++ b/doc/administration/get_started.md @@ -1,5 +1,7 @@ --- info: For assistance with this TAM Onboarding page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments-to-other-projects-and-subjects. +stage: none +group: unassigned --- # Get started administering GitLab **(FREE)** diff --git a/doc/administration/git_protocol.md b/doc/administration/git_protocol.md index c8c532e9a01..29156d9b3e1 100644 --- a/doc/administration/git_protocol.md +++ b/doc/administration/git_protocol.md @@ -1,20 +1,17 @@ --- stage: Create group: Source Code -info: "To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments" -type: reference +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments description: "Set and configure Git protocol v2" --- # Configuring Git Protocol v2 **(FREE)** -> - [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/46555) in GitLab 11.4. -> - [Temporarily disabled](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/55769) in GitLab 11.5.8, 11.6.6, 11.7.1, and 11.8+. -> - [Re-enabled](https://gitlab.com/gitlab-org/gitlab/-/issues/27828) in GitLab 12.8. +> [Re-enabled](https://gitlab.com/gitlab-org/gitlab/-/issues/27828) in GitLab 12.8. Git protocol v2 improves the v1 wire protocol in several ways and is -enabled by default in GitLab for HTTP requests. In order to enable SSH, -further configuration is needed by the administrator. +enabled by default in GitLab for HTTP requests. To enable SSH, additional +configuration is required by an administrator. More details about the new features and improvements are available in the [Google Open Source Blog](https://opensource.googleblog.com/2018/05/introducing-git-protocol-version-2.html) @@ -51,7 +48,7 @@ sudo systemctl restart ssh ## Instructions -In order to use the new protocol, clients need to either pass the configuration +To use the new protocol, clients need to either pass the configuration `-c protocol.version=2` to the Git command, or set it globally: ```shell diff --git a/doc/administration/gitaly/configure_gitaly.md b/doc/administration/gitaly/configure_gitaly.md index b31a02aae0a..a0c959d5de9 100644 --- a/doc/administration/gitaly/configure_gitaly.md +++ b/doc/administration/gitaly/configure_gitaly.md @@ -565,6 +565,12 @@ Note the following: - You can configure Gitaly servers with both an unencrypted listening address `listen_addr` and an encrypted listening address `tls_listen_addr` at the same time. This allows you to gradually transition from unencrypted to encrypted traffic if necessary. +- When running Praefect sub-commands such as `dial-nodes` and `list-untracked-repositories` from the command line with Gitaly TLS enabled, you must set + the `SSL_CERT_DIR` or `SSL_CERT_FILE` environment variable so that the Gitaly certificate is trusted. For example: + + ```shell + sudo SSL_CERT_DIR=/etc/gitlab/trusted_certs /opt/gitlab/embedded/bin/praefect -config /var/opt/gitlab/praefect/config.toml dial-nodes + ``` To configure Gitaly with TLS: @@ -1125,3 +1131,66 @@ Example: "time":"2021-03-25T14:57:53.543Z" } ``` + +## Repository consistency checks + +Gitaly runs repository consistency checks: + +- When triggering a repository check. +- When changes are fetched from a mirrored repository. +- When users push changes into repository. + +These consistency checks verify that a repository has all required objects and +that these objects are valid objects. They can be categorized as: + +- Basic checks that assert that a repository doesn't become corrupt. This + includes connectivity checks and checks that objects can be parsed. +- Security checks that recognize objects that are suitable to exploit past + security-related bugs in Git. +- Cosmetic checks that verify that all object metadata is valid. Older Git + versions and other Git implementations may have produced objects with invalid + metadata, but newer versions can interpret these malformed objects. + +Removing malformed objects that fail the consistency checks requires a +rewrite of the repository's history, which often can't be done. Therefore, +Gitaly by default disables consistency checks for a range of cosmetic issues +that don't negatively impact repository consistency. + +By default, Gitaly doesn't disable basic or security-related checks so +to not distribute objects that can trigger known vulnerabilities in Git +clients. This also limits the ability to import repositories containing such +objects even if the project doesn't have malicious intent. + +### Override repository consistency checks + +Instance administrators can override consistency checks if they must +process repositories that do not pass consistency checks. + +For Omnibus GitLab installations, edit `/etc/gitlab/gitlab.rb` and set the +following keys (in this example, to disable the `hasDotgit` consistency check): + +```ruby +ignored_git_errors = ["hasDotgit = ignore"] +omnibus_gitconfig['system'] = { + "fsck" => ignored_git_errors, + "fetch.fsck" => ignored_git_errors, + "receive.fsck" => ignored_git_errors, +} +``` + +For source installs, edit the Gitaly configuration (`gitaly.toml`) to do the +equivalent: + +```toml +[[git.config]] +key = "fsck.hasDotgit" +value = "ignore" + +[[git.config]] +key = "fetch.fsck.hasDotgit" +value = "ignore" + +[[git.config]] +key = "receive.fsck.hasDotgit" +value = "ignore" +``` diff --git a/doc/administration/gitaly/index.md b/doc/administration/gitaly/index.md index f99bbf21840..4d60832720b 100644 --- a/doc/administration/gitaly/index.md +++ b/doc/administration/gitaly/index.md @@ -490,7 +490,15 @@ The following are useful queries for monitoring Gitaly: ### Monitor Gitaly Cluster -To monitor Gitaly Cluster (Praefect), you can use these Prometheus metrics: +To monitor Gitaly Cluster (Praefect), you can use these Prometheus metrics. There are two separate metrics +endpoints from which metrics can be scraped: + +- The default `/metrics` endpoint. +- `/db_metrics`, which contains metrics that require database queries. + +#### Default Prometheus `/metrics` endpoint + +The following metrics are available from the `/metrics` endpoint: - `gitaly_praefect_read_distribution`, a counter to track [distribution of reads](#distributed-reads). It has two labels: @@ -523,6 +531,18 @@ To monitor [strong consistency](#strong-consistency), you can use the following You can also monitor the [Praefect logs](../logs.md#praefect-logs). +#### Database metrics `/db_metrics` endpoint + +> [Introduced](https://gitlab.com/gitlab-org/gitaly/-/issues/3286) in GitLab 14.5. + +The following metrics are available from the `/db_metrics` endpoint: + +- `gitaly_praefect_unavailable_repositories`, the number of repositories that have no healthy, up to date replicas. +- `gitaly_praefect_read_only_repositories`, the number of repositories in read-only mode within a virtual storage. + This metric is available for backwards compatibility reasons. `gitaly_praefect_unavailable_repositories` is more + accurate. +- `gitaly_praefect_replication_queue_depth`, the number of jobs in the replication queue. + ## Recover from failure Gitaly Cluster can [recover from certain types of failure](recovery.md). diff --git a/doc/administration/gitaly/praefect.md b/doc/administration/gitaly/praefect.md index d3a8662080f..e2db30b8358 100644 --- a/doc/administration/gitaly/praefect.md +++ b/doc/administration/gitaly/praefect.md @@ -20,6 +20,9 @@ Configure Gitaly Cluster using either: Smaller GitLab installations may need only [Gitaly itself](index.md). +To upgrade a Gitaly Cluster, follow the documentation for +[zero downtime upgrades](../../update/zero_downtime.md#gitaly-cluster). + ## Requirements The minimum recommended configuration for a Gitaly Cluster requires: @@ -376,8 +379,8 @@ configuration option is set. For more details, consult the PgBouncer documentati If there are multiple Praefect nodes: -- Complete the following steps for **each** node. -- Designate one node as the "deploy node", and configure it first. +1. Designate one node as the deploy node, and configure it using the following steps. +1. Complete the following steps for each additional node. To complete this section you need a [configured PostgreSQL server](#postgresql), including: @@ -405,7 +408,7 @@ On the **Praefect** node: # Enable only the Praefect service praefect['enable'] = true - # Prevent database connections during 'gitlab-ctl reconfigure' + # Disable database migrations to prevent database connections during 'gitlab-ctl reconfigure' gitlab_rails['auto_migrate'] = false praefect['auto_migrate'] = false ``` @@ -415,10 +418,21 @@ On the **Praefect** node: ```ruby praefect['listen_addr'] = '0.0.0.0:2305' + ``` +1. Configure Prometheus metrics by editing + `/etc/gitlab/gitlab.rb`: + + ```ruby # Enable Prometheus metrics access to Praefect. You must use firewalls # to restrict access to this address/port. + # The default metrics endpoint is /metrics praefect['prometheus_listen_addr'] = '0.0.0.0:9652' + + # Some metrics run queries against the database. Enabling separate database metrics allows + # these metrics to be collected when the metrics are + # scraped on a separate /db_metrics endpoint. + praefect['separate_database_metrics'] = true ``` 1. Configure a strong `auth_token` for **Praefect** by editing @@ -517,7 +531,7 @@ On the **Praefect** node: 1. For: - The "deploy node": - 1. Enable Praefect auto-migration again by setting `praefect['auto_migrate'] = true` in + 1. Enable Praefect database auto-migration again by setting `praefect['auto_migrate'] = true` in `/etc/gitlab/gitlab.rb`. 1. To ensure database migrations are only run during reconfigure and not automatically on upgrade, run: @@ -556,8 +570,6 @@ On the **Praefect** node: edit `/etc/gitlab/gitlab.rb`, remember to run `sudo gitlab-ctl reconfigure` again before trying the `sql-ping` command. -**The steps above must be completed for each Praefect node!** - #### Enabling TLS support > [Introduced](https://gitlab.com/gitlab-org/gitaly/-/issues/1698) in GitLab 13.2. @@ -755,7 +767,7 @@ For more information on Gitaly server configuration, see our # Enable Prometheus if needed prometheus['enable'] = true - # Prevent database connections during 'gitlab-ctl reconfigure' + # Disable database migrations to prevent database connections during 'gitlab-ctl reconfigure' gitlab_rails['auto_migrate'] = false ``` @@ -883,9 +895,9 @@ of choice already. Some examples include [HAProxy](https://www.haproxy.org/) Big-IP LTM, and Citrix Net Scaler. This documentation outlines what ports and protocols you need configure. -WARNING: -Long-running background jobs can maintain an idle connection with Praefect for up 6 hours. Set your load balancer timeout to be at least -6 hours long. +NOTE: +We recommend the equivalent of HAProxy `leastconn` load-balancing strategy because long-running operations (for example, +clones) keep some connections open for extended periods. | LB Port | Backend Port | Protocol | |:--------|:-------------|:---------| @@ -1217,9 +1229,9 @@ To migrate existing clusters: 1. Praefect nodes didn't historically keep database records of every repository stored on the cluster. When the `per_repository` election strategy is configured, Praefect expects to have database records of - each repository. A [background migration](https://gitlab.com/gitlab-org/gitaly/-/merge_requests/2749) is - included in GitLab 13.6 and later to create any missing database records for repositories. Before migrating - you should verify the migration has run by checking Praefect's logs: + each repository. A [background database migration](https://gitlab.com/gitlab-org/gitaly/-/merge_requests/2749) is + included in GitLab 13.6 and later to create any missing database records for repositories. Before migrating, + check Praefect's logs to verify that the database migration ran. Check Praefect's logs for `repository importer finished` message. The `virtual_storages` field contains the names of virtual storages and whether they've had any missing database records created. @@ -1236,8 +1248,8 @@ To migrate existing clusters: {"level":"info","msg":"repository importer finished","pid":19752,"time":"2021-04-28T11:41:36.743Z","virtual_storages":{"default":false}} ``` - The migration is ran when Praefect starts up. If the migration is unsuccessful, you can restart - a Praefect node to reattempt it. The migration only runs with `sql` election strategy configured. + The database migration runs when Praefect starts. If the database migration is unsuccessful, you can restart + a Praefect node to reattempt it. 1. Running two different election strategies side by side can cause a split brain, where different Praefect nodes consider repositories to have different primaries. This can be avoided either: diff --git a/doc/administration/img/instance_review_button.png b/doc/administration/img/instance_review_button.png Binary files differdeleted file mode 100644 index b7604d7c7e5..00000000000 --- a/doc/administration/img/instance_review_button.png +++ /dev/null diff --git a/doc/administration/img/instance_review_v14_7.png b/doc/administration/img/instance_review_v14_7.png Binary files differnew file mode 100644 index 00000000000..e9f6316c237 --- /dev/null +++ b/doc/administration/img/instance_review_v14_7.png diff --git a/doc/administration/incoming_email.md b/doc/administration/incoming_email.md index 3f54f5dd576..84b30724dff 100644 --- a/doc/administration/incoming_email.md +++ b/doc/administration/incoming_email.md @@ -170,6 +170,15 @@ Reply by email should now be working. cd /home/git/gitlab ``` +1. Install the `gitlab-mail_room` gem manually: + + ```shell + gem install gitlab-mail_room + ``` + + NOTE: This step is necessary to avoid thread deadlocks and to support the latest MailRoom features. See + [this explanation](../development/emails.md#mailroom-gem-updates) for more details. + 1. Find the `incoming_email` section in `config/gitlab.yml`, enable the feature and fill in the details for your specific IMAP server and email account (see [examples](#configuration-examples) below). @@ -470,6 +479,10 @@ incoming_email: ##### Dedicated email address +NOTE: +Supports [Reply by Email](reply_by_email.md) only. +Cannot support [Service Desk](../user/project/service_desk.md). + Assumes the dedicated email address `incoming@exchange.example.com`. Example for Omnibus installs: @@ -531,19 +544,20 @@ enabled by default, and must be enabled through PowerShell. This series of PowerShell commands enables [sub-addressing](#email-sub-addressing) at the organization level in Office 365. This allows all mailboxes in the organization -to receive sub-addressed mail: +to receive sub-addressed mail. -```powershell -Set-ExecutionPolicy RemoteSigned -Scope CurrentUser +To enable sub-addressing: -$UserCredential = Get-Credential +1. Download and install the `ExchangeOnlineManagement` module from the [PowerShell gallery](https://www.powershellgallery.com/packages/ExchangeOnlineManagement/). +1. In PowerShell, run the following commands: -$Session = New-PSSession -ConfigurationName Microsoft.Exchange -ConnectionUri https://outlook.office365.com/powershell-liveid/ -Credential $UserCredential -Authentication Basic -AllowRedirection - -Import-PSSession $Session -DisableNameChecking - -Set-OrganizationConfig -AllowPlusAddressInRecipients $true -``` + ```powershell + Set-ExecutionPolicy RemoteSigned -Scope CurrentUser + Import-Module ExchangeOnlineManagement + Connect-ExchangeOnline + Set-OrganizationConfig -AllowPlusAddressInRecipients $true + Disconnect-ExchangeOnline + ``` This example for Omnibus GitLab assumes the mailbox `incoming@office365.example.com`: @@ -655,6 +669,10 @@ incoming_email: ##### Dedicated email address +NOTE: +Supports [Reply by Email](reply_by_email.md) only. +Cannot support [Service Desk](../user/project/service_desk.md). + This example for Omnibus installs assumes the dedicated email address `incoming@office365.example.com`: ```ruby diff --git a/doc/administration/index.md b/doc/administration/index.md index d78c9d80b5f..bd6549fca80 100644 --- a/doc/administration/index.md +++ b/doc/administration/index.md @@ -55,7 +55,7 @@ Learn how to install, configure, update, and maintain your GitLab instance. - [File hooks](file_hooks.md): With custom file hooks, GitLab administrators can introduce custom integrations without modifying GitLab source code. - [Enforcing Terms of Service](../user/admin_area/settings/terms.md) -- [Third party offers](../user/admin_area/settings/third_party_offers.md) +- [Customer experience improvement and third-party offers](../user/admin_area/settings/third_party_offers.md) - [Compliance](compliance.md): A collection of features from across the application that you may configure to help ensure that your GitLab instance and DevOps workflow meet compliance standards. @@ -172,7 +172,7 @@ Learn how to install, configure, update, and maintain your GitLab instance. - [Job artifacts](job_artifacts.md): Enable, disable, and configure job artifacts (a set of files and directories which are outputted by a job when it completes successfully). - [Job logs](job_logs.md): Information about the job logs. - [Register runners](../ci/runners/runners_scope.md): Learn how to register and configure runners. -- [Shared runners pipelines quota](../user/admin_area/settings/continuous_integration.md#shared-runners-pipeline-minutes-quota): Limit the usage of pipeline minutes for shared runners. +- [Shared runners quota of CI/CD minutes](../ci/pipelines/cicd_minutes.md): Limit the usage of CI/CD minutes for shared runners. - [Enable or disable Auto DevOps](../topics/autodevops/index.md#enable-or-disable-auto-devops): Enable or disable Auto DevOps for your instance. ## Snippet settings diff --git a/doc/administration/instance_limits.md b/doc/administration/instance_limits.md index bfe59d5277b..3bedb6b01bd 100644 --- a/doc/administration/instance_limits.md +++ b/doc/administration/instance_limits.md @@ -151,7 +151,7 @@ Plan.default.actual_limits.update!(web_hook_calls: 10) Set the limit to `0` to disable it. -- **Default rate limit**: Disabled. +- **Default rate limit**: Disabled (unlimited). ## Gitaly concurrency limit @@ -230,10 +230,8 @@ There is a limit when embedding metrics in GitLab Flavored Markdown (GFM) for pe ## Number of webhooks -On GitLab.com, the [maximum number of webhooks and their size](../user/gitlab_com/index.md#webhooks) per project, and per group, is limited. - -To set this limit for a self-managed installation, where the default is `100` project webhooks and `50` group webhooks, run the following in the -[GitLab Rails console](operations/rails_console.md#starting-a-rails-console-session): +To set the maximum number of group or project webhooks for a self-managed installation, +run the following in the [GitLab Rails console](operations/rails_console.md#starting-a-rails-console-session): ```ruby # If limits don't exist for the default plan, you can create one with: @@ -248,6 +246,11 @@ Plan.default.actual_limits.update!(group_hooks: 100) Set the limit to `0` to disable it. +- **Default maximum number of webhooks**: `100` per project, `50` per group +- **Maximum payload size**: 25 MB + +For GitLab.com, see the [webhook limits for GitLab.com](../user/gitlab_com/index.md#webhooks). + ## Pull Mirroring Interval > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/237891) in GitLab 13.7. @@ -640,7 +643,7 @@ To set this limit to `100` on a self-managed instance, run the following command [GitLab Rails console](operations/rails_console.md#starting-a-rails-console-session): ```ruby -Plan.default.actual_limits.update!(dotenv_variable_limit: 100) +Plan.default.actual_limits.update!(dotenv_variables: 100) ``` This limit is [enabled on GitLab.com](../user/gitlab_com/index.md#gitlab-cicd). @@ -658,7 +661,7 @@ To set this limit to 5KB on a self-managed installation, run the following in th [GitLab Rails console](operations/rails_console.md#starting-a-rails-console-session): ```ruby -Plan.default.actual_limits.update!(dotenv_size_limit: 5.kilobytes) +Plan.default.actual_limits.update!(dotenv_size: 5.kilobytes) ``` ## Instance monitoring and metrics diff --git a/doc/administration/instance_review.md b/doc/administration/instance_review.md index 872cdb239bd..f444589bdf3 100644 --- a/doc/administration/instance_review.md +++ b/doc/administration/instance_review.md @@ -4,26 +4,26 @@ group: Conversion info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# Instance Review **(FREE SELF)** +# Instance review **(FREE SELF)** -> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/6995) in GitLab 11.3. +If you run a self-managed instance with 50 or more users on the Free tier +([either Community Edition or unlicensed Enterprise Edition](https://about.gitlab.com/install/ce-or-ee/)), +you can request a free instance review. -If you run a medium-sized self-managed instance (50+ users) of a free version of GitLab, -[either Community Edition or unlicensed Enterprise Edition](https://about.gitlab.com/install/ce-or-ee/), -you qualify for a free Instance Review. +<!-- vale gitlab.FutureTense = NO --> -1. Sign in as an administrator. -1. In the top menu, click your user icon, and select - **Get a free instance review**: +After you submit the request, a GitLab team member will review your instance +details and contact you with suggestions to improve your use of GitLab. -  +<!-- vale gitlab.FutureTense = YES --> -1. GitLab redirects you to a form with prefilled data obtained from your instance. -1. Click **Submit** to see the initial report. +To request an instance review: -<!-- vale gitlab.FutureTense = NO --> +1. Sign in as an administrator. +1. On the top bar, in the top right corner, select your avatar. +1. Select **Get a free instance review**. -You will be contacted by a GitLab team member for further review, to provide suggestions -intended to help you improve your usage of GitLab. +  -<!-- vale gitlab.FutureTense = YES --> +1. On the instance review page, enter your contact details. +1. Select **Request Instance Review**. diff --git a/doc/administration/job_artifacts.md b/doc/administration/job_artifacts.md index 64b5ddbd165..62c403bfe43 100644 --- a/doc/administration/job_artifacts.md +++ b/doc/administration/job_artifacts.md @@ -6,7 +6,8 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Jobs artifacts administration **(FREE SELF)** -This is the administration documentation. For the user guide see [pipelines/job_artifacts](../ci/pipelines/job_artifacts.md). +This is the administration documentation. To learn how to use job artifacts in your GitLab CI/CD pipeline, +see the [job artifacts configuration documentation](../ci/pipelines/job_artifacts.md). An artifact is a list of files and directories attached to a job after it finishes. This feature is enabled by default in all GitLab installations. diff --git a/doc/administration/job_logs.md b/doc/administration/job_logs.md index f2748305c24..5c6ea7f52eb 100644 --- a/doc/administration/job_logs.md +++ b/doc/administration/job_logs.md @@ -150,8 +150,8 @@ server, these two locations on the file system have to be shared using NFS. To eliminate both file system requirements: -- [Enable the incremental logging feature](#enable-or-disable-incremental-logging), which uses Redis instead of disk space for temporary caching of job logs. -- Configure [object storage](job_artifacts.md#object-storage-settings) for storing archived job logs. +1. Configure [object storage](job_artifacts.md#object-storage-settings) for storing archived job logs. +1. [Enable the incremental logging feature](#enable-or-disable-incremental-logging), which uses Redis instead of disk space for temporary caching of job logs. ### Technical details diff --git a/doc/administration/logs.md b/doc/administration/logs.md index 263fe699529..0d7635405d6 100644 --- a/doc/administration/logs.md +++ b/doc/administration/logs.md @@ -20,6 +20,54 @@ including adjusting log retention, log forwarding, switching logs from JSON to plain text logging, and more. - [How to parse and analyze JSON logs](troubleshooting/log_parsing.md). +## Log Levels + +Each log message has an assigned log level that indicates its importance and verbosity. +Each logger has an assigned minimum log level. +A logger emits a log message only if its log level is equal to or above the minimum log level. + +The following log levels are supported: + +| Level | Name | +|:------|:----------| +| 0 | `DEBUG` | +| 1 | `INFO` | +| 2 | `WARN` | +| 3 | `ERROR` | +| 4 | `FATAL` | +| 5 | `UNKNOWN` | + +GitLab loggers emit all log messages because they are set to `DEBUG` by default. + +### Override default log level + +You can override the minimum log level for GitLab loggers using the `GITLAB_LOG_LEVEL` environment variable. +Valid values are either a value of `0` to `5`, or the name of the log level. + +Example: + +```shell +GITLAB_LOG_LEVEL=info +``` + +For some services, other log levels are in place that are not affected by this setting. +Some of these services have their own environment variables to override the log level. For example: + +| Service | Log level | Environment variable | +|:---------------------|:----------|:---------------------| +| GitLab API | `INFO` | | +| GitLab Cleanup | `INFO` | `DEBUG` | +| GitLab Doctor | `INFO` | `VERBOSE` | +| GitLab Export | `INFO` | `EXPORT_DEBUG` | +| GitLab Geo | `INFO` | | +| GitLab Import | `INFO` | `IMPORT_DEBUG` | +| GitLab QA Runtime | `ERROR` | `QA_DEBUG` | +| Google APIs | `INFO` | | +| Rack Timeout | `ERROR` | | +| Sidekiq (server) | `INFO` | | +| Snowplow Tracker | `FATAL` | | +| gRPC Client (Gitaly) | `WARN` | `GRPC_LOG_LEVEL` | + ## Log Rotation The logs for a given service may be managed and rotated by: @@ -36,26 +84,26 @@ are written to a file called `current`. The `logrotate` service built into GitLa [manages all logs](https://docs.gitlab.com/omnibus/settings/logs.html#logrotate) except those captured by `runit`. -| Log type | Managed by logrotate | Managed by svlogd/runit | -|-------------------------------------------------|------------------------|-------------------------| -| [Alertmanager Logs](#alertmanager-logs) | **{dotted-circle}** No | **{check-circle}** Yes | -| [Crond Logs](#crond-logs) | **{dotted-circle}** No | **{check-circle}** Yes | -| [Gitaly](#gitaly-logs) | **{check-circle}** Yes | **{check-circle}** Yes | -| [GitLab Exporter for Omnibus](#gitlab-exporter) | **{dotted-circle}** No | **{check-circle}** Yes | -| [GitLab Pages Logs](#pages-logs) | **{check-circle}** Yes | **{check-circle}** Yes | -| GitLab Rails | **{check-circle}** Yes | **{dotted-circle}** No | -| [GitLab Shell Logs](#gitlab-shelllog) | **{check-circle}** Yes | **{dotted-circle}** No | -| [Grafana Logs](#grafana-logs) | **{dotted-circle}** No | **{check-circle}** Yes | -| [LogRotate Logs](#logrotate-logs) | **{dotted-circle}** No | **{check-circle}** Yes | -| [Mailroom](#mail_room_jsonlog-default) | **{check-circle}** Yes | **{check-circle}** Yes | -| [NGINX](#nginx-logs) | **{check-circle}** Yes | **{check-circle}** Yes | -| [PostgreSQL Logs](#postgresql-logs) | **{dotted-circle}** No | **{check-circle}** Yes | -| [Praefect Logs](#praefect-logs) | **{dotted-circle}** Yes| **{check-circle}** Yes | -| [Prometheus Logs](#prometheus-logs) | **{dotted-circle}** No | **{check-circle}** Yes | -| [Puma](#puma-logs) | **{check-circle}** Yes | **{check-circle}** Yes | -| [Redis Logs](#redis-logs) | **{dotted-circle}** No | **{check-circle}** Yes | -| [Registry Logs](#registry-logs) | **{dotted-circle}** No | **{check-circle}** Yes | -| [Workhorse Logs](#workhorse-logs) | **{check-circle}** Yes | **{check-circle}** Yes | +| Log type | Managed by logrotate | Managed by svlogd/runit | +|:------------------------------------------------|:------------------------|:------------------------| +| [Alertmanager Logs](#alertmanager-logs) | **{dotted-circle}** No | **{check-circle}** Yes | +| [Crond Logs](#crond-logs) | **{dotted-circle}** No | **{check-circle}** Yes | +| [Gitaly](#gitaly-logs) | **{check-circle}** Yes | **{check-circle}** Yes | +| [GitLab Exporter for Omnibus](#gitlab-exporter) | **{dotted-circle}** No | **{check-circle}** Yes | +| [GitLab Pages Logs](#pages-logs) | **{check-circle}** Yes | **{check-circle}** Yes | +| GitLab Rails | **{check-circle}** Yes | **{dotted-circle}** No | +| [GitLab Shell Logs](#gitlab-shelllog) | **{check-circle}** Yes | **{dotted-circle}** No | +| [Grafana Logs](#grafana-logs) | **{dotted-circle}** No | **{check-circle}** Yes | +| [LogRotate Logs](#logrotate-logs) | **{dotted-circle}** No | **{check-circle}** Yes | +| [Mailroom](#mail_room_jsonlog-default) | **{check-circle}** Yes | **{check-circle}** Yes | +| [NGINX](#nginx-logs) | **{check-circle}** Yes | **{check-circle}** Yes | +| [PostgreSQL Logs](#postgresql-logs) | **{dotted-circle}** No | **{check-circle}** Yes | +| [Praefect Logs](#praefect-logs) | **{dotted-circle}** Yes | **{check-circle}** Yes | +| [Prometheus Logs](#prometheus-logs) | **{dotted-circle}** No | **{check-circle}** Yes | +| [Puma](#puma-logs) | **{check-circle}** Yes | **{check-circle}** Yes | +| [Redis Logs](#redis-logs) | **{dotted-circle}** No | **{check-circle}** Yes | +| [Registry Logs](#registry-logs) | **{dotted-circle}** No | **{check-circle}** Yes | +| [Workhorse Logs](#workhorse-logs) | **{check-circle}** Yes | **{check-circle}** Yes | ## `production_json.log` diff --git a/doc/administration/maintenance_mode/index.md b/doc/administration/maintenance_mode/index.md index 2d17062e955..50c0f0ecc63 100644 --- a/doc/administration/maintenance_mode/index.md +++ b/doc/administration/maintenance_mode/index.md @@ -193,7 +193,8 @@ Replication and verification continues to work but proxied Git pushes to primary ### Secure features -Features that depend on creating issues or creating or approving Merge Requests, do not work. +Features that depend on creating issues or creating or approving merge requests, +do not work. Exporting a vulnerability list from a Vulnerability Report page does not work. diff --git a/doc/administration/merge_request_diffs.md b/doc/administration/merge_request_diffs.md index d6b9fa2b8d3..01576eb4abf 100644 --- a/doc/administration/merge_request_diffs.md +++ b/doc/administration/merge_request_diffs.md @@ -1,14 +1,11 @@ --- stage: Create group: Editor -info: "To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments" -type: reference +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- # Merge request diffs storage **(FREE SELF)** -> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/52568) in GitLab 11.8. - Merge request diffs are size-limited copies of diffs associated with merge requests. When viewing a merge request, diffs are sourced from these copies wherever possible as a performance optimization. @@ -17,7 +14,8 @@ By default, merge request diffs are stored in the database, in a table named `merge_request_diff_files`. Larger installations may find this table grows too large, in which case, switching to external storage is recommended. -Merge request diffs can be stored on disk, or in object storage. In general, it +Merge request diffs can be stored [on disk](#using-external-storage), or in +[object storage](#using-object-storage). In general, it is better to store the diffs in the database than on disk. A compromise is available that only [stores outdated diffs](#alternative-in-database-storage) outside of database. @@ -41,6 +39,7 @@ that only [stores outdated diffs](#alternative-in-database-storage) outside of d ``` 1. Save the file and [reconfigure GitLab](restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. + GitLab then migrates your existing merge request diffs to external storage. **In installations from source:** @@ -64,6 +63,7 @@ that only [stores outdated diffs](#alternative-in-database-storage) outside of d ``` 1. Save the file and [restart GitLab](restart_gitlab.md#installations-from-source) for the changes to take effect. + GitLab then migrates your existing merge request diffs to external storage. ## Using object storage @@ -84,6 +84,7 @@ be configured already. 1. Set [object storage settings](#object-storage-settings). 1. Save the file and [reconfigure GitLab](restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. + GitLab then migrates your existing merge request diffs to external storage. **In installations from source:** @@ -97,6 +98,7 @@ be configured already. 1. Set [object storage settings](#object-storage-settings). 1. Save the file and [restart GitLab](restart_gitlab.md#installations-from-source) for the changes to take effect. + GitLab then migrates your existing merge request diffs to external storage. [Read more about using object storage with GitLab](object_storage.md). diff --git a/doc/administration/monitoring/github_imports.md b/doc/administration/monitoring/github_imports.md index cd35b8b3f9e..e91483eb79d 100644 --- a/doc/administration/monitoring/github_imports.md +++ b/doc/administration/monitoring/github_imports.md @@ -6,8 +6,6 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Monitoring GitHub imports **(FREE SELF)** -> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/14731) in GitLab 10.2. - The GitHub importer exposes various Prometheus metrics that you can use to monitor the health and progress of the importer. diff --git a/doc/administration/monitoring/ip_whitelist.md b/doc/administration/monitoring/ip_whitelist.md index b6df176ea87..75b09f8a366 100644 --- a/doc/administration/monitoring/ip_whitelist.md +++ b/doc/administration/monitoring/ip_whitelist.md @@ -6,8 +6,6 @@ info: To determine the technical writer assigned to the Stage/Group associated w # IP whitelist **(FREE SELF)** -> Introduced in GitLab 9.4. - NOTE: We intend to [rename IP whitelist as `IP allowlist`](https://gitlab.com/groups/gitlab-org/-/epics/3478). diff --git a/doc/administration/monitoring/prometheus/gitlab_metrics.md b/doc/administration/monitoring/prometheus/gitlab_metrics.md index 4a504b07a1b..c5b87afd94b 100644 --- a/doc/administration/monitoring/prometheus/gitlab_metrics.md +++ b/doc/administration/monitoring/prometheus/gitlab_metrics.md @@ -22,10 +22,8 @@ GitLab monitors its own internal service metrics, and makes them available at th `/-/metrics` endpoint. Unlike other [Prometheus](https://prometheus.io) exporters, to access the metrics, the client IP address must be [explicitly allowed](../ip_whitelist.md). -For [Omnibus GitLab](https://docs.gitlab.com/omnibus/) and Chart installations, -these metrics are enabled and collected as of -[GitLab 9.4](https://gitlab.com/gitlab-org/omnibus-gitlab/-/merge_requests/1702). -For source installations, these metrics must be enabled +These metrics are enabled and collected for [Omnibus GitLab](https://docs.gitlab.com/omnibus/) +and Chart installations. For source installations, these metrics must be enabled manually and collected by a Prometheus server. For enabling and viewing metrics from Sidekiq nodes, see [Sidekiq metrics](#sidekiq-metrics). @@ -107,7 +105,7 @@ The following metrics are available: | `pipelines_created_total` | Counter | 9.4 | Counter of pipelines created | | | `rack_uncaught_errors_total` | Counter | 9.4 | Rack connections handling uncaught errors count | | | `user_session_logins_total` | Counter | 9.4 | Counter of how many users have logged in since GitLab was started or restarted | | -| `upload_file_does_not_exist` | Counter | 10.7 | Number of times an upload record could not find its file. Made available in all tiers in GitLab 11.5. | | +| `upload_file_does_not_exist` | Counter | 10.7 | Number of times an upload record could not find its file. | | | `failed_login_captcha_total` | Gauge | 11.0 | Counter of failed CAPTCHA attempts during login | | | `successful_login_captcha_total` | Gauge | 11.0 | Counter of successful CAPTCHA attempts during login | | | `auto_devops_pipelines_completed_total` | Counter | 12.7 | Counter of completed Auto DevOps pipelines, labeled by status | | @@ -288,7 +286,7 @@ configuration option in `gitlab.yml`. These metrics are served from the | `geo_uploads_verified` | Gauge | 14.6 | Number of uploads verified on secondary | `url` | | `geo_uploads_verification_failed` | Gauge | 14.6 | Number of uploads verifications failed on secondary | `url` | | `gitlab_sli:rails_request_apdex:total` | Counter | 14.4 | The number of request-apdex measurements, [more information the development documentation](../../../development/application_slis/rails_request_apdex.md) | `endpoint_id`, `feature_category`, `request_urgency` | -| `gitlab_sli:rails_request_apdex:success_total` | Counter | 14.4 | The number of succesful requests that met the target duration for their urgency. Devide by `gitlab_sli:rails_requests_apdex:total` to get a success ratio | `endpoint_id`, `feature_category`, `request_urgency` | +| `gitlab_sli:rails_request_apdex:success_total` | Counter | 14.4 | The number of successful requests that met the target duration for their urgency. Divide by `gitlab_sli:rails_requests_apdex:total` to get a success ratio | `endpoint_id`, `feature_category`, `request_urgency` | ## Database load balancing metrics **(PREMIUM SELF)** diff --git a/doc/administration/monitoring/prometheus/index.md b/doc/administration/monitoring/prometheus/index.md index 3268c0fc14c..acdcdb41dca 100644 --- a/doc/administration/monitoring/prometheus/index.md +++ b/doc/administration/monitoring/prometheus/index.md @@ -1,6 +1,6 @@ --- -stage: Monitor -group: Monitor +stage: Enablement +group: Distribution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- @@ -59,8 +59,8 @@ Although possible, it's not recommended to change the port Prometheus listens on, as this might affect or conflict with other services running on the GitLab server. Proceed at your own risk. -To access Prometheus from outside the GitLab server, set an FQDN or IP in -`prometheus['listen_address']`. To change the address/port that Prometheus +To access Prometheus from outside the GitLab server, +change the address/port that Prometheus listens on: 1. Edit `/etc/gitlab/gitlab.rb` diff --git a/doc/administration/monitoring/prometheus/pgbouncer_exporter.md b/doc/administration/monitoring/prometheus/pgbouncer_exporter.md index d42c471ac71..aba1561500a 100644 --- a/doc/administration/monitoring/prometheus/pgbouncer_exporter.md +++ b/doc/administration/monitoring/prometheus/pgbouncer_exporter.md @@ -6,8 +6,6 @@ info: To determine the technical writer assigned to the Stage/Group associated w # PgBouncer exporter **(FREE SELF)** -> Introduced in [Omnibus GitLab 11.0](https://gitlab.com/gitlab-org/omnibus-gitlab/-/merge_requests/2493). - The [PgBouncer exporter](https://github.com/prometheus-community/pgbouncer_exporter) enables you to measure various [PgBouncer](https://www.pgbouncer.org/) metrics. diff --git a/doc/administration/monitoring/prometheus/registry_exporter.md b/doc/administration/monitoring/prometheus/registry_exporter.md index 87b51aaed08..3a2acd47338 100644 --- a/doc/administration/monitoring/prometheus/registry_exporter.md +++ b/doc/administration/monitoring/prometheus/registry_exporter.md @@ -6,8 +6,6 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Registry exporter **(FREE SELF)** -> [Introduced](https://gitlab.com/gitlab-org/omnibus-gitlab/-/merge_requests/2884) in GitLab 11.9. - The Registry exporter allows you to measure various Registry metrics. To enable it: diff --git a/doc/administration/nfs.md b/doc/administration/nfs.md index a0170e6c4ef..87a10a0ec04 100644 --- a/doc/administration/nfs.md +++ b/doc/administration/nfs.md @@ -12,6 +12,8 @@ recommended for performance reasons. For data objects such as LFS, Uploads, Artifacts, and so on, an [Object Storage service](object_storage.md) is recommended over NFS where possible, due to better performance. +When eliminating the usage of NFS, there are [additional steps you need to take](object_storage.md#other-alternatives-to-file-system-storage) +in addition to moving to Object Storage. File system performance can impact overall GitLab performance, especially for actions that read or write to Git repositories. For steps you can use to test diff --git a/doc/administration/object_storage.md b/doc/administration/object_storage.md index c6490e365a5..0976fc3684e 100644 --- a/doc/administration/object_storage.md +++ b/doc/administration/object_storage.md @@ -48,6 +48,9 @@ There are two ways of specifying object storage configuration in GitLab: For more information on the differences and to transition from one form to another, see [Transition to consolidated form](#transition-to-consolidated-form). +If you are currently storing data locally, see +[Migrate to object storage](#migrate-to-object-storage) for migration details. + ### Consolidated object storage configuration > [Introduced](https://gitlab.com/gitlab-org/omnibus-gitlab/-/merge_requests/4368) in GitLab 13.2. @@ -485,9 +488,9 @@ This is the list of valid `objects` that can be used: | `uploads` | [User uploads](uploads.md) | | `lfs` | [Git Large File Storage objects](lfs/index.md) | | `packages` | [Project packages (for example, PyPI, Maven, or NuGet)](packages/index.md) | -| `dependency_proxy` | [GitLab Dependency Proxy](packages/dependency_proxy.md) | +| `dependency_proxy` | [Dependency Proxy](packages/dependency_proxy.md) | | `terraform_state` | [Terraform state files](terraform_state.md) | -| `pages` | [GitLab Pages](pages/index.md) | +| `pages` | [Pages](pages/index.md) | Within each object type, three parameters can be defined: @@ -514,6 +517,19 @@ no bucket is needed if CI artifacts are disabled with this setting: gitlab_rails['artifacts_enabled'] = false ``` +### Migrate to object storage + +To migrate existing local data to object storage see the following guides: + +- [Job artifacts](job_artifacts.md#migrating-to-object-storage) including archived job logs +- [LFS objects](lfs/index.md#migrating-to-object-storage) +- [Uploads](raketasks/uploads/migrate.md#migrate-to-object-storage) +- [Merge request diffs](merge_request_diffs.md#using-object-storage) +- [Packages](packages/index.md#migrating-local-packages-to-object-storage) (optional feature) +- Dependency Proxy - [migration not yet supported](https://gitlab.com/gitlab-org/gitlab/-/issues/343064) +- [Terraform state files](terraform_state.md#migrate-to-object-storage) +- [Pages content](pages/index.md#migrate-pages-deployments-to-object-storage) + ### Transition to consolidated form Prior to GitLab 13.2: @@ -565,11 +581,11 @@ supported by consolidated configuration form, refer to the following guides: | [Merge request diffs](merge_request_diffs.md#using-object-storage) | **{check-circle}** Yes | | [Mattermost](https://docs.mattermost.com/administration/config-settings.html#file-storage)| **{dotted-circle}** No | | [Packages](packages/index.md#using-object-storage) (optional feature) | **{check-circle}** Yes | -| [Dependency Proxy](packages/dependency_proxy.md#using-object-storage) (optional feature) **(PREMIUM SELF)** | **{check-circle}** Yes | +| [Dependency Proxy](packages/dependency_proxy.md#using-object-storage) (optional feature) | **{check-circle}** Yes | | [Pseudonymizer](pseudonymizer.md) (optional feature) | **{dotted-circle}** No | | [Autoscale runner caching](https://docs.gitlab.com/runner/configuration/autoscale.html#distributed-runners-caching) (optional for improved performance) | **{dotted-circle}** No | | [Terraform state files](terraform_state.md#using-object-storage) | **{check-circle}** Yes | -| [GitLab Pages content](pages/index.md#using-object-storage) | **{check-circle}** Yes | +| [Pages content](pages/index.md#using-object-storage) | **{check-circle}** Yes | ### Other alternatives to file system storage @@ -585,6 +601,12 @@ See the following additional guides: ## Warnings, limitations, and known issues +### Objects are not included in GitLab backups + +As noted in [our backup documentation](../raketasks/backup_restore.md), +objects are not included in GitLab backups. You can enable backups with +your object storage provider instead. + ### Use separate buckets Using separate buckets for each data type is the recommended approach for GitLab. diff --git a/doc/administration/operations/cleaning_up_redis_sessions.md b/doc/administration/operations/cleaning_up_redis_sessions.md deleted file mode 100644 index ed5014b65e1..00000000000 --- a/doc/administration/operations/cleaning_up_redis_sessions.md +++ /dev/null @@ -1,9 +0,0 @@ ---- -redirect_to: 'index.md' -remove_date: '2021-12-10' ---- - -This document was moved to [another location](index.md). - -<!-- This redirect file can be deleted after 2021-12-10. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page --> diff --git a/doc/administration/operations/fast_ssh_key_lookup.md b/doc/administration/operations/fast_ssh_key_lookup.md index c99f94589d7..ddedb3fe76a 100644 --- a/doc/administration/operations/fast_ssh_key_lookup.md +++ b/doc/administration/operations/fast_ssh_key_lookup.md @@ -27,13 +27,13 @@ lookup of authorized SSH keys. WARNING: OpenSSH version 6.9+ is required because `AuthorizedKeysCommand` must be -able to accept a fingerprint. Check the version of OpenSSH on your server. +able to accept a fingerprint. Check the version of OpenSSH on your server with `sshd -V`. ## Fast lookup is required for Geo **(PREMIUM)** By default, GitLab manages an `authorized_keys` file that is located in the `git` user's home directory. For most installations, this will be located under -`/var/opt/gitlab/.ssh/authorized_keys`, but you can use the following command to locate the `authorized_keys` on your system.: +`/var/opt/gitlab/.ssh/authorized_keys`, but you can use the following command to locate the `authorized_keys` on your system: ```shell getent passwd git | cut -d: -f6 | awk '{print $1"/.ssh/authorized_keys"}' @@ -77,9 +77,13 @@ sudo service sshd reload ``` Confirm that SSH is working by commenting out your user's key in the `authorized_keys` -file (start the line with a `#` to comment it), and attempting to pull a repository. +file (start the line with a `#` to comment it), and from your local machine, attempt to pull a repository or run: -A successful pull would mean that GitLab was able to find the key in the database, +```shell +ssh -T git@gitlab.example.com +``` + +A successful pull or [welcome message](../../ssh/index.md#verify-that-you-can-connect) would mean that GitLab was able to find the key in the database, since it is not present in the file anymore. NOTE: @@ -114,7 +118,7 @@ adding a new one, and attempting to pull a repository. Then you can backup and delete your `authorized_keys` file for best performance. The current users' keys are already present in the database, so there is no need for migration -or for asking users to re-add their keys. +or for users to re-add their keys. ## How to go back to using the `authorized_keys` file diff --git a/doc/administration/operations/moving_repositories.md b/doc/administration/operations/moving_repositories.md index 84e6dca1f2b..f0eb5792a96 100644 --- a/doc/administration/operations/moving_repositories.md +++ b/doc/administration/operations/moving_repositories.md @@ -184,8 +184,8 @@ Each of the approaches we list can or does overwrite data in the target director ### Recommended approach in all cases -The GitLab [backup and restore capability](../../raketasks/backup_restore.md) should be used. Git -repositories are accessed, managed, and stored on GitLab servers by Gitaly as a database. Data loss +For either Gitaly or Gitaly Cluster targets, the GitLab [backup and restore capability](../../raketasks/backup_restore.md) +should be used. Git repositories are accessed, managed, and stored on GitLab servers by Gitaly as a database. Data loss can result from directly accessing and copying Gitaly's files using tools like `rsync`. - From GitLab 13.3, backup performance can be improved by @@ -193,13 +193,15 @@ can result from directly accessing and copying Gitaly's files using tools like ` - Backups can be created of just the repositories using the [skip feature](../../raketasks/backup_restore.md#excluding-specific-directories-from-the-backup). +No other method works for Gitaly Cluster targets. + ### Target directory is empty: use a `tar` pipe -If the target directory `/mnt/gitlab/repositories` is empty the -simplest thing to do is to use a `tar` pipe. This method has low -overhead and `tar` is almost always already installed on your system. -However, it is not possible to resume an interrupted `tar` pipe: if -that happens then all data must be copied again. +For Gitaly targets (use [recommended approach](#recommended-approach-in-all-cases) for Gitaly Cluster targets), if the +target directory `/mnt/gitlab/repositories` is empty the simplest thing to do is to use a `tar` pipe. This method has +low overhead and `tar` is almost always already installed on your system. + +However, it is not possible to resume an interrupted `tar` pipe; if that happens then all data must be copied again. ```shell sudo -u git sh -c 'tar -C /var/opt/gitlab/git-data/repositories -cf - -- . |\ @@ -210,9 +212,9 @@ If you want to see progress, replace `-xf` with `-xvf`. #### `tar` pipe to another server -You can also use a `tar` pipe to copy data to another server. If your -`git` user has SSH access to the new server as `git@newserver`, you -can pipe the data through SSH. +For Gitaly targets (use [recommended approach](#recommended-approach-in-all-cases) for Gitaly Cluster targets), you can +also use a `tar` pipe to copy data to another server. If your `git` user has SSH access to the new server as +`git@newserver`, you can pipe the data through SSH. ```shell sudo -u git sh -c 'tar -C /var/opt/gitlab/git-data/repositories -cf - -- . |\ @@ -228,11 +230,11 @@ WARNING: Using `rsync` to migrate Git data can cause data loss and repository corruption. [These instructions are being reviewed](https://gitlab.com/gitlab-org/gitlab/-/issues/270422). -If the target directory already contains a partial / outdated copy -of the repositories it may be wasteful to copy all the data again -with `tar`. In this scenario it is better to use `rsync`. This utility -is either already installed on your system, or installable -by using `apt` or `yum`. +If the target directory already contains a partial or outdated copy of the repositories it may be wasteful to copy all +the data again with `tar`. In this scenario it is better to use `rsync` for Gitaly targets (use +[recommended approach](#recommended-approach-in-all-cases) for Gitaly Cluster targets). + +This utility is either already installed on your system, or installable using `apt` or `yum`. ```shell sudo -u git sh -c 'rsync -a --delete /var/opt/gitlab/git-data/repositories/. \ @@ -249,8 +251,9 @@ WARNING: Using `rsync` to migrate Git data can cause data loss and repository corruption. [These instructions are being reviewed](https://gitlab.com/gitlab-org/gitlab/-/issues/270422). -If the `git` user on your source system has SSH access to the target -server you can send the repositories over the network with `rsync`. +For Gitaly targets (use [recommended approach](#recommended-approach-in-all-cases) for Gitaly Cluster targets), if the +`git` user on your source system has SSH access to the target server you can send the repositories over the network with +`rsync`. ```shell sudo -u git sh -c 'rsync -a --delete /var/opt/gitlab/git-data/repositories/. \ @@ -269,17 +272,18 @@ Every time you start an `rsync` job it must: - Inspect all files in the target directory. - Decide whether or not to copy files. -If the source or target directory -has many contents, this startup phase of `rsync` can become a burden -for your GitLab server. You can reduce the workload of `rsync` by dividing its -work in smaller pieces, and sync one repository at a time. +If the source or target directory has many contents, this startup phase of `rsync` can become a burden for your GitLab +server. You can reduce the workload of `rsync` by dividing its work into smaller pieces, and sync one repository at a +time. In addition to `rsync` we use [GNU Parallel](http://www.gnu.org/software/parallel/). This utility is not included in GitLab, so you must install it yourself with `apt` or `yum`. -This process does not clean up repositories at the target location that no -longer exist at the source. +This process: + +- Doesn't clean up repositories at the target location that no longer exist at the source. +- Only works for Gitaly targets. Use [recommended approach](#recommended-approach-in-all-cases) for Gitaly Cluster targets. #### Parallel `rsync` for all repositories known to GitLab diff --git a/doc/administration/operations/rails_console.md b/doc/administration/operations/rails_console.md index 8c366e311b8..a93365c08b2 100644 --- a/doc/administration/operations/rails_console.md +++ b/doc/administration/operations/rails_console.md @@ -35,7 +35,7 @@ sudo -u git -H bundle exec rails console -e production **For Kubernetes deployments** -The console is in the task-runner pod. Refer to our [Kubernetes cheat sheet](../troubleshooting/kubernetes_cheat_sheet.md#gitlab-specific-kubernetes-information) for details. +The console is in the toolbox pod. Refer to our [Kubernetes cheat sheet](../troubleshooting/kubernetes_cheat_sheet.md#gitlab-specific-kubernetes-information) for details. To exit the console, type: `quit`. diff --git a/doc/administration/package_information/defaults.md b/doc/administration/package_information/defaults.md index 95d6135c28c..54e68392aa5 100644 --- a/doc/administration/package_information/defaults.md +++ b/doc/administration/package_information/defaults.md @@ -1,7 +1,7 @@ --- stage: Enablement group: Distribution -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- # Package defaults **(FREE SELF)** @@ -14,42 +14,43 @@ the package will assume the defaults as noted below. See the table below for the list of ports that the Omnibus GitLab assigns by default: -| Component | On by default | Communicates via | Alternative | Connection port | -|:-------------------:|:-------------:|:----------------:|:-----------:|:------------------------------------------:| -| GitLab Rails | Yes | Port | X | 80 or 443 | -| GitLab Shell | Yes | Port | X | 22 | -| PostgreSQL | Yes | Socket | Port (5432) | X | -| Redis | Yes | Socket | Port (6379) | X | -| Puma | Yes | Socket | Port (8080) | X | -| GitLab Workhorse | Yes | Socket | Port (8181) | X | -| NGINX status | Yes | Port | X | 8060 | -| Prometheus | Yes | Port | X | 9090 | -| Node exporter | Yes | Port | X | 9100 | -| Redis exporter | Yes | Port | X | 9121 | -| PostgreSQL exporter | Yes | Port | X | 9187 | -| PgBouncer exporter | No | Port | X | 9188 | -| GitLab Exporter | Yes | Port | X | 9168 | -| Sidekiq exporter | Yes | Port | X | 8082 | -| Puma exporter | No | Port | X | 8083 | -| Geo PostgreSQL | No | Socket | Port (5431) | X | -| Redis Sentinel | No | Port | X | 26379 | -| Incoming email | No | Port | X | 143 | -| Elastic search | No | Port | X | 9200 | -| GitLab Pages | No | Port | X | 80 or 443 | -| GitLab Registry | No* | Port | X | 80, 443 or 5050 | -| GitLab Registry | No | Port | X | 5000 | -| LDAP | No | Port | X | Depends on the component configuration | -| Kerberos | No | Port | X | 8443 or 8088 | -| OmniAuth | Yes | Port | X | Depends on the component configuration | -| SMTP | No | Port | X | 465 | -| Remote syslog | No | Port | X | 514 | -| Mattermost | No | Port | X | 8065 | -| Mattermost | No | Port | X | 80 or 443 | -| PgBouncer | No | Port | X | 6432 | -| Consul | No | Port | X | 8300, 8301(UDP), 8500, 8600[^Consul-notes] | -| Patroni | No | Port | X | 8008 | -| GitLab KAS | No | Port | X | 8150 | -| Gitaly | No | Port | X | 8075 | +| Component | On by default | Communicates via | Alternative | Connection port | +|:--------------------:|:-------------:|:----------------:|:-----------:|:------------------------------------------:| +| GitLab Rails | Yes | Port | X | 80 or 443 | +| GitLab Shell | Yes | Port | X | 22 | +| PostgreSQL | Yes | Socket | Port (5432) | X | +| Redis | Yes | Socket | Port (6379) | X | +| Puma | Yes | Socket | Port (8080) | X | +| GitLab Workhorse | Yes | Socket | Port (8181) | X | +| NGINX status | Yes | Port | X | 8060 | +| Prometheus | Yes | Port | X | 9090 | +| Node exporter | Yes | Port | X | 9100 | +| Redis exporter | Yes | Port | X | 9121 | +| PostgreSQL exporter | Yes | Port | X | 9187 | +| PgBouncer exporter | No | Port | X | 9188 | +| GitLab Exporter | Yes | Port | X | 9168 | +| Sidekiq exporter | Yes | Port | X | 8082 | +| Sidekiq health check | No | Port | X | 8092[^Sidekiq-health] | +| Puma exporter | No | Port | X | 8083 | +| Geo PostgreSQL | No | Socket | Port (5431) | X | +| Redis Sentinel | No | Port | X | 26379 | +| Incoming email | No | Port | X | 143 | +| Elastic search | No | Port | X | 9200 | +| GitLab Pages | No | Port | X | 80 or 443 | +| GitLab Registry | No* | Port | X | 80, 443 or 5050 | +| GitLab Registry | No | Port | X | 5000 | +| LDAP | No | Port | X | Depends on the component configuration | +| Kerberos | No | Port | X | 8443 or 8088 | +| OmniAuth | Yes | Port | X | Depends on the component configuration | +| SMTP | No | Port | X | 465 | +| Remote syslog | No | Port | X | 514 | +| Mattermost | No | Port | X | 8065 | +| Mattermost | No | Port | X | 80 or 443 | +| PgBouncer | No | Port | X | 6432 | +| Consul | No | Port | X | 8300, 8301(UDP), 8500, 8600[^Consul-notes] | +| Patroni | No | Port | X | 8008 | +| GitLab KAS | No | Port | X | 8150 | +| Gitaly | No | Port | X | 8075 | Legend: @@ -70,3 +71,5 @@ NOTE: In some cases, the GitLab Registry will be automatically enabled by default. Please see [our documentation](../packages/container_registry.md) for more details [^Consul-notes]: If using additional Consul functionality, more ports may need to be opened. See the [official documentation](https://www.consul.io/docs/install/ports#ports-table) for the list. + + [^Sidekiq-health]: If Sidekiq health check settings are not set, they will default to the Sidekiq metrics exporter settings. This default is deprecated and is set to be removed in [GitLab 15.0](https://gitlab.com/gitlab-org/gitlab/-/issues/347509). diff --git a/doc/administration/package_information/deprecation_policy.md b/doc/administration/package_information/deprecation_policy.md index 905de387dcb..7298bce6c95 100644 --- a/doc/administration/package_information/deprecation_policy.md +++ b/doc/administration/package_information/deprecation_policy.md @@ -38,7 +38,7 @@ We can differentiate two different types of configuration: - Sensitive: Configuration that can cause major service outage (like data integrity, installation integrity, or preventing users from reaching the installation) - Regular: Configuration that can make a feature unavailable but still makes the - installation useable (like a change in default project/group settings, or + installation usable (like a change in default project/group settings, or miscommunication with other components) We also need to differentiate deprecation and removal procedure. diff --git a/doc/administration/package_information/index.md b/doc/administration/package_information/index.md index ab4b1edfa30..2781f789409 100644 --- a/doc/administration/package_information/index.md +++ b/doc/administration/package_information/index.md @@ -76,7 +76,7 @@ characters on each line. ## Init system detection -Omnibus GitLab attempts to query the underlaying system in order to +Omnibus GitLab attempts to query the underlying system in order to check which init system it uses. This manifests itself as a `WARNING` during the `sudo gitlab-ctl reconfigure` run. diff --git a/doc/administration/package_information/postgresql_versions.md b/doc/administration/package_information/postgresql_versions.md index 252e0cad76d..97a35fb29ed 100644 --- a/doc/administration/package_information/postgresql_versions.md +++ b/doc/administration/package_information/postgresql_versions.md @@ -26,7 +26,7 @@ Read more about update policies and warnings in the PostgreSQL | GitLab version | PostgreSQL versions | Default version for fresh installs | Default version for upgrades | Notes | | -------------- | --------------------- | ---------------------------------- | ---------------------------- | ----- | -| 14.1 | 12.6, 13.3 | 12.6 | 12.6 | PostgreSQL 13 available for fresh installations if not using Geo or High Availability. | +| 14.1 | 12.6, 13.3 | 12.6 | 12.6 | PostgreSQL 13 available for fresh installations if not using [Geo](../geo/index.md#requirements-for-running-geo) or [Patroni](../postgresql/index.md#postgresql-replication-and-failover-with-omnibus-gitlab). | 14.0 | 12.6 | 12.6 | 12.6 | HA installations with repmgr are no longer supported and will be prevented from upgrading to Omnibus GitLab 14.0 | | 13.8 | 11.9, 12.4 | 12.4 | 12.4 | Package upgrades automatically performed PostgreSQL upgrade for nodes that are not part of a Geo or HA cluster.). | | 13.7 | 11.9, 12.4 | 12.4 | 11.9 | For upgrades users can manually upgrade to 12.4 following the [upgrade docs](https://docs.gitlab.com/omnibus/settings/database.html#gitlab-133-and-later). | diff --git a/doc/administration/package_information/supported_os.md b/doc/administration/package_information/supported_os.md index fcc2fef3e63..78f496a3998 100644 --- a/doc/administration/package_information/supported_os.md +++ b/doc/administration/package_information/supported_os.md @@ -17,6 +17,7 @@ The following lists the currently supported OSs and their possible EOL dates. | OS Version | First supported GitLab version | Arch | OS EOL | Details | | ---------------- | ------------------------------ | --------------- | ------------- | ------------------------------------------------------------ | +| AlmaLinux 8 | GitLab CE / GitLab EE 14.5.0 | x86_64, aarch64 | 2029 | <https://almalinux.org/> | | CentOS 7 | GitLab CE / GitLab EE 7.10.0 | x86_64 | June 2024 | <https://wiki.centos.org/About/Product> | | CentOS 8 | GitLab CE / GitLab EE 12.8.1 | x86_64, aarch64 | Dec 2021 | <https://wiki.centos.org/About/Product> | | Debian 9 | GitLab CE / GitLab EE 9.3.0 | amd64 | 2022 | <https://wiki.debian.org/LTS> | diff --git a/doc/administration/packages/container_registry.md b/doc/administration/packages/container_registry.md index 0877fe510de..43293385ed9 100644 --- a/doc/administration/packages/container_registry.md +++ b/doc/administration/packages/container_registry.md @@ -694,7 +694,7 @@ project, you can [disable it from your project's settings](../../user/project/se ## Use an external container registry with GitLab as an auth endpoint If you use an external container registry, some features associated with the -container registry may be unavailable or have [inherent risks](../../user/packages/container_registry/index.md#use-with-external-container-registries). +container registry may be unavailable or have [inherent risks](../../user/packages/container_registry/reduce_container_registry_storage.md#use-with-external-container-registries). For the integration to work, the external registry must be configured to use a JSON Web Token to authenticate with GitLab. The @@ -883,7 +883,7 @@ project.container_repositories.find_each do |repo| end ``` -You can also [run cleanup on a schedule](../../user/packages/container_registry/index.md#cleanup-policy). +You can also [run cleanup on a schedule](../../user/packages/container_registry/reduce_container_registry_storage.md#cleanup-policy). ## Container Registry garbage collection @@ -965,8 +965,6 @@ understand the implications. ### Removing untagged manifests and unreferenced layers -> [Introduced](https://gitlab.com/gitlab-org/omnibus-gitlab/-/merge_requests/3097) in Omnibus GitLab 11.10. - WARNING: This is a destructive operation. @@ -1341,7 +1339,10 @@ Start with a value between `25000000` (25MB) and `50000000` (50MB). ### Supporting older Docker clients -As of GitLab 11.9, we began shipping version 2.7.1 of the Docker container registry, which disables the schema1 manifest by default. If you are still using older Docker clients (1.9 or older), you may experience an error pushing images. See [omnibus-4145](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/4145) for more details. +The Docker container registry shipped with GitLab disables the schema1 manifest +by default. If you are still using older Docker clients (1.9 or older), you may +experience an error pushing images. See +[omnibus-4145](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/4145) for more details. You can add a configuration option for backwards compatibility. diff --git a/doc/administration/packages/dependency_proxy.md b/doc/administration/packages/dependency_proxy.md index a394a32fc18..b3dc6ffc2b2 100644 --- a/doc/administration/packages/dependency_proxy.md +++ b/doc/administration/packages/dependency_proxy.md @@ -14,68 +14,84 @@ GitLab can be used as a dependency proxy for a variety of common package manager This is the administration documentation. If you want to learn how to use the dependency proxies, see the [user guide](../../user/packages/dependency_proxy/index.md). -## Enabling the Dependency Proxy feature +The GitLab Dependency Proxy: -NOTE: -Dependency proxy requires the Puma web server to be enabled. +- Is turned on by default. +- Can be turned off by an administrator. +- Requires the [Puma web server](../operations/puma.md) + to be enabled. Puma is enabled by default in GitLab 13.0 and later. -To enable the dependency proxy feature: +## Turn off the Dependency Proxy -**Omnibus GitLab installations** +The Dependency Proxy is enabled by default. If you are an administrator, you +can turn off the Dependency Proxy. To turn off the Dependency Proxy, follow the instructions that +correspond to your GitLab installation: + +- [Omnibus GitLab installations](#omnibus-gitlab-installations) +- [Helm chart installations](#helm-chart-installations) +- [Installations from source](#installations-from-source) + +### Omnibus GitLab installations 1. Edit `/etc/gitlab/gitlab.rb` and add the following line: ```ruby - gitlab_rails['dependency_proxy_enabled'] = true + gitlab_rails['dependency_proxy_enabled'] = false ``` -1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. -1. Enable the [Puma web server](../operations/puma.md). +1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) + for the changes to take effect. -**Helm chart installations** +### Helm chart installations -1. After the installation is complete, update the global `appConfig` to enable the feature: +After the installation is complete, update the global `appConfig` to turn off the Dependency Proxy: - ```yaml - global: - appConfig: - dependencyProxy: - enabled: true - bucket: gitlab-dependency-proxy - connection: {} - secret: - key: - ``` +```yaml +global: + appConfig: + dependencyProxy: + enabled: false + bucket: gitlab-dependency-proxy + connection: {} + secret: + key: +``` For more information, see [Configure Charts using Globals](https://docs.gitlab.com/charts/charts/globals.html#configure-appconfig-settings). -**Installations from source** +### Installations from source -1. After the installation is complete, configure the `dependency_proxy` - section in `config/gitlab.yml`. Set to `true` to enable it: +1. After the installation is complete, configure the `dependency_proxy` section in + `config/gitlab.yml`. Set `enabled` to `false` to turn off the Dependency Proxy: ```yaml dependency_proxy: - enabled: true + enabled: false ``` -1. [Restart GitLab](../restart_gitlab.md#installations-from-source "How to restart GitLab") for the changes to take effect. +1. [Restart GitLab](../restart_gitlab.md#installations-from-source "How to restart GitLab") + for the changes to take effect. + +### Multi-node GitLab installations -Since Puma is already the default web server for installations from source as of GitLab 12.9, -no further changes are needed. +Follow the steps for [Omnibus GitLab installations](#omnibus-gitlab-installations) +for each Web and Sidekiq node. -**Multi-node GitLab installations** +## Turn on the Dependency Proxy -Follow the steps for **Omnibus GitLab installation** for each Web and Sidekiq nodes. +The Dependency Proxy is turned on by default, but can be turned off by an +administrator. To turn on the Dependency Proxy, follow the instructions in +[Turn off the Dependency Proxy](#turn-off-the-dependency-proxy), +but set the `enabled` fields to `true`. ## Changing the storage path -By default, the dependency proxy files are stored locally, but you can change the default +By default, the Dependency Proxy files are stored locally, but you can change the default local location or even use object storage. ### Changing the local storage path -The dependency proxy files for Omnibus GitLab installations are stored under +The Dependency Proxy files for Omnibus GitLab installations are stored under `/var/opt/gitlab/gitlab-rails/shared/dependency_proxy/` and for source installations under `shared/dependency_proxy/` (relative to the Git home directory). To change the local storage path: @@ -105,7 +121,7 @@ To change the local storage path: ### Using object storage Instead of relying on the local storage, you can use an object storage to -store the blobs of the dependency proxy. +store the blobs of the Dependency Proxy. [Read more about using object storage with GitLab](../object_storage.md). @@ -199,5 +215,3 @@ Feature.disable(:dependency_proxy_for_private_groups) # Re-enable the authentication Feature.enable(:dependency_proxy_for_private_groups) ``` - -The ability to disable this feature will be [removed in 13.9](https://gitlab.com/gitlab-org/gitlab/-/issues/276777). diff --git a/doc/administration/pages/index.md b/doc/administration/pages/index.md index f3ad474771c..53510688794 100644 --- a/doc/administration/pages/index.md +++ b/doc/administration/pages/index.md @@ -192,6 +192,33 @@ to use the HTTPS protocol. WARNING: Multiple wildcards for one instance is not supported. Only one wildcard per instance can be assigned. +### Wildcard domains with TLS-terminating Load Balancer + +**Requirements:** + +- [Wildcard DNS setup](#dns-configuration) +- [TLS-terminating load balancer](../../install/aws/manual_install_aws.md#load-balancer) + +--- + +URL scheme: `https://<namespace>.example.io/<project_slug>` + +This setup is primarily intended to be used when [installing a GitLab POC on Amazon Web Services](../../install/aws/manual_install_aws.md). This includes a TLS-terminating [classic load balancer](../../install/aws/manual_install_aws.md#load-balancer) that listens for HTTPS connections, manages TLS certificates, and forwards HTTP traffic to the instance. + +1. In `/etc/gitlab/gitlab.rb` specify the following configuration: + + ```ruby + external_url "https://gitlab.example.com" # external_url here is only for reference + pages_external_url "https://pages.example.com" # not a subdomain of external_url + + pages_nginx['enable'] = true + pages_nginx['listen_port'] = 80 + pages_nginx['listen_https'] = false + pages_nginx['redirect_http_to_https'] = true + ``` + +1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). + ### Global settings Below is a table of all configuration settings known to Pages in Omnibus GitLab, @@ -262,6 +289,8 @@ control over how the Pages daemon runs and serves content in your environment. | `use_legacy_storage` | Temporarily-introduced parameter allowing to use legacy domain configuration source and storage. [Removed in 14.3](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/6166). | | `rate_limit_source_ip` | Rate limit per source IP in number of requests per second. Set to `0` to disable this feature. | | `rate_limit_source_ip_burst` | Rate limit per source IP maximum burst allowed per second. | +| `rate_limit_domain` | Rate limit per domain in number of requests per second. Set to `0` to disable this feature. | +| `rate_limit_domain_burst` | Rate limit per domain maximum burst allowed per second. | ## Advanced configuration @@ -518,6 +547,14 @@ After an archive reaches `zip_cache_expiration`, it's marked as expired and remo  +### HTTP Strict Transport Security (HSTS) support + +HTTP Strict Transport Security (HSTS) can be enabled through the `gitlab_pages['headers']` configuration option. HSTS informs browsers that the website they are visiting should always provide its content over HTTPS to ensure that attackers cannot force subsequent connections to happen unencrypted. It can also improve loading speed of pages as it prevents browsers from attempting to connect over an unencrypted HTTP channel before being redirected to HTTPS. + +```ruby +gitlab_pages['headers'] = ['Strict-Transport-Security: max-age=63072000'] +``` + ## Activate verbose logging for daemon Follow the steps below to configure verbose logging of GitLab Pages daemon. @@ -695,6 +732,9 @@ database encryption. Proceed with caution. pages_external_url "http://<pages_server_URL>" gitlab_pages['gitlab_server'] = 'http://<gitlab_server_IP_or_URL>' + + ## If access control was enabled on step 3 + gitlab_pages['access_control'] = true ``` 1. If you have custom UID/GID settings on the **GitLab server**, add them to the **Pages server** `/etc/gitlab/gitlab.rb` as well, @@ -717,7 +757,7 @@ database encryption. Proceed with caution. mv /var/opt/gitlab/gitlab-rails/shared/pages/gitlab-secrets.json /etc/gitlab/gitlab-secrets.json ``` -1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. +1. [Reconfigure the **Pages server**](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. 1. On the **GitLab server**, make the following changes to `/etc/gitlab/gitlab.rb`: @@ -727,7 +767,7 @@ database encryption. Proceed with caution. pages_nginx['enable'] = false ``` -1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. +1. [Reconfigure the **GitLab server**](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. It's possible to run GitLab Pages on multiple servers if you wish to distribute the load. You can do this through standard load balancing practices such as @@ -925,7 +965,7 @@ However, some projects may fail to be migrated for different reasons. To verify that all projects have been migrated successfully, you can manually run the migration: ```shell -gitlab-rake gitlab:pages:migrate_legacy_storage +sudo gitlab-rake gitlab:pages:migrate_legacy_storage ``` It's safe to interrupt this task and run it multiple times. @@ -1039,15 +1079,22 @@ than GitLab to prevent XSS attacks. > [Introduced](https://gitlab.com/gitlab-org/gitlab-pages/-/issues/631) in GitLab 14.5. -You can enforce source-IP rate limits to help minimize the risk of a Denial of Service (DoS) attack. GitLab Pages +You can enforce rate limits to help minimize the risk of a Denial of Service (DoS) attack. GitLab Pages uses a [token bucket algorithm](https://en.wikipedia.org/wiki/Token_bucket) to enforce rate limiting. By default, requests that exceed the specified limits are reported but not rejected. -Source-IP rate limits are enforced using the following: +GitLab Pages supports the following types of rate limiting: + +- Per `source_ip`. It limits how many requests are allowed from the single client IP address. +- Per `domain`. It limits how many requests are allowed per domain hosted on GitLab Pages. It can be a custom domain like `example.com`, or group domain like `group.gitlab.io`. -- `rate_limit_source_ip`: Set the maximum threshold in number of requests per second. Set to 0 to disable this feature. -- `rate_limit_source_ip_burst`: Sets the maximum threshold of number of requests allowed in an initial outburst of requests. +Rate limits are enforced using the following: + +- `rate_limit_source_ip`: Set the maximum threshold in number of requests per client IP per second. Set to 0 to disable this feature. +- `rate_limit_source_ip_burst`: Sets the maximum threshold of number of requests allowed in an initial outburst of requests per client IP. For example, when you load a web page that loads a number of resources at the same time. +- `rate_limit_domain_ip`: Set the maximum threshold in number of requests per hosted pages domain per second. Set to 0 to disable this feature. +- `rate_limit_domain_burst`: Sets the maximum threshold of number of requests allowed in an initial outburst of requests per hosted pages domain. #### Enable source-IP rate limits @@ -1067,6 +1114,24 @@ Source-IP rate limits are enforced using the following: 1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). +#### Enable domain rate limits + +1. Set rate limits in `/etc/gitlab/gitlab.rb`: + + ```ruby + gitlab_pages['rate_limit_domain'] = 1000 + gitlab_pages['rate_limit_domain_burst'] = 5000 + ``` + +1. To reject requests that exceed the specified limits, enable the `FF_ENFORCE_DOMAIN_RATE_LIMITS` feature flag in + `/etc/gitlab/gitlab.rb`: + + ```ruby + gitlab_pages['env'] = {'FF_ENFORCE_DOMAIN_RATE_LIMITS' => 'true'} + ``` + +1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). + <!-- ## Troubleshooting Include any troubleshooting steps that you can foresee. If you know beforehand what issues @@ -1282,6 +1347,15 @@ This can happen if your `gitlab-secrets.json` file is out of date between GitLab Pages. Follow steps 8-10 of [Running GitLab Pages on a separate server](#running-gitlab-pages-on-a-separate-server), in all of your GitLab Pages instances. +### Intermittent 502 errors when using an AWS Network Load Balancer and GitLab Pages is running on multiple application servers + +Connections will time out when using a Network Load Balancer with client IP preservation enabled and [the request is looped back to the source server](https://docs.aws.amazon.com/elasticloadbalancing/latest/network/load-balancer-troubleshooting.html#loopback-timeout). +This can happen to GitLab instances with multiple servers +running both the core GitLab application and GitLab Pages. + +AWS [recommends using an IP target type](https://aws.amazon.com/premiumsupport/knowledge-center/target-connection-fails-load-balancer/) +to resolve this issue. + ### 500 error with `securecookie: failed to generate random iv` and `Failed to save the session` This problem most likely results from an [out-dated operating system](../package_information/supported_os.md#os-versions-that-are-no-longer-supported). @@ -1332,8 +1406,11 @@ Once added, reconfigure with `sudo gitlab-ctl reconfigure` and restart GitLab wi ### `The redirect URI included is not valid.` when using Pages Access Control -Verify that the **Callback URL**/Redirect URI in the GitLab Pages [System OAuth application](../../integration/oauth_provider.md#instance-wide-applications) +You may see this error if `pages_external_url` was updated at some point of time. Verify the following: + +1. The **Callback URL**/Redirect URI in the GitLab Pages [System OAuth application](../../integration/oauth_provider.md#instance-wide-applications) is using the protocol (HTTP or HTTPS) that `pages_external_url` is configured to use. +1. The domain and path components of `Redirect URI` are valid: they should look like `projects.<pages_external_url>/auth`. ### 500 error `cannot serve from disk` diff --git a/doc/administration/pages/source.md b/doc/administration/pages/source.md index 45e9dadd1cf..c4b1756d8a1 100644 --- a/doc/administration/pages/source.md +++ b/doc/administration/pages/source.md @@ -35,16 +35,17 @@ In the case of [custom domains](#custom-domains) (but not ports `80` and/or `443`. For that reason, there is some flexibility in the way which you can set it up: -1. Run the Pages daemon in the same server as GitLab, listening on a secondary IP. -1. Run the Pages daemon in a separate server. In that case, the - [Pages path](#change-storage-path) must also be present in the server that - the Pages daemon is installed, so you must share it through the network. -1. Run the Pages daemon in the same server as GitLab, listening on the same IP - but on different ports. In that case, you must proxy the traffic with - a load balancer. If you choose that route, you should use TCP load - balancing for HTTPS. If you use TLS-termination (HTTPS-load balancing), the - pages aren't able to be served with user-provided certificates. For - HTTP, it's OK to use HTTP or TCP load balancing. +- Run the Pages daemon in the same server as GitLab, listening on a secondary + IP. +- Run the Pages daemon in a separate server. In that case, the + [Pages path](#change-storage-path) must also be present in the server that + the Pages daemon is installed, so you must share it through the network. +- Run the Pages daemon in the same server as GitLab, listening on the same IP + but on different ports. In that case, you must proxy the traffic with a load + balancer. If you choose that route, you should use TCP load balancing for + HTTPS. If you use TLS-termination (HTTPS-load balancing), the pages aren't + able to be served with user-provided certificates. For HTTP, you can use HTTP + or TCP load balancing. In this document, we proceed assuming the first option. If you aren't supporting custom domains, a secondary IP isn't needed. @@ -53,16 +54,16 @@ supporting custom domains, a secondary IP isn't needed. Before proceeding with the Pages configuration, make sure that: -1. You have a separate domain to serve GitLab Pages from. In - this document we assume that to be `example.io`. -1. You have configured a **wildcard DNS record** for that domain. -1. You have installed the `zip` and `unzip` packages in the same server that - GitLab is installed since they are needed to compress and decompress the - Pages artifacts. -1. Optional. You have a **wildcard certificate** for the Pages domain if you - decide to serve Pages (`*.example.io`) under HTTPS. -1. Optional but recommended. You have configured and enabled the [shared runners](../../ci/runners/index.md) - so that your users don't have to bring their own. +- You have a separate domain to serve GitLab Pages from. In this document we + assume that to be `example.io`. +- You have configured a **wildcard DNS record** for that domain. +- You have installed the `zip` and `unzip` packages in the same server that + GitLab is installed since they are needed to compress and decompress the + Pages artifacts. +- Optional. You have a **wildcard certificate** for the Pages domain if you + decide to serve Pages (`*.example.io`) under HTTPS. +- Optional but recommended. You have configured and enabled the [shared runners](../../ci/runners/index.md) + so your users don't have to bring their own. ### DNS configuration @@ -417,8 +418,6 @@ server_name ~^.*\.pages\.example\.io$; ## Access control -> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/33422) in GitLab 11.5. - GitLab Pages access control can be configured per-project, and allows access to a Pages site to be controlled based on a user's membership to that project. diff --git a/doc/administration/pseudonymizer.md b/doc/administration/pseudonymizer.md index bd6982bea12..24d9792dcb0 100644 --- a/doc/administration/pseudonymizer.md +++ b/doc/administration/pseudonymizer.md @@ -4,7 +4,12 @@ group: Distribution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# Pseudonymizer **(ULTIMATE)** +# Pseudonymizer (DEPRECATED) **(ULTIMATE)** + +> [Deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/219952) in GitLab 14.7. + +WARNING: +This feature was [deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/219952) in GitLab 14.7. Your GitLab database contains sensitive information. To protect sensitive information when you run analytics on your database, you can use the Pseudonymizer service, which: diff --git a/doc/administration/raketasks/check.md b/doc/administration/raketasks/check.md index 1d60b8c6f50..fba151fefe1 100644 --- a/doc/administration/raketasks/check.md +++ b/doc/administration/raketasks/check.md @@ -7,6 +7,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Integrity check Rake task **(FREE SELF)** GitLab provides Rake tasks to check the integrity of various components. +See also the [check GitLab configuration Rake task](maintenance.md#check-gitlab-configuration). ## Repository integrity @@ -118,9 +119,9 @@ and these checks verify them against current files. Integrity checks are supported for the following types of file: -- CI artifacts (Available from version 10.7.0) -- LFS objects (Available from version 10.6.0) -- User uploads (Available from version 10.6.0) +- CI artifacts (introduced in GitLab 10.7.0) +- LFS objects (introduced in GitLab 10.6.0) +- User uploads (introduced in GitLab 10.6.0) **Omnibus Installation** @@ -200,6 +201,84 @@ The LDAP check Rake task tests the bind DN and password credentials executed as part of the `gitlab:check` task, but can run independently. See [LDAP Rake Tasks - LDAP Check](ldap.md#check) for details. +## Verify database values can be decrypted using the current secrets + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/20069) in GitLab 13.1. + +This task runs through all possible encrypted values in the +database, verifying that they are decryptable using the current +secrets file (`gitlab-secrets.json`). + +Automatic resolution is not yet implemented. If you have values that +cannot be decrypted, you can follow steps to reset them, see our +docs on what to do [when the secrets file is lost](../../raketasks/backup_restore.md#when-the-secrets-file-is-lost). + +This can take a very long time, depending on the size of your +database, as it checks all rows in all tables. + +**Omnibus Installation** + +```shell +sudo gitlab-rake gitlab:doctor:secrets +``` + +**Source Installation** + +```shell +bundle exec rake gitlab:doctor:secrets RAILS_ENV=production +``` + +**Example output** + +```plaintext +I, [2020-06-11T17:17:54.951815 #27148] INFO -- : Checking encrypted values in the database +I, [2020-06-11T17:18:12.677708 #27148] INFO -- : - ApplicationSetting failures: 0 +I, [2020-06-11T17:18:12.823692 #27148] INFO -- : - User failures: 0 +[...] other models possibly containing encrypted data +I, [2020-06-11T17:18:14.938335 #27148] INFO -- : - Group failures: 1 +I, [2020-06-11T17:18:15.559162 #27148] INFO -- : - Operations::FeatureFlagsClient failures: 0 +I, [2020-06-11T17:18:15.575533 #27148] INFO -- : - ScimOauthAccessToken failures: 0 +I, [2020-06-11T17:18:15.575678 #27148] INFO -- : Total: 1 row(s) affected +I, [2020-06-11T17:18:15.575711 #27148] INFO -- : Done! +``` + +### Verbose mode + +To get more detailed information about which rows and columns can't be +decrypted, you can pass a `VERBOSE` environment variable: + +**Omnibus Installation** + +```shell +sudo gitlab-rake gitlab:doctor:secrets VERBOSE=1 +``` + +**Source Installation** + +```shell +bundle exec rake gitlab:doctor:secrets RAILS_ENV=production VERBOSE=1 +``` + +**Example verbose output** + +<!-- vale gitlab.SentenceSpacing = NO --> + +```plaintext +I, [2020-06-11T17:17:54.951815 #27148] INFO -- : Checking encrypted values in the database +I, [2020-06-11T17:18:12.677708 #27148] INFO -- : - ApplicationSetting failures: 0 +I, [2020-06-11T17:18:12.823692 #27148] INFO -- : - User failures: 0 +[...] other models possibly containing encrypted data +D, [2020-06-11T17:19:53.224344 #27351] DEBUG -- : > Something went wrong for Group[10].runners_token: Validation failed: Route can't be blank +I, [2020-06-11T17:19:53.225178 #27351] INFO -- : - Group failures: 1 +D, [2020-06-11T17:19:53.225267 #27351] DEBUG -- : - Group[10]: runners_token +I, [2020-06-11T17:18:15.559162 #27148] INFO -- : - Operations::FeatureFlagsClient failures: 0 +I, [2020-06-11T17:18:15.575533 #27148] INFO -- : - ScimOauthAccessToken failures: 0 +I, [2020-06-11T17:18:15.575678 #27148] INFO -- : Total: 1 row(s) affected +I, [2020-06-11T17:18:15.575711 #27148] INFO -- : Done! +``` + +<!-- vale gitlab.SentenceSpacing = YES --> + ## Troubleshooting The following are solutions to problems you might discover using the Rake tasks documented diff --git a/doc/administration/raketasks/doctor.md b/doc/administration/raketasks/doctor.md index 02d1557b6a4..bed3cdcbcfe 100644 --- a/doc/administration/raketasks/doctor.md +++ b/doc/administration/raketasks/doctor.md @@ -1,88 +1,9 @@ --- -stage: Enablement -group: Geo -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +redirect_to: 'check.md#verify-database-values-can-be-decrypted-using-the-current-secrets' +remove_date: '2022-03-04' --- -# Doctor Rake tasks **(FREE SELF)** +This document was moved to [another location](check.md#verify-database-values-can-be-decrypted-using-the-current-secrets). -This is a collection of tasks to help investigate and repair -problems caused by data integrity issues. - -## Verify database values can be decrypted using the current secrets - -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/20069) in GitLab 13.1. - -This task runs through all possible encrypted values in the -database, verifying that they are decryptable using the current -secrets file (`gitlab-secrets.json`). - -Automatic resolution is not yet implemented. If you have values that -cannot be decrypted, you can follow steps to reset them, see our -docs on what to do [when the secrets file is lost](../../raketasks/backup_restore.md#when-the-secrets-file-is-lost). - -This can take a very long time, depending on the size of your -database, as it checks all rows in all tables. - -**Omnibus Installation** - -```shell -sudo gitlab-rake gitlab:doctor:secrets -``` - -**Source Installation** - -```shell -bundle exec rake gitlab:doctor:secrets RAILS_ENV=production -``` - -**Example output** - -```plaintext -I, [2020-06-11T17:17:54.951815 #27148] INFO -- : Checking encrypted values in the database -I, [2020-06-11T17:18:12.677708 #27148] INFO -- : - ApplicationSetting failures: 0 -I, [2020-06-11T17:18:12.823692 #27148] INFO -- : - User failures: 0 -[...] other models possibly containing encrypted data -I, [2020-06-11T17:18:14.938335 #27148] INFO -- : - Group failures: 1 -I, [2020-06-11T17:18:15.559162 #27148] INFO -- : - Operations::FeatureFlagsClient failures: 0 -I, [2020-06-11T17:18:15.575533 #27148] INFO -- : - ScimOauthAccessToken failures: 0 -I, [2020-06-11T17:18:15.575678 #27148] INFO -- : Total: 1 row(s) affected -I, [2020-06-11T17:18:15.575711 #27148] INFO -- : Done! -``` - -### Verbose mode - -To get more detailed information about which rows and columns can't be -decrypted, you can pass a `VERBOSE` environment variable: - -**Omnibus Installation** - -```shell -sudo gitlab-rake gitlab:doctor:secrets VERBOSE=1 -``` - -**Source Installation** - -```shell -bundle exec rake gitlab:doctor:secrets RAILS_ENV=production VERBOSE=1 -``` - -**Example verbose output** - -<!-- vale gitlab.SentenceSpacing = NO --> - -```plaintext -I, [2020-06-11T17:17:54.951815 #27148] INFO -- : Checking encrypted values in the database -I, [2020-06-11T17:18:12.677708 #27148] INFO -- : - ApplicationSetting failures: 0 -I, [2020-06-11T17:18:12.823692 #27148] INFO -- : - User failures: 0 -[...] other models possibly containing encrypted data -D, [2020-06-11T17:19:53.224344 #27351] DEBUG -- : > Something went wrong for Group[10].runners_token: Validation failed: Route can't be blank -I, [2020-06-11T17:19:53.225178 #27351] INFO -- : - Group failures: 1 -D, [2020-06-11T17:19:53.225267 #27351] DEBUG -- : - Group[10]: runners_token -I, [2020-06-11T17:18:15.559162 #27148] INFO -- : - Operations::FeatureFlagsClient failures: 0 -I, [2020-06-11T17:18:15.575533 #27148] INFO -- : - ScimOauthAccessToken failures: 0 -I, [2020-06-11T17:18:15.575678 #27148] INFO -- : Total: 1 row(s) affected -I, [2020-06-11T17:18:15.575711 #27148] INFO -- : Done! -``` - -<!-- vale gitlab.SentenceSpacing = YES --> +<!-- This redirect file can be deleted after 2022-03-04. --> +<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page --> diff --git a/doc/administration/raketasks/geo.md b/doc/administration/raketasks/geo.md index 3112e5f61b1..6306e54bdc6 100644 --- a/doc/administration/raketasks/geo.md +++ b/doc/administration/raketasks/geo.md @@ -7,6 +7,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Geo Rake Tasks **(PREMIUM SELF)** The following Rake tasks are for [Geo installations](../geo/index.md). +See also [troubleshooting Geo](../geo/replication/troubleshooting.md) for additional Geo Rake tasks. ## Git housekeeping diff --git a/doc/administration/raketasks/ldap.md b/doc/administration/raketasks/ldap.md index 62ddd9be2b3..e6adb98a92a 100644 --- a/doc/administration/raketasks/ldap.md +++ b/doc/administration/raketasks/ldap.md @@ -1,6 +1,6 @@ --- stage: Manage -group: Access +group: Authentication & Authorization info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/administration/raketasks/maintenance.md b/doc/administration/raketasks/maintenance.md index 950b508ab0c..d66f3b1ed35 100644 --- a/doc/administration/raketasks/maintenance.md +++ b/doc/administration/raketasks/maintenance.md @@ -106,17 +106,22 @@ The `gitlab:check` Rake task runs the following Rake tasks: - `gitlab:gitlab_shell:check` - `gitlab:gitaly:check` - `gitlab:sidekiq:check` +- `gitlab:incoming_email:check` +- `gitlab:ldap:check` - `gitlab:app:check` It checks that each component was set up according to the installation guide and suggest fixes for issues found. This command must be run from your application server and doesn't work correctly on component servers like [Gitaly](../gitaly/configure_gitaly.md#run-gitaly-on-its-own-server). +If you're running Geo, see also the [Geo Health check Rake task](../geo/replication/troubleshooting.md#health-check-rake-task). You may also have a look at our troubleshooting guides for: - [GitLab](../index.md#troubleshooting) - [Omnibus GitLab](https://docs.gitlab.com/omnibus/index.html#troubleshooting) +Additionally you should also [verify database values can be decrypted using the current secrets](check.md#verify-database-values-can-be-decrypted-using-the-current-secrets). + To run `gitlab:check`, run: **Omnibus Installation** diff --git a/doc/administration/redis/replication_and_failover.md b/doc/administration/redis/replication_and_failover.md index db652a80780..086057d80b4 100644 --- a/doc/administration/redis/replication_and_failover.md +++ b/doc/administration/redis/replication_and_failover.md @@ -648,6 +648,7 @@ persistence classes. | `actioncable` | Pub/Sub queue backend for ActionCable. | | `trace_chunks` | Store [CI trace chunks](../job_logs.md#enable-or-disable-incremental-logging) data. | | `rate_limiting` | Store [rate limiting](../../user/admin_area/settings/user_and_ip_rate_limits.md) state. | +| `sessions` | Store [sessions](../../../ee/development/session.md#gitlabsession). | To make this work with Sentinel: @@ -661,6 +662,7 @@ To make this work with Sentinel: gitlab_rails['redis_actioncable_instance'] = REDIS_ACTIONCABLE_URL gitlab_rails['redis_trace_chunks_instance'] = REDIS_TRACE_CHUNKS_URL gitlab_rails['redis_rate_limiting_instance'] = REDIS_RATE_LIMITING_URL + gitlab_rails['redis_sessions_instance'] = REDIS_SESSIONS_URL # Configure the Sentinels gitlab_rails['redis_cache_sentinels'] = [ @@ -687,6 +689,10 @@ To make this work with Sentinel: { host: RATE_LIMITING_SENTINEL_HOST, port: 26379 }, { host: RATE_LIMITING_SENTINEL_HOST2, port: 26379 } ] + gitlab_rails['redis_sessions_sentinels'] = [ + { host: SESSIONS_SENTINEL_HOST, port: 26379 }, + { host: SESSIONS_SENTINEL_HOST2, port: 26379 } + ] ``` - Redis URLs should be in the format: `redis://:PASSWORD@SENTINEL_PRIMARY_NAME`, where: diff --git a/doc/administration/reference_architectures/10k_users.md b/doc/administration/reference_architectures/10k_users.md index fa8dfdf667b..a5c60af47b1 100644 --- a/doc/administration/reference_architectures/10k_users.md +++ b/doc/administration/reference_architectures/10k_users.md @@ -1833,7 +1833,7 @@ On each node perform the following: external_url 'https://gitlab.example.com' # git_data_dirs get configured for the Praefect virtual storage - # Address is Interal Load Balancer for Praefect + # Address is Internal Load Balancer for Praefect # Token is praefect_external_token git_data_dirs({ "default" => { @@ -2228,17 +2228,16 @@ use Google Cloud's Kubernetes Engine (GKE) and associated machine types, but the and CPU requirements should translate to most other providers. We hope to update this in the future with further specific cloud provider details. -| Service | Nodes<sup>1</sup> | Configuration | GCP | Allocatable CPUs and Memory | -|-------------------------------------------------------|-------------------|-------------------------|------------------|-----------------------------| -| Webservice | 4 | 32 vCPU, 28.8 GB memory | `n1-highcpu-32` | 127.5 vCPU, 118 GB memory | -| Sidekiq | 4 | 4 vCPU, 15 GB memory | `n1-standard-4` | 15.5 vCPU, 50 GB memory | -| Supporting services such as NGINX, Prometheus | 2 | 4 vCPU, 15 GB memory | `n1-standard-4` | 7.75 vCPU, 25 GB memory | +| Service | Nodes | Configuration | GCP | Allocatable CPUs and Memory | +|-------------------------------------------------------|-------|-------------------------|------------------|-----------------------------| +| Webservice | 4 | 32 vCPU, 28.8 GB memory | `n1-highcpu-32` | 127.5 vCPU, 118 GB memory | +| Sidekiq | 4 | 4 vCPU, 15 GB memory | `n1-standard-4` | 15.5 vCPU, 50 GB memory | +| Supporting services such as NGINX, Prometheus | 2 | 4 vCPU, 15 GB memory | `n1-standard-4` | 7.75 vCPU, 25 GB memory | -<!-- Disable ordered list rule https://github.com/DavidAnson/markdownlint/blob/main/doc/Rules.md#md029---ordered-list-item-prefix --> -<!-- markdownlint-disable MD029 --> -1. Nodes configuration is shown as it is forced to ensure pod vcpu / memory ratios and avoid scaling during **performance testing**. - In production deployments there is no need to assign pods to nodes. A minimum of three nodes in three different availability zones is strongly recommended to align with resilient cloud architecture practices. -<!-- markdownlint-enable MD029 --> +- For this setup, we **recommend** and regularly [test](index.md#testing-process-and-results) +[Google Kubernetes Engine (GKE)](https://cloud.google.com/kubernetes-engine) and [Amazon Elastic Kubernetes Service (EKS)](https://aws.amazon.com/eks/). Other Kubernetes services may also work, but your mileage may vary. +- Nodes configuration is shown as it is forced to ensure pod vcpu / memory ratios and avoid scaling during **performance testing**. + - In production deployments, there is no need to assign pods to nodes. A minimum of three nodes in three different availability zones is strongly recommended to align with resilient cloud architecture practices. Next are the backend components that run on static compute VMs via Omnibus (or External PaaS services where applicable): diff --git a/doc/administration/reference_architectures/25k_users.md b/doc/administration/reference_architectures/25k_users.md index 24b3350bd75..8cc355db951 100644 --- a/doc/administration/reference_architectures/25k_users.md +++ b/doc/administration/reference_architectures/25k_users.md @@ -26,7 +26,7 @@ full list of reference architectures, see | PgBouncer<sup>1</sup> | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | | Internal load balancing node<sup>3</sup> | 1 | 4 vCPU, 3.6GB memory | `n1-highcpu-4` | `c5.large` | `F2s v2` | | Redis/Sentinel - Cache<sup>2</sup> | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | `D4s v3` | -| Redis/Sentinel - Persistent<sup>2</sup> | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.large` | `D4s v3` | +| Redis/Sentinel - Persistent<sup>2</sup> | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | `D4s v3` | | Gitaly<sup>5</sup> | 3 | 32 vCPU, 120 GB memory | `n1-standard-32` | `m5.8xlarge` | `D32s v3` | | Praefect<sup>5</sup> | 3 | 4 vCPU, 3.6 GB memory | `n1-highcpu-4` | `c5.xlarge` | `F4s v2` | | Praefect PostgreSQL<sup>1</sup> | 1+ | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | @@ -2226,17 +2226,16 @@ use Google Cloud's Kubernetes Engine (GKE) and associated machine types, but the and CPU requirements should translate to most other providers. We hope to update this in the future with further specific cloud provider details. -| Service | Nodes<sup>1</sup> | Configuration | GCP | Allocatable CPUs and Memory | -|-------------------------------------------------------|-------------------|-------------------------|------------------|-----------------------------| -| Webservice | 7 | 32 vCPU, 28.8 GB memory | `n1-highcpu-32` | 223 vCPU, 206.5 GB memory | -| Sidekiq | 4 | 4 vCPU, 15 GB memory | `n1-standard-4` | 15.5 vCPU, 50 GB memory | -| Supporting services such as NGINX, Prometheus | 2 | 4 vCPU, 15 GB memory | `n1-standard-4` | 7.75 vCPU, 25 GB memory | +| Service | Nodes | Configuration | GCP | Allocatable CPUs and Memory | +|-------------------------------------------------------|-------|-------------------------|------------------|-----------------------------| +| Webservice | 7 | 32 vCPU, 28.8 GB memory | `n1-highcpu-32` | 223 vCPU, 206.5 GB memory | +| Sidekiq | 4 | 4 vCPU, 15 GB memory | `n1-standard-4` | 15.5 vCPU, 50 GB memory | +| Supporting services such as NGINX, Prometheus | 2 | 4 vCPU, 15 GB memory | `n1-standard-4` | 7.75 vCPU, 25 GB memory | -<!-- Disable ordered list rule https://github.com/DavidAnson/markdownlint/blob/main/doc/Rules.md#md029---ordered-list-item-prefix --> -<!-- markdownlint-disable MD029 --> -1. Nodes configuration is shown as it is forced to ensure pod vcpu / memory ratios and avoid scaling during **performance testing**. - In production deployments there is no need to assign pods to nodes. A minimum of three nodes in three different availability zones is strongly recommended to align with resilient cloud architecture practices. -<!-- markdownlint-enable MD029 --> +- For this setup, we **recommend** and regularly [test](index.md#testing-process-and-results) +[Google Kubernetes Engine (GKE)](https://cloud.google.com/kubernetes-engine) and [Amazon Elastic Kubernetes Service (EKS)](https://aws.amazon.com/eks/). Other Kubernetes services may also work, but your mileage may vary. +- Nodes configuration is shown as it is forced to ensure pod vcpu / memory ratios and avoid scaling during **performance testing**. + - In production deployments, there is no need to assign pods to nodes. A minimum of three nodes in three different availability zones is strongly recommended to align with resilient cloud architecture practices. Next are the backend components that run on static compute VMs via Omnibus (or External PaaS services where applicable): diff --git a/doc/administration/reference_architectures/2k_users.md b/doc/administration/reference_architectures/2k_users.md index f72c0877ddb..467c64b8279 100644 --- a/doc/administration/reference_architectures/2k_users.md +++ b/doc/administration/reference_architectures/2k_users.md @@ -1016,17 +1016,16 @@ use Google Cloud's Kubernetes Engine (GKE) and associated machine types, but the and CPU requirements should translate to most other providers. We hope to update this in the future with further specific cloud provider details. -| Service | Nodes<sup>1</sup> | Configuration | GCP | Allocatable CPUs and Memory | -|-------------------------------------------------------|-------------------|------------------------|-----------------|-----------------------------| -| Webservice | 3 | 8 vCPU, 7.2 GB memory | `n1-highcpu-8` | 23.7 vCPU, 16.9 GB memory | -| Sidekiq | 2 | 2 vCPU, 7.5 GB memory | `n1-standard-2` | 3.9 vCPU, 11.8 GB memory | -| Supporting services such as NGINX, Prometheus | 2 | 1 vCPU, 3.75 GB memory | `n1-standard-1` | 1.9 vCPU, 5.5 GB memory | - -<!-- Disable ordered list rule https://github.com/DavidAnson/markdownlint/blob/main/doc/Rules.md#md029---ordered-list-item-prefix --> -<!-- markdownlint-disable MD029 --> -1. Nodes configuration is shown as it is forced to ensure pod vcpu / memory ratios and avoid scaling during **performance testing**. - In production deployments there is no need to assign pods to nodes. A minimum of three nodes in three different availability zones is strongly recommended to align with resilient cloud architecture practices. -<!-- markdownlint-enable MD029 --> +| Service | Nodes | Configuration | GCP | Allocatable CPUs and Memory | +|-------------------------------------------------------|-------|------------------------|-----------------|-----------------------------| +| Webservice | 3 | 8 vCPU, 7.2 GB memory | `n1-highcpu-8` | 23.7 vCPU, 16.9 GB memory | +| Sidekiq | 2 | 2 vCPU, 7.5 GB memory | `n1-standard-2` | 3.9 vCPU, 11.8 GB memory | +| Supporting services such as NGINX, Prometheus | 2 | 1 vCPU, 3.75 GB memory | `n1-standard-1` | 1.9 vCPU, 5.5 GB memory | + +- For this setup, we **recommend** and regularly [test](index.md#testing-process-and-results) +[Google Kubernetes Engine (GKE)](https://cloud.google.com/kubernetes-engine) and [Amazon Elastic Kubernetes Service (EKS)](https://aws.amazon.com/eks/). Other Kubernetes services may also work, but your mileage may vary. +- Nodes configuration is shown as it is forced to ensure pod vcpu / memory ratios and avoid scaling during **performance testing**. + - In production deployments, there is no need to assign pods to nodes. A minimum of three nodes in three different availability zones is strongly recommended to align with resilient cloud architecture practices. Next are the backend components that run on static compute VMs via Omnibus (or External PaaS services where applicable): diff --git a/doc/administration/reference_architectures/3k_users.md b/doc/administration/reference_architectures/3k_users.md index c788a73753b..01d9987739b 100644 --- a/doc/administration/reference_architectures/3k_users.md +++ b/doc/administration/reference_architectures/3k_users.md @@ -1794,7 +1794,7 @@ On each node perform the following: external_url 'https://gitlab.example.com' # git_data_dirs get configured for the Praefect virtual storage - # Address is Interal Load Balancer for Praefect + # Address is Internal Load Balancer for Praefect # Token is praefect_external_token git_data_dirs({ "default" => { @@ -2185,17 +2185,16 @@ use Google Cloud's Kubernetes Engine (GKE) and associated machine types, but the and CPU requirements should translate to most other providers. We hope to update this in the future with further specific cloud provider details. -| Service | Nodes<sup>1</sup> | Configuration | GCP | Allocatable CPUs and Memory | -|-------------------------------------------------------|-------------------|-------------------------|------------------|-----------------------------| -| Webservice | 2 | 16 vCPU, 14.4 GB memory | `n1-highcpu-16` | 31.8 vCPU, 24.8 GB memory | -| Sidekiq | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | 11.8 vCPU, 38.9 GB memory | -| Supporting services such as NGINX, Prometheus | 2 | 2 vCPU, 7.5 GB memory | `n1-standard-2` | 3.9 vCPU, 11.8 GB memory | +| Service | Nodes | Configuration | GCP | Allocatable CPUs and Memory | +|-------------------------------------------------------|-------|-------------------------|------------------|-----------------------------| +| Webservice | 2 | 16 vCPU, 14.4 GB memory | `n1-highcpu-16` | 31.8 vCPU, 24.8 GB memory | +| Sidekiq | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | 11.8 vCPU, 38.9 GB memory | +| Supporting services such as NGINX, Prometheus | 2 | 2 vCPU, 7.5 GB memory | `n1-standard-2` | 3.9 vCPU, 11.8 GB memory | -<!-- Disable ordered list rule https://github.com/DavidAnson/markdownlint/blob/main/doc/Rules.md#md029---ordered-list-item-prefix --> -<!-- markdownlint-disable MD029 --> -1. Nodes configuration is shown as it is forced to ensure pod vcpu / memory ratios and avoid scaling during **performance testing**. - In production deployments there is no need to assign pods to nodes. A minimum of three nodes in three different availability zones is strongly recommended to align with resilient cloud architecture practices. -<!-- markdownlint-enable MD029 --> +- For this setup, we **recommend** and regularly [test](index.md#testing-process-and-results) +[Google Kubernetes Engine (GKE)](https://cloud.google.com/kubernetes-engine) and [Amazon Elastic Kubernetes Service (EKS)](https://aws.amazon.com/eks/). Other Kubernetes services may also work, but your mileage may vary. +- Nodes configuration is shown as it is forced to ensure pod vcpu / memory ratios and avoid scaling during **performance testing**. + - In production deployments, there is no need to assign pods to nodes. A minimum of three nodes in three different availability zones is strongly recommended to align with resilient cloud architecture practices. Next are the backend components that run on static compute VMs via Omnibus (or External PaaS services where applicable): diff --git a/doc/administration/reference_architectures/50k_users.md b/doc/administration/reference_architectures/50k_users.md index 4f576fc1c19..d5bb9c4ad64 100644 --- a/doc/administration/reference_architectures/50k_users.md +++ b/doc/administration/reference_architectures/50k_users.md @@ -1855,7 +1855,7 @@ On each node perform the following: external_url 'https://gitlab.example.com' # git_data_dirs get configured for the Praefect virtual storage - # Address is Interal Load Balancer for Praefect + # Address is Internal Load Balancer for Praefect # Token is praefect_external_token git_data_dirs({ "default" => { @@ -2242,17 +2242,16 @@ use Google Cloud's Kubernetes Engine (GKE) and associated machine types, but the and CPU requirements should translate to most other providers. We hope to update this in the future with further specific cloud provider details. -| Service | Nodes<sup>1</sup> | Configuration | GCP | Allocatable CPUs and Memory | -|-------------------------------------------------------|-------------------|-------------------------|------------------|-----------------------------| -| Webservice | 16 | 32 vCPU, 28.8 GB memory | `n1-highcpu-32` | 510 vCPU, 472 GB memory | -| Sidekiq | 4 | 4 vCPU, 15 GB memory | `n1-standard-4` | 15.5 vCPU, 50 GB memory | -| Supporting services such as NGINX, Prometheus | 2 | 4 vCPU, 15 GB memory | `n1-standard-4` | 7.75 vCPU, 25 GB memory | +| Service | Nodes | Configuration | GCP | Allocatable CPUs and Memory | +|-------------------------------------------------------|-------|-------------------------|------------------|-----------------------------| +| Webservice | 16 | 32 vCPU, 28.8 GB memory | `n1-highcpu-32` | 510 vCPU, 472 GB memory | +| Sidekiq | 4 | 4 vCPU, 15 GB memory | `n1-standard-4` | 15.5 vCPU, 50 GB memory | +| Supporting services such as NGINX, Prometheus | 2 | 4 vCPU, 15 GB memory | `n1-standard-4` | 7.75 vCPU, 25 GB memory | -<!-- Disable ordered list rule https://github.com/DavidAnson/markdownlint/blob/main/doc/Rules.md#md029---ordered-list-item-prefix --> -<!-- markdownlint-disable MD029 --> -1. Nodes configuration is shown as it is forced to ensure pod vcpu / memory ratios and avoid scaling during **performance testing**. - In production deployments there is no need to assign pods to nodes. A minimum of three nodes in three different availability zones is strongly recommended to align with resilient cloud architecture practices. -<!-- markdownlint-enable MD029 --> +- For this setup, we **recommend** and regularly [test](index.md#testing-process-and-results) +[Google Kubernetes Engine (GKE)](https://cloud.google.com/kubernetes-engine) and [Amazon Elastic Kubernetes Service (EKS)](https://aws.amazon.com/eks/). Other Kubernetes services may also work, but your mileage may vary. +- Nodes configuration is shown as it is forced to ensure pod vcpu / memory ratios and avoid scaling during **performance testing**. + - In production deployments, there is no need to assign pods to nodes. A minimum of three nodes in three different availability zones is strongly recommended to align with resilient cloud architecture practices. Next are the backend components that run on static compute VMs via Omnibus (or External PaaS services where applicable): diff --git a/doc/administration/reference_architectures/5k_users.md b/doc/administration/reference_architectures/5k_users.md index 92950806cfb..33ca4e4899f 100644 --- a/doc/administration/reference_architectures/5k_users.md +++ b/doc/administration/reference_architectures/5k_users.md @@ -1791,7 +1791,7 @@ On each node perform the following: external_url 'https://gitlab.example.com' # git_data_dirs get configured for the Praefect virtual storage - # Address is Interal Load Balancer for Praefect + # Address is Internal Load Balancer for Praefect # Token is praefect_external_token git_data_dirs({ "default" => { @@ -2161,17 +2161,16 @@ use Google Cloud's Kubernetes Engine (GKE) and associated machine types, but the and CPU requirements should translate to most other providers. We hope to update this in the future with further specific cloud provider details. -| Service | Nodes<sup>1</sup> | Configuration | GCP | Allocatable CPUs and Memory | -|-------------------------------------------------------|-------------------|-------------------------|------------------|-----------------------------| -| Webservice | 5 | 16 vCPU, 14.4 GB memory | `n1-highcpu-16` | 79.5 vCPU, 62 GB memory | -| Sidekiq | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | 11.8 vCPU, 38.9 GB memory | -| Supporting services such as NGINX, Prometheus | 2 | 2 vCPU, 7.5 GB memory | `n1-standard-2` | 3.9 vCPU, 11.8 GB memory | +| Service | Nodes | Configuration | GCP | Allocatable CPUs and Memory | +|-------------------------------------------------------|-------|-------------------------|------------------|-----------------------------| +| Webservice | 5 | 16 vCPU, 14.4 GB memory | `n1-highcpu-16` | 79.5 vCPU, 62 GB memory | +| Sidekiq | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | 11.8 vCPU, 38.9 GB memory | +| Supporting services such as NGINX, Prometheus | 2 | 2 vCPU, 7.5 GB memory | `n1-standard-2` | 3.9 vCPU, 11.8 GB memory | -<!-- Disable ordered list rule https://github.com/DavidAnson/markdownlint/blob/main/doc/Rules.md#md029---ordered-list-item-prefix --> -<!-- markdownlint-disable MD029 --> -1. Nodes configuration is shown as it is forced to ensure pod vcpu / memory ratios and avoid scaling during **performance testing**. - In production deployments there is no need to assign pods to nodes. A minimum of three nodes in three different availability zones is strongly recommended to align with resilient cloud architecture practices. -<!-- markdownlint-enable MD029 --> +- For this setup, we **recommend** and regularly [test](index.md#testing-process-and-results) +[Google Kubernetes Engine (GKE)](https://cloud.google.com/kubernetes-engine) and [Amazon Elastic Kubernetes Service (EKS)](https://aws.amazon.com/eks/). Other Kubernetes services may also work, but your mileage may vary. +- Nodes configuration is shown as it is forced to ensure pod vcpu / memory ratios and avoid scaling during **performance testing**. + - In production deployments, there is no need to assign pods to nodes. A minimum of three nodes in three different availability zones is strongly recommended to align with resilient cloud architecture practices. Next are the backend components that run on static compute VMs via Omnibus (or External PaaS services where applicable): diff --git a/doc/administration/reference_architectures/index.md b/doc/administration/reference_architectures/index.md index 6bf35ba6e22..bd796600564 100644 --- a/doc/administration/reference_architectures/index.md +++ b/doc/administration/reference_architectures/index.md @@ -208,19 +208,16 @@ Note the following about the testing process: - We aim to have a "test smart" approach where architectures tested have a good range that can also apply to others. Testing focuses on 10k Omnibus on GCP as the testing has shown this is a good bellwether for the other architectures and cloud providers as well as Cloud Native Hybrids. - Testing is done publicly and all results are shared. -Τhe following table details the testing done against the reference architectures along with the frequency and results. - -| Reference Architecture | Tests Run<sup>1</sup> | -|------------------------|----------------------------------------------------------------------------------------------------------------------| -| 1k | [Omnibus - Daily (GCP)](https://gitlab.com/gitlab-org/quality/performance/-/wikis/Benchmarks/Latest/1k)<sup>2</sup> | -| 2k | [Omnibus - Daily (GCP)](https://gitlab.com/gitlab-org/quality/performance/-/wikis/Benchmarks/Latest/2k)<sup>2</sup> | -| 3k | [Omnibus - Weekly (GCP)](https://gitlab.com/gitlab-org/quality/performance/-/wikis/Benchmarks/Latest/3k)<sup>2</sup> | -| 5k | [Omnibus - Weekly (GCP)](https://gitlab.com/gitlab-org/quality/performance/-/wikis/Benchmarks/Latest/5k)<sup>2</sup> | -| 10k | [Omnibus - Daily (GCP)](https://gitlab.com/gitlab-org/quality/performance/-/wikis/Benchmarks/Latest/10k)<sup>2</sup><br/>[Omnibus - Ad-Hoc (GCP, AWS, Azure)](https://gitlab.com/gitlab-org/quality/performance/-/wikis/Past-Results/10k)<br/><br/>[Cloud Native Hybrid - Ad-Hoc (GCP, AWS)](https://gitlab.com/gitlab-org/quality/performance/-/wikis/Past-Results/10k-Cloud-Native-Hybrid) | -| 25k | [Omnibus - Weekly (GCP)](https://gitlab.com/gitlab-org/quality/performance/-/wikis/Benchmarks/Latest/25k)<sup>2</sup><br/>[Omnibus - Ad-Hoc (Azure)](https://gitlab.com/gitlab-org/quality/performance/-/wikis/Past-Results/25k) | -| 50k | [Omnibus - Weekly (GCP)](https://gitlab.com/gitlab-org/quality/performance/-/wikis/Benchmarks/Latest/50k)<sup>2</sup><br/>[Omnibus - Ad-Hoc (AWS)](https://gitlab.com/gitlab-org/quality/performance/-/wikis/Past-Results/50k) | - -Note that: - -1. The list above is non exhaustive. Additional testing is continuously evaluated and iterated on, and the table is updated regularly. -1. The Omnibus reference architectures are VM-based only and testing has shown that they perform similarly on equivalently specced hardware regardless of Cloud Provider or if run on premises. +Τhe following table details the testing done against the reference architectures along with the frequency and results. Note that this list above is non exhaustive. Additional testing is continuously evaluated and iterated on, and the table is updated accordingly. + +| Reference<br/>Architecture<br/>Size | Bare-Metal | GCP | AWS | Azure | +|-----------------------------|------------|-----|-----|-------| +| 1k | <i>Refer to GCP<sup>1</sup></i> | [Standard - Weekly](https://gitlab.com/gitlab-org/quality/performance/-/wikis/Benchmarks/Latest/1k)<sup>1</sup> | - | - | +| 2k | <i>Refer to GCP<sup>1</sup></i> | [Standard - Weekly](https://gitlab.com/gitlab-org/quality/performance/-/wikis/Benchmarks/Latest/2k)<sup>1</sup> | - | - | +| 3k | <i>Refer to GCP<sup>1</sup></i> | [Standard - Weekly](https://gitlab.com/gitlab-org/quality/performance/-/wikis/Benchmarks/Latest/3k)<sup>1</sup> | - | - | +| 5k | <i>Refer to GCP<sup>1</sup></i> | [Standard - Weekly](https://gitlab.com/gitlab-org/quality/performance/-/wikis/Benchmarks/Latest/5k)<sup>1</sup> | - | - | +| 10k | <i>Refer to GCP<sup>1</sup></i> | [Standard - Daily](https://gitlab.com/gitlab-org/quality/performance/-/wikis/Benchmarks/Latest/10k)<sup>1</sup> <br/> [Standard (inc Cloud Services) - Ad-Hoc](https://gitlab.com/gitlab-org/quality/performance/-/wikis/Past-Results/10k) <br/> [Cloud Native Hybrid - Ad-Hoc](https://gitlab.com/gitlab-org/quality/performance/-/wikis/Past-Results/10k-Cloud-Native-Hybrid) | [Standard (inc Cloud Services) - Ad-Hoc](https://gitlab.com/gitlab-org/quality/performance/-/wikis/Past-Results/10k) <br/> [Cloud Native Hybrid - Ad-Hoc](https://gitlab.com/gitlab-org/quality/performance/-/wikis/Past-Results/10k-Cloud-Native-Hybrid) | [Standard - Ad-Hoc](https://gitlab.com/gitlab-org/quality/performance/-/wikis/Past-Results/10k) | +| 25k | <i>Refer to GCP<sup>1</sup></i> | [Standard - Weekly](https://gitlab.com/gitlab-org/quality/performance/-/wikis/Benchmarks/Latest/25k)<sup>1</sup> | - | [Standard - Ad-Hoc](https://gitlab.com/gitlab-org/quality/performance/-/wikis/Past-Results/25k) | +| 50k | <i>Refer to GCP<sup>1</sup></i> | [Standard - Weekly](https://gitlab.com/gitlab-org/quality/performance/-/wikis/Benchmarks/Latest/50k)<sup>1</sup> | [Standard (inc Cloud Services) - Ad-Hoc](https://gitlab.com/gitlab-org/quality/performance/-/wikis/Past-Results/50k) | - | + +1. The Standard Reference Architectures are designed to be platform agnostic, with everything being run on VMs via [Omnibus GitLab](https://docs.gitlab.com/omnibus/). While testing occurs primarily on GCP, ad-hoc testing has shown that they perform similarly on equivalently specced hardware on other Cloud Providers or if run on premises (bare-metal). diff --git a/doc/administration/sidekiq.md b/doc/administration/sidekiq.md index 8f1119f6868..989a024d6ab 100644 --- a/doc/administration/sidekiq.md +++ b/doc/administration/sidekiq.md @@ -2,7 +2,6 @@ stage: Enablement group: Distribution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments -type: reference --- # Configuring Sidekiq **(FREE SELF)** @@ -21,8 +20,6 @@ you want using steps 1 and 2 from the GitLab downloads page. 1. Generate the Sidekiq configuration: ```ruby - sidekiq['listen_address'] = "10.10.1.48" - ## Optional: Enable extra Sidekiq processes sidekiq_cluster['enable'] = true sidekiq['queue_groups'] = [ @@ -128,6 +125,34 @@ you want using steps 1 and 2 from the GitLab downloads page. external_url 'https://gitlab.example.com' ``` +1. (Optional) If you want to collect Sidekiq metrics, enable the Sidekiq metrics server. + To make metrics available from `localhost:8082/metrics`, set the following values: + + ```ruby + sidekiq['metrics_enabled'] = true + sidekiq['listen_address'] = "localhost" + sidekiq['listen_port'] = "8082" + + # Optionally log all the metrics server logs to log/sidekiq_exporter.log + sidekiq['exporter_log_enabled'] = true + ``` + +1. (Optional) If you use health check probes to observe Sidekiq, + set a separate port for health checks. + Configuring health checks is only necessary if there is something that actually probes them. + For more information about health checks, see the [Sidekiq health check page](sidekiq_health_check.md). + Enable health checks for Sidekiq: + + ```ruby + sidekiq['health_checks_enabled'] = true + sidekiq['health_checks_listen_address'] = "localhost" + sidekiq['health_checks_listen_port'] = "8092" + ``` + + NOTE: + If health check settings are not set, they will default to the metrics exporter settings. + This default is deprecated and is set to be removed in [GitLab 15.0](https://gitlab.com/gitlab-org/gitlab/-/issues/347509). + 1. Run `gitlab-ctl reconfigure`. You will need to restart the Sidekiq nodes after an update has occurred and database @@ -196,7 +221,13 @@ gitlab_rails['auto_migrate'] = false ####################################### ### Sidekiq configuration ### ####################################### -sidekiq['listen_address'] = "10.10.1.48" +sidekiq['metrics_enabled'] = true +sidekiq['exporter_log_enabled'] = false +sidekiq['listen_port'] = "8082" + +sidekiq['health_checks_enabled'] = true +sidekiq['health_checks_listen_address'] = "localhost" +sidekiq['health_checks_listen_port'] = "8092" ####################################### ### Monitoring configuration ### @@ -230,3 +261,4 @@ Related Sidekiq configuration: 1. [Extra Sidekiq processes](operations/extra_sidekiq_processes.md) 1. [Extra Sidekiq routing](operations/extra_sidekiq_routing.md) 1. [Using the GitLab-Sidekiq chart](https://docs.gitlab.com/charts/charts/gitlab/sidekiq/) +1. [Sidekiq health checks](sidekiq_health_check.md) diff --git a/doc/administration/sidekiq_health_check.md b/doc/administration/sidekiq_health_check.md new file mode 100644 index 00000000000..2ed736bac2c --- /dev/null +++ b/doc/administration/sidekiq_health_check.md @@ -0,0 +1,60 @@ +--- +stage: Enablement +group: Distribution +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Sidekiq Health Check **(FREE SELF)** + +GitLab provides liveness and readiness probes to indicate service health and +reachability to the Sidekiq cluster. These endpoints +[can be provided to schedulers like Kubernetes](https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/) +to hold traffic until the system is ready or restart the container as needed. + +The health check server can be set up when [configuring Sidekiq](sidekiq.md). + +## Readiness + +The readiness probe checks whether the Sidekiq workers are ready to process jobs. + +```plaintext +GET /readiness +``` + +Assuming you set up Sidekiq's address and port to be `localhost` and `8092` respectively, +here's an example request: + +```shell +curl "http://localhost:8092/readiness" +``` + +On success, the endpoint returns a `200` HTTP status code, and a response like the following: + +```json +{ + "status": "ok" +} +``` + +## Liveness + +Checks whether the Sidekiq cluster is running. + +```plaintext +GET /liveness +``` + +Assuming you set up Sidekiq's address and port to be `localhost` and `8092` respectively, +here's an example request: + +```shell +curl "http://localhost:8092/liveness" +``` + +On success, the endpoint returns a `200` HTTP status code, and a response like the following: + +```json +{ + "status": "ok" +} +``` diff --git a/doc/administration/terraform_state.md b/doc/administration/terraform_state.md index 582ffc9dc9c..d1113125c4e 100644 --- a/doc/administration/terraform_state.md +++ b/doc/administration/terraform_state.md @@ -101,6 +101,11 @@ The following settings are: > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/247042) in GitLab 13.9. +WARNING: +It's not possible to migrate Terraform state files from object storage back to local storage, +so proceed with caution. [An issue exists](https://gitlab.com/gitlab-org/gitlab/-/issues/350187) +to change this behavior. + To migrate Terraform state files to object storage, follow the instructions below. - For Omnibus package installations: diff --git a/doc/administration/troubleshooting/defcon.md b/doc/administration/troubleshooting/defcon.md index 1b263f70b46..a7cc47f8547 100644 --- a/doc/administration/troubleshooting/defcon.md +++ b/doc/administration/troubleshooting/defcon.md @@ -23,7 +23,7 @@ Side effects: ## `ci_queueing_disaster_recovery_disable_quota` -This feature flag, if temporarily enabled, disables enforcing CI minutes quota +This feature flag, if temporarily enabled, disables enforcing CI/CD minutes quota on shared runners. This can help to reduce system resource usage on the `jobs/request` endpoint by significantly reducing the computations being performed. diff --git a/doc/administration/troubleshooting/gitlab_rails_cheat_sheet.md b/doc/administration/troubleshooting/gitlab_rails_cheat_sheet.md index ccfa93d9bc8..33a81c12811 100644 --- a/doc/administration/troubleshooting/gitlab_rails_cheat_sheet.md +++ b/doc/administration/troubleshooting/gitlab_rails_cheat_sheet.md @@ -2,7 +2,6 @@ stage: Enablement group: Distribution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments -type: reference --- # GitLab Rails Console Cheat Sheet **(FREE SELF)** @@ -526,7 +525,7 @@ master f05321a5b5728bd8a89b7bf530aa44043c951dce...7d02e575fd790e76 ### Find mirrors with "bad decrypt" errors -This content has been converted to a Rake task, see the [Doctor Rake tasks docs](../raketasks/doctor.md). +This content has been converted to a Rake task, see [verify database values can be decrypted using the current secrets](../raketasks/check.md#verify-database-values-can-be-decrypted-using-the-current-secrets). ### Transfer mirror users and tokens to a single service account @@ -909,6 +908,14 @@ end Gitlab::CurrentSettings.current_application_settings.runners_registration_token ``` +### Seed runners registration token + +```ruby +appSetting = Gitlab::CurrentSettings.current_application_settings +appSetting.set_runners_registration_token('<new-runners-registration-token>') +appSetting.save! +``` + ### Run pipeline schedules manually You can run pipeline schedules manually through the Rails console to reveal any errors that are usually not visible. @@ -979,7 +986,7 @@ This is needed for example in a known edge-case with ### Remove licenses -To clean up the [License History table](../../user/admin_area/license.md#license-history): +To clean up the [License History table](../../user/admin_area/license.md#view-license-details-and-history): ```ruby TYPE = :trial? @@ -1065,7 +1072,7 @@ area on disk. It remains to be seen exactly how or whether the deletion is usefu ### Bad Decrypt Script (for encrypted variables) -This content has been converted to a Rake task, see the [Doctor Rake tasks docs](../raketasks/doctor.md). +This content has been converted to a Rake task, see [verify database values can be decrypted using the current secrets](../raketasks/check.md#verify-database-values-can-be-decrypted-using-the-current-secrets). As an example of repairing, if `ProjectImportData Bad count:` is detected and the decision is made to delete the encrypted credentials to allow manual reentry: @@ -1108,7 +1115,7 @@ gitlab-rails runner /tmp/encrypted-tokens.rb ### Decrypt Script for encrypted tokens -This content has been converted to a Rake task, see the [Doctor Rake tasks docs](../raketasks/doctor.md). +This content has been converted to a Rake task, see [verify database values can be decrypted using the current secrets](../raketasks/check.md#verify-database-values-can-be-decrypted-using-the-current-secrets). ## Geo @@ -1262,10 +1269,20 @@ registry = Geo::SnippetRepositoryRegistry.find(registry_id) registry.replicator.send(:sync_repository) ``` +## Gitaly + +### Find available and used space + +A Gitaly storage resource can be polled through Rails to determine the available and used space. + +```ruby +Gitlab::GitalyClient::ServerService.new("default").storage_disk_statistics +``` + ## Generate Service Ping -The [Service Ping Guide](../../development/service_ping/index.md) in our developer documentation -has more information about Service Ping. +The [Service Ping Guide](../../development/service_ping/index.md) in our developer documentation +has more information about Service Ping. ### Generate or get the cached Service Ping @@ -1291,7 +1308,7 @@ rake gitlab:usage_data:generate Generates Service Ping data in YAML format: -```shell +```shell rake gitlab:usage_data:dump_sql_in_yaml ``` diff --git a/doc/administration/troubleshooting/group_saml_scim.md b/doc/administration/troubleshooting/group_saml_scim.md index d052688363c..8ec6b35ec39 100644 --- a/doc/administration/troubleshooting/group_saml_scim.md +++ b/doc/administration/troubleshooting/group_saml_scim.md @@ -1,6 +1,6 @@ --- stage: Manage -group: Access +group: Authentication & Authorization info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments type: reference --- diff --git a/doc/administration/troubleshooting/img/okta_setting_username.png b/doc/administration/troubleshooting/img/okta_setting_username.png Binary files differindex c413b9d3a27..5f87d14bb8b 100644 --- a/doc/administration/troubleshooting/img/okta_setting_username.png +++ b/doc/administration/troubleshooting/img/okta_setting_username.png diff --git a/doc/administration/troubleshooting/kubernetes_cheat_sheet.md b/doc/administration/troubleshooting/kubernetes_cheat_sheet.md index 64a6979c016..290d6d9f21d 100644 --- a/doc/administration/troubleshooting/kubernetes_cheat_sheet.md +++ b/doc/administration/troubleshooting/kubernetes_cheat_sheet.md @@ -118,15 +118,18 @@ and they will assist you with any issues you are having. kubectl get events -w --namespace=gitlab ``` -- Most of the useful GitLab tools (console, Rake tasks, etc) are found in the task-runner - pod. You may enter it and run commands inside or run them from the outside: +- Most of the useful GitLab tools (console, Rake tasks, etc) are found in the toolbox + pod. You may enter it and run commands inside or run them from the outside. + + NOTE: + The `task-runner` pod was renamed to `toolbox` in GitLab 14.2 (charts 5.2). ```shell # find the pod - kubectl get pods | grep task-runner + kubectl --namespace gitlab get pods -lapp=toolbox # enter it - kubectl exec -it <task-runner-pod-name> -- bash + kubectl exec -it <toolbox-pod-name> -- bash # open rails console # rails console can be also called from other GitLab pods @@ -139,10 +142,10 @@ and they will assist you with any issues you are having. /usr/local/bin/gitlab-rake gitlab:check # open console without entering pod - kubectl exec -it <task-runner-pod-name> -- /srv/gitlab/bin/rails console + kubectl exec -it <toolbox-pod-name> -- /srv/gitlab/bin/rails console # check the status of DB migrations - kubectl exec -it <task-runner-pod-name> -- /usr/local/bin/gitlab-rake db:migrate:status + kubectl exec -it <toolbox-pod-name> -- /usr/local/bin/gitlab-rake db:migrate:status ``` You can also use `gitlab-rake`, instead of `/usr/local/bin/gitlab-rake`. @@ -163,12 +166,15 @@ and they will assist you with any issues you are having. kubectl get secret <secret-name> -ojsonpath={.data.password} | base64 --decode ; echo ``` -- How to connect to a GitLab PostgreSQL database: +- How to connect to a GitLab PostgreSQL database. + + NOTE: + The `task-runner` pod was renamed to `toolbox` in GitLab 14.2 (charts 5.2). In GitLab 14.2 (chart 5.2) and later: ```shell - kubectl exec -it <task-runner-pod-name> -- /srv/gitlab/bin/rails dbconsole --include-password --database main + kubectl exec -it <toolbox-pod-name> -- /srv/gitlab/bin/rails dbconsole --include-password --database main ``` In GitLab 14.1 (chart 5.1) and earlier: @@ -215,9 +221,9 @@ all Kubernetes resources and dependent charts: helm get manifest <release name> ``` -## Installation of minimal GitLab configuration via Minikube on macOS +## Installation of minimal GitLab configuration via minikube on macOS -This section is based on [Developing for Kubernetes with Minikube](https://docs.gitlab.com/charts/development/minikube/index.html) +This section is based on [Developing for Kubernetes with minikube](https://docs.gitlab.com/charts/development/minikube/index.html) and [Helm](https://docs.gitlab.com/charts/installation/tools.html#helm). Refer to those documents for details. @@ -227,13 +233,13 @@ to those documents for details. brew install kubernetes-cli ``` -- Install Minikube via Homebrew: +- Install minikube via Homebrew: ```shell brew cask install minikube ``` -- Start Minikube and configure it. If Minikube cannot start, try running `minikube delete && minikube start` +- Start minikube and configure it. If minikube cannot start, try running `minikube delete && minikube start` and repeat the steps: ```shell @@ -247,7 +253,7 @@ to those documents for details. brew install helm ``` -- Copy the [Minikube minimum values YAML file](https://gitlab.com/gitlab-org/charts/gitlab/raw/master/examples/values-minikube-minimum.yaml) +- Copy the [minikube minimum values YAML file](https://gitlab.com/gitlab-org/charts/gitlab/raw/master/examples/values-minikube-minimum.yaml) to your workstation: ```shell diff --git a/doc/administration/troubleshooting/postgresql.md b/doc/administration/troubleshooting/postgresql.md index f8cb45dd00d..e4d1696ea93 100644 --- a/doc/administration/troubleshooting/postgresql.md +++ b/doc/administration/troubleshooting/postgresql.md @@ -179,3 +179,43 @@ Once saved, [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure NOTE: These are Omnibus GitLab settings. If an external database, such as a customer's PostgreSQL installation or Amazon RDS is being used, these values don't get set, and would have to be set externally. + +### Temporarily changing the statement timeout + +WARNING: +The following advice does not apply in case +[PgBouncer](../postgresql/pgbouncer.md) is enabled, +because the changed timeout might affect more transactions than intended. + +In some situations, it may be desirable to set a different statement timeout +without having to [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure), +which in this case would restart Puma and Sidekiq. + +For example, a backup may fail with the following errors in the output of the +[backup command](../../raketasks/backup_restore.md#back-up-gitlab) +because the statement timeout was too short: + +```plaintext +pg_dump: error: Error message from server: server closed the connection unexpectedly +``` + +You may also see errors in the [PostgreSQL logs](../logs.md#postgresql-logs): + +```plaintext +canceling statement due to statement timeout +``` + +To temporarily change the statement timeout: + +1. Open `/var/opt/gitlab/gitlab-rails/etc/database.yml` in an editor +1. Set the value of `statement_timeout` to `0`, which sets an unlimited statement timeout. +1. [Confirm in a new Rails console session](../operations/rails_console.md#using-the-rails-runner) + that this value is used: + + ```shell + sudo gitlab-rails runner "ActiveRecord::Base.connection_config[:variables]" + ``` + +1. Perform the action for which you need a different timeout + (for example the backup or the Rails command). +1. Revert the edit in `/var/opt/gitlab/gitlab-rails/etc/database.yml`. diff --git a/doc/administration/troubleshooting/sidekiq.md b/doc/administration/troubleshooting/sidekiq.md index a606a3712ba..62ea3bcfa3c 100644 --- a/doc/administration/troubleshooting/sidekiq.md +++ b/doc/administration/troubleshooting/sidekiq.md @@ -387,7 +387,7 @@ blocking all jobs on that worker from proceeding. If Rugged calls performed by S background task processing. By default, Rugged is used when Git repository data is stored on local storage or on an NFS mount. -[Using Rugged is recommened when using NFS](../nfs.md#improving-nfs-performance-with-gitlab), but if +[Using Rugged is recommended when using NFS](../nfs.md#improving-nfs-performance-with-gitlab), but if you are using local storage, disabling Rugged can improve Sidekiq performance: ```shell diff --git a/doc/administration/troubleshooting/ssl.md b/doc/administration/troubleshooting/ssl.md index 01a4c4af65b..83df9ba19ff 100644 --- a/doc/administration/troubleshooting/ssl.md +++ b/doc/administration/troubleshooting/ssl.md @@ -18,7 +18,7 @@ main SSL docs available here: ## Using an internal CA certificate with GitLab After configuring a GitLab instance with an internal CA certificate, you might -not be able to access it by using various CLI tools. You may see experience the +not be able to access it by using various CLI tools. You may experience the following issues: - `curl` fails: diff --git a/doc/administration/user_settings.md b/doc/administration/user_settings.md index 891c11afaf4..2c1bb781882 100644 --- a/doc/administration/user_settings.md +++ b/doc/administration/user_settings.md @@ -1,6 +1,6 @@ --- stage: Manage -group: Access +group: Authentication & Authorization info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- |
