diff options
| author | GitLab Bot <gitlab-bot@gitlab.com> | 2022-07-20 15:40:28 +0000 |
|---|---|---|
| committer | GitLab Bot <gitlab-bot@gitlab.com> | 2022-07-20 15:40:28 +0000 |
| commit | b595cb0c1dec83de5bdee18284abe86614bed33b (patch) | |
| tree | 8c3d4540f193c5ff98019352f554e921b3a41a72 /doc/administration | |
| parent | 2f9104a328fc8a4bddeaa4627b595166d24671d0 (diff) | |
| download | gitlab-ce-b595cb0c1dec83de5bdee18284abe86614bed33b.tar.gz | |
Add latest changes from gitlab-org/gitlab@15-2-stable-eev15.2.0-rc42
Diffstat (limited to 'doc/administration')
107 files changed, 1901 insertions, 1734 deletions
diff --git a/doc/administration/audit_event_streaming.md b/doc/administration/audit_event_streaming.md index ad235ead992..817f22debbc 100644 --- a/doc/administration/audit_event_streaming.md +++ b/doc/administration/audit_event_streaming.md @@ -9,6 +9,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/332747) in GitLab 14.5 [with a flag](../administration/feature_flags.md) named `ff_external_audit_events_namespace`. Disabled by default. > - [Enabled on GitLab.com and by default on self-managed](https://gitlab.com/gitlab-org/gitlab/-/issues/338939) in GitLab 14.7. > - [Feature flag `ff_external_audit_events_namespace`](https://gitlab.com/gitlab-org/gitlab/-/issues/349588) removed in GitLab 14.8. +> - [Subgroup events recording](https://gitlab.com/gitlab-org/gitlab/-/issues/366878) fixed in GitLab 15.2. Users can set an HTTP endpoint for a top-level group to receive all audit events about the group, its subgroups, and projects as structured JSON. @@ -29,11 +30,11 @@ Event streaming destinations receive **all** audit event data, which could inclu Users with at least the Owner role for a group can add event streaming destinations for it: 1. On the top bar, select **Menu > Groups** and find your group. -1. On the left sidebar, select **Security & Compliance > Audit events** +1. On the left sidebar, select **Security & Compliance > Audit events**. 1. On the main area, select **Streams** tab. - - When the destination list is empty, select **Add stream** activate edit mode and add a new destination. - - When the destination list is not empty, select **{plus}** under the **Streams** tab to activate edit mode. -1. Enter the endpoint you wish to add and select **Add**. + - When the destination list is empty, select **Add stream** to show the section for adding destinations. + - When the destination list is not empty, select **{plus}** to show the section for adding destinations. +1. Enter the destination URL to add and select **Add**. Event streaming is enabled if: @@ -76,7 +77,7 @@ Users with at least the Owner role for a group can list event streaming destinat Users with at least the Owner role for a group can list event streaming destinations: 1. On the top bar, select **Menu > Groups** and find your group. -1. On the left sidebar, select **Security & Compliance > Audit events** +1. On the left sidebar, select **Security & Compliance > Audit events**. 1. On the main area, select **Streams** tab. ### Use the API @@ -115,7 +116,7 @@ When the last destination is successfully deleted, event streaming is disabled f Users with at least the Owner role for a group can delete event streaming destinations. 1. On the top bar, select **Menu > Groups** and find your group. -1. On the left sidebar, select **Security & Compliance > Audit events** +1. On the left sidebar, select **Security & Compliance > Audit events**. 1. On the main area, select **Streams** tab. 1. Select **{remove}** at the right side of each item. @@ -143,20 +144,26 @@ Destination is deleted if: - The returned `errors` object is empty. - The API responds with `200 OK`. -## Custom HTTP header values +## Custom HTTP headers -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/361216) in GitLab 15.1 [with a flag](feature_flags.md) named `streaming_audit_event_headers`. Disabled by default. +> - API [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/361216) in GitLab 15.1 [with a flag](feature_flags.md) named `streaming_audit_event_headers`. Disabled by default. +> - API [enabled on GitLab.com and self-managed](https://gitlab.com/gitlab-org/gitlab/-/issues/362941) in GitLab 15.2. +> - UI [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/361630) in GitLab 15.2 [with a flag](feature_flags.md) named `custom_headers_streaming_audit_events_ui`. Disabled by default. FLAG: -On self-managed GitLab, by default this feature is not available. To make it available per group, ask an administrator to [enable the feature flag](../administration/feature_flags.md) named `streaming_audit_event_headers`. -On GitLab.com, this feature is not available. -The feature is not ready for production use. +On self-managed GitLab, by default the API for this feature is available. To hide the feature, ask an administrator to [disable the feature flag](../administration/feature_flags.md) named `streaming_audit_event_headers`. +On GitLab.com, the API for this feature is available. Each streaming destination can have up to 20 custom HTTP headers included with each streamed event. -### Add with the API +### Adding custom HTTP headers -Group owners can add a HTTP header using the GraphQL `auditEventsStreamingHeadersCreate` mutation. +Add custom HTTP headers with the API or GitLab UI. + +#### Use the API + +Group owners can add a HTTP header using the GraphQL `auditEventsStreamingHeadersCreate` mutation. You can retrieve the destination ID +by [listing the external audit destinations](#list-streaming-destinations) on the group. ```graphql mutation { @@ -166,19 +173,111 @@ mutation { } ``` -### Delete with the API +The header is created if the returned `errors` object is empty. + +#### Use the GitLab UI + +FLAG: +On self-managed GitLab, by default the UI for this feature is not available. To make it available per group, ask an administrator to +[enable the feature flag](../administration/feature_flags.md) named `custom_headers_streaming_audit_events_ui`. On GitLab.com, the UI for this feature is +not available. The UI for this feature is not ready for production use. + +Users with at least the Owner role for a group can add event streaming destinations and custom HTTP headers for it: + +1. On the top bar, select **Menu > Groups** and find your group. +1. On the left sidebar, select **Security & Compliance > Audit events**. +1. On the main area, select **Streams** tab. + - When the destination list is empty, select **Add stream** to show the section for adding destinations. + - When the destination list is not empty, select **{plus}** to show the section for adding destinations. +1. Enter the destination URL to add. +1. Locate the **Custom HTTP headers** table. +1. In the **Header** column, add the header's name. +1. In the **Value** column, add the header's value. +1. Ignore the **Active** checkbox because it isn't functional. To track progress on adding functionality to the **Active** checkbox, see the + [relevant issue](https://gitlab.com/gitlab-org/gitlab/-/issues/361925). +1. Enter as many name and value pairs as required. When you enter a unique name and a value for a header, a new row in the table automatically appears. You can add up to + 20 headers per endpoint. +1. After all headers have been filled out, select **Add** to add the new endpoint. + +Event streaming is enabled if: + +- No warning is shown. +- The added endpoint is displayed in the UI. + +### Updating custom HTTP headers + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/361964) in GitLab 15.2. -Group owners can remove a HTTP header using the GraphQL `auditEventsStreamingHeadersDestroy` mutation. +Group owners can update a HTTP header using the GraphQL `auditEventsStreamingHeadersCreate` mutation. ```graphql mutation { - auditEventsStreamingHeadersDestroy(input: { headerId: "gid://gitlab/AuditEvents::ExternalAuditEventDestination/24601" }) { + auditEventsStreamingHeadersUpdate(input: { headerId: "gid://gitlab/AuditEvents::Streaming::Header/24255", key: "new-foo", value: "new-bar" }) { errors } } ``` -The header is created if the returned `errors` object is empty. +### Deleting custom HTTP headers + +Group owners can remove a HTTP header using the GraphQL `auditEventsStreamingHeadersDestroy` mutation. You can retrieve the header ID +by [listing all the custom headers](#list-all-custom-headers) on the group. + +```graphql +mutation { + auditEventsStreamingHeadersDestroy(input: { headerId: "gid://gitlab/AuditEvents::Streaming::Header/1" }) { + errors + } +} +``` + +The header is deleted if the returned `errors` object is empty. + +### List all custom headers + +List all custom HTTP headers with the API or GitLab UI. + +#### Use the API + +You can list all custom headers for a top-level group as well as their value and ID using the GraphQL `externalAuditEventDestinations` query. The ID +value returned by this query is what you need to pass to the `deletion` mutation. + +```graphql +query { + group(fullPath: "your-group") { + id + externalAuditEventDestinations { + nodes { + destinationUrl + id + headers { + nodes { + key + value + id + } + } + } + } + } +} +``` + +#### Use the GitLab UI + +FLAG: +On self-managed GitLab, by default the UI for this feature is not available. To make it available per group, ask an administrator to +[enable the feature flag](../administration/feature_flags.md) named `custom_headers_streaming_audit_events_ui`. On GitLab.com, the UI for this feature is +not available. The UI for this feature is not ready for production use. + +Users with at least the Owner role for a group can add event streaming destinations and custom HTTP headers for it: + +1. On the top bar, select **Menu > Groups** and find your group. +1. On the left sidebar, select **Security & Compliance > Audit events**. +1. On the main area, select **Streams** tab. +1. Select **{pencil}** at the right side of an item. +1. A read-only view of the items custom headers is shown. To track progress on adding editing functionality, see the [relevant issue](https://gitlab.com/gitlab-org/gitlab/-/issues/361925). +1. Select **Cancel** to close the read-only view. ## Verify event authenticity @@ -190,6 +289,17 @@ token is generated when the event destination is created and cannot be changed. Each streamed event contains a random alphanumeric identifier for the `X-Gitlab-Event-Streaming-Token` HTTP header that can be verified against the destination's value when [listing streaming destinations](#list-streaming-destinations). +### Use the GitLab UI + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/360814) in GitLab 15.2. + +Users with at least the Owner role for a group can list event streaming destinations and see the verification tokens: + +1. On the top bar, select **Menu > Groups** and find your group. +1. On the left sidebar, select **Security & Compliance > Audit events**. +1. On the main area, select **Streams**. +1. View the verification token on the right side of each item. + ## Audit event streaming on Git operations > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/332747) in GitLab 14.9 [with a flag](../administration/feature_flags.md) named `audit_event_streaming_git_operations`. Disabled by default. @@ -427,3 +537,215 @@ X-Gitlab-Audit-Event-Type: audit_operation "event_type": "audit_operation" } ``` + +## Audit event streaming on merge request create actions + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/90911) in GitLab 15.2. + +Stream audit events that relate to merge request create actions using the `/logs` endpoint. + +Send API requests that contain the `X-Gitlab-Audit-Event-Type` header with value `merge_request_create`. GitLab responds with JSON payloads with an +`event_type` field set to `merge_request_create`. + +### Headers + +Headers are formatted as follows: + +```plaintext +POST /logs HTTP/1.1 +Host: <DESTINATION_HOST> +Content-Type: application/x-www-form-urlencoded +X-Gitlab-Audit-Event-Type: merge_request_create +X-Gitlab-Event-Streaming-Token: <DESTINATION_TOKEN> +``` + +### Example payload + +```json +{ + "id": 1, + "author_id": 1, + "entity_id": 24, + "entity_type": "Project", + "details": { + "author_name": "example_user", + "target_id": 132, + "target_type": "MergeRequest", + "target_details": "Update test.md", + "custom_message": "Added merge request", + "ip_address": "127.0.0.1", + "entity_path": "example-group/example-project" + }, + "ip_address": "127.0.0.1", + "author_name": "Administrator", + "entity_path": "example-group/example-project", + "target_details": "Update test.md", + "created_at": "2022-07-04T00:19:22.675Z", + "target_type": "MergeRequest", + "target_id": 132, + "event_type": "merge_request_create" +} +``` + +## Audit event streaming on project fork actions + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/90916) in GitLab 15.2. + +Stream audit events that relate to project fork actions using the `/logs` endpoint. + +Send API requests that contain the `X-Gitlab-Audit-Event-Type` header with value `project_fork_operation`. GitLab responds with JSON payloads with an +`event_type` field set to `project_fork_operation`. + +### Headers + +Headers are formatted as follows: + +```plaintext +POST /logs HTTP/1.1 +Host: <DESTINATION_HOST> +Content-Type: application/x-www-form-urlencoded +X-Gitlab-Audit-Event-Type: project_fork_operation +X-Gitlab-Event-Streaming-Token: <DESTINATION_TOKEN> +``` + +### Example payload + +```json +{ + "id": 1, + "author_id": 1, + "entity_id": 24, + "entity_type": "Project", + "details": { + "author_name": "example_username", + "target_id": 24, + "target_type": "Project", + "target_details": "example-project", + "custom_message": "Forked project to another-group/example-project-forked", + "ip_address": "127.0.0.1", + "entity_path": "example-group/example-project" + }, + "ip_address": "127.0.0.1", + "author_name": "example_username", + "entity_path": "example-group/example-project", + "target_details": "example-project", + "created_at": "2022-06-30T03:43:35.384Z", + "target_type": "Project", + "target_id": 24, + "event_type": "project_fork_operation" +} +``` + +## Audit event streaming on project group link actions + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/90955) in GitLab 15.2. + +Stream audit events that relate to project group link creation, updates, and deletion using the `/logs` endpoint. + +Send API requests that contain the `X-Gitlab-Audit-Event-Type` header with value of either: + +- `project_group_link_create`. +- `project_group_link_update`. +- `project_group_link_destroy`. + +GitLab responds with JSON payloads with an `event_type` field set to either: + +- `project_group_link_create`. +- `project_group_link_update`. +- `project_group_link_destroy`. + +### Example Headers + +Headers are formatted as follows: + +```plaintext +POST /logs HTTP/1.1 +Host: <DESTINATION_HOST> +Content-Type: application/x-www-form-urlencoded +X-Gitlab-Audit-Event-Type: project_group_link_create +X-Gitlab-Event-Streaming-Token: <DESTINATION_TOKEN> +``` + +### Example payload for project group link create + +```json +{ + "id": 1, + "author_id": 1, + "entity_id": 24, + "entity_type": "Project", + "details": { + "author_name": "example-user", + "target_id": 31, + "target_type": "Group", + "target_details": "another-group", + "custom_message": "Added project group link", + "ip_address": "127.0.0.1", + "entity_path": "example-group/example-project" + }, + "ip_address": "127.0.0.1", + "author_name": "example-user", + "entity_path": "example-group/example-project", + "target_details": "another-group", + "created_at": "2022-07-04T00:43:09.318Z", + "target_type": "Group", + "target_id": 31, + "event_type": "project_group_link_create" +} +``` + +### Example payload for project group link update + +```json +{ + "id": 1, + "author_id": 1, + "entity_id": 24, + "entity_type": "Project", + "details": { + "author_name": "example-user", + "target_id": 31, + "target_type": "Group", + "target_details": "another-group", + "custom_message": "Changed project group link profile group_access from Developer to Guest", + "ip_address": "127.0.0.1", + "entity_path": "example-group/example-project" + }, + "ip_address": "127.0.0.1", + "author_name": "example-user", + "entity_path": "example-group/example-project", + "target_details": "another-group", + "created_at": "2022-07-04T00:43:28.328Z", + "target_type": "Group", + "target_id": 31, + "event_type": "project_group_link_update" +} +``` + +### Example payload for project group link delete + +```json +{ + "id": 1, + "author_id": 1, + "entity_id": 24, + "entity_type": "Project", + "details": { + "author_name": "example-user", + "target_id": 31, + "target_type": "Group", + "target_details": "another-group", + "custom_message": "Removed project group link", + "ip_address": "127.0.0.1", + "entity_path": "example-group/example-project" + }, + "ip_address": "127.0.0.1", + "author_name": "example-user", + "entity_path": "example-group/example-project", + "target_details": "another-group", + "created_at": "2022-07-04T00:42:56.279Z", + "target_type": "Group", + "target_id": 31, + "event_type": "project_group_link_destroy" +} +``` diff --git a/doc/administration/auth/index.md b/doc/administration/auth/index.md index ad8d78fdfd6..a412875501e 100644 --- a/doc/administration/auth/index.md +++ b/doc/administration/auth/index.md @@ -25,13 +25,15 @@ UltraAuth has removed their software which supports OmniAuth integration. We hav The external authentication and authorization providers may support the following capabilities. For more information, see the links shown on this page for each external provider. -| Capability | SaaS | Self-Managed | +| Capability | SaaS | Self-managed | |-------------------------------------------------|-----------------------------------------|------------------------------------| -| **User Provisioning** | SCIM<br>Just-In-Time (JIT) Provisioning | LDAP Sync | +| **User Provisioning** | SCIM<br>SAML <sup>1</sup> | LDAP <sup>1</sup><br>SAML <sup>1</sup><br>[OmniAuth Providers](../../integration/omniauth.md#supported-providers) <sup>1</sup> | | **User Detail Updating** (not group management) | Not Available | LDAP Sync | -| **Authentication** | SAML at top-level group (1 provider) | LDAP (multiple providers)<br>Generic OAuth2<br>SAML (only 1 permitted per unique provider)<br>Kerberos<br>JWT<br>Smartcard<br>OmniAuth Providers (only 1 permitted per unique provider) | +| **Authentication** | SAML at top-level group (1 provider) | LDAP (multiple providers)<br>Generic OAuth2<br>SAML (only 1 permitted per unique provider)<br>Kerberos<br>JWT<br>Smartcard<br>[OmniAuth Providers](../../integration/omniauth.md#supported-providers) (only 1 permitted per unique provider) | | **Provider-to-GitLab Role Sync** | SAML Group Sync | LDAP Group Sync<br>SAML Group Sync ([GitLab 15.1](https://gitlab.com/gitlab-org/gitlab/-/issues/285150) and later) | -| **User Removal** | SCIM (remove user from top-level group) | LDAP (Blocking User from Instance) | +| **User Removal** | SCIM (remove user from top-level group) | LDAP (remove user from groups and block from the instance) | + +1. Using Just-In-Time (JIT) provisioning, user accounts are created when the user first signs in. ## Change apps or configuration diff --git a/doc/administration/auth/ldap/img/multi_login.gif b/doc/administration/auth/ldap/img/multi_login.gif Binary files differdeleted file mode 100644 index 5aee6090793..00000000000 --- a/doc/administration/auth/ldap/img/multi_login.gif +++ /dev/null diff --git a/doc/administration/auth/ldap/img/multi_login.png b/doc/administration/auth/ldap/img/multi_login.png Binary files differnew file mode 100644 index 00000000000..512f403a442 --- /dev/null +++ b/doc/administration/auth/ldap/img/multi_login.png diff --git a/doc/administration/auth/ldap/index.md b/doc/administration/auth/ldap/index.md index 55b57193bf3..05eee338e64 100644 --- a/doc/administration/auth/ldap/index.md +++ b/doc/administration/auth/ldap/index.md @@ -7,7 +7,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Integrate LDAP with GitLab **(FREE SELF)** -GitLab integrates with [LDAP](https://en.wikipedia.org/wiki/Lightweight_Directory_Access_Protocol) +GitLab integrates with [LDAP - Lightweight Directory Access Protocol](https://en.wikipedia.org/wiki/Lightweight_Directory_Access_Protocol) to support user authentication. This integration works with most LDAP-compliant directory servers, including: @@ -274,7 +274,7 @@ gitlab_rails['ldap_servers'] = { This example results in the following sign-in page: - + ### Set up LDAP user filter diff --git a/doc/administration/auth/ldap/ldap_synchronization.md b/doc/administration/auth/ldap/ldap_synchronization.md index 0f0d301bfa9..b0ada1c11dd 100644 --- a/doc/administration/auth/ldap/ldap_synchronization.md +++ b/doc/administration/auth/ldap/ldap_synchronization.md @@ -42,11 +42,6 @@ The process also updates the following user information: - SSH public keys (if `sync_ssh_keys` is set) - Kerberos identity (if Kerberos is enabled) -The LDAP sync process: - -- Updates existing users. -- Creates new users on first sign in. - ### Adjust LDAP user sync schedule By default, GitLab runs a worker once per day at 01:30 a.m. server time to diff --git a/doc/administration/auth/oidc.md b/doc/administration/auth/oidc.md index deb47160d98..60a4cc8706f 100644 --- a/doc/administration/auth/oidc.md +++ b/doc/administration/auth/oidc.md @@ -339,7 +339,7 @@ The trailing forward slash is required. This configuration corresponds with the `Supported account types` setting used when creating the `IdentityExperienceFramework` app. -#### Keycloak +### Keycloak GitLab works with OpenID providers that use HTTPS. Although a Keycloak server can be set up using HTTP, GitLab can only communicate @@ -380,7 +380,7 @@ gitlab_rails['omniauth_providers'] = [ ] ``` -##### Configure Keycloak with a symmetric key algorithm +#### Configure Keycloak with a symmetric key algorithm > Introduced in GitLab 14.2. @@ -461,7 +461,7 @@ To use symmetric key encryption: If after reconfiguring, you see the error `JSON::JWS::VerificationFailed` error message, this means the incorrect secret was specified. -#### Casdoor +### Casdoor GitLab works with OpenID providers that use HTTPS. To connect to GitLab using OpenID with Casdoor, use HTTPS instead of HTTP. @@ -471,7 +471,7 @@ For your app, complete the following steps on Casdoor: 1. Add your GitLab redirect URL. For example, if your GitLab domain is `gitlab.example.com`, ensure the Casdoor app has the following `Redirect URI`: `https://gitlab.example.com/users/auth/openid_connect/callback`. -See the [Casdoor documentation](https://casdoor.org/docs/integration/gitlab/) for more details. +See the [Casdoor documentation](https://casdoor.org/docs/integration/gitlab) for more details. Example Omnibus GitLab configuration (file path: `/etc/gitlab/gitlab.rb`): diff --git a/doc/administration/consul.md b/doc/administration/consul.md index e1f2bd90e05..e282960c857 100644 --- a/doc/administration/consul.md +++ b/doc/administration/consul.md @@ -80,14 +80,14 @@ Nodes should be: - Upgraded one node at a time. Identify any existing health issues in the cluster by running the following command -within each node. The command will return an empty array if the cluster is healthy: +in each node. The command returns an empty array if the cluster is healthy: ```shell curl "http://127.0.0.1:8500/v1/health/state/critical" ``` -If the Consul version has changed, you'll see a notice at the end of `gitlab-ctl reconfigure` -informing you that Consul needs to be restarted for the new version to be used. +If the Consul version has changed, you see a notice at the end of `gitlab-ctl reconfigure` +informing you that Consul must be restarted for the new version to be used. Restart Consul one node at a time: @@ -96,7 +96,7 @@ sudo gitlab-ctl restart consul ``` Consul nodes communicate using the raft protocol. If the current leader goes -offline, there needs to be a leader election. A leader node must exist to facilitate +offline, there must be a leader election. A leader node must exist to facilitate synchronization across the cluster. If too many nodes go offline at the same time, the cluster loses quorum and doesn't elect a leader due to [broken consensus](https://www.consul.io/docs/architecture/consensus). @@ -111,7 +111,7 @@ bundled Consul wasn't used by any process other than GitLab itself, you can ## Troubleshooting Consul -Below are some useful operations should you need to debug any issues. +Below are some operations should you debug any issues. You can see any error logs by running: ```shell @@ -149,7 +149,7 @@ To be safe, it's recommended that you only restart Consul in one node at a time ensure the cluster remains intact. For larger clusters, it is possible to restart multiple nodes at a time. See the [Consul consensus document](https://www.consul.io/docs/architecture/consensus#deployment-table) -for the number of failures it can tolerate. This will be the number of simultaneous +for the number of failures it can tolerate. This is the number of simultaneous restarts it can sustain. To restart Consul: diff --git a/doc/administration/docs_self_host.md b/doc/administration/docs_self_host.md index 338601a054f..3f5d4adc95c 100644 --- a/doc/administration/docs_self_host.md +++ b/doc/administration/docs_self_host.md @@ -149,7 +149,7 @@ To extract the HTML files of the Docs site: docker cp gitlab-docs:/usr/share/nginx/html /srv/gitlab/ ``` - You will end up with a `/srv/gitlab/html/` directory that holds the documentation website. + You end up with a `/srv/gitlab/html/` directory that holds the documentation website. 1. Remove the container: diff --git a/doc/administration/encrypted_configuration.md b/doc/administration/encrypted_configuration.md index 38f033e260a..a96a4c4405d 100644 --- a/doc/administration/encrypted_configuration.md +++ b/doc/administration/encrypted_configuration.md @@ -14,12 +14,12 @@ GitLab can read settings for certain features from encrypted settings files. The - [LDAP `bind_dn` and `password`](auth/ldap/index.md#use-encrypted-credentials). - [SMTP `user_name` and `password`](raketasks/smtp.md#secrets). -In order to enable the encrypted configuration settings, a new base key needs to be generated for +To enable the encrypted configuration settings, a new base key must be generated for `encrypted_settings_key_base`. The secret can be generated in the following ways: **Omnibus Installation** -Starting with 13.7 the new secret is automatically generated for you, but you need to ensure your +Starting with 13.7 the new secret is automatically generated for you, but you must ensure your `/etc/gitlab/gitlab-secrets.json` contains the same values on all nodes. **GitLab Cloud Native Helm Chart** diff --git a/doc/administration/feature_flags.md b/doc/administration/feature_flags.md index 8a021c6d588..77f7f621a07 100644 --- a/doc/administration/feature_flags.md +++ b/doc/administration/feature_flags.md @@ -64,7 +64,7 @@ the status of the flag and the command to enable or disable it. ### Start the GitLab Rails console -The first thing you need to enable or disable a feature behind a flag is to +The first thing you must do to enable or disable a feature behind a flag is to start a session on GitLab Rails console. For Omnibus installations: @@ -83,7 +83,7 @@ For details, see [starting a Rails console session](operations/rails_console.md# ### Enable or disable the feature -Once the Rails console session has started, run the `Feature.enable` or +After the Rails console session has started, run the `Feature.enable` or `Feature.disable` commands accordingly. The specific flag can be found in the feature's documentation itself. @@ -123,11 +123,11 @@ For example, to enable the [`:product_analytics`](../operations/product_analytic Feature.enable(:product_analytics, Project.find(1234)) ``` -`Feature.enable` and `Feature.disable` always return `nil`, this is not an indication that the command failed: +`Feature.enable` and `Feature.disable` always return `true`, even if the application doesn't use the flag: ```ruby irb(main):001:0> Feature.enable(:my_awesome_feature) -=> nil +=> true ``` When the feature is ready, GitLab removes the feature flag, and the option for @@ -159,7 +159,7 @@ Feature.all ### Unset feature flag -You can unset a feature flag so that GitLab will fall back to the current defaults for that flag: +You can unset a feature flag so that GitLab falls back to the current defaults for that flag: ```ruby Feature.remove(:my_awesome_feature) diff --git a/doc/administration/geo/disaster_recovery/background_verification.md b/doc/administration/geo/disaster_recovery/background_verification.md index 57f8664ba11..97c9a6c5576 100644 --- a/doc/administration/geo/disaster_recovery/background_verification.md +++ b/doc/administration/geo/disaster_recovery/background_verification.md @@ -13,29 +13,25 @@ disable or enable this feature manually by following [these instructions](#disabling-or-enabling-the-automatic-background-verification). Automatic background verification ensures that the transferred data matches a -calculated checksum. If the checksum of the data on the **primary** node matches checksum of the -data on the **secondary** node, the data transferred successfully. Following a planned failover, +calculated checksum. If the checksum of the data on the **primary** site matches checksum of the +data on the **secondary** site, the data transferred successfully. Following a planned failover, any corrupted data may be **lost**, depending on the extent of the corruption. -If verification fails on the **primary** node, this indicates Geo is replicating a corrupted object. -You can restore it from backup or remove it from the **primary** node to resolve the issue. +If verification fails on the **primary** site, this indicates Geo is replicating a corrupted object. +You can restore it from backup or remove it from the **primary** site to resolve the issue. -If verification succeeds on the **primary** node but fails on the **secondary** node, +If verification succeeds on the **primary** site but fails on the **secondary** site, this indicates that the object was corrupted during the replication process. Geo actively try to correct verification failures marking the repository to be resynced with a back-off period. If you want to reset the verification for these failures, so you should follow [these instructions](background_verification.md#reset-verification-for-projects-where-verification-has-failed). If verification is lagging significantly behind replication, consider giving -the node more time before scheduling a planned failover. +the site more time before scheduling a planned failover. ## Disabling or enabling the automatic background verification -Run the following commands in a Rails console on the **primary** node: - -```shell -gitlab-rails console -``` +Run the following commands in a [Rails console](../../operations/rails_console.md) on a **Rails node on the primary** site. To check if automatic background verification is enabled: @@ -57,33 +53,33 @@ Feature.enable('geo_repository_verification') ## Repository verification -On the **primary** node: +On the **primary** site: 1. On the top bar, select **Menu > Admin**. -1. On the left sidebar, select **Geo > Nodes**. -1. Expand **Verification information** tab for that node to view automatic checksumming +1. On the left sidebar, select **Geo > Sites**. +1. Expand **Verification information** tab for that site to view automatic checksumming status for repositories and wikis. Successes are shown in green, pending work in gray, and failures in red.  -On the **secondary** node: +On the **secondary** site: 1. On the top bar, select **Menu > Admin**. -1. On the left sidebar, select **Geo > Nodes**. -1. Expand **Verification information** tab for that node to view automatic checksumming +1. On the left sidebar, select **Geo > Sites**. +1. Expand **Verification information** tab for that site to view automatic checksumming status for repositories and wikis. Successes are shown in green, pending work in gray, and failures in red.  -## Using checksums to compare Geo nodes +## Using checksums to compare Geo sites -To check the health of Geo **secondary** nodes, we use a checksum over the list of +To check the health of Geo **secondary** sites, we use a checksum over the list of Git references and their values. The checksum includes `HEAD`, `heads`, `tags`, -`notes`, and GitLab-specific references to ensure true consistency. If two nodes +`notes`, and GitLab-specific references to ensure true consistency. If two sites have the same checksum, then they definitely hold the same references. We compute -the checksum for every node after every update to make sure that they are all +the checksum for every site after every update to make sure that they are all in sync. ## Repository re-verification @@ -95,22 +91,17 @@ data. The default and recommended re-verification interval is 7 days, though an interval as short as 1 day can be set. Shorter intervals reduce risk but increase load and vice versa. -On the **primary** node: +On the **primary** site: 1. On the top bar, select **Menu > Admin**. -1. On the left sidebar, select **Geo > Nodes**. -1. Select **Edit** for the **primary** node to customize the minimum +1. On the left sidebar, select **Geo > Sites**. +1. Select **Edit** for the **primary** site to customize the minimum re-verification interval:  The automatic background re-verification is enabled by default, but you can -disable if you need. Run the following commands in a Rails console on the -**primary** node: - -```shell -gitlab-rails console -``` +disable if you need. Run the following commands in a [Rails console](../../operations/rails_console.md) on a **Rails node on the primary** site: To disable automatic background re-verification: @@ -126,11 +117,13 @@ Feature.enable('geo_repository_reverification') ## Reset verification for projects where verification has failed -Geo actively try to correct verification failures marking the repository to +Geo actively tries to correct verification failures marking the repository to be resynced with a back-off period. If you want to reset them manually, this Rake task marks projects where verification has failed or the checksum mismatch to be resynced without the back-off period: +Run the appropriate commands on a **Rails node on the primary** site. + For repositories: ```shell @@ -145,9 +138,9 @@ sudo gitlab-rake geo:verification:wiki:reset ## Reconcile differences with checksum mismatches -If the **primary** and **secondary** nodes have a checksum verification mismatch, the cause may not be apparent. To find the cause of a checksum mismatch: +If the **primary** and **secondary** sites have a checksum verification mismatch, the cause may not be apparent. To find the cause of a checksum mismatch: -1. On the **primary** node: +1. On the **primary** site: 1. On the top bar, select **Menu > Admin**. 1. On the left sidebar, select **Overview > Projects**. 1. Find the project that you want to check the checksum differences and @@ -157,31 +150,32 @@ If the **primary** and **secondary** nodes have a checksum verification mismatch  -1. Go to the project's repository directory on both **primary** and **secondary** nodes - (the path is usually `/var/opt/gitlab/git-data/repositories`). If `git_data_dirs` +1. On a **Gitaly node on the primary** site and a **Gitaly node on the secondary** site, go to the project's repository directory. If using Gitaly Cluster, [check that it is in a healthy state](../../gitaly/troubleshooting.md#check-cluster-health) prior to running these commands. + + The default path is `/var/opt/gitlab/git-data/repositories`. If `git_data_dirs` is customized, check the directory layout on your server to be sure: ```shell cd /var/opt/gitlab/git-data/repositories ``` -1. Run the following command on the **primary** node, redirecting the output to a file: + 1. Run the following command on the **primary** site, redirecting the output to a file: - ```shell - git show-ref --head | grep -E "HEAD|(refs/(heads|tags|keep-around|merge-requests|environments|notes)/)" > primary-node-refs - ``` + ```shell + git show-ref --head | grep -E "HEAD|(refs/(heads|tags|keep-around|merge-requests|environments|notes)/)" > primary-site-refs + ``` -1. Run the following command on the **secondary** node, redirecting the output to a file: + 1. Run the following command on the **secondary** site, redirecting the output to a file: - ```shell - git show-ref --head | grep -E "HEAD|(refs/(heads|tags|keep-around|merge-requests|environments|notes)/)" > secondary-node-refs - ``` + ```shell + git show-ref --head | grep -E "HEAD|(refs/(heads|tags|keep-around|merge-requests|environments|notes)/)" > secondary-site-refs + ``` -1. Copy the files from the previous steps on the same system, and do a diff between the contents: + 1. Copy the files from the previous steps on the same system, and do a diff between the contents: - ```shell - diff primary-node-refs secondary-node-refs - ``` + ```shell + diff primary-site-refs secondary-site-refs + ``` ## Current limitations @@ -191,10 +185,10 @@ progress to include them in [Geo: Verify all replicated data](https://gitlab.com For now, you can verify their integrity manually by following [these instructions](../../raketasks/check.md) on both -nodes, and comparing the output between them. +sites, and comparing the output between them. In GitLab EE 12.1, Geo calculates checksums for attachments, LFS objects, and -archived traces on secondary nodes after the transfer, compares it with the +archived traces on secondary sites after the transfer, compares it with the stored checksums, and rejects transfers if mismatched. Geo currently does not support an automatic way to verify these data if they have been synced before GitLab EE 12.1. diff --git a/doc/administration/geo/disaster_recovery/index.md b/doc/administration/geo/disaster_recovery/index.md index e5b9a14cfbd..f457cb4b0a2 100644 --- a/doc/administration/geo/disaster_recovery/index.md +++ b/doc/administration/geo/disaster_recovery/index.md @@ -413,7 +413,7 @@ required: 1. Promote the replica database associated with the **secondary** site. This sets the database to read-write. The instructions vary depending on where your database is hosted: - [Amazon RDS](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_ReadRepl.html#USER_ReadRepl.Promote) - - [Azure PostgreSQL](https://docs.microsoft.com/en-us/azure/postgresql/howto-read-replicas-portal#stop-replication) + - [Azure PostgreSQL](https://docs.microsoft.com/en-us/azure/postgresql/single-server/how-to-read-replicas-portal#stop-replication) - [Google Cloud SQL](https://cloud.google.com/sql/docs/mysql/replication/manage-replicas#promote-replica) - For other external PostgreSQL databases, save the following script in your secondary site, for example `/tmp/geo_promote.sh`, and modify the connection diff --git a/doc/administration/geo/disaster_recovery/planned_failover.md b/doc/administration/geo/disaster_recovery/planned_failover.md index c351b4031b5..5a5d896c20a 100644 --- a/doc/administration/geo/disaster_recovery/planned_failover.md +++ b/doc/administration/geo/disaster_recovery/planned_failover.md @@ -60,8 +60,8 @@ Alternatively, you can [back up](../../../raketasks/backup_restore.md#back-up-gi the container registry on the primary site and restore it onto the secondary site: -1. On your primary site, back up only the registry and [exclude specific directories -from the backup](../../../raketasks/backup_restore.md#excluding-specific-directories-from-the-backup): +1. On your primary site, back up only the registry and + [exclude specific directories from the backup](../../../raketasks/backup_gitlab.md#excluding-specific-directories-from-the-backup): ```shell # Create a backup in the /var/opt/gitlab/backups folder diff --git a/doc/administration/geo/index.md b/doc/administration/geo/index.md index c5472e412d3..cf7d2649142 100644 --- a/doc/administration/geo/index.md +++ b/doc/administration/geo/index.md @@ -123,7 +123,7 @@ The following are required to run Geo: - PostgreSQL 13 is not supported for Geo, see [epic 3832](https://gitlab.com/groups/gitlab-org/-/epics/3832) - Git 2.9 or later - Git-lfs 2.4.2 or later on the user side when using LFS -- All sites must run the same GitLab version. +- All sites must run [the same GitLab and PostgreSQL versions](setup/database.md#postgresql-replication). Additionally, check the GitLab [minimum requirements](../../install/requirements.md), and we recommend you use the latest version of GitLab for a better experience. diff --git a/doc/administration/geo/replication/configuration.md b/doc/administration/geo/replication/configuration.md index 043b96478c9..7666a450648 100644 --- a/doc/administration/geo/replication/configuration.md +++ b/doc/administration/geo/replication/configuration.md @@ -12,7 +12,9 @@ type: howto NOTE: This is the final step in setting up a **secondary** Geo site. Stages of the setup process must be completed in the documented order. -Before attempting the steps in this stage, [complete all prior stages](../setup/index.md#using-omnibus-gitlab). +If not, [complete all prior stages](../setup/index.md#using-omnibus-gitlab) before procceed. + +Make sure you [set up the database replication](../setup/database.md), and [configured fast lookup of authorized SSH keys](../../operations/fast_ssh_key_lookup.md) in **both primary and secondary sites**. The basic steps of configuring a **secondary** site are to: @@ -318,7 +320,7 @@ the **primary** site. After you sign in: 1. Verify that it's correctly identified as a **secondary** Geo site, and that Geo is enabled. -The initial replication, or 'backfill', is probably still in progress. You +The initial replication may take some time. The status of the site or the ‘backfill’ may still in progress. You can monitor the synchronization process on each Geo site from the **primary** site's **Geo Sites** dashboard in your browser. diff --git a/doc/administration/geo/replication/datatypes.md b/doc/administration/geo/replication/datatypes.md index e49d5260233..acd27d5feed 100644 --- a/doc/administration/geo/replication/datatypes.md +++ b/doc/administration/geo/replication/datatypes.md @@ -202,8 +202,8 @@ successfully, you must replicate their data using some other means. |[External merge request diffs](../../merge_request_diffs.md) | **Yes** (13.5) | **Yes** (14.6) | [**Yes** (15.1)](https://gitlab.com/groups/gitlab-org/-/epics/5551) | [No](object_storage.md#verification-of-files-in-object-storage) | Replication is behind the feature flag `geo_merge_request_diff_replication`, enabled by default. Verification was behind the feature flag `geo_merge_request_diff_verification`, removed in 14.7.| |[Versioned snippets](../../../user/snippets.md#versioned-snippets) | [**Yes** (13.7)](https://gitlab.com/groups/gitlab-org/-/epics/2809) | [**Yes** (14.2)](https://gitlab.com/groups/gitlab-org/-/epics/2810) | N/A | N/A | Verification was implemented behind the feature flag `geo_snippet_repository_verification` in 13.11, and the feature flag was removed in 14.2. | |[GitLab Pages](../../pages/index.md) | [**Yes** (14.3)](https://gitlab.com/groups/gitlab-org/-/epics/589) | **Yes** (14.6) | [**Yes** (15.1)](https://gitlab.com/groups/gitlab-org/-/epics/5551) | [No](object_storage.md#verification-of-files-in-object-storage) | Behind feature flag `geo_pages_deployment_replication`, enabled by default. Verification was behind the feature flag `geo_pages_deployment_verification`, removed in 14.7. | -|[Incident Metric Images](../../../operations/incident_management/incidents.md#metrics) | [Planned](https://gitlab.com/gitlab-org/gitlab/-/issues/352326) | [No](https://gitlab.com/gitlab-org/gitlab/-/issues/362561) | No | No | | -|[Alert Metric Images](../../../operations/incident_management/alerts.md#metrics-tab) | [Planned](https://gitlab.com/gitlab-org/gitlab/-/issues/352326) | [No](https://gitlab.com/gitlab-org/gitlab/-/issues/362561) | No | No | | +|[Incident Metric Images](../../../operations/incident_management/incidents.md#metrics) | [Planned](https://gitlab.com/gitlab-org/gitlab/-/issues/362561) | [No](https://gitlab.com/gitlab-org/gitlab/-/issues/362561) | No | No | | +|[Alert Metric Images](../../../operations/incident_management/alerts.md#metrics-tab) | [Planned](https://gitlab.com/gitlab-org/gitlab/-/issues/362564) | [No](https://gitlab.com/gitlab-org/gitlab/-/issues/362564) | No | No | | |[Server-side Git hooks](../../server_hooks.md) | [Not planned](https://gitlab.com/groups/gitlab-org/-/epics/1867) | No | N/A | N/A | Not planned because of current implementation complexity, low customer interest, and availability of alternatives to hooks. | |[Elasticsearch integration](../../../integration/elasticsearch.md) | [Not planned](https://gitlab.com/gitlab-org/gitlab/-/issues/1186) | No | No | No | Not planned because further product discovery is required and Elasticsearch (ES) clusters can be rebuilt. Secondaries use the same ES cluster as the primary. | |[Dependency proxy images](../../../user/packages/dependency_proxy/index.md) | [Not planned](https://gitlab.com/gitlab-org/gitlab/-/issues/259694) | No | No | No | Blocked by [Geo: Secondary Mimicry](https://gitlab.com/groups/gitlab-org/-/epics/1528). Replication of this cache is not needed for disaster recovery purposes because it can be recreated from external sources. | diff --git a/doc/administration/geo/replication/object_storage.md b/doc/administration/geo/replication/object_storage.md index dd1f982b3a1..d2e10678f8c 100644 --- a/doc/administration/geo/replication/object_storage.md +++ b/doc/administration/geo/replication/object_storage.md @@ -34,8 +34,7 @@ See [Object storage replication tests](geo_validation_tests.md#object-storage-re ## Enabling GitLab-managed object storage replication -> The beta feature was [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/10586) in GitLab 12.4. -> The feature was made [generally available] in GitLab 15.1. +> The feature was made [generally available](https://gitlab.com/groups/gitlab-org/-/epics/5551) in GitLab 15.1. **Secondary** sites can replicate files stored on the **primary** site regardless of whether they are stored on the local file system or in object storage. @@ -79,7 +78,7 @@ the bucket used by **secondary** sites. If you are using Google Cloud Storage, consider using [Multi-Regional Storage](https://cloud.google.com/storage/docs/storage-classes#multi-regional). -Or you can use the [Storage Transfer Service](https://cloud.google.com/storage-transfer/docs/), +Or you can use the [Storage Transfer Service](https://cloud.google.com/storage-transfer/docs/overview), although this only supports daily synchronization. For manual synchronization, or scheduled by `cron`, see: diff --git a/doc/administration/geo/replication/troubleshooting.md b/doc/administration/geo/replication/troubleshooting.md index bb7dbef214b..082ecbbb208 100644 --- a/doc/administration/geo/replication/troubleshooting.md +++ b/doc/administration/geo/replication/troubleshooting.md @@ -183,7 +183,7 @@ This machine's Geo node name matches a database record ... no ``` Learn more about recommended site names in the description of the Name field in -[Geo Admin Area Common Settings](../../../user/admin_area/geo_nodes.md#common-settings). +[Geo Admin Area Common Settings](../../../user/admin_area/geo_sites.md#common-settings). ### Message: `WARNING: oldest xmin is far in the past` and `pg_wal` size growing @@ -519,7 +519,7 @@ To solve this: curl --request GET --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/<first_failed_geo_sync_ID>" ``` -1. Enter the [Rails console](../../troubleshooting/navigating_gitlab_via_rails_console.md) and run: +1. Enter the [Rails console](../../operations/rails_console.md) and run: ```ruby failed_geo_syncs = Geo::ProjectRegistry.failed.pluck(:id) @@ -651,8 +651,11 @@ to start again from scratch, there are a few steps that can help you: 1. Reset the Tracking Database. + WARNING: + If you skipped the optional step 3, be sure both `geo-postgresql` and `postgresql` services are running. + ```shell - gitlab-rake db:drop:geo # on a secondary app node + gitlab-rake db:drop:geo DISABLE_DATABASE_ENVIRONMENT_CHECK=1 # on a secondary app node gitlab-ctl reconfigure # on the tracking database node gitlab-rake db:migrate:geo # on a secondary app node ``` @@ -805,7 +808,7 @@ You can work around this by marking the objects as synced and succeeded verifica be aware that can also mark objects that may be [missing from the primary](#missing-files-on-the-geo-primary-site). -To do that, enter the [Rails console](../../troubleshooting/navigating_gitlab_via_rails_console.md) +To do that, enter the [Rails console](../../operations/rails_console.md) and run: ```ruby @@ -817,16 +820,16 @@ end ### Message: curl 18 transfer closed with outstanding read data remaining & fetch-pack: unexpected disconnect while reading sideband packet -Unstable networking conditions can cause Gitaly to fail when trying to fetch large repository +Unstable networking conditions can cause Gitaly to fail when trying to fetch large repository data from the primary site. This is more likely to happen if a repository has to be replicated from scratch between sites. Geo retries several times, but if the transmission is consistently interrupted -by network hiccups, an alternative method such as `rsync` can be used to circumvent `git` and +by network hiccups, an alternative method such as `rsync` can be used to circumvent `git` and create the initial copy of any repository that fails to be replicated by Geo. We recommend transferring each failing repository individually and checking for consistency -after each transfer. Follow the [single target `rsync` instructions](../../operations/moving_repositories.md#single-rsync-to-another-server) +after each transfer. Follow the [single target `rsync` instructions](../../operations/moving_repositories.md#single-rsync-to-another-server) to transfer each affected repository from the primary to the secondary site. ## Fixing errors during a failover or when promoting a secondary to a primary node diff --git a/doc/administration/geo/secondary_proxy/index.md b/doc/administration/geo/secondary_proxy/index.md index 9a1aab8c238..e8c290e197b 100644 --- a/doc/administration/geo/secondary_proxy/index.md +++ b/doc/administration/geo/secondary_proxy/index.md @@ -124,7 +124,7 @@ sudo gitlab-rails runner "Feature.disable(:geo_secondary_proxy_separate_urls)" ``` In Kubernetes, you can run the same command in the toolbox pod. Refer to the -[Kubernetes cheat sheet](../../troubleshooting/kubernetes_cheat_sheet.md#gitlab-specific-kubernetes-information) +[Kubernetes cheat sheet](https://docs.gitlab.com/charts/troubleshooting/kubernetes_cheat_sheet.html#gitlab-specific-kubernetes-information) for details. ## Limitations diff --git a/doc/administration/geo/setup/database.md b/doc/administration/geo/setup/database.md index 7dfca4c8f73..c0ed7829fce 100644 --- a/doc/administration/geo/setup/database.md +++ b/doc/administration/geo/setup/database.md @@ -17,8 +17,11 @@ PostgreSQL instances, the Omnibus roles cannot perform all necessary configuration steps. In this case, use the [Geo with external PostgreSQL instances](external_database.md) process instead. +NOTE: The stages of the setup process must be completed in the documented order. -Before you attempt the steps in this stage, [complete all prior stages](../setup/index.md#using-omnibus-gitlab). +If not, [complete all prior stages](../setup/index.md#using-omnibus-gitlab) before proceeding. + +Ensure the **secondary** site is running the same version of GitLab Enterprise Edition as the **primary** site. Confirm you have added the [premium or higher licenses](https://about.gitlab.com/pricing/) to your **primary** site. Be sure to read and review all of these steps before you execute them in your testing or production environments. @@ -52,8 +55,9 @@ The following guide assumes that: which includes the [`pg_basebackup` tool](https://www.postgresql.org/docs/12/app-pgbasebackup.html). - You have a **primary** node already set up (the GitLab server you are replicating from), running Omnibus' PostgreSQL (or equivalent version), and - you have a new **secondary** server set up with the same versions of the OS, - PostgreSQL, and GitLab on all nodes. + you have a new **secondary** server set up with the same + [versions of PostgreSQL](../index.md#requirements-for-running-geo), + OS, and GitLab on all nodes. WARNING: Geo works with streaming replication. Logical replication is not supported at this time. @@ -478,7 +482,7 @@ data before running `pg_basebackup`. ```shell gitlab-ctl replicate-geo-database \ --slot-name=<secondary_node_name> \ - --host=<primary_node_ip> + --host=<primary_node_ip> \ --sslmode=verify-ca ``` diff --git a/doc/administration/geo/setup/external_database.md b/doc/administration/geo/setup/external_database.md index 7e8ffa829f9..b87baa658a1 100644 --- a/doc/administration/geo/setup/external_database.md +++ b/doc/administration/geo/setup/external_database.md @@ -10,14 +10,19 @@ This document is relevant if you are using a PostgreSQL instance that is *not managed by Omnibus*. This includes cloud-managed instances like Amazon RDS, or manually installed and configured PostgreSQL instances. +Ensure that you are using one of the PostgreSQL versions that +[Omnibus ships with](../../package_information/postgresql_versions.md) +to [avoid version mismatches](../index.md#requirements-for-running-geo) +in case a Geo site has to be rebuilt. + NOTE: We strongly recommend running Omnibus-managed instances as they are actively developed and tested. We aim to be compatible with most external (not managed by Omnibus) databases but we do not guarantee compatibility. -## **Primary** node +## **Primary** site -1. SSH into a GitLab **primary** application server and login as root: +1. SSH into a **Rails node on your primary** site and login as root: ```shell sudo -i @@ -39,13 +44,13 @@ developed and tested. We aim to be compatible with most external gitlab_rails['geo_node_name'] = '<site_name_here>' ``` -1. Reconfigure the **primary** node for the change to take effect: +1. Reconfigure the **Rails node** for the change to take effect: ```shell gitlab-ctl reconfigure ``` -1. Execute the command below to define the node as **primary** node: +1. Execute the command below on the **Rails node** to define the site as **primary** site: ```shell gitlab-ctl set-geo-primary-node @@ -62,10 +67,10 @@ To set up an external database, you can either: #### Leverage your cloud provider's tools to replicate the primary database -Given you have a primary node set up on AWS EC2 that uses RDS. +Given you have a primary site set up on AWS EC2 that uses RDS. You can now just create a read-only replica in a different region and the replication process is managed by AWS. Make sure you've set Network ACL (Access Control List), Subnet, and -Security Group according to your needs, so the secondary application node can access the database. +Security Group according to your needs, so the secondary Rails node(s) can access the database. The following instructions detail how to create a read-only replica for common cloud providers: @@ -74,7 +79,7 @@ cloud providers: - Azure Database for PostgreSQL - [Create and manage read replicas in Azure Database for PostgreSQL](https://docs.microsoft.com/en-us/azure/postgresql/single-server/how-to-read-replicas-portal) - Google Cloud SQL - [Creating read replicas](https://cloud.google.com/sql/docs/postgres/replication/create-replica) -Once your read-only replica is set up, you can skip to [configure your secondary application node](#configure-secondary-application-nodes-to-use-the-external-read-replica). +Once your read-only replica is set up, you can skip to [configure your secondary site](#configure-secondary-site-to-use-the-external-read-replica) #### Manually configure the primary database for replication @@ -107,7 +112,7 @@ max_replication_slots = 1 # number of secondary instances hot_standby = on ``` -## **Secondary** nodes +## **Secondary** sites ### Manually configure the replica database @@ -136,7 +141,7 @@ wal_keep_segments = 10 hot_standby = on ``` -### Configure **secondary** application nodes to use the external read-replica +### Configure **secondary** site to use the external read-replica With Omnibus, the [`geo_secondary_role`](https://docs.gitlab.com/omnibus/roles/#gitlab-geo-roles) @@ -148,7 +153,7 @@ has three main functions: To configure the connection to the external read-replica database and enable Log Cursor: -1. SSH into a GitLab **secondary** application server and login as root: +1. SSH into each **Rails, Sidekiq and Geo Log Cursor** node on your **secondary** site and login as root: ```shell sudo -i @@ -179,7 +184,7 @@ To configure the connection to the external read-replica database and enable Log ### Configure the tracking database -**Secondary** nodes use a separate PostgreSQL installation as a tracking +**Secondary** sites use a separate PostgreSQL installation as a tracking database to keep track of replication status and automatically recover from potential replication issues. Omnibus automatically configures a tracking database when `roles ['geo_secondary_role']` is set. @@ -209,7 +214,7 @@ the tracking database on port 5432. [database requirements document](../../../install/requirements.md#database). 1. Set up a `gitlab_geo` user with a password of your choice, create the `gitlabhq_geo_production` database, and make the user an owner of the database. You can see an example of this setup in the [installation from source documentation](../../../install/installation.md#6-database). 1. If you are **not** using a cloud-managed PostgreSQL database, ensure that your secondary - node can communicate with your tracking database by manually changing the + site can communicate with your tracking database by manually changing the `pg_hba.conf` that is associated with your tracking database. Remember to restart PostgreSQL afterwards for the changes to take effect: diff --git a/doc/administration/geo/setup/index.md b/doc/administration/geo/setup/index.md index f4c21293782..5ddfee6774e 100644 --- a/doc/administration/geo/setup/index.md +++ b/doc/administration/geo/setup/index.md @@ -12,21 +12,25 @@ These instructions assume you have a working instance of GitLab. They guide you 1. Making your existing instance the **primary** site. 1. Adding **secondary** sites. +You must use a [GitLab Premium](https://about.gitlab.com/pricing/) license or higher, +but you only need one license for all the sites. + WARNING: -The steps below should be followed in the order they appear. **Make sure the GitLab version is the same on all sites.** +The steps below should be followed in the order they appear. **Make sure the GitLab version is the same on all sites. Do not create an account or log in to the new secondary.** ## Using Omnibus GitLab If you installed GitLab using the Omnibus packages (highly recommended): -1. [Install GitLab Enterprise Edition](https://about.gitlab.com/install/) on the nodes that serve as the **secondary** site. Do not create an account or log in to the new **secondary** site. The **GitLab version must match** across primary and secondary sites. +1. [Install GitLab Enterprise Edition](https://about.gitlab.com/install/) on the nodes that serve as the **secondary** site. **Do not create an account or log in** to the new **secondary** site. The **GitLab version must match** across primary and secondary sites. 1. [Add the GitLab License](../../../user/admin_area/license.md) on the **primary** site to unlock Geo. The license must be for [GitLab Premium](https://about.gitlab.com/pricing/) or higher. 1. [Set up the database replication](database.md) (`primary (read-write) <-> secondary (read-only)` topology). 1. [Configure fast lookup of authorized SSH keys in the database](../../operations/fast_ssh_key_lookup.md). This step is required and needs to be done on **both** the **primary** and **secondary** sites. 1. [Configure GitLab](../replication/configuration.md) to set the **primary** and **secondary** sites. +1. Optional: [Configure Object storage](../../object_storage.md) 1. Optional: [Configure a secondary LDAP server](../../auth/ldap/index.md) for the **secondary** sites. See [notes on LDAP](../index.md#ldap). +1. Optional: [Configure Geo secondary proxying](../secondary_proxy/index.md) to use a single, unified URL for all Geo sites. This step is recommended to accelerate most read requests while transparently proxying writes to the primary Geo site. 1. Follow the [Using a Geo Site](../replication/usage.md) guide. -1. [Configure Geo secondary proxying](../secondary_proxy/index.md) to use a single, unified URL for all Geo sites. This step is recommended to accelerate most read requests while transparently proxying writes to the primary Geo site. ## Post-installation documentation diff --git a/doc/administration/gitaly/configure_gitaly.md b/doc/administration/gitaly/configure_gitaly.md index 426d07b154d..4b2832bebc0 100644 --- a/doc/administration/gitaly/configure_gitaly.md +++ b/doc/administration/gitaly/configure_gitaly.md @@ -1,5 +1,5 @@ --- -stage: Create +stage: Systems group: Gitaly info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- @@ -130,57 +130,49 @@ Install Gitaly on each Gitaly server using either Omnibus GitLab or install it f - To install from source, follow the steps at [Install Gitaly](../../install/installation.md#install-gitaly). -### Configure authentication +### Configure Gitaly servers -Gitaly and GitLab use two shared secrets for authentication: +To configure Gitaly servers, you must: -- One to authenticate gRPC requests to Gitaly. -- A second for authentication callbacks from GitLab Shell to the GitLab internal API. +- Configure authentication. +- Configure storage paths. +- Enable the network listener. -**For Omnibus GitLab** +The `git` user must be able to read, write, and set permissions on the configured storage path. -To configure the Gitaly token: +To avoid downtime while rotating Gitaly's token, you can temporarily disable authentication using the `gitaly['auth_transitioning']` setting. For more information, see the documentation on +[enabling "auth transitioning mode"](#enable-auth-transitioning-mode). -1. On the Gitaly clients, edit `/etc/gitlab/gitlab.rb`: +#### Configure authentication - ```ruby - gitlab_rails['gitaly_token'] = 'abc123secret' - ``` +Gitaly and GitLab use two shared secrets for authentication: -1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). -1. On the Gitaly server, edit `/etc/gitlab/gitlab.rb`: +- _Gitaly token_: used to authenticate gRPC requests to Gitaly +- _GitLab Shell token_: used for authentication callbacks from GitLab Shell to the GitLab internal API + +**For Omnibus GitLab** + +To configure the _Gitaly token_, edit `/etc/gitlab/gitlab.rb`: ```ruby gitaly['auth_token'] = 'abc123secret' ``` -1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). - -There are two ways to configure the GitLab Shell token. +There are two ways to configure the _GitLab Shell token_. -Method 1: +Method 1 (recommended): -1. Copy `/etc/gitlab/gitlab-secrets.json` from the Gitaly client to same path on the Gitaly servers +Copy `/etc/gitlab/gitlab-secrets.json` from the Gitaly client to same path on the Gitaly servers (and any other Gitaly clients). -1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) on Gitaly servers. Method 2: -1. On the Gitaly clients, edit `/etc/gitlab/gitlab.rb`: +Edit `/etc/gitlab/gitlab.rb`: ```ruby gitlab_shell['secret_token'] = 'shellsecret' ``` -1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). -1. On the Gitaly servers, edit `/etc/gitlab/gitlab.rb`: - - ```ruby - gitlab_shell['secret_token'] = 'shellsecret' - ``` - -1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). - **For installations from source** 1. Copy `/home/git/gitlab/.gitlab_shell_secret` from the Gitaly client to the same path on the @@ -203,14 +195,7 @@ Method 2: 1. Save the file and [restart GitLab](../restart_gitlab.md#installations-from-source). -### Configure Gitaly servers - -On the Gitaly servers, you must configure storage paths and enable the network listener. -The Gitaly server must be able to read, write, and set permissions on the configured path. - -If you want to reduce the risk of downtime when you enable authentication, you can temporarily -disable enforcement. For more information, see the documentation on configuring -[Gitaly authentication](https://gitlab.com/gitlab-org/gitaly/blob/master/doc/configuration/README.md#authentication). +#### Configure Gitaly server **For Omnibus GitLab** @@ -904,7 +889,7 @@ gitaly['cgroups_repositories_cpu_shares'] => 512 which represents 100% of CPU. This value cannot exceed that of the top level`cgroups_cpu_shares`. -#### Configure cgroups (legacy method) +#### Configure cgroups (legacy method) To configure cgroups in Gitaly for GitLab versions using the legacy method, add `gitaly['cgroups']` to `/etc/gitlab/gitlab.rb`. For example: diff --git a/doc/administration/gitaly/index.md b/doc/administration/gitaly/index.md index 160313073a5..c543f62f135 100644 --- a/doc/administration/gitaly/index.md +++ b/doc/administration/gitaly/index.md @@ -1,5 +1,5 @@ --- -stage: Create +stage: Systems group: Gitaly info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- @@ -71,7 +71,7 @@ the current status of these issues, please refer to the referenced issues and ep | Issue | Summary | How to avoid | |:--------------------------------------------------------------------------------------|:------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:--------------------------------| | Gitaly Cluster + Geo - Issues retrying failed syncs | If Gitaly Cluster is used on a Geo secondary site, repositories that have failed to sync could continue to fail when Geo tries to resync them. Recovering from this state requires assistance from support to run manual steps. Work is in-progress to update Gitaly Cluster to [identify repositories with a unique and persistent identifier](https://gitlab.com/gitlab-org/gitaly/-/issues/3485), which is expected to resolve the issue. | No known solution at this time. | -| Praefect unable to insert data into the database due to migrations not being applied after an upgrade | If the database is not kept up to date with completed migrations, then the Praefect node is unable to perform normal operation. | Make sure the Praefect database is up and running with all migrations completed (For example: `/opt/gitlab/embedded/bin/praefect -config /var/opt/gitlab/praefect/config.toml sql-migrate-status` should show a list of all applied migrations). Consider [requesting live upgrade assistance](https://about.gitlab.com/support/scheduling-upgrade-assistance/) so your upgrade plan can be reviewed by support. | +| Praefect unable to insert data into the database due to migrations not being applied after an upgrade | If the database is not kept up to date with completed migrations, then the Praefect node is unable to perform normal operation. | Make sure the Praefect database is up and running with all migrations completed (For example: `/opt/gitlab/embedded/bin/praefect -config /var/opt/gitlab/praefect/config.toml sql-migrate-status` should show a list of all applied migrations). Consider [requesting upgrade assistance](https://about.gitlab.com/support/scheduling-upgrade-assistance/) so your upgrade plan can be reviewed by support. | | Restoring a Gitaly Cluster node from a snapshot in a running cluster | Because the Gitaly Cluster runs with consistent state, introducing a single node that is behind will result in the cluster not being able to reconcile the nodes data and other nodes data | Don't restore a single Gitaly Cluster node from a backup snapshot. If you must restore from backup, it's best to snapshot all Gitaly Cluster nodes at the same time and take a database dump of the Praefect database. | ### Snapshot backup and recovery limitations @@ -523,6 +523,48 @@ For more information on configuring Gitaly Cluster, see [Configure Gitaly Cluste To upgrade a Gitaly Cluster, follow the documentation for [zero downtime upgrades](../../update/zero_downtime.md#gitaly-or-gitaly-cluster). +### Downgrade Gitaly Cluster to a previous version + +If you need to roll back a Gitaly Cluster to an earlier version, some Praefect database migrations may need to be reverted. In a cluster with: + +- A single Praefect node, this happens when GitLab itself is downgraded. +- Multiple Praefect nodes, additional steps are required. + +To downgrade a Gitaly Cluster with multiple Praefect nodes: + +1. Stop the Praefect service on all Praefect nodes: + + ```shell + gitlab-ctl stop praefect + ``` + +1. Downgrade the GitLab package to the older version on one of the Praefect nodes. +1. On the downgraded node, check the state of Praefect migrations: + + ```shell + /opt/gitlab/embedded/bin/praefect -config /var/opt/gitlab/praefect/config.toml sql-migrate-status + ``` + +1. Count the number of migrations with `unknown migration` in the `APPLIED` column. +1. On a Praefect node that has **not** been downgraded, perform a dry run of the rollback to validate which migrations to revert. `<CT_UNKNOWN>` + is the number of unknown migrations reported by the downgraded node. + + ```shell + /opt/gitlab/embedded/bin/praefect -config /var/opt/gitlab/praefect/config.toml sql-migrate <CT_UNKNOWN> + ``` + +1. If the results look correct, run the same command with the `-f` option to revert the migrations: + + ```shell + /opt/gitlab/embedded/bin/praefect -config /var/opt/gitlab/praefect/config.toml sql-migrate -f <CT_UNKNOWN> + ``` + +1. Downgrade the GitLab package on the remaining Praefect nodes and start the Praefect service again: + + ```shell + gitlab-ctl start praefect + ``` + ## Migrate to Gitaly Cluster WARNING: diff --git a/doc/administration/gitaly/monitoring.md b/doc/administration/gitaly/monitoring.md index ac0c4cf139d..0dd9e0d7128 100644 --- a/doc/administration/gitaly/monitoring.md +++ b/doc/administration/gitaly/monitoring.md @@ -1,5 +1,5 @@ --- -stage: Create +stage: Systems group: Gitaly info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/administration/gitaly/praefect.md b/doc/administration/gitaly/praefect.md index f6166d713b1..5635898293b 100644 --- a/doc/administration/gitaly/praefect.md +++ b/doc/administration/gitaly/praefect.md @@ -1,5 +1,5 @@ --- -stage: Create +stage: Systems group: Gitaly info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- @@ -833,6 +833,7 @@ For more information on Gitaly server configuration, see our sidekiq['enable'] = false gitlab_workhorse['enable'] = false prometheus_monitoring['enable'] = false + gitlab_kas['enable'] = false # Enable only the Gitaly service gitaly['enable'] = true diff --git a/doc/administration/gitaly/recovery.md b/doc/administration/gitaly/recovery.md index 2b5916be916..551a7ab289a 100644 --- a/doc/administration/gitaly/recovery.md +++ b/doc/administration/gitaly/recovery.md @@ -1,5 +1,5 @@ --- -stage: Create +stage: Systems group: Gitaly info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/administration/gitaly/reference.md b/doc/administration/gitaly/reference.md index d1e802111cd..91780ec5661 100644 --- a/doc/administration/gitaly/reference.md +++ b/doc/administration/gitaly/reference.md @@ -1,5 +1,5 @@ --- -stage: Create +stage: Systems group: Gitaly info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/administration/gitaly/troubleshooting.md b/doc/administration/gitaly/troubleshooting.md index a1f542ddac8..fb3a31d61b8 100644 --- a/doc/administration/gitaly/troubleshooting.md +++ b/doc/administration/gitaly/troubleshooting.md @@ -1,5 +1,5 @@ --- -stage: Create +stage: Systems group: Gitaly info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- @@ -220,6 +220,12 @@ on the Gitaly server matches the one on Gitaly client. If it doesn't match, update the secrets file on the Gitaly server to match the Gitaly client, then [reconfigure](../restart_gitlab.md#omnibus-gitlab-reconfigure). +If you've confirmed that your `gitlab-secrets.json` file is the same on all Gitaly servers and clients, +the application might be fetching this secret from a different file. Your Gitaly server's +`config.toml file` indicates the secrets file in use. +If that setting is missing, GitLab defaults to using `.gitlab_shell_secret` under +`/opt/gitlab/embedded/service/gitlab-rails/.gitlab_shell_secret`. + ### Repository pushes fail with a `deny updating a hidden ref` error Due to [a change](https://gitlab.com/gitlab-org/gitaly/-/merge_requests/3426) diff --git a/doc/administration/housekeeping.md b/doc/administration/housekeeping.md index 93409b54c0d..31b7ac9fd3e 100644 --- a/doc/administration/housekeeping.md +++ b/doc/administration/housekeeping.md @@ -6,7 +6,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Housekeeping **(FREE SELF)** -GitLab supports and automates housekeeping tasks within your current repository such as: +GitLab supports and automates housekeeping tasks in your current repository such as: - Compressing Git objects. - Removing unreachable objects. diff --git a/doc/administration/index.md b/doc/administration/index.md index 29fb65e2deb..4565c0a8e00 100644 --- a/doc/administration/index.md +++ b/doc/administration/index.md @@ -207,15 +207,14 @@ Learn how to install, configure, update, and maintain your GitLab instance. ## Troubleshooting -- [Debugging tips](troubleshooting/debug.md): Tips to debug problems when things go wrong. - [Log system](logs.md): Where to look for logs. - [Sidekiq Troubleshooting](troubleshooting/sidekiq.md): Debug when Sidekiq appears hung and is not processing jobs. - [Troubleshooting Elasticsearch](troubleshooting/elasticsearch.md) -- [Navigating GitLab via Rails console](troubleshooting/navigating_gitlab_via_rails_console.md) +- [Navigating GitLab via Rails console](operations/rails_console.md) - [GitLab application limits](instance_limits.md) - [Responding to security incidents](../security/responding_to_security_incidents.md) -### Support Team Docs +### Support Team Documentation The GitLab Support Team has collected a lot of information about troubleshooting GitLab. The following documents are used by the Support Team or by customers @@ -231,7 +230,7 @@ who are aware of the risks. - [Diagnostics tools](troubleshooting/diagnostics_tools.md) - [Linux commands](troubleshooting/linux_cheat_sheet.md) -- [Troubleshooting Kubernetes](troubleshooting/kubernetes_cheat_sheet.md) +- [Troubleshooting Kubernetes](https://docs.gitlab.com/charts/troubleshooting/kubernetes_cheat_sheet.html) - [Troubleshooting PostgreSQL](troubleshooting/postgresql.md) - [Guide to test environments](troubleshooting/test_environments.md) (for Support Engineers) - [GitLab Rails console commands](troubleshooting/gitlab_rails_cheat_sheet.md) (for Support Engineers) diff --git a/doc/administration/instance_limits.md b/doc/administration/instance_limits.md index d86414ae285..2007f5edb3b 100644 --- a/doc/administration/instance_limits.md +++ b/doc/administration/instance_limits.md @@ -7,7 +7,7 @@ type: reference # GitLab application limits **(FREE SELF)** -GitLab, like most large applications, enforces limits within certain features to maintain a +GitLab, like most large applications, enforces limits in certain features to maintain a minimum quality of performance. Allowing some features to be limitless could affect security, performance, data, or could even exhaust the allocated resources for the application. @@ -156,7 +156,7 @@ Set the limit to `0` to disable it. ### Search rate limit -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/80631) in GitLab 14.9 +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/80631) in GitLab 14.9. This setting limits global search requests. @@ -309,7 +309,7 @@ Blocked recursive webhook calls are logged in `auth.log` with the message `"Recu The [minimum wait time between pull refreshes](../user/project/repository/mirror/index.md) defaults to 300 seconds (5 minutes). For example, a pull refresh only runs once in a given 300 second period, regardless of how many times you trigger it. -This setting applies in the context of pull refreshes invoked via the [projects API](../api/projects.md#start-the-pull-mirroring-process-for-a-project), or when forcing an update by selecting the **Update now** (**{retry}**) button within **Settings > Repository > Mirroring repositories**. This setting has no effect on the automatic 30 minute interval schedule used by Sidekiq for [pull mirroring](../user/project/repository/mirror/pull.md). +This setting applies in the context of pull refreshes invoked via the [projects API](../api/projects.md#start-the-pull-mirroring-process-for-a-project), or when forcing an update by selecting **Update now** (**{retry}**) in **Settings > Repository > Mirroring repositories**. This setting has no effect on the automatic 30 minute interval schedule used by Sidekiq for [pull mirroring](../user/project/repository/mirror/pull.md). To change this limit for a self-managed installation, run the following in the [GitLab Rails console](operations/rails_console.md#starting-a-rails-console-session): @@ -342,7 +342,7 @@ and to limit memory consumption. When using offset-based pagination in the REST API, there is a limit to the maximum requested offset into the set of results. This limit is only applied to endpoints that support keyset-based pagination. More information about pagination options can be -found in the [API docs section on pagination](../api/index.md#pagination). +found in the [API documentation section on pagination](../api/index.md#pagination). To set this limit for a self-managed installation, run the following in the [GitLab Rails console](operations/rails_console.md#starting-a-rails-console-session): @@ -793,19 +793,9 @@ Plan.default.actual_limits.update!(dotenv_size: 5.kilobytes) > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/17859) in GitLab 12.5. -You can limit the number of inbound alerts for [incidents](../operations/incident_management/incidents.md) -that can be created in a period of time. The inbound [incident management](../operations/incident_management/index.md) -alert limit can help prevent overloading your incident responders by reducing the -number of alerts or duplicate issues. +This setting limits the number of inbound alert payloads over a period of time. -To set inbound incident management alert limits: - -1. On the top bar, select **Menu > Admin**. -1. On the left sidebar, select **Settings > Network**. -1. Expand General **Incident Management Limits**. -1. Select the **Enable Incident Management inbound alert limit** checkbox. -1. Optional. Input a custom value for **Maximum requests per project per rate limit period**. Default is 3600. -1. Optional. Input a custom value for **Rate limit period**. Default is 3600 seconds. +Read more about [incident management rate limits](../user/admin_area/settings/rate_limit_on_pipelines_creation.md). ### Prometheus Alert JSON payloads @@ -954,7 +944,7 @@ The maximum allowed [push size](../user/admin_area/settings/account_and_limit_se Total number of changes (branches or tags) in a single push. If changes are more than the specified limit, hooks are not executed. -More information can be found in these docs: +More information can be found in these documentations: - [Webhooks push events](../user/project/integrations/webhook_events.md#push-events) - [Project services push hooks limit](../user/project/integrations/index.md#push-hooks-limit) @@ -1050,7 +1040,7 @@ In addition to application-based limits, GitLab.com is configured to use Cloudfl ## Container Repository tag deletion limit -Container repository tags are in the Container Registry and, as such, each tag deletion will trigger network requests to the Container Registry. Because of this, we limit the number of tags that a single API call can delete to 20. +Container repository tags are in the Container Registry and, as such, each tag deletion triggers network requests to the Container Registry. Because of this, we limit the number of tags that a single API call can delete to 20. ## Project-level Secure Files API limits diff --git a/doc/administration/instance_review.md b/doc/administration/instance_review.md index f444589bdf3..d9126036a16 100644 --- a/doc/administration/instance_review.md +++ b/doc/administration/instance_review.md @@ -1,6 +1,6 @@ --- stage: Growth -group: Conversion +group: Acquisition info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/administration/integration/mailgun.md b/doc/administration/integration/mailgun.md index fbad201be7d..6b0cb0466fc 100644 --- a/doc/administration/integration/mailgun.md +++ b/doc/administration/integration/mailgun.md @@ -23,7 +23,7 @@ After completing the integration, Mailgun `temporary_failure` and `permanent_fai Before you can enable Mailgun in GitLab, set up your own Mailgun endpoints to receive the webhooks. -Using the [Mailgun webhook guide](https://www.mailgun.com/blog/a-guide-to-using-mailguns-webhooks/): +Using the [Mailgun webhook guide](https://www.mailgun.com/blog/product/a-guide-to-using-mailguns-webhooks/): 1. Add a webhook with the **Event type** set to **Permanent Failure**. 1. Enter the URL of your instance and include the `/-/mailgun/webhooks` path. diff --git a/doc/administration/integration/plantuml.md b/doc/administration/integration/plantuml.md index cdf6d48ad41..58c9e01a151 100644 --- a/doc/administration/integration/plantuml.md +++ b/doc/administration/integration/plantuml.md @@ -7,9 +7,11 @@ type: reference, howto # PlantUML and GitLab **(FREE)** -When [PlantUML](https://plantuml.com) integration is enabled and configured in -GitLab, you can create diagrams in snippets, wikis, and repositories. To set up -the integration, you must: +When the [PlantUML](https://plantuml.com) integration is enabled and configured in +GitLab, you can create diagrams in snippets, wikis, and repositories. This integration +is enabled on GitLab.com for all SaaS users and does not require any additional configuration. + +To set up the integration on a self-managed instance, you must: 1. [Configure your PlantUML server](#configure-your-plantuml-server). 1. [Configure local PlantUML access](#configure-local-plantuml-access). diff --git a/doc/administration/invalidate_markdown_cache.md b/doc/administration/invalidate_markdown_cache.md index 855910fec6a..20a77691bbc 100644 --- a/doc/administration/invalidate_markdown_cache.md +++ b/doc/administration/invalidate_markdown_cache.md @@ -5,18 +5,25 @@ info: "To determine the technical writer assigned to the Stage/Group associated type: reference --- -# Invalidate Markdown Cache **(FREE)** +# Markdown cache **(FREE)** -For performance reasons, GitLab caches the HTML version of Markdown text -in fields like comments, issue descriptions, and merge request descriptions. These -cached versions can become outdated, such as -when the `external_url` configuration option is changed. Links +For performance reasons, GitLab caches the HTML version of Markdown text in fields such as: + +- Comments. +- Issue descriptions. +- Merge request descriptions. + +These cached versions can become outdated, such as when the `external_url` configuration option is changed. Links in the cached text would still refer to the old URL. -To avoid this problem, the administrator can invalidate the existing cache by -increasing the `local_markdown_version` setting in application settings. This can -be done by changing the application settings -[through the API](../api/settings.md#change-application-settings): +## Invalidate the cache + +Pre-requisite: + +- You must be an administrator. + +To avoid problems caused by cached HTML versions, invalidate the existing cache by increasing the `local_markdown_version` +setting in application settings [using the API](../api/settings.md#change-application-settings): ```shell curl --request PUT --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/application/settings?local_markdown_version=<increased_number>" diff --git a/doc/administration/lfs/index.md b/doc/administration/lfs/index.md index ebf76cc6aec..2fcf7d2f276 100644 --- a/doc/administration/lfs/index.md +++ b/doc/administration/lfs/index.md @@ -184,7 +184,11 @@ Verify `objectstg` below (where `store=2`) has count of all LFS objects: ```shell gitlabhq_production=# SELECT count(*) AS total, sum(case when file_store = '1' then 1 else 0 end) AS filesystem, sum(case when file_store = '2' then 1 else 0 end) AS objectstg FROM lfs_objects; +``` + +**Example Output** +```shell total | filesystem | objectstg ------+------------+----------- 2409 | 0 | 2409 diff --git a/doc/administration/libravatar.md b/doc/administration/libravatar.md index 442c233a536..eac7c6f848b 100644 --- a/doc/administration/libravatar.md +++ b/doc/administration/libravatar.md @@ -12,7 +12,7 @@ GitLab by default supports the [Gravatar](https://gravatar.com) avatar service. Libravatar is another service that delivers your avatar (profile picture) to other websites. The Libravatar API is [heavily based on gravatar](https://wiki.libravatar.org/api/), so you can -easily switch to the Libravatar avatar service or even your own Libravatar +switch to the Libravatar avatar service or even your own Libravatar server. ## Configuration @@ -45,7 +45,7 @@ the URL is different in the configuration, but you must provide the same placeholders so GitLab can parse the URL correctly. For example, you host a service on `http://libravatar.example.com` and the -`plain_url` you need to supply in `gitlab.yml` is +`plain_url` you must supply in `gitlab.yml` is `http://libravatar.example.com/avatar/%{hash}?s=%{size}&d=identicon` diff --git a/doc/administration/load_balancer.md b/doc/administration/load_balancer.md index e6a1c0c25d5..988bddcfe62 100644 --- a/doc/administration/load_balancer.md +++ b/doc/administration/load_balancer.md @@ -11,8 +11,8 @@ traffic to the application servers. The specifics on which load balancer to use or the exact configuration is beyond the scope of GitLab documentation. We hope that if you're managing HA systems like GitLab you have a load balancer of choice already. Some examples including HAProxy (open-source), F5 Big-IP LTM, -and Citrix Net Scaler. This documentation outlines what ports and protocols -you need to use with GitLab. +and Citrix NetScaler. This documentation outlines what ports and protocols +to use with GitLab. ## SSL @@ -40,7 +40,7 @@ Configure your load balancers to use the 'HTTP(S)' protocol rather than 'TCP'. The load balancers is be responsible for managing SSL certificates and terminating SSL. -Since communication between the load balancers and GitLab isn't secure, +Because communication between the load balancers and GitLab isn't secure, there is some additional configuration needed. See [NGINX Proxied SSL documentation](https://docs.gitlab.com/omnibus/settings/nginx.html#supporting-proxied-ssl) for details. @@ -52,7 +52,7 @@ The load balancers is responsible for managing SSL certificates that end users see. Traffic is secure between the load balancers and NGINX in this -scenario. There is no need to add configuration for proxied SSL since the +scenario. There is no need to add configuration for proxied SSL because the connection is secure all the way. However, configuration must be added to GitLab to configure SSL certificates. See [NGINX HTTPS documentation](https://docs.gitlab.com/omnibus/settings/nginx.html#enable-https) diff --git a/doc/administration/logs.md b/doc/administration/logs.md index 5a2c1190896..671c264ed85 100644 --- a/doc/administration/logs.md +++ b/doc/administration/logs.md @@ -1063,23 +1063,23 @@ For Omnibus GitLab installations, Mattermost logs are in these locations: ## Workhorse Logs -For Omnibus GitLab installations, Workhorse logs are in `/var/log/gitlab/gitlab-workhorse/`. +For Omnibus GitLab installations, Workhorse logs are in `/var/log/gitlab/gitlab-workhorse/current`. ## PostgreSQL Logs -For Omnibus GitLab installations, PostgreSQL logs are in `/var/log/gitlab/postgresql/`. +For Omnibus GitLab installations, PostgreSQL logs are in `/var/log/gitlab/postgresql/current`. ## Prometheus Logs -For Omnibus GitLab installations, Prometheus logs are in `/var/log/gitlab/prometheus/`. +For Omnibus GitLab installations, Prometheus logs are in `/var/log/gitlab/prometheus/current`. ## Redis Logs -For Omnibus GitLab installations, Redis logs are in `/var/log/gitlab/redis/`. +For Omnibus GitLab installations, Redis logs are in `/var/log/gitlab/redis/current`. ## Alertmanager Logs -For Omnibus GitLab installations, Alertmanager logs are in `/var/log/gitlab/alertmanager/`. +For Omnibus GitLab installations, Alertmanager logs are in `/var/log/gitlab/alertmanager/current`. <!-- vale gitlab.Spelling = NO --> @@ -1091,11 +1091,11 @@ For Omnibus GitLab installations, crond logs are in `/var/log/gitlab/crond/`. ## Grafana Logs -For Omnibus GitLab installations, Grafana logs are in `/var/log/gitlab/grafana/`. +For Omnibus GitLab installations, Grafana logs are in `/var/log/gitlab/grafana/current`. ## LogRotate Logs -For Omnibus GitLab installations, `logrotate` logs are in `/var/log/gitlab/logrotate/`. +For Omnibus GitLab installations, `logrotate` logs are in `/var/log/gitlab/logrotate/current`. ## GitLab Monitor Logs @@ -1103,12 +1103,12 @@ For Omnibus GitLab installations, GitLab Monitor logs are in `/var/log/gitlab/gi ## GitLab Exporter -For Omnibus GitLab installations, GitLab Exporter logs are in `/var/log/gitlab/gitlab-exporter/`. +For Omnibus GitLab installations, GitLab Exporter logs are in `/var/log/gitlab/gitlab-exporter/current`. ## GitLab agent server For Omnibus GitLab installations, GitLab agent server logs are -in `/var/log/gitlab/gitlab-kas/`. +in `/var/log/gitlab/gitlab-kas/current`. ## Praefect Logs diff --git a/doc/administration/monitoring/performance/grafana_configuration.md b/doc/administration/monitoring/performance/grafana_configuration.md index 79612145327..3a76e2e4578 100644 --- a/doc/administration/monitoring/performance/grafana_configuration.md +++ b/doc/administration/monitoring/performance/grafana_configuration.md @@ -14,7 +14,7 @@ and Grafana allows you to query the data to display useful graphs. Omnibus GitLab can [help you install Grafana (recommended)](https://docs.gitlab.com/omnibus/settings/grafana.html) or Grafana supplies package repositories (Yum/Apt) for easy installation. -See [Grafana installation documentation](https://grafana.com/docs/grafana/latest/installation/) +See [Grafana installation documentation](https://grafana.com/docs/grafana/latest/setup-grafana/installation/) for detailed steps. Before starting Grafana for the first time, set the administration user @@ -133,8 +133,8 @@ However, you should **not** reinstate your old data _except_ under one of the fo If you require access to your old Grafana data but don't meet one of these criteria, you may consider: 1. Reinstating it temporarily. -1. [Exporting the dashboards](https://grafana.com/docs/grafana/latest/reference/export_import/#exporting-a-dashboard) you need. -1. Refreshing the data and [re-importing your dashboards](https://grafana.com/docs/grafana/latest/reference/export_import/#importing-a-dashboard). +1. [Exporting the dashboards](https://grafana.com/docs/grafana/latest/dashboards/export-import/#exporting-a-dashboard) you need. +1. Refreshing the data and [re-importing your dashboards](https://grafana.com/docs/grafana/latest/dashboards/export-import/#import-dashboard). WARNING: These actions pose a temporary vulnerability while your old Grafana data is in use. diff --git a/doc/administration/monitoring/performance/performance_bar.md b/doc/administration/monitoring/performance/performance_bar.md index aefd7a49b8b..759f485c109 100644 --- a/doc/administration/monitoring/performance/performance_bar.md +++ b/doc/administration/monitoring/performance/performance_bar.md @@ -54,7 +54,7 @@ From left to right, the performance bar displays: - [**First Contentful Paint**](https://web.dev/first-contentful-paint/): Time until something was visible to the user. Displays `NaN` if your browser does not support this feature. - - [**DomContentLoaded**](https://developers.google.com/web/fundamentals/performance/critical-rendering-path/measure-crp) Event. + - [**DomContentLoaded**](https://web.dev/critical-rendering-path-measure-crp/) Event. - **Total number of requests** the page loaded. - **Memory**: the amount of memory consumed and objects allocated during the selected request. Select it to display a window with more details. diff --git a/doc/administration/monitoring/prometheus/gitlab_metrics.md b/doc/administration/monitoring/prometheus/gitlab_metrics.md index 4f8fbd0c07e..d6575766b17 100644 --- a/doc/administration/monitoring/prometheus/gitlab_metrics.md +++ b/doc/administration/monitoring/prometheus/gitlab_metrics.md @@ -42,6 +42,8 @@ The following metrics are available: | `gitlab_ci_pipeline_builder_scoped_variables_duration` | Histogram | 14.5 | Time in seconds it takes to create the scoped variables for a CI/CD job | `gitlab_ci_pipeline_creation_duration_seconds` | Histogram | 13.0 | Time in seconds it takes to create a CI/CD pipeline | | | `gitlab_ci_pipeline_size_builds` | Histogram | 13.1 | Total number of builds within a pipeline grouped by a pipeline source | `source` | +| `gitlab_ci_runner_authentication_success_total` | Counter | 15.2 | Total number of times that runner authentication has succeeded | `type` | +| `gitlab_ci_runner_authentication_failure_total` | Counter | 15.2 | Total number of times that runner authentication has failed | `job_waiter_started_total` | Counter | 12.9 | Number of batches of jobs started where a web request is waiting for the jobs to complete | `worker` | | `job_waiter_timeouts_total` | Counter | 12.9 | Number of batches of jobs that timed out where a web request is waiting for the jobs to complete | `worker` | | `gitlab_ci_active_jobs` | Histogram | 14.2 | Count of active jobs when pipeline is created | | @@ -136,8 +138,8 @@ The following metrics are available: | `pipeline_graph_links_per_job_ratio` | Histogram | 13.9 | Ratio of links to job per graph | | | `gitlab_ci_pipeline_security_orchestration_policy_processing_duration_seconds` | Histogram | 13.12 | Time in seconds it takes to process Security Policies in CI/CD pipeline | | | `gitlab_spamcheck_request_duration_seconds` | Histogram | 13.12 | The duration for requests between Rails and the anti-spam engine | | -| `service_desk_thank_you_email` | Counter | 14.0 | Total number of email responses to new service desk emails | | -| `service_desk_new_note_email` | Counter | 14.0 | Total number of email notifications on new service desk comment | | +| `service_desk_thank_you_email` | Counter | 14.0 | Total number of email responses to new Service Desk emails | | +| `service_desk_new_note_email` | Counter | 14.0 | Total number of email notifications on new Service Desk comment | | | `email_receiver_error` | Counter | 14.1 | Total number of errors when processing incoming emails | | | `gitlab_snowplow_events_total` | Counter | 14.1 | Total number of GitLab Snowplow product intelligence events emitted | | | `gitlab_snowplow_failed_events_total` | Counter | 14.1 | Total number of GitLab Snowplow product intelligence events emission failures | | @@ -176,6 +178,7 @@ configuration option in `gitlab.yml`. These metrics are served from the | `sidekiq_jobs_queue_duration_seconds` | Histogram | 12.5 | Duration in seconds that a Sidekiq job was queued before being executed | `queue`, `boundary`, `external_dependencies`, `feature_category`, `urgency` | | `sidekiq_jobs_failed_total` | Counter | 12.2 | Sidekiq jobs failed | `queue`, `boundary`, `external_dependencies`, `feature_category`, `urgency` | | `sidekiq_jobs_retried_total` | Counter | 12.2 | Sidekiq jobs retried | `queue`, `boundary`, `external_dependencies`, `feature_category`, `urgency` | +| `sidekiq_jobs_interrupted_total` | Counter | 15.2 | Sidekiq jobs interrupted | `queue`, `boundary`, `external_dependencies`, `feature_category`, `urgency` | | `sidekiq_jobs_dead_total` | Counter | 13.7 | Sidekiq dead jobs (jobs that have run out of retries) | `queue`, `boundary`, `external_dependencies`, `feature_category`, `urgency` | | `sidekiq_redis_requests_total` | Counter | 13.1 | Redis requests during a Sidekiq job execution | `queue`, `boundary`, `external_dependencies`, `feature_category`, `job_status`, `urgency` | | `sidekiq_elasticsearch_requests_total` | Counter | 13.1 | Elasticsearch requests during a Sidekiq job execution | `queue`, `boundary`, `external_dependencies`, `feature_category`, `job_status`, `urgency` | @@ -347,6 +350,7 @@ Some basic Ruby runtime metrics are available: |:---------------------------------------- |:--------- |:----- |:----------- | | `ruby_gc_duration_seconds` | Counter | 11.1 | Time spent by Ruby in GC | | `ruby_gc_stat_...` | Gauge | 11.1 | Various metrics from [GC.stat](https://ruby-doc.org/core-2.6.5/GC.html#method-c-stat) | +| `ruby_gc_stat_ext_heap_fragmentation` | Gauge | 15.2 | Degree of Ruby heap fragmentation as live objects versus eden slots (range 0 to 1) | | `ruby_file_descriptors` | Gauge | 11.1 | File descriptors per process | | `ruby_sampler_duration_seconds` | Counter | 11.1 | Time spent collecting stats | | `ruby_process_cpu_seconds_total` | Gauge | 12.0 | Total amount of CPU time per process | diff --git a/doc/administration/monitoring/prometheus/web_exporter.md b/doc/administration/monitoring/prometheus/web_exporter.md index 4b449a1d74e..fe140b15ed2 100644 --- a/doc/administration/monitoring/prometheus/web_exporter.md +++ b/doc/administration/monitoring/prometheus/web_exporter.md @@ -51,3 +51,23 @@ To enable the dedicated server: for the changes to take effect. Metrics can now be served and scraped from `localhost:8083/metrics`. + +## Enable HTTPS + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/364771) in GitLab 15.2. + +To serve metrics via HTTPS instead of HTTP, enable TLS in the exporter settings: + +1. Edit `/etc/gitlab/gitlab.rb` to add (or find and uncomment) the following lines: + + ```ruby + puma['exporter_tls_enabled'] = true + puma['exporter_tls_cert_path'] = "/path/to/certificate.pem" + puma['exporter_tls_key_path'] = "/path/to/private-key.pem" + ``` + +1. Save the file and [reconfigure GitLab](../../restart_gitlab.md#omnibus-gitlab-reconfigure) + for the changes to take effect. + +When TLS is enabled, the same `port` and `address` will be used as described above. +The metrics server cannot serve both HTTP and HTTPS at the same time. diff --git a/doc/administration/nfs.md b/doc/administration/nfs.md index d9866a6c09f..51fa627b8d4 100644 --- a/doc/administration/nfs.md +++ b/doc/administration/nfs.md @@ -24,7 +24,7 @@ file system performance, see Starting with GitLab version 14.0, support for NFS to store Git repository data is deprecated. Technical customer support and engineering support is available for the 14.x releases. Engineering is fixing bugs and security vulnerabilities consistent with our [release and maintenance policy](../policy/maintenance.md#security-releases). -Upon the release of GitLab 15.6 technical and engineering support for using NFS to store Git repository data will be officially at end-of-life. There will be no product changes or troubleshooting provided via Engineering, Security or Paid Support channels after the release date of 15.6, regardless of your GitLab version. +Upon the release of GitLab 15.6 technical and engineering support for using NFS to store Git repository data is officially at end-of-life. There are no product changes or troubleshooting provided via Engineering, Security or Paid Support channels after the release date of 15.6, regardless of your GitLab version. Until the release of 15.6, for customers running 14.x releases, we continue to help with Git related tickets from customers running one or more Gitaly servers with its data stored on NFS. Examples may include: @@ -268,9 +268,9 @@ version of a directory. From the [Linux man page](https://linux.die.net/man/5/nfs), the important parts: -> If the nocto option is specified, the client uses a non-standard heuristic to determine when files on the server have changed. +> If the `nocto` option is specified, the client uses a non-standard heuristic to determine when files on the server have changed. > -> Using the nocto option may improve performance for read-only mounts, but should be used only if the data on the server changes only occasionally. +> Using the `nocto` option may improve performance for read-only mounts, but should be used only if the data on the server changes only occasionally. We have noticed this behavior in an issue about [refs not found after a push](https://gitlab.com/gitlab-org/gitlab/-/issues/326066), where newly added loose refs can be seen as missing on a different client with a local dentry cache, as diff --git a/doc/administration/object_storage.md b/doc/administration/object_storage.md index 7e6dc51d27c..ddeaf0280eb 100644 --- a/doc/administration/object_storage.md +++ b/doc/administration/object_storage.md @@ -76,7 +76,7 @@ because it does not require a shared folder. Consolidated object storage configuration can't be used for backups or Mattermost. See the [full table for a complete list](#storage-specific-configuration). -However, backups can be configured with [server side encryption](../raketasks/backup_restore.md#s3-encrypted-buckets) separately. +However, backups can be configured with [server side encryption](../raketasks/backup_gitlab.md#s3-encrypted-buckets) separately. Enabling consolidated object storage enables object storage for all object types. If not all buckets are specified, `sudo gitlab-ctl reconfigure` may fail with the error like: @@ -528,7 +528,7 @@ supported by consolidated configuration form, refer to the following guides: | Object storage type | Supported by consolidated configuration? | |---------------------|------------------------------------------| -| [Backups](../raketasks/backup_restore.md#uploading-backups-to-a-remote-cloud-storage) | **{dotted-circle}** No | +| [Backups](../raketasks/backup_gitlab.md#uploading-backups-to-a-remote-cloud-storage) | **{dotted-circle}** No | | [Job artifacts](job_artifacts.md#using-object-storage) including archived job logs | **{check-circle}** Yes | | [LFS objects](lfs/index.md#storing-lfs-objects-in-remote-object-storage) | **{check-circle}** Yes | | [Uploads](uploads.md#using-object-storage) | **{check-circle}** Yes | @@ -542,7 +542,7 @@ supported by consolidated configuration form, refer to the following guides: | [Pages content](pages/index.md#using-object-storage) | **{check-circle}** Yes | WARNING: -The use of [encrypted S3 buckets](#encrypted-s3-buckets) with non-consolidated configuration is not supported. +The use of [encrypted S3 buckets](#encrypted-s3-buckets) with non-consolidated configuration is not supported. You may start getting [ETag mismatch errors](#etag-mismatch) if you use it. ### Other alternatives to file system storage @@ -587,7 +587,7 @@ Helm-based installs require separate buckets to ### S3 API compatibility issues -Not all S3 providers [are fully compatible](../raketasks/backup_restore.md#other-s3-providers) +Not all S3 providers [are fully compatible](../raketasks/backup_gitlab.md#other-s3-providers) with the Fog library that GitLab uses. Symptoms include an error in `production.log`: ```plaintext diff --git a/doc/administration/operations/extra_sidekiq_processes.md b/doc/administration/operations/extra_sidekiq_processes.md index cffe682a80b..373017eefa7 100644 --- a/doc/administration/operations/extra_sidekiq_processes.md +++ b/doc/administration/operations/extra_sidekiq_processes.md @@ -9,7 +9,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w GitLab allows you to start multiple Sidekiq processes. These processes can be used to consume a dedicated set of queues. This can be used to ensure certain queues always have dedicated -workers, no matter the number of jobs that need to be processed. +workers, no matter the number of jobs to be processed. NOTE: The information in this page applies only to Omnibus GitLab. @@ -325,17 +325,16 @@ you'd use the following: The `sidekiq-cluster` command does not terminate once it has started the desired amount of Sidekiq processes. Instead, the process continues running and -forward any signals to the child processes. This makes it easy to stop all -Sidekiq processes as you simply send a signal to the `sidekiq-cluster` process, +forwards any signals to the child processes. This allows you to stop all +Sidekiq processes as you send a signal to the `sidekiq-cluster` process, instead of having to send it to the individual processes. If the `sidekiq-cluster` process crashes or receives a `SIGKILL`, the child processes terminate themselves after a few seconds. This ensures you don't end up with zombie Sidekiq processes. -All of this makes monitoring the processes fairly easy. Simply hook up -`sidekiq-cluster` to your supervisor of choice (for example, runit) and you're good to -go. +This allows you to monitor the processes by hooking up +`sidekiq-cluster` to your supervisor of choice (for example, runit). If a child process died the `sidekiq-cluster` command signals all remaining process to terminate, then terminate itself. This removes the need for diff --git a/doc/administration/operations/extra_sidekiq_routing.md b/doc/administration/operations/extra_sidekiq_routing.md index 2a14fa1d312..a6ad3e62bb7 100644 --- a/doc/administration/operations/extra_sidekiq_routing.md +++ b/doc/administration/operations/extra_sidekiq_routing.md @@ -75,7 +75,7 @@ workers. ## Worker matching query -GitLab provides a simple query syntax to match a worker based on its +GitLab provides a query syntax to match a worker based on its attributes. This query syntax is employed by both [Queue routing rules](#queue-routing-rules) and [Queue selector](extra_sidekiq_processes.md#queue-selector). A query includes two @@ -102,12 +102,8 @@ based on a subset of worker attributes: quickly. Can be `high`, `low`, or `throttled`. For example, the `authorized_projects` queue is used to refresh user permissions, and is `high` urgency. -- `worker_name` - the worker name. The other attributes are typically more useful as - they are more general, but this is available in case a particular worker needs - to be selected. -- `name` - the queue name generated from the worker name. The other attributes - are typically more useful as they are more general, but this is available in - case a particular queue needs to be selected. Because this is generated from +- `worker_name` - the worker name. Use this attribute to select a specific worker. +- `name` - the queue name generated from the worker name. Use this attribute to select a specific queue. Because this is generated from the worker name, it does not change based on the result of other routing rules. - `resource_boundary` - if the queue is bound by `cpu`, `memory`, or @@ -137,7 +133,7 @@ GitLab Enterprise Edition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee to lowest precedence: - `|` - the logical `OR` operator. For example, `query_a|query_b` (where `query_a` - and `query_b` are queries made up of the other operators here) will include + and `query_b` are queries made up of the other operators here) includes queues that match either query. - `&` - the logical `AND` operator. For example, `query_a&query_b` (where `query_a` and `query_b` are queries made up of the other operators here) will @@ -161,7 +157,7 @@ entire queue group selects all queues. ### Migration -After the Sidekiq routing rules are changed, administrators need to take care +After the Sidekiq routing rules are changed, administrators must take care with the migration to avoid losing jobs entirely, especially in a system with long queues of jobs. The migration can be done by following the migration steps mentioned in [Sidekiq job @@ -181,7 +177,7 @@ sidekiq['routing_rules'] = [ ] ``` -These queues will also need to be included in at least one [Sidekiq +These queues must also be included in at least one [Sidekiq queue group](extra_sidekiq_processes.md#start-multiple-processes). The following table shows the workers that should have their own queue: diff --git a/doc/administration/operations/filesystem_benchmarking.md b/doc/administration/operations/filesystem_benchmarking.md index 9403634d905..a5c4795efea 100644 --- a/doc/administration/operations/filesystem_benchmarking.md +++ b/doc/administration/operations/filesystem_benchmarking.md @@ -36,7 +36,7 @@ fio --randrepeat=1 --ioengine=libaio --direct=1 --gtod_reduce=1 --name=test --bs ``` This creates a 4GB file in `/path/to/git-data/testfile`. It performs -4KB reads and writes using a 75%/25% split within the file, with 64 +4KB reads and writes using a 75%/25% split in the file, with 64 operations running at a time. Be sure to delete the file after the test completes. @@ -72,7 +72,7 @@ operations per second. ### Simple benchmarking NOTE: -This test is naive but may be useful if `fio` is not +This test is naive but can be used if `fio` is not available on the system. It's possible to receive good results on this test but still have poor performance due to read speed and various other factors. @@ -83,7 +83,7 @@ executed, and then reads the same 1,000 files. 1. Change into the root of the appropriate [repository storage path](../repository_storage_paths.md). -1. Create a temporary directory for the test so it's easy to remove the files later: +1. Create a temporary directory for the test so it can be removed later: ```shell mkdir test; cd test diff --git a/doc/administration/operations/index.md b/doc/administration/operations/index.md index fe531407eb2..41a9a0e6d67 100644 --- a/doc/administration/operations/index.md +++ b/doc/administration/operations/index.md @@ -15,7 +15,7 @@ Keep your GitLab instance up and running smoothly. by GitLab to another file system or another server. - [Sidekiq MemoryKiller](sidekiq_memory_killer.md): Configure Sidekiq MemoryKiller to restart Sidekiq. -- [Multiple Sidekiq processes](extra_sidekiq_processes.md): Configure multiple Sidekiq processes to ensure certain queues always have dedicated workers, no matter the number of jobs that need to be processed. **(FREE SELF)** +- [Multiple Sidekiq processes](extra_sidekiq_processes.md): Configure multiple Sidekiq processes to ensure certain queues always have dedicated workers, no matter the number of jobs that must be processed. **(FREE SELF)** - [Sidekiq routing rules](extra_sidekiq_routing.md): Configure the routing rules to route a job from a worker to a desirable queue. **(FREE SELF)** - [Puma](puma.md): Understand Puma and puma-worker-killer. - Speed up SSH operations by [Authorizing SSH users via a fast, @@ -24,10 +24,10 @@ Keep your GitLab instance up and running smoothly. of SSH certificates](ssh_certificates.md). - [File System Performance Benchmarking](filesystem_benchmarking.md): File system performance can have a big impact on GitLab performance, especially for actions - that read or write Git repositories. This information will help benchmark + that read or write Git repositories. This information helps benchmark file system performance against known good and bad real-world systems. - [The Rails Console](rails_console.md): Provides a way to interact with your GitLab instance from the command line. Used for troubleshooting a problem or retrieving some data that can only be done through direct access to GitLab. - [ChatOps Scripts](https://gitlab.com/gitlab-com/chatops): The GitLab.com Infrastructure team uses this repository to house common ChatOps scripts they use to troubleshoot and maintain the production instance of GitLab.com. - These scripts are likely useful to administrators of GitLab instances of all sizes. + These scripts can be used by administrators of GitLab instances of all sizes. diff --git a/doc/administration/operations/moving_repositories.md b/doc/administration/operations/moving_repositories.md index 1459b707e5c..4228b792fdc 100644 --- a/doc/administration/operations/moving_repositories.md +++ b/doc/administration/operations/moving_repositories.md @@ -1,8 +1,7 @@ --- -stage: Create +stage: Systems group: Gitaly info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments -type: reference --- # Moving repositories managed by GitLab **(FREE SELF)** @@ -189,9 +188,9 @@ should be used. Git repositories are accessed, managed, and stored on GitLab ser can result from directly accessing and copying Gitaly's files using tools like `rsync`. - From GitLab 13.3, backup performance can be improved by - [processing multiple repositories concurrently](../../raketasks/backup_restore.md#back-up-git-repositories-concurrently). + [processing multiple repositories concurrently](../../raketasks/backup_gitlab.md#back-up-git-repositories-concurrently). - Backups can be created of just the repositories using the - [skip feature](../../raketasks/backup_restore.md#excluding-specific-directories-from-the-backup). + [skip feature](../../raketasks/backup_gitlab.md#excluding-specific-directories-from-the-backup). No other method works for Gitaly Cluster targets. @@ -385,6 +384,6 @@ See the following for information on troubleshooting repository moves. ### Repository move fails for archived projects Because of a [known issue](https://gitlab.com/gitlab-org/gitlab/-/issues/363670), -[archived projects](../../user/project/settings/index.md#advanced-settings) fail to move even though the data is cloned +[archived projects](../../user/project/settings/index.md#advanced-project-settings) fail to move even though the data is cloned by Gitaly. Make sure archived projects are -[unarchived](../../user/project/settings/index.md#unarchiving-a-project) before initiating a move. +[unarchived](../../user/project/settings/index.md#unarchive-a-project) before initiating a move. diff --git a/doc/administration/operations/puma.md b/doc/administration/operations/puma.md index 0025ca109e5..bc95ee626ce 100644 --- a/doc/administration/operations/puma.md +++ b/doc/administration/operations/puma.md @@ -111,7 +111,7 @@ To change the worker timeout to 600 seconds: WARNING: This is an experimental [Alpha feature](../../policy/alpha-beta-support.md#alpha-features) and subject to change without notice. The feature is not ready for production use. If you want to use this feature, we recommend testing -with non-production data first. See the [known issues](#puma-single-mode-known-issues) +outside of production first. See the [known issues](#puma-single-mode-known-issues) for additional details. In a memory-constrained environment with less than 4GB of RAM available, consider disabling Puma @@ -137,6 +137,10 @@ For details on Puma worker and thread settings, see the [Puma requirements](../. The downside of running Puma in this configuration is the reduced throughput, which can be considered a fair tradeoff in a memory-constrained environment. +Remember to have sufficient swap available to avoid out of memory (OOM) +conditions. View the [Memory requirements](../../install/requirements.md#memory) +for details. + ### Puma single mode known issues When running Puma in single mode, some features are not supported: @@ -279,6 +283,124 @@ To switch from Unicorn to Puma: 1. Optional. For multi-node deployments, configure the load balancer to use the [readiness check](../load_balancer.md#readiness-check). +## Troubleshooting Puma + +### 502 Gateway Timeout after Puma spins at 100% CPU + +This error occurs when the Web server times out (default: 60 s) after not +hearing back from the Puma worker. If the CPU spins to 100% while this is in +progress, there may be something taking longer than it should. + +To fix this issue, we first need to figure out what is happening. The +following tips are only recommended if you do not mind users being affected by +downtime. Otherwise, skip to the next section. + +1. Load the problematic URL +1. Run `sudo gdb -p <PID>` to attach to the Puma process. +1. In the GDB window, type: + + ```plaintext + call (void) rb_backtrace() + ``` + +1. This forces the process to generate a Ruby backtrace. Check + `/var/log/gitlab/puma/puma_stderr.log` for the backtrace. For example, you may see: + + ```plaintext + from /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/metrics/sampler.rb:33:in `block in start' + from /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/metrics/sampler.rb:33:in `loop' + from /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/metrics/sampler.rb:36:in `block (2 levels) in start' + from /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/metrics/sampler.rb:44:in `sample' + from /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/metrics/sampler.rb:68:in `sample_objects' + from /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/metrics/sampler.rb:68:in `each_with_object' + from /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/metrics/sampler.rb:68:in `each' + from /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/metrics/sampler.rb:69:in `block in sample_objects' + from /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/metrics/sampler.rb:69:in `name' + ``` + +1. To see the current threads, run: + + ```plaintext + thread apply all bt + ``` + +1. Once you're done debugging with `gdb`, be sure to detach from the process and exit: + + ```plaintext + detach + exit + ``` + +GDB reports an error if the Puma process terminates before you can run these commands. +To buy more time, you can always raise the +Puma worker timeout. For omnibus users, you can edit `/etc/gitlab/gitlab.rb` and +increase it from 60 seconds to 600: + +```ruby +gitlab_rails['env'] = { + 'GITLAB_RAILS_RACK_TIMEOUT' => 600 +} +``` + +For source installations, set the environment variable. +Refer to [Puma Worker timeout](../operations/puma.md#change-the-worker-timeout). + +[Reconfigure](../restart_gitlab.md#omnibus-gitlab-reconfigure) GitLab for the changes to take effect. + +#### Troubleshooting without affecting other users + +The previous section attached to a running Puma process, which may have +undesirable effects on users trying to access GitLab during this time. If you +are concerned about affecting others during a production system, you can run a +separate Rails process to debug the issue: + +1. Log in to your GitLab account. +1. Copy the URL that is causing problems (for example, `https://gitlab.com/ABC`). +1. Create a Personal Access Token for your user (User Settings -> Access Tokens). +1. Bring up the [GitLab Rails console.](../operations/rails_console.md#starting-a-rails-console-session) +1. At the Rails console, run: + + ```ruby + app.get '<URL FROM STEP 2>/?private_token=<TOKEN FROM STEP 3>' + ``` + + For example: + + ```ruby + app.get 'https://gitlab.com/gitlab-org/gitlab-foss/-/issues/1?private_token=123456' + ``` + +1. In a new window, run `top`. It should show this Ruby process using 100% CPU. Write down the PID. +1. Follow step 2 from the previous section on using GDB. + +### GitLab: API is not accessible + +This often occurs when GitLab Shell attempts to request authorization via the +[internal API](../../development/internal_api/index.md) (for example, `http://localhost:8080/api/v4/internal/allowed`), and +something in the check fails. There are many reasons why this may happen: + +1. Timeout connecting to a database (for example, PostgreSQL or Redis) +1. Error in Git hooks or push rules +1. Error accessing the repository (for example, stale NFS handles) + +To diagnose this problem, try to reproduce the problem and then see if there +is a Puma worker that is spinning via `top`. Try to use the `gdb` +techniques above. In addition, using `strace` may help isolate issues: + +```shell +strace -ttTfyyy -s 1024 -p <PID of puma worker> -o /tmp/puma.txt +``` + +If you cannot isolate which Puma worker is the issue, try to run `strace` +on all the Puma workers to see where the +[`/internal/allowed`](../../development/internal_api/index.md) endpoint gets stuck: + +```shell +ps auwx | grep puma | awk '{ print " -p " $2}' | xargs strace -ttTfyyy -s 1024 -o /tmp/puma.txt +``` + +The output in `/tmp/puma.txt` may help diagnose the root cause. + ## Related topics - [Use a dedicated metrics server to export web metrics](../monitoring/prometheus/puma_exporter.md) diff --git a/doc/administration/operations/rails_console.md b/doc/administration/operations/rails_console.md index df039ee734f..660a99faaf8 100644 --- a/doc/administration/operations/rails_console.md +++ b/doc/administration/operations/rails_console.md @@ -6,8 +6,10 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Rails console **(FREE SELF)** +At the heart of GitLab is a web application [built using the Ruby on Rails +framework](https://about.gitlab.com/blog/2018/10/29/why-we-use-rails-to-build-gitlab/). The [Rails console](https://guides.rubyonrails.org/command_line.html#rails-console). -provides a way to interact with your GitLab instance from the command line. +provides a way to interact with your GitLab instance from the command line, and also grants access to the amazing tools built right into Rails. WARNING: The Rails console interacts directly with GitLab. In many cases, @@ -17,7 +19,9 @@ with no consequences, you are strongly advised to do so in a test environment. The Rails console is for GitLab system administrators who are troubleshooting a problem or need to retrieve some data that can only be done through direct -access of the GitLab application. +access of the GitLab application. Basic knowledge of Ruby is needed (try [this +30-minute tutorial](https://try.ruby-lang.org/) for a quick introduction). +Rails experience is useful but not required. ## Starting a Rails console session @@ -35,10 +39,39 @@ sudo -u git -H bundle exec rails console -e production **For Kubernetes deployments** -The console is in the toolbox pod. Refer to our [Kubernetes cheat sheet](../troubleshooting/kubernetes_cheat_sheet.md#gitlab-specific-kubernetes-information) for details. +The console is in the toolbox pod. Refer to our [Kubernetes cheat sheet](https://docs.gitlab.com/charts/troubleshooting/kubernetes_cheat_sheet.html#gitlab-specific-kubernetes-information) for details. To exit the console, type: `quit`. +## Enable Active Record logging + +You can enable output of Active Record debug logging in the Rails console +session by running: + +```ruby +ActiveRecord::Base.logger = Logger.new($stdout) +``` + +This shows information about database queries triggered by any Ruby code +you may run in the console. To turn off logging again, run: + +```ruby +ActiveRecord::Base.logger = nil +``` + +## Disable database statement timeout + +You can disable the PostgreSQL statement timeout for the current Rails console +session by running: + +```ruby +ActiveRecord::Base.connection.execute('SET statement_timeout TO 0') +``` + +This change only affects the current Rails console session and is +not persisted in the GitLab production environment or in the next Rails +console session. + ## Output Rails console session history Enter the following command on the rails console to display @@ -168,3 +201,435 @@ sudo chown -R git:git /scripts sudo chmod 700 /scripts sudo gitlab-rails runner /scripts/helloworld.rb ``` + +## Active Record objects + +### Looking up database-persisted objects + +Under the hood, Rails uses [Active Record](https://guides.rubyonrails.org/active_record_basics.html), +an object-relational mapping system, to read, write, and map application objects +to the PostgreSQL database. These mappings are handled by Active Record models, +which are Ruby classes defined in a Rails app. For GitLab, the model classes +can be found at `/opt/gitlab/embedded/service/gitlab-rails/app/models`. + +Let's enable debug logging for Active Record so we can see the underlying +database queries made: + +```ruby +ActiveRecord::Base.logger = Logger.new($stdout) +``` + +Now, let's try retrieving a user from the database: + +```ruby +user = User.find(1) +``` + +Which would return: + +```ruby +D, [2020-03-05T16:46:25.571238 #910] DEBUG -- : User Load (1.8ms) SELECT "users".* FROM "users" WHERE "users"."id" = 1 LIMIT 1 +=> #<User id:1 @root> +``` + +We can see that we've queried the `users` table in the database for a row whose +`id` column has the value `1`, and Active Record has translated that database +record into a Ruby object that we can interact with. Try some of the following: + +- `user.username` +- `user.created_at` +- `user.admin` + +By convention, column names are directly translated into Ruby object attributes, +so you should be able to do `user.<column_name>` to view the attribute's value. + +Also by convention, Active Record class names (singular and in camel case) map +directly onto table names (plural and in snake case) and vice versa. For example, +the `users` table maps to the `User` class, while the `application_settings` +table maps to the `ApplicationSetting` class. + +You can find a list of tables and column names in the Rails database schema, +available at `/opt/gitlab/embedded/service/gitlab-rails/db/schema.rb`. + +You can also look up an object from the database by attribute name: + +```ruby +user = User.find_by(username: 'root') +``` + +Which would return: + +```ruby +D, [2020-03-05T17:03:24.696493 #910] DEBUG -- : User Load (2.1ms) SELECT "users".* FROM "users" WHERE "users"."username" = 'root' LIMIT 1 +=> #<User id:1 @root> +``` + +Give the following a try: + +- `User.find_by(email: 'admin@example.com')` +- `User.where.not(admin: true)` +- `User.where('created_at < ?', 7.days.ago)` + +Did you notice that the last two commands returned an `ActiveRecord::Relation` +object that appeared to contain multiple `User` objects? + +Up to now, we've been using `.find` or `.find_by`, which are designed to return +only a single object (notice the `LIMIT 1` in the generated SQL query?). +`.where` is used when it is desirable to get a collection of objects. + +Let's get a collection of non-administrator users and see what we can do with it: + +```ruby +users = User.where.not(admin: true) +``` + +Which would return: + +```ruby +D, [2020-03-05T17:11:16.845387 #910] DEBUG -- : User Load (2.8ms) SELECT "users".* FROM "users" WHERE "users"."admin" != TRUE LIMIT 11 +=> #<ActiveRecord::Relation [#<User id:3 @support-bot>, #<User id:7 @alert-bot>, #<User id:5 @carrie>, #<User id:4 @bernice>, #<User id:2 @anne>]> +``` + +Now, try the following: + +- `users.count` +- `users.order(created_at: :desc)` +- `users.where(username: 'support-bot')` + +In the last command, we see that we can chain `.where` statements to generate +more complex queries. Notice also that while the collection returned contains +only a single object, we cannot directly interact with it: + +```ruby +users.where(username: 'support-bot').username +``` + +Which would return: + +```ruby +Traceback (most recent call last): + 1: from (irb):37 +D, [2020-03-05T17:18:25.637607 #910] DEBUG -- : User Load (1.6ms) SELECT "users".* FROM "users" WHERE "users"."admin" != TRUE AND "users"."username" = 'support-bot' LIMIT 11 +NoMethodError (undefined method `username' for #<ActiveRecord::Relation [#<User id:3 @support-bot>]>) +Did you mean? by_username +``` + +Let's retrieve the single object from the collection by using the `.first` +method to get the first item in the collection: + +```ruby +users.where(username: 'support-bot').first.username +``` + +We now get the result we wanted: + +```ruby +D, [2020-03-05T17:18:30.406047 #910] DEBUG -- : User Load (2.6ms) SELECT "users".* FROM "users" WHERE "users"."admin" != TRUE AND "users"."username" = 'support-bot' ORDER BY "users"."id" ASC LIMIT 1 +=> "support-bot" +``` + +For more on different ways to retrieve data from the database using Active +Record, please see the [Active Record Query Interface documentation](https://guides.rubyonrails.org/active_record_querying.html). + +### Modifying Active Record objects + +In the previous section, we learned about retrieving database records using +Active Record. Now, let's learn how to write changes to the database. + +First, let's retrieve the `root` user: + +```ruby +user = User.find_by(username: 'root') +``` + +Next, let's try updating the user's password: + +```ruby +user.password = 'password' +user.save +``` + +Which would return: + +```ruby +Enqueued ActionMailer::MailDeliveryJob (Job ID: 05915c4e-c849-4e14-80bb-696d5ae22065) to Sidekiq(mailers) with arguments: "DeviseMailer", "password_change", "deliver_now", #<GlobalID:0x00007f42d8ccebe8 @uri=#<URI::GID gid://gitlab/User/1>> +=> true +``` + +Here, we see that the `.save` command returned `true`, indicating that the +password change was successfully saved to the database. + +We also see that the save operation triggered some other action -- in this case +a background job to deliver an email notification. This is an example of an +[Active Record callback](https://guides.rubyonrails.org/active_record_callbacks.html) +-- code which is designated to run in response to events in the Active Record +object life cycle. This is also why using the Rails console is preferred when +direct changes to data is necessary as changes made via direct database queries +do not trigger these callbacks. + +It's also possible to update attributes in a single line: + +```ruby +user.update(password: 'password') +``` + +Or update multiple attributes at once: + +```ruby +user.update(password: 'password', email: 'hunter2@example.com') +``` + +Now, let's try something different: + +```ruby +# Retrieve the object again so we get its latest state +user = User.find_by(username: 'root') +user.password = 'password' +user.password_confirmation = 'hunter2' +user.save +``` + +This returns `false`, indicating that the changes we made were not saved to the +database. You can probably guess why, but let's find out for sure: + +```ruby +user.save! +``` + +This should return: + +```ruby +Traceback (most recent call last): + 1: from (irb):64 +ActiveRecord::RecordInvalid (Validation failed: Password confirmation doesn't match Password) +``` + +Aha! We've tripped an [Active Record Validation](https://guides.rubyonrails.org/active_record_validations.html). +Validations are business logic put in place at the application-level to prevent +unwanted data from being saved to the database and in most cases come with +helpful messages letting you know how to fix the problem inputs. + +We can also add the bang (Ruby speak for `!`) to `.update`: + +```ruby +user.update!(password: 'password', password_confirmation: 'hunter2') +``` + +In Ruby, method names ending with `!` are commonly known as "bang methods". By +convention, the bang indicates that the method directly modifies the object it +is acting on, as opposed to returning the transformed result and leaving the +underlying object untouched. For Active Record methods that write to the +database, bang methods also serve an additional function: they raise an +explicit exception whenever an error occurs, instead of just returning `false`. + +We can also skip validations entirely: + +```ruby +# Retrieve the object again so we get its latest state +user = User.find_by(username: 'root') +user.password = 'password' +user.password_confirmation = 'hunter2' +user.save!(validate: false) +``` + +This is not recommended, as validations are usually put in place to ensure the +integrity and consistency of user-provided data. + +A validation error prevents the entire object from being saved to +the database. You can see a little of this in the section below. If you're getting +a mysterious red banner in the GitLab UI when submitting a form, this can often +be the fastest way to get to the root of the problem. + +### Interacting with Active Record objects + +At the end of the day, Active Record objects are just normal Ruby objects. As +such, we can define methods on them which perform arbitrary actions. + +For example, GitLab developers have added some methods which help with +two-factor authentication: + +```ruby +def disable_two_factor! + transaction do + update( + otp_required_for_login: false, + encrypted_otp_secret: nil, + encrypted_otp_secret_iv: nil, + encrypted_otp_secret_salt: nil, + otp_grace_period_started_at: nil, + otp_backup_codes: nil + ) + self.u2f_registrations.destroy_all # rubocop: disable DestroyAll + end +end + +def two_factor_enabled? + two_factor_otp_enabled? || two_factor_u2f_enabled? +end +``` + +(See: `/opt/gitlab/embedded/service/gitlab-rails/app/models/user.rb`) + +We can then use these methods on any user object: + +```ruby +user = User.find_by(username: 'root') +user.two_factor_enabled? +user.disable_two_factor! +``` + +Some methods are defined by gems, or Ruby software packages, which GitLab uses. +For example, the [StateMachines](https://github.com/state-machines/state_machines-activerecord) +gem which GitLab uses to manage user state: + +```ruby +state_machine :state, initial: :active do + event :block do + + ... + + event :activate do + + ... + +end +``` + +Give it a try: + +```ruby +user = User.find_by(username: 'root') +user.state +user.block +user.state +user.activate +user.state +``` + +Earlier, we mentioned that a validation error prevents the entire object +from being saved to the database. Let's see how this can have unexpected +interactions: + +```ruby +user.password = 'password' +user.password_confirmation = 'hunter2' +user.block +``` + +We get `false` returned! Let's find out what happened by adding a bang as we did +earlier: + +```ruby +user.block! +``` + +Which would return: + +```ruby +Traceback (most recent call last): + 1: from (irb):87 +StateMachines::InvalidTransition (Cannot transition state via :block from :active (Reason(s): Password confirmation doesn't match Password)) +``` + +We see that a validation error from what feels like a completely separate +attribute comes back to haunt us when we try to update the user in any way. + +In practical terms, we sometimes see this happen with GitLab administration settings -- +validations are sometimes added or changed in a GitLab update, resulting in +previously saved settings now failing validation. Because you can only update +a subset of settings at once through the UI, in this case the only way to get +back to a good state is direct manipulation via Rails console. + +### Commonly used Active Record models and how to look up objects + +**Get a user by primary email address or username:** + +```ruby +User.find_by(email: 'admin@example.com') +User.find_by(username: 'root') +``` + +**Get a user by primary OR secondary email address:** + +```ruby +User.find_by_any_email('user@example.com') +``` + +The `find_by_any_email` method is a custom method added by GitLab developers rather +than a Rails-provided default method. + +**Get a collection of administrator users:** + +```ruby +User.admins +``` + +`admins` is a [scope convenience method](https://guides.rubyonrails.org/active_record_querying.html#scopes) +which does `where(admin: true)` under the hood. + +**Get a project by its path:** + +```ruby +Project.find_by_full_path('group/subgroup/project') +``` + +`find_by_full_path` is a custom method added by GitLab developers rather +than a Rails-provided default method. + +**Get a project's issue or merge request by its numeric ID:** + +```ruby +project = Project.find_by_full_path('group/subgroup/project') +project.issues.find_by(iid: 42) +project.merge_requests.find_by(iid: 42) +``` + +`iid` means "internal ID" and is how we keep issue and merge request IDs +scoped to each GitLab project. + +**Get a group by its path:** + +```ruby +Group.find_by_full_path('group/subgroup') +``` + +**Get a group's related groups:** + +```ruby +group = Group.find_by_full_path('group/subgroup') + +# Get a group's parent group +group.parent + +# Get a group's child groups +group.children +``` + +**Get a group's projects:** + +```ruby +group = Group.find_by_full_path('group/subgroup') + +# Get group's immediate child projects +group.projects + +# Get group's child projects, including those in subgroups +group.all_projects +``` + +**Get CI pipeline or builds:** + +```ruby +Ci::Pipeline.find(4151) +Ci::Build.find(66124) +``` + +The pipeline and job ID numbers increment globally across your GitLab +instance, so there's no requirement to use an internal ID attribute to look them up, +unlike with issues or merge requests. + +**Get the current application settings object:** + +```ruby +ApplicationSetting.current +``` diff --git a/doc/administration/operations/ssh_certificates.md b/doc/administration/operations/ssh_certificates.md index 3141312612e..a777f5484fd 100644 --- a/doc/administration/operations/ssh_certificates.md +++ b/doc/administration/operations/ssh_certificates.md @@ -96,7 +96,7 @@ Match User git AuthorizedPrincipalsCommand /opt/gitlab/embedded/service/gitlab-shell/bin/gitlab-shell-authorized-principals-check %i sshUsers ``` -This command will emit output that looks something like: +This command emits output that looks something like: ```shell command="/opt/gitlab/embedded/service/gitlab-shell/bin/gitlab-shell username-{KEY_ID}",no-port-forwarding,no-X11-forwarding,no-agent-forwarding,no-pty {PRINCIPAL} diff --git a/doc/administration/package_information/deprecation_policy.md b/doc/administration/package_information/deprecation_policy.md index 420690866e5..422339c014c 100644 --- a/doc/administration/package_information/deprecation_policy.md +++ b/doc/administration/package_information/deprecation_policy.md @@ -41,7 +41,7 @@ We can differentiate two different types of configuration: installation usable (like a change in default project/group settings, or miscommunication with other components) -We also need to differentiate deprecation and removal procedure. +We must also differentiate deprecation and removal procedure. #### Deprecating configuration @@ -78,7 +78,7 @@ The final comment in the issue **has to have**: 1. Text snippet for the release blog post section 1. Documentation MR ( or snippet ) for introducing the change -1. Draft MR removing the configuration OR details on what needs to be done. See [Adding deprecation messages](https://docs.gitlab.com/omnibus/development/adding-deprecation-messages.html) for more on this +1. Draft MR removing the configuration or details on what must be done. See [Adding deprecation messages](https://docs.gitlab.com/omnibus/development/adding-deprecation-messages.html) for more on this ## Example diff --git a/doc/administration/package_information/postgresql_versions.md b/doc/administration/package_information/postgresql_versions.md index 8aab4121a0b..afdc63c4851 100644 --- a/doc/administration/package_information/postgresql_versions.md +++ b/doc/administration/package_information/postgresql_versions.md @@ -29,18 +29,18 @@ Read more about update policies and warnings in the PostgreSQL | GitLab version | PostgreSQL versions | Default version for fresh installs | Default version for upgrades | Notes | | -------------- | --------------------- | ---------------------------------- | ---------------------------- | ----- | -| 15.0 | 12.10, 13.6 | 13.6 | 12.10 | For upgrades, users can manually upgrade to 13.6 following the [upgrade docs](https://docs.gitlab.com/omnibus/settings/database.html#gitlab-150-and-later). | +| 15.0 | 12.10, 13.6 | 13.6 | 12.10 | For upgrades, users can manually upgrade to 13.6 following the [upgrade documentation](https://docs.gitlab.com/omnibus/settings/database.html#gitlab-150-and-later). | | 14.1 | 12.7, 13.3 | 12.7 | 12.7 | PostgreSQL 13 available for fresh installations if not using [Geo](../geo/index.md#requirements-for-running-geo) or [Patroni](../postgresql/index.md#postgresql-replication-and-failover-with-omnibus-gitlab). -| 14.0 | 12.7 | 12.7 | 12.7 | HA installations with repmgr are no longer supported and will be prevented from upgrading to Omnibus GitLab 14.0 | +| 14.0 | 12.7 | 12.7 | 12.7 | HA installations with repmgr are no longer supported and are prevented from upgrading to Omnibus GitLab 14.0 | | 13.8 | 11.9, 12.4 | 12.4 | 12.4 | Package upgrades automatically performed PostgreSQL upgrade for nodes that are not part of a Geo or HA cluster.). | -| 13.7 | 11.9, 12.4 | 12.4 | 11.9 | For upgrades users can manually upgrade to 12.4 following the [upgrade docs](https://docs.gitlab.com/omnibus/settings/database.html#gitlab-133-and-later). | +| 13.7 | 11.9, 12.4 | 12.4 | 11.9 | For upgrades users can manually upgrade to 12.4 following the [upgrade documentation](https://docs.gitlab.com/omnibus/settings/database.html#gitlab-133-and-later). | | 13.4 | 11.9, 12.4 | 11.9 | 11.9 | Package upgrades aborted if users not running PostgreSQL 11 already | | 13.3 | 11.7, 12.3 | 11.7 | 11.7 | Package upgrades aborted if users not running PostgreSQL 11 already | | 13.0 | 11.7 | 11.7 | 11.7 | Package upgrades aborted if users not running PostgreSQL 11 already | | 12.10 | 9.6.17, 10.12, and 11.7 | 11.7 | 11.7 | Package upgrades automatically performed PostgreSQL upgrade for nodes that are not part of a Geo or repmgr cluster. | -| 12.8 | 9.6.17, 10.12, and 11.7 | 10.12 | 10.12 | Users can manually upgrade to 11.7 following the upgrade docs. | +| 12.8 | 9.6.17, 10.12, and 11.7 | 10.12 | 10.12 | Users can manually upgrade to 11.7 following the upgrade documentation. | | 12.0 | 9.6.11 and 10.7 | 10.7 | 10.7 | Package upgrades automatically performed PostgreSQL upgrade. | -| 11.11 | 9.6.11 and 10.7 | 9.6.11 | 9.6.11 | Users can manually upgrade to 10.7 following the upgrade docs. | +| 11.11 | 9.6.11 and 10.7 | 9.6.11 | 9.6.11 | Users can manually upgrade to 10.7 following the upgrade documentation. | | 10.0 | 9.6.3 | 9.6.3 | 9.6.3 | Package upgrades aborted if users still on 9.2. | | 9.0 | 9.2.18 and 9.6.1 | 9.6.1 | 9.6.1 | Package upgrades automatically performed PostgreSQL upgrade. | -| 8.14 | 9.2.18 and 9.6.1 | 9.2.18 | 9.2.18 | Users can manually upgrade to 9.6 following the upgrade docs. | +| 8.14 | 9.2.18 and 9.6.1 | 9.2.18 | 9.2.18 | Users can manually upgrade to 9.6 following the upgrade documentation. | diff --git a/doc/administration/package_information/supported_os.md b/doc/administration/package_information/supported_os.md index f2838cd779b..4f31981a05c 100644 --- a/doc/administration/package_information/supported_os.md +++ b/doc/administration/package_information/supported_os.md @@ -15,28 +15,27 @@ page](https://about.gitlab.com/install/). The following lists the currently supported OSs and their possible EOL dates. -| OS Version | First supported GitLab version | Arch | Install Docs | OS EOL | Details | +| OS Version | First supported GitLab version | Arch | Install Documentation | OS EOL | Details | | ------------------------------------------------------------ | ------------------------------ | --------------- | :----------------------------------------------------------: | ---------- | ------------------------------------------------------------ | -| AlmaLinux 8 | GitLab CE / GitLab EE 14.5.0 | x86_64, aarch64 | [AlmaLinux Install Docs](https://about.gitlab.com/install/#almalinux-8) | 2029 | <https://almalinux.org/> | -| CentOS 7 | GitLab CE / GitLab EE 7.10.0 | x86_64 | [CentOS Install Docs](https://about.gitlab.com/install/#centos-7) | June 2024 | <https://wiki.centos.org/About/Product> | -| Debian 9 | GitLab CE / GitLab EE 9.3.0 | amd64 | [Debian Install Docs](https://about.gitlab.com/install/#debian) | 2022 | <https://wiki.debian.org/LTS> | -| Debian 10 | GitLab CE / GitLab EE 12.2.0 | amd64, arm64 | [Debian Install Docs](https://about.gitlab.com/install/#debian) | 2024 | <https://wiki.debian.org/LTS> | -| Debian 11 | GitLab CE / GitLab EE 14.6.0 | amd64, arm64 | [Debian Install Docs](https://about.gitlab.com/install/#debian) | 2026 | <https://wiki.debian.org/LTS> | -| OpenSUSE 15.3 | GitLab CE / GitLab EE 14.5.0 | x86_64, aarch64 | [OpenSUSE Install Docs](https://about.gitlab.com/install/#opensuse-leap-15-3) | Nov 2022 | <https://en.opensuse.org/Lifetime> | -| RHEL 8 | GitLab CE / GitLab EE 12.8.1 | x86_64, arm64 | [Use CentOS Install Docs](https://about.gitlab.com/install/#centos-7) | May 2024 | [RHEL Details](https://access.redhat.com/support/policy/updates/errata/#Life_Cycle_Dates) | -| SLES 12 | GitLab EE 9.0.0 | x86_64 | [Use OpenSUSE Install Docs](https://about.gitlab.com/install/#opensuse-leap-15-3) | Oct 2027 | <https://www.suse.com/lifecycle/> | - -| Oracle Linux | GitLab CE / GitLab EE 8.14.0 | x86_64 | [Use CentOS Install Docs](https://about.gitlab.com/install/#centos-7) | Jul 2024 | <https://www.oracle.com/a/ocom/docs/elsp-lifetime-069338.pdf> | -| Scientific Linux | GitLab CE / GitLab EE 8.14.0 | x86_64 | [Use CentOS Install Docs](https://about.gitlab.com/install/#centos-7) | June 2024 | <https://scientificlinux.org/downloads/sl-versions/sl7/> | -| Ubuntu 18.04 | GitLab CE / GitLab EE 10.7.0 | amd64 | [Ubuntu Install Docs](https://about.gitlab.com/install/#ubuntu) | April 2023 | <https://wiki.ubuntu.com/Releases> | -| Ubuntu 20.04 | GitLab CE / GitLab EE 13.2.0 | amd64, arm64 | [Ubuntu Install Docs](https://about.gitlab.com/install/#ubuntu) | April 2025 | <https://wiki.ubuntu.com/Releases> | -| Amazon Linux 2 | GitLab CE / GitLab EE 14.9.0 | amd64, arm64 | [Amazon Linux 2 Instal Docsl](https://about.gitlab.com/install/#amazonlinux-2) | June 2023 | <https://aws.amazon.com/amazon-linux-2/faqs/> | -| Raspberry Pi OS (Buster) (formerly known as Raspbian Buster) | GitLab CE 12.2.0 | armhf | [Raspberry Pi Install Docs](https://about.gitlab.com/install/#raspberry-pi-os) | 2024 | [Raspberry Pi Details](https://www.raspberrypi.com/news/new-old-functionality-with-raspberry-pi-os-legacy/) | +| AlmaLinux 8 | GitLab CE / GitLab EE 14.5.0 | x86_64, aarch64 | [AlmaLinux Install Documentation](https://about.gitlab.com/install/#almalinux-8) | 2029 | <https://almalinux.org/> | +| CentOS 7 | GitLab CE / GitLab EE 7.10.0 | x86_64 | [CentOS Install Documentation](https://about.gitlab.com/install/#centos-7) | June 2024 | <https://wiki.centos.org/About/Product> | +| Debian 9 | GitLab CE / GitLab EE 9.3.0 | amd64 | [Debian Install Documentation](https://about.gitlab.com/install/#debian) | 2022 | <https://wiki.debian.org/LTS> | +| Debian 10 | GitLab CE / GitLab EE 12.2.0 | amd64, arm64 | [Debian Install Documentation](https://about.gitlab.com/install/#debian) | 2024 | <https://wiki.debian.org/LTS> | +| Debian 11 | GitLab CE / GitLab EE 14.6.0 | amd64, arm64 | [Debian Install Documentation](https://about.gitlab.com/install/#debian) | 2026 | <https://wiki.debian.org/LTS> | +| OpenSUSE 15.3 | GitLab CE / GitLab EE 14.5.0 | x86_64, aarch64 | [OpenSUSE Install Documentation](https://about.gitlab.com/install/#opensuse-leap-15-3) | Nov 2022 | <https://en.opensuse.org/Lifetime> | +| RHEL 8 | GitLab CE / GitLab EE 12.8.1 | x86_64, arm64 | [Use CentOS Install Documentation](https://about.gitlab.com/install/#centos-7) | May 2024 | [RHEL Details](https://access.redhat.com/support/policy/updates/errata/#Life_Cycle_Dates) | +| SLES 12 | GitLab EE 9.0.0 | x86_64 | [Use OpenSUSE Install Documentation](https://about.gitlab.com/install/#opensuse-leap-15-3) | Oct 2027 | <https://www.suse.com/lifecycle/> | +| Oracle Linux | GitLab CE / GitLab EE 8.14.0 | x86_64 | [Use CentOS Install Documentation](https://about.gitlab.com/install/#centos-7) | Jul 2024 | <https://www.oracle.com/a/ocom/docs/elsp-lifetime-069338.pdf> | +| Scientific Linux | GitLab CE / GitLab EE 8.14.0 | x86_64 | [Use CentOS Install Documentation](https://about.gitlab.com/install/#centos-7) | June 2024 | <https://scientificlinux.org/downloads/sl-versions/sl7/> | +| Ubuntu 18.04 | GitLab CE / GitLab EE 10.7.0 | amd64 | [Ubuntu Install Documentation](https://about.gitlab.com/install/#ubuntu) | April 2023 | <https://wiki.ubuntu.com/Releases> | +| Ubuntu 20.04 | GitLab CE / GitLab EE 13.2.0 | amd64, arm64 | [Ubuntu Install Documentation](https://about.gitlab.com/install/#ubuntu) | April 2025 | <https://wiki.ubuntu.com/Releases> | +| Amazon Linux 2 | GitLab CE / GitLab EE 14.9.0 | amd64, arm64 | [Amazon Linux 2 Install Documentation](https://about.gitlab.com/install/#amazonlinux-2) | June 2023 | <https://aws.amazon.com/amazon-linux-2/faqs/> | +| Raspberry Pi OS (Buster) (formerly known as Raspbian Buster) | GitLab CE 12.2.0 | armhf | [Raspberry Pi Install Documentation](https://about.gitlab.com/install/#raspberry-pi-os) | 2024 | [Raspberry Pi Details](https://www.raspberrypi.com/news/new-old-functionality-with-raspberry-pi-os-legacy/) | NOTE: CentOS 8 was EOL on December 31, 2021. In GitLab 14.5 and later, [CentOS builds work in AlmaLinux](https://gitlab.com/gitlab-org/distribution/team-tasks/-/issues/954#note_730198505). -We will officially support all distributions that are binary compatible with Red Hat Enterprise Linux. +We officially support all distributions that are binary compatible with Red Hat Enterprise Linux. This gives users a path forward for their CentOS 8 builds at its end of life. ## Update GitLab package sources after upgrading the OS @@ -48,7 +47,7 @@ If your package manager reports that no further updates are available, although [new versions have been released](https://about.gitlab.com/releases/categories/releases/), repeat the "Add the GitLab package repository" instructions of the [Linux package install guide](https://about.gitlab.com/install/#content). -Future GitLab upgrades will now be fetched according to your upgraded OS. +Future GitLab upgrades are fetched according to your upgraded OS. ## Packages for ARM64 @@ -65,7 +64,7 @@ running GitLab on ARM. ## OS Versions that are no longer supported GitLab provides omnibus packages for operating systems only until their -EOL (End-Of-Life). After the EOL date of the OS, GitLab will stop releasing +EOL (End-Of-Life). After the EOL date of the OS, GitLab stops releasing official packages. The list of deprecated operating systems and the final GitLab release for them can be found below: diff --git a/doc/administration/packages/container_registry.md b/doc/administration/packages/container_registry.md index 2fa3eea54bd..3bfb3ceca86 100644 --- a/doc/administration/packages/container_registry.md +++ b/doc/administration/packages/container_registry.md @@ -887,7 +887,7 @@ administrators can clean up image tags and [run garbage collection](#container-registry-garbage-collection). To remove image tags by running the cleanup policy, run the following commands in the -[GitLab Rails console](../troubleshooting/navigating_gitlab_via_rails_console.md): +[GitLab Rails console](../operations/rails_console.md): ```ruby # Numeric ID of the project whose container registry should be cleaned up @@ -927,9 +927,9 @@ these controls should migrate to the GitLab interface. Users who have the [Maintainer role](../../user/permissions.md) for the project can [delete Container Registry tags in bulk](../../api/container_registry.md#delete-registry-repository-tags-in-bulk) -periodically based on their own criteria, however, this alone does not recycle data, +periodically based on their own criteria. However, deleting the tags alone does not recycle data, it only unlinks tags from manifests and image blobs. To recycle the Container -Registry data in the whole GitLab instance, you can use the built-in command +Registry data in the whole GitLab instance, you can use the built-in garbage collection command provided by `gitlab-ctl`. Prerequisites: @@ -938,7 +938,7 @@ Prerequisites: [cloud native chart](https://docs.gitlab.com/charts/charts/registry/#garbage-collection). - You must set the Registry to [read-only mode](#performing-garbage-collection-without-downtime). Running garbage collection causes downtime for the Container Registry. When you run this command - on an instance in an environment where another instances is still writing to the Registry storage, + on an instance in an environment where another instance is still writing to the Registry storage, referenced manifests are removed. ### Understanding the content-addressable layers @@ -1389,7 +1389,7 @@ project or branch name. Special characters can include: - Double hyphen/dash To get around this, you can [change the group path](../../user/group/index.md#change-a-groups-path), -[change the project path](../../user/project/settings/index.md#renaming-a-repository) or change the +[change the project path](../../user/project/settings/index.md#rename-a-repository) or change the branch name. Another option is to create a [push rule](../../user/project/repository/push_rules.md) to prevent this at the instance level. @@ -1452,7 +1452,7 @@ curl "localhost:5001/debug/vars" ### Access old schema v1 Docker images -Support for the [Docker registry v1 API](https://www.docker.com/blog/registry-v1-api-deprecation/), +Support for the Docker registry API V1, including [schema V1 image manifests](https://docs.docker.com/registry/spec/manifest-v2-1/), was: @@ -1738,7 +1738,7 @@ In this case, follow these steps: 1. Try the removal again. If you still can't remove the repository using the common methods, you can use the -[GitLab Rails console](../troubleshooting/navigating_gitlab_via_rails_console.md) +[GitLab Rails console](../operations/rails_console.md) to remove the project by force: ```ruby diff --git a/doc/administration/pages/index.md b/doc/administration/pages/index.md index 1535de0c2c2..b58eeff48cf 100644 --- a/doc/administration/pages/index.md +++ b/doc/administration/pages/index.md @@ -269,6 +269,9 @@ control over how the Pages daemon runs and serves content in your environment. | `max_uri_length` | The maximum length of URIs accepted by GitLab Pages. Set to 0 for unlimited length. [Introduced](https://gitlab.com/gitlab-org/gitlab-pages/-/issues/659) in GitLab 14.5. | `metrics_address` | The address to listen on for metrics requests. | | `redirect_http` | Redirect pages from HTTP to HTTPS, true/false. | +| `redirects_max_config_size` | The maximum size of the _redirects file, in bytes (default: 65536). | +| `redirects_max_path_segments` | The maximum number of path segments allowed in _redirects rules URLs (default: 25). | +| `redirects_max_rule_count` | The maximum number of rules allowed in _redirects (default: 1000). | | `sentry_dsn` | The address for sending Sentry crash reporting to. | | `sentry_enabled` | Enable reporting and logging with Sentry, true/false. | | `sentry_environment` | The environment for Sentry crash reporting. | @@ -489,11 +492,6 @@ To do that: 1. Select the **Disable public access to Pages sites** checkbox. 1. Select **Save changes**. -WARNING: -For self-managed installations, all public websites remain private until they are -redeployed. Resolve this issue by -[sourcing domain configuration from the GitLab API](https://gitlab.com/gitlab-org/gitlab/-/issues/218357). - ### Running behind a proxy Like the rest of GitLab, Pages can be used in those environments where external @@ -582,6 +580,19 @@ HTTP Strict Transport Security (HSTS) can be enabled through the `gitlab_pages[' gitlab_pages['headers'] = ['Strict-Transport-Security: max-age=63072000'] ``` +### Pages project redirects limits + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-pages/-/merge_requests/778) in GitLab 15.2. + +GitLab Pages comes with a set of default limits for the [`_redirects` file](../../user/project/pages/redirects.md) +to minimize the impact on performance. You can configure these limits if you'd like to increase or decrease the limits. + +```ruby +gitlab_pages['redirects_max_config_size'] = 131072 +gitlab_pages['redirects_max_path_segments'] = 50 +gitlab_pages['redirects_max_rule_count'] = 2000 +``` + ## Activate verbose logging for daemon Follow the steps below to configure verbose logging of GitLab Pages daemon. @@ -1400,7 +1411,7 @@ this setting needs to be configured on the main GitLab server. If the wildcard DNS [prerequisite](#prerequisites) can't be met, you can still use GitLab Pages in a limited fashion: -1. [Move](../../user/project/settings/index.md#transferring-an-existing-project-into-another-namespace) +1. [Move](../../user/project/settings/index.md#transfer-a-project-to-another-namespace) all projects you need to use Pages with into a single group namespace, for example `pages`. 1. Configure a [DNS entry](#dns-configuration) without the `*.`-wildcard, for example `pages.example.io`. 1. Configure `pages_external_url http://example.io/` in your `gitlab.rb` file. diff --git a/doc/administration/postgresql/external.md b/doc/administration/postgresql/external.md index 5d693793a92..7036502e377 100644 --- a/doc/administration/postgresql/external.md +++ b/doc/administration/postgresql/external.md @@ -21,7 +21,7 @@ If you use a cloud-managed service, or provide your own PostgreSQL instance: 1. If you are using a cloud-managed service, you may need to grant additional roles to your `gitlab` user: - Amazon RDS requires the [`rds_superuser`](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Appendix.PostgreSQL.CommonDBATasks.html#Appendix.PostgreSQL.CommonDBATasks.Roles) role. - - Azure Database for PostgreSQL requires the [`azure_pg_admin`](https://docs.microsoft.com/en-us/azure/postgresql/howto-create-users#how-to-create-additional-admin-users-in-azure-database-for-postgresql) role. Azure Database for PostgreSQL - Flexible Server requires [allow-listing extensions before they can be installed](https://docs.microsoft.com/en-us/azure/postgresql/flexible-server/concepts-extensions#how-to-use-postgresql-extensions). + - Azure Database for PostgreSQL requires the [`azure_pg_admin`](https://docs.microsoft.com/en-us/azure/postgresql/single-server/how-to-create-users#how-to-create-additional-admin-users-in-azure-database-for-postgresql) role. Azure Database for PostgreSQL - Flexible Server requires [allow-listing extensions before they can be installed](https://docs.microsoft.com/en-us/azure/postgresql/flexible-server/concepts-extensions#how-to-use-postgresql-extensions). - Google Cloud SQL requires the [`cloudsqlsuperuser`](https://cloud.google.com/sql/docs/postgres/users#default-users) role. This is for the installation of extensions during installation and upgrades. As an alternative, @@ -48,3 +48,20 @@ If you use a cloud-managed service, or provide your own PostgreSQL instance: ```shell sudo gitlab-ctl reconfigure ``` + +## Troubleshooting + +### Resolve `SSL SYSCALL error: EOF detected` error + +When using an external PostgreSQL instance, you may see an error like: + +```shell +pg_dump: error: Error message from server: SSL SYSCALL error: EOF detected +``` + +To resolve this error, ensure that you are meeting the +[minimum PostgreSQL requirements](../../install/requirements.md#postgresql-requirements). After +upgrading your RDS instance to a suitable version, you should be able to perform a backup without +this error. Refer to issue #64763 +([Segmentation fault citing `LooseForeignKeys::CleanupWorker` causes complete database restart](https://gitlab.com/gitlab-org/gitlab/-/issues/364763)) +for more information. diff --git a/doc/administration/postgresql/moving.md b/doc/administration/postgresql/moving.md new file mode 100644 index 00000000000..23d377dba37 --- /dev/null +++ b/doc/administration/postgresql/moving.md @@ -0,0 +1,65 @@ +--- +stage: Data Stores +group: Database +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Moving GitLab databases to a different PostgreSQL instance **(FREE SELF)** + +Sometimes it is necessary to move your databases from one PostgreSQL instance to +another. For example, if you are using AWS Aurora and are preparing to +enable Database Load Balancing, you will need to move your databases to +RDS for PostgreSQL. + +To move databases from one instance to another: + +1. Gather the source and destination PostgreSQL endpoint information: + + ```shell + SRC_PGHOST=<source postgresql host> + SRC_PGUSER=<source postgresql user> + + DST_PGHOST=<destination postgresql host> + DST_PGUSER=<destination postgresql user> + ``` + +1. Stop GitLab: + + ```shell + sudo gitlab-ctl stop + ``` + +1. Dump the databases from the source: + + ```shell + /opt/gitlab/embedded/bin/pg_dump -h $SRC_PGHOST -U $SRC_PGUSER -c -C -f gitlabhq_production.sql gitlabhq_production + /opt/gitlab/embedded/bin/pg_dump -h $SRC_PGHOST -U $SRC_PGUSER -c -C -f praefect_production.sql praefect_production + ``` + +1. Restore the databases to the destination (this will overwrite any existing databases with the same names): + + ```shell + /opt/gitlab/embedded/bin/psql -h $DST_PGHOST -U $DST_PGUSER -f praefect_production.sql postgres + /opt/gitlab/embedded/bin/psql -h $DST_PGHOST -U $DST_PGUSER -f gitlabhq_production.sql postgres + ``` + +1. Configure the GitLab application servers with the appropriate connection details + for your destination PostgreSQL instance in your `/etc/gitlab/gitlab.rb` file: + + ```ruby + gitlab_rails['db_host'] = '<destination postgresql host>' + ``` + + For more information on GitLab multi-node setups, refer to the [reference architectures](../reference_architectures/index.md). + +1. Reconfigure for the changes to take effect: + + ```shell + sudo gitlab-ctl reconfigure + ``` + +1. Restart GitLab: + + ```shell + sudo gitlab-ctl start + ``` diff --git a/doc/administration/postgresql/pgbouncer.md b/doc/administration/postgresql/pgbouncer.md index 8ae2b6497f8..b12edd4b9ad 100644 --- a/doc/administration/postgresql/pgbouncer.md +++ b/doc/administration/postgresql/pgbouncer.md @@ -173,7 +173,7 @@ ote_pid | tls Some database changes have to be done directly, and not through PgBouncer. Read more about the affected tasks: [database restores](../../raketasks/backup_restore.md#back-up-and-restore-for-installations-using-pgbouncer) -and [GitLab upgrades](../../update/zero_downtime.md#use-postgresql-ha). +and [GitLab upgrades](../../update/zero_downtime.md#postgresql). 1. To find the primary node, run the following on a database node: diff --git a/doc/administration/postgresql/replication_and_failover.md b/doc/administration/postgresql/replication_and_failover.md index e9b607ad5d4..37471a4f491 100644 --- a/doc/administration/postgresql/replication_and_failover.md +++ b/doc/administration/postgresql/replication_and_failover.md @@ -996,7 +996,7 @@ You can switch an exiting database cluster to use Patroni instead of repmgr with ### Upgrading PostgreSQL major version in a Patroni cluster -As of GitLab 13.3, PostgreSQL 11.7 and 12.3 are both shipped with Omnibus GitLab by default. As of GitLab 13.7, PostgreSQL 12 is the default. If you want to upgrade to PostgreSQL 12 in versions prior to GitLab 13.7, you must ask for it explicitly. +As of GitLab 14.1, PostgreSQL 12.6 and 13.3 are both shipped with Omnibus GitLab by default. As of GitLab 15.0, PostgreSQL 13 is the default. If you want to upgrade to PostgreSQL 13 in versions prior to GitLab 15.0, you must ask for it explicitly. WARNING: The procedure for upgrading PostgreSQL in a Patroni cluster is different than when upgrading using repmgr. @@ -1046,7 +1046,7 @@ Considering these, you should carefully plan your PostgreSQL upgrade: 1. Upgrade PostgreSQL on **the leader node** and make sure that the upgrade is completed successfully: ```shell - sudo gitlab-ctl pg-upgrade -V 12 + sudo gitlab-ctl pg-upgrade -V 13 ``` NOTE: @@ -1073,7 +1073,7 @@ Considering these, you should carefully plan your PostgreSQL upgrade: 1. Upgrade PostgreSQL **on replicas** (you can do this in parallel on all of them): ```shell - sudo gitlab-ctl pg-upgrade -V 12 + sudo gitlab-ctl pg-upgrade -V 13 ``` NOTE: diff --git a/doc/administration/raketasks/check.md b/doc/administration/raketasks/check.md index d66ff9afb94..cf569cd81d0 100644 --- a/doc/administration/raketasks/check.md +++ b/doc/administration/raketasks/check.md @@ -211,7 +211,7 @@ secrets file (`gitlab-secrets.json`). Automatic resolution is not yet implemented. If you have values that cannot be decrypted, you can follow steps to reset them, see our -docs on what to do [when the secrets file is lost](../../raketasks/backup_restore.md#when-the-secrets-file-is-lost). +documentation on what to do [when the secrets file is lost](../../raketasks/backup_restore.md#when-the-secrets-file-is-lost). This can take a very long time, depending on the size of your database, as it checks all rows in all tables. @@ -352,23 +352,23 @@ When this scenario is detected, the Rake task displays an error message. For exa ```shell Checking integrity of Job artifacts -- 3..8: Failures: 2 - - Job artifact: 3: #<Errno::ENOENT: No such file or directory @ rb_sysopen - /var/opt/gitlab/gitlab-rails/shared/artifacts/4e/07/4e07408562bedb8b60ce05c1decfe3ad16b72230967de01f640b7e4729b49fce/2021_05_26/5/3/job.log> - - Job artifact: 8: #<Errno::ENOENT: No such file or directory @ rb_sysopen - /var/opt/gitlab/gitlab-rails/shared/artifacts/4e/07/4e07408562bedb8b60ce05c1decfe3ad16b72230967de01f640b7e4729b49fce/2021_05_26/6/8/job.log> +- 1..15: Failures: 2 + - Job artifact: 9: #<Errno::ENOENT: No such file or directory @ rb_sysopen - /var/opt/gitlab/gitlab-rails/shared/artifacts/4b/22/4b227777d4dd1fc61c6f884f48641d02b4d121d3fd328cb08b5531fcacdabf8a/2022_06_30/8/9/job.log> + - Job artifact: 15: Remote object does not exist Done! ``` -To delete these references to missing local artifacts (`job.log` files): +To delete these references to missing local and/or remote artifacts (`job.log` files): 1. Open the [GitLab Rails Console](../operations/rails_console.md#starting-a-rails-console-session). 1. Run the following Ruby code: ```ruby artifacts_deleted = 0 - ::Ci::JobArtifact.find_each do |artifact| ### Iterate artifacts + ::Ci::JobArtifact.find_each do |artifact| ### Iterate artifacts # next if artifact.file.filename != "job.log" ### Uncomment if only `job.log` files' references are to be processed - next if artifact.file.exists? ### Skip if the file reference is valid + next if artifact.file.file.exists? ### Skip if the file reference is valid artifacts_deleted += 1 puts "#{artifact.id} #{artifact.file.path} is missing." ### Allow verification before destroy # artifact.destroy! ### Uncomment to actually destroy diff --git a/doc/administration/raketasks/github_import.md b/doc/administration/raketasks/github_import.md index 30c8518c209..0d724bfd4dc 100644 --- a/doc/administration/raketasks/github_import.md +++ b/doc/administration/raketasks/github_import.md @@ -11,13 +11,13 @@ A username should be passed as the second argument to the Rake task, which becomes the owner of the project. You can resume an import with the same command. -Bear in mind that the syntax is very specific. Remove any spaces within the argument block and +Bear in mind that the syntax is very specific. Remove any spaces in the argument block and before/after the brackets. Also, some shells (for example, Zsh) can interpret the open/close brackets -(`[]`) separately. You may need to either escape the brackets or use double quotes. +(`[]`) separately. You may want to either escape the brackets or use double quotes. ## Caveats -If the GitHub [rate limit](https://docs.github.com/en/rest/reference/rate-limit) is reached while +If the GitHub [rate limit](https://docs.github.com/en/rest/rate-limit) is reached while importing, the importing process waits (`sleep()`) until it can continue importing. ## Importing multiple projects diff --git a/doc/administration/raketasks/maintenance.md b/doc/administration/raketasks/maintenance.md index f4bd6f255e5..6de47c37957 100644 --- a/doc/administration/raketasks/maintenance.md +++ b/doc/administration/raketasks/maintenance.md @@ -186,7 +186,7 @@ Checking GitLab ... Finished ## Rebuild `authorized_keys` file In some cases it is necessary to rebuild the `authorized_keys` file, -for example, if after an upgrade you receive `Permission denied (publickey)` when pushing [via SSH](../../ssh/index.md) +for example, if after an upgrade you receive `Permission denied (publickey)` when pushing [via SSH](../../user/ssh.md) and find `404 Key Not Found` errors in [the `gitlab-shell.log` file](../logs.md#gitlab-shelllog). To rebuild `authorized_keys`, run: @@ -234,7 +234,7 @@ sudo -u git -H bundle exec rake cache:clear RAILS_ENV=production Sometimes during version upgrades you might end up with some wrong CSS or missing some icons. In that case, try to precompile the assets again. -This only applies to source installations and does NOT apply to +This only applies to source installations and does not apply to Omnibus packages. **Source Installation** diff --git a/doc/administration/raketasks/praefect.md b/doc/administration/raketasks/praefect.md index d2fd4943c68..b7db3f26a60 100644 --- a/doc/administration/raketasks/praefect.md +++ b/doc/administration/raketasks/praefect.md @@ -1,8 +1,7 @@ --- -stage: Create +stage: Systems group: Gitaly info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments -type: reference --- # Praefect Rake tasks **(FREE SELF)** diff --git a/doc/administration/raketasks/smtp.md b/doc/administration/raketasks/smtp.md index bc78d54aa1b..49274501809 100644 --- a/doc/administration/raketasks/smtp.md +++ b/doc/administration/raketasks/smtp.md @@ -55,7 +55,7 @@ bundle exec rake gitlab:smtp:secret:edit RAILS_ENV=production EDITOR=vim ### Write raw secret -Write new secret content by providing it on STDIN. +Write new secret content by providing it on `STDIN`. **Omnibus Installation** diff --git a/doc/administration/read_only_gitlab.md b/doc/administration/read_only_gitlab.md index a601ba4cd7a..387a1a75393 100644 --- a/doc/administration/read_only_gitlab.md +++ b/doc/administration/read_only_gitlab.md @@ -63,7 +63,7 @@ sudo gitlab-ctl start puma ## Make the database read-only -If you want to allow users to use the GitLab UI, then you need to ensure that +If you want to allow users to use the GitLab UI, ensure that the database is read-only: 1. Take a [GitLab backup](../raketasks/backup_restore.md) @@ -113,7 +113,7 @@ the database is read-only: sudo gitlab-ctl restart postgresql ``` -When you're ready to revert the read-only state, you need to remove the added +When you're ready to revert the read-only state, remove the added lines in `/etc/gitlab/gitlab.rb`, and reconfigure GitLab and restart PostgreSQL: ```shell @@ -121,5 +121,5 @@ sudo gitlab-ctl reconfigure sudo gitlab-ctl restart postgresql ``` -Once you verify all works as expected, you can remove the `gitlab_read_only` +After you verify all works as expected, remove the `gitlab_read_only` user from the database. diff --git a/doc/administration/redis/index.md b/doc/administration/redis/index.md index 3581b27a4c8..fb5f6cc3a54 100644 --- a/doc/administration/redis/index.md +++ b/doc/administration/redis/index.md @@ -12,7 +12,7 @@ multiple ways to configure Redis. You can choose to install and manage Redis and Sentinel yourself, use a hosted cloud solution, or you can use the ones that come bundled with the Omnibus GitLab -packages so you only need to focus on configuration. Pick the one that suits your needs. +packages so you can only focus on configuration. Pick the one that suits your needs. ## Redis replication and failover using Omnibus GitLab diff --git a/doc/administration/redis/standalone.md b/doc/administration/redis/standalone.md index 6e0e1eb886e..bd5b30efb7b 100644 --- a/doc/administration/redis/standalone.md +++ b/doc/administration/redis/standalone.md @@ -44,7 +44,7 @@ Omnibus GitLab: 1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. 1. Note the Redis node's IP address or hostname, port, and - Redis password. These will be necessary when [configuring the GitLab application servers](#set-up-the-gitlab-rails-application-instance). + Redis password. These are necessary when [configuring the GitLab application servers](#set-up-the-gitlab-rails-application-instance). [Advanced configuration options](https://docs.gitlab.com/omnibus/settings/redis.html) are supported and can be added if needed. diff --git a/doc/administration/redis/troubleshooting.md b/doc/administration/redis/troubleshooting.md index 4426adc5d51..ca52fe0a29a 100644 --- a/doc/administration/redis/troubleshooting.md +++ b/doc/administration/redis/troubleshooting.md @@ -7,7 +7,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Troubleshooting Redis **(FREE SELF)** -There are a lot of moving parts that needs to be taken care carefully +There are a lot of moving parts that must be taken care carefully in order for the HA setup to work as expected. Before proceeding with the troubleshooting below, check your firewall rules: @@ -41,7 +41,7 @@ You can check if everything is correct by connecting to each server using /opt/gitlab/embedded/bin/redis-cli -h <redis-host-or-ip> -a '<redis-password>' info replication ``` -When connected to a `Primary` Redis, you will see the number of connected +When connected to a `Primary` Redis, you see the number of connected `replicas`, and a list of each with connection details: ```plaintext @@ -56,7 +56,7 @@ repl_backlog_first_byte_offset:206989083 repl_backlog_histlen:1048576 ``` -When it's a `replica`, you will see details of the primary connection and if +When it's a `replica`, you see details of the primary connection and if its `up` or `down`: ```plaintext @@ -138,7 +138,7 @@ there may be something wrong with your configuration files or it can be related to [this upstream issue](https://github.com/redis/redis-rb/issues/531). You must make sure that `resque.yml` and `sentinel.conf` are configured correctly, -otherwise `redis-rb` will not work properly. +otherwise `redis-rb` does not work properly. The `master-group-name` (`gitlab-redis`) defined in (`sentinel.conf`) **must** be used as the hostname in GitLab (`resque.yml`): diff --git a/doc/administration/reference_architectures/10k_users.md b/doc/administration/reference_architectures/10k_users.md index 99272cdd226..a64828be4be 100644 --- a/doc/administration/reference_architectures/10k_users.md +++ b/doc/administration/reference_architectures/10k_users.md @@ -38,7 +38,7 @@ full list of reference architectures, see <!-- Disable ordered list rule https://github.com/DavidAnson/markdownlint/blob/main/doc/Rules.md#md029---ordered-list-item-prefix --> <!-- markdownlint-disable MD029 --> -1. Can be optionally run on reputable third-party external PaaS PostgreSQL solutions. [Google Cloud SQL](https://cloud.google.com/sql/docs/postgres/high-availability#normal) and [Amazon RDS](https://aws.amazon.com/rds/) are known to work, however Azure Database for PostgreSQL is **not recommended** due to [performance issues](https://gitlab.com/gitlab-org/quality/reference-architectures/-/issues/61). Consul is primarily used for PostgreSQL high availability so can be ignored when using a PostgreSQL PaaS setup. However it is also used optionally by Prometheus for Omnibus auto host discovery. +1. Can be optionally run on reputable third-party external PaaS PostgreSQL solutions. [Google Cloud SQL](https://cloud.google.com/sql/docs/postgres/high-availability#normal) and [Amazon RDS](https://aws.amazon.com/rds/) are known to work. However, Amazon Aurora is **incompatible** with load balancing enabled by default in [14.4.0](../../update/index.md#1440), and Azure Database for PostgreSQL is **not recommended** due to [performance issues](https://gitlab.com/gitlab-org/quality/reference-architectures/-/issues/61). Consul is primarily used for PostgreSQL high availability so can be ignored when using a PostgreSQL PaaS setup. However, Consul is also used optionally by Prometheus for Omnibus auto host discovery. 2. Can be optionally run on reputable third-party external PaaS Redis solutions. Google Memorystore and AWS ElastiCache are known to work. 3. Can be optionally run on reputable third-party load balancing services (LB PaaS). AWS ELB is known to work. 4. Should be run on reputable third-party object storage (storage PaaS) for cloud implementations. Google Cloud Storage and AWS S3 are known to work. @@ -153,6 +153,7 @@ As a general guidance, GitLab should run on most infrastructure such as reputabl Be aware of the following specific call outs: +- [Amazon Aurora](https://aws.amazon.com/rds/aurora/) is incompatible. See [14.4.0](../../update/index.md#1440) for more details. - [Azure Database for PostgreSQL](https://docs.microsoft.com/en-us/azure/postgresql/#:~:text=Azure%20Database%20for%20PostgreSQL%20is,high%20availability%2C%20and%20dynamic%20scalability.) is [not recommended](https://gitlab.com/gitlab-org/quality/reference-architectures/-/issues/61) due to known performance issues or missing features. - [Azure Blob Storage](https://docs.microsoft.com/en-us/azure/storage/blobs/) is recommended to be configured with [Premium accounts](https://docs.microsoft.com/en-us/azure/storage/blobs/storage-blob-block-blob-premium) to ensure consistent performance. @@ -502,7 +503,7 @@ cluster to be used with GitLab. If you're hosting GitLab on a cloud provider, you can optionally use a managed service for PostgreSQL. -A reputable provider or solution should be used for this. [Google Cloud SQL](https://cloud.google.com/sql/docs/postgres/high-availability#normal) and [Amazon RDS](https://aws.amazon.com/rds/) are known to work, however Azure Database for PostgreSQL is **not recommended** due to [performance issues](https://gitlab.com/gitlab-org/quality/reference-architectures/-/issues/61). +A reputable provider or solution should be used for this. [Google Cloud SQL](https://cloud.google.com/sql/docs/postgres/high-availability#normal) and [Amazon RDS](https://aws.amazon.com/rds/) are known to work. However, Amazon Aurora is **incompatible** with load balancing enabled by default in [14.4.0](../../update/index.md#1440), and Azure Database for PostgreSQL is **not recommended** due to [performance issues](https://gitlab.com/gitlab-org/quality/reference-architectures/-/issues/61). If you use a cloud-managed service, or provide your own PostgreSQL: @@ -1271,7 +1272,7 @@ you are using Geo, where separate database instances are required for handling r In this setup, the specs of the main database setup shouldn't need to be changed as the impact should be minimal. -A reputable provider or solution should be used for this. [Google Cloud SQL](https://cloud.google.com/sql/docs/postgres/high-availability#normal) and [Amazon RDS](https://aws.amazon.com/rds/) are known to work, however Azure Database for PostgreSQL is **not recommended** due to [performance issues](https://gitlab.com/gitlab-org/quality/reference-architectures/-/issues/61). +A reputable provider or solution should be used for this. [Google Cloud SQL](https://cloud.google.com/sql/docs/postgres/high-availability#normal) and [Amazon RDS](https://aws.amazon.com/rds/) are known to work. However, Amazon Aurora is **incompatible** with load balancing enabled by default in [14.4.0](../../update/index.md#1440), and Azure Database for PostgreSQL is **not recommended** due to [performance issues](https://gitlab.com/gitlab-org/quality/reference-architectures/-/issues/61). Examples of the above could include [Google's Cloud SQL](https://cloud.google.com/sql/docs/postgres/high-availability#normal) or [Amazon RDS](https://aws.amazon.com/rds/). @@ -1351,6 +1352,9 @@ To configure the Praefect nodes, on each one: on the page. 1. Edit the `/etc/gitlab/gitlab.rb` file to configure Praefect: + NOTE: + You can't remove the `default` entry from `virtual_storages` because [GitLab requires it](../gitaly/configure_gitaly.md#gitlab-requires-a-default-repository-storage). + <!-- Updates to example must be made at: - https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/administration/gitaly/praefect.md @@ -2202,7 +2206,7 @@ on what features you intend to use: |Object storage type|Supported by consolidated configuration?| |-------------------|----------------------------------------| -| [Backups](../../raketasks/backup_restore.md#uploading-backups-to-a-remote-cloud-storage) | No | +| [Backups](../../raketasks/backup_gitlab.md#uploading-backups-to-a-remote-cloud-storage) | No | | [Job artifacts](../job_artifacts.md#using-object-storage) including archived job logs | Yes | | [LFS objects](../lfs/index.md#storing-lfs-objects-in-remote-object-storage) | Yes | | [Uploads](../uploads.md#using-object-storage) | Yes | @@ -2328,7 +2332,7 @@ services where applicable): <!-- Disable ordered list rule https://github.com/DavidAnson/markdownlint/blob/main/doc/Rules.md#md029---ordered-list-item-prefix --> <!-- markdownlint-disable MD029 --> -1. Can be optionally run on reputable third-party external PaaS PostgreSQL solutions. [Google Cloud SQL](https://cloud.google.com/sql/docs/postgres/high-availability#normal) and [Amazon RDS](https://aws.amazon.com/rds/) are known to work, however Azure Database for PostgreSQL is **not recommended** due to [performance issues](https://gitlab.com/gitlab-org/quality/reference-architectures/-/issues/61). Consul is primarily used for PostgreSQL high availability so can be ignored when using a PostgreSQL PaaS setup. However it is also used optionally by Prometheus for Omnibus auto host discovery. +1. Can be optionally run on reputable third-party external PaaS PostgreSQL solutions. [Google Cloud SQL](https://cloud.google.com/sql/docs/postgres/high-availability#normal) and [Amazon RDS](https://aws.amazon.com/rds/) are known to work. However, Amazon Aurora is **incompatible** with load balancing enabled by default in [14.4.0](../../update/index.md#1440), and Azure Database for PostgreSQL is **not recommended** due to [performance issues](https://gitlab.com/gitlab-org/quality/reference-architectures/-/issues/61). Consul is primarily used for PostgreSQL high availability so can be ignored when using a PostgreSQL PaaS setup. However, Consul is also used optionally by Prometheus for Omnibus auto host discovery. 2. Can be optionally run on reputable third-party external PaaS Redis solutions. Google Memorystore and AWS ElastiCache are known to work. 3. Can be optionally run on reputable third-party load balancing services (LB PaaS). AWS ELB is known to work. 4. Should be run on reputable third-party object storage (storage PaaS) for cloud implementations. Google Cloud Storage and AWS S3 are known to work. diff --git a/doc/administration/reference_architectures/1k_users.md b/doc/administration/reference_architectures/1k_users.md index c356c7224cb..1a774603863 100644 --- a/doc/administration/reference_architectures/1k_users.md +++ b/doc/administration/reference_architectures/1k_users.md @@ -10,7 +10,7 @@ This page describes GitLab reference architecture for up to 1,000 users. For a full list of reference architectures, see [Available reference architectures](index.md#available-reference-architectures). -If you need to serve up to 1,000 users and you don't have strict availability +If you are serving up to 1,000 users and you don't have strict availability requirements, a single-node solution with [frequent backups](index.md#automated-backups) is appropriate for many organizations. @@ -83,6 +83,7 @@ As a general guidance, GitLab should run on most infrastructure such as reputabl Be aware of the following specific call outs: +- [Amazon Aurora](https://aws.amazon.com/rds/aurora/) is incompatible. See [14.4.0](../../update/index.md#1440) for more details. - [Azure Database for PostgreSQL](https://docs.microsoft.com/en-us/azure/postgresql/#:~:text=Azure%20Database%20for%20PostgreSQL%20is,high%20availability%2C%20and%20dynamic%20scalability.) is [not recommended](https://gitlab.com/gitlab-org/quality/reference-architectures/-/issues/61) due to known performance issues or missing features. - [Azure Blob Storage](https://docs.microsoft.com/en-us/azure/storage/blobs/) is recommended to be configured with [Premium accounts](https://docs.microsoft.com/en-us/azure/storage/blobs/storage-blob-block-blob-premium) to ensure consistent performance. @@ -93,7 +94,8 @@ swap on your server, even if you currently have enough available memory. Having swap helps to reduce the chance of errors occurring if your available memory changes. We also recommend configuring the kernel's swappiness setting to a lower value (such as `10`) to make the most of your memory, while still having -the swap available when needed. +the swap available when needed. View the +[Memory requirements](../../install/requirements.md#memory) for details. ## Setup instructions @@ -121,4 +123,4 @@ components are deployed in Kubernetes via our official [Helm Charts](https://doc and _stateful_ components are deployed in compute VMs with Omnibus. The [2k GitLab Cloud Native Hybrid](2k_users.md#cloud-native-hybrid-reference-architecture-with-helm-charts-alternative) (non HA) and [3k GitLab Cloud Native Hybrid](3k_users.md#cloud-native-hybrid-reference-architecture-with-helm-charts-alternative) (HA) reference architectures are the smallest we recommend in Kubernetes. -For environments that need to serve less users, you can lower the node specs. Depending on your user count, you can lower all suggested node specs as desired. However, it's recommended that you don't go lower than the [general requirements](../../install/requirements.md). +For environments that serve fewer users, you can lower the node specs. Depending on your user count, you can lower all suggested node specs as desired. However, it's recommended that you don't go lower than the [general requirements](../../install/requirements.md). diff --git a/doc/administration/reference_architectures/25k_users.md b/doc/administration/reference_architectures/25k_users.md index 367654b8e59..5b1e8bfc16b 100644 --- a/doc/administration/reference_architectures/25k_users.md +++ b/doc/administration/reference_architectures/25k_users.md @@ -38,7 +38,7 @@ full list of reference architectures, see <!-- Disable ordered list rule https://github.com/DavidAnson/markdownlint/blob/main/doc/Rules.md#md029---ordered-list-item-prefix --> <!-- markdownlint-disable MD029 --> -1. Can be optionally run on reputable third-party external PaaS PostgreSQL solutions. [Google Cloud SQL](https://cloud.google.com/sql/docs/postgres/high-availability#normal) and [Amazon RDS](https://aws.amazon.com/rds/) are known to work, however Azure Database for PostgreSQL is **not recommended** due to [performance issues](https://gitlab.com/gitlab-org/quality/reference-architectures/-/issues/61). Consul is primarily used for PostgreSQL high availability so can be ignored when using a PostgreSQL PaaS setup. However it is also used optionally by Prometheus for Omnibus auto host discovery. +1. Can be optionally run on reputable third-party external PaaS PostgreSQL solutions. [Google Cloud SQL](https://cloud.google.com/sql/docs/postgres/high-availability#normal) and [Amazon RDS](https://aws.amazon.com/rds/) are known to work. However, Amazon Aurora is **incompatible** with load balancing enabled by default in [14.4.0](../../update/index.md#1440), and Azure Database for PostgreSQL is **not recommended** due to [performance issues](https://gitlab.com/gitlab-org/quality/reference-architectures/-/issues/61). Consul is primarily used for PostgreSQL high availability so can be ignored when using a PostgreSQL PaaS setup. However, Consul is also used optionally by Prometheus for Omnibus auto host discovery. 2. Can be optionally run on reputable third-party external PaaS Redis solutions. Google Memorystore and AWS ElastiCache are known to work. 3. Can be optionally run on reputable third-party load balancing services (LB PaaS). AWS ELB is known to work. 4. Should be run on reputable third-party object storage (storage PaaS) for cloud implementations. Google Cloud Storage and AWS S3 are known to work. @@ -153,6 +153,7 @@ As a general guidance, GitLab should run on most infrastructure such as reputabl Be aware of the following specific call outs: +- [Amazon Aurora](https://aws.amazon.com/rds/aurora/) is incompatible. See [14.4.0](../../update/index.md#1440) for more details. - [Azure Database for PostgreSQL](https://docs.microsoft.com/en-us/azure/postgresql/#:~:text=Azure%20Database%20for%20PostgreSQL%20is,high%20availability%2C%20and%20dynamic%20scalability.) is [not recommended](https://gitlab.com/gitlab-org/quality/reference-architectures/-/issues/61) due to known performance issues or missing features. - [Azure Blob Storage](https://docs.microsoft.com/en-us/azure/storage/blobs/) is recommended to be configured with [Premium accounts](https://docs.microsoft.com/en-us/azure/storage/blobs/storage-blob-block-blob-premium) to ensure consistent performance. @@ -505,7 +506,7 @@ cluster to be used with GitLab. If you're hosting GitLab on a cloud provider, you can optionally use a managed service for PostgreSQL. -A reputable provider or solution should be used for this. [Google Cloud SQL](https://cloud.google.com/sql/docs/postgres/high-availability#normal) and [Amazon RDS](https://aws.amazon.com/rds/) are known to work, however Azure Database for PostgreSQL is **not recommended** due to [performance issues](https://gitlab.com/gitlab-org/quality/reference-architectures/-/issues/61). +A reputable provider or solution should be used for this. [Google Cloud SQL](https://cloud.google.com/sql/docs/postgres/high-availability#normal) and [Amazon RDS](https://aws.amazon.com/rds/) are known to work. However, Amazon Aurora is **incompatible** with load balancing enabled by default in [14.4.0](../../update/index.md#1440), and Azure Database for PostgreSQL is **not recommended** due to [performance issues](https://gitlab.com/gitlab-org/quality/reference-architectures/-/issues/61). If you use a cloud-managed service, or provide your own PostgreSQL: @@ -1277,7 +1278,7 @@ you are using Geo, where separate database instances are required for handling r In this setup, the specs of the main database setup shouldn't need to be changed as the impact should be minimal. -A reputable provider or solution should be used for this. [Google Cloud SQL](https://cloud.google.com/sql/docs/postgres/high-availability#normal) and [Amazon RDS](https://aws.amazon.com/rds/) are known to work, however Azure Database for PostgreSQL is **not recommended** due to [performance issues](https://gitlab.com/gitlab-org/quality/reference-architectures/-/issues/61). +A reputable provider or solution should be used for this. [Google Cloud SQL](https://cloud.google.com/sql/docs/postgres/high-availability#normal) and [Amazon RDS](https://aws.amazon.com/rds/) are known to work. However, Amazon Aurora is **incompatible** with load balancing enabled by default in [14.4.0](../../update/index.md#1440), and Azure Database for PostgreSQL is **not recommended** due to [performance issues](https://gitlab.com/gitlab-org/quality/reference-architectures/-/issues/61). Once the database is set up, follow the [post configuration](#praefect-postgresql-post-configuration). @@ -1355,6 +1356,9 @@ To configure the Praefect nodes, on each one: on the page. 1. Edit the `/etc/gitlab/gitlab.rb` file to configure Praefect: + NOTE: + You can't remove the `default` entry from `virtual_storages` because [GitLab requires it](../gitaly/configure_gitaly.md#gitlab-requires-a-default-repository-storage). + <!-- Updates to example must be made at: - https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/administration/gitaly/praefect.md @@ -2206,7 +2210,7 @@ on what features you intend to use: |Object storage type|Supported by consolidated configuration?| |-------------------|----------------------------------------| -| [Backups](../../raketasks/backup_restore.md#uploading-backups-to-a-remote-cloud-storage) | No | +| [Backups](../../raketasks/backup_gitlab.md#uploading-backups-to-a-remote-cloud-storage) | No | | [Job artifacts](../job_artifacts.md#using-object-storage) including archived job logs | Yes | | [LFS objects](../lfs/index.md#storing-lfs-objects-in-remote-object-storage) | Yes | | [Uploads](../uploads.md#using-object-storage) | Yes | @@ -2326,7 +2330,7 @@ services where applicable): <!-- Disable ordered list rule https://github.com/DavidAnson/markdownlint/blob/main/doc/Rules.md#md029---ordered-list-item-prefix --> <!-- markdownlint-disable MD029 --> -1. Can be optionally run on reputable third-party external PaaS PostgreSQL solutions. [Google Cloud SQL](https://cloud.google.com/sql/docs/postgres/high-availability#normal) and [Amazon RDS](https://aws.amazon.com/rds/) are known to work, however Azure Database for PostgreSQL is **not recommended** due to [performance issues](https://gitlab.com/gitlab-org/quality/reference-architectures/-/issues/61). Consul is primarily used for PostgreSQL high availability so can be ignored when using a PostgreSQL PaaS setup. However it is also used optionally by Prometheus for Omnibus auto host discovery. +1. Can be optionally run on reputable third-party external PaaS PostgreSQL solutions. [Google Cloud SQL](https://cloud.google.com/sql/docs/postgres/high-availability#normal) and [Amazon RDS](https://aws.amazon.com/rds/) are known to work. However, Amazon Aurora is **incompatible** with load balancing enabled by default in [14.4.0](../../update/index.md#1440), and Azure Database for PostgreSQL is **not recommended** due to [performance issues](https://gitlab.com/gitlab-org/quality/reference-architectures/-/issues/61). Consul is primarily used for PostgreSQL high availability so can be ignored when using a PostgreSQL PaaS setup. However, Consul is also used optionally by Prometheus for Omnibus auto host discovery. 2. Can be optionally run on reputable third-party external PaaS Redis solutions. Google Memorystore and AWS ElastiCache are known to work. 3. Can be optionally run on reputable third-party load balancing services (LB PaaS). AWS ELB is known to work. 4. Should be run on reputable third-party object storage (storage PaaS) for cloud implementations. Google Cloud Storage and AWS S3 are known to work. diff --git a/doc/administration/reference_architectures/2k_users.md b/doc/administration/reference_architectures/2k_users.md index 3b9a8f966c8..df84bf0402d 100644 --- a/doc/administration/reference_architectures/2k_users.md +++ b/doc/administration/reference_architectures/2k_users.md @@ -31,7 +31,7 @@ For a full list of reference architectures, see | NFS server (non-Gitaly) | 1 | 4 vCPU, 3.6 GB memory | `n1-highcpu-4` | `c5.xlarge` | `F4s v2` | <!-- markdownlint-disable MD029 --> -1. Can be optionally run on reputable third-party external PaaS PostgreSQL solutions. [Google Cloud SQL](https://cloud.google.com/sql/docs/postgres/high-availability#normal) and [Amazon RDS](https://aws.amazon.com/rds/) are known to work, however Azure Database for PostgreSQL is **not recommended** due to [performance issues](https://gitlab.com/gitlab-org/quality/reference-architectures/-/issues/61). Consul is primarily used for PostgreSQL high availability so can be ignored when using a PostgreSQL PaaS setup. However it is also used optionally by Prometheus for Omnibus auto host discovery. +1. Can be optionally run on reputable third-party external PaaS PostgreSQL solutions. [Google Cloud SQL](https://cloud.google.com/sql/docs/postgres/high-availability#normal) and [Amazon RDS](https://aws.amazon.com/rds/) are known to work. However, Amazon Aurora is **incompatible** with load balancing enabled by default in [14.4.0](../../update/index.md#1440), and Azure Database for PostgreSQL is **not recommended** due to [performance issues](https://gitlab.com/gitlab-org/quality/reference-architectures/-/issues/61). Consul is primarily used for PostgreSQL high availability so can be ignored when using a PostgreSQL PaaS setup. However, Consul is also used optionally by Prometheus for Omnibus auto host discovery. 2. Can be optionally run as reputable third-party external PaaS Redis solutions. Google Memorystore and AWS ElastiCache are known to work. 3. Can be optionally run as reputable third-party load balancing services (LB PaaS). AWS ELB is known to work. 4. Should be run on reputable third-party object storage (storage PaaS) for cloud implementations. Google Cloud Storage and AWS S3 are known to work. @@ -89,6 +89,7 @@ As a general guidance, GitLab should run on most infrastructure such as reputabl Be aware of the following specific call outs: +- [Amazon Aurora](https://aws.amazon.com/rds/aurora/) is incompatible. See [14.4.0](../../update/index.md#1440) for more details. - [Azure Database for PostgreSQL](https://docs.microsoft.com/en-us/azure/postgresql/#:~:text=Azure%20Database%20for%20PostgreSQL%20is,high%20availability%2C%20and%20dynamic%20scalability.) is [not recommended](https://gitlab.com/gitlab-org/quality/reference-architectures/-/issues/61) due to known performance issues or missing features. - [Azure Blob Storage](https://docs.microsoft.com/en-us/azure/storage/blobs/) is recommended to be configured with [Premium accounts](https://docs.microsoft.com/en-us/azure/storage/blobs/storage-blob-block-blob-premium) to ensure consistent performance. @@ -246,7 +247,7 @@ to be used with GitLab. If you're hosting GitLab on a cloud provider, you can optionally use a managed service for PostgreSQL. -A reputable provider or solution should be used for this. [Google Cloud SQL](https://cloud.google.com/sql/docs/postgres/high-availability#normal) and [Amazon RDS](https://aws.amazon.com/rds/) are known to work, however Azure Database for PostgreSQL is **not recommended** due to [performance issues](https://gitlab.com/gitlab-org/quality/reference-architectures/-/issues/61). +A reputable provider or solution should be used for this. [Google Cloud SQL](https://cloud.google.com/sql/docs/postgres/high-availability#normal) and [Amazon RDS](https://aws.amazon.com/rds/) are known to work. However, Amazon Aurora is **incompatible** with load balancing enabled by default in [14.4.0](../../update/index.md#1440), and Azure Database for PostgreSQL is **not recommended** due to [performance issues](https://gitlab.com/gitlab-org/quality/reference-architectures/-/issues/61). If you use a cloud-managed service, or provide your own PostgreSQL: @@ -444,6 +445,9 @@ To configure the Gitaly server, on the server node you want to use for Gitaly: 1. Edit the Gitaly server node's `/etc/gitlab/gitlab.rb` file to configure storage paths, enable the network listener, and to configure the token: + NOTE: + You can't remove the `default` entry from `git_data_dirs` because [GitLab requires it](../gitaly/configure_gitaly.md#gitlab-requires-a-default-repository-storage). + <!-- Updates to example must be made at: - https://gitlab.com/gitlab-org/charts/gitlab/blob/master/doc/advanced/external-gitaly/external-omnibus-gitaly.md#configure-omnibus-gitlab @@ -916,7 +920,7 @@ on what features you intend to use: |Object storage type|Supported by consolidated configuration?| |-------------------|----------------------------------------| -| [Backups](../../raketasks/backup_restore.md#uploading-backups-to-a-remote-cloud-storage) | No | +| [Backups](../../raketasks/backup_gitlab.md#uploading-backups-to-a-remote-cloud-storage) | No | | [Job artifacts](../job_artifacts.md#using-object-storage) including archived job logs | Yes | | [LFS objects](../lfs/index.md#storing-lfs-objects-in-remote-object-storage) | Yes | | [Uploads](../uploads.md#using-object-storage) | Yes | @@ -1027,7 +1031,7 @@ services where applicable): <!-- Disable ordered list rule https://github.com/DavidAnson/markdownlint/blob/main/doc/Rules.md#md029---ordered-list-item-prefix --> <!-- markdownlint-disable MD029 --> -1. Can be optionally run on reputable third-party external PaaS PostgreSQL solutions. [Google Cloud SQL](https://cloud.google.com/sql/docs/postgres/high-availability#normal) and [Amazon RDS](https://aws.amazon.com/rds/) are known to work, however Azure Database for PostgreSQL is **not recommended** due to [performance issues](https://gitlab.com/gitlab-org/quality/reference-architectures/-/issues/61). Consul is primarily used for PostgreSQL high availability so can be ignored when using a PostgreSQL PaaS setup. However it is also used optionally by Prometheus for Omnibus auto host discovery. +1. Can be optionally run on reputable third-party external PaaS PostgreSQL solutions. [Google Cloud SQL](https://cloud.google.com/sql/docs/postgres/high-availability#normal) and [Amazon RDS](https://aws.amazon.com/rds/) are known to work. However, Amazon Aurora is **incompatible** with load balancing enabled by default in [14.4.0](../../update/index.md#1440), and Azure Database for PostgreSQL is **not recommended** due to [performance issues](https://gitlab.com/gitlab-org/quality/reference-architectures/-/issues/61). Consul is primarily used for PostgreSQL high availability so can be ignored when using a PostgreSQL PaaS setup. However, Consul is also used optionally by Prometheus for Omnibus auto host discovery. 2. Can be optionally run on reputable third-party external PaaS Redis solutions. Google Memorystore and AWS ElastiCache are known to work. 3. Should be run on reputable third-party object storage (storage PaaS) for cloud implementations. Google Cloud Storage and AWS S3 are known to work. <!-- markdownlint-enable MD029 --> diff --git a/doc/administration/reference_architectures/3k_users.md b/doc/administration/reference_architectures/3k_users.md index a43d33a4d4a..6a70739ca31 100644 --- a/doc/administration/reference_architectures/3k_users.md +++ b/doc/administration/reference_architectures/3k_users.md @@ -47,7 +47,7 @@ For a full list of reference architectures, see <!-- Disable ordered list rule https://github.com/DavidAnson/markdownlint/blob/main/doc/Rules.md#md029---ordered-list-item-prefix --> <!-- markdownlint-disable MD029 --> -1. Can be optionally run on reputable third-party external PaaS PostgreSQL solutions. [Google Cloud SQL](https://cloud.google.com/sql/docs/postgres/high-availability#normal) and [Amazon RDS](https://aws.amazon.com/rds/) are known to work, however Azure Database for PostgreSQL is **not recommended** due to [performance issues](https://gitlab.com/gitlab-org/quality/reference-architectures/-/issues/61). Consul is primarily used for PostgreSQL high availability so can be ignored when using a PostgreSQL PaaS setup. However it is also used optionally by Prometheus for Omnibus auto host discovery. +1. Can be optionally run on reputable third-party external PaaS PostgreSQL solutions. [Google Cloud SQL](https://cloud.google.com/sql/docs/postgres/high-availability#normal) and [Amazon RDS](https://aws.amazon.com/rds/) are known to work. However, Amazon Aurora is **incompatible** with load balancing enabled by default in [14.4.0](../../update/index.md#1440), and Azure Database for PostgreSQL is **not recommended** due to [performance issues](https://gitlab.com/gitlab-org/quality/reference-architectures/-/issues/61). Consul is primarily used for PostgreSQL high availability so can be ignored when using a PostgreSQL PaaS setup. However, Consul is also used optionally by Prometheus for Omnibus auto host discovery. 2. Can be optionally run on reputable third-party external PaaS Redis solutions. Google Memorystore and AWS ElastiCache are known to work. 3. Can be optionally run on reputable third-party load balancing services (LB PaaS). AWS ELB is known to work. 4. Should be run on reputable third-party object storage (storage PaaS) for cloud implementations. Google Cloud Storage and AWS S3 are known to work. @@ -159,6 +159,7 @@ As a general guidance, GitLab should run on most infrastructure such as reputabl Be aware of the following specific call outs: +- [Amazon Aurora](https://aws.amazon.com/rds/aurora/) is incompatible. See [14.4.0](../../update/index.md#1440) for more details. - [Azure Database for PostgreSQL](https://docs.microsoft.com/en-us/azure/postgresql/#:~:text=Azure%20Database%20for%20PostgreSQL%20is,high%20availability%2C%20and%20dynamic%20scalability.) is [not recommended](https://gitlab.com/gitlab-org/quality/reference-architectures/-/issues/61) due to known performance issues or missing features. - [Azure Blob Storage](https://docs.microsoft.com/en-us/azure/storage/blobs/) is recommended to be configured with [Premium accounts](https://docs.microsoft.com/en-us/azure/storage/blobs/storage-blob-block-blob-premium) to ensure consistent performance. @@ -787,7 +788,7 @@ cluster to be used with GitLab. If you're hosting GitLab on a cloud provider, you can optionally use a managed service for PostgreSQL. -A reputable provider or solution should be used for this. [Google Cloud SQL](https://cloud.google.com/sql/docs/postgres/high-availability#normal) and [Amazon RDS](https://aws.amazon.com/rds/) are known to work, however Azure Database for PostgreSQL is **not recommended** due to [performance issues](https://gitlab.com/gitlab-org/quality/reference-architectures/-/issues/61). +A reputable provider or solution should be used for this. [Google Cloud SQL](https://cloud.google.com/sql/docs/postgres/high-availability#normal) and [Amazon RDS](https://aws.amazon.com/rds/) are known to work. However, Amazon Aurora is **incompatible** with load balancing enabled by default in [14.4.0](../../update/index.md#1440), and Azure Database for PostgreSQL is **not recommended** due to [performance issues](https://gitlab.com/gitlab-org/quality/reference-architectures/-/issues/61). If you use a cloud-managed service, or provide your own PostgreSQL: @@ -1217,7 +1218,7 @@ you are using Geo, where separate database instances are required for handling r In this setup, the specs of the main database setup shouldn't need to be changed as the impact should be minimal. -A reputable provider or solution should be used for this. [Google Cloud SQL](https://cloud.google.com/sql/docs/postgres/high-availability#normal) and [Amazon RDS](https://aws.amazon.com/rds/) are known to work, however Azure Database for PostgreSQL is **not recommended** due to [performance issues](https://gitlab.com/gitlab-org/quality/reference-architectures/-/issues/61). +A reputable provider or solution should be used for this. [Google Cloud SQL](https://cloud.google.com/sql/docs/postgres/high-availability#normal) and [Amazon RDS](https://aws.amazon.com/rds/) are known to work. However, Amazon Aurora is **incompatible** with load balancing enabled by default in [14.4.0](../../update/index.md#1440), and Azure Database for PostgreSQL is **not recommended** due to [performance issues](https://gitlab.com/gitlab-org/quality/reference-architectures/-/issues/61). Once the database is set up, follow the [post configuration](#praefect-postgresql-post-configuration). @@ -1295,6 +1296,9 @@ To configure the Praefect nodes, on each one: on the page. 1. Edit the `/etc/gitlab/gitlab.rb` file to configure Praefect: + NOTE: + You can't remove the `default` entry from `virtual_storages` because [GitLab requires it](../gitaly/configure_gitaly.md#gitlab-requires-a-default-repository-storage). + <!-- Updates to example must be made at: - https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/administration/gitaly/praefect.md @@ -2141,7 +2145,7 @@ on what features you intend to use: |Object storage type|Supported by consolidated configuration?| |-------------------|----------------------------------------| -| [Backups](../../raketasks/backup_restore.md#uploading-backups-to-a-remote-cloud-storage) | No | +| [Backups](../../raketasks/backup_gitlab.md#uploading-backups-to-a-remote-cloud-storage) | No | | [Job artifacts](../job_artifacts.md#using-object-storage) including archived job logs | Yes | | [LFS objects](../lfs/index.md#storing-lfs-objects-in-remote-object-storage) | Yes | | [Uploads](../uploads.md#using-object-storage) | Yes | @@ -2285,7 +2289,7 @@ services where applicable): <!-- Disable ordered list rule https://github.com/DavidAnson/markdownlint/blob/main/doc/Rules.md#md029---ordered-list-item-prefix --> <!-- markdownlint-disable MD029 --> -1. Can be optionally run on reputable third-party external PaaS PostgreSQL solutions. [Google Cloud SQL](https://cloud.google.com/sql/docs/postgres/high-availability#normal) and [Amazon RDS](https://aws.amazon.com/rds/) are known to work, however Azure Database for PostgreSQL is **not recommended** due to [performance issues](https://gitlab.com/gitlab-org/quality/reference-architectures/-/issues/61). Consul is primarily used for PostgreSQL high availability so can be ignored when using a PostgreSQL PaaS setup. However it is also used optionally by Prometheus for Omnibus auto host discovery. +1. Can be optionally run on reputable third-party external PaaS PostgreSQL solutions. [Google Cloud SQL](https://cloud.google.com/sql/docs/postgres/high-availability#normal) and [Amazon RDS](https://aws.amazon.com/rds/) are known to work. However, Amazon Aurora is **incompatible** with load balancing enabled by default in [14.4.0](../../update/index.md#1440), and Azure Database for PostgreSQL is **not recommended** due to [performance issues](https://gitlab.com/gitlab-org/quality/reference-architectures/-/issues/61). Consul is primarily used for PostgreSQL high availability so can be ignored when using a PostgreSQL PaaS setup. However, Consul is also used optionally by Prometheus for Omnibus auto host discovery. 2. Can be optionally run on reputable third-party external PaaS Redis solutions. Google Memorystore and AWS ElastiCache are known to work. 3. Can be optionally run on reputable third-party load balancing services (LB PaaS). AWS ELB is known to work. 4. Should be run on reputable third-party object storage (storage PaaS) for cloud implementations. Google Cloud Storage and AWS S3 are known to work. diff --git a/doc/administration/reference_architectures/50k_users.md b/doc/administration/reference_architectures/50k_users.md index 316969dfacc..0d0e44e2364 100644 --- a/doc/administration/reference_architectures/50k_users.md +++ b/doc/administration/reference_architectures/50k_users.md @@ -38,7 +38,7 @@ full list of reference architectures, see <!-- Disable ordered list rule https://github.com/DavidAnson/markdownlint/blob/main/doc/Rules.md#md029---ordered-list-item-prefix --> <!-- markdownlint-disable MD029 --> -1. Can be optionally run on reputable third-party external PaaS PostgreSQL solutions. [Google Cloud SQL](https://cloud.google.com/sql/docs/postgres/high-availability#normal) and [Amazon RDS](https://aws.amazon.com/rds/) are known to work, however Azure Database for PostgreSQL is **not recommended** due to [performance issues](https://gitlab.com/gitlab-org/quality/reference-architectures/-/issues/61). Consul is primarily used for PostgreSQL high availability so can be ignored when using a PostgreSQL PaaS setup. However it is also used optionally by Prometheus for Omnibus auto host discovery. +1. Can be optionally run on reputable third-party external PaaS PostgreSQL solutions. [Google Cloud SQL](https://cloud.google.com/sql/docs/postgres/high-availability#normal) and [Amazon RDS](https://aws.amazon.com/rds/) are known to work. However, Amazon Aurora is **incompatible** with load balancing enabled by default in [14.4.0](../../update/index.md#1440), and Azure Database for PostgreSQL is **not recommended** due to [performance issues](https://gitlab.com/gitlab-org/quality/reference-architectures/-/issues/61). Consul is primarily used for PostgreSQL high availability so can be ignored when using a PostgreSQL PaaS setup. However, Consul is also used optionally by Prometheus for Omnibus auto host discovery. 2. Can be optionally run on reputable third-party external PaaS Redis solutions. Google Memorystore and AWS ElastiCache are known to work. 3. Can be optionally run on reputable third-party load balancing services (LB PaaS). AWS ELB is known to work. 4. Should be run on reputable third-party object storage (storage PaaS) for cloud implementations. Google Cloud Storage and AWS S3 are known to work. @@ -153,6 +153,7 @@ As a general guidance, GitLab should run on most infrastructure such as reputabl Be aware of the following specific call outs: +- [Amazon Aurora](https://aws.amazon.com/rds/aurora/) is incompatible. See [14.4.0](../../update/index.md#1440) for more details. - [Azure Database for PostgreSQL](https://docs.microsoft.com/en-us/azure/postgresql/#:~:text=Azure%20Database%20for%20PostgreSQL%20is,high%20availability%2C%20and%20dynamic%20scalability.) is [not recommended](https://gitlab.com/gitlab-org/quality/reference-architectures/-/issues/61) due to known performance issues or missing features. - [Azure Blob Storage](https://docs.microsoft.com/en-us/azure/storage/blobs/) is recommended to be configured with [Premium accounts](https://docs.microsoft.com/en-us/azure/storage/blobs/storage-blob-block-blob-premium) to ensure consistent performance. @@ -511,7 +512,7 @@ cluster to be used with GitLab. If you're hosting GitLab on a cloud provider, you can optionally use a managed service for PostgreSQL. -A reputable provider or solution should be used for this. [Google Cloud SQL](https://cloud.google.com/sql/docs/postgres/high-availability#normal) and [Amazon RDS](https://aws.amazon.com/rds/) are known to work, however Azure Database for PostgreSQL is **not recommended** due to [performance issues](https://gitlab.com/gitlab-org/quality/reference-architectures/-/issues/61). +A reputable provider or solution should be used for this. [Google Cloud SQL](https://cloud.google.com/sql/docs/postgres/high-availability#normal) and [Amazon RDS](https://aws.amazon.com/rds/) are known to work. However, Amazon Aurora is **incompatible** with load balancing enabled by default in [14.4.0](../../update/index.md#1440), and Azure Database for PostgreSQL is **not recommended** due to [performance issues](https://gitlab.com/gitlab-org/quality/reference-architectures/-/issues/61). If you use a cloud-managed service, or provide your own PostgreSQL: @@ -1284,7 +1285,7 @@ you are using Geo, where separate database instances are required for handling r In this setup, the specs of the main database setup shouldn't need to be changed as the impact should be minimal. -A reputable provider or solution should be used for this. [Google Cloud SQL](https://cloud.google.com/sql/docs/postgres/high-availability#normal) and [Amazon RDS](https://aws.amazon.com/rds/) are known to work, however Azure Database for PostgreSQL is **not recommended** due to [performance issues](https://gitlab.com/gitlab-org/quality/reference-architectures/-/issues/61). +A reputable provider or solution should be used for this. [Google Cloud SQL](https://cloud.google.com/sql/docs/postgres/high-availability#normal) and [Amazon RDS](https://aws.amazon.com/rds/) are known to work. However, Amazon Aurora is **incompatible** with load balancing enabled by default in [14.4.0](../../update/index.md#1440), and Azure Database for PostgreSQL is **not recommended** due to [performance issues](https://gitlab.com/gitlab-org/quality/reference-architectures/-/issues/61). Examples of the above could include [Google's Cloud SQL](https://cloud.google.com/sql/docs/postgres/high-availability#normal) or [Amazon RDS](https://aws.amazon.com/rds/). @@ -1364,6 +1365,9 @@ To configure the Praefect nodes, on each one: on the page. 1. Edit the `/etc/gitlab/gitlab.rb` file to configure Praefect: + NOTE: + You can't remove the `default` entry from `virtual_storages` because [GitLab requires it](../gitaly/configure_gitaly.md#gitlab-requires-a-default-repository-storage). + <!-- Updates to example must be made at: - https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/administration/gitaly/praefect.md @@ -2222,7 +2226,7 @@ on what features you intend to use: |Object storage type|Supported by consolidated configuration?| |-------------------|----------------------------------------| -| [Backups](../../raketasks/backup_restore.md#uploading-backups-to-a-remote-cloud-storage) | No | +| [Backups](../../raketasks/backup_gitlab.md#uploading-backups-to-a-remote-cloud-storage) | No | | [Job artifacts](../job_artifacts.md#using-object-storage) including archived job logs | Yes | | [LFS objects](../lfs/index.md#storing-lfs-objects-in-remote-object-storage) | Yes | | [Uploads](../uploads.md#using-object-storage) | Yes | @@ -2342,7 +2346,7 @@ services where applicable): <!-- Disable ordered list rule https://github.com/DavidAnson/markdownlint/blob/main/doc/Rules.md#md029---ordered-list-item-prefix --> <!-- markdownlint-disable MD029 --> -1. Can be optionally run on reputable third-party external PaaS PostgreSQL solutions. [Google Cloud SQL](https://cloud.google.com/sql/docs/postgres/high-availability#normal) and [Amazon RDS](https://aws.amazon.com/rds/) are known to work, however Azure Database for PostgreSQL is **not recommended** due to [performance issues](https://gitlab.com/gitlab-org/quality/reference-architectures/-/issues/61). Consul is primarily used for PostgreSQL high availability so can be ignored when using a PostgreSQL PaaS setup. However it is also used optionally by Prometheus for Omnibus auto host discovery. +1. Can be optionally run on reputable third-party external PaaS PostgreSQL solutions. [Google Cloud SQL](https://cloud.google.com/sql/docs/postgres/high-availability#normal) and [Amazon RDS](https://aws.amazon.com/rds/) are known to work. However, Amazon Aurora is **incompatible** with load balancing enabled by default in [14.4.0](../../update/index.md#1440), and Azure Database for PostgreSQL is **not recommended** due to [performance issues](https://gitlab.com/gitlab-org/quality/reference-architectures/-/issues/61). Consul is primarily used for PostgreSQL high availability so can be ignored when using a PostgreSQL PaaS setup. However, Consul is also used optionally by Prometheus for Omnibus auto host discovery. 2. Can be optionally run on reputable third-party external PaaS Redis solutions. Google Memorystore and AWS ElastiCache are known to work. 3. Can be optionally run on reputable third-party load balancing services (LB PaaS). AWS ELB is known to work. 4. Should be run on reputable third-party object storage (storage PaaS) for cloud implementations. Google Cloud Storage and AWS S3 are known to work. diff --git a/doc/administration/reference_architectures/5k_users.md b/doc/administration/reference_architectures/5k_users.md index ef44d688f31..ef2e48baa33 100644 --- a/doc/administration/reference_architectures/5k_users.md +++ b/doc/administration/reference_architectures/5k_users.md @@ -44,7 +44,7 @@ costly-to-operate environment by using the <!-- Disable ordered list rule https://github.com/DavidAnson/markdownlint/blob/main/doc/Rules.md#md029---ordered-list-item-prefix --> <!-- markdownlint-disable MD029 --> -1. Can be optionally run on reputable third-party external PaaS PostgreSQL solutions. [Google Cloud SQL](https://cloud.google.com/sql/docs/postgres/high-availability#normal) and [Amazon RDS](https://aws.amazon.com/rds/) are known to work, however Azure Database for PostgreSQL is **not recommended** due to [performance issues](https://gitlab.com/gitlab-org/quality/reference-architectures/-/issues/61). Consul is primarily used for PostgreSQL high availability so can be ignored when using a PostgreSQL PaaS setup. However it is also used optionally by Prometheus for Omnibus auto host discovery. +1. Can be optionally run on reputable third-party external PaaS PostgreSQL solutions. [Google Cloud SQL](https://cloud.google.com/sql/docs/postgres/high-availability#normal) and [Amazon RDS](https://aws.amazon.com/rds/) are known to work. However, Amazon Aurora is **incompatible** with load balancing enabled by default in [14.4.0](../../update/index.md#1440), and Azure Database for PostgreSQL is **not recommended** due to [performance issues](https://gitlab.com/gitlab-org/quality/reference-architectures/-/issues/61). Consul is primarily used for PostgreSQL high availability so can be ignored when using a PostgreSQL PaaS setup. However, Consul is also used optionally by Prometheus for Omnibus auto host discovery. 2. Can be optionally run on reputable third-party external PaaS Redis solutions. Google Memorystore and AWS ElastiCache are known to work. 3. Can be optionally run on reputable third-party load balancing services (LB PaaS). AWS ELB is known to work. 4. Should be run on reputable third-party object storage (storage PaaS) for cloud implementations. Google Cloud Storage and AWS S3 are known to work. @@ -156,13 +156,14 @@ As a general guidance, GitLab should run on most infrastructure such as reputabl Be aware of the following specific call outs: +- [Amazon Aurora](https://aws.amazon.com/rds/aurora/) is incompatible. See [14.4.0](../../update/index.md#1440) for more details. - [Azure Database for PostgreSQL](https://docs.microsoft.com/en-us/azure/postgresql/#:~:text=Azure%20Database%20for%20PostgreSQL%20is,high%20availability%2C%20and%20dynamic%20scalability.) is [not recommended](https://gitlab.com/gitlab-org/quality/reference-architectures/-/issues/61) due to known performance issues or missing features. - [Azure Blob Storage](https://docs.microsoft.com/en-us/azure/storage/blobs/) is recommended to be configured with [Premium accounts](https://docs.microsoft.com/en-us/azure/storage/blobs/storage-blob-block-blob-premium) to ensure consistent performance. ### Praefect PostgreSQL It's worth noting that at this time [Praefect requires its own database server](../gitaly/praefect.md#postgresql) and -that to achieve full High Availability a third-party PostgreSQL database solution will be required. +that to achieve full High Availability a third-party PostgreSQL database solution is required. We hope to offer a built in solutions for these restrictions in the future but in the meantime a non HA PostgreSQL server can be set up via Omnibus GitLab, which the above specs reflect. Refer to the following issues for more information: [`omnibus-gitlab#5919`](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/5919) & [`gitaly#3398`](https://gitlab.com/gitlab-org/gitaly/-/issues/3398). @@ -232,7 +233,7 @@ The following list includes descriptions of each server and its assigned IP: ## Configure the external load balancer -In a multi-node GitLab configuration, you'll need a load balancer to route +In a multi-node GitLab configuration, you need a load balancer to route traffic to the application servers. The specifics on which load balancer to use or its exact configuration is beyond the scope of GitLab documentation. We assume that if you're managing multi-node systems like GitLab, you already have a load @@ -245,7 +246,7 @@ This architecture has been tested and validated with [HAProxy](https://www.hapro as the load balancer. Although other load balancers with similar feature sets could also be used, those load balancers have not been validated. -The next question is how you will handle SSL in your environment. +The next question is how you handle SSL in your environment. There are several different options: - [The application node terminates SSL](#application-node-terminates-ssl). @@ -257,8 +258,8 @@ There are several different options: ### Application node terminates SSL Configure your load balancer to pass connections on port 443 as `TCP` rather -than `HTTP(S)` protocol. This will pass the connection to the application node's -NGINX service untouched. NGINX will have the SSL certificate and listen on port 443. +than `HTTP(S)` protocol. This passes the connection to the application node's +NGINX service untouched. NGINX has the SSL certificate and listen on port 443. See the [NGINX HTTPS documentation](https://docs.gitlab.com/omnibus/settings/nginx.html#enable-https) for details on managing SSL certificates and configuring NGINX. @@ -266,10 +267,10 @@ for details on managing SSL certificates and configuring NGINX. ### Load balancer terminates SSL without backend SSL Configure your load balancer to use the `HTTP(S)` protocol rather than `TCP`. -The load balancer will then be responsible for managing SSL certificates and +The load balancer is then responsible for managing SSL certificates and terminating SSL. -Since communication between the load balancer and GitLab will not be secure, +Since communication between the load balancer and GitLab is not secure, there is some additional configuration needed. See the [NGINX proxied SSL documentation](https://docs.gitlab.com/omnibus/settings/nginx.html#supporting-proxied-ssl) for details. @@ -277,12 +278,12 @@ for details. ### Load balancer terminates SSL with backend SSL Configure your load balancers to use the 'HTTP(S)' protocol rather than 'TCP'. -The load balancers will be responsible for managing SSL certificates that -end users will see. +The load balancers are responsible for managing SSL certificates that +end users see. -Traffic will also be secure between the load balancers and NGINX in this +Traffic is also secure between the load balancers and NGINX in this scenario. There is no need to add configuration for proxied SSL since the -connection will be secure all the way. However, configuration will need to be +connection is secure all the way. However, configuration needs to be added to GitLab to configure SSL certificates. See [NGINX HTTPS documentation](https://docs.gitlab.com/omnibus/settings/nginx.html#enable-https) for details on managing SSL certificates and configuring NGINX. @@ -292,7 +293,7 @@ for details on managing SSL certificates and configuring NGINX. Ensure the external load balancer only routes to working services with built in monitoring endpoints. The [readiness checks](../../user/admin_area/monitoring/health_check.md) all require [additional configuration](../monitoring/ip_whitelist.md) -on the nodes being checked, otherwise, the external load balancer will not be able to +on the nodes being checked, otherwise, the external load balancer is not able to connect. ### Ports @@ -311,11 +312,11 @@ The basic ports to be used are shown in the table below. to pass through the `Connection` and `Upgrade` hop-by-hop headers. See the [web terminal](../integration/terminal.md) integration guide for more details. -- (*2*): When using HTTPS protocol for port 443, you will need to add an SSL +- (*2*): When using HTTPS protocol for port 443, you need to add an SSL certificate to the load balancers. If you wish to terminate SSL at the GitLab application server instead, use TCP protocol. -If you're using GitLab Pages with custom domain support you will need some +If you're using GitLab Pages with custom domain support you need some additional port configurations. GitLab Pages requires a separate virtual IP address. Configure DNS to point the `pages_external_url` from `/etc/gitlab/gitlab.rb` at the new virtual IP address. See the @@ -337,7 +338,7 @@ GitLab Pages requires a separate virtual IP address. Configure DNS to point the Some organizations have policies against opening SSH port 22. In this case, it may be helpful to configure an alternate SSH hostname that allows users -to use SSH on port 443. An alternate SSH hostname will require a new virtual IP address +to use SSH on port 443. An alternate SSH hostname requires a new virtual IP address compared to the other GitLab HTTP configuration above. Configure DNS for an alternate SSH hostname such as `altssh.gitlab.example.com`. @@ -359,7 +360,7 @@ such as connections to [PgBouncer](#configure-pgbouncer) and [Praefect](#configu It's a separate node from the External Load Balancer and shouldn't have any access externally. -The following IP will be used as an example: +The following IP is used as an example: - `10.6.0.40`: Internal Load Balancer @@ -433,8 +434,8 @@ You are highly encouraged to read the [Redis Sentinel](https://redis.io/topics/s before configuring Redis with GitLab to fully understand the topology and architecture. -In this section, you'll be guided through configuring an external Redis instance -to be used with GitLab. The following IPs will be used as an example: +In this section, you are guided through configuring an external Redis instance +to be used with GitLab. The following IPs are used as an example: - `10.6.0.61`: Redis Primary - `10.6.0.62`: Redis Replica 1 @@ -442,7 +443,7 @@ to be used with GitLab. The following IPs will be used as an example: ### Provide your own Redis instance -Managed Redis from cloud providers such as AWS ElastiCache will work. If these +Managed Redis from cloud providers such as AWS ElastiCache works. If these services support high availability, be sure it is **not** the Redis Cluster type. Redis version 5.0 or higher is required, as this is what ships with @@ -451,7 +452,7 @@ do not support an optional count argument to SPOP which is now required for [Merge Trains](../../ci/pipelines/merge_trains.md). Note the Redis node's IP address or hostname, port, and password (if required). -These will be necessary when configuring the +These are necessary when configuring the [GitLab application servers](#configure-gitlab-rails) later. ### Standalone Redis using Omnibus GitLab @@ -617,8 +618,8 @@ You can specify multiple roles, like sentinel and Redis, as: [roles](https://docs.gitlab.com/omnibus/roles/). These values don't have to be changed again in `/etc/gitlab/gitlab.rb` after -a failover, as the nodes will be managed by the [Sentinels](#configure-consul-and-sentinel), and even after a -`gitlab-ctl reconfigure`, they will get their configuration restored by +a failover, as the nodes are managed by the [Sentinels](#configure-consul-and-sentinel), and even after a +`gitlab-ctl reconfigure`, they get their configuration restored by the same Sentinels. Advanced [configuration options](https://docs.gitlab.com/omnibus/settings/redis.html) @@ -633,7 +634,7 @@ are supported and can be added if needed. ## Configure Consul and Sentinel Now that the Redis servers are all set up, let's configure the Sentinel -servers. The following IPs will be used as an example: +servers. The following IPs are used as an example: - `10.6.0.11`: Consul/Sentinel 1 - `10.6.0.12`: Consul/Sentinel 2 @@ -647,7 +648,7 @@ clients to report `NOAUTH Authentication required.`. To configure the Sentinel: -1. SSH in to the server that will host Consul/Sentinel. +1. SSH in to the server that hosts Consul/Sentinel. 1. [Download and install](https://about.gitlab.com/install/) the Omnibus GitLab package of your choice. Be sure to both follow _only_ installation steps 1 and 2 on the page, and to select the correct Omnibus GitLab package, with the same version @@ -776,7 +777,7 @@ run: sentinel: (pid 30098) 76832s; run: log: (pid 29704) 76850s ## Configure PostgreSQL -In this section, you'll be guided through configuring a highly available PostgreSQL +In this section, you are guided through configuring a highly available PostgreSQL cluster to be used with GitLab. ### Provide your own PostgreSQL instance @@ -784,7 +785,7 @@ cluster to be used with GitLab. If you're hosting GitLab on a cloud provider, you can optionally use a managed service for PostgreSQL. -A reputable provider or solution should be used for this. [Google Cloud SQL](https://cloud.google.com/sql/docs/postgres/high-availability#normal) and [Amazon RDS](https://aws.amazon.com/rds/) are known to work, however Azure Database for PostgreSQL is **not recommended** due to [performance issues](https://gitlab.com/gitlab-org/quality/reference-architectures/-/issues/61). +A reputable provider or solution should be used for this. [Google Cloud SQL](https://cloud.google.com/sql/docs/postgres/high-availability#normal) and [Amazon RDS](https://aws.amazon.com/rds/) are known to work. However, Amazon Aurora is **incompatible** with load balancing enabled by default in [14.4.0](../../update/index.md#1440), and Azure Database for PostgreSQL is **not recommended** due to [performance issues](https://gitlab.com/gitlab-org/quality/reference-architectures/-/issues/61). If you use a cloud-managed service, or provide your own PostgreSQL: @@ -813,7 +814,7 @@ replication and failover requires: A local PgBouncer service to be configured on each PostgreSQL node. Note that this is separate from the main PgBouncer cluster that tracks the primary. -The following IPs will be used as an example: +The following IPs are used as an example: - `10.6.0.31`: PostgreSQL primary - `10.6.0.32`: PostgreSQL secondary 1 @@ -828,8 +829,8 @@ in the second step, do not supply the `EXTERNAL_URL` value. #### PostgreSQL nodes 1. SSH in to one of the PostgreSQL nodes. -1. Generate a password hash for the PostgreSQL username/password pair. This assumes you will use the default - username of `gitlab` (recommended). The command will request a password +1. Generate a password hash for the PostgreSQL username/password pair. This assumes you use the default + username of `gitlab` (recommended). The command requests a password and confirmation. Use the value that is output by this command in the next step as the value of `<postgresql_password_hash>`: @@ -837,8 +838,8 @@ in the second step, do not supply the `EXTERNAL_URL` value. sudo gitlab-ctl pg-password-md5 gitlab ``` -1. Generate a password hash for the PgBouncer username/password pair. This assumes you will use the default - username of `pgbouncer` (recommended). The command will request a password +1. Generate a password hash for the PgBouncer username/password pair. This assumes you use the default + username of `pgbouncer` (recommended). The command requests a password and confirmation. Use the value that is output by this command in the next step as the value of `<pgbouncer_password_hash>`: @@ -846,8 +847,8 @@ in the second step, do not supply the `EXTERNAL_URL` value. sudo gitlab-ctl pg-password-md5 pgbouncer ``` -1. Generate a password hash for the PostgreSQL replication username/password pair. This assumes you will use the default - username of `gitlab_replicator` (recommended). The command will request a password +1. Generate a password hash for the PostgreSQL replication username/password pair. This assumes you use the default + username of `gitlab_replicator` (recommended). The command requests a password and a confirmation. Use the value that is output by this command in the next step as the value of `<postgresql_replication_password_hash>`: @@ -855,8 +856,8 @@ in the second step, do not supply the `EXTERNAL_URL` value. sudo gitlab-ctl pg-password-md5 gitlab_replicator ``` -1. Generate a password hash for the Consul database username/password pair. This assumes you will use the default - username of `gitlab-consul` (recommended). The command will request a password +1. Generate a password hash for the Consul database username/password pair. This assumes you use the default + username of `gitlab-consul` (recommended). The command requests a password and confirmation. Use the value that is output by this command in the next step as the value of `<consul_password_hash>`: @@ -935,7 +936,7 @@ in the second step, do not supply the `EXTERNAL_URL` value. # END user configuration ``` -PostgreSQL, with Patroni managing its failover, will default to use `pg_rewind` by default to handle conflicts. +PostgreSQL, with Patroni managing its failover, defaults to use `pg_rewind` by default to handle conflicts. Like most failover handling methods, this has a small chance of leading to data loss. Learn more about the various [Patroni replication methods](../postgresql/replication_and_failover.md#selecting-the-appropriate-patroni-replication-method). @@ -987,7 +988,7 @@ If the 'State' column for any node doesn't say "running", check the Now that the PostgreSQL servers are all set up, let's configure PgBouncer for tracking and handling reads/writes to the primary database. -The following IPs will be used as an example: +The following IPs are used as an example: - `10.6.0.21`: PgBouncer 1 - `10.6.0.22`: PgBouncer 2 @@ -1111,9 +1112,9 @@ The recommended cluster setup includes the following components: - 1 Praefect PostgreSQL node: Database server for Praefect. A third-party solution is required for Praefect database connections to be made highly available. - 1 load balancer: A load balancer is required for Praefect. The - [internal load balancer](#configure-the-internal-load-balancer) will be used. + [internal load balancer](#configure-the-internal-load-balancer) is used. -This section will detail how to configure the recommended standard setup in order. +This section details how to configure the recommended standard setup in order. For more advanced setups refer to the [standalone Gitaly Cluster documentation](../gitaly/praefect.md). ### Configure Praefect PostgreSQL @@ -1125,7 +1126,7 @@ A built-in solution is being [worked on](https://gitlab.com/gitlab-org/omnibus-g #### Praefect non-HA PostgreSQL standalone using Omnibus GitLab -The following IPs will be used as an example: +The following IPs are used as an example: - `10.6.0.141`: Praefect PostgreSQL @@ -1137,8 +1138,8 @@ in the second step, do not supply the `EXTERNAL_URL` value. 1. SSH in to the Praefect PostgreSQL node. 1. Create a strong password to be used for the Praefect PostgreSQL user. Take note of this password as `<praefect_postgresql_password>`. -1. Generate the password hash for the Praefect PostgreSQL username/password pair. This assumes you will use the default - username of `praefect` (recommended). The command will request the password `<praefect_postgresql_password>` +1. Generate the password hash for the Praefect PostgreSQL username/password pair. This assumes you use the default + username of `praefect` (recommended). The command requests the password `<praefect_postgresql_password>` and confirmation. Use the value that is output by this command in the next step as the value of `<praefect_postgresql_password_hash>`: @@ -1215,13 +1216,13 @@ you are using Geo, where separate database instances are required for handling r In this setup, the specs of the main database setup shouldn't need to be changed as the impact should be minimal. -A reputable provider or solution should be used for this. [Google Cloud SQL](https://cloud.google.com/sql/docs/postgres/high-availability#normal) and [Amazon RDS](https://aws.amazon.com/rds/) are known to work, however Azure Database for PostgreSQL is **not recommended** due to [performance issues](https://gitlab.com/gitlab-org/quality/reference-architectures/-/issues/61). +A reputable provider or solution should be used for this. [Google Cloud SQL](https://cloud.google.com/sql/docs/postgres/high-availability#normal) and [Amazon RDS](https://aws.amazon.com/rds/) are known to work. However, Amazon Aurora is **incompatible** with load balancing enabled by default in [14.4.0](../../update/index.md#1440), and Azure Database for PostgreSQL is **not recommended** due to [performance issues](https://gitlab.com/gitlab-org/quality/reference-architectures/-/issues/61). Once the database is set up, follow the [post configuration](#praefect-postgresql-post-configuration). #### Praefect PostgreSQL post-configuration -After the Praefect PostgreSQL server has been set up, you'll then need to configure the user and database for Praefect to use. +After the Praefect PostgreSQL server has been set up, you then need to configure the user and database for Praefect to use. We recommend the user be named `praefect` and the database `praefect_production`, and these can be configured as standard in PostgreSQL. The password for the user is the same as the one you configured earlier as `<praefect_postgresql_password>`. @@ -1274,12 +1275,12 @@ Praefect requires several secret tokens to secure communications across the Clus Gitaly Cluster nodes are configured in Praefect via a `virtual storage`. Each storage contains the details of each Gitaly node that makes up the cluster. Each storage is also given a name -and this name is used in several areas of the configuration. In this guide, the name of the storage will be +and this name is used in several areas of the configuration. In this guide, the name of the storage is `default`. Also, this guide is geared towards new installs, if upgrading an existing environment to use Gitaly Cluster, you may need to use a different name. Refer to the [Praefect documentation](../gitaly/praefect.md#praefect) for more information. -The following IPs will be used as an example: +The following IPs are used as an example: - `10.6.0.131`: Praefect 1 - `10.6.0.132`: Praefect 2 @@ -1293,6 +1294,9 @@ To configure the Praefect nodes, on each one: on the page. 1. Edit the `/etc/gitlab/gitlab.rb` file to configure Praefect: + NOTE: + You can't remove the `default` entry from `virtual_storages` because [GitLab requires it](../gitaly/configure_gitaly.md#gitlab-requires-a-default-repository-storage). + <!-- Updates to example must be made at: - https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/administration/gitaly/praefect.md @@ -1429,7 +1433,7 @@ For configuring Gitaly you should note the following: - `git_data_dirs` should be configured to reflect the storage path for the specific Gitaly node - `auth_token` should be the same as `praefect_internal_token` -The following IPs will be used as an example: +The following IPs are used as an example: - `10.6.0.91`: Gitaly 1 - `10.6.0.92`: Gitaly 2 @@ -1811,7 +1815,7 @@ On each node perform the following: ``` 1. Specify the necessary NFS mounts in `/etc/fstab`. - The exact contents of `/etc/fstab` will depend on how you chose + The exact contents of `/etc/fstab` depends on how you chose to configure your NFS server. See the [NFS documentation](../nfs.md) for examples and the various options. @@ -1827,9 +1831,9 @@ On each node perform the following: on the page. 1. Create or edit `/etc/gitlab/gitlab.rb` and use the following configuration. To maintain uniformity of links across nodes, the `external_url` - on the application server should point to the external URL that users will use + on the application server should point to the external URL that users use to access GitLab. This would be the URL of the [external load balancer](#configure-the-external-load-balancer) - which will route traffic to the GitLab application server: + which routes traffic to the GitLab application server: ```ruby external_url 'https://gitlab.example.com' @@ -1999,7 +2003,7 @@ On each node perform the following: When you specify `https` in the `external_url`, as in the previous example, GitLab expects that the SSL certificates are in `/etc/gitlab/ssl/`. If the -certificates aren't present, NGINX will fail to start. For more information, see +certificates aren't present, NGINX fails to start. For more information, see the [NGINX documentation](https://docs.gitlab.com/omnibus/settings/nginx.html#enable-https). ### GitLab Rails post-configuration @@ -2141,7 +2145,7 @@ on what features you intend to use: |Object storage type|Supported by consolidated configuration?| |-------------------|----------------------------------------| -| [Backups](../../raketasks/backup_restore.md#uploading-backups-to-a-remote-cloud-storage) | No | +| [Backups](../../raketasks/backup_gitlab.md#uploading-backups-to-a-remote-cloud-storage) | No | | [Job artifacts](../job_artifacts.md#using-object-storage) including archived job logs | Yes | | [LFS objects](../lfs/index.md#storing-lfs-objects-in-remote-object-storage) | Yes | | [Uploads](../uploads.md#using-object-storage) | Yes | @@ -2260,7 +2264,7 @@ services where applicable): <!-- Disable ordered list rule https://github.com/DavidAnson/markdownlint/blob/main/doc/Rules.md#md029---ordered-list-item-prefix --> <!-- markdownlint-disable MD029 --> -1. Can be optionally run on reputable third-party external PaaS PostgreSQL solutions. [Google Cloud SQL](https://cloud.google.com/sql/docs/postgres/high-availability#normal) and [Amazon RDS](https://aws.amazon.com/rds/) are known to work, however Azure Database for PostgreSQL is **not recommended** due to [performance issues](https://gitlab.com/gitlab-org/quality/reference-architectures/-/issues/61). Consul is primarily used for PostgreSQL high availability so can be ignored when using a PostgreSQL PaaS setup. However it is also used optionally by Prometheus for Omnibus auto host discovery. +1. Can be optionally run on reputable third-party external PaaS PostgreSQL solutions. [Google Cloud SQL](https://cloud.google.com/sql/docs/postgres/high-availability#normal) and [Amazon RDS](https://aws.amazon.com/rds/) are known to work. However, Amazon Aurora is **incompatible** with load balancing enabled by default in [14.4.0](../../update/index.md#1440), and Azure Database for PostgreSQL is **not recommended** due to [performance issues](https://gitlab.com/gitlab-org/quality/reference-architectures/-/issues/61). Consul is primarily used for PostgreSQL high availability so can be ignored when using a PostgreSQL PaaS setup. However, Consul is also used optionally by Prometheus for Omnibus auto host discovery. 2. Can be optionally run on reputable third-party external PaaS Redis solutions. Google Memorystore and AWS ElastiCache are known to work. 3. Can be optionally run on reputable third-party load balancing services (LB PaaS). AWS ELB is known to work. 4. Should be run on reputable third-party object storage (storage PaaS) for cloud implementations. Google Cloud Storage and AWS S3 are known to work. diff --git a/doc/administration/reference_architectures/index.md b/doc/administration/reference_architectures/index.md index d9fa6c7bdd9..e996acb1157 100644 --- a/doc/administration/reference_architectures/index.md +++ b/doc/administration/reference_architectures/index.md @@ -296,7 +296,7 @@ The following table details the cost to run the different reference architecture NOTE: The following lists are non exhaustive. Generally, other cloud providers not listed -here will likely work with the same specs, but this hasn't been validated. +here likely work with the same specs, but this hasn't been validated. Additionally, when it comes to other cloud provider services not listed here, it's advised to be cautious as each implementation can be notably different and should be tested thoroughly before production use. @@ -366,9 +366,9 @@ Additionally, the following cloud provider services are validated and supported The following specific cloud provider services have been found to have issues in terms of either functionality or performance. As such, they either have caveats that should be considered or are not recommended: +- [Amazon Aurora](https://aws.amazon.com/rds/aurora/) is incompatible. See [14.4.0](../../update/index.md#1440) for more details. - [Azure Blob Storage](https://azure.microsoft.com/en-gb/services/storage/blobs/) has been found to have performance limits that can impact production use at certain times. For larger Reference Architectures the service may not be sufficient for production use and an alternative is recommended for use instead. - [Azure Database for PostgreSQL Server](https://azure.microsoft.com/en-gb/services/postgresql/#overview) (Single / Flexible) is not recommended for use due to notable performance issues or missing functionality. -- [AWS Aurora Database](https://aws.amazon.com/rds/aurora/) is not recommended due to compatibility issues. NOTE: As a general rule we unfortunately don't recommend Azure Services at this time. @@ -389,7 +389,7 @@ most complex: As you implement these components, begin with a single server and then do backups. Only after completing the first server should you proceed to the next. -Also, not implementing extra servers for GitLab doesn't necessarily mean that you'll have +Also, not implementing extra servers for GitLab doesn't necessarily mean that you have more downtime. Depending on your needs and experience level, single servers can have more actual perceived uptime for your users. @@ -401,7 +401,7 @@ have more actual perceived uptime for your users. This solution is appropriate for many teams that have the default GitLab installation. With automatic backups of the GitLab repositories, configuration, and the database, this can be an optimal solution if you don't have strict requirements. -[Automated backups](../../raketasks/backup_restore.md#configuring-cron-to-make-daily-backups) +[Automated backups](../../raketasks/backup_gitlab.md#configuring-cron-to-make-daily-backups) is the least complex to setup. This provides a point-in-time recovery of a predetermined schedule. ### Traffic load balancer **(PREMIUM SELF)** @@ -410,7 +410,7 @@ is the least complex to setup. This provides a point-in-time recovery of a prede > - Required domain knowledge: HAProxy, shared storage, distributed systems This requires separating out GitLab into multiple application nodes with an added -[load balancer](../load_balancer.md). The load balancer will distribute traffic +[load balancer](../load_balancer.md). The load balancer distributes traffic across GitLab application nodes. Meanwhile, each application node connects to a shared file server and database systems on the back end. This way, if one of the application servers fails, the workflow is not interrupted. @@ -434,7 +434,7 @@ to any of the [available reference architectures](#available-reference-architect GitLab supports [zero-downtime upgrades](../../update/zero_downtime.md). Single GitLab nodes can be updated with only a [few minutes of downtime](../../update/index.md#upgrade-based-on-installation-method). To avoid this, we recommend to separate GitLab into several application nodes. -As long as at least one of each component is online and capable of handling the instance's usage load, your team's productivity will not be interrupted during the update. +As long as at least one of each component is online and capable of handling the instance's usage load, your team's productivity is not interrupted during the update. ### Automated database failover **(PREMIUM SELF)** @@ -459,8 +459,8 @@ that can also be promoted in case of disaster. ## Deviating from the suggested reference architectures As a general guideline, the further away you move from the Reference Architectures, -the harder it will be get support for it. With any deviation, you're introducing -a layer of complexity that will add challenges to finding out where potential +the harder it is to get support for it. With any deviation, you're introducing +a layer of complexity that adds challenges to finding out where potential issues might lie. The reference architectures use the official GitLab Linux packages (Omnibus @@ -474,7 +474,7 @@ However, it is still an additional layer and may still add some support complexi Other technologies, like [Docker swarm](https://docs.docker.com/engine/swarm/) are not officially supported, but can be implemented at your own risk. In that -case, GitLab Support will not be able to help you. +case, GitLab Support is not able to help you. ## Supported modifications for lower user count HA reference architectures diff --git a/doc/administration/reference_architectures/troubleshooting.md b/doc/administration/reference_architectures/troubleshooting.md index 2d10d05da37..6e0aaacd598 100644 --- a/doc/administration/reference_architectures/troubleshooting.md +++ b/doc/administration/reference_architectures/troubleshooting.md @@ -13,7 +13,7 @@ the [reference architectures](index.md#reference-architectures). ### S3 API compatibility issues -Not all S3 providers [are fully compatible](../../raketasks/backup_restore.md#other-s3-providers) +Not all S3 providers [are fully compatible](../../raketasks/backup_gitlab.md#other-s3-providers) with the Fog library that GitLab uses. Symptoms include: ```plaintext @@ -54,20 +54,20 @@ This can result in some of the following problems: - If GitLab is using non-secure HTTP to access the object storage, clients may generate `https->http` downgrade errors and refuse to process the redirect. The solution to this -is for GitLab to use HTTPS. LFS, for example, will generate this error: +is for GitLab to use HTTPS. LFS, for example, generates this error: ```plaintext LFS: lfsapi/client: refusing insecure redirect, https->http ``` -- Clients will need to trust the certificate authority that issued the object storage +- Clients must trust the certificate authority that issued the object storage certificate, or may return common TLS errors such as: ```plaintext x509: certificate signed by unknown authority ``` -- Clients will need network access to the object storage. Errors that might result +- Clients need network access to the object storage. Errors that might result if this access is not in place include: ```plaintext @@ -113,7 +113,7 @@ You can check if everything is correct by connecting to each server using /opt/gitlab/embedded/bin/redis-cli -h <redis-host-or-ip> -a '<redis-password>' info replication ``` -When connected to a `Primary` Redis, you will see the number of connected +When connected to a `Primary` Redis, you see the number of connected `replicas`, and a list of each with connection details: ```plaintext @@ -128,7 +128,7 @@ repl_backlog_first_byte_offset:206989083 repl_backlog_histlen:1048576 ``` -When it's a `replica`, you will see details of the primary connection and if +When it's a `replica`, you see details of the primary connection and if its `up` or `down`: ```plaintext @@ -254,7 +254,7 @@ To start a session: sudo gitlab-ctl pgb-console ``` -The password you will be prompted for is the `pgbouncer_user_password` +The password you are prompted for is the `pgbouncer_user_password` To get some basic information about the instance, run @@ -315,7 +315,7 @@ sudo gitlab-ctl tail patroni ### Consul and PostgreSQL with Patroni changes not taking effect -Due to the potential impacts, `gitlab-ctl reconfigure` only reloads Consul and PostgreSQL, it will not restart the services. However, not all changes can be activated by reloading. +Due to the potential impacts, `gitlab-ctl reconfigure` only reloads Consul and PostgreSQL, it does not restart the services. However, not all changes can be activated by reloading. To restart either service, run `gitlab-ctl restart consul` or `gitlab-ctl restart patroni` respectively. @@ -335,7 +335,7 @@ PG::ConnectionBad: ERROR: pgbouncer cannot connect to server The problem may be that your PgBouncer node's IP address is not included in the `trust_auth_cidr_addresses` setting in `/etc/gitlab/gitlab.rb` on the database nodes. -You can confirm that this is the issue by checking the PostgreSQL log on the master +You can confirm that this is the issue by checking the PostgreSQL log on the primary database node. If you see the following error then `trust_auth_cidr_addresses` is the problem. diff --git a/doc/administration/reply_by_email_postfix_setup.md b/doc/administration/reply_by_email_postfix_setup.md index 2652ad20329..0fbb4562995 100644 --- a/doc/administration/reply_by_email_postfix_setup.md +++ b/doc/administration/reply_by_email_postfix_setup.md @@ -6,7 +6,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Set up Postfix for incoming email **(FREE SELF)** -This document will take you through the steps of setting up a basic Postfix mail +This document takes you through the steps of setting up a basic Postfix mail server with IMAP authentication on Ubuntu, to be used with [incoming email](incoming_email.md). The instructions make the assumption that you are using the email address `incoming@gitlab.example.com`, that is, username `incoming` on host `gitlab.example.com`. Don't forget to change it to your actual host when executing the example code snippets. @@ -46,7 +46,7 @@ The instructions make the assumption that you are using the email address `incom sudo passwd incoming ``` - Be sure not to forget this, you'll need it later. + Be sure not to forget this, you will need it later. ## Test the out-of-the-box setup diff --git a/doc/administration/repository_checks.md b/doc/administration/repository_checks.md index c2233f70a9a..8cfefdb9b56 100644 --- a/doc/administration/repository_checks.md +++ b/doc/administration/repository_checks.md @@ -1,8 +1,7 @@ --- -stage: Create +stage: Systems group: Gitaly -info: "To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments" -type: reference +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- # Repository checks **(FREE SELF)** diff --git a/doc/administration/repository_storage_paths.md b/doc/administration/repository_storage_paths.md index 3acaad8231e..f001ba35bca 100644 --- a/doc/administration/repository_storage_paths.md +++ b/doc/administration/repository_storage_paths.md @@ -1,8 +1,7 @@ --- -stage: Create +stage: Systems group: Gitaly info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments -type: reference, howto --- # Repository storage **(FREE SELF)** @@ -169,7 +168,6 @@ NOTE: If all storage weights are `0` (for example, when `default` does not exist), GitLab attempts to create new repositories on `default`, regardless of the configuration or if `default` exists. See [the tracking issue](https://gitlab.com/gitlab-org/gitlab/-/issues/36175) for more information. -information. ## Move repositories diff --git a/doc/administration/repository_storage_types.md b/doc/administration/repository_storage_types.md index 1a593c8c6bd..7ce629b5d71 100644 --- a/doc/administration/repository_storage_types.md +++ b/doc/administration/repository_storage_types.md @@ -1,8 +1,7 @@ --- -stage: Create +stage: Systems group: Gitaly info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments -type: reference, howto --- # Repository storage types **(FREE SELF)** diff --git a/doc/administration/restart_gitlab.md b/doc/administration/restart_gitlab.md index 06b6cbd4271..6ac031ea6bd 100644 --- a/doc/administration/restart_gitlab.md +++ b/doc/administration/restart_gitlab.md @@ -31,7 +31,7 @@ GitLab Rails application (Puma) as well as the other components, like: ### Omnibus GitLab restart There may be times in the documentation where you are asked to _restart_ -GitLab. In that case, you need to run the following command: +GitLab. To restart, run the following command: ```shell sudo gitlab-ctl restart diff --git a/doc/administration/server_hooks.md b/doc/administration/server_hooks.md index a83a17d6d52..1a47dc4ccf2 100644 --- a/doc/administration/server_hooks.md +++ b/doc/administration/server_hooks.md @@ -1,8 +1,7 @@ --- -stage: Create +stage: Systems group: Gitaly info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments -type: reference, howto disqus_identifier: 'https://docs.gitlab.com/ee/administration/custom_hooks.html' --- @@ -27,9 +26,9 @@ alternatives to server hooks include: [Geo](geo/index.md) doesn't replicate server hooks to secondary nodes. -## Create a server hook for a single repository +## Create server hooks for a repository -To create a server hook for a single repository: +To create server hooks for a repository: 1. On the top bar, select **Menu > Admin**. 1. Go to **Overview > Projects** and select the project you want to add a server hook to. @@ -41,16 +40,21 @@ To create a server hook for a single repository: - For Omnibus GitLab installations, the path is usually `/var/opt/gitlab/git-data/repositories/<group>/<project>.git`. - For an installation from source, the path is usually `/home/git/repositories/<group>/<project>.git`. 1. On the file system, create a new directory in the correct location called `custom_hooks`. -1. In the new `custom_hooks` directory, create a file with a name that matches the hook type. For example, for a - `pre-receive` server hook, the filename should be `pre-receive` with no extension. -1. Make the server hook file executable and ensure that it's owned by the Git user. +1. In the new `custom_hooks` directory: + - To create a single server hook, create a file with a name that matches the hook type. For example, for a + `pre-receive` server hook, the filename should be `pre-receive` with no extension. + - To create many server hooks, create a directory for the hooks that matches the hook type. For example, for a + `pre-receive` server hook, the directory name should be `pre-receive.d`. Put the files for the hook in that directory. +1. Make the server hook files executable and ensure that they are owned by the Git user. 1. Write the code to make the server hook function as expected. Server hooks can be in any programming language. Ensure the [shebang](https://en.wikipedia.org/wiki/Shebang_(Unix)) at the top reflects the language type. For example, if the script is in Ruby the shebang is probably `#!/usr/bin/env ruby`. +1. Make the hook file executable, ensure that it's owned by the Git user, and ensure it does not match the backup file + pattern (`*~`). If the server hook code is properly implemented, it should execute when the Git hook is next triggered. -## Create a global server hook for all repositories +## Create global server hooks for all repositories To create a Git hook that applies to all repositories, set a global server hook. The default global server hook directory is in the GitLab Shell directory. Any server hook added there applies to all repositories, including: @@ -80,8 +84,11 @@ To use a different directory for global server hooks, set `custom_hooks_dir` in To create a global server hook for all repositories: 1. On the GitLab server, go to the configured global server hook directory. -1. Create a new directory in this location called `pre-receive.d`, `post-receive.d`, or `update.d`, depending on the type - of server hook. Any other names are ignored. +1. In the configured global server hook directory: + - To create a single server hook, create a file with a name that matches the hook type. For example, for a + `pre-receive` server hook, the filename should be `pre-receive` with no extension. + - To create many server hooks, create a directory for the hooks that matches the hook type. For example, for a + `pre-receive` server hook, the directory name should be `pre-receive.d`. Put the files for the hook in that directory. 1. Inside this new directory, add your server hook. Server hooks can be in any programming language. Ensure the [shebang](https://en.wikipedia.org/wiki/Shebang_(Unix)) at the top reflects the language type. For example, if the script is in Ruby the shebang is probably `#!/usr/bin/env ruby`. diff --git a/doc/administration/sidekiq.md b/doc/administration/sidekiq.md index 528ecf12df9..fc24c764330 100644 --- a/doc/administration/sidekiq.md +++ b/doc/administration/sidekiq.md @@ -6,19 +6,92 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Configure an external Sidekiq instance **(FREE SELF)** -You can configure an external Sidekiq instance by using the Sidekiq that's -bundled in the GitLab package. Sidekiq requires connection to the Redis, +You can configure an external Sidekiq instance by using the Sidekiq that's bundled in the GitLab package. Sidekiq requires connection to the Redis, PostgreSQL, and Gitaly instances. -## Required configuration +## Configure TCP access for PostgreSQL, Gitaly, and Redis -To configure Sidekiq: +By default, GitLab uses UNIX sockets and is not set up to communicate via TCP. To change this: + +1. Edit the `/etc/gitlab/gitlab.rb` file on your GitLab instance, and add the following: + + ```ruby + + ## PostgreSQL + + # Replace POSTGRESQL_PASSWORD_HASH with a generated md5 value + postgresql['sql_user_password'] = 'POSTGRESQL_PASSWORD_HASH' + postgresql['listen_address'] = '0.0.0.0' + postgresql['port'] = 5432 + + # Add the Sidekiq nodes to PostgreSQL's trusted addresses. + # In the following example, 10.10.1.30/32 is the private IP + # of the Sidekiq server. + postgresql['md5_auth_cidr_addresses'] = %w(127.0.0.1/32 10.10.1.30/32) + postgresql['trust_auth_cidr_addresses'] = %w(127.0.0.1/32 10.10.1.30/32) + + ## Gitaly + + # Make Gitaly accept connections on all network interfaces + gitaly['listen_addr'] = "0.0.0.0:8075" + ## Set up the Gitaly token as a form of authentication since you are accessing Gitaly over the network + ## https://docs.gitlab.com/ee/administration/gitaly/configure_gitaly.html#about-the-gitaly-token + gitaly['auth_token'] = 'abc123secret' + praefect['auth_token'] = 'abc123secret' + gitlab_rails['gitaly_token'] = 'abc123secret' + + ## Redis configuration + + redis['bind'] = '0.0.0.0' + redis['port'] = 6379 + # Password to Authenticate Redis + redis['password'] = 'redis-password-goes-here' + gitlab_rails['redis_password'] = 'redis-password-goes-here' + + gitlab_rails['auto_migrate'] = false + ``` + +1. Run `reconfigure`: + + ```shell + sudo gitlab-ctl reconfigure + ``` + +1. Restart the `PostgreSQL` server: + + ```shell + sudo gitlab-ctl restart postgresql + ``` + +1. After the restart, set `auto_migrate` to `true` or comment to use the default settings: + + ```ruby + gitlab_rails['auto_migrate'] = true + ``` + +1. Run `reconfigure` again: + + ```shell + sudo gitlab-ctl reconfigure + ``` + +## Set up Sidekiq instance 1. SSH into the Sidekiq server. + +1. Confirm that you can access the PostgreSQL, Gitaly, and Redis ports: + + ```shell + telnet <GitLab host> 5432 # PostgreSQL + telnet <GitLab host> 8075 # Gitaly + telnet <GitLab host> 6379 # Redis + ``` + 1. [Download and install](https://about.gitlab.com/install/) the Omnibus GitLab package using steps 1 and 2. **Do not complete any other steps.** -1. Edit `/etc/gitlab/gitlab.rb` with the following information and make sure - to replace with your values: + +1. Copy the `/etc/gitlab/gitlab.rb` file from the GitLab instance and add the following settings. Make sure + to replace them with your values: <!-- Updates to example must be made at: @@ -59,16 +132,25 @@ Updates to example must be made at: ## external_url 'https://gitlab.example.com' + # Configure the gitlab-shell API callback URL. Without this, `git push` will + # fail. This can be your 'front door' GitLab URL or an internal load + # balancer. + gitlab_rails['internal_api_url'] = 'GITLAB_URL' + gitlab_shell['secret_token'] = 'SHELL_TOKEN' + ######################################## #### Redis ### ######################################## - ## Must be the same in every sentinel node - redis['master_name'] = 'gitlab-redis' - + ## Must be the same in every sentinel node. + redis['master_name'] = 'gitlab-redis' # Required if you have setup redis cluster ## The same password for Redis authentication you set up for the master node. redis['master_password'] = '<redis_master_password>' + ### If redis is running on the main Gitlab instance and you have opened the TCP port as above add the following + gitlab_rails['redis_host'] = '<gitlab_host>' + gitlab_rails['redis_port'] = 6379 + ####################################### ### Gitaly ### ####################################### @@ -77,8 +159,8 @@ Updates to example must be made at: ## https://docs.gitlab.com/ee/administration/gitaly/configure_gitaly.html#about-the-gitaly-token git_data_dirs({ "default" => { - "gitaly_address" => "tcp://gitaly:8075", - "gitaly_token" => "<gitaly_token>" + "gitaly_address" => "tcp://<gitlab_host>:8075", + "gitaly_token" => "<gitaly_token>" } }) @@ -90,12 +172,6 @@ Updates to example must be made at: gitlab_rails['db_host'] = '<database_host>' gitlab_rails['db_port'] = '5432' gitlab_rails['db_password'] = '<database_password>' - - # Add the Sidekiq nodes to PostgreSQL's trusted addresses. - # In the following example, 10.10.1.30/32 is the private IP - # of the Sidekiq server. - postgresql['trust_auth_cidr_addresses'] = %w(127.0.0.1/32 10.10.1.30/32) - ## Prevent database migrations from running on upgrade automatically gitlab_rails['auto_migrate'] = false @@ -112,13 +188,15 @@ Updates to example must be made at: sidekiq['max_concurrency'] = 10 ``` +1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the GitLab instance and replace the file in the Sidekiq instance. + 1. Reconfigure GitLab: ```shell sudo gitlab-ctl reconfigure ``` -1. Restart the Sidekiq nodes after completing the process and finishing the database migrations. +1. Restart the Sidekiq instance after completing the process and finishing the database migrations. ## Configure multiple Sidekiq nodes with shared storage @@ -191,6 +269,26 @@ To configure the metrics server: sudo gitlab-ctl reconfigure ``` +### Enable HTTPS + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/364771) in GitLab 15.2. + +To serve metrics via HTTPS instead of HTTP, enable TLS in the exporter settings: + +1. Edit `/etc/gitlab/gitlab.rb` to add (or find and uncomment) the following lines: + + ```ruby + sidekiq['exporter_tls_enabled'] = true + sidekiq['exporter_tls_cert_path'] = "/path/to/certificate.pem" + sidekiq['exporter_tls_key_path'] = "/path/to/private-key.pem" + ``` + +1. Save the file and [reconfigure GitLab](restart_gitlab.md#omnibus-gitlab-reconfigure) + for the changes to take effect. + +When TLS is enabled, the same `port` and `address` are used as described above. +The metrics server cannot serve both HTTP and HTTPS at the same time. + ## Configure health checks If you use health check probes to observe Sidekiq, enable the Sidekiq health check server. @@ -198,7 +296,7 @@ To make health checks available from `localhost:8092`: 1. Edit `/etc/gitlab/gitlab.rb`: - ```ruby + ```ruby sidekiq['health_checks_enabled'] = true sidekiq['health_checks_listen_address'] = "localhost" sidekiq['health_checks_listen_port'] = "8092" diff --git a/doc/administration/smime_signing_email.md b/doc/administration/smime_signing_email.md index 943fd91c885..c9647129104 100644 --- a/doc/administration/smime_signing_email.md +++ b/doc/administration/smime_signing_email.md @@ -24,7 +24,7 @@ files must be provided: - Only RSA keys are supported. Optionally, you can also provide a bundle of CA certs (PEM-encoded) to be -included on each signature. This will typically be an intermediate CA. +included on each signature. This is typically an intermediate CA. WARNING: Be mindful of the access levels for your private keys and visibility to @@ -44,7 +44,7 @@ third parties. 1. Save the file and [reconfigure GitLab](restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. -The key needs to be readable by the GitLab system user (`git` by default). +The key must be readable by the GitLab system user (`git` by default). **For installations from source:** @@ -67,7 +67,7 @@ The key needs to be readable by the GitLab system user (`git` by default). 1. Save the file and [restart GitLab](restart_gitlab.md#installations-from-source) for the changes to take effect. -The key needs to be readable by the GitLab system user (`git` by default). +The key must be readable by the GitLab system user (`git` by default). ### How to convert S/MIME PKCS #12 format to PEM encoding diff --git a/doc/administration/troubleshooting/debug.md b/doc/administration/troubleshooting/debug.md index 81ca1bda5d0..d5019f1aa4a 100644 --- a/doc/administration/troubleshooting/debug.md +++ b/doc/administration/troubleshooting/debug.md @@ -1,289 +1,11 @@ --- -stage: Systems -group: Distribution -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +redirect_to: '../reference_architectures/troubleshooting.md' +remove_date: '2022-10-19' --- -# Debugging tips **(FREE SELF)** +This document was moved to [another location](../reference_architectures/troubleshooting.md). -Sometimes things don't work the way they should. Here are some tips on debugging issues out -in production. - -## Starting a Rails console session - -Troubleshooting and debugging your GitLab instance often requires a Rails console. - -Your type of GitLab installation determines how -[to start a rails console](../operations/rails_console.md). -See also: - -- [GitLab Rails Console Cheat Sheet](gitlab_rails_cheat_sheet.md). -- [Navigating GitLab via Rails console](navigating_gitlab_via_rails_console.md). - -### Enabling Active Record logging - -You can enable output of Active Record debug logging in the Rails console -session by running: - -```ruby -ActiveRecord::Base.logger = Logger.new($stdout) -``` - -This will show information about database queries triggered by any Ruby code -you may run in the console. To turn off logging again, run: - -```ruby -ActiveRecord::Base.logger = nil -``` - -### Disabling database statement timeout - -You can disable the PostgreSQL statement timeout for the current Rails console -session by running: - -```ruby -ActiveRecord::Base.connection.execute('SET statement_timeout TO 0') -``` - -This change only affects the current Rails console session and will -not be persisted in the GitLab production environment or in the next Rails -console session. - -### Output Rails console session history - -If you'd like to output your Rails console command history in a format that's -easy to copy and save for future reference, you can run: - -```ruby -puts Readline::HISTORY.to_a -``` - -## Using the Rails runner - -If you need to run some Ruby code in the context of your GitLab production -environment, you can do so using the [Rails runner](https://guides.rubyonrails.org/command_line.html#rails-runner). When executing a script file, the script must be accessible by the `git` user. - -**For Omnibus installations** - -```shell -sudo gitlab-rails runner "RAILS_COMMAND" - -# Example with a two-line Ruby script -sudo gitlab-rails runner "user = User.first; puts user.username" - -# Example with a ruby script file (make sure to use the full path) -sudo gitlab-rails runner /path/to/script.rb -``` - -**For installations from source** - -```shell -sudo -u git -H bundle exec rails runner -e production "RAILS_COMMAND" - -# Example with a two-line Ruby script -sudo -u git -H bundle exec rails runner -e production "user = User.first; puts user.username" - -# Example with a ruby script file (make sure to use the full path) -sudo -u git -H bundle exec rails runner -e production /path/to/script.rb -``` - -## Mail not working - -A common problem is that mails are not being sent for some reason. Suppose you configured -an SMTP server, but you're not seeing mail delivered. Here's how to check the settings: - -1. Run a [Rails console](../operations/rails_console.md#starting-a-rails-console-session). - -1. Look at the ActionMailer `delivery_method` to make sure it matches what you - intended. If you configured SMTP, it should say `:smtp`. If you're using - Sendmail, it should say `:sendmail`: - - ```ruby - irb(main):001:0> ActionMailer::Base.delivery_method - => :smtp - ``` - -1. If you're using SMTP, check the mail settings: - - ```ruby - irb(main):002:0> ActionMailer::Base.smtp_settings - => {:address=>"localhost", :port=>25, :domain=>"localhost.localdomain", :user_name=>nil, :password=>nil, :authentication=>nil, :enable_starttls_auto=>true} - ``` - - In the example above, the SMTP server is configured for the local machine. If this is intended, you may need to check your local mail - logs (for example, `/var/log/mail.log`) for more details. - -1. Send a test message via the console. - - ```ruby - irb(main):003:0> Notify.test_email('youremail@email.com', 'Hello World', 'This is a test message').deliver_now - ``` - - If you do not receive an email and/or see an error message, then check - your mail server settings. - -## Advanced Issues - -For more advanced issues, `gdb` is a must-have tool for debugging issues. - -### The GNU Project Debugger (GDB) - -To install on Ubuntu/Debian: - -```shell -sudo apt-get install gdb -``` - -On CentOS: - -```shell -sudo yum install gdb -``` - -<!-- vale gitlab.Spelling = NO --> - -### rbtrace - -<!-- vale gitlab.Spelling = YES --> - -GitLab 11.2 ships with [`rbtrace`](https://github.com/tmm1/rbtrace), which -allows you to trace Ruby code, view all running threads, take memory dumps, -and more. However, this is not enabled by default. To enable it, define the -`ENABLE_RBTRACE` variable to the environment. For example, in Omnibus: - -```ruby -gitlab_rails['env'] = {"ENABLE_RBTRACE" => "1"} -``` - -Then reconfigure the system and restart Puma and Sidekiq. To run this -in Omnibus, run as root: - -```ruby -/opt/gitlab/embedded/bin/ruby /opt/gitlab/embedded/bin/rbtrace -``` - -## Common Problems - -Many of the tips to diagnose issues below apply to many different situations. We'll use one -concrete example to illustrate what you can do to learn what is going wrong. - -### 502 Gateway Timeout after Unicorn spins at 100% CPU - -This error occurs when the Web server times out (default: 60 s) after not -hearing back from the Unicorn worker. If the CPU spins to 100% while this in -progress, there may be something taking longer than it should. - -To fix this issue, we first need to figure out what is happening. The -following tips are only recommended if you do not mind users being affected by -downtime. Otherwise skip to the next section. - -1. Load the problematic URL -1. Run `sudo gdb -p <PID>` to attach to the Puma process. -1. In the GDB window, type: - - ```plaintext - call (void) rb_backtrace() - ``` - -1. This forces the process to generate a Ruby backtrace. Check - `/var/log/gitlab/puma/puma_stderr.log` for the backtrace. For example, you may see: - - ```plaintext - from /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/metrics/sampler.rb:33:in `block in start' - from /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/metrics/sampler.rb:33:in `loop' - from /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/metrics/sampler.rb:36:in `block (2 levels) in start' - from /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/metrics/sampler.rb:44:in `sample' - from /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/metrics/sampler.rb:68:in `sample_objects' - from /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/metrics/sampler.rb:68:in `each_with_object' - from /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/metrics/sampler.rb:68:in `each' - from /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/metrics/sampler.rb:69:in `block in sample_objects' - from /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/metrics/sampler.rb:69:in `name' - ``` - -1. To see the current threads, run: - - ```plaintext - thread apply all bt - ``` - -1. Once you're done debugging with `gdb`, be sure to detach from the process and exit: - - ```plaintext - detach - exit - ``` - -If the Puma process terminates before you are able to run these -commands, GDB will report an error. To buy more time, you can always raise the -Puma worker timeout. For omnibus users, you can edit `/etc/gitlab/gitlab.rb` and -increase it from 60 seconds to 600: - -```ruby -gitlab_rails['env'] = { - 'GITLAB_RAILS_RACK_TIMEOUT' => 600 -} -``` - -For source installations, set the environment variable. -Refer to [Puma Worker timeout](../operations/puma.md#change-the-worker-timeout). - -[Reconfigure](../restart_gitlab.md#omnibus-gitlab-reconfigure) GitLab for the changes to take effect. - -#### Troubleshooting without affecting other users - -The previous section attached to a running Unicorn process, and this may have -undesirable effects for users trying to access GitLab during this time. If you -are concerned about affecting others during a production system, you can run a -separate Rails process to debug the issue: - -1. Log in to your GitLab account. -1. Copy the URL that is causing problems (for example, `https://gitlab.com/ABC`). -1. Create a Personal Access Token for your user (User Settings -> Access Tokens). -1. Bring up the [GitLab Rails console.](../operations/rails_console.md#starting-a-rails-console-session) -1. At the Rails console, run: - - ```ruby - app.get '<URL FROM STEP 2>/?private_token=<TOKEN FROM STEP 3>' - ``` - - For example: - - ```ruby - app.get 'https://gitlab.com/gitlab-org/gitlab-foss/-/issues/1?private_token=123456' - ``` - -1. In a new window, run `top`. It should show this Ruby process using 100% CPU. Write down the PID. -1. Follow step 2 from the previous section on using GDB. - -### GitLab: API is not accessible - -This often occurs when GitLab Shell attempts to request authorization via the -[internal API](../../development/internal_api/index.md) (for example, `http://localhost:8080/api/v4/internal/allowed`), and -something in the check fails. There are many reasons why this may happen: - -1. Timeout connecting to a database (for example, PostgreSQL or Redis) -1. Error in Git hooks or push rules -1. Error accessing the repository (for example, stale NFS handles) - -To diagnose this problem, try to reproduce the problem and then see if there -is a Unicorn worker that is spinning via `top`. Try to use the `gdb` -techniques above. In addition, using `strace` may help isolate issues: - -```shell -strace -ttTfyyy -s 1024 -p <PID of puma worker> -o /tmp/puma.txt -``` - -If you cannot isolate which Unicorn worker is the issue, try to run `strace` -on all the Unicorn workers to see where the -[`/internal/allowed`](../../development/internal_api/index.md) endpoint gets stuck: - -```shell -ps auwx | grep puma | awk '{ print " -p " $2}' | xargs strace -ttTfyyy -s 1024 -o /tmp/puma.txt -``` - -The output in `/tmp/puma.txt` may help diagnose the root cause. - -## More information - -- [Debugging Stuck Ruby Processes](https://newrelic.com/blog/best-practices/debugging-stuck-ruby-processes-what-to-do-before-you-kill-9) -- [Cheat sheet of using GDB and Ruby processes](gdb-stuck-ruby.txt) +<!-- This redirect file can be deleted after 2022-10-19. --> +<!-- Redirects that point to other docs in the same project expire in three months. --> +<!-- Redirects that point to docs in a different project or site (for example, link is not relative and starts with `https:`) expire in one year. --> +<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html --> diff --git a/doc/administration/troubleshooting/defcon.md b/doc/administration/troubleshooting/defcon.md index 292b4b13967..f2554f523f0 100644 --- a/doc/administration/troubleshooting/defcon.md +++ b/doc/administration/troubleshooting/defcon.md @@ -1,35 +1,11 @@ --- -stage: Systems -group: Distribution -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments -type: reference +redirect_to: '../../ci/troubleshooting.md#disaster-recovery' +remove_date: '2022-10-04' --- -# Disaster recovery **(FREE SELF)** +This document was moved to [another location](../../ci/troubleshooting.md#disaster-recovery). -This document describes a feature that allows you to disable some important but computationally -expensive parts of the application to relieve stress on the database during an ongoing downtime. - -## `ci_queueing_disaster_recovery_disable_fair_scheduling` - -This feature flag, if temporarily enabled, disables fair scheduling on shared runners. -This can help to reduce system resource usage on the `jobs/request` endpoint -by significantly reducing the computations being performed. - -Side effects: - -- In case of a large backlog of jobs, the jobs are processed in the order - they were put in the system, instead of balancing the jobs across many projects. - -## `ci_queueing_disaster_recovery_disable_quota` - -This feature flag, if temporarily enabled, disables enforcing CI/CD minutes quota -on shared runners. This can help to reduce system resource usage on the -`jobs/request` endpoint by significantly reducing the computations being -performed. - -Side effects: - -- Projects which are out of quota will be run. This affects - only jobs created during the last hour, as prior jobs are canceled - by a periodic background worker (`StuckCiJobsWorker`). +<!-- This redirect file can be deleted after <2022-10-04>. --> +<!-- Redirects that point to other docs in the same project expire in three months. --> +<!-- Redirects that point to docs in a different project or site (for example, link is not relative and starts with `https:`) expire in one year. --> +<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html --> diff --git a/doc/administration/troubleshooting/diagnostics_tools.md b/doc/administration/troubleshooting/diagnostics_tools.md index 6055746238f..faddf12b97d 100644 --- a/doc/administration/troubleshooting/diagnostics_tools.md +++ b/doc/administration/troubleshooting/diagnostics_tools.md @@ -8,7 +8,7 @@ type: reference # Diagnostics tools **(FREE SELF)** These are some of the diagnostics tools the GitLab Support team uses during troubleshooting. -They are listed here for transparency, and they may be useful for users with experience +They are listed here for transparency, and for users with experience with troubleshooting GitLab. If you are currently having an issue with GitLab, you may want to check your [support options](https://about.gitlab.com/support/) first, before attempting to use these tools. @@ -24,6 +24,6 @@ running on. [strace-parser](https://gitlab.com/wchandler/strace-parser) is a small tool to analyze and summarize raw `strace` data. -## kubesos +## `kubesos` The [`kubesos`](https://gitlab.com/gitlab-com/support/toolbox/kubesos/) utility retrieves GitLab cluster configuration and logs from GitLab Cloud Native chart deployments. diff --git a/doc/administration/troubleshooting/gdb-stuck-ruby.txt b/doc/administration/troubleshooting/gdb-stuck-ruby.txt deleted file mode 100644 index de8d704f9e3..00000000000 --- a/doc/administration/troubleshooting/gdb-stuck-ruby.txt +++ /dev/null @@ -1,142 +0,0 @@ -# Here's the script I'll use to demonstrate - it just loops forever: - -$ cat test.rb -#!/usr/bin/env ruby - -loop do - sleep 1 -end - -# Now, I'll start the script in the background, and redirect stdout and stderr -# to /dev/null: - -$ ruby ./test.rb >/dev/null 2>/dev/null & -[1] 1343 - -# Next, I'll grab the PID of the script (1343): - -$ ps aux | grep test.rb -vagrant 1343 0.0 0.4 3884 1652 pts/0 S 14:42 0:00 ruby ./test.rb -vagrant 1345 0.0 0.2 4624 852 pts/0 S+ 14:42 0:00 grep --color=auto test.rb - -# Now I start gdb. Note that I'm using sudo here. This may or may not be -# necessary in your setup. I'd try without sudo first, and fall back to adding -# it if the next step fails: - -$ sudo gdb -GNU gdb (Ubuntu/Linaro 7.4-2012.04-0ubuntu2.1) 7.4-2012.04 -Copyright (C) 2012 Free Software Foundation, Inc. -License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> -This is free software: you are free to change and redistribute it. -There is NO WARRANTY, to the extent permitted by law. Type "show copying" -and "show warranty" for details. -This GDB was configured as "i686-linux-gnu". -For bug reporting instructions, please see: -<http://bugs.launchpad.net/gdb-linaro/>. - -# OK, now I'm in gdb, and I want to instruct it to attach to our Ruby process. -# I can do that using the 'attach' command, which takes a PID (the one we -# gathered above): - -(gdb) attach 1343 -Attaching to process 1343 -Reading symbols from /opt/vagrant_ruby/bin/ruby...done. -Reading symbols from /lib/i386-linux-gnu/librt.so.1...(no debugging symbols found)...done. -Loaded symbols for /lib/i386-linux-gnu/librt.so.1 -Reading symbols from /lib/i386-linux-gnu/libdl.so.2...(no debugging symbols found)...done. -Loaded symbols for /lib/i386-linux-gnu/libdl.so.2 -Reading symbols from /lib/i386-linux-gnu/libcrypt.so.1...(no debugging symbols found)...done. -Loaded symbols for /lib/i386-linux-gnu/libcrypt.so.1 -Reading symbols from /lib/i386-linux-gnu/libm.so.6...(no debugging symbols found)...done. -Loaded symbols for /lib/i386-linux-gnu/libm.so.6 -Reading symbols from /lib/i386-linux-gnu/libc.so.6...(no debugging symbols found)...done. -Loaded symbols for /lib/i386-linux-gnu/libc.so.6 -Reading symbols from /lib/i386-linux-gnu/libpthread.so.0...(no debugging symbols found)...done. -[Thread debugging using libthread_db enabled] -Using host libthread_db library "/lib/i386-linux-gnu/libthread_db.so.1". -Loaded symbols for /lib/i386-linux-gnu/libpthread.so.0 -Reading symbols from /lib/ld-linux.so.2...(no debugging symbols found)...done. -Loaded symbols for /lib/ld-linux.so.2 -0xb770c424 in __kernel_vsyscall () - -# Great, now gdb is attached to the target process. If the step above fails, try -# going back and running gdb under sudo. The next thing I want to do is gather -# C-level backtraces from all threads in the process. The following command -# stands for 'thread apply all backtrace': - -(gdb) t a a bt - -Thread 1 (Thread 0xb74d76c0 (LWP 1343)): -#0 0xb770c424 in __kernel_vsyscall () -#1 0xb75d7abd in select () from /lib/i386-linux-gnu/libc.so.6 -#2 0x08069c56 in rb_thread_wait_for (time=...) at eval.c:11376 -#3 0x080a20fd in rb_f_sleep (argc=1, argv=0xbf85f490) at process.c:1633 -#4 0x0805e0e2 in call_cfunc (argv=0xbf85f490, argc=1, len=-1, recv=3075299660, func=0x80a20b0 <rb_f_sleep>) - at eval.c:5778 -#5 rb_call0 (klass=3075304600, recv=3075299660, id=9393, oid=9393, argc=1, argv=0xbf85f490, body=0xb74c85a8, flags=2) - at eval.c:5928 -#6 0x0805e35d in rb_call (klass=3075304600, recv=3075299660, mid=9393, argc=1, argv=0xbf85f490, scope=1, - self=<optimized out>) at eval.c:6176 -#7 0x080651ec in rb_eval (self=3075299660, n=0xb74c4e1c) at eval.c:3521 -#8 0x0805c31c in rb_yield_0 (val=6, self=3075299660, klass=<optimized out>, flags=0, avalue=0) at eval.c:5095 -#9 0x0806a1e5 in loop_i () at eval.c:5227 -#10 0x08058dbd in rb_rescue2 (b_proc=0x806a1c0 <loop_i>, data1=0, r_proc=0, data2=0) at eval.c:5491 -#11 0x08058f28 in rb_f_loop () at eval.c:5252 -#12 0x0805e0c1 in call_cfunc (argv=0x0, argc=0, len=0, recv=3075299660, func=0x8058ef0 <rb_f_loop>) at eval.c:5781 -#13 rb_call0 (klass=3075304600, recv=3075299660, id=4121, oid=4121, argc=0, argv=0x0, body=0xb74d4dbc, flags=2) - at eval.c:5928 -#14 0x0805e35d in rb_call (klass=3075304600, recv=3075299660, mid=4121, argc=0, argv=0x0, scope=1, self=<optimized out>) - at eval.c:6176 -#15 0x080651ec in rb_eval (self=3075299660, n=0xb74c4dcc) at eval.c:3521 -#16 0x080662c6 in rb_eval (self=3075299660, n=0xb74c4de0) at eval.c:3236 -#17 0x08068ee4 in ruby_exec_internal () at eval.c:1654 -#18 0x08068f24 in ruby_exec () at eval.c:1674 -#19 0x0806b2cd in ruby_run () at eval.c:1684 -#20 0x08053771 in main (argc=2, argv=0xbf860204, envp=0xbf860210) at main.c:48 - -# C backtraces are sometimes sufficient, but often Ruby backtraces are necessary -# for debugging as well. Ruby has a built-in function called rb_backtrace() that -# we can use to dump out a Ruby backtrace, but it prints to stdout or stderr -# (depending on your Ruby version), which might have been redirected to a file -# or to /dev/null (as in our example) when the process started up. -# -# To get aroundt this, we'll do a little trick and redirect the target process's -# stdout and stderr to the current TTY, so that any output from the process -# will appear directly on our screen. - -# First, let's close the existing file descriptors for stdout and stderr -# (FD 1 and 2, respectively): -(gdb) call (void) close(1) -(gdb) call (void) close(2) - -# Next, we need to figure out the device name for the current TTY: -(gdb) shell tty -/dev/pts/0 - -# OK, now we can pass the device name obtained above to open() and attach -# file descriptors 1 and 2 back to the current TTY with these calls: - -(gdb) call (int) open("/dev/pts/0", 2, 0) -$1 = 1 -(gdb) call (int) open("/dev/pts/0", 2, 0) -$2 = 2 - -# Finally, we call rb_backtrace() in order to dump the Ruby backtrace: - -(gdb) call (void) rb_backtrace() - from ./test.rb:4:in `sleep' - from ./test.rb:4 - from ./test.rb:3:in `loop' - from ./test.rb:3 - -# And here's how we get out of gdb. Once you've quit, you'll probably want to -# clean up the stuck process by killing it. - -(gdb) quit -A debugging session is active. - - Inferior 1 [process 1343] will be detached. - -Quit anyway? (y or n) y -Detaching from program: /opt/vagrant_ruby/bin/ruby, process 1343 -$ diff --git a/doc/administration/troubleshooting/gitlab_rails_cheat_sheet.md b/doc/administration/troubleshooting/gitlab_rails_cheat_sheet.md index b57bc0a1119..0ff1afa86ed 100644 --- a/doc/administration/troubleshooting/gitlab_rails_cheat_sheet.md +++ b/doc/administration/troubleshooting/gitlab_rails_cheat_sheet.md @@ -8,9 +8,9 @@ info: To determine the technical writer assigned to the Stage/Group associated w This is the GitLab Support Team's collection of information regarding the GitLab Rails console, for use while troubleshooting. It is listed here for transparency, -and it may be useful for users with experience with these tools. If you are currently +and for users with experience with these tools. If you are currently having an issue with GitLab, it is highly recommended that you first check -our guide on [navigating our Rails console](navigating_gitlab_via_rails_console.md), +our guide on [our Rails console](../operations/rails_console.md), and your [support options](https://about.gitlab.com/support/), before attempting to use this information. @@ -80,7 +80,7 @@ Notify.test_email(u.email, "Test email for #{u.name}", 'Test email').deliver_now ## Limiting output -Adding a semicolon(`;`) and a follow-up statement at the end of a statement prevents the default implicit return output. This is useful if you are already explicitly printing details and potentially have a lot of return output: +Adding a semicolon(`;`) and a follow-up statement at the end of a statement prevents the default implicit return output. This can be used if you are already explicitly printing details and potentially have a lot of return output: ```ruby puts ActiveRecord::Base.descendants; :ok @@ -100,9 +100,9 @@ project.id # => 2537 ``` -## Open object in irb +## Open object in `irb` -Sometimes it is easier to navigate through a method if you are within the context of the object. You can shim into the namespace of `Object` to let you open `irb` within the context of any object: +Sometimes it is easier to go through a method if you are in the context of the object. You can shim into the namespace of `Object` to let you open `irb` in the context of any object: ```ruby Object.define_method(:irb) { binding.irb } @@ -311,7 +311,7 @@ end ### Bulk update push rules for _all_ projects -For example, enable **Check whether the commit author is a GitLab user** and **Do not allow users to remove Git tags with `git push`** checkboxes, and create a filter for allowing commits from a specific e-mail domain only: +For example, enable **Check whether the commit author is a GitLab user** and **Do not allow users to remove Git tags with `git push`** checkboxes, and create a filter for allowing commits from a specific email domain only: ``` ruby Project.find_each do |p| @@ -443,7 +443,7 @@ p.create_wiki ### creates the wiki project on the filesystem ## Issue boards -### In case of issue boards not loading properly and it's getting time out. We need to call the Issue Rebalancing service to fix this +### In case of issue boards not loading properly and it's getting time out. Call the Issue Rebalancing service to fix this ```ruby p = Project.find_by_full_path('<username-or-group>/<project-name>') @@ -515,9 +515,9 @@ If this all runs successfully, you see an output like the following before being => nil ``` -The exported project is located within a `.tar.gz` file in `/var/opt/gitlab/gitlab-rails/uploads/-/system/import_export_upload/export_file/`. +The exported project is located in a `.tar.gz` file in `/var/opt/gitlab/gitlab-rails/uploads/-/system/import_export_upload/export_file/`. -If this fails, [enable verbose logging](navigating_gitlab_via_rails_console.md#looking-up-database-persisted-objects), +If this fails, [enable verbose logging](../operations/rails_console.md#looking-up-database-persisted-objects), repeat the above procedure after, and report the output to [GitLab Support](https://about.gitlab.com/support/). @@ -611,6 +611,13 @@ user.skip_reconfirmation! ### Disable 2fa for single user +**In GitLab 13.5 and later:** + +Use the code under [Disable 2FA | For a single user](../../security/two_factor_authentication.md#for-a-single-user) so that the target user +is notified that 2FA has been disabled. + +**In GitLab 13.4 and earlier:** + ```ruby user = User.find_by_username('<username>') user.disable_two_factor! @@ -629,7 +636,7 @@ User.billable.count ::HistoricalData.max_historical_user_count(from: 1.year.ago.beginning_of_day, to: Time.current.end_of_day) ``` -Using cURL and jq (up to a max 100, see the [pagination docs](../../api/index.md#pagination)): +Using cURL and jq (up to a max 100, see [Pagination](../../api/index.md#pagination)): ```shell curl --silent --header "Private-Token: ********************" \ @@ -1053,6 +1060,9 @@ License.current.expires_at # Is this a trial license? License.current.trial? + +# License ID for lookup on CustomersDot +License.current.license_id ``` ### Check if a project feature is available on the instance @@ -1101,10 +1111,10 @@ License.select(&TYPE).each(&:destroy!) ### Registry Disk Space Usage by Project -As a GitLab administrator, you may need to reduce disk space consumption. +As a GitLab administrator, you may want to reduce disk space consumption. A common culprit is Docker Registry images that are no longer in use. To find the storage broken down by each project, run the following in the -[GitLab Rails console](../troubleshooting/navigating_gitlab_via_rails_console.md): +[GitLab Rails console](../operations/rails_console.md): ```ruby projects_and_size = [["project_id", "creator_id", "registry_size_bytes", "project path"]] @@ -1135,11 +1145,11 @@ end ### Run the Cleanup policy now -Find this content in the [Container Registry troubleshooting docs](../packages/container_registry.md#run-the-cleanup-policy-now). +Find this content in the [Container Registry troubleshooting documentation](../packages/container_registry.md#run-the-cleanup-policy-now). ## Sidekiq -This content has been moved to the [Troubleshooting Sidekiq docs](sidekiq.md). +This content has been moved to [Troubleshooting Sidekiq](sidekiq.md). ## Redis @@ -1275,11 +1285,16 @@ project = Project.find_by_full_path('<group/project>') Geo::RepositorySyncService.new(project).execute ``` -### Blob types newer than uploads/artifacts/LFS +### Blob types +- `Ci::JobArtifact` +- `Ci::PipelineArtifact` +- `LfsObject` +- `MergeRequestDiff` - `Packages::PackageFile` +- `PagesDeployment` - `Terraform::StateVersion` -- `MergeRequestDiff` +- `Upload` `Packages::PackageFile` is used in the following examples, but things generally work the same for the other Blob types. diff --git a/doc/administration/troubleshooting/group_saml_scim.md b/doc/administration/troubleshooting/group_saml_scim.md index c6a102e87ee..145eb5f65ae 100644 --- a/doc/administration/troubleshooting/group_saml_scim.md +++ b/doc/administration/troubleshooting/group_saml_scim.md @@ -60,6 +60,10 @@ User claims and attributes: IdP links and certificate: +NOTE: +Google Workspace displays a SHA256 fingerprint. To retrieve the SHA1 fingerprint required by GitLab for configuring SAML, download the certificate and calculate the SHA1 certificate +fingerprint. +  ## Okta @@ -109,3 +113,95 @@ Adding a user: SSO settings:  + +## SAML response example + +When a user signs in using SAML, GitLab receives a SAML response. The SAML response can be found in `production.log` logs as a base64-encoded message. Locate the response by +searching for `SAMLResponse`. The decoded SAML response is in XML format. For example: + +```xml +<?xml version="1.0" encoding="UTF-8"?> +<saml2p:Response xmlns:saml2p="urn:oasis:names:tc:SAML:2.0:protocol" xmlns:xs="http://www.w3.org/2001/XMLSchema" Destination="https://gitlabexample/-/saml/callback" ID="id4898983630840142426821432" InResponseTo="_c65e4c88-9425-4472-b42c-37f4186ac0ee" IssueInstant="2022-05-30T21:30:35.696Z" Version="2.0"> + <saml2:Issuer xmlns:saml2="urn:oasis:names:tc:SAML:2.0:assertion" Format="urn:oasis:names:tc:SAML:2.0:nameid-format:entity">http://www.okta.com/exk2y6j57o1Pdr2lI8qh7</saml2:Issuer> + <ds:Signature xmlns:ds="http://www.w3.org/2000/09/xmldsig#"> + <ds:SignedInfo> + <ds:CanonicalizationMethod Algorithm="http://www.w3.org/2001/10/xml-exc-c14n#"/> + <ds:SignatureMethod Algorithm="http://www.w3.org/2001/04/xmldsig-more#rsa-sha256"/> + <ds:Reference URI="#id4898983630840142426821432"> + <ds:Transforms> + <ds:Transform Algorithm="http://www.w3.org/2000/09/xmldsig#enveloped-signature"/> + <ds:Transform Algorithm="http://www.w3.org/2001/10/xml-exc-c14n#"> + <ec:InclusiveNamespaces xmlns:ec="http://www.w3.org/2001/10/xml-exc-c14n#" PrefixList="xs"/> + </ds:Transform> + </ds:Transforms> + <ds:DigestMethod Algorithm="http://www.w3.org/2001/04/xmlenc#sha256"/> + <ds:DigestValue>neiQvv9d3OgS4GZW8Nptp4JhjpKs3GCefibn+vmRgk4=</ds:DigestValue> + </ds:Reference> + </ds:SignedInfo> + <ds:SignatureValue>dMsQX8ivi...HMuKGhyLRvabGU6CuPrf7==</ds:SignatureValue> + <ds:KeyInfo> + <ds:X509Data> + <ds:X509Certificate>MIIDq...cptGr3vN9TQ==</ds:X509Certificate> + </ds:X509Data> + </ds:KeyInfo> + </ds:Signature> + <saml2p:Status xmlns:saml2p="urn:oasis:names:tc:SAML:2.0:protocol"> + <saml2p:StatusCode Value="urn:oasis:names:tc:SAML:2.0:status:Success"/> + </saml2p:Status> + <saml2:Assertion xmlns:saml2="urn:oasis:names:tc:SAML:2.0:assertion" xmlns:xs="http://www.w3.org/2001/XMLSchema" ID="id489" IssueInstant="2022-05-30T21:30:35.696Z" Version="2.0"> + <saml2:Issuer xmlns:saml2="urn:oasis:names:tc:SAML:2.0:assertion" Format="urn:oasis:names:tc:SAML:2.0:nameid-format:entity">http://www.okta.com/exk2y6j57o1Pdr2lI8qh7</saml2:Issuer> + <ds:Signature xmlns:ds="http://www.w3.org/2000/09/xmldsig#"> + <ds:SignedInfo> + <ds:CanonicalizationMethod Algorithm="http://www.w3.org/2001/10/xml-exc-c14n#"/> + <ds:SignatureMethod Algorithm="http://www.w3.org/2001/04/xmldsig-more#rsa-sha256"/> + <ds:Reference URI="#id48989836309833801859473359"> + <ds:Transforms> + <ds:Transform Algorithm="http://www.w3.org/2000/09/xmldsig#enveloped-signature"/> + <ds:Transform Algorithm="http://www.w3.org/2001/10/xml-exc-c14n#"> + <ec:InclusiveNamespaces xmlns:ec="http://www.w3.org/2001/10/xml-exc-c14n#" PrefixList="xs"/> + </ds:Transform> + </ds:Transforms> + <ds:DigestMethod Algorithm="http://www.w3.org/2001/04/xmlenc#sha256"/> + <ds:DigestValue>MaIsoi8hbT9gsi/mNZsz449mUuAcuEWY0q3bc4asOQs=</ds:DigestValue> + </ds:Reference> + </ds:SignedInfo> + <ds:SignatureValue>dMsQX8ivi...HMuKGhyLRvabGU6CuPrf7==<</ds:SignatureValue> + <ds:KeyInfo> + <ds:X509Data> + <ds:X509Certificate>MIIDq...cptGr3vN9TQ==</ds:X509Certificate> + </ds:X509Data> + </ds:KeyInfo> + </ds:Signature> + <saml2:Subject xmlns:saml2="urn:oasis:names:tc:SAML:2.0:assertion"> + <saml2:NameID Format="urn:oasis:names:tc:SAML:2.0:nameid-format:persistent">useremail@domain.com</saml2:NameID> + <saml2:SubjectConfirmation Method="urn:oasis:names:tc:SAML:2.0:cm:bearer"> + <saml2:SubjectConfirmationData InResponseTo="_c65e4c88-9425-4472-b42c-37f4186ac0ee" NotOnOrAfter="2022-05-30T21:35:35.696Z" Recipient="https://gitlab.example.com/-/saml/callback"/> + </saml2:SubjectConfirmation> + </saml2:Subject> + <saml2:Conditions xmlns:saml2="urn:oasis:names:tc:SAML:2.0:assertion" NotBefore="2022-05-30T21:25:35.696Z" NotOnOrAfter="2022-05-30T21:35:35.696Z"> + <saml2:AudienceRestriction> + <saml2:Audience>https://gitlab.example.com/</saml2:Audience> + </saml2:AudienceRestriction> + </saml2:Conditions> + <saml2:AuthnStatement xmlns:saml2="urn:oasis:names:tc:SAML:2.0:assertion" AuthnInstant="2022-05-30T21:30:35.696Z" SessionIndex="_c65e4c88-9425-4472-b42c-37f4186ac0ee"> + <saml2:AuthnContext> + <saml2:AuthnContextClassRef>urn:oasis:names:tc:SAML:2.0:ac:classes:PasswordProtectedTransport</saml2:AuthnContextClassRef> + </saml2:AuthnContext> + </saml2:AuthnStatement> + <saml2:AttributeStatement xmlns:saml2="urn:oasis:names:tc:SAML:2.0:assertion"> + <saml2:Attribute Name="email" NameFormat="urn:oasis:names:tc:SAML:2.0:attrname-format:unspecified"> + <saml2:AttributeValue xmlns:xs="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="xs:string">useremail@domain.com</saml2:AttributeValue> + </saml2:Attribute> + <saml2:Attribute Name="firtname" NameFormat="urn:oasis:names:tc:SAML:2.0:attrname-format:unspecified"> + <saml2:AttributeValue xmlns:xs="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="xs:string">John</saml2:AttributeValue> + </saml2:Attribute> + <saml2:Attribute Name="lastname" NameFormat="urn:oasis:names:tc:SAML:2.0:attrname-format:unspecified"> + <saml2:AttributeValue xmlns:xs="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="xs:string">Doe</saml2:AttributeValue> + </saml2:Attribute> + <saml2:Attribute Name="Groups" NameFormat="urn:oasis:names:tc:SAML:2.0:attrname-format:unspecified"> + <saml2:AttributeValue xmlns:xs="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="xs:string">Super-awesome-group</saml2:AttributeValue> + </saml2:Attribute> + </saml2:AttributeStatement> + </saml2:Assertion> +</saml2p:Response> +``` diff --git a/doc/administration/troubleshooting/index.md b/doc/administration/troubleshooting/index.md index 7d40a9e9683..7fe731bda66 100644 --- a/doc/administration/troubleshooting/index.md +++ b/doc/administration/troubleshooting/index.md @@ -17,13 +17,14 @@ installation. - [Sidekiq](sidekiq.md) - [GitLab Rails console cheat sheet](gitlab_rails_cheat_sheet.md) - [Group SAML and SCIM troubleshooting](group_saml_scim.md) **(PREMIUM SAAS)** -- [Kubernetes cheat sheet](kubernetes_cheat_sheet.md) +- [Kubernetes cheat sheet](https://docs.gitlab.com/charts/troubleshooting/kubernetes_cheat_sheet.html) - [Linux cheat sheet](linux_cheat_sheet.md) - [Parsing GitLab logs with `jq`](log_parsing.md) -- [Navigating GitLab via Rails console](navigating_gitlab_via_rails_console.md) - [Diagnostics tools](diagnostics_tools.md) -- [Debugging tips](debug.md) - [Tracing requests with correlation ID](tracing_correlation_id.md) +Some feature documentation pages also have a troubleshooting section at the end +that you can check for feature-specific help. + If you need a testing environment to troubleshoot, see the [apps for a testing environment](test_environments.md). diff --git a/doc/administration/troubleshooting/kubernetes_cheat_sheet.md b/doc/administration/troubleshooting/kubernetes_cheat_sheet.md index 0c93d1ab3ee..15ec8d5940b 100644 --- a/doc/administration/troubleshooting/kubernetes_cheat_sheet.md +++ b/doc/administration/troubleshooting/kubernetes_cheat_sheet.md @@ -1,295 +1,11 @@ --- -stage: Systems -group: Distribution -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments -type: reference +redirect_to: 'https://docs.gitlab.com/charts/troubleshooting/kubernetes_cheat_sheet.html' +remove_date: '2022-10-05' --- -# Kubernetes, GitLab, and you **(FREE SELF)** +This document was moved to [another location](https://docs.gitlab.com/charts/troubleshooting/kubernetes_cheat_sheet.html). -This is a list of useful information regarding Kubernetes that the GitLab Support -Team sometimes uses while troubleshooting. GitLab is making this public, so that anyone -can make use of the Support team's collected knowledge - -WARNING: -These commands **can alter or break** your Kubernetes components so use these at your own risk. - -If you are on a [paid tier](https://about.gitlab.com/pricing/) and are not sure how -to use these commands, it is best to [contact Support](https://about.gitlab.com/support/) -and they will assist you with any issues you are having. - -## Generic Kubernetes commands - -- How to authorize to your GCP project (can be especially useful if you have projects - under different GCP accounts): - - ```shell - gcloud auth login - ``` - -- How to access Kubernetes dashboard: - - ```shell - # for minikube: - minikube dashboard —url - # for non-local installations if access via Kubectl is configured: - kubectl proxy - ``` - -- How to SSH to a Kubernetes node and enter the container as root - <https://github.com/kubernetes/kubernetes/issues/30656>: - - - For GCP, you may find the node name and run `gcloud compute ssh node-name`. - - List containers using `docker ps`. - - Enter container using `docker exec --user root -ti container-id bash`. - -- How to copy a file from local machine to a pod: - - ```shell - kubectl cp file-name pod-name:./destination-path - ``` - -- What to do with pods in `CrashLoopBackoff` status: - - - Check logs via Kubernetes dashboard. - - Check logs via Kubectl: - - ```shell - kubectl logs <webservice pod> -c dependencies - ``` - -- How to tail all Kubernetes cluster events in real time: - - ```shell - kubectl get events -w --all-namespaces - ``` - -- How to get logs of the previously terminated pod instance: - - ```shell - kubectl logs <pod-name> --previous - ``` - - No logs are kept in the containers/pods themselves. Everything is written to `stdout`. - This is the principle of Kubernetes, read [Twelve-factor app](https://12factor.net/) - for details. - -- How to get cron jobs configured on a cluster - - ```shell - kubectl get cronjobs - ``` - - When one configures [cron-based backups](https://docs.gitlab.com/charts/backup-restore/backup.html#cron-based-backup), - you will be able to see the new schedule here. Some details about the schedules can be found - in [Running Automated Tasks with a CronJob](https://kubernetes.io/docs/tasks/job/automated-tasks-with-cron-jobs/#creating-a-cron-job) - -## GitLab-specific Kubernetes information - -- Minimal configuration that can be used to [test a Kubernetes Helm chart](https://gitlab.com/gitlab-org/charts/gitlab/-/issues/620). - -- Tailing logs of a separate pod. An example for a `webservice` pod: - - ```shell - kubectl logs gitlab-webservice-54fbf6698b-hpckq -c webservice - ``` - -- Tail and follow all pods that share a label (in this case, `webservice`): - - ```shell - # all containers in the webservice pods - kubectl logs -f -l app=webservice --all-containers=true --max-log-requests=50 - - # only the webservice containers in all webservice pods - kubectl logs -f -l app=webservice -c webservice --max-log-requests=50 - ``` - -- One can stream logs from all containers at once, similar to the Omnibus - command `gitlab-ctl tail`: - - ```shell - kubectl logs -f -l release=gitlab --all-containers=true --max-log-requests=100 - ``` - -- Check all events in the `gitlab` namespace (the namespace name can be different if you - specified a different one when deploying the Helm chart): - - ```shell - kubectl get events -w --namespace=gitlab - ``` - -- Most of the useful GitLab tools (console, Rake tasks, etc) are found in the toolbox - pod. You may enter it and run commands inside or run them from the outside. - - NOTE: - The `task-runner` pod was renamed to `toolbox` in GitLab 14.2 (charts 5.2). - - ```shell - # find the pod - kubectl --namespace gitlab get pods -lapp=toolbox - - # enter it - kubectl exec -it <toolbox-pod-name> -- bash - - # open rails console - # rails console can be also called from other GitLab pods - /srv/gitlab/bin/rails console - - # source-style commands should also work - cd /srv/gitlab && bundle exec rake gitlab:check RAILS_ENV=production - - # run GitLab check. The output can be confusing and invalid because of the specific structure of GitLab installed via helm chart - /usr/local/bin/gitlab-rake gitlab:check - - # open console without entering pod - kubectl exec -it <toolbox-pod-name> -- /srv/gitlab/bin/rails console - - # check the status of DB migrations - kubectl exec -it <toolbox-pod-name> -- /usr/local/bin/gitlab-rake db:migrate:status - ``` - - You can also use `gitlab-rake`, instead of `/usr/local/bin/gitlab-rake`. - -- Troubleshooting **Infrastructure > Kubernetes clusters** integration: - - - Check the output of `kubectl get events -w --all-namespaces`. - - Check the logs of pods within `gitlab-managed-apps` namespace. - - On the side of GitLab check Sidekiq log and Kubernetes log. When GitLab is installed - via Helm Chart, `kubernetes.log` can be found inside the Sidekiq pod. - -- How to get your initial administrator password <https://docs.gitlab.com/charts/installation/deployment.html#initial-login>: - - ```shell - # find the name of the secret containing the password - kubectl get secrets | grep initial-root - # decode it - kubectl get secret <secret-name> -ojsonpath={.data.password} | base64 --decode ; echo - ``` - -- How to connect to a GitLab PostgreSQL database. - - NOTE: - The `task-runner` pod was renamed to `toolbox` in GitLab 14.2 (charts 5.2). - - In GitLab 14.2 (chart 5.2) and later: - - ```shell - kubectl exec -it <toolbox-pod-name> -- /srv/gitlab/bin/rails dbconsole --include-password --database main - ``` - - In GitLab 14.1 (chart 5.1) and earlier: - - ```shell - kubectl exec -it <task-runner-pod-name> -- /srv/gitlab/bin/rails dbconsole --include-password - ``` - -- How to get information about Helm installation status: - - ```shell - helm status name-of-installation - ``` - -- How to update GitLab installed using Helm Chart: - - ```shell - helm repo upgrade - - # get current values and redirect them to yaml file (analogue of gitlab.rb values) - helm get values <release name> > gitlab.yaml - - # run upgrade itself - helm upgrade <release name> <chart path> -f gitlab.yaml - ``` - - After <https://gitlab.com/gitlab-org/charts/gitlab/-/issues/780> is fixed, it should - be possible to use [Updating GitLab using the Helm Chart](https://docs.gitlab.com/charts/index.html#updating-gitlab-using-the-helm-chart) - for upgrades. - -- How to apply changes to GitLab configuration: - - - Modify the `gitlab.yaml` file. - - Run the following command to apply changes: - - ```shell - helm upgrade <release name> <chart path> -f gitlab.yaml - ``` - -- How to get the manifest for a release. It can be useful because it contains the information about -all Kubernetes resources and dependent charts: - - ```shell - helm get manifest <release name> - ``` - -## Installation of minimal GitLab configuration via minikube on macOS - -This section is based on [Developing for Kubernetes with minikube](https://docs.gitlab.com/charts/development/minikube/index.html) -and [Helm](https://docs.gitlab.com/charts/installation/tools.html#helm). Refer -to those documents for details. - -- Install Kubectl via Homebrew: - - ```shell - brew install kubernetes-cli - ``` - -- Install minikube via Homebrew: - - ```shell - brew cask install minikube - ``` - -- Start minikube and configure it. If minikube cannot start, try running `minikube delete && minikube start` - and repeat the steps: - - ```shell - minikube start --cpus 3 --memory 8192 # minimum amount for GitLab to work - minikube addons enable ingress - ``` - -- Install Helm via Homebrew and initialize it: - - ```shell - brew install helm - ``` - -- Copy the [minikube minimum values YAML file](https://gitlab.com/gitlab-org/charts/gitlab/raw/master/examples/values-minikube-minimum.yaml) - to your workstation: - - ```shell - curl --output values.yaml "https://gitlab.com/gitlab-org/charts/gitlab/raw/master/examples/values-minikube-minimum.yaml" - ``` - -- Find the IP address in the output of `minikube ip` and update the YAML file with - this IP address. - -- Install the GitLab Helm Chart: - - ```shell - helm repo add gitlab https://charts.gitlab.io - helm install gitlab -f <path-to-yaml-file> gitlab/gitlab - ``` - - If you want to modify some GitLab settings, you can use the above-mentioned configuration - as a base and create your own YAML file. - -- Monitor the installation progress via `helm status gitlab` and `minikube dashboard`. - The installation could take up to 20-30 minutes depending on the amount of resources - on your workstation. - -- When all the pods show either a `Running` or `Completed` status, get the GitLab password as - described in [Initial login](https://docs.gitlab.com/charts/installation/deployment.html#initial-login), - and log in to GitLab via the UI. It will be accessible via `https://gitlab.domain` - where `domain` is the value provided in the YAML file. - -<!-- ## Troubleshooting - -Include any troubleshooting steps that you can foresee. If you know beforehand what issues -one might have when setting this up, or when something is changed, or on upgrading, it's -important to describe those, too. Think of things that may go wrong and include them here. -This is important to minimize requests for support, and to avoid doc comments with -questions that you know someone might ask. - -Each scenario can be a third-level heading, e.g. `### Getting error message X`. -If you have none to add when creating a doc, leave this section in place -but commented out to help encourage others to add to it in the future. --> +<!-- This redirect file can be deleted after 2022-10-05. --> +<!-- Redirects that point to other docs in the same project expire in three months. --> +<!-- Redirects that point to docs in a different project or site (for example, link is not relative and starts with `https:`) expire in one year. --> +<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html --> diff --git a/doc/administration/troubleshooting/linux_cheat_sheet.md b/doc/administration/troubleshooting/linux_cheat_sheet.md index 0245af39e45..66d5fb82936 100644 --- a/doc/administration/troubleshooting/linux_cheat_sheet.md +++ b/doc/administration/troubleshooting/linux_cheat_sheet.md @@ -9,7 +9,7 @@ type: reference This is the GitLab Support Team's collection of information regarding Linux, that they sometimes use while troubleshooting. It is listed here for transparency, -and it may be useful for users with experience with Linux. If you are currently +and for users with experience with Linux. If you are currently having an issue with GitLab, you may want to check your [support options](https://about.gitlab.com/support/) first, before attempting to use this information. @@ -107,6 +107,9 @@ grep -2 search_term <filename> # Search on all files in directory (recursively) grep -r search_term <directory> +# Grep namespace/project/name of a GitLab repository +grep 'fullpath' /var/opt/gitlab/git-data/repositories/@hashed/<repo hash>/.git/config + # search through *.gz files is the same except with zgrep zgrep search_term <filename> @@ -126,6 +129,7 @@ history # Search through command history <ctrl>-R + # Execute last command with sudo sudo !! ``` diff --git a/doc/administration/troubleshooting/log_parsing.md b/doc/administration/troubleshooting/log_parsing.md index 4cc62c08f4f..0320b2e52ce 100644 --- a/doc/administration/troubleshooting/log_parsing.md +++ b/doc/administration/troubleshooting/log_parsing.md @@ -47,7 +47,7 @@ grep <TERM> <FILE> | jq . jq -cR 'fromjson?' file.json | jq <COMMAND> ``` -By default `jq` will error out when it encounters a line that is not valid JSON. +By default `jq` errors out when it encounters a line that is not valid JSON. This skips over all invalid lines and parses the rest. #### Print a JSON log's time range @@ -152,6 +152,51 @@ CT: 297 ROUTE: /api/:version/projects/:id/repository/tags DURS: 731.39, CT: 190 ROUTE: /api/:version/projects/:id/repository/commits DURS: 1079.02, 979.68, 958.21 ``` +### Print top API user agents + +```shell +jq --raw-output '[.route, .ua] | @tsv' api_json.log | sort | uniq -c | sort -n +``` + +**Example output**: + +```plaintext + 89 /api/:version/usage_data/increment_unique_users # plus browser details + 567 /api/:version/jobs/:id/trace gitlab-runner # plus version details +1234 /api/:version/internal/allowed GitLab-Shell +``` + +This sample response seems normal. A custom tool or script might be causing a high load +if the output contains many: + +- Third party libraries like `python-requests` or `curl`. +- [GitLab CLI clients](https://about.gitlab.com/partners/technology-partners/#cli-clients). + +You can also [use `fast-stats top`](#parsing-gitlab-logs-with-jq) to extract performance statistics. + +### Parsing `gitlab-workhorse/current` + +### Print top Workhorse user agents + +```shell +jq --raw-output '[.uri, .user_agent] | @tsv' current | sort | uniq -c | sort -n +``` + +**Example output**: + +```plaintext + 89 /api/graphql # plus browser details + 567 /api/v4/internal/allowed GitLab-Shell +1234 /api/v4/jobs/request gitlab-runner # plus version details +``` + +Similar to the [API `ua` data](#print-top-api-user-agents), +deviations from this common order might indicate scripts that could be optimized. + +The performance impact of runners checking for new jobs can be reduced by increasing +[the `check_interval` setting](https://docs.gitlab.com/runner/configuration/advanced-configuration.html#the-global-section), +for example. + ### Parsing `gitlab-rails/geo.log` #### Find most common Geo sync errors @@ -166,7 +211,7 @@ jq --raw-output 'select(.severity == "ERROR") | [.project_path, .message] | @tsv ### Parsing `gitaly/current` -The following examples are useful to [troubleshoot Gitaly](../gitaly/troubleshooting.md). +Use the following examples to [troubleshoot Gitaly](../gitaly/troubleshooting.md). #### Find all Gitaly requests sent from web UI diff --git a/doc/administration/troubleshooting/navigating_gitlab_via_rails_console.md b/doc/administration/troubleshooting/navigating_gitlab_via_rails_console.md index 51ef3d95a4e..09a5cb8d185 100644 --- a/doc/administration/troubleshooting/navigating_gitlab_via_rails_console.md +++ b/doc/administration/troubleshooting/navigating_gitlab_via_rails_console.md @@ -1,465 +1,11 @@ --- -stage: Systems -group: Distribution -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +redirect_to: '../operations/rails_console.md' +remove_date: '2022-10-05' --- -# Navigating GitLab via Rails console **(FREE SELF)** +This document was moved to [another location](../operations/rails_console.md). -At the heart of GitLab is a web application [built using the Ruby on Rails -framework](https://about.gitlab.com/blog/2018/10/29/why-we-use-rails-to-build-gitlab/). -Thanks to this, we also get access to the amazing tools built right into Rails. -This guide introduces the [Rails console](../operations/rails_console.md#starting-a-rails-console-session) -and the basics of interacting with your GitLab instance from the command line. - -WARNING: -The Rails console interacts directly with your GitLab instance. In many cases, -there are no handrails to prevent you from permanently modifying, corrupting -or destroying production data. If you would like to explore the Rails console -with no consequences, you are strongly advised to do so in a test environment. - -This guide is targeted at GitLab system administrators who are troubleshooting -a problem or must retrieve some data that can only be done through direct -access of the GitLab application. Basic knowledge of Ruby is needed (try [this -30-minute tutorial](https://try.ruby-lang.org/) for a quick introduction). -Rails experience is helpful to have but not a must. - -## Starting a Rails console session - -Your type of GitLab installation determines how -[to start a rails console](../operations/rails_console.md). - -The following code examples take place inside the Rails console and also -assume an Omnibus GitLab installation. - -## Active Record objects - -### Looking up database-persisted objects - -Under the hood, Rails uses [Active Record](https://guides.rubyonrails.org/active_record_basics.html), -an object-relational mapping system, to read, write, and map application objects -to the PostgreSQL database. These mappings are handled by Active Record models, -which are Ruby classes defined in a Rails app. For GitLab, the model classes -can be found at `/opt/gitlab/embedded/service/gitlab-rails/app/models`. - -Let's enable debug logging for Active Record so we can see the underlying -database queries made: - -```ruby -ActiveRecord::Base.logger = Logger.new($stdout) -``` - -Now, let's try retrieving a user from the database: - -```ruby -user = User.find(1) -``` - -Which would return: - -```ruby -D, [2020-03-05T16:46:25.571238 #910] DEBUG -- : User Load (1.8ms) SELECT "users".* FROM "users" WHERE "users"."id" = 1 LIMIT 1 -=> #<User id:1 @root> -``` - -We can see that we've queried the `users` table in the database for a row whose -`id` column has the value `1`, and Active Record has translated that database -record into a Ruby object that we can interact with. Try some of the following: - -- `user.username` -- `user.created_at` -- `user.admin` - -By convention, column names are directly translated into Ruby object attributes, -so you should be able to do `user.<column_name>` to view the attribute's value. - -Also by convention, Active Record class names (singular and in camel case) map -directly onto table names (plural and in snake case) and vice versa. For example, -the `users` table maps to the `User` class, while the `application_settings` -table maps to the `ApplicationSetting` class. - -You can find a list of tables and column names in the Rails database schema, -available at `/opt/gitlab/embedded/service/gitlab-rails/db/schema.rb`. - -You can also look up an object from the database by attribute name: - -```ruby -user = User.find_by(username: 'root') -``` - -Which would return: - -```ruby -D, [2020-03-05T17:03:24.696493 #910] DEBUG -- : User Load (2.1ms) SELECT "users".* FROM "users" WHERE "users"."username" = 'root' LIMIT 1 -=> #<User id:1 @root> -``` - -Give the following a try: - -- `User.find_by(email: 'admin@example.com')` -- `User.where.not(admin: true)` -- `User.where('created_at < ?', 7.days.ago)` - -Did you notice that the last two commands returned an `ActiveRecord::Relation` -object that appeared to contain multiple `User` objects? - -Up to now, we've been using `.find` or `.find_by`, which are designed to return -only a single object (notice the `LIMIT 1` in the generated SQL query?). -`.where` is used when it is desirable to get a collection of objects. - -Let's get a collection of non-administrator users and see what we can do with it: - -```ruby -users = User.where.not(admin: true) -``` - -Which would return: - -```ruby -D, [2020-03-05T17:11:16.845387 #910] DEBUG -- : User Load (2.8ms) SELECT "users".* FROM "users" WHERE "users"."admin" != TRUE LIMIT 11 -=> #<ActiveRecord::Relation [#<User id:3 @support-bot>, #<User id:7 @alert-bot>, #<User id:5 @carrie>, #<User id:4 @bernice>, #<User id:2 @anne>]> -``` - -Now, try the following: - -- `users.count` -- `users.order(created_at: :desc)` -- `users.where(username: 'support-bot')` - -In the last command, we see that we can chain `.where` statements to generate -more complex queries. Notice also that while the collection returned contains -only a single object, we cannot directly interact with it: - -```ruby -users.where(username: 'support-bot').username -``` - -Which would return: - -```ruby -Traceback (most recent call last): - 1: from (irb):37 -D, [2020-03-05T17:18:25.637607 #910] DEBUG -- : User Load (1.6ms) SELECT "users".* FROM "users" WHERE "users"."admin" != TRUE AND "users"."username" = 'support-bot' LIMIT 11 -NoMethodError (undefined method `username' for #<ActiveRecord::Relation [#<User id:3 @support-bot>]>) -Did you mean? by_username -``` - -Let's retrieve the single object from the collection by using the `.first` -method to get the first item in the collection: - -```ruby -users.where(username: 'support-bot').first.username -``` - -We now get the result we wanted: - -```ruby -D, [2020-03-05T17:18:30.406047 #910] DEBUG -- : User Load (2.6ms) SELECT "users".* FROM "users" WHERE "users"."admin" != TRUE AND "users"."username" = 'support-bot' ORDER BY "users"."id" ASC LIMIT 1 -=> "support-bot" -``` - -For more on different ways to retrieve data from the database using Active -Record, please see the [Active Record Query Interface documentation](https://guides.rubyonrails.org/active_record_querying.html). - -### Modifying Active Record objects - -In the previous section, we learned about retrieving database records using -Active Record. Now, let's learn how to write changes to the database. - -First, let's retrieve the `root` user: - -```ruby -user = User.find_by(username: 'root') -``` - -Next, let's try updating the user's password: - -```ruby -user.password = 'password' -user.save -``` - -Which would return: - -```ruby -Enqueued ActionMailer::MailDeliveryJob (Job ID: 05915c4e-c849-4e14-80bb-696d5ae22065) to Sidekiq(mailers) with arguments: "DeviseMailer", "password_change", "deliver_now", #<GlobalID:0x00007f42d8ccebe8 @uri=#<URI::GID gid://gitlab/User/1>> -=> true -``` - -Here, we see that the `.save` command returned `true`, indicating that the -password change was successfully saved to the database. - -We also see that the save operation triggered some other action -- in this case -a background job to deliver an email notification. This is an example of an -[Active Record callback](https://guides.rubyonrails.org/active_record_callbacks.html) --- code which is designated to run in response to events in the Active Record -object life cycle. This is also why using the Rails console is preferred when -direct changes to data is necessary as changes made via direct database queries -do not trigger these callbacks. - -It's also possible to update attributes in a single line: - -```ruby -user.update(password: 'password') -``` - -Or update multiple attributes at once: - -```ruby -user.update(password: 'password', email: 'hunter2@example.com') -``` - -Now, let's try something different: - -```ruby -# Retrieve the object again so we get its latest state -user = User.find_by(username: 'root') -user.password = 'password' -user.password_confirmation = 'hunter2' -user.save -``` - -This returns `false`, indicating that the changes we made were not saved to the -database. You can probably guess why, but let's find out for sure: - -```ruby -user.save! -``` - -This should return: - -```ruby -Traceback (most recent call last): - 1: from (irb):64 -ActiveRecord::RecordInvalid (Validation failed: Password confirmation doesn't match Password) -``` - -Aha! We've tripped an [Active Record Validation](https://guides.rubyonrails.org/active_record_validations.html). -Validations are business logic put in place at the application-level to prevent -unwanted data from being saved to the database and in most cases come with -helpful messages letting you know how to fix the problem inputs. - -We can also add the bang (Ruby speak for `!`) to `.update`: - -```ruby -user.update!(password: 'password', password_confirmation: 'hunter2') -``` - -In Ruby, method names ending with `!` are commonly known as "bang methods". By -convention, the bang indicates that the method directly modifies the object it -is acting on, as opposed to returning the transformed result and leaving the -underlying object untouched. For Active Record methods that write to the -database, bang methods also serve an additional function: they raise an -explicit exception whenever an error occurs, instead of just returning `false`. - -We can also skip validations entirely: - -```ruby -# Retrieve the object again so we get its latest state -user = User.find_by(username: 'root') -user.password = 'password' -user.password_confirmation = 'hunter2' -user.save!(validate: false) -``` - -This is not recommended, as validations are usually put in place to ensure the -integrity and consistency of user-provided data. - -A validation error prevents the entire object from being saved to -the database. You can see a little of this in the section below. If you're getting -a mysterious red banner in the GitLab UI when submitting a form, this can often -be the fastest way to get to the root of the problem. - -### Interacting with Active Record objects - -At the end of the day, Active Record objects are just normal Ruby objects. As -such, we can define methods on them which perform arbitrary actions. - -For example, GitLab developers have added some methods which help with -two-factor authentication: - -```ruby -def disable_two_factor! - transaction do - update( - otp_required_for_login: false, - encrypted_otp_secret: nil, - encrypted_otp_secret_iv: nil, - encrypted_otp_secret_salt: nil, - otp_grace_period_started_at: nil, - otp_backup_codes: nil - ) - self.u2f_registrations.destroy_all # rubocop: disable DestroyAll - end -end - -def two_factor_enabled? - two_factor_otp_enabled? || two_factor_u2f_enabled? -end -``` - -(See: `/opt/gitlab/embedded/service/gitlab-rails/app/models/user.rb`) - -We can then use these methods on any user object: - -```ruby -user = User.find_by(username: 'root') -user.two_factor_enabled? -user.disable_two_factor! -``` - -Some methods are defined by gems, or Ruby software packages, which GitLab uses. -For example, the [StateMachines](https://github.com/state-machines/state_machines-activerecord) -gem which GitLab uses to manage user state: - -```ruby -state_machine :state, initial: :active do - event :block do - - ... - - event :activate do - - ... - -end -``` - -Give it a try: - -```ruby -user = User.find_by(username: 'root') -user.state -user.block -user.state -user.activate -user.state -``` - -Earlier, we mentioned that a validation error prevents the entire object -from being saved to the database. Let's see how this can have unexpected -interactions: - -```ruby -user.password = 'password' -user.password_confirmation = 'hunter2' -user.block -``` - -We get `false` returned! Let's find out what happened by adding a bang as we did -earlier: - -```ruby -user.block! -``` - -Which would return: - -```ruby -Traceback (most recent call last): - 1: from (irb):87 -StateMachines::InvalidTransition (Cannot transition state via :block from :active (Reason(s): Password confirmation doesn't match Password)) -``` - -We see that a validation error from what feels like a completely separate -attribute comes back to haunt us when we try to update the user in any way. - -In practical terms, we sometimes see this happen with GitLab administration settings -- -validations are sometimes added or changed in a GitLab update, resulting in -previously saved settings now failing validation. Because you can only update -a subset of settings at once through the UI, in this case the only way to get -back to a good state is direct manipulation via Rails console. - -### Commonly used Active Record models and how to look up objects - -**Get a user by primary email address or username:** - -```ruby -User.find_by(email: 'admin@example.com') -User.find_by(username: 'root') -``` - -**Get a user by primary OR secondary email address:** - -```ruby -User.find_by_any_email('user@example.com') -``` - -The `find_by_any_email` method is a custom method added by GitLab developers rather -than a Rails-provided default method. - -**Get a collection of administrator users:** - -```ruby -User.admins -``` - -`admins` is a [scope convenience method](https://guides.rubyonrails.org/active_record_querying.html#scopes) -which does `where(admin: true)` under the hood. - -**Get a project by its path:** - -```ruby -Project.find_by_full_path('group/subgroup/project') -``` - -`find_by_full_path` is a custom method added by GitLab developers rather -than a Rails-provided default method. - -**Get a project's issue or merge request by its numeric ID:** - -```ruby -project = Project.find_by_full_path('group/subgroup/project') -project.issues.find_by(iid: 42) -project.merge_requests.find_by(iid: 42) -``` - -`iid` means "internal ID" and is how we keep issue and merge request IDs -scoped to each GitLab project. - -**Get a group by its path:** - -```ruby -Group.find_by_full_path('group/subgroup') -``` - -**Get a group's related groups:** - -```ruby -group = Group.find_by_full_path('group/subgroup') - -# Get a group's parent group -group.parent - -# Get a group's child groups -group.children -``` - -**Get a group's projects:** - -```ruby -group = Group.find_by_full_path('group/subgroup') - -# Get group's immediate child projects -group.projects - -# Get group's child projects, including those in subgroups -group.all_projects -``` - -**Get CI pipeline or builds:** - -```ruby -Ci::Pipeline.find(4151) -Ci::Build.find(66124) -``` - -The pipeline and job ID numbers increment globally across your GitLab -instance, so there's no requirement to use an internal ID attribute to look them up, -unlike with issues or merge requests. - -**Get the current application settings object:** - -```ruby -ApplicationSetting.current -``` +<!-- This redirect file can be deleted after <2022-10-05>. --> +<!-- Redirects that point to other docs in the same project expire in three months. --> +<!-- Redirects that point to docs in a different project or site (link is not relative and starts with `https:`) expire in one year. --> +<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html --> diff --git a/doc/administration/troubleshooting/postgresql.md b/doc/administration/troubleshooting/postgresql.md index cdbf786bdb2..61b661d45f8 100644 --- a/doc/administration/troubleshooting/postgresql.md +++ b/doc/administration/troubleshooting/postgresql.md @@ -54,7 +54,7 @@ This section is for links to information elsewhere in the GitLab documentation. - Including [troubleshooting](../postgresql/replication_and_failover.md#troubleshooting) `gitlab-ctl patroni check-leader` and PgBouncer errors. -- [Developer database documentation](../../development/index.md#database-guides), +- [Developer database documentation](../../development/feature_development.md#database-guides), some of which is absolutely not for production use. Including: - Understanding EXPLAIN plans. diff --git a/doc/administration/troubleshooting/ssl.md b/doc/administration/troubleshooting/ssl.md index d5d50127ad5..e1cd92a788f 100644 --- a/doc/administration/troubleshooting/ssl.md +++ b/doc/administration/troubleshooting/ssl.md @@ -9,7 +9,7 @@ type: reference This page contains a list of common SSL-related errors and scenarios that you may encounter while working with GitLab. It should serve as an addition to the -main SSL docs available here: +main SSL documentation: - [Omnibus SSL Configuration](https://docs.gitlab.com/omnibus/settings/ssl.html). - [Self-signed certificates or custom Certification Authorities for GitLab Runner](https://docs.gitlab.com/runner/configuration/tls-self-signed.html). @@ -110,8 +110,7 @@ https://gitlab.domain.tld/api/v4/jobs/request: Post https://gitlab.domain.tld/ap x509: certificate signed by unknown authority ``` -If you encounter a similar problem, add your certificate to `/etc/gitlab-runner/certs`, -and the restart the runner by running `gitlab-runner restart`. +Follow the details in [Self-signed certificates or custom Certification Authorities for GitLab Runner](https://docs.gitlab.com/runner/configuration/tls-self-signed.html). ## Mirroring a remote GitLab repository that uses a self-signed SSL certificate @@ -237,7 +236,7 @@ remote server that is serving only HTTP. One scenario is that you're using [object storage](../object_storage.md), which isn't served under HTTPS. GitLab is misconfigured and attempts a TLS handshake, -but the object storage will respond with plain HTTP. +but the object storage responds with plain HTTP. ## `schannel: SEC_E_UNTRUSTED_ROOT` @@ -247,7 +246,7 @@ If you're on Windows and get the following error: Fatal: unable to access 'https://gitlab.domain.tld/group/project.git': schannel: SEC_E_UNTRUSTED_ROOT (0x80090325) - The certificate chain was issued by an authority that is not trusted." ``` -You may need to specify that Git should use OpenSSL: +You must specify that Git should use OpenSSL: ```shell git config --system http.sslbackend openssl diff --git a/doc/administration/troubleshooting/test_environments.md b/doc/administration/troubleshooting/test_environments.md index 94d17ba714e..d240103a51c 100644 --- a/doc/administration/troubleshooting/test_environments.md +++ b/doc/administration/troubleshooting/test_environments.md @@ -45,7 +45,7 @@ docker run --name gitlab_saml -p 8080:8080 -p 8443:8443 \ -d jamedjo/test-saml-idp ``` -The following will also need to go in your `/etc/gitlab/gitlab.rb`. See [our SAML docs](../../integration/saml.md) +The following must also go in your `/etc/gitlab/gitlab.rb`. See [our SAML docs](../../integration/saml.md) for more, as well as the list of [default usernames, passwords, and emails](https://hub.docker.com/r/jamedjo/test-saml-idp/#usage). ```ruby diff --git a/doc/administration/troubleshooting/tracing_correlation_id.md b/doc/administration/troubleshooting/tracing_correlation_id.md index fed3604057b..418dd729066 100644 --- a/doc/administration/troubleshooting/tracing_correlation_id.md +++ b/doc/administration/troubleshooting/tracing_correlation_id.md @@ -32,7 +32,7 @@ documentation for some popular browsers. To locate a relevant request and view its correlation ID: -1. Enable persistent logging in your network monitor. Some actions in GitLab will redirect you quickly after you submit a form, so this will help capture all relevant activity. +1. Enable persistent logging in your network monitor. Some actions in GitLab redirect you quickly after you submit a form, so this helps capture all relevant activity. 1. To help isolate the requests you are looking for, you can filter for `document` requests. 1. Select the request of interest to view further detail. 1. Go to the **Headers** section and look for **Response Headers**. There you should find an `x-request-id` header with a @@ -121,7 +121,7 @@ find /var/log/gitlab -type f -mtime 0 -exec grep 'LOt9hgi1TV4' '{}' '+' ### Searching in distributed architectures If you have done some horizontal scaling in your GitLab infrastructure, then -you will need to search across _all_ of your GitLab nodes. You can do this with +you must search across _all_ of your GitLab nodes. You can do this with some sort of log aggregation software like Loki, ELK, Splunk, or others. You can use a tool like Ansible or PSSH (parallel SSH) that can execute identical commands across your servers in diff --git a/doc/administration/uploads.md b/doc/administration/uploads.md index e618c4787ab..0bd46193d10 100644 --- a/doc/administration/uploads.md +++ b/doc/administration/uploads.md @@ -66,7 +66,7 @@ For source installations the following settings are nested under `uploads:` and | Setting | Description | Default | |---------|-------------|---------| | `enabled` | Enable/disable object storage | `false` | -| `remote_directory` | The bucket name where Uploads will be stored| | +| `remote_directory` | The bucket name where Uploads are stored| | | `proxy_download` | Set to `true` to enable proxying all files served. Option allows to reduce egress traffic as this allows clients to download directly from remote storage instead of proxying all data | `false` | | `connection` | Various connection options described below | | diff --git a/doc/administration/whats-new.md b/doc/administration/whats-new.md index beb47254573..05c769cf20c 100644 --- a/doc/administration/whats-new.md +++ b/doc/administration/whats-new.md @@ -1,7 +1,7 @@ --- -stage: Growth -group: Adoption -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +stage: none +group: unassigned +info: For assistance with this What's new page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments-to-other-projects-and-subjects. --- # What's new **(FREE)** |
