diff options
Diffstat (limited to 'doc/administration')
59 files changed, 1649 insertions, 393 deletions
diff --git a/doc/administration/audit_event_streaming.md b/doc/administration/audit_event_streaming.md index ca59dff7ef4..6cf630ad079 100644 --- a/doc/administration/audit_event_streaming.md +++ b/doc/administration/audit_event_streaming.md @@ -23,11 +23,11 @@ GitLab can stream a single event more than once to the same destination. Use the ## Add a new event streaming destination WARNING: -Event streaming destinations will receive **all** audit event data, which could include sensitive information. Make sure you trust the destination endpoint. +Event streaming destinations receive **all** audit event data, which could include sensitive information. Make sure you trust the destination endpoint. -### Add event streaming destination using GitLab UI +### Use the GitLab UI -Users with at least the Owner role of a group can add event streaming destinations for it: +Users with at least the Owner role for a group can add event streaming destinations for it: 1. On the top bar, select **Menu > Groups** and find your group. 1. On the left sidebar, select **Security & Compliance > Audit events** @@ -41,10 +41,10 @@ Event streaming is enabled if: - No warning is shown. - The added endpoint is displayed in the UI. -### Add event streaming destination using the API +### Use the API -To enable event streaming, a group owner must add a new event streaming destination using the `externalAuditEventDestinationCreate` mutation -in the GraphQL API. +To enable event streaming and add a destination, users with at least the Owner role for a group must use the +`externalAuditEventDestinationCreate` mutation in the GraphQL API. ```graphql mutation { @@ -66,19 +66,24 @@ Event streaming is enabled if: - The returned `errors` object is empty. - The API responds with `200 OK`. -## List currently enabled streaming destinations +## List streaming destinations -### List currently enabled streaming destination using GitLab UI +Users with at least the Owner role for a group can list event streaming destinations. -Users with at least the Owner role of a group can list event streaming destinations: +### Use the GitLab UI + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/336411) in GitLab 14.9. + +Users with at least the Owner role for a group can list event streaming destinations: 1. On the top bar, select **Menu > Groups** and find your group. 1. On the left sidebar, select **Security & Compliance > Audit events** 1. On the main area, select **Streams** tab. -### List currently enabled streaming destinations using the API +### Use the API -Group owners can view a list of event streaming destinations at any time using the `externalAuditEventDestinations` query type. +Users with at least the Owner role for a group can view a list of event streaming destinations at any time using the +`externalAuditEventDestinations` query type. ```graphql query { @@ -97,27 +102,34 @@ query { If the resulting list is empty, then audit event streaming is not enabled for that group. -## Delete currently enabled streaming destination +## Delete streaming destinations -Group Owners can delete event streaming destinations at any time using the `deleteAuditEventDestinations` mutation type. +Users with at least the Owner role for a group can delete event streaming destinations using the +`deleteAuditEventDestinations` mutation type. + +When the last destination is successfully deleted, event streaming is disabled for the group. -### Delete currently enabled streaming using GitLab UI +### Use the GitLab UI -Uses with at least the Owner role of a group can delete event streaming destination. +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/336411) in GitLab 14.9. + +Users with at least the Owner role for a group can delete event streaming destinations. 1. On the top bar, select **Menu > Groups** and find your group. 1. On the left sidebar, select **Security & Compliance > Audit events** 1. On the main area, select **Streams** tab. 1. Select **{remove}** at the right side of each item. -The external streaming destination is delete when: +The external streaming destination is deleted when: - No warning is shown. - The deleted endpoint is not displayed in the UI. -### Delete currently enabled streaming using the API +### Use the API -You can delete an event streaming destination by specifying an ID. Get the required ID by [listing the details](audit_event_streaming.md#list-currently-enabled-streaming-destinations-using-the-api) of event streaming destinations. +Delete an event streaming destination by specifying an ID. Get the required ID by +[listing the details](audit_event_streaming.md#use-the-api-1) of event +streaming destinations. ```graphql @@ -134,8 +146,6 @@ Destination is deleted if: - The returned `errors` object is empty. - The API responds with `200 OK`. -When the last destination is successfully deleted, event streaming is disabled for the group. - ## Verify event authenticity > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/345424) in GitLab 14.8. @@ -144,7 +154,7 @@ Each streaming destination has a unique verification token (`verificationToken`) token is generated when the event destination is created and cannot be changed. Each streamed event contains a random alphanumeric identifier for the `X-Gitlab-Event-Streaming-Token` HTTP header that can be verified against -the destination's value when [listing streaming destinations](#list-currently-enabled-streaming-destinations). +the destination's value when [listing streaming destinations](#list-streaming-destinations). ## Audit event streaming on Git operations @@ -155,7 +165,7 @@ On self-managed GitLab, by default this feature is not available. To make it ava Streaming audit events can be sent when signed-in users push or pull a project's remote Git repositories: -- [Using SSH](../ssh/index.md). +- [Using SSH](../user/ssh.md). - Using HTTP or HTTPS. - Using the **Download** button (**{download}**) in GitLab UI. @@ -163,9 +173,9 @@ Audit events are not captured for users that are not signed in. For example, whe To configure streaming audit events for Git operations, see [Add a new event streaming destination](#add-a-new-event-streaming-destination). -### Request headers +### Headers -Request headers are formatted as follows: +Headers are formatted as follows: ```plaintext POST /logs HTTP/1.1 @@ -174,7 +184,7 @@ Content-Type: application/x-www-form-urlencoded X-Gitlab-Event-Streaming-Token: <DESTINATION_TOKEN> ``` -### Example responses for SSH events +### Example payloads for SSH events Fetch: @@ -236,7 +246,7 @@ Push: } ``` -### Example responses for HTTP and HTTPS events +### Example payloads for HTTP and HTTPS events Fetch: @@ -298,7 +308,7 @@ Push: } ``` -### Example responses for events from GitLab UI download button +### Example payloads for events from GitLab UI download button Fetch: @@ -333,9 +343,9 @@ Fetch: Stream audit events that relate to merge approval actions performed within a project. -### Request headers +### Headers -Request headers are formatted as follows: +Headers are formatted as follows: ```plaintext POST /logs HTTP/1.1 @@ -344,7 +354,7 @@ Content-Type: application/x-www-form-urlencoded X-Gitlab-Event-Streaming-Token: <DESTINATION_TOKEN> ``` -### Example request body +### Example payload ```json { diff --git a/doc/administration/auditor_users.md b/doc/administration/auditor_users.md index 7c89f34ce61..b3304fd1cbd 100644 --- a/doc/administration/auditor_users.md +++ b/doc/administration/auditor_users.md @@ -61,8 +61,7 @@ To create an Auditor user: 1. On the left sidebar, select **Overview > Users**. 1. Create a new user or edit an existing one, and in the **Access** section select Auditor. -1. Select **Create user** or **Save changes** if you created a new user or - edited an existing one respectively. +1. If you created a user, select **Create user**. For an existing user, select **Save changes**. To revoke Auditor permissions from a user, make them a Regular user by following the previous steps. diff --git a/doc/administration/auth/index.md b/doc/administration/auth/index.md index d09344629e2..fca4f163075 100644 --- a/doc/administration/auth/index.md +++ b/doc/administration/auth/index.md @@ -31,7 +31,6 @@ providers: - [Salesforce](../../integration/salesforce.md) - [SAML](../../integration/saml.md) - [SAML for GitLab.com groups](../../user/group/saml_sso/index.md) **(PREMIUM SAAS)** -- [Shibboleth](../../integration/saml.md) - [Smartcard](smartcard.md) **(PREMIUM SELF)** - [Twitter](../../integration/twitter.md) diff --git a/doc/administration/auth/ldap/index.md b/doc/administration/auth/ldap/index.md index 2a396c4d53a..a7e070b755a 100644 --- a/doc/administration/auth/ldap/index.md +++ b/doc/administration/auth/ldap/index.md @@ -518,7 +518,7 @@ GitLab doesn't support TLS client authentication. Complete these steps on your L The TLS client authentication setting in your LDAP server cannot be mandatory and clients cannot be authenticated with the TLS protocol. -## Deleting users +## Users deleted from LDAP Users deleted from the LDAP server: @@ -531,6 +531,16 @@ However, these users can continue to use Git with SSH until the next time the To delete the account immediately, you can manually [block the user](../../../user/admin_area/moderate_users.md#block-a-user). +## Updating user email addresses + +Email addresses on the LDAP server are considered the source of truth for users when LDAP is used to sign in. Updating user email +addresses must be done on the LDAP server that manages the user. The email address for GitLab is updated either: + +- When the user next signs in. +- When the next [user sync](ldap_synchronization.md#user-sync) is run. + +The updated user's previous email address becomes the secondary email address to preserve that user's commit history. + ## Google Secure LDAP > Introduced in GitLab 11.9. diff --git a/doc/administration/auth/ldap/ldap-troubleshooting.md b/doc/administration/auth/ldap/ldap-troubleshooting.md index b8391bec72f..5c5d5aaffe8 100644 --- a/doc/administration/auth/ldap/ldap-troubleshooting.md +++ b/doc/administration/auth/ldap/ldap-troubleshooting.md @@ -71,7 +71,28 @@ options = { filter: Net::LDAP::Filter.eq('cn', '*'), # :attributes is optional - # the attributes we want to get returnedk + # the attributes we want to get returned + attributes: %w(dn cn memberuid member submember uniquemember memberof) +} +adapter.ldap_search(options) +``` + +When using OIDs in the filter, replace `Net::LDAP::Filter.eq` with `Net::LDAP::Filter.construct`: + +```ruby +adapter = Gitlab::Auth::Ldap::Adapter.new('ldapmain') +options = { + # :base is required + # use .base or .group_base + base: adapter.config.base, + + # :filter is optional + # This filter includes OID 1.2.840.113556.1.4.1941 + # It will search for all direct and nested members of the group gitlab_grp in the LDAP directory + filter: Net::LDAP::Filter.construct("(memberOf:1.2.840.113556.1.4.1941:=CN=gitlab_grp,DC=example,DC=com)"), + + # :attributes is optional + # the attributes we want to get returned attributes: %w(dn cn memberuid member submember uniquemember memberof) } adapter.ldap_search(options) diff --git a/doc/administration/auth/oidc.md b/doc/administration/auth/oidc.md index a099b9c76f8..561cbd1b3ae 100644 --- a/doc/administration/auth/oidc.md +++ b/doc/administration/auth/oidc.md @@ -471,7 +471,7 @@ For your app, complete the following steps on Casdoor: 1. Add your GitLab redirect URL. For example, if your GitLab domain is `gitlab.example.com`, ensure the Casdoor app has the following `Redirect URI`: `https://gitlab.example.com/users/auth/openid_connect/callback`. -See the [Casdoor documentation](https://casdoor.org/docs/integration/gitlab) for more details. +See the [Casdoor documentation](https://casdoor.org/docs/integration/gitlab/) for more details. Example Omnibus GitLab configuration (file path: `/etc/gitlab/gitlab.rb`): diff --git a/doc/administration/clusters/kas.md b/doc/administration/clusters/kas.md index e5c371b9d40..82bb1a35e02 100644 --- a/doc/administration/clusters/kas.md +++ b/doc/administration/clusters/kas.md @@ -116,7 +116,7 @@ If you get the following error message: The project specified by the manifest (`root/kas-manifest001`) doesn't exist or the project where the manifest is kept is private. To fix this issue, -ensure the project path is correct and that the project's visibility is [set to public](../../public_access/public_access.md). +ensure the project path is correct and that the project's visibility is [set to public](../../user/public_access.md). ### Configuration file not found diff --git a/doc/administration/compliance.md b/doc/administration/compliance.md index 7c3f36c71c2..99c291379f1 100644 --- a/doc/administration/compliance.md +++ b/doc/administration/compliance.md @@ -82,7 +82,7 @@ These features can help provide visibility into GitLab and audit what is happeni These features can also help with compliance requirements: -- [**Email all users of a project, group, or entire server**](../tools/email.md) +- [**Email all users of a project, group, or entire server**](../user/admin_area/email_from_gitlab.md) (for instances): An administrator can email groups of users based on project or group membership, or email everyone using the GitLab instance. These emails are great for scheduled maintenance or upgrades. diff --git a/doc/administration/file_hooks.md b/doc/administration/file_hooks.md index e041f1e11c6..4ddca58c774 100644 --- a/doc/administration/file_hooks.md +++ b/doc/administration/file_hooks.md @@ -15,12 +15,12 @@ without modifying the GitLab source code. A file hook runs on each event. You can filter events or projects in a file hook's code, and create many file hooks as you need. Each file hook is triggered by GitLab asynchronously in case of an event. For a list of events -see the [system hooks](../system_hooks/system_hooks.md) documentation. +see the [system hooks](system_hooks.md) documentation. NOTE: File hooks must be configured on the file system of the GitLab server. Only GitLab server administrators can complete these tasks. Explore -[system hooks](../system_hooks/system_hooks.md) or [webhooks](../user/project/integrations/webhooks.md) +[system hooks](system_hooks.md) or [webhooks](../user/project/integrations/webhooks.md) as an option if you do not have file system access. Instead of writing and supporting your own file hook, you can also make changes @@ -52,7 +52,7 @@ Follow the steps below to set up a custom hook: language type. For example, if the script is in Ruby the shebang will probably be `#!/usr/bin/env ruby`. 1. The data to the file hook is provided as JSON on STDIN. It is exactly the - same as for [system hooks](../system_hooks/system_hooks.md). + same as for [system hooks](system_hooks.md). That's it! Assuming the file hook code is properly implemented, the hook fires as appropriate. The file hooks file list is updated for each event, there is no diff --git a/doc/administration/geo/disaster_recovery/planned_failover.md b/doc/administration/geo/disaster_recovery/planned_failover.md index 939b4aec968..2a21bda23d7 100644 --- a/doc/administration/geo/disaster_recovery/planned_failover.md +++ b/doc/administration/geo/disaster_recovery/planned_failover.md @@ -42,6 +42,38 @@ Repository-centric strategies for using `rsync` effectively can be found in the [moving repositories](../../operations/moving_repositories.md) documentation; these strategies can be adapted for use with any other file-based data, such as [GitLab Pages](../../pages/index.md#change-storage-path). +### Container registry + +By default, the container registry is not automatically replicated to secondary +sites and this needs to be manually configured, see [Docker Registry for a secondary site](../replication/docker_registry.md). + +If you are using local storage on your current primary site for the container +registry, you can `rsync` the container registry objects to the secondary +site you are about to failover to: + +```shell +# Run from the secondary site +rsync --archive --perms --delete root@<geo-primary>:/var/opt/gitlab/gitlab-rails/shared/registry/. /var/opt/gitlab/gitlab-rails/shared/registry +``` + +Alternatively, you can [back up](../../../raketasks/backup_restore.md#back-up-gitlab) +the container registry on the primary site and restore it onto the secondary +site: + +1. On your primary site, back up only the registry and [exclude specific directories +from the backup](../../../raketasks/backup_restore.md#excluding-specific-directories-from-the-backup): + + ```shell + # Create a backup in the /var/opt/gitlab/backups folder + sudo gitlab-backup create SKIP=db,uploads,builds,artifacts,lfs,terraform_state,pages,repositories,packages + ``` + +1. Copy the backup tarball generated from your primary site to the `/var/opt/gitlab/backups` folder +on your secondary site. + +1. On your secondary site, restore the registry following the [Restore GitLab](../../../raketasks/backup_restore.md#restore-gitlab) +documentation. + ## Preflight checks NOTE: diff --git a/doc/administration/geo/replication/geo_validation_tests.md b/doc/administration/geo/replication/geo_validation_tests.md index 70b3eaf5fea..e4432c41ff1 100644 --- a/doc/administration/geo/replication/geo_validation_tests.md +++ b/doc/administration/geo/replication/geo_validation_tests.md @@ -208,4 +208,16 @@ The following are additional validation tests we performed. - Description: Tested the average time it takes for a single image to replicate from the primary object storage location to the secondary when using Azure based object storage replication and [GitLab based object storage replication](object_storage.md#enabling-gitlab-managed-object-storage-replication). This was tested by uploading a 1mb image to a project on the primary site every second for 60 seconds. The time was then measured until a image was available on the secondary site. This was achieved using a [Ruby Script](https://gitlab.com/gitlab-org/quality/geo-replication-tester). - Outcome: When using Azure based replication the average time for an image to replicate from the primary object storage to the secondary was recorded as 40 seconds, the longest replication time was 70 seconds and the quickest was 11 seconds. When using GitLab based replication the average time for replication to complete was 5 seconds, the longest replication time was 10 seconds and the quickest was 3 seconds. - Follow up issue: - - [Test and validate object storage replication performance on reference architectures](https://gitlab.com/gitlab-org/gitlab/-/issues/347314) + - [Validate Cross Region Object storage replication using Azure based object storage](https://gitlab.com/gitlab-org/gitlab/-/issues/358154) + +### April 2022 + +[Validate Object storage replication using AWS based object storage](https://gitlab.com/gitlab-org/gitlab/-/issues/351463): + +- Description: Tested the average time it takes for a single image to replicate from the primary object storage location to the secondary when using AWS based object storage replication and [GitLab based object storage replication](object_storage.md#enabling-gitlab-managed-object-storage-replication). This was tested by uploading a 1mb image to a project on the primary site every second for 60 seconds. The time was then measured until a image was available on the secondary site. This was achieved using a [Ruby Script](https://gitlab.com/gitlab-org/quality/geo-replication-tester). +- Outcome: When using AWS managed replication the average time for an image to replicate between sites is about 49 seconds, this is true for when sites are located within the same region and when they are further apart (Europe to America). When using Geo managed replication within the same region the average time for replication took just 5 seconds, however when replicating cross region the average time rose to 33 seconds. + +[Validate Object storage replication using GCP based object storage](https://gitlab.com/gitlab-org/gitlab/-/issues/351464): + +- Description: Tested the average time it takes for a single image to replicate from the primary object storage location to the secondary when using GCP based object storage replication and [GitLab based object storage replication](object_storage.md#enabling-gitlab-managed-object-storage-replication). This was tested by uploading a 1mb image to a project on the primary site every second for 60 seconds. The time was then measured until a image was available on the secondary site. This was achieved using a [Ruby Script](https://gitlab.com/gitlab-org/quality/geo-replication-tester). +- Outcome: GCP handles replication differently than other Cloud Providers. In GCP, the process is to a create single bucket that is either multi, dual or single region based. This means that the bucket will automatically store replicas in a region based on the option chosen. Even when using multi region, this will still only replicate within a single continent, the options being America, Europe, or Asia. At current there doesn't seem to be any way to replicate objects between continents using GCP based replication. For Geo managed replication the average time when replicating within the same region was 6 seconds, and when replicating cross region this rose to just 9 seconds. diff --git a/doc/administration/geo/replication/troubleshooting.md b/doc/administration/geo/replication/troubleshooting.md index 8f55ce99787..871d6041066 100644 --- a/doc/administration/geo/replication/troubleshooting.md +++ b/doc/administration/geo/replication/troubleshooting.md @@ -88,29 +88,49 @@ node running Rails (Puma, Sidekiq, or Geo Log Cursor) on the Geo **secondary** s sudo gitlab-rake geo:status ``` -Example output: +The output includes: + +- a count of "failed" items if any failures occurred +- the percentage of "succeeded" items, relative to the "total" + +Example: ```plaintext http://secondary.example.com/ ----------------------------------------------------- - GitLab Version: 11.10.4-ee + GitLab Version: 14.9.2-ee Geo Role: Secondary Health Status: Healthy - Repositories: 289/289 (100%) - Verified Repositories: 289/289 (100%) - Wikis: 289/289 (100%) - Verified Wikis: 289/289 (100%) - LFS Objects: 8/8 (100%) - Attachments: 5/5 (100%) - CI job artifacts: 0/0 (0%) - Repositories Checked: 0/289 (0%) + Repositories: succeeded 12345 / total 12345 (100%) + Verified Repositories: succeeded 12345 / total 12345 (100%) + Wikis: succeeded 6789 / total 6789 (100%) + Verified Wikis: succeeded 6789 / total 6789 (100%) + Attachments: succeeded 4 / total 4 (100%) + CI job artifacts: succeeded 0 / total 0 (0%) + Design repositories: succeeded 1 / total 1 (100%) + LFS Objects: failed 1 / succeeded 2 / total 3 (67%) + Merge Request Diffs: succeeded 0 / total 0 (0%) + Package Files: failed 1 / succeeded 2 / total 3 (67%) + Terraform State Versions: failed 1 / succeeded 2 / total 3 (67%) + Snippet Repositories: failed 1 / succeeded 2 / total 3 (67%) + Group Wiki Repositories: succeeded 4 / total 4 (100%) + Pipeline Artifacts: failed 3 / succeeded 0 / total 3 (0%) + Pages Deployments: succeeded 0 / total 0 (0%) + Repositories Checked: failed 5 / succeeded 0 / total 5 (0%) + Package Files Verified: succeeded 0 / total 10 (0%) + Terraform State Versions Verified: succeeded 0 / total 10 (0%) + Snippet Repositories Verified: succeeded 99 / total 100 (99%) + Pipeline Artifacts Verified: succeeded 0 / total 10 (0%) Sync Settings: Full Database replication lag: 0 seconds - Last event ID seen from primary: 10215 (about 2 minutes ago) - Last event ID processed by cursor: 10215 (about 2 minutes ago) - Last status report was: 2 minutes ago + Last event ID seen from primary: 12345 (about 2 minutes ago) + Last event ID processed by cursor: 12345 (about 2 minutes ago) + Last status report was: 1 minute ago ``` +To find more details about failed items, check +[the `gitlab-rails/geo.log` file](../../troubleshooting/log_parsing.md#find-most-common-geo-sync-errors) + ### Check if PostgreSQL replication is working To check if PostgreSQL replication is working, check if: diff --git a/doc/administration/geo/replication/usage.md b/doc/administration/geo/replication/usage.md index b1183e56cd0..d96bfae78a1 100644 --- a/doc/administration/geo/replication/usage.md +++ b/doc/administration/geo/replication/usage.md @@ -27,7 +27,7 @@ Everything up-to-date ``` NOTE: -If you're using HTTPS instead of [SSH](../../../ssh/index.md) to push to the secondary, +If you're using HTTPS instead of [SSH](../../../user/ssh.md) to push to the secondary, you can't store credentials in the URL like `user:password@URL`. Instead, you can use a [`.netrc` file](https://www.gnu.org/software/inetutils/manual/html_node/The-_002enetrc-file.html) for Unix-like operating systems or `_netrc` for Windows. In that case, the credentials diff --git a/doc/administration/geo/replication/version_specific_updates.md b/doc/administration/geo/replication/version_specific_updates.md index d3bc54775f4..b0797445890 100644 --- a/doc/administration/geo/replication/version_specific_updates.md +++ b/doc/administration/geo/replication/version_specific_updates.md @@ -12,7 +12,7 @@ for updating Geo sites. ## Updating to 14.9 -**DO NOT** update to GitLab 14.9.0. +**DO NOT** update to GitLab 14.9.0. We've discovered an issue with Geo's CI verification feature that may [cause job traces to be lost](https://gitlab.com/gitlab-com/gl-infra/production/-/issues/6664). This issue will be fixed in the next patch release. diff --git a/doc/administration/get_started.md b/doc/administration/get_started.md index e6a2e833af3..b2bdd876499 100644 --- a/doc/administration/get_started.md +++ b/doc/administration/get_started.md @@ -20,7 +20,7 @@ Authentication is the first step in making your installation secure. This one-time secret code is an additional safeguard that keeps intruders out, even if they have your password. - Add a backup email. If you lose access to your account, the GitLab Support team can help you more quickly. - Save or print your recovery codes. If you can't access your authentication device, you can use these recovery codes to sign in to your GitLab account. - - Add [an SSH key](../ssh/index.md) to your profile. You can generate new recovery codes as needed with SSH. + - Add [an SSH key](../user/ssh.md) to your profile. You can generate new recovery codes as needed with SSH. - Enable [personal access tokens](../user/profile/personal_access_tokens.md). When using 2FA, you can use these tokens to access the GitLab API. ## Projects and groups @@ -62,9 +62,10 @@ You may need to import projects from external sources like GitHub, Bitbucket, or ### Popular project imports -- [GitHub Enterprise to self-managed GitLab](../integration/github.md): Enabling OAuth makes it easier for developers to find and import their projects. -- [Bitbucket Server](../user/project/import/bitbucket_server.md#limitations): There are certain data limitations. - For assistance with these data types, contact your GitLab account manager or GitLab Support about our professional migration services. +- [GitHub Enterprise to self-managed GitLab](../integration/github.md) +- [Bitbucket Server](../user/project/import/bitbucket_server.md) + +For assistance with these data types, contact your GitLab account manager or GitLab Support about our professional migration services. ## GitLab instance security diff --git a/doc/administration/gitaly/configure_gitaly.md b/doc/administration/gitaly/configure_gitaly.md index 88efd1885db..0fb285c50d6 100644 --- a/doc/administration/gitaly/configure_gitaly.md +++ b/doc/administration/gitaly/configure_gitaly.md @@ -300,7 +300,7 @@ disable enforcement. For more information, see the documentation on configuring ```toml listen_addr = '0.0.0.0:8075' - internal_socket_dir = '/var/opt/gitlab/gitaly' + runtime_dir = '/var/opt/gitlab/gitaly' [logging] format = 'json' @@ -308,6 +308,9 @@ disable enforcement. For more information, see the documentation on configuring dir = '/var/log/gitaly' ``` + For GitLab 14.9 and earlier, set `internal_socket_dir = '/var/opt/gitlab/gitaly'` instead + of `runtime_dir`. + 1. Append the following to `/home/git/gitaly/config.toml` for each respective Gitaly server: On `gitaly1.internal`: diff --git a/doc/administration/gitaly/index.md b/doc/administration/gitaly/index.md index f88a1aa2b4d..8e4464bba43 100644 --- a/doc/administration/gitaly/index.md +++ b/doc/administration/gitaly/index.md @@ -43,23 +43,26 @@ repository storage is either: - Read requests are distributed between multiple Gitaly nodes, which can improve performance. - Write requests are broadcast to repository replicas. -## Guidance regarding Gitaly Cluster +## Before deploying Gitaly Cluster -Gitaly Cluster provides the benefits of fault tolerance, but comes with additional complexity of setup and management. Please review existing technical limitations and considerations prior to deploying Gitaly Cluster. +Gitaly Cluster provides the benefits of fault tolerance, but comes with additional complexity of setup and management. +Before deploying Gitaly Cluster, please review: -- [Known issues](#known-issues) +- Existing [known issues](#known-issues). - [Snapshot limitations](#snapshot-backup-and-recovery-limitations). +- [Configuration guidance](configure_gitaly.md) and [Repository storage options](../repository_storage_paths.md) to make + sure that Gitaly Cluster is the best setup for you. -Please also review the [configuration guidance](configure_gitaly.md) and [Repository storage options](../repository_storage_paths.md) to make sure that Gitaly Cluster is the best set-up for you. Finally, refer to the following guidance: +If you have: -- If you have not yet migrated to Gitaly Cluster and want to continue using NFS, remain on the - service you are using. NFS is supported in 14.x releases. -- If you have not yet migrated to Gitaly Cluster but want to migrate away from NFS, you have two options - a sharded Gitaly instance or Gitaly Cluster. -- If you have migrated to Gitaly Cluster and the limitations and tradeoffs are not suitable for your environment, your options are: - 1. [Migrate off Gitaly Cluster](#migrate-off-gitaly-cluster) back to your NFS solution - 1. [Migrate off Gitaly Cluster](#migrate-off-gitaly-cluster) to NFS solution or to a sharded Gitaly instance. +- Not yet migrated to Gitaly Cluster and want to continue using NFS, remain on the service you are using. NFS is + supported in 14.x releases but is [deprecated](../../update/deprecations.md#nfs-for-git-repository-storage). +Support for storing Git repository data on NFS will end for all versions of GitLab with the release of 15.0. +- Not yet migrated to Gitaly Cluster but want to migrate away from NFS, you have two options: + - A sharded Gitaly instance. + - Gitaly Cluster. -Reach out to your Technical Account Manager or customer support if you have any questions. +Contact your Technical Account Manager or customer support if you have any questions. ### Known issues @@ -69,19 +72,20 @@ the current status of these issues, please refer to the referenced issues and ep | Issue | Summary | How to avoid | |:--------------------------------------------------------------------------------------|:------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:--------------------------------| | Gitaly Cluster + Geo - Issues retrying failed syncs | If Gitaly Cluster is used on a Geo secondary site, repositories that have failed to sync could continue to fail when Geo tries to resync them. Recovering from this state requires assistance from support to run manual steps. Work is in-progress to update Gitaly Cluster to [identify repositories with a unique and persistent identifier](https://gitlab.com/gitlab-org/gitaly/-/issues/3485), which is expected to resolve the issue. | No known solution at this time. | -| Database inconsistencies due to repository access outside of Gitaly Cluster's control | Operations that write to the repository storage that do not go through normal Gitaly Cluster methods can cause database inconsistencies. These can include (but are not limited to) snapshot restoration for cluster node disks, node upgrades which modify files under Git control, or any other disk operation that may touch repository storage external to GitLab. The Gitaly team is actively working to provide manual commands to [reconcile the Praefect database with the repository storage](https://gitlab.com/groups/gitlab-org/-/epics/6723). | Don't directly change repositories on any Gitaly Cluster node at this time. | | Praefect unable to insert data into the database due to migrations not being applied after an upgrade | If the database is not kept up to date with completed migrations, then the Praefect node is unable to perform normal operation. | Make sure the Praefect database is up and running with all migrations completed (For example: `/opt/gitlab/embedded/bin/praefect -config /var/opt/gitlab/praefect/config.toml sql-migrate-status` should show a list of all applied migrations). Consider [requesting live upgrade assistance](https://about.gitlab.com/support/scheduling-live-upgrade-assistance.html) so your upgrade plan can be reviewed by support. | | Restoring a Gitaly Cluster node from a snapshot in a running cluster | Because the Gitaly Cluster runs with consistent state, introducing a single node that is behind will result in the cluster not being able to reconcile the nodes data and other nodes data | Don't restore a single Gitaly Cluster node from a backup snapshot. If you must restore from backup, it's best to snapshot all Gitaly Cluster nodes at the same time and take a database dump of the Praefect database. | ### Snapshot backup and recovery limitations -Gitaly Cluster does not support snapshot backups because these can cause issues where the Praefect -database becomes out of sync with the disk storage. Because of how Praefect rebuilds the replication -metadata of Gitaly disk information during a restore, we recommend using the -[official backup and restore Rake tasks](../../raketasks/backup_restore.md). If you are unable to use this method, please contact customer support for restoration help. +Gitaly Cluster does not support snapshot backups. Snapshot backups can cause issues where the Praefect database becomes +out of sync with the disk storage. Because of how Praefect rebuilds the replication metadata of Gitaly disk information +during a restore, we recommend using the [official backup and restore Rake tasks](../../raketasks/backup_restore.md). -To track progress on work on a solution for manually re-synchronizing the Praefect database with -disk storage, see [this epic](https://gitlab.com/groups/gitlab-org/-/epics/6575). +If you are unable to use this method, please contact customer support for restoration help. + +We are tracking in [this issue](https://gitlab.com/gitlab-org/gitlab/-/issues/351383) improvements to the +[official backup and restore Rake tasks](../../raketasks/backup_restore.md) to add support for incremental backups. For +more information, see [this epic](https://gitlab.com/groups/gitlab-org/-/epics/2094). ### What to do if you are on Gitaly Cluster experiencing an issue or limitation @@ -361,7 +365,8 @@ For more information on configuring Gitaly Cluster, see [Configure Gitaly Cluste ## Migrating to Gitaly Cluster -Please see [current guidance on Gitaly Cluster](#guidance-regarding-gitaly-cluster). The basic process for migrating to Gitaly Cluster involves: +See the [Before deploying Gitaly Cluster](#before-deploying-gitaly-cluster) section before continuing. The basic process +for migrating to Gitaly Cluster involves: 1. Create the required storage. Refer to [repository storage recommendations](faq.md#what-are-some-repository-storage-recommendations). @@ -378,16 +383,14 @@ NOTE: GitLab requires a `default` repository storage to be configured. [Read more about this limitation](configure_gitaly.md#gitlab-requires-a-default-repository-storage). -### Migrate off Gitaly Cluster +## Migrate off Gitaly Cluster -If you have repositories stored on a Gitaly Cluster, but you'd like to migrate -them back to direct Gitaly storage: +If the limitations and tradeoffs of Gitaly Cluster are found to be not suitable for your environment, you can Migrate +off Gitaly Cluster to a sharded Gitaly instance: -1. Create and configure a new - [Gitaly server](configure_gitaly.md#run-gitaly-on-its-own-server). -1. [Move the repositories](../operations/moving_repositories.md#move-repositories) - to the newly created storage. You can move them by shard or by group, which gives you the opportunity to spread them over - multiple Gitaly servers. +1. Create and configure a new [Gitaly server](configure_gitaly.md#run-gitaly-on-its-own-server). +1. [Move the repositories](../operations/moving_repositories.md#move-repositories) to the newly created storage. You can + move them by shard or by group, which gives you the opportunity to spread them over multiple Gitaly servers. ## Monitor Gitaly and Gitaly Cluster diff --git a/doc/administration/gitaly/praefect.md b/doc/administration/gitaly/praefect.md index d7a06994b6c..7dee12a6690 100644 --- a/doc/administration/gitaly/praefect.md +++ b/doc/administration/gitaly/praefect.md @@ -231,6 +231,16 @@ instructions only work on Omnibus-provided PostgreSQL: The database used by Praefect is now configured. +You can now configure Praefect to use the database: + +```ruby +praefect['database_host'] = POSTGRESQL_HOST +praefect['database_port'] = 5432 +praefect['database_user'] = 'praefect' +praefect['database_password'] = PRAEFECT_SQL_PASSWORD +praefect['database_dbname'] = 'praefect_production' +``` + If you see Praefect database errors after configuring PostgreSQL, see [troubleshooting steps](troubleshooting.md#relation-does-not-exist-errors). @@ -619,7 +629,7 @@ Note the following: environment variable so that the Gitaly certificate is trusted. For example: ```shell - sudo SSL_CERT_DIR=/etc/gitlab/trusted_certs /opt/gitlab/embedded/bin/praefect -config /var/opt/gitlab/praefect/config.toml dial-nodes + sudo SSL_CERT_DIR=/etc/gitlab/trusted-certs /opt/gitlab/embedded/bin/praefect -config /var/opt/gitlab/praefect/config.toml dial-nodes ``` - You can configure Praefect servers with both an unencrypted listening address @@ -1197,7 +1207,7 @@ You can configure: ``` If `default_replication_factor` is unset, the repositories are always replicated on every node defined in `virtual_storages`. If a new -node is introduced to the virtual storage, both new and existing repositories are replicated to the node automatically. +node is introduced to the virtual storage, both new and existing repositories are replicated to the node automatically. ## Automatic failover and primary election strategies diff --git a/doc/administration/gitaly/troubleshooting.md b/doc/administration/gitaly/troubleshooting.md index 516af4ca469..1be0bf23ed5 100644 --- a/doc/administration/gitaly/troubleshooting.md +++ b/doc/administration/gitaly/troubleshooting.md @@ -16,7 +16,8 @@ Before troubleshooting, see the Gitaly and Gitaly Cluster The following sections provide possible solutions to Gitaly errors. -See also [Gitaly timeout](../../user/admin_area/settings/gitaly_timeouts.md) settings. +See also [Gitaly timeout](../../user/admin_area/settings/gitaly_timeouts.md) settings, +and our advice on [parsing the `gitaly/current` file](../troubleshooting/log_parsing.md#parsing-gitalycurrent). ### Check versions when using standalone Gitaly servers @@ -614,7 +615,7 @@ Possible solutions: ## Profiling Gitaly Gitaly exposes several of Golang's built-in performance profiling tools on the Prometheus listen port. For example, if Prometheus is listening -on port `9236` of the GitLab server: +on port `9236` of the GitLab server: - Get a list of running `goroutines` and their backtraces: diff --git a/doc/administration/housekeeping.md b/doc/administration/housekeeping.md index 4de48aa3f14..71b8439f70a 100644 --- a/doc/administration/housekeeping.md +++ b/doc/administration/housekeeping.md @@ -40,8 +40,6 @@ The following housekeeping options are available: - **Enable automatic repository housekeeping**: Regularly run `git repack` and `git gc`. If you keep this setting disabled for a long time, Git repository access on your GitLab server becomes slower and your repositories use more disk space. -- **Enable Git pack file bitmap creation**: Create pack file bitmaps which accelerates `git clone` - performance. Makes housekeeping take a little longer. - **Incremental repack period**: Number of Git pushes after which an incremental `git repack` is run. - **Full repack period**: Number of Git pushes after which a full `git repack` is run. diff --git a/doc/administration/incoming_email.md b/doc/administration/incoming_email.md index caade32b7c2..63f8ce3c4cb 100644 --- a/doc/administration/incoming_email.md +++ b/doc/administration/incoming_email.md @@ -564,7 +564,7 @@ to receive sub-addressed mail. To enable sub-addressing: -1. Download and install the `ExchangeOnlineManagement` module from the [PowerShell gallery](https://www.powershellgallery.com/packages/ExchangeOnlineManagement/). +1. Download and install the `ExchangeOnlineManagement` module from the [PowerShell gallery](https://www.powershellgallery.com/packages/ExchangeOnlineManagement/2.0.5). 1. In PowerShell, run the following commands: ```powershell @@ -766,6 +766,8 @@ This example for Omnibus GitLab assumes you're using the following mailbox: `inc ##### Configure Microsoft Graph +> Alternative Azure deployments [introduced](https://gitlab.com/gitlab-org/omnibus-gitlab/-/merge_requests/5978) in GitLab 14.9. + ```ruby gitlab_rails['incoming_email_enabled'] = true @@ -788,4 +790,19 @@ gitlab_rails['incoming_email_inbox_options'] = { } ``` +For Microsoft Cloud for US Government or [other Azure deployments](https://docs.microsoft.com/en-us/graph/deployments), configure the `azure_ad_endpoint` and `graph_endpoint` settings. + +- Example for Microsoft Cloud for US Government: + +```ruby +gitlab_rails['incoming_email_inbox_options'] = { + 'azure_ad_endpoint': 'https://login.microsoftonline.us', + 'graph_endpoint': 'https://graph.microsoft.us', + 'tenant_id': '<YOUR-TENANT-ID>', + 'client_id': '<YOUR-CLIENT-ID>', + 'client_secret': '<YOUR-CLIENT-SECRET>', + 'poll_interval': 60 # Optional +} +``` + The Microsoft Graph API is not yet supported in source installations. See [this issue](https://gitlab.com/gitlab-org/gitlab/-/issues/326169) for more details. diff --git a/doc/administration/index.md b/doc/administration/index.md index 50dc2bc905a..73ea4a8e1d0 100644 --- a/doc/administration/index.md +++ b/doc/administration/index.md @@ -32,7 +32,7 @@ Learn how to install, configure, update, and maintain your GitLab instance. - [Install](../install/index.md): Requirements, directory structures, and installation methods. - [Reference architectures](reference_architectures/index.md): Add additional resources to support more users. - - [Installing GitLab on Amazon Web Services (AWS)](../install/aws/index.md): Set up GitLab on Amazon AWS. +- [Installing GitLab on Amazon Web Services (AWS)](../install/aws/index.md): Set up GitLab on Amazon AWS. - [Geo](geo/index.md): Replicate your GitLab instance to other geographic locations as a read-only fully operational version. - [Disaster Recovery](geo/disaster_recovery/index.md): Quickly fail-over to a different site with minimal effort in a disaster situation. - [Add License](../user/admin_area/license.md): Add a license at install time to unlock features that are in paid tiers of GitLab. @@ -40,7 +40,7 @@ Learn how to install, configure, update, and maintain your GitLab instance. ### Configuring GitLab - [Adjust your instance's time zone](timezone.md): Customize the default time zone of GitLab. -- [System hooks](../system_hooks/system_hooks.md): Notifications when users, projects and keys are changed. +- [System hooks](system_hooks.md): Notifications when users, projects and keys are changed. - [Security](../security/index.md): Learn what you can do to further secure your GitLab instance. - [Usage statistics, version check, and Service Ping](../user/admin_area/settings/usage_statistics.md): Enable or disable information about your instance to be sent to GitLab, Inc. - [Global user settings](user_settings.md): Configure instance-wide user permissions. @@ -126,7 +126,7 @@ Learn how to install, configure, update, and maintain your GitLab instance. - [Sync LDAP](auth/ldap/index.md) - [Kerberos authentication](../integration/kerberos.md) - See also other [authentication](../topics/authentication/index.md#gitlab-administrators) topics (for example, enforcing 2FA). -- [Email users](../tools/email.md): Email GitLab users from GitLab. +- [Email users](../user/admin_area/email_from_gitlab.md): Email GitLab users from GitLab. - [User Cohorts](../user/admin_area/user_cohorts.md): Display the monthly cohorts of new users and their activities over time. - [Audit events](audit_events.md): View the changes made on the GitLab server for: - Groups and projects. @@ -146,7 +146,7 @@ Learn how to install, configure, update, and maintain your GitLab instance. - [Issue closing pattern](issue_closing_pattern.md): Customize how to close an issue from commit messages. - [Gitaly](gitaly/index.md): Configuring Gitaly, the Git repository storage service for GitLab. - [Default labels](../user/admin_area/labels.md): Create labels that are automatically added to every new project. -- [Restrict the use of public or internal projects](../public_access/public_access.md#restrict-use-of-public-or-internal-projects): Restrict the use of visibility levels for users when they create a project or a snippet. +- [Restrict the use of public or internal projects](../user/public_access.md#restrict-use-of-public-or-internal-projects): Restrict the use of visibility levels for users when they create a project or a snippet. - [Custom project templates](../user/admin_area/custom_project_templates.md): Configure a set of projects to be used as custom templates when creating a new project. ## Package Registry administration @@ -218,6 +218,7 @@ Learn how to install, configure, update, and maintain your GitLab instance. - [Troubleshooting Elasticsearch](troubleshooting/elasticsearch.md) - [Navigating GitLab via Rails console](troubleshooting/navigating_gitlab_via_rails_console.md) - [GitLab application limits](instance_limits.md) +- [Responding to security incidents](../security/responding_to_security_incidents.md) ### Support Team Docs diff --git a/doc/administration/instance_limits.md b/doc/administration/instance_limits.md index 11a34f5b5f8..0a0c538caa2 100644 --- a/doc/administration/instance_limits.md +++ b/doc/administration/instance_limits.md @@ -240,7 +240,7 @@ Activity history for projects and individuals' profiles was limited to one year > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/14939) in GitLab 12.7. -There is a limit when embedding metrics in GitLab Flavored Markdown (GFM) for performance reasons. +There is a limit when embedding metrics in GitLab Flavored Markdown (GLFM) for performance reasons. - **Max limit**: 100 embeds. @@ -383,6 +383,20 @@ Plan.default.actual_limits.update!(ci_active_jobs: 500) Set the limit to `0` to disable it. +### Maximum time jobs can run + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/16777) in GitLab 12.3. + +The default maximum time that jobs can run for is 60 minutes. Jobs that run for +more than 60 minutes time out. + +You can change the maximum time a job can run before it times out: + +- At the project-level in the [project's CI/CD settings](../ci/pipelines/settings.md#set-a-limit-for-how-long-jobs-can-run) + for a given project. This limit must be between 10 minutes and 1 month. +- At the [runner level](../ci/runners/configure_runners.md#set-maximum-job-timeout-for-a-runner). + This limit must be 10 minutes or longer. + ### Maximum number of deployment jobs in a pipeline > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/46931) in GitLab 13.7. @@ -858,7 +872,7 @@ See the [documentation about Snippets settings](snippets/index.md). ## Design Management limits -See the [Design Management Limitations](../user/project/issues/design_management.md#limitations) section. +See the limits in the [Add a design to an issue](../user/project/issues/design_management.md#add-a-design-to-an-issue) section. ## Push Event Limits @@ -962,3 +976,16 @@ retarget merge requests pointing to the now-merged branch. To learn more, read ## CDN-based limits on GitLab.com In addition to application-based limits, GitLab.com is configured to use Cloudflare's standard DDoS protection and Spectrum to protect Git over SSH. Cloudflare terminates client TLS connections but is not application aware and cannot be used for limits tied to users or groups. Cloudflare page rules and rate limits are configured with Terraform. These configurations are [not public](https://about.gitlab.com/handbook/communication/#not-public) because they include security and abuse implementations that detect malicious activities and making them public would undermine those operations. + +## Container Repository tag deletion limit + +Container repository tags are in the Container Registry and, as such, each tag deletion will trigger network requests to the Container Registry. Because of this, we limit the number of tags that a single API call can delete to 20. + +## Project-level Secure Files API limits + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/78227) in GitLab 14.8. + +The [secure files API](../api/secure_files.md) enforces the following limits: + +- Files must be smaller than 5 MB. +- Projects cannot have more than 100 secure files. diff --git a/doc/administration/integration/kroki.md b/doc/administration/integration/kroki.md index 008d33c6c94..0f02e3783a5 100644 --- a/doc/administration/integration/kroki.md +++ b/doc/administration/integration/kroki.md @@ -10,7 +10,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w > - Support for reStructuredText and Textile documents [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/324766) in GitLab 13.12. When [Kroki](https://kroki.io) integration is enabled and configured in -GitLab you can use it to create diagrams in AsciiDoc, Markdown, reStructuredText, and Textile documents. +GitLab, you can use it to create diagrams in AsciiDoc, Markdown, reStructuredText, and Textile documents. ## Kroki Server diff --git a/doc/administration/integration/plantuml.md b/doc/administration/integration/plantuml.md index 5a9699b3a39..cdf6d48ad41 100644 --- a/doc/administration/integration/plantuml.md +++ b/doc/administration/integration/plantuml.md @@ -213,7 +213,7 @@ After configuring your local PlantUML server, you're ready to enable the PlantUM 1. On the left sidebar, go to **Settings > General** and expand the **PlantUML** section. 1. Select the **Enable PlantUML** checkbox. 1. Set the PlantUML instance as `https://gitlab.example.com/-/plantuml/`, - and click **Save changes**. + and select **Save changes**. Depending on your PlantUML and GitLab version numbers, you may also need to take these steps: diff --git a/doc/administration/job_artifacts.md b/doc/administration/job_artifacts.md index 544a5973052..03a0162d526 100644 --- a/doc/administration/job_artifacts.md +++ b/doc/administration/job_artifacts.md @@ -307,33 +307,6 @@ To migrate back to local storage: ## Expiring artifacts -> [In GitLab 14.6](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/76504), we improved the performance of removing expired artifacts, introduced [with a flag](feature_flags.md) named `ci_destroy_all_expired_service`. Disabled by default. - -FLAG: -On self-managed GitLab, by default this feature is not available. To make it available, ask an administrator to -[enable the feature flag](feature_flags.md) named `ci_destroy_all_expired_service`. The feature is not ready for -production use. -On GitLab.com, this feature is not available. - -### Removing expired job artifacts on GitLab self-managed instances - -In the process of migrating old artifacts for our SaaS customers, we are working to resolve any potential unrecoverable data loss for self-managed customers for artifacts that they may not want deleted yet. Before we can use the more performant way of cleaning up expired artifacts, we need to do some remediation to make sure customers don't lose their data, which is part of our effort in [the relevant epic](https://gitlab.com/groups/gitlab-org/-/epics/7097). - -Two options are available: - -- If you don't need any artifacts created before 2020-06-23, an Administrator can enable the worker for removing expired CI/CD artifacts: - - ```ruby - Feature.enable(:ci_destroy_all_expired_service) - ``` - -- If you want to keep any artifacts (including job logs) before 2020-06-23, follow the [progress of the migration effort](https://gitlab.com/groups/gitlab-org/-/epics/7097) where we work on a resolution to have this flag fully enabled in a future release. - -Alternatively, Administrators can also run commands in the Rails console to -[delete artifacts from completed jobs prior to a specific date](#delete-job-artifacts-from-jobs-completed-before-a-specific-date). - -### Usage details - If [`artifacts:expire_in`](../ci/yaml/index.md#artifactsexpire_in) is used to set an expiry for the artifacts, they are marked for deletion right after that date passes. Otherwise, they expire per the [default artifacts expiration setting](../user/admin_area/settings/continuous_integration.md). @@ -641,4 +614,4 @@ WARNING: Uploading artifacts as "archive" to coordinator... POST https://gitlab. FATAL: invalid argument ``` -If a job artifact fails to upload with the above error when using consolidated object storage, make sure you are [using separate buckets](object_storage.md#use-separate-buckets) for each data type. +If a job artifact fails to upload with the above error when using consolidated object storage, make sure you are [using separate buckets](object_storage.md#use-separate-buckets) for each data type. diff --git a/doc/administration/monitoring/gitlab_self_monitoring_project/index.md b/doc/administration/monitoring/gitlab_self_monitoring_project/index.md index 09cef51d14c..6f2e3cce0fa 100644 --- a/doc/administration/monitoring/gitlab_self_monitoring_project/index.md +++ b/doc/administration/monitoring/gitlab_self_monitoring_project/index.md @@ -8,18 +8,18 @@ info: To determine the technical writer assigned to the Stage/Group associated w > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/32351) in GitLab 12.7 [with a flag](../../feature_flags.md) named `self_monitoring_project`. Disabled by default. > - Generally available in GitLab 12.8. [Feature flag `self_monitoring_project`](https://gitlab.com/gitlab-org/gitlab/-/issues/198511) removed. -> - [Deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/348909) in GitLab 14.9. Planned for removal in GitLab 15.0. +> - [Deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/348909) in GitLab 14.9. Planned for removal in GitLab 16.0. WARNING: This feature is in its end-of-life process. It is [deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/348909) -for use in GitLab 14.9, and is planned for removal in GitLab 15.0. +in GitLab 14.9, and is planned for removal in GitLab 16.0. GitLab provides administrators insights into the health of their GitLab instance. To provide a native experience (similar interacting with an application deployed using GitLab), a project called **Monitoring** is created: -- With [internal visibility](../../../public_access/public_access.md#internal-projects-and-groups). +- With [internal visibility](../../../user/public_access.md#internal-projects-and-groups). - Under a group called **GitLab Instance**. The project is created specifically for visualizing and configuring the monitoring of your GitLab diff --git a/doc/administration/monitoring/performance/request_profiling.md b/doc/administration/monitoring/performance/request_profiling.md index f8a66f6f48f..6cd20132092 100644 --- a/doc/administration/monitoring/performance/request_profiling.md +++ b/doc/administration/monitoring/performance/request_profiling.md @@ -10,7 +10,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w WARNING: This feature is in its end-of-life process. It is [deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/352488) -for use in GitLab 14.8, and is planned for removal in GitLab 15.0. +in GitLab 14.8, and is planned for removal in GitLab 15.0. To profile a request: diff --git a/doc/administration/monitoring/prometheus/gitlab_metrics.md b/doc/administration/monitoring/prometheus/gitlab_metrics.md index a1bfae678c7..08f7d5095da 100644 --- a/doc/administration/monitoring/prometheus/gitlab_metrics.md +++ b/doc/administration/monitoring/prometheus/gitlab_metrics.md @@ -88,7 +88,6 @@ The following metrics are available: | `gitlab_transaction_event_stuck_import_jobs_total` | Counter | 9.4 | Count of stuck import jobs | `projects_without_jid_count`, `projects_with_jid_count` | | `gitlab_transaction_event_update_build_total` | Counter | 9.4 | Counter for update build for API `/jobs/request/:id` | | | `gitlab_transaction_new_redis_connections_total` | Counter | 9.4 | Counter for new Redis connections | | -| `gitlab_transaction_queue_duration_total` | Counter | 9.4 | Duration jobs were enqueued before processing | | | `gitlab_transaction_rails_queue_duration_total` | Counter | 9.4 | Measures latency between GitLab Workhorse forwarding a request to Rails | `controller`, `action` | | `gitlab_transaction_view_duration_total` | Counter | 9.4 | Duration for views | `controller`, `action`, `view` | | `gitlab_view_rendering_duration_seconds` | Histogram | 10.2 | Duration for views (histogram) | `controller`, `action`, `view` | diff --git a/doc/administration/nfs.md b/doc/administration/nfs.md index 50c8102f366..ef5d26ac845 100644 --- a/doc/administration/nfs.md +++ b/doc/administration/nfs.md @@ -246,7 +246,7 @@ use the `hard` option, because (from the man page): > use the soft option only when client responsiveness is more important than data integrity Other vendors make similar recommendations, including -[System Applications and Products in Data Processing (SAP)](http://wiki.scn.sap.com/wiki/x/PARnFQ) and NetApp's +[System Applications and Products in Data Processing (SAP)](https://wiki.scn.sap.com/wiki/display/Basis/Recommended+mount+options+for+read-write+directories) and NetApp's [knowledge base](https://kb.netapp.com/Advice_and_Troubleshooting/Data_Storage_Software/ONTAP_OS/What_are_the_differences_between_hard_mount_and_soft_mount), they highlight that if the NFS client driver caches data, `soft` means there is no certainty if writes by GitLab are actually on disk. diff --git a/doc/administration/object_storage.md b/doc/administration/object_storage.md index 8ea4ad9019e..6ea433f95e9 100644 --- a/doc/administration/object_storage.md +++ b/doc/administration/object_storage.md @@ -17,7 +17,7 @@ GitLab has been tested by vendors and customers on a number of object storage pr - [Amazon S3](https://aws.amazon.com/s3/) - [Google Cloud Storage](https://cloud.google.com/storage) -- [Digital Ocean Spaces](https://www.digitalocean.com/products/spaces/) +- [Digital Ocean Spaces](https://www.digitalocean.com/products/spaces) - [Oracle Cloud Infrastructure](https://docs.cloud.oracle.com/en-us/iaas/Content/Object/Tasks/s3compatibleapi.htm) - [OpenStack Swift](https://docs.openstack.org/swift/latest/s3_compat.html) - [Azure Blob storage](https://docs.microsoft.com/en-us/azure/storage/blobs/storage-blobs-introduction) diff --git a/doc/administration/operations/fast_ssh_key_lookup.md b/doc/administration/operations/fast_ssh_key_lookup.md index 8877b1266e0..477a11dd899 100644 --- a/doc/administration/operations/fast_ssh_key_lookup.md +++ b/doc/administration/operations/fast_ssh_key_lookup.md @@ -31,7 +31,8 @@ able to accept a fingerprint. Check the version of OpenSSH on your server with ` ## Fast lookup is required for Geo **(PREMIUM)** -By default, GitLab manages an `authorized_keys` file that is located in the +Unlike [Cloud Native GitLab](https://docs.gitlab.com/charts/), Omnibus GitLab by default +manages an `authorized_keys` file that is located in the `git` user's home directory. For most installations, this will be located under `/var/opt/gitlab/.ssh/authorized_keys`, but you can use the following command to locate the `authorized_keys` on your system: @@ -83,7 +84,7 @@ file (start the line with a `#` to comment it), and from your local machine, att ssh -T git@gitlab.example.com ``` -A successful pull or [welcome message](../../ssh/index.md#verify-that-you-can-connect) would mean that GitLab was able to find the key in the database, +A successful pull or [welcome message](../../user/ssh.md#verify-that-you-can-connect) would mean that GitLab was able to find the key in the database, since it is not present in the file anymore. NOTE: diff --git a/doc/administration/operations/puma.md b/doc/administration/operations/puma.md index 9e828b39f46..c12f75989c3 100644 --- a/doc/administration/operations/puma.md +++ b/doc/administration/operations/puma.md @@ -108,6 +108,12 @@ To change the worker timeout to 600 seconds: ## Disable Puma clustered mode in memory-constrained environments +WARNING: +This is an experimental [Alpha feature](../../policy/alpha-beta-support.md#alpha-features) and subject to change without notice. The feature +is not ready for production use. If you want to use this feature, we recommend testing +with non-production data first. See the [known issues](#puma-single-mode-known-issues) +for additional details. + In a memory-constrained environment with less than 4GB of RAM available, consider disabling Puma [clustered mode](https://github.com/puma/puma#clustered-mode). @@ -131,6 +137,8 @@ For details on Puma worker and thread settings, see the [Puma requirements](../. The downside of running Puma in this configuration is the reduced throughput, which can be considered a fair tradeoff in a memory-constrained environment. +### Puma single mode known issues + When running Puma in single mode, some features are not supported: - [Phased restart](https://gitlab.com/gitlab-org/gitlab/-/issues/300665) diff --git a/doc/administration/package_information/postgresql_versions.md b/doc/administration/package_information/postgresql_versions.md index e707cb70187..c80437221c4 100644 --- a/doc/administration/package_information/postgresql_versions.md +++ b/doc/administration/package_information/postgresql_versions.md @@ -26,8 +26,8 @@ Read more about update policies and warnings in the PostgreSQL | GitLab version | PostgreSQL versions | Default version for fresh installs | Default version for upgrades | Notes | | -------------- | --------------------- | ---------------------------------- | ---------------------------- | ----- | -| 14.1 | 12.6, 13.3 | 12.6 | 12.6 | PostgreSQL 13 available for fresh installations if not using [Geo](../geo/index.md#requirements-for-running-geo) or [Patroni](../postgresql/index.md#postgresql-replication-and-failover-with-omnibus-gitlab). -| 14.0 | 12.6 | 12.6 | 12.6 | HA installations with repmgr are no longer supported and will be prevented from upgrading to Omnibus GitLab 14.0 | +| 14.1 | 12.7, 13.3 | 12.7 | 12.7 | PostgreSQL 13 available for fresh installations if not using [Geo](../geo/index.md#requirements-for-running-geo) or [Patroni](../postgresql/index.md#postgresql-replication-and-failover-with-omnibus-gitlab). +| 14.0 | 12.7 | 12.7 | 12.7 | HA installations with repmgr are no longer supported and will be prevented from upgrading to Omnibus GitLab 14.0 | | 13.8 | 11.9, 12.4 | 12.4 | 12.4 | Package upgrades automatically performed PostgreSQL upgrade for nodes that are not part of a Geo or HA cluster.). | | 13.7 | 11.9, 12.4 | 12.4 | 11.9 | For upgrades users can manually upgrade to 12.4 following the [upgrade docs](https://docs.gitlab.com/omnibus/settings/database.html#gitlab-133-and-later). | | 13.4 | 11.9, 12.4 | 11.9 | 11.9 | Package upgrades aborted if users not running PostgreSQL 11 already | diff --git a/doc/administration/package_information/supported_os.md b/doc/administration/package_information/supported_os.md index 547b8bf8658..647620858d9 100644 --- a/doc/administration/package_information/supported_os.md +++ b/doc/administration/package_information/supported_os.md @@ -24,14 +24,15 @@ The following lists the currently supported OSs and their possible EOL dates. | Debian 10 | GitLab CE / GitLab EE 12.2.0 | amd64, arm64 | 2024 | <https://wiki.debian.org/LTS> | | Debian 11 | GitLab CE / GitLab EE 14.6.0 | amd64, arm64 | 2026 | <https://wiki.debian.org/LTS> | | OpenSUSE 15.3 | GitLab CE / GitLab EE 14.5.0 | x86_64, aarch64 | Nov 2022 | <https://en.opensuse.org/Lifetime> | +| RHEL 8 | GitLab CE / GitLab EE 12.8.1 | x86_64, arm64 | May 2024 | <https://access.redhat.com/support/policy/updates/errata/#Life_Cycle_Dates> | | SLES 12 | GitLab EE 9.0.0 | x86_64 | Oct 2027 | <https://www.suse.com/lifecycle/> | | Ubuntu 18.04 | GitLab CE / GitLab EE 10.7.0 | amd64 | April 2023 | <https://wiki.ubuntu.com/Releases> | | Ubuntu 20.04 | GitLab CE / GitLab EE 13.2.0 | amd64, arm64 | April 2025 | <https://wiki.ubuntu.com/Releases> | -| Amazon Linux 2 | GitLab CE / GitLab EE 14.9.0 | amd64, arm64 | June 2023 | <https://aws.amazon.com/amazon-linux-2/faqs> | -| Raspbian Buster | GitLab CE 12.2.0 | armhf | 2022 | <https://wiki.debian.org/DebianReleases#Production_Releases> | +| Amazon Linux 2 | GitLab CE / GitLab EE 14.9.0 | amd64, arm64 | June 2023 | <https://aws.amazon.com/amazon-linux-2/faqs/> | +| Raspberry Pi OS (Buster) (formerly known as Raspbian Buster) | GitLab CE 12.2.0 | armhf | 2024 | <https://www.raspberrypi.com/news/new-old-functionality-with-raspberry-pi-os-legacy/> | NOTE: -CentOS 8 will be EOL on December 31, 2021. In GitLab 14.5 and later, +CentOS 8 was EOL on December 31, 2021. In GitLab 14.5 and later, [CentOS builds work in AlmaLinux](https://gitlab.com/gitlab-org/distribution/team-tasks/-/issues/954#note_730198505). We will officially support all distributions that are binary compatible with Red Hat Enterprise Linux. This gives users a path forward for their CentOS 8 builds at its end of life. @@ -78,7 +79,7 @@ release for them can be found below: | Ubuntu 14.04 | [April 2019](https://ubuntu.com/info/release-end-of-life) | [GitLab CE](https://packages.gitlab.com/app/gitlab/gitlab-ce/search?q=gitlab-ce_11.10&dist=ubuntu%2Ftrusty) / [GitLab EE](https://packages.gitlab.com/app/gitlab/gitlab-ee/search?q=gitlab-ee_11.10&dist=ubuntu%2Ftrusty) 11.10 | | OpenSUSE 42.3 | [July 2019](https://en.opensuse.org/Lifetime#Discontinued_distributions) | [GitLab CE](https://packages.gitlab.com/app/gitlab/gitlab-ce/search?q=gitlab-ce-12.1&dist=opensuse%2F42.3) / [GitLab EE](https://packages.gitlab.com/app/gitlab/gitlab-ee/search?q=gitlab-ee-12.1&dist=opensuse%2F42.3) 12.1 | | OpenSUSE 15.0 | [December 2019](https://en.opensuse.org/Lifetime#Discontinued_distributions) | [GitLab CE](https://packages.gitlab.com/app/gitlab/gitlab-ce/search?q=gitlab-ce-12.5&dist=opensuse%2F15.0) / [GitLab EE](https://packages.gitlab.com/app/gitlab/gitlab-ee/search?q=gitlab-ee-12.5&dist=opensuse%2F15.0) 12.5 | -| Raspbian Stretch | [June 2020](https://downloads.raspberrypi.org/raspbian/images/raspbian-2019-04-09/) | [GitLab CE](https://packages.gitlab.com/app/gitlab/raspberry-pi2/search?q=gitlab-ce_13.2&dist=raspbian%2Fstretch) 13.3 | +| Raspbian Stretch | [June 2020](https://downloads.raspberrypi.org/raspbian/images/raspbian-2019-04-09/) | [GitLab CE](https://packages.gitlab.com/app/gitlab/raspberry-pi2/search?q=gitlab-ce_13.3&dist=raspbian%2Fstretch) 13.3 | | Debian Jessie | [June 2020](https://www.debian.org/News/2020/20200709) | [GitLab CE](https://packages.gitlab.com/app/gitlab/gitlab-ce/search?q=gitlab-ce_13.2&dist=debian%2Fjessie) / [GitLab EE](https://packages.gitlab.com/app/gitlab/gitlab-ee/search?q=gitlab-ee_13.2&dist=debian%2Fjessie) 13.3 | | CentOS 6 | [November 2020](https://wiki.centos.org/About/Product) | [GitLab CE](https://packages.gitlab.com/app/gitlab/gitlab-ce/search?q=13.6&filter=all&filter=all&dist=el%2F6) / [GitLab EE](https://packages.gitlab.com/app/gitlab/gitlab-ee/search?q=13.6&filter=all&filter=all&dist=el%2F6) 13.6 | | OpenSUSE 15.1 | [November 2020](https://en.opensuse.org/Lifetime#Discontinued_distributions) | [GitLab CE](https://packages.gitlab.com/app/gitlab/gitlab-ce/search?q=gitlab-ce-13.12&dist=opensuse%2F15.1) / [GitLab EE](https://packages.gitlab.com/app/gitlab/gitlab-ee/search?q=gitlab-ee-13.12&dist=opensuse%2F15.1) 13.12 | diff --git a/doc/administration/packages/container_registry.md b/doc/administration/packages/container_registry.md index 7bf7362c68b..d6f5588ebc9 100644 --- a/doc/administration/packages/container_registry.md +++ b/doc/administration/packages/container_registry.md @@ -903,6 +903,10 @@ You can also [run cleanup on a schedule](../../user/packages/container_registry/ ## Container Registry garbage collection +NOTE: +Retention policies within your object storage provider, such as Amazon S3 Lifecycle, may prevent +objects from being properly deleted. + Container Registry can use considerable amounts of disk space. To clear up some unused layers, the registry includes a garbage collect command. diff --git a/doc/administration/packages/dependency_proxy.md b/doc/administration/packages/dependency_proxy.md index 93c6585230f..4c50cfeeb0f 100644 --- a/doc/administration/packages/dependency_proxy.md +++ b/doc/administration/packages/dependency_proxy.md @@ -264,3 +264,29 @@ Feature.disable(:dependency_proxy_for_private_groups) # Re-enable the authentication Feature.enable(:dependency_proxy_for_private_groups) ``` + +## Changing the JWT expiration + +The Dependency Proxy follows the [Docker v2 token authentication flow](https://docs.docker.com/registry/spec/auth/token/), +issuing the client a JWT to use for the pull requests. The token expiration time is a configurable +using the application setting `container_registry_token_expire_delay`. It can be changed from the +rails console: + +```ruby +# update the JWT expiration to 30 minutes +ApplicationSetting.update(container_registry_token_expire_delay: 30) +``` + +The default expiration and the expiration on GitLab.com is 15 minutes. + +## Using the dependency proxy behind a proxy + +1. Edit `/etc/gitlab/gitlab.rb` and add the following lines: + + ```ruby + gitlab_workhorse['env'] = { + "http_proxy" => "http://USERNAME:PASSWORD@example.com:8080", + "https_proxy" => "http://USERNAME:PASSWORD@example.com:8080" + ``` + +1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. diff --git a/doc/administration/packages/index.md b/doc/administration/packages/index.md index abf5c46114b..b122cb9db90 100644 --- a/doc/administration/packages/index.md +++ b/doc/administration/packages/index.md @@ -10,9 +10,7 @@ GitLab Packages allows organizations to use GitLab as a private repository for a variety of common package managers. Users are able to build and publish packages, which can be easily consumed as a dependency in downstream projects. -The Packages feature allows GitLab to act as a repository for the following: - -The Package Registry supports the following formats: +The Packages feature allows GitLab to act as a repository and supports the following formats: | Package type | GitLab version | |-------------------------------------------------------------------|----------------| diff --git a/doc/administration/pages/index.md b/doc/administration/pages/index.md index 56ccf70d169..114894c01f0 100644 --- a/doc/administration/pages/index.md +++ b/doc/administration/pages/index.md @@ -1,6 +1,6 @@ --- -stage: Release -group: Release +stage: Create +group: Editor info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments description: 'Learn how to administer GitLab Pages.' --- @@ -243,6 +243,7 @@ control over how the Pages daemon runs and serves content in your environment. | `enable` | Enable or disable GitLab Pages on the current system. | | `external_http` | Configure Pages to bind to one or more secondary IP addresses, serving HTTP requests. Multiple addresses can be given as an array, along with exact ports, for example `['1.2.3.4', '1.2.3.5:8063']`. Sets value for `listen_http`. | | `external_https` | Configure Pages to bind to one or more secondary IP addresses, serving HTTPS requests. Multiple addresses can be given as an array, along with exact ports, for example `['1.2.3.4', '1.2.3.5:8063']`. Sets value for `listen_https`. | +| `server_shutdown_timeout` | GitLab Pages server shutdown timeout in seconds (default: 30s). | | `gitlab_client_http_timeout` | GitLab API HTTP client connection timeout in seconds (default: 10s). | | `gitlab_client_jwt_expiry` | JWT Token expiry time in seconds (default: 30s). | | `gitlab_cache_expiry` | The maximum time a domain's configuration is stored in the cache (default: 600s). | @@ -292,6 +293,10 @@ control over how the Pages daemon runs and serves content in your environment. | `rate_limit_source_ip_burst` | Rate limit per source IP maximum burst allowed per second. | | `rate_limit_domain` | Rate limit per domain in number of requests per second. Set to `0` to disable this feature. | | `rate_limit_domain_burst` | Rate limit per domain maximum burst allowed per second. | +| `server_read_timeout` | Maximum duration to read the request headers and body. For no timeout, set to `0` or a negative value. Default: `5s` | +| `server_read_header_timeout` | Maximum duration to read the request headers. For no timeout, set to `0` or a negative value. Default: `1s` | +| `server_write_timeout` | Maximum duration to write all files in the response. Larger files require more time. For no timeout, set to `0` or a negative value. Default: `5m` | +| `server_keep_alive` | The `Keep-Alive` period for network connections accepted by this listener. If `0`, `Keep-Alive` is enabled if supported by the protocol and operating system. If negative, `Keep-Alive` is disabled. Default: `15s` | ## Advanced configuration @@ -1392,15 +1397,15 @@ in all of your GitLab Pages instances. Connections will time out when using a Network Load Balancer with client IP preservation enabled and [the request is looped back to the source server](https://docs.aws.amazon.com/elasticloadbalancing/latest/network/load-balancer-troubleshooting.html#loopback-timeout). This can happen to GitLab instances with multiple servers -running both the core GitLab application and GitLab Pages. This can also happen when a single +running both the core GitLab application and GitLab Pages. This can also happen when a single container is running both the core GitLab application and GitLab Pages. AWS [recommends using an IP target type](https://aws.amazon.com/premiumsupport/knowledge-center/target-connection-fails-load-balancer/) to resolve this issue. -Turning off [client IP preservation](https://docs.aws.amazon.com/elasticloadbalancing/latest/network/load-balancer-target-groups.html#client-ip-preservation) -may resolve this issue when the core GitLab application and GitLab Pages run on the same host or -container. +Turning off [client IP preservation](https://docs.aws.amazon.com/elasticloadbalancing/latest/network/load-balancer-target-groups.html#client-ip-preservation) +may resolve this issue when the core GitLab application and GitLab Pages run on the same host or +container. ### 500 error with `securecookie: failed to generate random iv` and `Failed to save the session` diff --git a/doc/administration/pages/source.md b/doc/administration/pages/source.md index c4b1756d8a1..6c148387d7d 100644 --- a/doc/administration/pages/source.md +++ b/doc/administration/pages/source.md @@ -1,6 +1,6 @@ --- -stage: Release -group: Release +stage: Create +group: Editor info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/administration/raketasks/doctor.md b/doc/administration/raketasks/doctor.md deleted file mode 100644 index 457077462a6..00000000000 --- a/doc/administration/raketasks/doctor.md +++ /dev/null @@ -1,11 +0,0 @@ ---- -redirect_to: 'check.md#verify-database-values-can-be-decrypted-using-the-current-secrets' -remove_date: '2022-03-04' ---- - -This document was moved to [another location](check.md#verify-database-values-can-be-decrypted-using-the-current-secrets). - -<!-- This redirect file can be deleted after 2022-03-04. --> -<!-- Redirects that point to other docs in the same project expire in three months. --> -<!-- Redirects that point to docs in a different project or site (for example, link is not relative and starts with `https:`) expire in one year. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html --> diff --git a/doc/administration/raketasks/maintenance.md b/doc/administration/raketasks/maintenance.md index d66f3b1ed35..82ebdcaab42 100644 --- a/doc/administration/raketasks/maintenance.md +++ b/doc/administration/raketasks/maintenance.md @@ -183,9 +183,12 @@ Redis version >= 2.0.0? ... yes Checking GitLab ... Finished ``` -## Rebuild authorized_keys file +## Rebuild `authorized_keys` file -In some case it is necessary to rebuild the `authorized_keys` file. To do this, run: +In some cases it is necessary to rebuild the `authorized_keys` file, +for example, if after an upgrade you receive `Permission denied (publickey)` when pushing [via SSH](../../ssh/index.md) +and find `404 Key Not Found` errors in [the `gitlab-shell.log` file](../logs.md#gitlab-shelllog). +To rebuild `authorized_keys`, run: **Omnibus Installation** diff --git a/doc/administration/reference_architectures/10k_users.md b/doc/administration/reference_architectures/10k_users.md index fcce44f62b2..2ca79bbeae4 100644 --- a/doc/administration/reference_architectures/10k_users.md +++ b/doc/administration/reference_architectures/10k_users.md @@ -14,7 +14,7 @@ full list of reference architectures, see > - **High Availability:** Yes ([Praefect](#configure-praefect-postgresql) needs a third-party PostgreSQL solution for HA) > - **Estimated Costs:** [See cost table](index.md#cost-to-run) > - **Cloud Native Hybrid Alternative:** [Yes](#cloud-native-hybrid-reference-architecture-with-helm-charts-alternative) -> - **Performance tested daily with the [GitLab Performance Tool](https://gitlab.com/gitlab-org/quality/performance)**: +> - **Validation and test results:** The Quality Engineering team does [regular smoke and performance tests](index.md#validation-and-test-results) to ensure the reference architectures remain compliant > - **Test requests per second (RPS) rates:** API: 200 RPS, Web: 20 RPS, Git (Pull): 20 RPS, Git (Push): 4 RPS > - **[Latest Results](https://gitlab.com/gitlab-org/quality/performance/-/wikis/Benchmarks/Latest/10k)** @@ -42,7 +42,7 @@ full list of reference architectures, see 2. Can be optionally run on reputable third-party external PaaS Redis solutions. Google Memorystore and AWS Elasticache are known to work. 3. Can be optionally run on reputable third-party load balancing services (LB PaaS). AWS ELB is known to work. 4. Should be run on reputable third-party object storage (storage PaaS) for cloud implementations. Google Cloud Storage and AWS S3 are known to work. -5. Gitaly Cluster provides the benefits of fault tolerance, but comes with additional complexity of setup and management. Please [review the existing technical limitations and considerations prior to deploying Gitaly Cluster](../gitaly/index.md#guidance-regarding-gitaly-cluster). If Gitaly Sharded is desired, the same specs listed above for `Gitaly` should be used. +5. Gitaly Cluster provides the benefits of fault tolerance, but comes with additional complexity of setup and management. Review the existing [technical limitations and considerations before deploying Gitaly Cluster](../gitaly/index.md#before-deploying-gitaly-cluster). If you want sharded Gitaly, use the same specs listed above for `Gitaly`. <!-- markdownlint-enable MD029 --> NOTE: @@ -1157,8 +1157,8 @@ are supported and can be added if needed. In this configuration, every Git repository is stored on every Gitaly node in the cluster, with one being designated the primary, and failover occurs automatically if the primary node goes down. NOTE: -Gitaly Cluster provides the benefits of fault tolerance, but comes with additional complexity of setup and management. Please [review the existing technical limitations and considerations prior to deploying Gitaly Cluster](../gitaly/index.md#guidance-regarding-gitaly-cluster). -For implementations with Gitaly Sharded, the same Gitaly specs should be used. Follow the [separate Gitaly documentation](../gitaly/configure_gitaly.md) instead of this section. +Gitaly Cluster provides the benefits of fault tolerance, but comes with additional complexity of setup and management. Review the existing [technical limitations and considerations before deploying Gitaly Cluster](../gitaly/index.md#before-deploying-gitaly-cluster). +For implementations with sharded Gitaly, use the same Gitaly specs. Follow the [separate Gitaly documentation](../gitaly/configure_gitaly.md) instead of this section. The recommended cluster setup includes the following components: @@ -1455,10 +1455,12 @@ To configure the Praefect nodes, on each one: ### Configure Gitaly The [Gitaly](../gitaly/index.md) server nodes that make up the cluster have -requirements that are dependent on data, specifically the number of projects -and those projects' sizes. It's recommended that a Gitaly Cluster stores -no more than 5 TB of data on each node. Depending on your -repository storage requirements, you may require additional Gitaly Clusters. +requirements that are dependent on data and load. + +NOTE: +The Reference Architecture specs have been designed with good headroom in mind +but for Gitaly, increased specs or additional +Gitaly Cluster arrays may be required for notably large data sets or load. Due to Gitaly having notable input and output requirements, we strongly recommend that all Gitaly nodes use solid-state drives (SSDs). These SSDs @@ -1531,6 +1533,26 @@ On each node: # Recommended to be enabled for improved performance but can notably increase disk I/O # Refer to https://docs.gitlab.com/ee/administration/gitaly/configure_gitaly.html#pack-objects-cache for more info gitaly['pack_objects_cache_enabled'] = true + + # Configure the Consul agent + consul['enable'] = true + ## Enable service discovery for Prometheus + consul['monitoring_service_discovery'] = true + + # START user configuration + # Please set the real values as explained in Required Information section + # + ## The IPs of the Consul server nodes + ## You can also use FQDNs and intermix them with IPs + consul['configuration'] = { + retry_join: %w(10.6.0.11 10.6.0.12 10.6.0.13), + } + + # Set the network addresses that the exporters will listen on for monitoring + node_exporter['listen_address'] = '0.0.0.0:9100' + gitaly['prometheus_listen_addr'] = '0.0.0.0:9236' + # + # END user configuration ``` 1. Append the following to `/etc/gitlab/gitlab.rb` for each respective server: @@ -1768,6 +1790,7 @@ To configure the Sidekiq nodes, on each one: # Object Storage ## This is an example for configuring Object Storage on GCP ## Replace this config with your chosen Object Storage provider as desired + gitlab_rails['object_store']['enabled'] = true gitlab_rails['object_store']['connection'] = { 'provider' => 'Google', 'google_project' => '<gcp-project-name>', @@ -1914,6 +1937,7 @@ On each node perform the following: # This is an example for configuring Object Storage on GCP # Replace this config with your chosen Object Storage provider as desired + gitlab_rails['object_store']['enabled'] = true gitlab_rails['object_store']['connection'] = { 'provider' => 'Google', 'google_project' => '<gcp-project-name>', @@ -2093,6 +2117,30 @@ To configure the Monitoring node: retry_join: %w(10.6.0.11 10.6.0.12 10.6.0.13) } + # Configure Prometheus to scrape services not covered by discovery + prometheus['scrape_configs'] = [ + { + 'job_name': 'pgbouncer', + 'static_configs' => [ + 'targets' => [ + "10.6.0.31:9188", + "10.6.0.32:9188", + "10.6.0.33:9188", + ], + ], + }, + { + 'job_name': 'praefect', + 'static_configs' => [ + 'targets' => [ + "10.6.0.131:9652", + "10.6.0.132:9652", + "10.6.0.133:9652", + ], + ], + }, + ] + # Nginx - For Grafana access nginx['enable'] = true ``` @@ -2119,7 +2167,7 @@ GitLab has been tested on a number of object storage providers: - [Amazon S3](https://aws.amazon.com/s3/) - [Google Cloud Storage](https://cloud.google.com/storage) -- [Digital Ocean Spaces](https://www.digitalocean.com/products/spaces/) +- [Digital Ocean Spaces](http://www.digitalocean.com/products/spaces) - [Oracle Cloud Infrastructure](https://docs.cloud.oracle.com/en-us/iaas/Content/Object/Tasks/s3compatibleapi.htm) - [OpenStack Swift](https://docs.openstack.org/swift/latest/s3_compat.html) - [Azure Blob storage](https://docs.microsoft.com/en-us/azure/storage/blobs/storage-blobs-introduction) @@ -2136,7 +2184,8 @@ There are two ways of specifying object storage configuration in GitLab: Starting with GitLab 13.2, consolidated object storage configuration is available. It simplifies your GitLab configuration since the connection details are shared across object types. Refer to [Consolidated object storage configuration](../object_storage.md#consolidated-object-storage-configuration) guide for instructions on how to set it up. GitLab Runner returns job logs in chunks which Omnibus GitLab caches temporarily on disk in `/var/opt/gitlab/gitlab-ci/builds` by default, even when using consolidated object storage. With default configuration, this directory needs to be shared via NFS on any GitLab Rails and Sidekiq nodes. -In GitLab 13.6 and later, it's recommended to switch to [Incremental logging](../job_logs.md#incremental-logging-architecture), which uses Redis instead of disk space for temporary caching of job logs. + +In GitLab 13.6 and later, it's also recommended to switch to [Incremental logging](../job_logs.md#incremental-logging-architecture), which uses Redis instead of disk space for temporary caching of job logs. This is required when no NFS node has been deployed. For configuring object storage in GitLab 13.1 and earlier, or for storage types not supported by consolidated configuration form, refer to the following guides based @@ -2236,11 +2285,11 @@ use Google Cloud's Kubernetes Engine (GKE) and associated machine types, but the and CPU requirements should translate to most other providers. We hope to update this in the future with further specific cloud provider details. -| Service | Nodes | Configuration | GCP | Allocatable CPUs and Memory | -|-------------------------------------------------------|-------|-------------------------|------------------|-----------------------------| -| Webservice | 4 | 32 vCPU, 28.8 GB memory | `n1-highcpu-32` | 127.5 vCPU, 118 GB memory | -| Sidekiq | 4 | 4 vCPU, 15 GB memory | `n1-standard-4` | 15.5 vCPU, 50 GB memory | -| Supporting services such as NGINX, Prometheus | 2 | 4 vCPU, 15 GB memory | `n1-standard-4` | 7.75 vCPU, 25 GB memory | +| Service | Nodes | Configuration | GCP | AWS | Min Allocatable CPUs and Memory | +|-----------------------------------------------|-------|-------------------------|-----------------|--------------|---------------------------------| +| Webservice | 4 | 32 vCPU, 28.8 GB memory | `n1-highcpu-32` | `c5.9xlarge` | 127.5 vCPU, 118 GB memory | +| Sidekiq | 4 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | 15.5 vCPU, 50 GB memory | +| Supporting services such as NGINX, Prometheus | 2 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | 7.75 vCPU, 25 GB memory | - For this setup, we **recommend** and regularly [test](index.md#validation-and-test-results) [Google Kubernetes Engine (GKE)](https://cloud.google.com/kubernetes-engine) and [Amazon Elastic Kubernetes Service (EKS)](https://aws.amazon.com/eks/). Other Kubernetes services may also work, but your mileage may vary. @@ -2250,18 +2299,18 @@ future with further specific cloud provider details. Next are the backend components that run on static compute VMs via Omnibus (or External PaaS services where applicable): -| Service | Nodes | Configuration | GCP | -|-----------------------------------------------------|-------|-------------------------|------------------| -| Consul<sup>1</sup> | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | -| PostgreSQL<sup>1</sup> | 3 | 8 vCPU, 30 GB memory | `n1-standard-8` | -| PgBouncer<sup>1</sup> | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | -| Internal load balancing node<sup>3</sup> | 1 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | -| Redis/Sentinel - Cache<sup>2</sup> | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | -| Redis/Sentinel - Persistent<sup>2</sup> | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | -| Gitaly<sup>5</sup> | 3 | 16 vCPU, 60 GB memory | `n1-standard-16` | -| Praefect<sup>5</sup> | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | -| Praefect PostgreSQL<sup>1</sup> | 1+ | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | -| Object storage<sup>4</sup> | n/a | n/a | n/a | +| Service | Nodes | Configuration | GCP | AWS | +|------------------------------------------|-------|-----------------------|------------------|--------------| +| Consul<sup>1</sup> | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | +| PostgreSQL<sup>1</sup> | 3 | 8 vCPU, 30 GB memory | `n1-standard-8` | `m5.2xlarge` | +| PgBouncer<sup>1</sup> | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | +| Internal load balancing node<sup>3</sup> | 1 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | +| Redis/Sentinel - Cache<sup>2</sup> | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | +| Redis/Sentinel - Persistent<sup>2</sup> | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | +| Gitaly<sup>5</sup> | 3 | 16 vCPU, 60 GB memory | `n1-standard-16` | `m5.4xlarge` | +| Praefect<sup>5</sup> | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | +| Praefect PostgreSQL<sup>1</sup> | 1+ | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | +| Object storage<sup>4</sup> | n/a | n/a | n/a | n/a | <!-- Disable ordered list rule https://github.com/DavidAnson/markdownlint/blob/main/doc/Rules.md#md029---ordered-list-item-prefix --> <!-- markdownlint-disable MD029 --> @@ -2269,7 +2318,7 @@ services where applicable): 2. Can be optionally run on reputable third-party external PaaS Redis solutions. Google Memorystore and AWS Elasticache are known to work. 3. Can be optionally run on reputable third-party load balancing services (LB PaaS). AWS ELB is known to work. 4. Should be run on reputable third-party object storage (storage PaaS) for cloud implementations. Google Cloud Storage and AWS S3 are known to work. -5. Gitaly Cluster provides the benefits of fault tolerance, but comes with additional complexity of setup and management. Please [review the existing technical limitations and considerations prior to deploying Gitaly Cluster](../gitaly/index.md#guidance-regarding-gitaly-cluster). If Gitaly Sharded is desired, the same specs listed above for `Gitaly` should be used. +5. Gitaly Cluster provides the benefits of fault tolerance, but comes with additional complexity of setup and management. Review the existing [technical limitations and considerations before deploying Gitaly Cluster](../gitaly/index.md#before-deploying-gitaly-cluster). If you want sharded Gitaly, use the same specs listed above for `Gitaly`. <!-- markdownlint-enable MD029 --> NOTE: diff --git a/doc/administration/reference_architectures/1k_users.md b/doc/administration/reference_architectures/1k_users.md index 0d0e7681ffd..7213a1eb92b 100644 --- a/doc/administration/reference_architectures/1k_users.md +++ b/doc/administration/reference_architectures/1k_users.md @@ -21,7 +21,7 @@ many organizations. > - **Estimated Costs:** [See cost table](index.md#cost-to-run) > - **Cloud Native Hybrid:** No. For a cloud native hybrid environment, you > can follow a [modified hybrid reference architecture](#cloud-native-hybrid-reference-architecture-with-helm-charts). -> - **Performance tested daily with the [GitLab Performance Tool (GPT)](https://gitlab.com/gitlab-org/quality/performance)**: +> - **Validation and test results:** The Quality Engineering team does [regular smoke and performance tests](index.md#validation-and-test-results) to ensure the reference architectures remain compliant > - **Test requests per second (RPS) rates:** API: 20 RPS, Web: 2 RPS, Git (Pull): 2 RPS, Git (Push): 1 RPS > - **[Latest Results](https://gitlab.com/gitlab-org/quality/performance/-/wikis/Benchmarks/Latest/1k)** diff --git a/doc/administration/reference_architectures/25k_users.md b/doc/administration/reference_architectures/25k_users.md index c08fe985b40..2a1d344508e 100644 --- a/doc/administration/reference_architectures/25k_users.md +++ b/doc/administration/reference_architectures/25k_users.md @@ -14,7 +14,7 @@ full list of reference architectures, see > - **High Availability:** Yes ([Praefect](#configure-praefect-postgresql) needs a third-party PostgreSQL solution for HA) > - **Estimated Costs:** [See cost table](index.md#cost-to-run) > - **Cloud Native Hybrid Alternative:** [Yes](#cloud-native-hybrid-reference-architecture-with-helm-charts-alternative) -> - **Performance tested weekly with the [GitLab Performance Tool (GPT)](https://gitlab.com/gitlab-org/quality/performance)**: +> - **Validation and test results:** The Quality Engineering team does [regular smoke and performance tests](index.md#validation-and-test-results) to ensure the reference architectures remain compliant > - **Test requests per second (RPS) rates:** API: 500 RPS, Web: 50 RPS, Git (Pull): 50 RPS, Git (Push): 10 RPS > - **[Latest Results](https://gitlab.com/gitlab-org/quality/performance/-/wikis/Benchmarks/Latest/25k)** @@ -24,7 +24,7 @@ full list of reference architectures, see | Consul<sup>1</sup> | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | | PostgreSQL<sup>1</sup> | 3 | 16 vCPU, 60 GB memory | `n1-standard-16` | `m5.4xlarge` | `D16s v3` | | PgBouncer<sup>1</sup> | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | `F2s v2` | -| Internal load balancing node<sup>3</sup> | 1 | 4 vCPU, 3.6GB memory | `n1-highcpu-4` | `c5.large` | `F2s v2` | +| Internal load balancing node<sup>3</sup> | 1 | 4 vCPU, 3.6GB memory | `n1-highcpu-4` | `c5.xlarge` | `F4s v2` | | Redis/Sentinel - Cache<sup>2</sup> | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | `D4s v3` | | Redis/Sentinel - Persistent<sup>2</sup> | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | `D4s v3` | | Gitaly<sup>5</sup> | 3 | 32 vCPU, 120 GB memory | `n1-standard-32` | `m5.8xlarge` | `D32s v3` | @@ -42,7 +42,7 @@ full list of reference architectures, see 2. Can be optionally run on reputable third-party external PaaS Redis solutions. Google Memorystore and AWS Elasticache are known to work. 3. Can be optionally run on reputable third-party load balancing services (LB PaaS). AWS ELB is known to work. 4. Should be run on reputable third-party object storage (storage PaaS) for cloud implementations. Google Cloud Storage and AWS S3 are known to work. -5. Gitaly Cluster provides the benefits of fault tolerance, but comes with additional complexity of setup and management. Please [review the existing technical limitations and considerations prior to deploying Gitaly Cluster](../gitaly/index.md#guidance-regarding-gitaly-cluster). If Gitaly Sharded is desired, the same specs listed above for `Gitaly` should be used. +5. Gitaly Cluster provides the benefits of fault tolerance, but comes with additional complexity of setup and management. Review the existing [technical limitations and considerations before deploying Gitaly Cluster](../gitaly/index.md#before-deploying-gitaly-cluster). If you want sharded Gitaly, use the same specs listed above for `Gitaly`. <!-- markdownlint-enable MD029 --> NOTE: @@ -86,7 +86,7 @@ card "Database" as database { card "redis" as redis { collections "**Redis Persistent** x3" as redis_persistent #FF6347 collections "**Redis Cache** x3" as redis_cache #FF6347 - + redis_cache -[hidden]-> redis_persistent } @@ -1163,8 +1163,8 @@ fault tolerant solution for storing Git repositories. In this configuration, every Git repository is stored on every Gitaly node in the cluster, with one being designated the primary, and failover occurs automatically if the primary node goes down. NOTE: -Gitaly Cluster provides the benefits of fault tolerance, but comes with additional complexity of setup and management. Please [review the existing technical limitations and considerations prior to deploying Gitaly Cluster](../gitaly/index.md#guidance-regarding-gitaly-cluster). -For implementations with Gitaly Sharded, the same Gitaly specs should be used. Follow the [separate Gitaly documentation](../gitaly/configure_gitaly.md) instead of this section. +Gitaly Cluster provides the benefits of fault tolerance, but comes with additional complexity of setup and management. Review the existing [technical limitations and considerations before deploying Gitaly Cluster](../gitaly/index.md#before-deploying-gitaly-cluster). +For implementations with sharded Gitaly, use the same Gitaly specs. Follow the [separate Gitaly documentation](../gitaly/configure_gitaly.md) instead of this section. The recommended cluster setup includes the following components: @@ -1459,10 +1459,12 @@ the file of the same name on this server. If this is the first Omnibus node you ### Configure Gitaly The [Gitaly](../gitaly/index.md) server nodes that make up the cluster have -requirements that are dependent on data, specifically the number of projects -and those projects' sizes. It's recommended that a Gitaly Cluster stores -no more than 5 TB of data on each node. Depending on your -repository storage requirements, you may require additional Gitaly Clusters. +requirements that are dependent on data and load. + +NOTE: +The Reference Architecture specs have been designed with good headroom in mind +but for Gitaly, increased specs or additional +Gitaly Cluster arrays may be required for notably large data sets or load. Due to Gitaly having notable input and output requirements, we strongly recommend that all Gitaly nodes use solid-state drives (SSDs). These SSDs @@ -1535,6 +1537,26 @@ On each node: # Recommended to be enabled for improved performance but can notably increase disk I/O # Refer to https://docs.gitlab.com/ee/administration/gitaly/configure_gitaly.html#pack-objects-cache for more info gitaly['pack_objects_cache_enabled'] = true + + # Configure the Consul agent + consul['enable'] = true + ## Enable service discovery for Prometheus + consul['monitoring_service_discovery'] = true + + # START user configuration + # Please set the real values as explained in Required Information section + # + ## The IPs of the Consul server nodes + ## You can also use FQDNs and intermix them with IPs + consul['configuration'] = { + retry_join: %w(10.6.0.11 10.6.0.12 10.6.0.13), + } + + # Set the network addresses that the exporters will listen on for monitoring + node_exporter['listen_address'] = '0.0.0.0:9100' + gitaly['prometheus_listen_addr'] = '0.0.0.0:9236' + # + # END user configuration ``` 1. Append the following to `/etc/gitlab/gitlab.rb` for each respective server: @@ -1772,6 +1794,7 @@ To configure the Sidekiq nodes, on each one: # Object Storage # This is an example for configuring Object Storage on GCP # Replace this config with your chosen Object Storage provider as desired + gitlab_rails['object_store']['enabled'] = true gitlab_rails['object_store']['connection'] = { 'provider' => 'Google', 'google_project' => '<gcp-project-name>', @@ -1920,6 +1943,7 @@ On each node perform the following: # This is an example for configuring Object Storage on GCP # Replace this config with your chosen Object Storage provider as desired + gitlab_rails['object_store']['enabled'] = true gitlab_rails['object_store']['connection'] = { 'provider' => 'Google', 'google_project' => '<gcp-project-name>', @@ -2098,6 +2122,30 @@ To configure the Monitoring node: retry_join: %w(10.6.0.11 10.6.0.12 10.6.0.13) } + # Configure Prometheus to scrape services not covered by discovery + prometheus['scrape_configs'] = [ + { + 'job_name': 'pgbouncer', + 'static_configs' => [ + 'targets' => [ + "10.6.0.31:9188", + "10.6.0.32:9188", + "10.6.0.33:9188", + ], + ], + }, + { + 'job_name': 'praefect', + 'static_configs' => [ + 'targets' => [ + "10.6.0.131:9652", + "10.6.0.132:9652", + "10.6.0.133:9652", + ], + ], + }, + ] + # Nginx - For Grafana access nginx['enable'] = true ``` @@ -2123,7 +2171,7 @@ GitLab has been tested on a number of object storage providers: - [Amazon S3](https://aws.amazon.com/s3/) - [Google Cloud Storage](https://cloud.google.com/storage) -- [Digital Ocean Spaces](https://www.digitalocean.com/products/spaces/) +- [Digital Ocean Spaces](http://www.digitalocean.com/products/spaces) - [Oracle Cloud Infrastructure](https://docs.cloud.oracle.com/en-us/iaas/Content/Object/Tasks/s3compatibleapi.htm) - [OpenStack Swift](https://docs.openstack.org/swift/latest/s3_compat.html) - [Azure Blob storage](https://docs.microsoft.com/en-us/azure/storage/blobs/storage-blobs-introduction) @@ -2140,7 +2188,8 @@ There are two ways of specifying object storage configuration in GitLab: Starting with GitLab 13.2, consolidated object storage configuration is available. It simplifies your GitLab configuration since the connection details are shared across object types. Refer to [Consolidated object storage configuration](../object_storage.md#consolidated-object-storage-configuration) guide for instructions on how to set it up. GitLab Runner returns job logs in chunks which Omnibus GitLab caches temporarily on disk in `/var/opt/gitlab/gitlab-ci/builds` by default, even when using consolidated object storage. With default configuration, this directory needs to be shared via NFS on any GitLab Rails and Sidekiq nodes. -In GitLab 13.6 and later, it's recommended to switch to [Incremental logging](../job_logs.md#incremental-logging-architecture), which uses Redis instead of disk space for temporary caching of job logs. + +In GitLab 13.6 and later, it's also recommended to switch to [Incremental logging](../job_logs.md#incremental-logging-architecture), which uses Redis instead of disk space for temporary caching of job logs. This is required when no NFS node has been deployed. For configuring object storage in GitLab 13.1 and earlier, or for storage types not supported by consolidated configuration form, refer to the following guides based @@ -2234,11 +2283,11 @@ use Google Cloud's Kubernetes Engine (GKE) and associated machine types, but the and CPU requirements should translate to most other providers. We hope to update this in the future with further specific cloud provider details. -| Service | Nodes | Configuration | GCP | Allocatable CPUs and Memory | -|-------------------------------------------------------|-------|-------------------------|------------------|-----------------------------| -| Webservice | 7 | 32 vCPU, 28.8 GB memory | `n1-highcpu-32` | 223 vCPU, 206.5 GB memory | -| Sidekiq | 4 | 4 vCPU, 15 GB memory | `n1-standard-4` | 15.5 vCPU, 50 GB memory | -| Supporting services such as NGINX, Prometheus | 2 | 4 vCPU, 15 GB memory | `n1-standard-4` | 7.75 vCPU, 25 GB memory | +| Service | Nodes | Configuration | GCP | AWS | Min Allocatable CPUs and Memory | +|-----------------------------------------------|-------|-------------------------|-----------------|-----------------|---------------------------------| +| Webservice | 7 | 32 vCPU, 28.8 GB memory | `n1-highcpu-32` | `c5.9xlarge` | 223 vCPU, 206.5 GB memory | +| Sidekiq | 4 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | 15.5 vCPU, 50 GB memory | +| Supporting services such as NGINX, Prometheus | 2 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | 7.75 vCPU, 25 GB memory | - For this setup, we **recommend** and regularly [test](index.md#validation-and-test-results) [Google Kubernetes Engine (GKE)](https://cloud.google.com/kubernetes-engine) and [Amazon Elastic Kubernetes Service (EKS)](https://aws.amazon.com/eks/). Other Kubernetes services may also work, but your mileage may vary. @@ -2248,18 +2297,18 @@ future with further specific cloud provider details. Next are the backend components that run on static compute VMs via Omnibus (or External PaaS services where applicable): -| Service | Nodes | Configuration | GCP | -|-----------------------------------------------------|-------|-------------------------|------------------| -| Consul<sup>1</sup> | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | -| PostgreSQL<sup>1</sup> | 3 | 16 vCPU, 60 GB memory | `n1-standard-16` | -| PgBouncer<sup>1</sup> | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | -| Internal load balancing node<sup>3</sup> | 1 | 4 vCPU, 3.6GB memory | `n1-highcpu-4` | -| Redis/Sentinel - Cache<sup>2</sup> | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | -| Redis/Sentinel - Persistent<sup>2</sup> | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | -| Gitaly<sup>5</sup> | 3 | 32 vCPU, 120 GB memory | `n1-standard-32` | -| Praefect<sup>5</sup> | 3 | 4 vCPU, 3.6 GB memory | `n1-highcpu-4` | -| Praefect PostgreSQL<sup>1</sup> | 1+ | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | -| Object storage<sup>4</sup> | n/a | n/a | n/a | +| Service | Nodes | Configuration | GCP | AWS | +|------------------------------------------|-------|------------------------|------------------|--------------| +| Consul<sup>1</sup> | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | +| PostgreSQL<sup>1</sup> | 3 | 16 vCPU, 60 GB memory | `n1-standard-16` | `m5.4xlarge` | +| PgBouncer<sup>1</sup> | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | +| Internal load balancing node<sup>3</sup> | 1 | 4 vCPU, 3.6GB memory | `n1-highcpu-4` | `c5.xlarge` | +| Redis/Sentinel - Cache<sup>2</sup> | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | +| Redis/Sentinel - Persistent<sup>2</sup> | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | +| Gitaly<sup>5</sup> | 3 | 32 vCPU, 120 GB memory | `n1-standard-32` | `m5.8xlarge` | +| Praefect<sup>5</sup> | 3 | 4 vCPU, 3.6 GB memory | `n1-highcpu-4` | `c5.xlarge` | +| Praefect PostgreSQL<sup>1</sup> | 1+ | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | +| Object storage<sup>4</sup> | n/a | n/a | n/a | n/a | <!-- Disable ordered list rule https://github.com/DavidAnson/markdownlint/blob/main/doc/Rules.md#md029---ordered-list-item-prefix --> <!-- markdownlint-disable MD029 --> @@ -2267,7 +2316,7 @@ services where applicable): 2. Can be optionally run on reputable third-party external PaaS Redis solutions. Google Memorystore and AWS Elasticache are known to work. 3. Can be optionally run on reputable third-party load balancing services (LB PaaS). AWS ELB is known to work. 4. Should be run on reputable third-party object storage (storage PaaS) for cloud implementations. Google Cloud Storage and AWS S3 are known to work. -5. Gitaly Cluster provides the benefits of fault tolerance, but comes with additional complexity of setup and management. Please [review the existing technical limitations and considerations prior to deploying Gitaly Cluster](../gitaly/index.md#guidance-regarding-gitaly-cluster). If Gitaly Sharded is desired, the same specs listed above for `Gitaly` should be used. +5. Gitaly Cluster provides the benefits of fault tolerance, but comes with additional complexity of setup and management. Review the existing [technical limitations and considerations before deploying Gitaly Cluster](../gitaly/index.md#before-deploying-gitaly-cluster). If you want sharded Gitaly, use the same specs listed above for `Gitaly`. <!-- markdownlint-enable MD029 --> NOTE: @@ -2312,7 +2361,7 @@ card "Database" as database { card "redis" as redis { collections "**Redis Persistent** x3" as redis_persistent #FF6347 collections "**Redis Cache** x3" as redis_cache #FF6347 - + redis_cache -[hidden]-> redis_persistent } diff --git a/doc/administration/reference_architectures/2k_users.md b/doc/administration/reference_architectures/2k_users.md index 6f6c02c309a..f417d1e2ab5 100644 --- a/doc/administration/reference_architectures/2k_users.md +++ b/doc/administration/reference_architectures/2k_users.md @@ -13,9 +13,9 @@ For a full list of reference architectures, see > - **Supported users (approximate):** 2,000 > - **High Availability:** No. For a highly-available environment, you can > follow a modified [3K reference architecture](3k_users.md#supported-modifications-for-lower-user-counts-ha). -> - **Estimated Costs:** [See cost table](index.md#cost-to-run) +> - **Estimated Costs:** [See cost table](index.md#cost-to-run) > - **Cloud Native Hybrid:** [Yes](#cloud-native-hybrid-reference-architecture-with-helm-charts-alternative) -> - **Performance tested daily with the [GitLab Performance Tool (GPT)](https://gitlab.com/gitlab-org/quality/performance)**: +> - **Validation and test results:** The Quality Engineering team does [regular smoke and performance tests](index.md#validation-and-test-results) to ensure the reference architectures remain compliant > - **Test requests per second (RPS) rates:** API: 40 RPS, Web: 4 RPS, Git (Pull): 4 RPS, Git (Push): 1 RPS > - **[Latest Results](https://gitlab.com/gitlab-org/quality/performance/-/wikis/Benchmarks/Latest/2k)** @@ -397,11 +397,13 @@ are supported and can be added if needed. ## Configure Gitaly -[Gitaly](../gitaly/index.md) server node requirements are dependent on data, -specifically the number of projects and those projects' sizes. It's recommended -that a Gitaly server node stores no more than 5TB of data. Although this -reference architecture includes a single Gitaly server node, you may require -additional nodes depending on your repository storage requirements. +[Gitaly](../gitaly/index.md) server node requirements are dependent on data size, +specifically the number of projects and those projects' sizes. + +NOTE: +The Reference Architecture specs have been designed with good headroom in mind +but for Gitaly, increased specs or switching to Gitaly Cluster +may be required for notably large data sets or load. Due to Gitaly having notable input and output requirements, we strongly recommend that all Gitaly nodes use solid-state drives (SSDs). These SSDs @@ -665,6 +667,7 @@ On each node perform the following: # Object Storage # This is an example for configuring Object Storage on GCP # Replace this config with your chosen Object Storage provider as desired + gitlab_rails['object_store']['enabled'] = true gitlab_rails['object_store']['connection'] = { 'provider' => 'Google', 'google_project' => '<gcp-project-name>', @@ -886,7 +889,7 @@ GitLab has been tested on a number of object storage providers: - [Amazon S3](https://aws.amazon.com/s3/) - [Google Cloud Storage](https://cloud.google.com/storage) -- [Digital Ocean Spaces](https://www.digitalocean.com/products/spaces/) +- [Digital Ocean Spaces](http://www.digitalocean.com/products/spaces) - [Oracle Cloud Infrastructure](https://docs.cloud.oracle.com/en-us/iaas/Content/Object/Tasks/s3compatibleapi.htm) - [OpenStack Swift](https://docs.openstack.org/swift/latest/s3_compat.html) - [Azure Blob storage](https://docs.microsoft.com/en-us/azure/storage/blobs/storage-blobs-introduction) @@ -903,7 +906,8 @@ There are two ways of specifying object storage configuration in GitLab: Starting with GitLab 13.2, consolidated object storage configuration is available. It simplifies your GitLab configuration since the connection details are shared across object types. Refer to [Consolidated object storage configuration](../object_storage.md#consolidated-object-storage-configuration) guide for instructions on how to set it up. GitLab Runner returns job logs in chunks which Omnibus GitLab caches temporarily on disk in `/var/opt/gitlab/gitlab-ci/builds` by default, even when using consolidated object storage. With default configuration, this directory needs to be shared via NFS on any GitLab Rails and Sidekiq nodes. -In GitLab 13.6 and later, it's recommended to switch to [Incremental logging](../job_logs.md#incremental-logging-architecture), which uses Redis instead of disk space for temporary caching of job logs. + +In GitLab 13.6 and later, it's also recommended to switch to [Incremental logging](../job_logs.md#incremental-logging-architecture), which uses Redis instead of disk space for temporary caching of job logs. This is required when no NFS node has been deployed. For configuring object storage in GitLab 13.1 and earlier, or for storage types not supported by consolidated configuration form, refer to the following guides based @@ -1000,11 +1004,11 @@ use Google Cloud's Kubernetes Engine (GKE) and associated machine types, but the and CPU requirements should translate to most other providers. We hope to update this in the future with further specific cloud provider details. -| Service | Nodes | Configuration | GCP | Allocatable CPUs and Memory | -|-------------------------------------------------------|-------|------------------------|-----------------|-----------------------------| -| Webservice | 3 | 8 vCPU, 7.2 GB memory | `n1-highcpu-8` | 23.7 vCPU, 16.9 GB memory | -| Sidekiq | 2 | 2 vCPU, 7.5 GB memory | `n1-standard-2` | 3.9 vCPU, 11.8 GB memory | -| Supporting services such as NGINX, Prometheus | 2 | 1 vCPU, 3.75 GB memory | `n1-standard-1` | 1.9 vCPU, 5.5 GB memory | +| Service | Nodes | Configuration | GCP | AWS | Min Allocatable CPUs and Memory | +|-----------------------------------------------|-------|------------------------|-----------------|--------------|---------------------------------| +| Webservice | 3 | 8 vCPU, 7.2 GB memory | `n1-highcpu-8` | `c5.2xlarge` | 23.7 vCPU, 16.9 GB memory | +| Sidekiq | 2 | 2 vCPU, 7.5 GB memory | `n1-standard-2` | `m5.large` | 3.9 vCPU, 11.8 GB memory | +| Supporting services such as NGINX, Prometheus | 2 | 1 vCPU, 3.75 GB memory | `n1-standard-1` | `m5.large` | 1.9 vCPU, 5.5 GB memory | - For this setup, we **recommend** and regularly [test](index.md#validation-and-test-results) [Google Kubernetes Engine (GKE)](https://cloud.google.com/kubernetes-engine) and [Amazon Elastic Kubernetes Service (EKS)](https://aws.amazon.com/eks/). Other Kubernetes services may also work, but your mileage may vary. @@ -1014,12 +1018,12 @@ future with further specific cloud provider details. Next are the backend components that run on static compute VMs via Omnibus (or External PaaS services where applicable): -| Service | Nodes | Configuration | GCP | -|--------------------------------------------|-------|-------------------------|------------------| -| PostgreSQL<sup>1</sup> | 1 | 2 vCPU, 7.5 GB memory | `n1-standard-2` | -| Redis<sup>2</sup> | 1 | 1 vCPU, 3.75 GB memory | `n1-standard-1` | -| Gitaly | 1 | 4 vCPU, 15 GB memory | `n1-standard-4` | -| Object storage<sup>3</sup> | n/a | n/a | n/a | +| Service | Nodes | Configuration | GCP | AWS | +|----------------------------|-------|------------------------|-----------------|-------------| +| PostgreSQL<sup>1</sup> | 1 | 2 vCPU, 7.5 GB memory | `n1-standard-2` | `m5.large` | +| Redis<sup>2</sup> | 1 | 1 vCPU, 3.75 GB memory | `n1-standard-1` | `m5.large` | +| Gitaly | 1 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | +| Object storage<sup>3</sup> | n/a | n/a | n/a | n/a | <!-- Disable ordered list rule https://github.com/DavidAnson/markdownlint/blob/main/doc/Rules.md#md029---ordered-list-item-prefix --> <!-- markdownlint-disable MD029 --> diff --git a/doc/administration/reference_architectures/3k_users.md b/doc/administration/reference_architectures/3k_users.md index 76f81e65580..9d8be3e90b6 100644 --- a/doc/administration/reference_architectures/3k_users.md +++ b/doc/administration/reference_architectures/3k_users.md @@ -24,7 +24,7 @@ For a full list of reference architectures, see > - **High Availability:** Yes, although [Praefect](#configure-praefect-postgresql) needs a third-party PostgreSQL solution > - **Estimated Costs:** [See cost table](index.md#cost-to-run) > - **Cloud Native Hybrid Alternative:** [Yes](#cloud-native-hybrid-reference-architecture-with-helm-charts-alternative) -> - **Performance tested weekly with the [GitLab Performance Tool (GPT)](https://gitlab.com/gitlab-org/quality/performance)**: +> - **Validation and test results:** The Quality Engineering team does [regular smoke and performance tests](index.md#validation-and-test-results) to ensure the reference architectures remain compliant > - **Test requests per second (RPS) rates:** API: 60 RPS, Web: 6 RPS, Git (Pull): 6 RPS, Git (Push): 1 RPS > - **[Latest Results](https://gitlab.com/gitlab-org/quality/performance/-/wikis/Benchmarks/Latest/3k)** @@ -51,7 +51,7 @@ For a full list of reference architectures, see 2. Can be optionally run on reputable third-party external PaaS Redis solutions. Google Memorystore and AWS Elasticache are known to work. 3. Can be optionally run on reputable third-party load balancing services (LB PaaS). AWS ELB is known to work. 4. Should be run on reputable third-party object storage (storage PaaS) for cloud implementations. Google Cloud Storage and AWS S3 are known to work. -5. Gitaly Cluster provides the benefits of fault tolerance, but comes with additional complexity of setup and management. Please [review the existing technical limitations and considerations prior to deploying Gitaly Cluster](../gitaly/index.md#guidance-regarding-gitaly-cluster). If Gitaly Sharded is desired, the same specs listed above for `Gitaly` should be used. +5. Gitaly Cluster provides the benefits of fault tolerance, but comes with additional complexity of setup and management. Review the existing [technical limitations and considerations before deploying Gitaly Cluster](../gitaly/index.md#before-deploying-gitaly-cluster). If you want sharded Gitaly, use the same specs listed above for `Gitaly`. <!-- markdownlint-enable MD029 --> NOTE: @@ -1104,8 +1104,8 @@ The following IPs will be used as an example: In this configuration, every Git repository is stored on every Gitaly node in the cluster, with one being designated the primary, and failover occurs automatically if the primary node goes down. NOTE: -Gitaly Cluster provides the benefits of fault tolerance, but comes with additional complexity of setup and management. Please [review the existing technical limitations and considerations prior to deploying Gitaly Cluster](../gitaly/index.md#guidance-regarding-gitaly-cluster). -For implementations with Gitaly Sharded, the same Gitaly specs should be used. Follow the [separate Gitaly documentation](../gitaly/configure_gitaly.md) instead of this section. +Gitaly Cluster provides the benefits of fault tolerance, but comes with additional complexity of setup and management. Review the existing [technical limitations and considerations before deploying Gitaly Cluster](../gitaly/index.md#before-deploying-gitaly-cluster). +For implementations with sharded Gitaly, use the same Gitaly specs. Follow the [separate Gitaly documentation](../gitaly/configure_gitaly.md) instead of this section. The recommended cluster setup includes the following components: @@ -1399,10 +1399,12 @@ the file of the same name on this server. If this is the first Omnibus node you ### Configure Gitaly The [Gitaly](../gitaly/index.md) server nodes that make up the cluster have -requirements that are dependent on data, specifically the number of projects -and those projects' sizes. It's recommended that a Gitaly Cluster stores -no more than 5 TB of data on each node. Depending on your -repository storage requirements, you may require additional Gitaly Clusters. +requirements that are dependent on data and load. + +NOTE: +The Reference Architecture specs have been designed with good headroom in mind +but for Gitaly, increased specs or additional +Gitaly Cluster arrays may be required for notably large data sets or load. Due to Gitaly having notable input and output requirements, we strongly recommend that all Gitaly nodes use solid-state drives (SSDs). These SSDs @@ -1475,6 +1477,26 @@ On each node: # Recommended to be enabled for improved performance but can notably increase disk I/O # Refer to https://docs.gitlab.com/ee/administration/gitaly/configure_gitaly.html#pack-objects-cache for more info gitaly['pack_objects_cache_enabled'] = true + + # Configure the Consul agent + consul['enable'] = true + ## Enable service discovery for Prometheus + consul['monitoring_service_discovery'] = true + + # START user configuration + # Please set the real values as explained in Required Information section + # + ## The IPs of the Consul server nodes + ## You can also use FQDNs and intermix them with IPs + consul['configuration'] = { + retry_join: %w(10.6.0.11 10.6.0.12 10.6.0.13), + } + + # Set the network addresses that the exporters will listen on for monitoring + node_exporter['listen_address'] = '0.0.0.0:9100' + gitaly['prometheus_listen_addr'] = '0.0.0.0:9236' + # + # END user configuration ``` 1. Append the following to `/etc/gitlab/gitlab.rb` for each respective server: @@ -1696,6 +1718,7 @@ To configure the Sidekiq nodes, one each one: # Object Storage ## This is an example for configuring Object Storage on GCP ## Replace this config with your chosen Object Storage provider as desired + gitlab_rails['object_store']['enabled'] = true gitlab_rails['object_store']['connection'] = { 'provider' => 'Google', 'google_project' => '<gcp-project-name>', @@ -1882,6 +1905,7 @@ On each node perform the following: # Object storage # This is an example for configuring Object Storage on GCP # Replace this config with your chosen Object Storage provider as desired + gitlab_rails['object_store']['enabled'] = true gitlab_rails['object_store']['connection'] = { 'provider' => 'Google', 'google_project' => '<gcp-project-name>', @@ -2018,6 +2042,30 @@ running [Prometheus](../monitoring/prometheus/index.md) and retry_join: %w(10.6.0.11 10.6.0.12 10.6.0.13) } + # Configure Prometheus to scrape services not covered by discovery + prometheus['scrape_configs'] = [ + { + 'job_name': 'pgbouncer', + 'static_configs' => [ + 'targets' => [ + "10.6.0.21:9188", + "10.6.0.22:9188", + "10.6.0.23:9188", + ], + ], + }, + { + 'job_name': 'praefect', + 'static_configs' => [ + 'targets' => [ + "10.6.0.131:9652", + "10.6.0.132:9652", + "10.6.0.133:9652", + ], + ], + }, + ] + # Nginx - For Grafana access nginx['enable'] = true ``` @@ -2058,7 +2106,7 @@ GitLab has been tested on a number of object storage providers: - [Amazon S3](https://aws.amazon.com/s3/) - [Google Cloud Storage](https://cloud.google.com/storage) -- [Digital Ocean Spaces](https://www.digitalocean.com/products/spaces/) +- [Digital Ocean Spaces](http://www.digitalocean.com/products/spaces) - [Oracle Cloud Infrastructure](https://docs.cloud.oracle.com/en-us/iaas/Content/Object/Tasks/s3compatibleapi.htm) - [OpenStack Swift](https://docs.openstack.org/swift/latest/s3_compat.html) - [Azure Blob storage](https://docs.microsoft.com/en-us/azure/storage/blobs/storage-blobs-introduction) @@ -2075,7 +2123,8 @@ There are two ways of specifying object storage configuration in GitLab: Starting with GitLab 13.2, consolidated object storage configuration is available. It simplifies your GitLab configuration since the connection details are shared across object types. Refer to [Consolidated object storage configuration](../object_storage.md#consolidated-object-storage-configuration) guide for instructions on how to set it up. GitLab Runner returns job logs in chunks which Omnibus GitLab caches temporarily on disk in `/var/opt/gitlab/gitlab-ci/builds` by default, even when using consolidated object storage. With default configuration, this directory needs to be shared via NFS on any GitLab Rails and Sidekiq nodes. -In GitLab 13.6 and later, it's recommended to switch to [Incremental logging](../job_logs.md#incremental-logging-architecture), which uses Redis instead of disk space for temporary caching of job logs. + +In GitLab 13.6 and later, it's also recommended to switch to [Incremental logging](../job_logs.md#incremental-logging-architecture), which uses Redis instead of disk space for temporary caching of job logs. This is required when no NFS node has been deployed. For configuring object storage in GitLab 13.1 and earlier, or for storage types not supported by consolidated configuration form, refer to the following guides based @@ -2157,6 +2206,7 @@ but with smaller performance requirements, several modifications can be consider - Reducing the node counts: Some node types do not need consensus and can run with fewer nodes (but more than one for redundancy). This will also lead to reduced performance. - GitLab Rails and Sidekiq: Stateless services don't have a minimum node count. Two are enough for redundancy. - Gitaly and Praefect: A quorum is not strictly necessary. Two Gitaly nodes and two Praefect nodes are enough for redundancy. + - PostgreSQL and PgBouncer: A quorum is not strictly necessary. Two PostgreSQL nodes and two PgBouncer nodes are enough for redundancy. - Running select components in reputable Cloud PaaS solutions: Select components of the GitLab setup can instead be run on Cloud Provider PaaS solutions. By doing this, additional dependent components can also be removed: - PostgreSQL: Can be run on reputable Cloud PaaS solutions such as Google Cloud SQL or Amazon RDS. In this setup, the PgBouncer and Consul nodes are no longer required: - Consul may still be desired if [Prometheus](../monitoring/prometheus/index.md) auto discovery is a requirement, otherwise you would need to [manually add scrape configurations](../monitoring/prometheus/index.md#adding-custom-scrape-configurations) for all nodes. @@ -2193,11 +2243,11 @@ use Google Cloud's Kubernetes Engine (GKE) and associated machine types, but the and CPU requirements should translate to most other providers. We hope to update this in the future with further specific cloud provider details. -| Service | Nodes | Configuration | GCP | Allocatable CPUs and Memory | -|-------------------------------------------------------|-------|-------------------------|------------------|-----------------------------| -| Webservice | 2 | 16 vCPU, 14.4 GB memory | `n1-highcpu-16` | 31.8 vCPU, 24.8 GB memory | -| Sidekiq | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | 11.8 vCPU, 38.9 GB memory | -| Supporting services such as NGINX, Prometheus | 2 | 2 vCPU, 7.5 GB memory | `n1-standard-2` | 3.9 vCPU, 11.8 GB memory | +| Service | Nodes | Configuration | GCP | AWS | Min Allocatable CPUs and Memory | +|-----------------------------------------------|-------|-------------------------|-----------------|--------------|---------------------------------| +| Webservice | 2 | 16 vCPU, 14.4 GB memory | `n1-highcpu-16` | `c5.4xlarge` | 31.8 vCPU, 24.8 GB memory | +| Sidekiq | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | 11.8 vCPU, 38.9 GB memory | +| Supporting services such as NGINX, Prometheus | 2 | 2 vCPU, 7.5 GB memory | `n1-standard-2` | `m5.large` | 3.9 vCPU, 11.8 GB memory | - For this setup, we **recommend** and regularly [test](index.md#validation-and-test-results) [Google Kubernetes Engine (GKE)](https://cloud.google.com/kubernetes-engine) and [Amazon Elastic Kubernetes Service (EKS)](https://aws.amazon.com/eks/). Other Kubernetes services may also work, but your mileage may vary. @@ -2207,17 +2257,17 @@ future with further specific cloud provider details. Next are the backend components that run on static compute VMs via Omnibus (or External PaaS services where applicable): -| Service | Nodes | Configuration | GCP | -|--------------------------------------------|-------|-------------------------|------------------| -| Redis<sup>2</sup> | 3 | 2 vCPU, 7.5 GB memory | `n1-standard-2` | -| Consul<sup>1</sup> + Sentinel<sup>2</sup> | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | -| PostgreSQL<sup>1</sup> | 3 | 2 vCPU, 7.5 GB memory | `n1-standard-2` | -| PgBouncer<sup>1</sup> | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | -| Internal load balancing node<sup>3</sup> | 1 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | -| Gitaly<sup>5</sup> | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | -| Praefect<sup>5</sup> | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | -| Praefect PostgreSQL<sup>1</sup> | 1+ | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | -| Object storage<sup>4</sup> | n/a | n/a | n/a | +| Service | Nodes | Configuration | GCP | AWS | +|-------------------------------------------|-------|-----------------------|-----------------|-------------| +| Redis<sup>2</sup> | 3 | 2 vCPU, 7.5 GB memory | `n1-standard-2` | `m5.large` | +| Consul<sup>1</sup> + Sentinel<sup>2</sup> | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | +| PostgreSQL<sup>1</sup> | 3 | 2 vCPU, 7.5 GB memory | `n1-standard-2` | `m5.large` | +| PgBouncer<sup>1</sup> | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | +| Internal load balancing node<sup>3</sup> | 1 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | +| Gitaly<sup>5</sup> | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | +| Praefect<sup>5</sup> | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | +| Praefect PostgreSQL<sup>1</sup> | 1+ | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | +| Object storage<sup>4</sup> | n/a | n/a | n/a | n/a | <!-- Disable ordered list rule https://github.com/DavidAnson/markdownlint/blob/main/doc/Rules.md#md029---ordered-list-item-prefix --> <!-- markdownlint-disable MD029 --> @@ -2225,7 +2275,7 @@ services where applicable): 2. Can be optionally run on reputable third-party external PaaS Redis solutions. Google Memorystore and AWS Elasticache are known to work. 3. Can be optionally run on reputable third-party load balancing services (LB PaaS). AWS ELB is known to work. 4. Should be run on reputable third-party object storage (storage PaaS) for cloud implementations. Google Cloud Storage and AWS S3 are known to work. -5. Gitaly Cluster provides the benefits of fault tolerance, but comes with additional complexity of setup and management. Please [review the existing technical limitations and considerations prior to deploying Gitaly Cluster](../gitaly/index.md#guidance-regarding-gitaly-cluster). If Gitaly Sharded is desired, the same specs listed above for `Gitaly` should be used. +5. Gitaly Cluster provides the benefits of fault tolerance, but comes with additional complexity of setup and management. Review the existing [technical limitations and considerations before deploying Gitaly Cluster](../gitaly/index.md#before-deploying-gitaly-cluster). If you want sharded Gitaly, use the same specs listed above for `Gitaly`. <!-- markdownlint-enable MD029 --> NOTE: diff --git a/doc/administration/reference_architectures/50k_users.md b/doc/administration/reference_architectures/50k_users.md index dfa963d1ad0..6c4e2227b18 100644 --- a/doc/administration/reference_architectures/50k_users.md +++ b/doc/administration/reference_architectures/50k_users.md @@ -14,7 +14,7 @@ full list of reference architectures, see > - **High Availability:** Yes ([Praefect](#configure-praefect-postgresql) needs a third-party PostgreSQL solution for HA) > - **Estimated Costs:** [See cost table](index.md#cost-to-run) > - **Cloud Native Hybrid Alternative:** [Yes](#cloud-native-hybrid-reference-architecture-with-helm-charts-alternative) -> - **Performance tested weekly with the [GitLab Performance Tool (GPT)](https://gitlab.com/gitlab-org/quality/performance)**: +> - **Validation and test results:** The Quality Engineering team does [regular smoke and performance tests](index.md#validation-and-test-results) to ensure the reference architectures remain compliant > - **Test requests per second (RPS) rates:** API: 1000 RPS, Web: 100 RPS, Git (Pull): 100 RPS, Git (Push): 20 RPS > - **[Latest Results](https://gitlab.com/gitlab-org/quality/performance/-/wikis/Benchmarks/Latest/50k)** @@ -42,7 +42,7 @@ full list of reference architectures, see 2. Can be optionally run on reputable third-party external PaaS Redis solutions. Google Memorystore and AWS Elasticache are known to work. 3. Can be optionally run on reputable third-party load balancing services (LB PaaS). AWS ELB is known to work. 4. Should be run on reputable third-party object storage (storage PaaS) for cloud implementations. Google Cloud Storage and AWS S3 are known to work. -5. Gitaly Cluster provides the benefits of fault tolerance, but comes with additional complexity of setup and management. Please [review the existing technical limitations and considerations prior to deploying Gitaly Cluster](../gitaly/index.md#guidance-regarding-gitaly-cluster). If Gitaly Sharded is desired, the same specs listed above for `Gitaly` should be used. +5. Gitaly Cluster provides the benefits of fault tolerance, but comes with additional complexity of setup and management. Review the existing [technical limitations and considerations before deploying Gitaly Cluster](../gitaly/index.md#before-deploying-gitaly-cluster). If you want sharded Gitaly, use the same specs listed above for `Gitaly`. <!-- markdownlint-enable MD029 --> NOTE: @@ -1170,8 +1170,8 @@ Advanced [configuration options](https://docs.gitlab.com/omnibus/settings/redis. In this configuration, every Git repository is stored on every Gitaly node in the cluster, with one being designated the primary, and failover occurs automatically if the primary node goes down. NOTE: -Gitaly Cluster provides the benefits of fault tolerance, but comes with additional complexity of setup and management. Please [review the existing technical limitations and considerations prior to deploying Gitaly Cluster](../gitaly/index.md#guidance-regarding-gitaly-cluster). -For implementations with Gitaly Sharded, the same Gitaly specs should be used. Follow the [separate Gitaly documentation](../gitaly/configure_gitaly.md) instead of this section. +Gitaly Cluster provides the benefits of fault tolerance, but comes with additional complexity of setup and management. Review the existing [technical limitations and considerations before deploying Gitaly Cluster](../gitaly/index.md#before-deploying-gitaly-cluster). +For implementations with sharded Gitaly, use the same Gitaly specs. Follow the [separate Gitaly documentation](../gitaly/configure_gitaly.md) instead of this section. The recommended cluster setup includes the following components: @@ -1468,10 +1468,12 @@ the file of the same name on this server. If this is the first Omnibus node you ### Configure Gitaly The [Gitaly](../gitaly/index.md) server nodes that make up the cluster have -requirements that are dependent on data, specifically the number of projects -and those projects' sizes. It's recommended that a Gitaly Cluster stores -no more than 5 TB of data on each node. Depending on your -repository storage requirements, you may require additional Gitaly Clusters. +requirements that are dependent on data and load. + +NOTE: +The Reference Architecture specs have been designed with good headroom in mind +but for Gitaly, increased specs or additional +Gitaly Cluster arrays may be required for notably large data sets or load. Due to Gitaly having notable input and output requirements, we strongly recommend that all Gitaly nodes use solid-state drives (SSDs). These SSDs @@ -1544,6 +1546,26 @@ On each node: # Recommended to be enabled for improved performance but can notably increase disk I/O # Refer to https://docs.gitlab.com/ee/administration/gitaly/configure_gitaly.html#pack-objects-cache for more info gitaly['pack_objects_cache_enabled'] = true + + # Configure the Consul agent + consul['enable'] = true + ## Enable service discovery for Prometheus + consul['monitoring_service_discovery'] = true + + # START user configuration + # Please set the real values as explained in Required Information section + # + ## The IPs of the Consul server nodes + ## You can also use FQDNs and intermix them with IPs + consul['configuration'] = { + retry_join: %w(10.6.0.11 10.6.0.12 10.6.0.13), + } + + # Set the network addresses that the exporters will listen on for monitoring + node_exporter['listen_address'] = '0.0.0.0:9100' + gitaly['prometheus_listen_addr'] = '0.0.0.0:9236' + # + # END user configuration ``` 1. Append the following to `/etc/gitlab/gitlab.rb` for each respective server: @@ -1781,6 +1803,7 @@ To configure the Sidekiq nodes, on each one: # Object storage ## This is an example for configuring Object Storage on GCP ## Replace this config with your chosen Object Storage provider as desired + gitlab_rails['object_store']['enabled'] = true gitlab_rails['object_store']['connection'] = { 'provider' => 'Google', 'google_project' => '<gcp-project-name>', @@ -1936,6 +1959,7 @@ On each node perform the following: # This is an example for configuring Object Storage on GCP # Replace this config with your chosen Object Storage provider as desired + gitlab_rails['object_store']['enabled'] = true gitlab_rails['object_store']['connection'] = { 'provider' => 'Google', 'google_project' => '<gcp-project-name>', @@ -2114,6 +2138,30 @@ To configure the Monitoring node: retry_join: %w(10.6.0.11 10.6.0.12 10.6.0.13) } + # Configure Prometheus to scrape services not covered by discovery + prometheus['scrape_configs'] = [ + { + 'job_name': 'pgbouncer', + 'static_configs' => [ + 'targets' => [ + "10.6.0.31:9188", + "10.6.0.32:9188", + "10.6.0.33:9188", + ], + ], + }, + { + 'job_name': 'praefect', + 'static_configs' => [ + 'targets' => [ + "10.6.0.131:9652", + "10.6.0.132:9652", + "10.6.0.133:9652", + ], + ], + }, + ] + # Nginx - For Grafana access nginx['enable'] = true ``` @@ -2139,7 +2187,7 @@ GitLab has been tested on a number of object storage providers: - [Amazon S3](https://aws.amazon.com/s3/) - [Google Cloud Storage](https://cloud.google.com/storage) -- [Digital Ocean Spaces](https://www.digitalocean.com/products/spaces/) +- [Digital Ocean Spaces](http://www.digitalocean.com/products/spaces) - [Oracle Cloud Infrastructure](https://docs.cloud.oracle.com/en-us/iaas/Content/Object/Tasks/s3compatibleapi.htm) - [OpenStack Swift](https://docs.openstack.org/swift/latest/s3_compat.html) - [Azure Blob storage](https://docs.microsoft.com/en-us/azure/storage/blobs/storage-blobs-introduction) @@ -2156,7 +2204,8 @@ There are two ways of specifying object storage configuration in GitLab: Starting with GitLab 13.2, consolidated object storage configuration is available. It simplifies your GitLab configuration since the connection details are shared across object types. Refer to [Consolidated object storage configuration](../object_storage.md#consolidated-object-storage-configuration) guide for instructions on how to set it up. GitLab Runner returns job logs in chunks which Omnibus GitLab caches temporarily on disk in `/var/opt/gitlab/gitlab-ci/builds` by default, even when using consolidated object storage. With default configuration, this directory needs to be shared via NFS on any GitLab Rails and Sidekiq nodes. -In GitLab 13.6 and later, it's recommended to switch to [Incremental logging](../job_logs.md#incremental-logging-architecture), which uses Redis instead of disk space for temporary caching of job logs. + +In GitLab 13.6 and later, it's also recommended to switch to [Incremental logging](../job_logs.md#incremental-logging-architecture), which uses Redis instead of disk space for temporary caching of job logs. This is required when no NFS node has been deployed. For configuring object storage in GitLab 13.1 and earlier, or for storage types not supported by consolidated configuration form, refer to the following guides based @@ -2250,12 +2299,12 @@ use Google Cloud's Kubernetes Engine (GKE) and associated machine types, but the and CPU requirements should translate to most other providers. We hope to update this in the future with further specific cloud provider details. -| Service | Nodes | Configuration | GCP | Allocatable CPUs and Memory | -|-------------------------------------------------------|-------|-------------------------|------------------|-----------------------------| -| Webservice | 16 | 32 vCPU, 28.8 GB memory | `n1-highcpu-32` | 510 vCPU, 472 GB memory | -| Sidekiq | 4 | 4 vCPU, 15 GB memory | `n1-standard-4` | 15.5 vCPU, 50 GB memory | -| Supporting services such as NGINX, Prometheus | 2 | 4 vCPU, 15 GB memory | `n1-standard-4` | 7.75 vCPU, 25 GB memory | - +| Service | Nodes | Configuration | GCP | AWS | Min Allocatable CPUs and Memory | +|-----------------------------------------------|-------|-------------------------|-----------------|--------------|---------------------------------| +| Webservice | 16 | 32 vCPU, 28.8 GB memory | `n1-highcpu-32` | `m5.8xlarge` | 510 vCPU, 472 GB memory | +| Sidekiq | 4 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | 15.5 vCPU, 50 GB memory | +| Supporting services such as NGINX, Prometheus | 2 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | 7.75 vCPU, 25 GB memory | + - For this setup, we **recommend** and regularly [test](index.md#validation-and-test-results) [Google Kubernetes Engine (GKE)](https://cloud.google.com/kubernetes-engine) and [Amazon Elastic Kubernetes Service (EKS)](https://aws.amazon.com/eks/). Other Kubernetes services may also work, but your mileage may vary. - Nodes configuration is shown as it is forced to ensure pod vcpu / memory ratios and avoid scaling during **performance testing**. @@ -2264,18 +2313,18 @@ future with further specific cloud provider details. Next are the backend components that run on static compute VMs via Omnibus (or External PaaS services where applicable): -| Service | Nodes | Configuration | GCP | -|-----------------------------------------------------|-------|-------------------------|------------------| -| Consul<sup>1</sup> | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | -| PostgreSQL<sup>1</sup> | 3 | 32 vCPU, 120 GB memory | `n1-standard-32` | -| PgBouncer<sup>1</sup> | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | -| Internal load balancing node<sup>3</sup> | 1 | 8 vCPU, 7.2 GB memory | `n1-highcpu-8` | -| Redis/Sentinel - Cache<sup>2</sup> | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | -| Redis/Sentinel - Persistent<sup>2</sup> | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | -| Gitaly<sup>5</sup> | 3 | 64 vCPU, 240 GB memory | `n1-standard-64` | -| Praefect<sup>5</sup> | 3 | 4 vCPU, 3.6 GB memory | `n1-highcpu-4` | -| Praefect PostgreSQL<sup>1</sup> | 1+ | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | -| Object storage<sup>4</sup> | n/a | n/a | n/a | +| Service | Nodes | Configuration | GCP | AWS | +|------------------------------------------|-------|------------------------|------------------|---------------| +| Consul<sup>1</sup> | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | +| PostgreSQL<sup>1</sup> | 3 | 32 vCPU, 120 GB memory | `n1-standard-32` | `m5.8xlarge` | +| PgBouncer<sup>1</sup> | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | +| Internal load balancing node<sup>3</sup> | 1 | 8 vCPU, 7.2 GB memory | `n1-highcpu-8` | `c5.2xlarge` | +| Redis/Sentinel - Cache<sup>2</sup> | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | +| Redis/Sentinel - Persistent<sup>2</sup> | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | +| Gitaly<sup>5</sup> | 3 | 64 vCPU, 240 GB memory | `n1-standard-64` | `m5.16xlarge` | +| Praefect<sup>5</sup> | 3 | 4 vCPU, 3.6 GB memory | `n1-highcpu-4` | `c5.xlarge` | +| Praefect PostgreSQL<sup>1</sup> | 1+ | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | +| Object storage<sup>4</sup> | n/a | n/a | n/a | n/a | <!-- Disable ordered list rule https://github.com/DavidAnson/markdownlint/blob/main/doc/Rules.md#md029---ordered-list-item-prefix --> <!-- markdownlint-disable MD029 --> @@ -2283,7 +2332,7 @@ services where applicable): 2. Can be optionally run on reputable third-party external PaaS Redis solutions. Google Memorystore and AWS Elasticache are known to work. 3. Can be optionally run on reputable third-party load balancing services (LB PaaS). AWS ELB is known to work. 4. Should be run on reputable third-party object storage (storage PaaS) for cloud implementations. Google Cloud Storage and AWS S3 are known to work. -5. Gitaly Cluster provides the benefits of fault tolerance, but comes with additional complexity of setup and management. Please [review the existing technical limitations and considerations prior to deploying Gitaly Cluster](../gitaly/index.md#guidance-regarding-gitaly-cluster). If Gitaly Sharded is desired, the same specs listed above for `Gitaly` should be used. +5. Gitaly Cluster provides the benefits of fault tolerance, but comes with additional complexity of setup and management. Review the existing [technical limitations and considerations before deploying Gitaly Cluster](../gitaly/index.md#before-deploying-gitaly-cluster). If you want sharded Gitaly, use the same specs listed above for `Gitaly`. <!-- markdownlint-enable MD029 --> NOTE: diff --git a/doc/administration/reference_architectures/5k_users.md b/doc/administration/reference_architectures/5k_users.md index f2463afbf3b..dd7209a3a3a 100644 --- a/doc/administration/reference_architectures/5k_users.md +++ b/doc/administration/reference_architectures/5k_users.md @@ -21,7 +21,7 @@ costly-to-operate environment by using the > - **High Availability:** Yes ([Praefect](#configure-praefect-postgresql) needs a third-party PostgreSQL solution for HA) > - **Estimated Costs:** [See cost table](index.md#cost-to-run) > - **Cloud Native Hybrid Alternative:** [Yes](#cloud-native-hybrid-reference-architecture-with-helm-charts-alternative) -> - **Performance tested weekly with the [GitLab Performance Tool (GPT)](https://gitlab.com/gitlab-org/quality/performance)**: +> - **Validation and test results:** The Quality Engineering team does [regular smoke and performance tests](index.md#validation-and-test-results) to ensure the reference architectures remain compliant > - **Test requests per second (RPS) rates:** API: 100 RPS, Web: 10 RPS, Git (Pull): 10 RPS, Git (Push): 2 RPS > - **[Latest Results](https://gitlab.com/gitlab-org/quality/performance/-/wikis/Benchmarks/Latest/5k)** @@ -48,7 +48,7 @@ costly-to-operate environment by using the 2. Can be optionally run on reputable third-party external PaaS Redis solutions. Google Memorystore and AWS Elasticache are known to work. 3. Can be optionally run on reputable third-party load balancing services (LB PaaS). AWS ELB is known to work. 4. Should be run on reputable third-party object storage (storage PaaS) for cloud implementations. Google Cloud Storage and AWS S3 are known to work. -5. Gitaly Cluster provides the benefits of fault tolerance, but comes with additional complexity of setup and management. Please [review the existing technical limitations and considerations prior to deploying Gitaly Cluster](../gitaly/index.md#guidance-regarding-gitaly-cluster). If Gitaly Sharded is desired, the same specs listed above for `Gitaly` should be used. +5. Gitaly Cluster provides the benefits of fault tolerance, but comes with additional complexity of setup and management. Review the existing [technical limitations and considerations before deploying Gitaly Cluster](../gitaly/index.md#before-deploying-gitaly-cluster). If you want sharded Gitaly, use the same specs listed above for `Gitaly`. <!-- markdownlint-enable MD029 --> NOTE: @@ -1101,8 +1101,8 @@ The following IPs will be used as an example: In this configuration, every Git repository is stored on every Gitaly node in the cluster, with one being designated the primary, and failover occurs automatically if the primary node goes down. NOTE: -Gitaly Cluster provides the benefits of fault tolerance, but comes with additional complexity of setup and management. Please [review the existing technical limitations and considerations prior to deploying Gitaly Cluster](../gitaly/index.md#guidance-regarding-gitaly-cluster). -For implementations with Gitaly Sharded, the same Gitaly specs should be used. Follow the [separate Gitaly documentation](../gitaly/configure_gitaly.md) instead of this section. +Gitaly Cluster provides the benefits of fault tolerance, but comes with additional complexity of setup and management. Review the existing [technical limitations and considerations before deploying Gitaly Cluster](../gitaly/index.md#before-deploying-gitaly-cluster). +For implementations with sharded Gitaly, use the same Gitaly specs. Follow the [separate Gitaly documentation](../gitaly/configure_gitaly.md) instead of this section. The recommended cluster setup includes the following components: @@ -1397,10 +1397,12 @@ the file of the same name on this server. If this is the first Omnibus node you ### Configure Gitaly The [Gitaly](../gitaly/index.md) server nodes that make up the cluster have -requirements that are dependent on data, specifically the number of projects -and those projects' sizes. It's recommended that a Gitaly Cluster stores -no more than 5 TB of data on each node. Depending on your -repository storage requirements, you may require additional Gitaly Clusters. +requirements that are dependent on data and load. + +NOTE: +The Reference Architecture specs have been designed with good headroom in mind +but for Gitaly, increased specs or additional +Gitaly Cluster arrays may be required for notably large data sets or load. Due to Gitaly having notable input and output requirements, we strongly recommend that all Gitaly nodes use solid-state drives (SSDs). These SSDs @@ -1473,6 +1475,26 @@ On each node: # Recommended to be enabled for improved performance but can notably increase disk I/O # Refer to https://docs.gitlab.com/ee/administration/gitaly/configure_gitaly.html#pack-objects-cache for more info gitaly['pack_objects_cache_enabled'] = true + + # Configure the Consul agent + consul['enable'] = true + ## Enable service discovery for Prometheus + consul['monitoring_service_discovery'] = true + + # START user configuration + # Please set the real values as explained in Required Information section + # + ## The IPs of the Consul server nodes + ## You can also use FQDNs and intermix them with IPs + consul['configuration'] = { + retry_join: %w(10.6.0.11 10.6.0.12 10.6.0.13), + } + + # Set the network addresses that the exporters will listen on for monitoring + node_exporter['listen_address'] = '0.0.0.0:9100' + gitaly['prometheus_listen_addr'] = '0.0.0.0:9236' + # + # END user configuration ``` 1. Append the following to `/etc/gitlab/gitlab.rb` for each respective server: @@ -1693,6 +1715,7 @@ To configure the Sidekiq nodes, one each one: # Object Storage ## This is an example for configuring Object Storage on GCP ## Replace this config with your chosen Object Storage provider as desired + gitlab_rails['object_store']['enabled'] = true gitlab_rails['object_store']['connection'] = { 'provider' => 'Google', 'google_project' => '<gcp-project-name>', @@ -1867,6 +1890,7 @@ On each node perform the following: # This is an example for configuring Object Storage on GCP # Replace this config with your chosen Object Storage provider as desired + gitlab_rails['object_store']['enabled'] = true gitlab_rails['object_store']['connection'] = { 'provider' => 'Google', 'google_project' => '<gcp-project-name>', @@ -2018,6 +2042,30 @@ running [Prometheus](../monitoring/prometheus/index.md) and retry_join: %w(10.6.0.11 10.6.0.12 10.6.0.13) } + # Configure Prometheus to scrape services not covered by discovery + prometheus['scrape_configs'] = [ + { + 'job_name': 'pgbouncer', + 'static_configs' => [ + 'targets' => [ + "10.6.0.31:9188", + "10.6.0.32:9188", + "10.6.0.33:9188", + ], + ], + }, + { + 'job_name': 'praefect', + 'static_configs' => [ + 'targets' => [ + "10.6.0.131:9652", + "10.6.0.132:9652", + "10.6.0.133:9652", + ], + ], + }, + ] + # Nginx - For Grafana access nginx['enable'] = true ``` @@ -2058,7 +2106,7 @@ GitLab has been tested on a number of object storage providers: - [Amazon S3](https://aws.amazon.com/s3/) - [Google Cloud Storage](https://cloud.google.com/storage) -- [Digital Ocean Spaces](https://www.digitalocean.com/products/spaces/) +- [Digital Ocean Spaces](http://www.digitalocean.com/products/spaces) - [Oracle Cloud Infrastructure](https://docs.cloud.oracle.com/en-us/iaas/Content/Object/Tasks/s3compatibleapi.htm) - [OpenStack Swift](https://docs.openstack.org/swift/latest/s3_compat.html) - [Azure Blob storage](https://docs.microsoft.com/en-us/azure/storage/blobs/storage-blobs-introduction) @@ -2075,7 +2123,8 @@ There are two ways of specifying object storage configuration in GitLab: Starting with GitLab 13.2, consolidated object storage configuration is available. It simplifies your GitLab configuration since the connection details are shared across object types. Refer to [Consolidated object storage configuration](../object_storage.md#consolidated-object-storage-configuration) guide for instructions on how to set it up. GitLab Runner returns job logs in chunks which Omnibus GitLab caches temporarily on disk in `/var/opt/gitlab/gitlab-ci/builds` by default, even when using consolidated object storage. With default configuration, this directory needs to be shared via NFS on any GitLab Rails and Sidekiq nodes. -In GitLab 13.6 and later, it's recommended to switch to [Incremental logging](../job_logs.md#incremental-logging-architecture), which uses Redis instead of disk space for temporary caching of job logs. + +In GitLab 13.6 and later, it's also recommended to switch to [Incremental logging](../job_logs.md#incremental-logging-architecture), which uses Redis instead of disk space for temporary caching of job logs. This is required when no NFS node has been deployed. For configuring object storage in GitLab 13.1 and earlier, or for storage types not supported by consolidated configuration form, refer to the following guides based @@ -2169,11 +2218,11 @@ use Google Cloud's Kubernetes Engine (GKE) and associated machine types, but the and CPU requirements should translate to most other providers. We hope to update this in the future with further specific cloud provider details. -| Service | Nodes | Configuration | GCP | Allocatable CPUs and Memory | -|-------------------------------------------------------|-------|-------------------------|------------------|-----------------------------| -| Webservice | 5 | 16 vCPU, 14.4 GB memory | `n1-highcpu-16` | 79.5 vCPU, 62 GB memory | -| Sidekiq | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | 11.8 vCPU, 38.9 GB memory | -| Supporting services such as NGINX, Prometheus | 2 | 2 vCPU, 7.5 GB memory | `n1-standard-2` | 3.9 vCPU, 11.8 GB memory | +| Service | Nodes | Configuration | GCP | AWS | Min Allocatable CPUs and Memory | +|-----------------------------------------------|-------|-------------------------|-----------------|--------------|---------------------------------| +| Webservice | 5 | 16 vCPU, 14.4 GB memory | `n1-highcpu-16` | `c5.4xlarge` | 79.5 vCPU, 62 GB memory | +| Sidekiq | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | 11.8 vCPU, 38.9 GB memory | +| Supporting services such as NGINX, Prometheus | 2 | 2 vCPU, 7.5 GB memory | `n1-standard-2` | `m5.large` | 3.9 vCPU, 11.8 GB memory | - For this setup, we **recommend** and regularly [test](index.md#validation-and-test-results) [Google Kubernetes Engine (GKE)](https://cloud.google.com/kubernetes-engine) and [Amazon Elastic Kubernetes Service (EKS)](https://aws.amazon.com/eks/). Other Kubernetes services may also work, but your mileage may vary. @@ -2183,17 +2232,17 @@ future with further specific cloud provider details. Next are the backend components that run on static compute VMs via Omnibus (or External PaaS services where applicable): -| Service | Nodes | Configuration | GCP | -|--------------------------------------------|-------|-------------------------|------------------| -| Redis<sup>2</sup> | 3 | 2 vCPU, 7.5 GB memory | `n1-standard-2` | -| Consul<sup>1</sup> + Sentinel<sup>2</sup> | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | -| PostgreSQL<sup>1</sup> | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | -| PgBouncer<sup>1</sup> | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | -| Internal load balancing node<sup>3</sup> | 1 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | -| Gitaly<sup>5</sup> | 3 | 8 vCPU, 30 GB memory | `n1-standard-8` | -| Praefect<sup>5</sup> | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | -| Praefect PostgreSQL<sup>1</sup> | 1+ | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | -| Object storage<sup>4</sup> | n/a | n/a | n/a | +| Service | Nodes | Configuration | GCP | AWS | +|-------------------------------------------|-------|-----------------------|-----------------|--------------| +| Redis<sup>2</sup> | 3 | 2 vCPU, 7.5 GB memory | `n1-standard-2` | `m5.large` | +| Consul<sup>1</sup> + Sentinel<sup>2</sup> | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | +| PostgreSQL<sup>1</sup> | 3 | 4 vCPU, 15 GB memory | `n1-standard-4` | `m5.xlarge` | +| PgBouncer<sup>1</sup> | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | +| Internal load balancing node<sup>3</sup> | 1 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | +| Gitaly<sup>5</sup> | 3 | 8 vCPU, 30 GB memory | `n1-standard-8` | `m5.2xlarge` | +| Praefect<sup>5</sup> | 3 | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | +| Praefect PostgreSQL<sup>1</sup> | 1+ | 2 vCPU, 1.8 GB memory | `n1-highcpu-2` | `c5.large` | +| Object storage<sup>4</sup> | n/a | n/a | n/a | n/a | <!-- Disable ordered list rule https://github.com/DavidAnson/markdownlint/blob/main/doc/Rules.md#md029---ordered-list-item-prefix --> <!-- markdownlint-disable MD029 --> @@ -2201,7 +2250,7 @@ services where applicable): 2. Can be optionally run on reputable third-party external PaaS Redis solutions. Google Memorystore and AWS Elasticache are known to work. 3. Can be optionally run on reputable third-party load balancing services (LB PaaS). AWS ELB is known to work. 4. Should be run on reputable third-party object storage (storage PaaS) for cloud implementations. Google Cloud Storage and AWS S3 are known to work. -5. Gitaly Cluster provides the benefits of fault tolerance, but comes with additional complexity of setup and management. Please [review the existing technical limitations and considerations prior to deploying Gitaly Cluster](../gitaly/index.md#guidance-regarding-gitaly-cluster). If Gitaly Sharded is desired, the same specs listed above for `Gitaly` should be used. +5. Gitaly Cluster provides the benefits of fault tolerance, but comes with additional complexity of setup and management. Review the existing [technical limitations and considerations before deploying Gitaly Cluster](../gitaly/index.md#before-deploying-gitaly-cluster). If you want sharded Gitaly, use the same specs listed above for `Gitaly`. <!-- markdownlint-enable MD029 --> NOTE: diff --git a/doc/administration/reference_architectures/index.md b/doc/administration/reference_architectures/index.md index 3fcd6d7ae4e..bb741c39c08 100644 --- a/doc/administration/reference_architectures/index.md +++ b/doc/administration/reference_architectures/index.md @@ -18,20 +18,6 @@ you scale GitLab accordingly.  <!-- Internal link: https://docs.google.com/spreadsheets/d/1obYP4fLKkVVDOljaI3-ozhmCiPtEeMblbBKkf2OADKs/edit#gid=1403207183 --> -Testing on these reference architectures was performed with the -[GitLab Performance Tool](https://gitlab.com/gitlab-org/quality/performance) -at specific coded workloads, and the throughputs used for testing were -calculated based on sample customer data. Select the -[reference architecture](#available-reference-architectures) that matches your scale. - -Each endpoint type is tested with the following number of requests per second (RPS) -per 1,000 users: - -- API: 20 RPS -- Web: 2 RPS -- Git (Pull): 2 RPS -- Git (Push): 0.4 RPS (rounded to nearest integer) - For GitLab instances with less than 2,000 users, it's recommended that you use the [default setup](#automated-backups) by [installing GitLab](../../install/index.md) on a single machine to minimize @@ -48,7 +34,8 @@ When scaling GitLab, there are several factors to consider: - A load balancer is added in front to distribute traffic across the application nodes. - The application nodes connects to a shared file server and PostgreSQL and Redis services on the backend. -NOTE: +## Available reference architectures + Depending on your workflow, the following recommended reference architectures may need to be adapted accordingly. Your workload is influenced by factors including how active your users are, how much automation you use, mirroring, @@ -57,12 +44,10 @@ provided by [GCP machine types](https://cloud.google.com/compute/docs/machine-ty For different cloud vendors, attempt to select options that best match the provided architecture. -## Available reference architectures - -The following reference architectures are available. - ### GitLab package (Omnibus) +The following reference architectures, where the GitLab package is used, are available: + - [Up to 1,000 users](1k_users.md) - [Up to 2,000 users](2k_users.md) - [Up to 3,000 users](3k_users.md) @@ -87,17 +72,53 @@ to get assistance from Support with troubleshooting the [2,000 users](2k_users.m and higher reference architectures. [Read more about our definition of scaled architectures](https://about.gitlab.com/support/#definition-of-scaled-architecture). -### Validation and test results +## Validation and test results + +The [Quality Engineering team](https://about.gitlab.com/handbook/engineering/quality/quality-engineering/) +does regular smoke and performance tests for the reference architectures to ensure they +remain compliant. + +### Why we perform the tests + +The Quality Department has a focus on measuring and improving the performance +of GitLab, as well as creating and validating reference architectures that +self-managed customers can rely on as performant configurations. + +For more information, see our [handbook page](https://about.gitlab.com/handbook/engineering/quality/performance-and-scalability/). + +### How we perform the tests + +Testing occurs against all reference architectures and cloud providers in an automated and ad-hoc fashion. This is done by two tools: + +- The [GitLab Environment Toolkit](https://gitlab.com/gitlab-org/gitlab-environment-toolkit) for building the environments. +- The [GitLab Performance Tool](https://gitlab.com/gitlab-org/quality/performance) for performance testing. + +Network latency on the test environments between components on all Cloud Providers were measured at <5ms. Note that this is shared as an observation and not as an implicit recommendation. + +We aim to have a "test smart" approach where architectures tested have a good range that can also apply to others. Testing focuses on 10k Omnibus on GCP as the testing has shown this is a good bellwether for the other architectures and cloud providers as well as Cloud Native Hybrids. + +The Standard Reference Architectures are designed to be platform agnostic, with everything being run on VMs via [Omnibus GitLab](https://docs.gitlab.com/omnibus/). While testing occurs primarily on GCP, ad-hoc testing has shown that they perform similarly on equivalently specced hardware on other Cloud Providers or if run on premises (bare-metal). + +Testing on these reference architectures is performed with the +[GitLab Performance Tool](https://gitlab.com/gitlab-org/quality/performance) +at specific coded workloads, and the throughputs used for testing are +calculated based on sample customer data. Select the +[reference architecture](#available-reference-architectures) that matches your scale. -The [Quality Engineering - Enablement team](https://about.gitlab.com/handbook/engineering/quality/quality-engineering/) does regular smoke and performance tests for the reference architectures to ensure they remain compliant. +Each endpoint type is tested with the following number of requests per second (RPS) +per 1,000 users: -- Testing occurs against all reference architectures and cloud providers in an automated and ad-hoc fashion. This is done by two tools: - - The [GitLab Environment Toolkit](https://gitlab.com/gitlab-org/gitlab-environment-toolkit) for building the environments. - - The [GitLab Performance Tool](https://gitlab.com/gitlab-org/quality/performance) for performance testing. -- Network latency on the test environments between components on all Cloud Providers were measured at <5ms. Note that this is shared as an observation and not as an implicit recommendation. -- We aim to have a "test smart" approach where architectures tested have a good range that can also apply to others. Testing focuses on 10k Omnibus on GCP as the testing has shown this is a good bellwether for the other architectures and cloud providers as well as Cloud Native Hybrids. -- Testing is done publicly and all results are shared. -- For more information about performance testing at GitLab, read [how our QA team leverages GitLab’s performance testing tool (and you can too)](https://about.gitlab.com/blog/2020/02/18/how-were-building-up-performance-testing-of-gitlab/). +- API: 20 RPS +- Web: 2 RPS +- Git (Pull): 2 RPS +- Git (Push): 0.4 RPS (rounded to nearest integer) + +### How to interpret the results + +NOTE: +Read our blog post on [how our QA team leverages GitLab’s performance testing tool](https://about.gitlab.com/blog/2020/02/18/how-were-building-up-performance-testing-of-gitlab/). + +Testing is done publicly and all results are shared. The following table details the testing done against the reference architectures along with the frequency and results. Additional testing is continuously evaluated, and the table is updated accordingly. @@ -192,9 +213,7 @@ table.test-coverage th { </tr> </table> -The Standard Reference Architectures are designed to be platform agnostic, with everything being run on VMs via [Omnibus GitLab](https://docs.gitlab.com/omnibus/). While testing occurs primarily on GCP, ad-hoc testing has shown that they perform similarly on equivalently specced hardware on other Cloud Providers or if run on premises (bare-metal). - -### Cost to run +## Cost to run <table class="test-coverage"> <col> @@ -217,61 +236,61 @@ The Standard Reference Architectures are designed to be platform agnostic, with <th scope="row">1k</th> <td><a href="https://cloud.google.com/products/calculator#id=a6d6a94a-c7dc-4c22-85c4-7c5747f272ed">Calculated cost</a></td> <td></td> + <td><a href="https://calculator.aws/#/estimate?id=b51f178f4403b69a63f6eb33ea425f82de3bf249">Calculated cost</a></td> <td></td> - <td></td> - <td></td> + <td><a href="https://azure.com/e/1adf30bef7e34ceba9efa97c4470417b">Calculated cost</a></td> </tr> <tr> <th scope="row">2k</th> <td><a href="https://cloud.google.com/products/calculator#id=84d11491-d72a-493c-a16e-650931faa658">Calculated cost</a></td> <td></td> + <td><a href="https://calculator.aws/#/estimate?id=dce36b5cb6ab25211f74e47233d77f58fefb54e2">Calculated cost</a></td> <td></td> - <td></td> - <td></td> + <td><a href="https://azure.com/e/72764902f3854f798407fb03c3de4b6f">Calculated cost</a></td> </tr> <tr> <th scope="row">3k</th> <td><a href="https://cloud.google.com/products/calculator/#id=ac4838e6-9c40-4a36-ac43-6d1bc1843e08">Calculated cost</a></td> <td></td> + <td><a href="https://calculator.aws/#/estimate?id=b1c5b4e32e990eaeb035a148255132bd28988760">Calculated cost</a></td> <td></td> - <td></td> - <td></td> + <td><a href="https://azure.com/e/0dbfc575051943b9970e5d8ace03680d">Calculated cost</a></td> </tr> <tr> <th scope="row">5k</th> <td><a href="https://cloud.google.com/products/calculator/#id=8742e8ea-c08f-4e0a-b058-02f3a1c38a2f">Calculated cost</a></td> <td></td> + <td><a href="https://calculator.aws/#/estimate?id=2bf1af883096e6f4c6efddb4f3c35febead7fec2">Calculated cost</a></td> <td></td> - <td></td> - <td></td> + <td><a href="https://azure.com/e/8f618711ffec4b039f1581871ca6a7c9">Calculated cost</a></td> </tr> <tr> <th scope="row">10k</th> <td><a href="https://cloud.google.com/products/calculator#id=e77713f6-dc0b-4bb3-bcef-cea904ac8efd">Calculated cost</a></td> <td></td> + <td><a href="https://calculator.aws/#/estimate?id=1d374df13c0f2088d332ab0134f5b1d0f717259e">Calculated cost</a></td> <td></td> - <td></td> - <td></td> + <td><a href="https://azure.com/e/de3da8286dda4d4db1362932bc75410b">Calculated cost</a></td> </tr> <tr> <th scope="row">25k</th> <td><a href="https://cloud.google.com/products/calculator#id=925386e1-c01c-4c0a-8d7d-ebde1824b7b0">Calculated cost</a></td> <td></td> + <td><a href="https://calculator.aws/#/estimate?id=46fe6a6e9256d9b7779fae59fbbfa7e836942b7d">Calculated cost</a></td> <td></td> - <td></td> - <td></td> + <td><a href="https://azure.com/e/69724ebd82914a60857da6a3ace05a64">Calculate cost</a></td> </tr> <tr> <th scope="row">50k</th> <td><a href="https://cloud.google.com/products/calculator/#id=8006396b-88ee-40cd-a1c8-77cdefa4d3c8">Calculated cost</a></td> <td></td> + <td><a href="https://calculator.aws/#/estimate?id=e15926b1a3c7139e4faf390a3875ff807d2ab91c">Calculated cost</a></td> <td></td> - <td></td> - <td></td> + <td><a href="https://azure.com/e/3f973040ebc14023933d35f576c89846">Calculated cost</a></td> </tr> </table> -### Recommended cloud providers and services +## Recommended cloud providers and services NOTE: The following lists are non exhaustive. Generally, other cloud providers not listed @@ -347,7 +366,7 @@ The following specific cloud provider services have been found to have issues in - [Azure Blob Storage](https://azure.microsoft.com/en-gb/services/storage/blobs/) has been found to have performance limits that can impact production use at certain times. For larger Reference Architectures the service may not be sufficient for production use and an alternative is recommended for use instead. - [Azure Database for PostgreSQL Server](https://azure.microsoft.com/en-gb/services/postgresql/#overview) (Single / Flexible) is not recommended for use due to notable performance issues or missing functionality. -- [AWS Aurora Database](https://aws.amazon.com/rds/aurora) is not recommended due to compatibility issues. +- [AWS Aurora Database](https://aws.amazon.com/rds/aurora/) is not recommended due to compatibility issues. NOTE: As a general rule we unfortunately don't recommend Azure Services at this time. @@ -411,7 +430,7 @@ to any of the [available reference architectures](#available-reference-architect > - Required domain knowledge: PostgreSQL, HAProxy, shared storage, distributed systems GitLab supports [zero-downtime upgrades](../../update/zero_downtime.md). -Single GitLab nodes can be updated with only a [few minutes of downtime](../../update/zero_downtime.md#single-node-deployment). +Single GitLab nodes can be updated with only a [few minutes of downtime](../../update/index.md#upgrade-based-on-installation-method). To avoid this, we recommend to separate GitLab into several application nodes. As long as at least one of each component is online and capable of handling the instance's usage load, your team's productivity will not be interrupted during the update. diff --git a/doc/administration/repository_checks.md b/doc/administration/repository_checks.md index 6dd8faac65b..c2233f70a9a 100644 --- a/doc/administration/repository_checks.md +++ b/doc/administration/repository_checks.md @@ -34,7 +34,7 @@ When enabled, GitLab periodically runs a repository check on all project reposit repositories to detect possible data corruption. A project is checked no more than once per month. Administrators can configure the frequency of repository checks. To edit the frequency: -- For Omnibus GitLab installations, edit `gitlab_rails['repository_check_worker_cron']` in +- For Omnibus GitLab installations, edit `gitlab_rails['repository_check_worker_cron']` in `/etc/gitlab/gitlab.rb`. - For source-based installations, edit `[gitlab.cron_jobs.repository_check_worker]` in `/home/git/gitlab/config/gitlab.yml`. diff --git a/doc/administration/sidekiq_health_check.md b/doc/administration/sidekiq_health_check.md index 2ed736bac2c..ae02711f46f 100644 --- a/doc/administration/sidekiq_health_check.md +++ b/doc/administration/sidekiq_health_check.md @@ -21,7 +21,7 @@ The readiness probe checks whether the Sidekiq workers are ready to process jobs GET /readiness ``` -Assuming you set up Sidekiq's address and port to be `localhost` and `8092` respectively, +If you set Sidekiq's address as `localhost` and port as `8092`, here's an example request: ```shell @@ -44,7 +44,7 @@ Checks whether the Sidekiq cluster is running. GET /liveness ``` -Assuming you set up Sidekiq's address and port to be `localhost` and `8092` respectively, +If you set Sidekiq's address as `localhost` and port as `8092`, here's an example request: ```shell diff --git a/doc/administration/system_hooks.md b/doc/administration/system_hooks.md new file mode 100644 index 00000000000..71d7e7f1426 --- /dev/null +++ b/doc/administration/system_hooks.md @@ -0,0 +1,771 @@ +--- +stage: Ecosystem +group: Integrations +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +type: reference +--- + +# System hooks **(FREE SELF)** + +Your GitLab instance can perform HTTP POST requests on the following events: + +- `group_create` +- `group_destroy` +- `group_rename` +- `key_create` +- `key_destroy` +- `project_create` +- `project_destroy` +- `project_rename` +- `project_transfer` +- `project_update` +- `repository_update` +- `user_add_to_group` +- `user_add_to_team` +- `user_create` +- `user_destroy` +- `user_failed_login` +- `user_remove_from_group` +- `user_remove_from_team` +- `user_rename` +- `user_update_for_group` +- `user_update_for_team` + +The triggers for most of these are self-explanatory, but `project_update` and +`project_rename` deserve some clarification: `project_update` is fired any time +an attribute of a project is changed (including name, description, and tags) +_unless_ the `path` attribute is also changed. In that case, a `project_rename` +is triggered instead (so that, for instance, if all you care about is the +repository URL, you can just listen for `project_rename`). + +`user_failed_login` is sent whenever a _blocked_ user attempts to sign in and is +denied access. + +System hooks can be used, for example, for logging or changing information in an +LDAP server. + +In addition to these default events, you can enable triggers for other events, +such as push events, and disable the `repository_update` event +when you create a system hook. + +NOTE: +We follow the same structure and deprecations as [Webhooks](../user/project/integrations/webhooks.md) +for Push and Tag events, but we never display commits. + +## Create a system hook + +To create a system hook: + +1. On the top bar, select **Menu > Admin**. +1. On the left sidebar, select **System Hooks**. +1. Provide the **URL** and **Secret Token**. +1. Select the checkbox next to each optional **Trigger** you want to enable. +1. Select **Enable SSL verification**, if desired. +1. Click **Add system hook**. + +## Hooks request example + +**Request header**: + +```plaintext +X-Gitlab-Event: System Hook +``` + +**Project created:** + +```json +{ + "created_at": "2012-07-21T07:30:54Z", + "updated_at": "2012-07-21T07:38:22Z", + "event_name": "project_create", + "name": "StoreCloud", + "owner_email": "johnsmith@example.com", + "owner_name": "John Smith", + "owners": [{ + "name": "John", + "email": "user1@example.com" + }], + "path": "storecloud", + "path_with_namespace": "jsmith/storecloud", + "project_id": 74, + "project_visibility": "private" +} +``` + +**Project destroyed:** + +```json +{ + "created_at": "2012-07-21T07:30:58Z", + "updated_at": "2012-07-21T07:38:22Z", + "event_name": "project_destroy", + "name": "Underscore", + "owner_email": "johnsmith@example.com", + "owner_name": "John Smith", + "owners": [{ + "name": "John", + "email": "user1@example.com" + }], + "path": "underscore", + "path_with_namespace": "jsmith/underscore", + "project_id": 73, + "project_visibility": "internal" +} +``` + +**Project renamed:** + +```json +{ + "created_at": "2012-07-21T07:30:58Z", + "updated_at": "2012-07-21T07:38:22Z", + "event_name": "project_rename", + "name": "Underscore", + "path": "underscore", + "path_with_namespace": "jsmith/underscore", + "project_id": 73, + "owner_name": "John Smith", + "owner_email": "johnsmith@example.com", + "owners": [{ + "name": "John", + "email": "user1@example.com" + }], + "project_visibility": "internal", + "old_path_with_namespace": "jsmith/overscore" +} +``` + +Note that `project_rename` is not triggered if the namespace changes. +Please refer to `group_rename` and `user_rename` for that case. + +**Project transferred:** + +```json +{ + "created_at": "2012-07-21T07:30:58Z", + "updated_at": "2012-07-21T07:38:22Z", + "event_name": "project_transfer", + "name": "Underscore", + "path": "underscore", + "path_with_namespace": "scores/underscore", + "project_id": 73, + "owner_name": "John Smith", + "owner_email": "johnsmith@example.com", + "owners": [{ + "name": "John", + "email": "user1@example.com" + }], + "project_visibility": "internal", + "old_path_with_namespace": "jsmith/overscore" +} +``` + +**Project updated:** + +```json +{ + "created_at": "2012-07-21T07:30:54Z", + "updated_at": "2012-07-21T07:38:22Z", + "event_name": "project_update", + "name": "StoreCloud", + "owner_email": "johnsmith@example.com", + "owner_name": "John Smith", + "owners": [{ + "name": "John", + "email": "user1@example.com" + }], + "path": "storecloud", + "path_with_namespace": "jsmith/storecloud", + "project_id": 74, + "project_visibility": "private" +} +``` + +**New Team Member:** + +```json +{ + "created_at": "2012-07-21T07:30:56Z", + "updated_at": "2012-07-21T07:38:22Z", + "event_name": "user_add_to_team", + "access_level": "Maintainer", + "project_id": 74, + "project_name": "StoreCloud", + "project_path": "storecloud", + "project_path_with_namespace": "jsmith/storecloud", + "user_email": "johnsmith@example.com", + "user_name": "John Smith", + "user_username": "johnsmith", + "user_id": 41, + "project_visibility": "private" +} +``` + +**Team Member Removed:** + +```json +{ + "created_at": "2012-07-21T07:30:56Z", + "updated_at": "2012-07-21T07:38:22Z", + "event_name": "user_remove_from_team", + "access_level": "Maintainer", + "project_id": 74, + "project_name": "StoreCloud", + "project_path": "storecloud", + "project_path_with_namespace": "jsmith/storecloud", + "user_email": "johnsmith@example.com", + "user_name": "John Smith", + "user_username": "johnsmith", + "user_id": 41, + "project_visibility": "private" +} +``` + +**Team Member Updated:** + +```json +{ + "created_at": "2012-07-21T07:30:56Z", + "updated_at": "2012-07-21T07:38:22Z", + "event_name": "user_update_for_team", + "access_level": "Maintainer", + "project_id": 74, + "project_name": "StoreCloud", + "project_path": "storecloud", + "project_path_with_namespace": "jsmith/storecloud", + "user_email": "johnsmith@example.com", + "user_name": "John Smith", + "user_username": "johnsmith", + "user_id": 41, + "project_visibility": "private" +} +``` + +**User created:** + +```json +{ + "created_at": "2012-07-21T07:44:07Z", + "updated_at": "2012-07-21T07:38:22Z", + "email": "js@gitlabhq.com", + "event_name": "user_create", + "name": "John Smith", + "username": "js", + "user_id": 41 +} +``` + +**User removed:** + +```json +{ + "created_at": "2012-07-21T07:44:07Z", + "updated_at": "2012-07-21T07:38:22Z", + "email": "js@gitlabhq.com", + "event_name": "user_destroy", + "name": "John Smith", + "username": "js", + "user_id": 41 +} +``` + +**User failed login:** + +```json +{ + "event_name": "user_failed_login", + "created_at": "2017-10-03T06:08:48Z", + "updated_at": "2018-01-15T04:52:06Z", + "name": "John Smith", + "email": "user4@example.com", + "user_id": 26, + "username": "user4", + "state": "blocked" +} +``` + +If the user is blocked via LDAP, `state` is `ldap_blocked`. + +**User renamed:** + +```json +{ + "event_name": "user_rename", + "created_at": "2017-11-01T11:21:04Z", + "updated_at": "2017-11-01T14:04:47Z", + "name": "new-name", + "email": "best-email@example.tld", + "user_id": 58, + "username": "new-exciting-name", + "old_username": "old-boring-name" +} +``` + +**Key added** + +```json +{ + "event_name": "key_create", + "created_at": "2014-08-18 18:45:16 UTC", + "updated_at": "2012-07-21T07:38:22Z", + "username": "root", + "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC58FwqHUbebw2SdT7SP4FxZ0w+lAO/erhy2ylhlcW/tZ3GY3mBu9VeeiSGoGz8hCx80Zrz+aQv28xfFfKlC8XQFpCWwsnWnQqO2Lv9bS8V1fIHgMxOHIt5Vs+9CAWGCCvUOAurjsUDoE2ALIXLDMKnJxcxD13XjWdK54j6ZXDB4syLF0C2PnAQSVY9X7MfCYwtuFmhQhKaBussAXpaVMRHltie3UYSBUUuZaB3J4cg/7TxlmxcNd+ppPRIpSZAB0NI6aOnqoBCpimscO/VpQRJMVLr3XiSYeT6HBiDXWHnIVPfQc03OGcaFqOit6p8lYKMaP/iUQLm+pgpZqrXZ9vB john@localhost", + "id": 4 +} +``` + +**Key removed** + +```json +{ + "event_name": "key_destroy", + "created_at": "2014-08-18 18:45:16 UTC", + "updated_at": "2012-07-21T07:38:22Z", + "username": "root", + "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC58FwqHUbebw2SdT7SP4FxZ0w+lAO/erhy2ylhlcW/tZ3GY3mBu9VeeiSGoGz8hCx80Zrz+aQv28xfFfKlC8XQFpCWwsnWnQqO2Lv9bS8V1fIHgMxOHIt5Vs+9CAWGCCvUOAurjsUDoE2ALIXLDMKnJxcxD13XjWdK54j6ZXDB4syLF0C2PnAQSVY9X7MfCYwtuFmhQhKaBussAXpaVMRHltie3UYSBUUuZaB3J4cg/7TxlmxcNd+ppPRIpSZAB0NI6aOnqoBCpimscO/VpQRJMVLr3XiSYeT6HBiDXWHnIVPfQc03OGcaFqOit6p8lYKMaP/iUQLm+pgpZqrXZ9vB john@localhost", + "id": 4 +} +``` + +**Group created:** + +```json +{ + "created_at": "2012-07-21T07:30:54Z", + "updated_at": "2012-07-21T07:38:22Z", + "event_name": "group_create", + "name": "StoreCloud", + "path": "storecloud", + "group_id": 78 +} +``` + +**Group removed:** + +```json +{ + "created_at": "2012-07-21T07:30:54Z", + "updated_at": "2012-07-21T07:38:22Z", + "event_name": "group_destroy", + "name": "StoreCloud", + "path": "storecloud", + "group_id": 78 +} +``` + +**Group renamed:** + +```json +{ + "event_name": "group_rename", + "created_at": "2017-10-30T15:09:00Z", + "updated_at": "2017-11-01T10:23:52Z", + "name": "Better Name", + "path": "better-name", + "full_path": "parent-group/better-name", + "group_id": 64, + "old_path": "old-name", + "old_full_path": "parent-group/old-name" +} +``` + +**New Group Member:** + +```json +{ + "created_at": "2012-07-21T07:30:56Z", + "updated_at": "2012-07-21T07:38:22Z", + "event_name": "user_add_to_group", + "group_access": "Maintainer", + "group_id": 78, + "group_name": "StoreCloud", + "group_path": "storecloud", + "user_email": "johnsmith@example.com", + "user_name": "John Smith", + "user_username": "johnsmith", + "user_id": 41 +} +``` + +**Group Member Removed:** + +```json +{ + "created_at": "2012-07-21T07:30:56Z", + "updated_at": "2012-07-21T07:38:22Z", + "event_name": "user_remove_from_group", + "group_access": "Maintainer", + "group_id": 78, + "group_name": "StoreCloud", + "group_path": "storecloud", + "user_email": "johnsmith@example.com", + "user_name": "John Smith", + "user_username": "johnsmith", + "user_id": 41 +} +``` + +**Group Member Updated:** + +```json +{ + "created_at": "2012-07-21T07:30:56Z", + "updated_at": "2012-07-21T07:38:22Z", + "event_name": "user_update_for_group", + "group_access": "Maintainer", + "group_id": 78, + "group_name": "StoreCloud", + "group_path": "storecloud", + "user_email": "johnsmith@example.com", + "user_name": "John Smith", + "user_username": "johnsmith", + "user_id": 41 +} +``` + +## Push events + +Triggered when you push to the repository, except when pushing tags. +It generates one event per modified branch. + +**Request header**: + +```plaintext +X-Gitlab-Event: System Hook +``` + +**Request body:** + +```json +{ + "event_name": "push", + "before": "95790bf891e76fee5e1747ab589903a6a1f80f22", + "after": "da1560886d4f094c3e6c9ef40349f7d38b5d27d7", + "ref": "refs/heads/master", + "checkout_sha": "da1560886d4f094c3e6c9ef40349f7d38b5d27d7", + "user_id": 4, + "user_name": "John Smith", + "user_email": "john@example.com", + "user_avatar": "https://s.gravatar.com/avatar/d4c74594d841139328695756648b6bd6?s=8://s.gravatar.com/avatar/d4c74594d841139328695756648b6bd6?s=80", + "project_id": 15, + "project":{ + "name":"Diaspora", + "description":"", + "web_url":"http://example.com/mike/diaspora", + "avatar_url":null, + "git_ssh_url":"git@example.com:mike/diaspora.git", + "git_http_url":"http://example.com/mike/diaspora.git", + "namespace":"Mike", + "visibility_level":0, + "path_with_namespace":"mike/diaspora", + "default_branch":"master", + "homepage":"http://example.com/mike/diaspora", + "url":"git@example.com:mike/diaspora.git", + "ssh_url":"git@example.com:mike/diaspora.git", + "http_url":"http://example.com/mike/diaspora.git" + }, + "repository":{ + "name": "Diaspora", + "url": "git@example.com:mike/diaspora.git", + "description": "", + "homepage": "http://example.com/mike/diaspora", + "git_http_url":"http://example.com/mike/diaspora.git", + "git_ssh_url":"git@example.com:mike/diaspora.git", + "visibility_level":0 + }, + "commits": [ + { + "id": "c5feabde2d8cd023215af4d2ceeb7a64839fc428", + "message": "Add simple search to projects in public area", + "timestamp": "2013-05-13T18:18:08+00:00", + "url": "https://dev.gitlab.org/gitlab/gitlabhq/commit/c5feabde2d8cd023215af4d2ceeb7a64839fc428", + "author": { + "name": "Example User", + "email": "user@example.com" + } + } + ], + "total_commits_count": 1 +} +``` + +## Tag events + +Triggered when you create (or delete) tags to the repository. +It generates one event per modified tag. + +**Request header**: + +```plaintext +X-Gitlab-Event: System Hook +``` + +**Request body:** + +```json +{ + "event_name": "tag_push", + "before": "0000000000000000000000000000000000000000", + "after": "82b3d5ae55f7080f1e6022629cdb57bfae7cccc7", + "ref": "refs/tags/v1.0.0", + "checkout_sha": "5937ac0a7beb003549fc5fd26fc247adbce4a52e", + "user_id": 1, + "user_name": "John Smith", + "user_avatar": "https://s.gravatar.com/avatar/d4c74594d841139328695756648b6bd6?s=8://s.gravatar.com/avatar/d4c74594d841139328695756648b6bd6?s=80", + "project_id": 1, + "project":{ + "name":"Example", + "description":"", + "web_url":"http://example.com/jsmith/example", + "avatar_url":null, + "git_ssh_url":"git@example.com:jsmith/example.git", + "git_http_url":"http://example.com/jsmith/example.git", + "namespace":"Jsmith", + "visibility_level":0, + "path_with_namespace":"jsmith/example", + "default_branch":"master", + "homepage":"http://example.com/jsmith/example", + "url":"git@example.com:jsmith/example.git", + "ssh_url":"git@example.com:jsmith/example.git", + "http_url":"http://example.com/jsmith/example.git" + }, + "repository":{ + "name": "Example", + "url": "ssh://git@example.com/jsmith/example.git", + "description": "", + "homepage": "http://example.com/jsmith/example", + "git_http_url":"http://example.com/jsmith/example.git", + "git_ssh_url":"git@example.com:jsmith/example.git", + "visibility_level":0 + }, + "commits": [], + "total_commits_count": 0 +} +``` + +## Merge request events + +Triggered when a new merge request is created, an existing merge request was +updated/merged/closed or a commit is added in the source branch. + +**Request header**: + +```plaintext +X-Gitlab-Event: System Hook +``` + +```json +{ + "object_kind": "merge_request", + "event_type": "merge_request", + "user": { + "id": 1, + "name": "Administrator", + "username": "root", + "avatar_url": "http://www.gravatar.com/avatar/e64c7d89f26bd1972efa854d13d7dd61?s=40\u0026d=identicon", + "email": "admin@example.com" + }, + "project": { + "id": 1, + "name":"Gitlab Test", + "description":"Aut reprehenderit ut est.", + "web_url":"http://example.com/gitlabhq/gitlab-test", + "avatar_url":null, + "git_ssh_url":"git@example.com:gitlabhq/gitlab-test.git", + "git_http_url":"http://example.com/gitlabhq/gitlab-test.git", + "namespace":"GitlabHQ", + "visibility_level":20, + "path_with_namespace":"gitlabhq/gitlab-test", + "default_branch":"master", + "homepage":"http://example.com/gitlabhq/gitlab-test", + "url":"http://example.com/gitlabhq/gitlab-test.git", + "ssh_url":"git@example.com:gitlabhq/gitlab-test.git", + "http_url":"http://example.com/gitlabhq/gitlab-test.git" + }, + "repository": { + "name": "Gitlab Test", + "url": "http://example.com/gitlabhq/gitlab-test.git", + "description": "Aut reprehenderit ut est.", + "homepage": "http://example.com/gitlabhq/gitlab-test" + }, + "object_attributes": { + "id": 99, + "target_branch": "master", + "source_branch": "ms-viewport", + "source_project_id": 14, + "author_id": 51, + "assignee_id": 6, + "title": "MS-Viewport", + "created_at": "2013-12-03T17:23:34Z", + "updated_at": "2013-12-03T17:23:34Z", + "milestone_id": null, + "state": "opened", + "merge_status": "unchecked", + "target_project_id": 14, + "iid": 1, + "description": "", + "source": { + "name":"Awesome Project", + "description":"Aut reprehenderit ut est.", + "web_url":"http://example.com/awesome_space/awesome_project", + "avatar_url":null, + "git_ssh_url":"git@example.com:awesome_space/awesome_project.git", + "git_http_url":"http://example.com/awesome_space/awesome_project.git", + "namespace":"Awesome Space", + "visibility_level":20, + "path_with_namespace":"awesome_space/awesome_project", + "default_branch":"master", + "homepage":"http://example.com/awesome_space/awesome_project", + "url":"http://example.com/awesome_space/awesome_project.git", + "ssh_url":"git@example.com:awesome_space/awesome_project.git", + "http_url":"http://example.com/awesome_space/awesome_project.git" + }, + "target": { + "name":"Awesome Project", + "description":"Aut reprehenderit ut est.", + "web_url":"http://example.com/awesome_space/awesome_project", + "avatar_url":null, + "git_ssh_url":"git@example.com:awesome_space/awesome_project.git", + "git_http_url":"http://example.com/awesome_space/awesome_project.git", + "namespace":"Awesome Space", + "visibility_level":20, + "path_with_namespace":"awesome_space/awesome_project", + "default_branch":"master", + "homepage":"http://example.com/awesome_space/awesome_project", + "url":"http://example.com/awesome_space/awesome_project.git", + "ssh_url":"git@example.com:awesome_space/awesome_project.git", + "http_url":"http://example.com/awesome_space/awesome_project.git" + }, + "last_commit": { + "id": "da1560886d4f094c3e6c9ef40349f7d38b5d27d7", + "message": "fixed readme", + "timestamp": "2012-01-03T23:36:29+02:00", + "url": "http://example.com/awesome_space/awesome_project/commits/da1560886d4f094c3e6c9ef40349f7d38b5d27d7", + "author": { + "name": "GitLab dev user", + "email": "gitlabdev@dv6700.(none)" + } + }, + "work_in_progress": false, + "url": "http://example.com/diaspora/merge_requests/1", + "action": "open", + "assignee": { + "name": "User1", + "username": "user1", + "avatar_url": "http://www.gravatar.com/avatar/e64c7d89f26bd1972efa854d13d7dd61?s=40\u0026d=identicon" + } + }, + "labels": [{ + "id": 206, + "title": "API", + "color": "#ffffff", + "project_id": 14, + "created_at": "2013-12-03T17:15:43Z", + "updated_at": "2013-12-03T17:15:43Z", + "template": false, + "description": "API related issues", + "type": "ProjectLabel", + "group_id": 41 + }], + "changes": { + "updated_by_id": { + "previous": null, + "current": 1 + }, + "updated_at": { + "previous": "2017-09-15 16:50:55 UTC", + "current":"2017-09-15 16:52:00 UTC" + }, + "labels": { + "previous": [{ + "id": 206, + "title": "API", + "color": "#ffffff", + "project_id": 14, + "created_at": "2013-12-03T17:15:43Z", + "updated_at": "2013-12-03T17:15:43Z", + "template": false, + "description": "API related issues", + "type": "ProjectLabel", + "group_id": 41 + }], + "current": [{ + "id": 205, + "title": "Platform", + "color": "#123123", + "project_id": 14, + "created_at": "2013-12-03T17:15:43Z", + "updated_at": "2013-12-03T17:15:43Z", + "template": false, + "description": "Platform related issues", + "type": "ProjectLabel", + "group_id": 41 + }] + } + } +} +``` + +## Repository Update events + +Triggered only once when you push to the repository (including tags). + +**Request header**: + +```plaintext +X-Gitlab-Event: System Hook +``` + +**Request body:** + +```json +{ + "event_name": "repository_update", + "user_id": 1, + "user_name": "John Smith", + "user_email": "admin@example.com", + "user_avatar": "https://s.gravatar.com/avatar/d4c74594d841139328695756648b6bd6?s=8://s.gravatar.com/avatar/d4c74594d841139328695756648b6bd6?s=80", + "project_id": 1, + "project": { + "name":"Example", + "description":"", + "web_url":"http://example.com/jsmith/example", + "avatar_url":null, + "git_ssh_url":"git@example.com:jsmith/example.git", + "git_http_url":"http://example.com/jsmith/example.git", + "namespace":"Jsmith", + "visibility_level":0, + "path_with_namespace":"jsmith/example", + "default_branch":"master", + "homepage":"http://example.com/jsmith/example", + "url":"git@example.com:jsmith/example.git", + "ssh_url":"git@example.com:jsmith/example.git", + "http_url":"http://example.com/jsmith/example.git" + }, + "changes": [ + { + "before":"8205ea8d81ce0c6b90fbe8280d118cc9fdad6130", + "after":"4045ea7a3df38697b3730a20fb73c8bed8a3e69e", + "ref":"refs/heads/master" + } + ], + "refs":["refs/heads/master"] +} +``` + +## Local requests in system hooks + +[Requests to local network by system hooks](../security/webhooks.md) can be allowed +or blocked by an administrator. + +<!-- ## Troubleshooting + +Include any troubleshooting steps that you can foresee. If you know beforehand what issues +one might have when setting this up, or when something is changed, or on upgrading, it's +important to describe those, too. Think of things that may go wrong and include them here. +This is important to minimize requests for support, and to avoid doc comments with +questions that you know someone might ask. + +Each scenario can be a third-level heading, e.g. `### Getting error message X`. +If you have none to add when creating a doc, leave this section in place +but commented out to help encourage others to add to it in the future. --> diff --git a/doc/administration/troubleshooting/gitlab_rails_cheat_sheet.md b/doc/administration/troubleshooting/gitlab_rails_cheat_sheet.md index 1c948771f5b..54d934c8986 100644 --- a/doc/administration/troubleshooting/gitlab_rails_cheat_sheet.md +++ b/doc/administration/troubleshooting/gitlab_rails_cheat_sheet.md @@ -257,6 +257,13 @@ ProjectDestroyWorker.perform_async(project.id, user.id, {}) # or Projects::DestroyService.new(project, user).execute ``` +If this fails, display why it doesn't work with: + +```ruby +project = Project.find_by_full_path('<project_path>') +project.delete_error +``` + ### Remove fork relationship manually ```ruby diff --git a/doc/administration/troubleshooting/group_saml_scim.md b/doc/administration/troubleshooting/group_saml_scim.md index a30ade058f6..c6a102e87ee 100644 --- a/doc/administration/troubleshooting/group_saml_scim.md +++ b/doc/administration/troubleshooting/group_saml_scim.md @@ -20,6 +20,7 @@ They may then set up a test configuration of the desired identity provider. We i This section includes relevant screenshots of the following example configurations of [Group SAML](../../user/group/saml_sso/index.md) and [Group SCIM](../../user/group/saml_sso/scim_setup.md): - [Azure Active Directory](#azure-active-directory) +- [Google Workspace](#google-workspace) - [Okta](#okta) - [OneLogin](#onelogin) @@ -47,6 +48,20 @@ Group Sync:  +## Google Workspace + +Basic SAML app configuration: + + + +User claims and attributes: + + + +IdP links and certificate: + + + ## Okta Basic SAML app configuration: diff --git a/doc/administration/troubleshooting/img/GoogleWorkspace-basic-SAML_v14_10.png b/doc/administration/troubleshooting/img/GoogleWorkspace-basic-SAML_v14_10.png Binary files differnew file mode 100644 index 00000000000..bc11e18fb6f --- /dev/null +++ b/doc/administration/troubleshooting/img/GoogleWorkspace-basic-SAML_v14_10.png diff --git a/doc/administration/troubleshooting/img/GoogleWorkspace-claims_v14_10.png b/doc/administration/troubleshooting/img/GoogleWorkspace-claims_v14_10.png Binary files differnew file mode 100644 index 00000000000..78bb1725e9c --- /dev/null +++ b/doc/administration/troubleshooting/img/GoogleWorkspace-claims_v14_10.png diff --git a/doc/administration/troubleshooting/img/GoogleWorkspace-linkscert_v14_10.png b/doc/administration/troubleshooting/img/GoogleWorkspace-linkscert_v14_10.png Binary files differnew file mode 100644 index 00000000000..e665e23058c --- /dev/null +++ b/doc/administration/troubleshooting/img/GoogleWorkspace-linkscert_v14_10.png diff --git a/doc/administration/troubleshooting/log_parsing.md b/doc/administration/troubleshooting/log_parsing.md index c5443c564f4..c5b1d302db2 100644 --- a/doc/administration/troubleshooting/log_parsing.md +++ b/doc/administration/troubleshooting/log_parsing.md @@ -11,6 +11,11 @@ but if they are not available you can still quickly parse [GitLab logs](../logs.md) in JSON format (the default in GitLab 12.0 and later) using [`jq`](https://stedolan.github.io/jq/). +NOTE: +Spefically for summarising error events and basic usage statistics, +the GitLab Support Team provides the specialised +[`fast-stats` tool](https://gitlab.com/gitlab-com/support/toolbox/fast-stats/#when-to-use-it). + ## What is JQ? As noted in its [manual](https://stedolan.github.io/jq/manual/), `jq` is a command-line JSON processor. The following examples @@ -18,6 +23,10 @@ include use cases targeted for parsing GitLab log files. ## Parsing Logs +The examples listed below address their respective log files by +their relative Omnibus paths and default filenames. +Find the respective full paths in the [GitLab logs sections](../logs.md#production_jsonlog). + ### General Commands #### Pipe colorized `jq` output into `less` @@ -61,7 +70,7 @@ zcat some_json.log.25.gz | (head -1; tail -1) | jq '.time' grep -hR <correlationID> | jq -c -R 'fromjson?' | jq -C -s 'sort_by(.time)' | less -R ``` -### Parsing `production_json.log` and `api_json.log` +### Parsing `gitlab-rails/production_json.log` and `gitlab-rails/api_json.log` #### Find all requests with a 5XX status code @@ -111,7 +120,7 @@ jq 'select(.queue_duration > 10000)' <FILE> jq -s 'map(select(.gitaly_calls != null)) | sort_by(-.gitaly_calls) | limit(10; .[])' <FILE> ``` -### Parsing `production_json.log` +### Parsing `gitlab-rails/production_json.log` #### Print the top three controller methods by request volume and their three longest durations @@ -127,7 +136,7 @@ CT: 2435 METHOD: MetricsController#index DURS: 299.29, 284.01, 158.57 CT: 1328 METHOD: Projects::NotesController#index DURS: 403.99, 386.29, 384.39 ``` -### Parsing `api_json.log` +### Parsing `gitlab-rails/api_json.log` #### Print top three routes with request count and their three longest durations @@ -143,8 +152,22 @@ CT: 297 ROUTE: /api/:version/projects/:id/repository/tags DURS: 731.39, CT: 190 ROUTE: /api/:version/projects/:id/repository/commits DURS: 1079.02, 979.68, 958.21 ``` +### Parsing `gitlab-rails/geo.log` + +#### Find most common Geo sync errors + +If [the `geo:status` Rake task](../geo/replication/troubleshooting.md#sync-status-rake-task) +repeatedly reports that some items never reach 100%, +the following command helps to focus on the most common errors. + +```shell +jq --raw-output 'select(.severity == "ERROR") | [.project_path, .message] | @tsv' geo.log | sort | uniq -c | sort | tail +``` + ### Parsing `gitaly/current` +The following examples are useful to [troubleshoot Gitaly](../gitaly/troubleshooting.md). + #### Find all Gitaly requests sent from web UI ```shell @@ -185,7 +208,7 @@ jq --raw-output --slurp ' .[2]."grpc.time_ms", .[0]."grpc.request.glProjectPath" ] - | @sh' /var/log/gitlab/gitaly/current \ + | @sh' current \ | awk 'BEGIN { printf "%7s %10s %10s %10s\t%s\n", "CT", "MAX DURS", "", "", "PROJECT" } { printf "%7u %7u ms, %7u ms, %7u ms\t%s\n", $1, $2, $3, $4, $5 }' ``` @@ -203,12 +226,12 @@ jq --raw-output --slurp ' #### Find all projects affected by a fatal Git problem ```shell -grep "fatal: " /var/log/gitlab/gitaly/current | \ +grep "fatal: " current | \ jq '."grpc.request.glProjectPath"' | \ sort | uniq ``` -### Parsing `gitlab-shell.log` +### Parsing `gitlab-shell/gitlab-shell.log` For investigating Git calls via SSH, from [GitLab 12.10](https://gitlab.com/gitlab-org/gitlab-shell/-/merge_requests/367). @@ -226,7 +249,7 @@ jq --raw-output --slurp ' | sort_by(-length) | limit(20; .[]) | "count: \(length)\tuser: \(.[0].username)\tproject: \(.[0].gl_project_path)" ' \ - /var/log/gitlab/gitlab-shell/gitlab-shell.log + gitlab-shell.log ``` Find the top 20 calls by project, user, and command: @@ -244,5 +267,5 @@ jq --raw-output --slurp ' | sort_by(-length) | limit(20; .[]) | "count: \(length)\tcommand: \(.[0].command)\tuser: \(.[0].username)\tproject: \(.[0].gl_project_path)" ' \ - /var/log/gitlab/gitlab-shell/gitlab-shell.log + gitlab-shell.log ``` |
