diff options
Diffstat (limited to 'doc/administration')
145 files changed, 5707 insertions, 4363 deletions
diff --git a/doc/administration/audit_events.md b/doc/administration/audit_events.md index e42982e6524..cc94d756f99 100644 --- a/doc/administration/audit_events.md +++ b/doc/administration/audit_events.md @@ -1,6 +1,6 @@ --- -stage: Monitor -group: APM +stage: Manage +group: Analytics info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers --- @@ -36,7 +36,7 @@ There are two kinds of events logged: ### Impersonation data **(PREMIUM)** -> [Introduced](https://gitlab.com/gitlab-org/gitlab/issues/536) in [GitLab Premium](https://about.gitlab.com/pricing/) 13.0. +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/536) in [GitLab Premium](https://about.gitlab.com/pricing/) 13.0. Impersonation is where an administrator uses credentials to perform an action as a different user. @@ -93,13 +93,15 @@ From there, you can see the following actions: - Release was added to a project - Release was updated - Release milestone associations changed -- Permission to approve merge requests by committers was updated ([introduced](https://gitlab.com/gitlab-org/gitlab/issues/7531) in GitLab 12.9) -- Permission to approve merge requests by authors was updated ([introduced](https://gitlab.com/gitlab-org/gitlab/issues/7531) in GitLab 12.9) -- Number of required approvals was updated ([introduced](https://gitlab.com/gitlab-org/gitlab/issues/7531) in GitLab 12.9) +- Permission to approve merge requests by committers was updated ([introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/7531) in GitLab 12.9) +- Permission to approve merge requests by authors was updated ([introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/7531) in GitLab 12.9) +- Number of required approvals was updated ([introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/7531) in GitLab 12.9) + +Project events can also be accessed via the [Project Audit Events API](../api/audit_events.md#project-audit-events-starter) ### Instance events **(PREMIUM ONLY)** -> [Introduced](https://gitlab.com/gitlab-org/gitlab/issues/2336) in [GitLab Premium](https://about.gitlab.com/pricing/) 9.3. +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/2336) in [GitLab Premium](https://about.gitlab.com/pricing/) 9.3. Server-wide audit logging introduces the ability to observe user actions across the entire instance of your GitLab server, making it easy to understand who @@ -118,10 +120,10 @@ recorded: - Ask for password reset - Grant OAuth access - Started or stopped user impersonation -- Changed username ([introduced](https://gitlab.com/gitlab-org/gitlab/issues/7797) in GitLab 12.8) -- User was deleted ([introduced](https://gitlab.com/gitlab-org/gitlab/issues/251) in GitLab 12.8) -- User was added ([introduced](https://gitlab.com/gitlab-org/gitlab/issues/251) in GitLab 12.8) -- User was blocked via Admin Area ([introduced](https://gitlab.com/gitlab-org/gitlab/issues/251) in GitLab 12.8) +- Changed username ([introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/7797) in GitLab 12.8) +- User was deleted ([introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/251) in GitLab 12.8) +- User was added ([introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/251) in GitLab 12.8) +- User was blocked via Admin Area ([introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/251) in GitLab 12.8) - User was blocked via API ([introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/25872) in GitLab 12.9) It's possible to filter particular actions by choosing an audit data type from @@ -152,7 +154,7 @@ It may make the user interface for your project or audit logs very busy, and the to prevent performance degradations on GitLab instances with very high Git write traffic. In an upcoming release, Audit Logs for Git push events will be enabled -by default. Follow [#7865](https://gitlab.com/gitlab-org/gitlab/issues/7865) for updates. +by default. Follow [#7865](https://gitlab.com/gitlab-org/gitlab/-/issues/7865) for updates. If you still wish to enable **Repository push** events in your instance, follow the steps bellow. diff --git a/doc/administration/auth/README.md b/doc/administration/auth/README.md index f30d6be1775..60e1dfb4637 100644 --- a/doc/administration/auth/README.md +++ b/doc/administration/auth/README.md @@ -1,6 +1,9 @@ --- comments: false type: index +stage: Manage +group: Access +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers --- # GitLab authentication and authorization @@ -21,10 +24,9 @@ providers: - [Google](../../integration/google.md) - [JWT](jwt.md) - [Kerberos](../../integration/kerberos.md) -- [LDAP](ldap.md): Includes Active Directory, Apple Open Directory, Open LDAP, +- [LDAP](ldap/index.md): Includes Active Directory, Apple Open Directory, Open LDAP, and 389 Server. - - [LDAP for GitLab EE](ldap-ee.md): LDAP additions to GitLab Enterprise Editions **(STARTER ONLY)** - - [Google Secure LDAP](google_secure_ldap.md) + - [Google Secure LDAP](ldap/google_secure_ldap.md) - [Okta](okta.md) - [Salesforce](../../integration/salesforce.md) - [SAML](../../integration/saml.md) @@ -32,4 +34,6 @@ providers: - [Shibboleth](../../integration/shibboleth.md) - [Smartcard](smartcard.md) **(PREMIUM ONLY)** - [Twitter](../../integration/twitter.md) -- [UltraAuth](../../integration/ultra_auth.md) + +NOTE: **Note:** +UltraAuth has removed their software which supports OmniAuth integration. We have therefore removed all references to UltraAuth integration. diff --git a/doc/administration/auth/authentiq.md b/doc/administration/auth/authentiq.md index e9b32b64160..56f6fddc1af 100644 --- a/doc/administration/auth/authentiq.md +++ b/doc/administration/auth/authentiq.md @@ -1,5 +1,8 @@ --- type: reference +stage: Manage +group: Access +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers --- # Authentiq OmniAuth Provider diff --git a/doc/administration/auth/cognito.md b/doc/administration/auth/cognito.md index 8d5580ccb6c..b4df6446835 100644 --- a/doc/administration/auth/cognito.md +++ b/doc/administration/auth/cognito.md @@ -1,3 +1,10 @@ +--- +type: concepts, howto +stage: Manage +group: Access +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + # Amazon Web Services Cognito Amazon Cognito lets you add user sign-up, sign-in, and access control to your GitLab instance. @@ -51,6 +58,8 @@ Include the code block in the `/etc/gitlab/gitlab.rb` file: gitlab_rails['omniauth_providers'] = [ { "name" => "cognito", + # "label" => "Cognito", + # "icon" => nil, # Optional icon URL "app_id" => "CLIENT ID", "app_secret" => "CLIENT SECRET", "args" => { @@ -79,3 +88,5 @@ Include the code block in the `/etc/gitlab/gitlab.rb` file: Your sign-in page should now display a Cognito button below the regular sign-in form. To begin the authentication process, click the icon, and AWS Cognito will ask the user to sign in and authorize the GitLab application. If successful, the user will be redirected and signed in to your GitLab instance. + +For more information, see the [Initial OmniAuth Configuration](../../integration/omniauth.md#initial-omniauth-configuration). diff --git a/doc/administration/auth/crowd.md b/doc/administration/auth/crowd.md index 71938d4fd2b..254bd259344 100644 --- a/doc/administration/auth/crowd.md +++ b/doc/administration/auth/crowd.md @@ -1,5 +1,8 @@ --- type: reference +stage: Manage +group: Access +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers --- # Atlassian Crowd OmniAuth Provider diff --git a/doc/administration/auth/google_secure_ldap.md b/doc/administration/auth/google_secure_ldap.md index b643dd2f7b9..d30efc6d3cc 100644 --- a/doc/administration/auth/google_secure_ldap.md +++ b/doc/administration/auth/google_secure_ldap.md @@ -1,219 +1,5 @@ --- -type: reference +redirect_to: 'ldap/google_secure_ldap.md' --- -# Google Secure LDAP **(CORE ONLY)** - -> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/issues/46391) in GitLab 11.9. - -[Google Cloud Identity](https://cloud.google.com/identity/) provides a Secure -LDAP service that can be configured with GitLab for authentication and group sync. - -Secure LDAP requires a slightly different configuration than standard LDAP servers. -The steps below cover: - -- Configuring the Secure LDAP Client in the Google Admin console. -- Required GitLab configuration. - -## Configuring Google LDAP client - -1. Navigate to <https://admin.google.com/Dashboard> and sign in as a GSuite domain administrator. - -1. Go to **Apps > LDAP > Add Client**. - -1. Provide an `LDAP client name` and an optional `Description`. Any descriptive - values are acceptable. For example, the name could be 'GitLab' and the - description could be 'GitLab LDAP Client'. Click the **Continue** button. - -  - -1. Set **Access Permission** according to your needs. You must choose either - 'Entire domain (GitLab)' or 'Selected organizational units' for both 'Verify user - credentials' and 'Read user information'. Select 'Add LDAP Client' - - TIP: **Tip:** If you plan to use GitLab [LDAP Group Sync](ldap-ee.md#group-sync) - , turn on 'Read group information'. - -  - -1. Download the generated certificate. This is required for GitLab to - communicate with the Google Secure LDAP service. Save the downloaded certificates - for later use. After downloading, click the **Continue to Client Details** button. - -1. Expand the **Service Status** section and turn the LDAP client 'ON for everyone'. - After selecting 'Save', click on the 'Service Status' bar again to collapse - and return to the rest of the settings. - -1. Expand the **Authentication** section and choose 'Generate New Credentials'. - Copy/note these credentials for later use. After selecting 'Close', click - on the 'Authentication' bar again to collapse and return to the rest of the settings. - -Now the Google Secure LDAP Client configuration is finished. The screenshot below -shows an example of the final settings. Continue on to configure GitLab. - - - -## Configuring GitLab - -Edit GitLab configuration, inserting the access credentials and certificate -obtained earlier. - -The following are the configuration keys that need to be modified using the -values obtained during the LDAP client configuration earlier: - -- `bind_dn`: The access credentials username -- `password`: The access credentials password -- `cert`: The `.crt` file text from the downloaded certificate bundle -- `key`: The `.key` file text from the downloaded certificate bundle - -**For Omnibus installations** - -1. Edit `/etc/gitlab/gitlab.rb`: - - ```ruby - gitlab_rails['ldap_enabled'] = true - gitlab_rails['ldap_servers'] = YAML.load <<-EOS # remember to close this block with 'EOS' below - main: # 'main' is the GitLab 'provider ID' of this LDAP server - label: 'Google Secure LDAP' - - host: 'ldap.google.com' - port: 636 - uid: 'uid' - bind_dn: 'DizzyHorse' - password: 'd6V5H8nhMUW9AuDP25abXeLd' - encryption: 'simple_tls' - verify_certificates: true - - tls_options: - cert: | - -----BEGIN CERTIFICATE----- - MIIDbDCCAlSgAwIBAgIGAWlzxiIfMA0GCSqGSIb3DQEBCwUAMHcxFDASBgNVBAoTC0dvb2dsZSBJ - bmMuMRYwFAYDVQQHEw1Nb3VudGFpbiBWaWV3MRQwEgYDVQQDEwtMREFQIENsaWVudDEPMA0GA1UE - CxMGR1N1aXRlMQswCQYDVQQGEwJVUzETMBEGA1UECBMKQ2FsaWZvcm5pYTAeFw0xOTAzMTIyMTE5 - MThaFw0yMjAzMTEyMTE5MThaMHcxFDASBgNVBAoTC0dvb2dsZSBJbmMuMRYwFAYDVQQHEw1Nb3Vu - dGFpbiBWaWV3MRQwEgYDVQQDEwtMREFQIENsaWVudDEPMA0GA1UECxMGR1N1aXRlMQswCQYDVQQG - EwJVUzETMBEGA1UECBMKQ2FsaWZvcm5pYTCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEB - ALOTy4aC38dyjESk6N8fRsKk8DN23ZX/GaNFL5OUmmA1KWzrvVC881OzNdtGm3vNOIxr9clteEG/ - tQwsmsJvQT5U+GkBt+tGKF/zm7zueHUYqTP7Pg5pxAnAei90qkIRFi17ulObyRHPYv1BbCt8pxNB - 4fG/gAXkFbCNxwh1eiQXXRTfruasCZ4/mHfX7MVm8JmWU9uAVIOLW+DSWOFhrDQduJdGBXJOyC2r - Gqoeg9+tkBmNH/jjxpnEkFW8q7io9DdOUqqNgoidA1h9vpKTs3084sy2DOgUvKN9uXWx14uxIyYU - Y1DnDy0wczcsuRt7l+EgtCEgpsLiLJQbKW+JS1UCAwEAATANBgkqhkiG9w0BAQsFAAOCAQEAf60J - yazhbHkDKIH2gFxfm7QLhhnqsmafvl4WP7JqZt0u0KdnvbDPfokdkM87yfbKJU1MTI86M36wEC+1 - P6bzklKz7kXbzAD4GggksAzxsEE64OWHC+Y64Tkxq2NiZTw/76POkcg9StiIXjG0ZcebHub9+Ux/ - rTncip92nDuvgEM7lbPFKRIS/YMhLCk09B/U0F6XLsf1yYjyf5miUTDikPkov23b/YGfpc8kh6hq - 1kqdi6a1cYPP34eAhtRhMqcZU9qezpJF6s9EeN/3YFfKzLODFSsVToBRAdZgGHzj//SAtLyQTD4n - KCSvK1UmaMxNaZyTHg8JnMf0ZuRpv26iSg== - -----END CERTIFICATE----- - - key: | - -----BEGIN PRIVATE KEY----- - MIIEvQIBADANBgkqhkiG9w0BAQEFAASCBKcwggSjAgEAAoIBAQCzk8uGgt/HcoxEpOjfH0bCpPAz - dt2V/xmjRS+TlJpgNSls671QvPNTszXbRpt7zTiMa/XJbXhBv7UMLJrCb0E+VPhpAbfrRihf85u8 - 7nh1GKkz+z4OacQJwHovdKpCERYte7pTm8kRz2L9QWwrfKcTQeHxv4AF5BWwjccIdXokF10U367m - rAmeP5h31+zFZvCZllPbgFSDi1vg0ljhYaw0HbiXRgVyTsgtqxqqHoPfrZAZjR/448aZxJBVvKu4 - qPQ3TlKqjYKInQNYfb6Sk7N9POLMtgzoFLyjfbl1sdeLsSMmFGNQ5w8tMHM3LLkbe5fhILQhIKbC - 4iyUGylviUtVAgMBAAECggEAIPb0CQy0RJoX+q/lGbRVmnyJpYDf+115WNnl+mrwjdGkeZyqw4v0 - BPzkWYzUFP1esJRO6buBNFybQRFdFW0z5lvVv/zzRKq71aVUBPInxaMRyHuJ8D5lIL8nDtgVOwyE - 7DOGyDtURUMzMjdUwoTe7K+O6QBU4X/1pVPZYgmissYSMmt68LiP8k0p601F4+r5xOi/QEy44aVp - aOJZBUOisKB8BmUXZqmQ4Cy05vU9Xi1rLyzkn9s7fxnZ+JO6Sd1r0Thm1mE0yuPgxkDBh/b4f3/2 - GsQNKKKCiij/6TfkjnBi8ZvWR44LnKpu760g/K7psVNrKwqJG6C/8RAcgISWQQKBgQDop7BaKGhK - 1QMJJ/vnlyYFTucfGLn6bM//pzTys5Gop0tpcfX/Hf6a6Dd+zBhmC3tBmhr80XOX/PiyAIbc0lOI - 31rafZuD/oVx5mlIySWX35EqS14LXmdVs/5vOhsInNgNiE+EPFf1L9YZgG/zA7OUBmqtTeYIPDVC - 7ViJcydItQKBgQDFmK0H0IA6W4opGQo+zQKhefooqZ+RDk9IIZMPOAtnvOM7y3rSVrfsSjzYVuMS - w/RP/vs7rwhaZejnCZ8/7uIqwg4sdUBRzZYR3PRNFeheW+BPZvb+2keRCGzOs7xkbF1mu54qtYTa - HZGZj1OsD83AoMwVLcdLDgO1kw32dkS8IQKBgFRdgoifAHqqVah7VFB9se7Y1tyi5cXWsXI+Wufr - j9U9nQ4GojK52LqpnH4hWnOelDqMvF6TQTyLIk/B+yWWK26Ft/dk9wDdSdystd8L+dLh4k0Y+Whb - +lLMq2YABw+PeJUnqdYE38xsZVHoDjBsVjFGRmbDybeQxauYT7PACy3FAoGBAK2+k9bdNQMbXp7I - j8OszHVkJdz/WXlY1cmdDAxDwXOUGVKIlxTAf7TbiijILZ5gg0Cb+hj+zR9/oI0WXtr+mAv02jWp - W8cSOLS4TnBBpTLjIpdu+BwbnvYeLF6MmEjNKEufCXKQbaLEgTQ/XNlchBSuzwSIXkbWqdhM1+gx - EjtBAoGARAdMIiDMPWIIZg3nNnFebbmtBP0qiBsYohQZ+6i/8s/vautEHBEN6Q0brIU/goo+nTHc - t9VaOkzjCmAJSLPUanuBC8pdYgLu5J20NXUZLD9AE/2bBT3OpezKcdYeI2jqoc1qlWHlNtVtdqQ2 - AcZSFJQjdg5BTyvdEDhaYUKGdRw= - -----END PRIVATE KEY----- - EOS - ``` - -1. Save the file and [reconfigure](../restart_gitlab.md#omnibus-gitlab-reconfigure) GitLab for the changes to take effect. - ---- - -**For installations from source** - -1. Edit `config/gitlab.yml`: - - ```yaml - ldap: - enabled: true - servers: - main: # 'main' is the GitLab 'provider ID' of this LDAP server - label: 'Google Secure LDAP' - - host: 'ldap.google.com' - port: 636 - uid: 'uid' - bind_dn: 'DizzyHorse' - password: 'd6V5H8nhMUW9AuDP25abXeLd' - encryption: 'simple_tls' - verify_certificates: true - - tls_options: - cert: | - -----BEGIN CERTIFICATE----- - MIIDbDCCAlSgAwIBAgIGAWlzxiIfMA0GCSqGSIb3DQEBCwUAMHcxFDASBgNVBAoTC0dvb2dsZSBJ - bmMuMRYwFAYDVQQHEw1Nb3VudGFpbiBWaWV3MRQwEgYDVQQDEwtMREFQIENsaWVudDEPMA0GA1UE - CxMGR1N1aXRlMQswCQYDVQQGEwJVUzETMBEGA1UECBMKQ2FsaWZvcm5pYTAeFw0xOTAzMTIyMTE5 - MThaFw0yMjAzMTEyMTE5MThaMHcxFDASBgNVBAoTC0dvb2dsZSBJbmMuMRYwFAYDVQQHEw1Nb3Vu - dGFpbiBWaWV3MRQwEgYDVQQDEwtMREFQIENsaWVudDEPMA0GA1UECxMGR1N1aXRlMQswCQYDVQQG - EwJVUzETMBEGA1UECBMKQ2FsaWZvcm5pYTCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEB - ALOTy4aC38dyjESk6N8fRsKk8DN23ZX/GaNFL5OUmmA1KWzrvVC881OzNdtGm3vNOIxr9clteEG/ - tQwsmsJvQT5U+GkBt+tGKF/zm7zueHUYqTP7Pg5pxAnAei90qkIRFi17ulObyRHPYv1BbCt8pxNB - 4fG/gAXkFbCNxwh1eiQXXRTfruasCZ4/mHfX7MVm8JmWU9uAVIOLW+DSWOFhrDQduJdGBXJOyC2r - Gqoeg9+tkBmNH/jjxpnEkFW8q7io9DdOUqqNgoidA1h9vpKTs3084sy2DOgUvKN9uXWx14uxIyYU - Y1DnDy0wczcsuRt7l+EgtCEgpsLiLJQbKW+JS1UCAwEAATANBgkqhkiG9w0BAQsFAAOCAQEAf60J - yazhbHkDKIH2gFxfm7QLhhnqsmafvl4WP7JqZt0u0KdnvbDPfokdkM87yfbKJU1MTI86M36wEC+1 - P6bzklKz7kXbzAD4GggksAzxsEE64OWHC+Y64Tkxq2NiZTw/76POkcg9StiIXjG0ZcebHub9+Ux/ - rTncip92nDuvgEM7lbPFKRIS/YMhLCk09B/U0F6XLsf1yYjyf5miUTDikPkov23b/YGfpc8kh6hq - 1kqdi6a1cYPP34eAhtRhMqcZU9qezpJF6s9EeN/3YFfKzLODFSsVToBRAdZgGHzj//SAtLyQTD4n - KCSvK1UmaMxNaZyTHg8JnMf0ZuRpv26iSg== - -----END CERTIFICATE----- - - key: | - -----BEGIN PRIVATE KEY----- - MIIEvQIBADANBgkqhkiG9w0BAQEFAASCBKcwggSjAgEAAoIBAQCzk8uGgt/HcoxEpOjfH0bCpPAz - dt2V/xmjRS+TlJpgNSls671QvPNTszXbRpt7zTiMa/XJbXhBv7UMLJrCb0E+VPhpAbfrRihf85u8 - 7nh1GKkz+z4OacQJwHovdKpCERYte7pTm8kRz2L9QWwrfKcTQeHxv4AF5BWwjccIdXokF10U367m - rAmeP5h31+zFZvCZllPbgFSDi1vg0ljhYaw0HbiXRgVyTsgtqxqqHoPfrZAZjR/448aZxJBVvKu4 - qPQ3TlKqjYKInQNYfb6Sk7N9POLMtgzoFLyjfbl1sdeLsSMmFGNQ5w8tMHM3LLkbe5fhILQhIKbC - 4iyUGylviUtVAgMBAAECggEAIPb0CQy0RJoX+q/lGbRVmnyJpYDf+115WNnl+mrwjdGkeZyqw4v0 - BPzkWYzUFP1esJRO6buBNFybQRFdFW0z5lvVv/zzRKq71aVUBPInxaMRyHuJ8D5lIL8nDtgVOwyE - 7DOGyDtURUMzMjdUwoTe7K+O6QBU4X/1pVPZYgmissYSMmt68LiP8k0p601F4+r5xOi/QEy44aVp - aOJZBUOisKB8BmUXZqmQ4Cy05vU9Xi1rLyzkn9s7fxnZ+JO6Sd1r0Thm1mE0yuPgxkDBh/b4f3/2 - GsQNKKKCiij/6TfkjnBi8ZvWR44LnKpu760g/K7psVNrKwqJG6C/8RAcgISWQQKBgQDop7BaKGhK - 1QMJJ/vnlyYFTucfGLn6bM//pzTys5Gop0tpcfX/Hf6a6Dd+zBhmC3tBmhr80XOX/PiyAIbc0lOI - 31rafZuD/oVx5mlIySWX35EqS14LXmdVs/5vOhsInNgNiE+EPFf1L9YZgG/zA7OUBmqtTeYIPDVC - 7ViJcydItQKBgQDFmK0H0IA6W4opGQo+zQKhefooqZ+RDk9IIZMPOAtnvOM7y3rSVrfsSjzYVuMS - w/RP/vs7rwhaZejnCZ8/7uIqwg4sdUBRzZYR3PRNFeheW+BPZvb+2keRCGzOs7xkbF1mu54qtYTa - HZGZj1OsD83AoMwVLcdLDgO1kw32dkS8IQKBgFRdgoifAHqqVah7VFB9se7Y1tyi5cXWsXI+Wufr - j9U9nQ4GojK52LqpnH4hWnOelDqMvF6TQTyLIk/B+yWWK26Ft/dk9wDdSdystd8L+dLh4k0Y+Whb - +lLMq2YABw+PeJUnqdYE38xsZVHoDjBsVjFGRmbDybeQxauYT7PACy3FAoGBAK2+k9bdNQMbXp7I - j8OszHVkJdz/WXlY1cmdDAxDwXOUGVKIlxTAf7TbiijILZ5gg0Cb+hj+zR9/oI0WXtr+mAv02jWp - W8cSOLS4TnBBpTLjIpdu+BwbnvYeLF6MmEjNKEufCXKQbaLEgTQ/XNlchBSuzwSIXkbWqdhM1+gx - EjtBAoGARAdMIiDMPWIIZg3nNnFebbmtBP0qiBsYohQZ+6i/8s/vautEHBEN6Q0brIU/goo+nTHc - t9VaOkzjCmAJSLPUanuBC8pdYgLu5J20NXUZLD9AE/2bBT3OpezKcdYeI2jqoc1qlWHlNtVtdqQ2 - AcZSFJQjdg5BTyvdEDhaYUKGdRw= - -----END PRIVATE KEY----- - ``` - -1. Save the file and [restart](../restart_gitlab.md#installations-from-source) GitLab for the changes to take effect. - -<!-- ## Troubleshooting - -Include any troubleshooting steps that you can foresee. If you know beforehand what issues -one might have when setting this up, or when something is changed, or on upgrading, it's -important to describe those, too. Think of things that may go wrong and include them here. -This is important to minimize requests for support, and to avoid doc comments with -questions that you know someone might ask. - -Each scenario can be a third-level heading, e.g. `### Getting error message X`. -If you have none to add when creating a doc, leave this section in place -but commented out to help encourage others to add to it in the future. --> +This document was moved to [another location](ldap/google_secure_ldap.md). diff --git a/doc/administration/auth/how_to_configure_ldap_gitlab_ce/img/gitlab_ou.png b/doc/administration/auth/how_to_configure_ldap_gitlab_ce/img/gitlab_ou.png Binary files differdeleted file mode 100644 index 223fd0ac401..00000000000 --- a/doc/administration/auth/how_to_configure_ldap_gitlab_ce/img/gitlab_ou.png +++ /dev/null diff --git a/doc/administration/auth/how_to_configure_ldap_gitlab_ce/img/ldap_ou.gif b/doc/administration/auth/how_to_configure_ldap_gitlab_ce/img/ldap_ou.gif Binary files differdeleted file mode 100644 index a6727a3d85f..00000000000 --- a/doc/administration/auth/how_to_configure_ldap_gitlab_ce/img/ldap_ou.gif +++ /dev/null diff --git a/doc/administration/auth/how_to_configure_ldap_gitlab_ce/img/user_auth.gif b/doc/administration/auth/how_to_configure_ldap_gitlab_ce/img/user_auth.gif Binary files differdeleted file mode 100644 index 36e6085259f..00000000000 --- a/doc/administration/auth/how_to_configure_ldap_gitlab_ce/img/user_auth.gif +++ /dev/null diff --git a/doc/administration/auth/how_to_configure_ldap_gitlab_ce/index.md b/doc/administration/auth/how_to_configure_ldap_gitlab_ce/index.md index 01da4528eab..40d021e180c 100644 --- a/doc/administration/auth/how_to_configure_ldap_gitlab_ce/index.md +++ b/doc/administration/auth/how_to_configure_ldap_gitlab_ce/index.md @@ -1,273 +1,5 @@ --- -type: howto +redirect_to: '../ldap/index.md' --- -# How to configure LDAP with GitLab CE - -Managing a large number of users in GitLab can become a burden for system administrators. As an organization grows so do user accounts. Keeping these user accounts in sync across multiple enterprise applications often becomes a time consuming task. - -In this guide we will focus on configuring GitLab with Active Directory. [Active Directory](https://en.wikipedia.org/wiki/Active_Directory) is a popular LDAP compatible directory service provided by Microsoft, included in all modern Windows Server operating systems. - -GitLab has supported LDAP integration since [version 2.2](https://about.gitlab.com/releases/2012/02/22/gitlab-version-2-2/). With GitLab LDAP [group syncing](../how_to_configure_ldap_gitlab_ee/index.md#group-sync) being added to GitLab Enterprise Edition in [version 6.0](https://about.gitlab.com/releases/2013/08/20/gitlab-6-dot-0-released/). LDAP integration has become one of the most popular features in GitLab. - -## Getting started - -### Choosing an LDAP Server - -The main reason organizations choose to utilize a LDAP server is to keep the entire organization's user base consolidated into a central repository. Users can access multiple applications and systems across the IT environment using a single login. Because LDAP is an open, vendor-neutral, industry standard application protocol, the number of applications using LDAP authentication continues to increase. - -There are many commercial and open source [directory servers](https://en.wikipedia.org/wiki/Directory_service#LDAP_implementations) that support the LDAP protocol. Deciding on the right directory server highly depends on the existing IT environment in which the server will be integrated with. - -For example, [Active Directory](https://docs.microsoft.com/en-us/previous-versions/windows/it-pro/windows-server-2012-R2-and-2012/hh831484(v=ws.11)) is generally favored in a primarily Windows environment, as this allows quick integration with existing services. Other popular directory services include: - -- [Oracle Internet Directory](https://www.oracle.com/middleware/technologies/internet-directory.html) -- [OpenLDAP](https://www.openldap.org/) -- [389 Directory](http://directory.fedoraproject.org/) -- [OpenDJ (Renamed to Forgerock Directory Services)](https://www.forgerock.com/platform/directory-services) -- [ApacheDS](https://directory.apache.org/) - -> GitLab uses the [Net::LDAP](https://rubygems.org/gems/net-ldap) library under the hood. This means it supports all [IETF](https://tools.ietf.org/html/rfc2251) compliant LDAPv3 servers. - -### Active Directory (AD) - -We won't cover the installation and configuration of Windows Server or Active Directory Domain Services in this tutorial. There are a number of resources online to guide you through this process: - -- Install Windows Server 2012 - (`technet.microsoft.com`) - [Installing Windows Server 2012](https://docs.microsoft.com/en-us/previous-versions/windows/it-pro/windows-server-2012-R2-and-2012/jj134246(v=ws.11)) - -- Install Active Directory Domain Services (AD DS) (`technet.microsoft.com`) - [Install Active Directory Domain Services](https://docs.microsoft.com/en-us/windows-server/identity/ad-ds/deploy/install-active-directory-domain-services--level-100-#BKMK_PS) - -> **Shortcut:** You can quickly install AD DS via PowerShell using -`Install-WindowsFeature AD-Domain-Services -IncludeManagementTools` - -### Creating an AD **OU** structure - -Configuring organizational units (**OU**s) is an important part of setting up Active Directory. **OU**s form the base for an entire organizational structure. Using GitLab as an example we have designed the **OU** structure below using the geographic **OU** model. In the Geographic Model we separate **OU**s for different geographic regions. - -| GitLab **OU** Design | GitLab AD Structure | -| :----------------------------: | :------------------------------: | -|  |  | - -Using PowerShell you can output the **OU** structure as a table (_all names are examples only_): - -```ps -Get-ADObject -LDAPFilter "(objectClass=*)" -SearchBase 'OU=GitLab INT,DC=GitLab,DC=org' -Properties CanonicalName | Format-Table Name,CanonicalName -A -``` - -```plaintext -OU CanonicalName ----- ------------- -GitLab INT GitLab.org/GitLab INT -United States GitLab.org/GitLab INT/United States -Developers GitLab.org/GitLab INT/United States/Developers -Gary Johnson GitLab.org/GitLab INT/United States/Developers/Gary Johnson -Ellis Matthews GitLab.org/GitLab INT/United States/Developers/Ellis Matthews -William Collins GitLab.org/GitLab INT/United States/Developers/William Collins -People Ops GitLab.org/GitLab INT/United States/People Ops -Margaret Baker GitLab.org/GitLab INT/United States/People Ops/Margaret Baker -Libby Hartzler GitLab.org/GitLab INT/United States/People Ops/Libby Hartzler -Victoria Ryles GitLab.org/GitLab INT/United States/People Ops/Victoria Ryles -The Netherlands GitLab.org/GitLab INT/The Netherlands -Developers GitLab.org/GitLab INT/The Netherlands/Developers -John Doe GitLab.org/GitLab INT/The Netherlands/Developers/John Doe -Jon Mealy GitLab.org/GitLab INT/The Netherlands/Developers/Jon Mealy -Jane Weingarten GitLab.org/GitLab INT/The Netherlands/Developers/Jane Weingarten -Production GitLab.org/GitLab INT/The Netherlands/Production -Sarah Konopka GitLab.org/GitLab INT/The Netherlands/Production/Sarah Konopka -Cynthia Bruno GitLab.org/GitLab INT/The Netherlands/Production/Cynthia Bruno -David George GitLab.org/GitLab INT/The Netherlands/Production/David George -United Kingdom GitLab.org/GitLab INT/United Kingdom -Developers GitLab.org/GitLab INT/United Kingdom/Developers -Leroy Fox GitLab.org/GitLab INT/United Kingdom/Developers/Leroy Fox -Christopher Alley GitLab.org/GitLab INT/United Kingdom/Developers/Christopher Alley -Norris Morita GitLab.org/GitLab INT/United Kingdom/Developers/Norris Morita -Support GitLab.org/GitLab INT/United Kingdom/Support -Laura Stanley GitLab.org/GitLab INT/United Kingdom/Support/Laura Stanley -Nikki Schuman GitLab.org/GitLab INT/United Kingdom/Support/Nikki Schuman -Harriet Butcher GitLab.org/GitLab INT/United Kingdom/Support/Harriet Butcher -Global Groups GitLab.org/GitLab INT/Global Groups -DevelopersNL GitLab.org/GitLab INT/Global Groups/DevelopersNL -DevelopersUK GitLab.org/GitLab INT/Global Groups/DevelopersUK -DevelopersUS GitLab.org/GitLab INT/Global Groups/DevelopersUS -ProductionNL GitLab.org/GitLab INT/Global Groups/ProductionNL -SupportUK GitLab.org/GitLab INT/Global Groups/SupportUK -People Ops US GitLab.org/GitLab INT/Global Groups/People Ops US -Global Admins GitLab.org/GitLab INT/Global Groups/Global Admins -``` - -> See [more information](https://docs.microsoft.com/en-us/previous-versions/windows/it-pro/windows-powershell-1.0/ff730967(v=technet.10)) on searching Active Directory with Windows PowerShell from [The Scripting Guys](https://devblogs.microsoft.com/scripting/) - -## GitLab LDAP configuration - -The initial configuration of LDAP in GitLab requires changes to the `gitlab.rb` configuration file (`/etc/gitlab/gitlab.rb`). Below is an example of a complete configuration using an Active Directory. - -The two Active Directory specific values are `active_directory: true` and `uid: 'sAMAccountName'`. `sAMAccountName` is an attribute returned by Active Directory used for GitLab usernames. See the example output from `ldapsearch` for a full list of attributes a "person" object (user) has in **AD** - [`ldapsearch` example](#using-ldapsearch-unix) - -> Both group_base and admin_group configuration options are only available in GitLab Enterprise Edition. See [GitLab EE - LDAP Features](../how_to_configure_ldap_gitlab_ee/index.md#gitlab-enterprise-edition---ldap-features) **(STARTER ONLY)** - -### Example `gitlab.rb` LDAP - -```ruby -gitlab_rails['ldap_enabled'] = true -gitlab_rails['ldap_servers'] = { -'main' => { - 'label' => 'GitLab AD', - 'host' => 'ad.example.org', - 'port' => 636, - 'uid' => 'sAMAccountName', - 'encryption' => 'simple_tls', - 'verify_certificates' => true, - 'bind_dn' => 'CN=GitLabSRV,CN=Users,DC=GitLab,DC=org', - 'password' => 'Password1', - 'active_directory' => true, - 'base' => 'OU=GitLab INT,DC=GitLab,DC=org', - 'group_base' => 'OU=Global Groups,OU=GitLab INT,DC=GitLab,DC=org', - 'admin_group' => 'Global Admins' - } -} -``` - -> **Note:** Remember to run `gitlab-ctl reconfigure` after modifying `gitlab.rb` - -## Security improvements (LDAPS) - -Security is an important aspect when deploying an LDAP server. By default, LDAP traffic is transmitted unsecured. LDAP can be secured using SSL/TLS called LDAPS, or commonly "LDAP over SSL". - -Securing LDAP (enabling LDAPS) on Windows Server 2012 involves installing a valid SSL certificate. For full details see Microsoft's guide [How to enable LDAP over SSL with a third-party certification authority](https://support.microsoft.com/en-us/help/321051/how-to-enable-ldap-over-ssl-with-a-third-party-certification-authority) - -> By default a LDAP service listens for connections on TCP and UDP port 389. LDAPS (LDAP over SSL) listens on port 636 - -### Testing your AD server - -#### Using **AdFind** (Windows) - -You can use the [`AdFind`](https://social.technet.microsoft.com/wiki/contents/articles/7535.adfind-command-examples.aspx) utility (on Windows based systems) to test that your LDAP server is accessible and authentication is working correctly. This is a freeware utility built by [Joe Richards](http://www.joeware.net/freetools/tools/adfind/index.htm). - -**Return all objects** - -You can use the filter `objectclass=*` to return all directory objects. - -```shell -adfind -h ad.example.org:636 -ssl -u "CN=GitLabSRV,CN=Users,DC=GitLab,DC=org" -up Password1 -b "OU=GitLab INT,DC=GitLab,DC=org" -f (objectClass=*) -``` - -**Return single object using filter** - -You can also retrieve a single object by **specifying** the object name or full **DN**. In this example we specify the object name only `CN=Leroy Fox`. - -```shell -adfind -h ad.example.org:636 -ssl -u "CN=GitLabSRV,CN=Users,DC=GitLab,DC=org" -up Password1 -b "OU=GitLab INT,DC=GitLab,DC=org" -f (&(objectcategory=person)(CN=Leroy Fox))” -``` - -#### Using **ldapsearch** (Unix) - -You can use the `ldapsearch` utility (on Unix based systems) to test that your LDAP server is accessible and authentication is working correctly. This utility is included in the [`ldap-utils`](https://wiki.debian.org/LDAP/LDAPUtils) package. - -**Return all objects** - -You can use the filter `objectclass=*` to return all directory objects. - -```shell -ldapsearch -D "CN=GitLabSRV,CN=Users,DC=GitLab,DC=org" \ --w Password1 -p 636 -h ad.example.org \ --b "OU=GitLab INT,DC=GitLab,DC=org" -Z \ --s sub "(objectclass=*)" -``` - -**Return single object using filter** - -You can also retrieve a single object by **specifying** the object name or full **DN**. In this example we specify the object name only `CN=Leroy Fox`. - -```shell -ldapsearch -D "CN=GitLabSRV,CN=Users,DC=GitLab,DC=org" -w Password1 -p 389 -h ad.example.org -b "OU=GitLab INT,DC=GitLab,DC=org" -Z -s sub "CN=Leroy Fox" -``` - -**Full output of `ldapsearch` command:** - Filtering for _CN=Leroy Fox_ - -```plaintext -# LDAPv3 -# base <OU=GitLab INT,DC=GitLab,DC=org> with scope subtree -# filter: CN=Leroy Fox -# requesting: ALL -# - -# Leroy Fox, Developers, United Kingdom, GitLab INT, GitLab.org -dn: CN=Leroy Fox,OU=Developers,OU=United Kingdom,OU=GitLab INT,DC=GitLab,DC=or - g -objectClass: top -objectClass: person -objectClass: organizationalPerson -objectClass: user -cn: Leroy Fox -sn: Fox -givenName: Leroy -distinguishedName: CN=Leroy Fox,OU=Developers,OU=United Kingdom,OU=GitLab INT, - DC=GitLab,DC=org -instanceType: 4 -whenCreated: 20170210030500.0Z -whenChanged: 20170213050128.0Z -displayName: Leroy Fox -uSNCreated: 16790 -memberOf: CN=DevelopersUK,OU=Global Groups,OU=GitLab INT,DC=GitLab,DC=org -uSNChanged: 20812 -name: Leroy Fox -objectGUID:: rBCAo6NR6E6vfSKgzcUILg== -userAccountControl: 512 -badPwdCount: 0 -codePage: 0 -countryCode: 0 -badPasswordTime: 0 -lastLogoff: 0 -lastLogon: 0 -pwdLastSet: 131311695009850084 -primaryGroupID: 513 -objectSid:: AQUAAAAAAAUVAAAA9GMAb7tdJZvsATf7ZwQAAA== -accountExpires: 9223372036854775807 -logonCount: 0 -sAMAccountName: Leroyf -sAMAccountType: 805306368 -userPrincipalName: Leroyf@GitLab.org -objectCategory: CN=Person,CN=Schema,CN=Configuration,DC=GitLab,DC=org -dSCorePropagationData: 16010101000000.0Z -lastLogonTimestamp: 131314356887754250 - -# search result -search: 2 -result: 0 Success - -# numResponses: 2 -# numEntries: 1 -``` - -## Basic user authentication - -After configuring LDAP, basic authentication will be available. Users can then login using their directory credentials. An extra tab is added to the GitLab login screen for the configured LDAP server (e.g "**GitLab AD**"). - - - -Users that are removed from the LDAP base group (e.g `OU=GitLab INT,DC=GitLab,DC=org`) will be **blocked** in GitLab. [More information](../ldap.md#security) on LDAP security. - -If `allow_username_or_email_login` is enabled in the LDAP configuration, GitLab will ignore everything after the first '@' in the LDAP username used on login. Example: The username `jon.doe@example.com` is converted to `jon.doe` when authenticating with the LDAP server. Disable this setting if you use `userPrincipalName` as the `uid`. - -## LDAP extended features on GitLab EE - -With [GitLab Enterprise Edition (EE)](https://about.gitlab.com/pricing/), besides everything we just described, you'll -have extended functionalities with LDAP, such as: - -- Group sync -- Group permissions -- Updating user permissions -- Multiple LDAP servers - -Read through the article on [LDAP for GitLab EE](../how_to_configure_ldap_gitlab_ee/index.md) **(STARTER ONLY)** for an overview. - -<!-- ## Troubleshooting - -Include any troubleshooting steps that you can foresee. If you know beforehand what issues -one might have when setting this up, or when something is changed, or on upgrading, it's -important to describe those, too. Think of things that may go wrong and include them here. -This is important to minimize requests for support, and to avoid doc comments with -questions that you know someone might ask. - -Each scenario can be a third-level heading, e.g. `### Getting error message X`. -If you have none to add when creating a doc, leave this section in place -but commented out to help encourage others to add to it in the future. --> +This document was moved to [another location](../ldap/index.md). diff --git a/doc/administration/auth/how_to_configure_ldap_gitlab_ee/img/admin_group.png b/doc/administration/auth/how_to_configure_ldap_gitlab_ee/img/admin_group.png Binary files differdeleted file mode 100644 index 9896379d669..00000000000 --- a/doc/administration/auth/how_to_configure_ldap_gitlab_ee/img/admin_group.png +++ /dev/null diff --git a/doc/administration/auth/how_to_configure_ldap_gitlab_ee/img/group_link_final.png b/doc/administration/auth/how_to_configure_ldap_gitlab_ee/img/group_link_final.png Binary files differdeleted file mode 100644 index 21fb5a7d0ce..00000000000 --- a/doc/administration/auth/how_to_configure_ldap_gitlab_ee/img/group_link_final.png +++ /dev/null diff --git a/doc/administration/auth/how_to_configure_ldap_gitlab_ee/img/group_linking.gif b/doc/administration/auth/how_to_configure_ldap_gitlab_ee/img/group_linking.gif Binary files differdeleted file mode 100644 index a0ec2d4f10a..00000000000 --- a/doc/administration/auth/how_to_configure_ldap_gitlab_ee/img/group_linking.gif +++ /dev/null diff --git a/doc/administration/auth/how_to_configure_ldap_gitlab_ee/img/manual_permissions.gif b/doc/administration/auth/how_to_configure_ldap_gitlab_ee/img/manual_permissions.gif Binary files differdeleted file mode 100644 index a9d5dd7e73e..00000000000 --- a/doc/administration/auth/how_to_configure_ldap_gitlab_ee/img/manual_permissions.gif +++ /dev/null diff --git a/doc/administration/auth/how_to_configure_ldap_gitlab_ee/img/multi_login.gif b/doc/administration/auth/how_to_configure_ldap_gitlab_ee/img/multi_login.gif Binary files differdeleted file mode 100644 index d317add9837..00000000000 --- a/doc/administration/auth/how_to_configure_ldap_gitlab_ee/img/multi_login.gif +++ /dev/null diff --git a/doc/administration/auth/how_to_configure_ldap_gitlab_ee/index.md b/doc/administration/auth/how_to_configure_ldap_gitlab_ee/index.md index d7b32c8a92a..40d021e180c 100644 --- a/doc/administration/auth/how_to_configure_ldap_gitlab_ee/index.md +++ b/doc/administration/auth/how_to_configure_ldap_gitlab_ee/index.md @@ -1,125 +1,5 @@ --- -type: howto +redirect_to: '../ldap/index.md' --- -# How to configure LDAP with GitLab EE **(STARTER ONLY)** - -This article expands on [How to Configure LDAP with GitLab CE](../how_to_configure_ldap_gitlab_ce/index.md). Make sure to read through it before moving forward. - -## GitLab Enterprise Edition - LDAP features - -[GitLab Enterprise Edition (EE)](https://about.gitlab.com/pricing/) has several advantages when it comes to integrating with Active Directory (LDAP): - -- [Administrator Sync](../ldap-ee.md#administrator-sync): As an extension of group sync, you can automatically manage your global GitLab administrators. Specify a group CN for `admin_group` and all members of the LDAP group will be given administrator privileges. -- [Group Sync](#group-sync): This allows GitLab group membership to be automatically updated based on LDAP group members. -- [Multiple LDAP servers](#multiple-ldap-servers): The ability to configure multiple LDAP servers. This is useful if an organization has different LDAP servers within departments. This is not designed for failover. We're working on [supporting LDAP failover](https://gitlab.com/gitlab-org/gitlab/issues/139) in GitLab. - -- Daily user synchronization: Once a day, GitLab will run a synchronization to check and update GitLab users against LDAP. This process updates all user details automatically. - -In the following section, you'll find a description of each of these features. Read through [LDAP GitLab EE docs](../ldap-ee.md) for complementary information. - - - -All members of the group `Global Admins` will be given **administrator** access to GitLab, allowing them to view the `/admin` dashboard. - -### Group Sync - -Group syncing allows AD (LDAP) groups to be mapped to GitLab groups. This provides more control over per-group user management. To configure group syncing edit the `group_base` **DN** (`'OU=Global Groups,OU=GitLab INT,DC=GitLab,DC=org'`). This **OU** contains all groups that will be associated with [GitLab groups](../../../user/group/index.md). - -#### Creating group links - example - -As an example, let's suppose we have a "UKGov" GitLab group, which deals with confidential government information. Therefore, users of this group must be given the correct permissions to projects contained within the group. Granular group permissions can be applied based on the AD group. - -**UK Developers** of our "UKGov" group are given **"developer"** permissions. - -_The developer permission allows the development staff to effectively manage all project code, issues, and merge requests._ - -**UK Support** staff of our "UKGov" group are given **"reporter"** permissions. - -_The reporter permission allows support staff to manage issues, labels, and review project code._ - -**US People Ops** of our "UKGov" group are given **"guest"** permissions. - - - -> Guest permissions allows people ops staff to review and lodge new issues while allowing no read or write access to project code or [confidential issues](../../../user/project/issues/confidential_issues.md#permissions-and-access-to-confidential-issues) created by other users. - -See the [permission list](../../../user/permissions.md) for complementary information. - -#### Group permissions - example - -Considering the previous example, our staff will have -access to our GitLab instance with the following structure: - - - -Using this permission structure in our example allows only UK staff access to sensitive information stored in the projects code, while still allowing other teams to work effectively. As all permissions are controlled via AD groups new users can be quickly added to existing groups. New group members will then automatically inherit the required permissions. - -> [More information](../ldap-ee.md#group-sync) on group syncing. - -### Updating user permissions - new feature - -Since GitLab [v8.15](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/822) LDAP user permissions can now be manually overridden by an admin user. To override a user's permissions visit the groups **Members** page and select **Edit permissions**. - - - -### Multiple LDAP servers - -GitLab EE can support multiple LDAP servers. Simply configure another server in the `gitlab.rb` file within the `ldap_servers` block. In the example below we configure a new secondary server with the label **GitLab Secondary AD**. This is shown on the GitLab login screen. Large enterprises often utilize multiple LDAP servers for segregating organizational departments. - - - -Considering the example illustrated on the image above, -our `gitlab.rb` configuration would look like: - -```ruby -gitlab_rails['ldap_enabled'] = true -gitlab_rails['ldap_servers'] = { -'main' => { - 'label' => 'GitLab AD', - 'host' => 'ad.example.org', - 'port' => 636, - 'uid' => 'sAMAccountName', - 'method' => 'ssl', - 'bind_dn' => 'CN=GitLabSRV,CN=Users,DC=GitLab,DC=org', - 'password' => 'Password1', - 'active_directory' => true, - 'base' => 'OU=GitLab INT,DC=GitLab,DC=org', - 'group_base' => 'OU=Global Groups,OU=GitLab INT,DC=GitLab,DC=org', - 'admin_group' => 'Global Admins' - }, - -'secondary' => { - 'label' => 'GitLab Secondary AD', - 'host' => 'ad-secondary.example.net', - 'port' => 636, - 'uid' => 'sAMAccountName', - 'method' => 'ssl', - 'bind_dn' => 'CN=GitLabSRV,CN=Users,DC=GitLab,DC=com', - 'password' => 'Password1', - 'active_directory' => true, - 'base' => 'OU=GitLab Secondary,DC=GitLab,DC=com', - 'group_base' => 'OU=Global Groups,OU=GitLab INT,DC=GitLab,DC=com', - 'admin_group' => 'Global Admins' - } -} -``` - -## Conclusion - -Integration of GitLab with Active Directory (LDAP) reduces the complexity of user management. -It has the advantage of improving user permission controls, while easing the deployment of GitLab into an existing [IT environment](https://www.techopedia.com/definition/29199/it-infrastructure). GitLab EE offers advanced group management and multiple LDAP servers. - -With the assistance of the [GitLab Support](https://about.gitlab.com/support/) team, setting up GitLab with an existing AD/LDAP solution will be a smooth and painless process. - -<!-- ## Troubleshooting - -Include any troubleshooting steps that you can foresee. If you know beforehand what issues -one might have when setting this up, or when something is changed, or on upgrading, it's -important to describe those, too. Think of things that may go wrong and include them here. -This is important to minimize requests for support, and to avoid doc comments with -questions that you know someone might ask. - -Each scenario can be a third-level heading, e.g. `### Getting error message X`. -If you have none to add when creating a doc, leave this section in place -but commented out to help encourage others to add to it in the future. --> +This document was moved to [another location](../ldap/index.md). diff --git a/doc/administration/auth/jwt.md b/doc/administration/auth/jwt.md index 5a773485842..29b192a4845 100644 --- a/doc/administration/auth/jwt.md +++ b/doc/administration/auth/jwt.md @@ -1,5 +1,8 @@ --- type: reference +stage: Manage +group: Access +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers --- # JWT OmniAuth provider diff --git a/doc/administration/auth/ldap-ee.md b/doc/administration/auth/ldap-ee.md index 9e193dba08c..f5565628af1 100644 --- a/doc/administration/auth/ldap-ee.md +++ b/doc/administration/auth/ldap-ee.md @@ -1,417 +1,5 @@ --- -type: reference +redirect_to: 'ldap/index.md' --- -# LDAP Additions in GitLab EE **(STARTER ONLY)** - -This section documents LDAP features specific to GitLab Enterprise Edition -[Starter](https://about.gitlab.com/pricing/#self-managed) and above. - -For documentation relevant to both Community Edition and Enterprise Edition, -see the main [LDAP documentation](ldap.md). - -NOTE: **Note:** -[Microsoft Active Directory Trusts](https://docs.microsoft.com/en-us/previous-versions/windows/it-pro/windows-server-2008-R2-and-2008/cc771568(v=ws.10)) are not supported - -## Use cases - -- User sync: Once a day, GitLab will update users against LDAP. -- Group sync: Once an hour, GitLab will update group membership - based on LDAP group members. - -## Multiple LDAP servers - -With GitLab Enterprise Edition Starter, you can configure multiple LDAP servers -that your GitLab instance will connect to. - -To add another LDAP server: - -1. Duplicating the settings under [the main configuration](ldap.md#configuration). -1. Edit them to match the additional LDAP server. - -Be sure to choose a different provider ID made of letters a-z and numbers 0-9. -This ID will be stored in the database so that GitLab can remember which LDAP -server a user belongs to. - -## User sync - -Once per day, GitLab will run a worker to check and update GitLab -users against LDAP. - -The process will execute the following access checks: - -- Ensure the user is still present in LDAP. -- If the LDAP server is Active Directory, ensure the user is active (not - blocked/disabled state). This will only be checked if - `active_directory: true` is set in the LDAP configuration. - -NOTE: **Note:** -In Active Directory, a user is marked as disabled/blocked if the user -account control attribute (`userAccountControl:1.2.840.113556.1.4.803`) -has bit 2 set. See <https://ctovswild.com/2009/09/03/bitmask-searches-in-ldap/> -for more information. - -The user will be set to `ldap_blocked` state in GitLab if the above conditions -fail. This means the user will not be able to login or push/pull code. - -The process will also update the following user information: - -- Email address. -- If `sync_ssh_keys` is set, SSH public keys. -- If Kerberos is enabled, Kerberos identity. - -NOTE: **Note:** -The LDAP sync process updates existing users while new users will -be created on first sign in. - -## Group Sync - -If your LDAP supports the `memberof` property, when the user signs in for the -first time GitLab will trigger a sync for groups the user should be a member of. -That way they don't need to wait for the hourly sync to be granted -access to their groups and projects. - -A group sync process will run every hour on the hour, and `group_base` must be set -in LDAP configuration for LDAP synchronizations based on group CN to work. This allows -GitLab group membership to be automatically updated based on LDAP group members. - -The `group_base` configuration should be a base LDAP 'container', such as an -'organization' or 'organizational unit', that contains LDAP groups that should -be available to GitLab. For example, `group_base` could be -`ou=groups,dc=example,dc=com`. In the config file it will look like the -following. - -**Omnibus configuration** - -1. Edit `/etc/gitlab/gitlab.rb`: - - ```ruby - gitlab_rails['ldap_servers'] = YAML.load <<-EOS - main: - ## snip... - ## - ## Base where we can search for groups - ## - ## Ex. ou=groups,dc=gitlab,dc=example - ## - ## - group_base: ou=groups,dc=example,dc=com - EOS - ``` - -1. [Apply your changes to GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). - -**Source configuration** - -1. Edit `/home/git/gitlab/config/gitlab.yml`: - - ```yaml - production: - ldap: - servers: - main: - # snip... - group_base: ou=groups,dc=example,dc=com - ``` - -1. [Restart GitLab](../restart_gitlab.md#installations-from-source) for the changes to take effect. - -To take advantage of group sync, group owners or maintainers will need to [create one -or more LDAP group links](#adding-group-links). - -NOTE: **Note:** -If an LDAP user is a group member when LDAP Synchronization is added, and they are not part of the LDAP group, they will be removed from the group. - -### Adding group links - -Once [group sync has been configured](#group-sync) on the instance, one or more LDAP -groups can be linked to a GitLab group to grant their members access to its -contents. - -Group owners or maintainers can add and use LDAP group links by: - -1. Navigating to the group's **Settings > LDAP Synchronization** page. Here, one or more - LDAP groups and [filters](#filters-premium-only) can be linked to this GitLab group, - each one with a configured [permission level](../../user/permissions.md#group-members-permissions) - for its members. -1. Updating the group's membership by navigating to the group's **Settings > Members** - page and clicking **Sync now**. - -### Filters **(PREMIUM ONLY)** - -In GitLab Premium, you can add an LDAP user filter for group synchronization. -Filters allow for complex logic without creating a special LDAP group. - -To sync GitLab group membership based on an LDAP filter: - -1. Open the **LDAP Synchronization** page for the GitLab group. -1. Select **LDAP user filter** as the **Sync method**. -1. Enter an LDAP user filter in the **LDAP user filter** field. - -The filter must comply with the -syntax defined in [RFC 2254](https://tools.ietf.org/search/rfc2254). - -## Administrator sync - -As an extension of group sync, you can automatically manage your global GitLab -administrators. Specify a group CN for `admin_group` and all members of the -LDAP group will be given administrator privileges. The configuration will look -like the following. - -NOTE: **Note:** -Administrators will not be synced unless `group_base` is also -specified alongside `admin_group`. Also, only specify the CN of the admin -group, as opposed to the full DN. - -**Omnibus configuration** - -1. Edit `/etc/gitlab/gitlab.rb`: - - ```ruby - gitlab_rails['ldap_servers'] = YAML.load <<-EOS - main: - ## snip... - ## - ## Base where we can search for groups - ## - ## Ex. ou=groups,dc=gitlab,dc=example - ## - ## - group_base: ou=groups,dc=example,dc=com - - ## - ## The CN of a group containing GitLab administrators - ## - ## Ex. administrators - ## - ## Note: Not `cn=administrators` or the full DN - ## - ## - admin_group: my_admin_group - - EOS - ``` - -1. [Apply your changes to GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). - -**Source configuration** - -1. Edit `/home/git/gitlab/config/gitlab.yml`: - - ```yaml - production: - ldap: - servers: - main: - # snip... - group_base: ou=groups,dc=example,dc=com - admin_group: my_admin_group - ``` - -1. [Restart GitLab](../restart_gitlab.md#installations-from-source) for the changes to take effect. - -## Global group memberships lock - -"Lock memberships to LDAP synchronization" setting allows instance administrators -to lock down user abilities to invite new members to a group. - -When enabled, the following applies: - -- Only administrator can manage memberships of any group including access levels. -- Users are not allowed to share project with other groups or invite members to - a project created in a group. - -## Adjusting LDAP user sync schedule - -> Introduced in GitLab Enterprise Edition Starter. - -NOTE: **Note:** -These are cron formatted values. You can use a crontab generator to create -these values, for example <http://www.crontabgenerator.com/>. - -By default, GitLab will run a worker once per day at 01:30 a.m. server time to -check and update GitLab users against LDAP. - -You can manually configure LDAP user sync times by setting the -following configuration values. The example below shows how to set LDAP user -sync to run once every 12 hours at the top of the hour. - -**Omnibus installations** - -1. Edit `/etc/gitlab/gitlab.rb`: - - ```ruby - gitlab_rails['ldap_sync_worker_cron'] = "0 */12 * * *" - ``` - -1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. - -**Source installations** - -1. Edit `config/gitlab.yaml`: - - ```yaml - cron_jobs: - ldap_sync_worker_cron: - "0 */12 * * *" - ``` - -1. [Restart GitLab](../restart_gitlab.md#installations-from-source) for the changes to take effect. - -## Adjusting LDAP group sync schedule - -NOTE: **Note:** -These are cron formatted values. You can use a crontab generator to create -these values, for example <http://www.crontabgenerator.com/>. - -By default, GitLab will run a group sync process every hour, on the hour. - -CAUTION: **Important:** -It's recommended that you do not run too short intervals as this -could lead to multiple syncs running concurrently. This is primarily a concern -for installations with a large number of LDAP users. Please review the -[LDAP group sync benchmark metrics](#benchmarks) to see how -your installation compares before proceeding. - -You can manually configure LDAP group sync times by setting the -following configuration values. The example below shows how to set group -sync to run once every 2 hours at the top of the hour. - -**Omnibus installations** - -1. Edit `/etc/gitlab/gitlab.rb`: - - ```ruby - gitlab_rails['ldap_group_sync_worker_cron'] = "0 */2 * * * *" - ``` - -1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. - -**Source installations** - -1. Edit `config/gitlab.yaml`: - - ```yaml - cron_jobs: - ldap_group_sync_worker_cron: - "*/30 * * * *" - ``` - -1. [Restart GitLab](../restart_gitlab.md#installations-from-source) for the changes to take effect. - -## External groups - -> Introduced in GitLab Enterprise Edition Starter 8.9. - -Using the `external_groups` setting will allow you to mark all users belonging -to these groups as [external users](../../user/permissions.md#external-users-core-only). -Group membership is checked periodically through the `LdapGroupSync` background -task. - -**Omnibus configuration** - -1. Edit `/etc/gitlab/gitlab.rb`: - - ```ruby - gitlab_rails['ldap_servers'] = YAML.load <<-EOS - main: - ## snip... - ## - ## An array of CNs of groups containing users that should be considered external - ## - ## Ex. ['interns', 'contractors'] - ## - ## Note: Not `cn=interns` or the full DN - ## - external_groups: ['interns', 'contractors'] - EOS - ``` - -1. [Apply your changes to GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). - -**Source configuration** - -1. Edit `config/gitlab.yaml`: - - ```yaml - production: - ldap: - servers: - main: - # snip... - external_groups: ['interns', 'contractors'] - ``` - -1. [Restart GitLab](../restart_gitlab.md#installations-from-source) for the changes to take effect. - -## Group sync technical details - -There is a lot going on with group sync 'under the hood'. This section -outlines what LDAP queries are executed and what behavior you can expect -from group sync. - -Group member access will be downgraded from a higher level if their LDAP group -membership changes. For example, if a user has 'Owner' rights in a group and the -next group sync reveals they should only have 'Developer' privileges, their -access will be adjusted accordingly. The only exception is if the user is the -*last* owner in a group. Groups need at least one owner to fulfill -administrative duties. - -### Supported LDAP group types/attributes - -GitLab supports LDAP groups that use member attributes: - -- `member` -- `submember` -- `uniquemember` -- `memberof` -- `memberuid`. - -This means group sync supports, at least, LDAP groups with object class: -`groupOfNames`, `posixGroup`, and `groupOfUniqueNames`. - -Other object classes should work fine as long as members -are defined as one of the mentioned attributes. This also means GitLab supports -Microsoft Active Directory, Apple Open Directory, Open LDAP, and 389 Server. -Other LDAP servers should work, too. - -Active Directory also supports nested groups. Group sync will recursively -resolve membership if `active_directory: true` is set in the configuration file. - -NOTE: **Note:** -Nested group membership will only be resolved if the nested group -also falls within the configured `group_base`. For example, if GitLab sees a -nested group with DN `cn=nested_group,ou=special_groups,dc=example,dc=com` but -the configured `group_base` is `ou=groups,dc=example,dc=com`, `cn=nested_group` -will be ignored. - -### Queries - -- Each LDAP group is queried a maximum of one time with base `group_base` and - filter `(cn=<cn_from_group_link>)`. -- If the LDAP group has the `memberuid` attribute, GitLab will execute another - LDAP query per member to obtain each user's full DN. These queries are - executed with base `base`, scope 'base object', and a filter depending on - whether `user_filter` is set. Filter may be `(uid=<uid_from_group>)` or a - joining of `user_filter`. - -### Benchmarks - -Group sync was written to be as performant as possible. Data is cached, database -queries are optimized, and LDAP queries are minimized. The last benchmark run -revealed the following metrics: - -For 20000 LDAP users, 11000 LDAP groups and 1000 GitLab groups with 10 -LDAP group links each: - -- Initial sync (no existing members assigned in GitLab) took 1.8 hours -- Subsequent syncs (checking membership, no writes) took 15 minutes - -These metrics are meant to provide a baseline and performance may vary based on -any number of factors. This was a pretty extreme benchmark and most instances will -not have near this many users or groups. Disk speed, database performance, -network and LDAP server response time will affect these metrics. - -## Troubleshooting - -Please see our [administrator guide to troubleshooting LDAP](ldap-troubleshooting.md). +This document was moved to [another location](ldap/index.md). diff --git a/doc/administration/auth/ldap-troubleshooting.md b/doc/administration/auth/ldap-troubleshooting.md index 77c9c30d140..a553f449abb 100644 --- a/doc/administration/auth/ldap-troubleshooting.md +++ b/doc/administration/auth/ldap-troubleshooting.md @@ -1,651 +1,5 @@ -# LDAP Troubleshooting for Administrators +--- +redirect_to: 'ldap/ldap-troubleshooting.md' +--- -## Common Problems & Workflows - -### Connection - -#### Connection refused - -If you are getting `Connection Refused` errors when trying to connect to the -LDAP server please double-check the LDAP `port` and `encryption` settings used by -GitLab. Common combinations are `encryption: 'plain'` and `port: 389`, OR -`encryption: 'simple_tls'` and `port: 636`. - -#### Connection times out - -If GitLab cannot reach your LDAP endpoint, you will see a message like this: - -```plaintext -Could not authenticate you from Ldapmain because "Connection timed out - user specified timeout". -``` - -If your configured LDAP provider and/or endpoint is offline or otherwise -unreachable by GitLab, no LDAP user will be able to authenticate and log in. -GitLab does not cache or store credentials for LDAP users to provide authentication -during an LDAP outage. - -Contact your LDAP provider or administrator if you are seeing this error. - -#### Referral error - -If you see `LDAP search error: Referral` in the logs, or when troubleshooting -LDAP Group Sync, this error may indicate a configuration problem. The LDAP -configuration `/etc/gitlab/gitlab.rb` (Omnibus) or `config/gitlab.yml` (source) -is in YAML format and is sensitive to indentation. Check that `group_base` and -`admin_group` configuration keys are indented 2 spaces past the server -identifier. The default identifier is `main` and an example snippet looks like -the following: - -```yaml -main: # 'main' is the GitLab 'provider ID' of this LDAP server - label: 'LDAP' - host: 'ldap.example.com' - ... - group_base: 'cn=my_group,ou=groups,dc=example,dc=com' - admin_group: 'my_admin_group' -``` - -#### Query LDAP **(STARTER ONLY)** - -The following allows you to perform a search in LDAP using the rails console. -Depending on what you're trying to do, it may make more sense to query [a -user](#query-a-user-in-ldap) or [a group](#query-a-group-in-ldap-starter-only) directly, or -even [use `ldapsearch`](#ldapsearch) instead. - -```ruby -adapter = Gitlab::Auth::Ldap::Adapter.new('ldapmain') -options = { - # :base is required - # use .base or .group_base - base: adapter.config.group_base, - - # :filter is optional - # 'cn' looks for all "cn"s under :base - # '*' is the search string - here, it's a wildcard - filter: Net::Ldap::Filter.eq('cn', '*'), - - # :attributes is optional - # the attributes we want to get returned - attributes: %w(dn cn memberuid member submember uniquemember memberof) -} -adapter.ldap_search(options) -``` - -For examples of how this is run, -[review the `Adapter` module](https://gitlab.com/gitlab-org/gitlab/blob/master/ee/lib/ee/gitlab/auth/ldap/adapter.rb). - -### User logins - -#### No users are found - -If [you've confirmed](#ldap-check) that a connection to LDAP can be -established but GitLab doesn't show you LDAP users in the output, one of the -following is most likely true: - -- The `bind_dn` user doesn't have enough permissions to traverse the user tree. -- The user(s) don't fall under the [configured `base`](ldap.md#configuration). -- The [configured `user_filter`](ldap.md#using-an-ldap-filter-to-limit-access-to-your-gitlab-server) blocks access to the user(s). - -In this case, you con confirm which of the above is true using -[ldapsearch](#ldapsearch) with the existing LDAP configuration in your -`/etc/gitlab/gitlab.rb`. - -#### User(s) cannot login - -A user can have trouble logging in for any number of reasons. To get started, -here are some questions to ask yourself: - -- Does the user fall under the [configured `base`](ldap.md#configuration) in - LDAP? The user must fall under this `base` to login. -- Does the user pass through the [configured `user_filter`](ldap.md#using-an-ldap-filter-to-limit-access-to-your-gitlab-server)? - If one is not configured, this question can be ignored. If it is, then the - user must also pass through this filter to be allowed to login. - - Refer to our docs on [debugging the `user_filter`](#debug-ldap-user-filter). - -If the above are both okay, the next place to look for the problem is -the logs themselves while reproducing the issue. - -- Ask the user to login and let it fail. -- [Look through the output](#gitlab-logs) for any errors or other - messages about the login. You may see one of the other error messages on - this page, in which case that section can help resolve the issue. - -If the logs don't lead to the root of the problem, use the -[rails console](#rails-console) to [query this user](#query-a-user-in-ldap) -to see if GitLab can read this user on the LDAP server. - -It can also be helpful to -[debug a user sync](#sync-all-users-starter-only) to -investigate further. - -#### Invalid credentials on login - -If that the login credentials used are accurate on LDAP, ensure the following -are true for the user in question: - -- Make sure the user you are binding with has enough permissions to read the user's - tree and traverse it. -- Check that the `user_filter` is not blocking otherwise valid users. -- Run [an LDAP check command](#ldap-check) to make sure that the LDAP settings - are correct and [GitLab can see your users](#no-users-are-found). - -#### Email has already been taken - -A user tries to login with the correct LDAP credentials, is denied access, -and the [production.log](../logs.md#productionlog) shows an error that looks like this: - -```plaintext -(LDAP) Error saving user <USER DN> (email@example.com): ["Email has already been taken"] -``` - -This error is referring to the email address in LDAP, `email@example.com`. Email -addresses must be unique in GitLab and LDAP links to a user's primary email (as opposed -to any of their possibly-numerous secondary emails). Another user (or even the -same user) has the email `email@example.com` set as a secondary email, which -is throwing this error. - -We can check where this conflicting email address is coming from using the -[rails console](#rails-console). Once in the console, run the following: - -```ruby -# This searches for an email among the primary AND secondary emails -user = User.find_by_any_email('email@example.com') -user.username -``` - -This will show you which user has this email address. One of two steps will -have to be taken here: - -- To create a new GitLab user/username for this user when logging in with LDAP, - remove the secondary email to remove the conflict. -- To use an existing GitLab user/username for this user to use with LDAP, - remove this email as a secondary email and make it a primary one so GitLab - will associate this profile to the LDAP identity. - -The user can do either of these steps [in their -profile](../../user/profile/index.md#user-profile) or an admin can do it. - -#### Debug LDAP user filter - -[`ldapsearch`](#ldapsearch) allows you to test your configured -[user filter](ldap.md#using-an-ldap-filter-to-limit-access-to-your-gitlab-server) -to confirm that it returns the users you expect it to return. - -```shell -ldapsearch -H ldaps://$host:$port -D "$bind_dn" -y bind_dn_password.txt -b "$base" "$user_filter" sAMAccountName -``` - -- Variables beginning with a `$` refer to a variable from the LDAP section of - your configuration file. -- Replace `ldaps://` with `ldap://` if you are using the plain authentication method. - Port `389` is the default `ldap://` port and `636` is the default `ldaps://` - port. -- We are assuming the password for the `bind_dn` user is in `bind_dn_password.txt`. - -#### Sync all users **(STARTER ONLY)** - -The output from a manual [user sync](ldap-ee.md#user-sync) can show you what happens when -GitLab tries to sync its users against LDAP. Enter the [rails console](#rails-console) -and then run: - -```ruby -Rails.logger.level = Logger::DEBUG - -LdapSyncWorker.new.perform -``` - -Next, [learn how to read the -output](#example-console-output-after-a-user-sync-starter-only). - -##### Example console output after a user sync **(STARTER ONLY)** - -The output from a [manual user sync](#sync-all-users-starter-only) will be very verbose, and a -single user's successful sync can look like this: - -```shell -Syncing user John, email@example.com - Identity Load (0.9ms) SELECT "identities".* FROM "identities" WHERE "identities"."user_id" = 20 AND (provider LIKE 'ldap%') LIMIT 1 -Instantiating Gitlab::Auth::Ldap::Person with LDIF: -dn: cn=John Smith,ou=people,dc=example,dc=com -cn: John Smith -mail: email@example.com -memberof: cn=admin_staff,ou=people,dc=example,dc=com -uid: John - - UserSyncedAttributesMetadata Load (0.9ms) SELECT "user_synced_attributes_metadata".* FROM "user_synced_attributes_metadata" WHERE "user_synced_attributes_metadata"."user_id" = 20 LIMIT 1 - (0.3ms) BEGIN - Namespace Load (1.0ms) SELECT "namespaces".* FROM "namespaces" WHERE "namespaces"."owner_id" = 20 AND "namespaces"."type" IS NULL LIMIT 1 - Route Load (0.8ms) SELECT "routes".* FROM "routes" WHERE "routes"."source_id" = 27 AND "routes"."source_type" = 'Namespace' LIMIT 1 - Ci::Runner Load (1.1ms) SELECT "ci_runners".* FROM "ci_runners" INNER JOIN "ci_runner_namespaces" ON "ci_runners"."id" = "ci_runner_namespaces"."runner_id" WHERE "ci_runner_namespaces"."namespace_id" = 27 - (0.7ms) COMMIT - (0.4ms) BEGIN - Route Load (0.8ms) SELECT "routes".* FROM "routes" WHERE (LOWER("routes"."path") = LOWER('John')) - Namespace Load (1.0ms) SELECT "namespaces".* FROM "namespaces" WHERE "namespaces"."id" = 27 LIMIT 1 - Route Exists (0.9ms) SELECT 1 AS one FROM "routes" WHERE LOWER("routes"."path") = LOWER('John') AND "routes"."id" != 50 LIMIT 1 - User Update (1.1ms) UPDATE "users" SET "updated_at" = '2019-10-17 14:40:59.751685', "last_credential_check_at" = '2019-10-17 14:40:59.738714' WHERE "users"."id" = 20 -``` - -There's a lot here, so let's go over what could be helpful when debugging. - -First, GitLab will look for all users that have previously -logged in with LDAP and iterate on them. Each user's sync will start with -the following line that contains the user's username and email, as they -exist in GitLab now: - -```shell -Syncing user John, email@example.com -``` - -If you don't find a particular user's GitLab email in the output, then that -user hasn't logged in with LDAP yet. - -Next, GitLab searches its `identities` table for the existing -link between this user and the configured LDAP provider(s): - -```sql - Identity Load (0.9ms) SELECT "identities".* FROM "identities" WHERE "identities"."user_id" = 20 AND (provider LIKE 'ldap%') LIMIT 1 -``` - -The identity object will have the DN that GitLab will use to look for the user -in LDAP. If the DN isn't found, the email is used instead. We can see that -this user is found in LDAP: - -```shell -Instantiating Gitlab::Auth::Ldap::Person with LDIF: -dn: cn=John Smith,ou=people,dc=example,dc=com -cn: John Smith -mail: email@example.com -memberof: cn=admin_staff,ou=people,dc=example,dc=com -uid: John -``` - -If the user wasn't found in LDAP with either the DN or email, you may see the -following message instead: - -```shell -LDAP search error: No Such Object -``` - -...in which case the user will be blocked: - -```shell - User Update (0.4ms) UPDATE "users" SET "state" = $1, "updated_at" = $2 WHERE "users"."id" = $3 [["state", "ldap_blocked"], ["updated_at", "2019-10-18 15:46:22.902177"], ["id", 20]] -``` - -Once the user is found in LDAP the rest of the output will update the GitLab -database with any changes. - -#### Query a user in LDAP - -This will test that GitLab can reach out to LDAP and read a particular user. -It can expose potential errors connecting to and/or querying LDAP -that may seem to fail silently in the GitLab UI. - -```ruby -Rails.logger.level = Logger::DEBUG - -adapter = Gitlab::Auth::Ldap::Adapter.new('ldapmain') # If `main` is the LDAP provider -Gitlab::Auth::Ldap::Person.find_by_uid('<uid>', adapter) -``` - -### Group memberships **(STARTER ONLY)** - -#### Membership(s) not granted **(STARTER ONLY)** - -Sometimes you may think a particular user should be added to a GitLab group via -LDAP group sync, but for some reason it's not happening. There are several -things to check to debug the situation. - -- Ensure LDAP configuration has a `group_base` specified. - [This configuration](ldap-ee.md#group-sync) is required for group sync to work properly. -- Ensure the correct [LDAP group link is added to the GitLab - group](ldap-ee.md#adding-group-links). -- Check that the user has an LDAP identity: - 1. Sign in to GitLab as an administrator user. - 1. Navigate to **Admin area -> Users**. - 1. Search for the user - 1. Open the user, by clicking on their name. Do not click 'Edit'. - 1. Navigate to the **Identities** tab. There should be an LDAP identity with - an LDAP DN as the 'Identifier'. If not, this user hasn't logged in with - LDAP yet and must do so first. -- You've waited an hour or [the configured - interval](ldap-ee.md#adjusting-ldap-group-sync-schedule) for the group to - sync. To speed up the process, either go to the GitLab group **Settings -> - Members** and press **Sync now** (sync one group) or [run the group sync Rake - task](../raketasks/ldap.md#run-a-group-sync-starter-only) (sync all groups). - -If all of the above looks good, jump in to a little more advanced debugging in -the rails console. - -1. Enter the [rails console](#rails-console). -1. Choose a GitLab group to test with. This group should have an LDAP group link - already configured. -1. [Enable debug logging, find the above GitLab group, and sync it with LDAP](#sync-one-group-starter-only). -1. Look through the output of the sync. See [example log - output](#example-console-output-after-a-group-sync-starter-only) - for how to read the output. -1. If you still aren't able to see why the user isn't being added, [query the - LDAP group directly](#query-a-group-in-ldap-starter-only) to see what members are listed. -1. Is the user's DN or UID in one of the lists from the above output? One of the DNs or - UIDs here should match the 'Identifier' from the LDAP identity checked earlier. If it doesn't, - the user does not appear to be in the LDAP group. - -#### Admin privileges not granted - -When [Administrator sync](ldap-ee.md#administrator-sync) has been configured -but the configured users aren't granted the correct admin privileges, confirm -the following are true: - -- A [`group_base` is also configured](ldap-ee.md#group-sync). -- The configured `admin_group` in the `gitlab.rb` is a CN, rather than a DN or an array. -- This CN falls under the scope of the configured `group_base`. -- The members of the `admin_group` have already logged into GitLab with their LDAP - credentials. GitLab will only grant this admin access to the users whose - accounts are already connected to LDAP. - -If all the above are true and the users are still not getting access, [run a manual -group sync](#sync-all-groups-starter-only) in the rails console and [look through the -output](#example-console-output-after-a-group-sync-starter-only) to see what happens when -GitLab syncs the `admin_group`. - -#### Sync all groups **(STARTER ONLY)** - -NOTE: **NOTE:** -To sync all groups manually when debugging is unnecessary, [use the Rake -task](../raketasks/ldap.md#run-a-group-sync-starter-only) instead. - -The output from a manual [group sync](ldap-ee.md#group-sync) can show you what happens -when GitLab syncs its LDAP group memberships against LDAP. - -```ruby -Rails.logger.level = Logger::DEBUG - -LdapAllGroupsSyncWorker.new.perform -``` - -Next, [learn how to read the -output](#example-console-output-after-a-group-sync-starter-only). - -##### Example console output after a group sync **(STARTER ONLY)** - -Like the output from the user sync, the output from the [manual group -sync](#sync-all-groups-starter-only) will also be very verbose. However, it contains lots -of helpful information. - -Indicates the point where syncing actually begins: - -```shell -Started syncing 'ldapmain' provider for 'my_group' group -``` - -The following entry shows an array of all user DNs GitLab sees in the LDAP server. -Note that these are the users for a single LDAP group, not a GitLab group. If -you have multiple LDAP groups linked to this GitLab group, you will see multiple -log entries like this - one for each LDAP group. If you don't see an LDAP user -DN in this log entry, LDAP is not returning the user when we do the lookup. -Verify the user is actually in the LDAP group. - -```shell -Members in 'ldap_group_1' LDAP group: ["uid=john0,ou=people,dc=example,dc=com", -"uid=mary0,ou=people,dc=example,dc=com", "uid=john1,ou=people,dc=example,dc=com", -"uid=mary1,ou=people,dc=example,dc=com", "uid=john2,ou=people,dc=example,dc=com", -"uid=mary2,ou=people,dc=example,dc=com", "uid=john3,ou=people,dc=example,dc=com", -"uid=mary3,ou=people,dc=example,dc=com", "uid=john4,ou=people,dc=example,dc=com", -"uid=mary4,ou=people,dc=example,dc=com"] -``` - -Shortly after each of the above entries, you will see a hash of resolved member -access levels. This hash represents all user DNs GitLab thinks should have -access to this group, and at which access level (role). This hash is additive, -and more DNs may be added, or existing entries modified, based on additional -LDAP group lookups. The very last occurrence of this entry should indicate -exactly which users GitLab believes should be added to the group. - -NOTE: **Note:** -10 is 'Guest', 20 is 'Reporter', 30 is 'Developer', 40 is 'Maintainer' -and 50 is 'Owner'. - -```shell -Resolved 'my_group' group member access: {"uid=john0,ou=people,dc=example,dc=com"=>30, -"uid=mary0,ou=people,dc=example,dc=com"=>30, "uid=john1,ou=people,dc=example,dc=com"=>30, -"uid=mary1,ou=people,dc=example,dc=com"=>30, "uid=john2,ou=people,dc=example,dc=com"=>30, -"uid=mary2,ou=people,dc=example,dc=com"=>30, "uid=john3,ou=people,dc=example,dc=com"=>30, -"uid=mary3,ou=people,dc=example,dc=com"=>30, "uid=john4,ou=people,dc=example,dc=com"=>30, -"uid=mary4,ou=people,dc=example,dc=com"=>30} -``` - -It's not uncommon to see warnings like the following. These indicate that GitLab -would have added the user to a group, but the user could not be found in GitLab. -Usually this is not a cause for concern. - -If you think a particular user should already exist in GitLab, but you're seeing -this entry, it could be due to a mismatched DN stored in GitLab. See -[User DN and/or email have changed](#user-dn-orand-email-have-changed) to update the user's LDAP identity. - -```shell -User with DN `uid=john0,ou=people,dc=example,dc=com` should have access -to 'my_group' group but there is no user in GitLab with that -identity. Membership will be updated once the user signs in for -the first time. -``` - -Finally, the following entry says syncing has finished for this group: - -```shell -Finished syncing all providers for 'my_group' group -``` - -Once all the configured group links have been synchronized, GitLab will look -for any Administrators or External users to sync: - -```shell -Syncing admin users for 'ldapmain' provider -``` - -The output will look similar to what happens with a single group, and then -this line will indicate the sync is finished: - -```shell -Finished syncing admin users for 'ldapmain' provider -``` - -If [admin sync](ldap-ee.md#administrator-sync) is not configured, you'll see a message -stating as such: - -```shell -No `admin_group` configured for 'ldapmain' provider. Skipping -``` - -#### Sync one group **(STARTER ONLY)** - -[Syncing all groups](#sync-all-groups-starter-only) can produce a lot of noise in the output, which can be -distracting when you're only interested in troubleshooting the memberships of -a single GitLab group. In that case, here's how you can just sync this group -and see its debug output: - -```ruby -Rails.logger.level = Logger::DEBUG - -# Find the GitLab group. -# If the output is `nil`, the group could not be found. -# If a bunch of group attributes are in the output, your group was found successfully. -group = Group.find_by(name: 'my_gitlab_group') - -# Sync this group against LDAP -EE::Gitlab::Auth::Ldap::Sync::Group.execute_all_providers(group) -``` - -The output will be similar to -[that you'd get from syncing all groups](#example-console-output-after-a-group-sync-starter-only). - -#### Query a group in LDAP **(STARTER ONLY)** - -When you'd like to confirm that GitLab can read a LDAP group and see all its members, -you can run the following: - -```ruby -# Find the adapter and the group itself -adapter = Gitlab::Auth::Ldap::Adapter.new('ldapmain') # If `main` is the LDAP provider -ldap_group = EE::Gitlab::Auth::Ldap::Group.find_by_cn('group_cn_here', adapter) - -# Find the members of the LDAP group -ldap_group.member_dns -ldap_group.member_uids -``` - -### User DN or/and email have changed - -When an LDAP user is created in GitLab, their LDAP DN is stored for later reference. - -If GitLab cannot find a user by their DN, it will fall back -to finding the user by their email. If the lookup is successful, GitLab will -update the stored DN to the new value so both values will now match what's in -LDAP. - -If the email has changed and the DN has not, GitLab will find the user with -the DN and update its own record of the user's email to match the one in LDAP. - -However, if the primary email _and_ the DN change in LDAP, then GitLab will -have no way of identifying the correct LDAP record of the user and, as a -result, the user will be blocked. To rectify this, the user's existing -profile will have to be updated with at least one of the new values (primary -email or DN) so the LDAP record can be found. - -The following script will update the emails for all provided users so they -won't be blocked or unable to access their accounts. - ->**NOTE**: The following script will require that any new accounts with the new -email address are removed first. This is because emails have to be unique in GitLab. - -Go to the [rails console](#rails-console) and then run: - -```ruby -# Each entry will have to include the old username and the new email -emails = { - 'ORIGINAL_USERNAME' => 'NEW_EMAIL_ADDRESS', - ... -} - -emails.each do |username, email| - user = User.find_by_username(username) - user.email = email - user.skip_reconfirmation! - user.save! -end -``` - -You can then [run a UserSync](#sync-all-users-starter-only) **(STARTER ONLY)** to sync the latest DN -for each of these users. - -## Debugging Tools - -### LDAP check - -The [Rake task to check LDAP](../raketasks/ldap.md#check) is a valuable tool -to help determine whether GitLab can successfully establish a connection to -LDAP and can get so far as to even read users. - -If a connection can't be established, it is likely either because of a problem -with your configuration or a firewall blocking the connection. - -- Ensure you don't have a firewall blocking the -connection, and that the LDAP server is accessible to the GitLab host. -- Look for an error message in the Rake check output, which may lead to your LDAP configuration to -confirm that the configuration values (specifically `host`, `port`, `bind_dn`, and -`password`) are correct. -- Look for [errors](#connection) in [the logs](#gitlab-logs) to further debug connection failures. - -If GitLab can successfully connect to LDAP but doesn't return any -users, [see what to do when no users are found](#no-users-are-found). - -### GitLab logs - -If a user account is blocked or unblocked due to the LDAP configuration, a -message will be [logged to `application.log`](../logs.md#applicationlog). - -If there is an unexpected error during an LDAP lookup (configuration error, -timeout), the login is rejected and a message will be [logged to -`production.log`](../logs.md#productionlog). - -### ldapsearch - -`ldapsearch` is a utility that will allow you to query your LDAP server. You can -use it to test your LDAP settings and ensure that the settings you're using -will get you the results you expect. - -When using `ldapsearch`, be sure to use the same settings you've already -specified in your `gitlab.rb` configuration so you can confirm what happens -when those exact settings are used. - -Running this command on the GitLab host will also help confirm that there's no -obstruction between the GitLab host and LDAP. - -For example, consider the following GitLab configuration: - -```shell -gitlab_rails['ldap_servers'] = YAML.load <<-'EOS' # remember to close this block with 'EOS' below - main: # 'main' is the GitLab 'provider ID' of this LDAP server - label: 'LDAP' - host: '127.0.0.1' - port: 389 - uid: 'uid' - encryption: 'plain' - bind_dn: 'cn=admin,dc=ldap-testing,dc=example,dc=com' - password: 'Password1' - active_directory: true - allow_username_or_email_login: false - block_auto_created_users: false - base: 'dc=ldap-testing,dc=example,dc=com' - user_filter: '' - attributes: - username: ['uid', 'userid', 'sAMAccountName'] - email: ['mail', 'email', 'userPrincipalName'] - name: 'cn' - first_name: 'givenName' - last_name: 'sn' - group_base: 'ou=groups,dc=ldap-testing,dc=example,dc=com' - admin_group: 'gitlab_admin' -EOS -``` - -You would run the following `ldapsearch` to find the `bind_dn` user: - -```shell -ldapsearch -D "cn=admin,dc=ldap-testing,dc=example,dc=com" \ - -w Password1 \ - -p 389 \ - -h 127.0.0.1 \ - -b "dc=ldap-testing,dc=example,dc=com" -``` - -Note that the `bind_dn`, `password`, `port`, `host`, and `base` are all -identical to what's configured in the `gitlab.rb`. - -Please see [the official -`ldapsearch` documentation](https://linux.die.net/man/1/ldapsearch) for more. - -### Rails console - -CAUTION: **CAUTION:** -Please note that it is very easy to create, read, modify, and destroy data on the -rails console, so please be sure to run commands exactly as listed. - -The rails console is a valuable tool to help debug LDAP problems. It allows you to -directly interact with the application by running commands and seeing how GitLab -responds to them. - -Please refer to [this guide](../troubleshooting/debug.md#starting-a-rails-console-session) -for instructions on how to use the rails console. - -#### Enable debug output - -This will provide debug output that will be useful to see -what GitLab is doing and with what. This value is not persisted, and will only -be enabled for this session in the rails console. - -To enable debug output in the rails console, [enter the rails -console](#rails-console) and run: - -```ruby -Rails.logger.level = Logger::DEBUG -``` +This document was moved to [another location](ldap/ldap-troubleshooting.md). diff --git a/doc/administration/auth/ldap.md b/doc/administration/auth/ldap.md index 1e91b539db5..f5565628af1 100644 --- a/doc/administration/auth/ldap.md +++ b/doc/administration/auth/ldap.md @@ -1,555 +1,5 @@ --- -type: reference +redirect_to: 'ldap/index.md' --- -<!-- If the change is EE-specific, put it in `ldap-ee.md`, NOT here. --> - -# LDAP - -GitLab integrates with LDAP to support user authentication. - -This integration works with most LDAP-compliant directory servers, including: - -- Microsoft Active Directory -- Apple Open Directory -- Open LDAP -- 389 Server. - -GitLab Enterprise Editions (EE) include enhanced integration, -including group membership syncing as well as multiple LDAP servers support. - -For more details about EE-specific LDAP features, see the -[LDAP Enterprise Edition documentation](ldap-ee.md). - -NOTE: **Note:** -The information on this page is relevant for both GitLab CE and EE. - -## Overview - -[LDAP](https://en.wikipedia.org/wiki/Lightweight_Directory_Access_Protocol) -stands for **Lightweight Directory Access Protocol**, which is a standard -application protocol for accessing and maintaining distributed directory -information services over an Internet Protocol (IP) network. - -## Security - -GitLab assumes that LDAP users: - -- Are not able to change their LDAP `mail`, `email`, or `userPrincipalName` attribute. - An LDAP user who is allowed to change their email on the LDAP server can potentially - [take over any account](#enabling-ldap-sign-in-for-existing-gitlab-users) - on your GitLab server. -- Have unique email addresses, otherwise it is possible for LDAP users with the same - email address to share the same GitLab account. - -We recommend against using LDAP integration if your LDAP users are -allowed to change their 'mail', 'email' or 'userPrincipalName' attribute on -the LDAP server or share email addresses. - -### User deletion - -If a user is deleted from the LDAP server, they will be blocked in GitLab as -well. Users will be immediately blocked from logging in. However, there is an -LDAP check cache time of one hour (see note) which means users that -are already logged in or are using Git over SSH will still be able to access -GitLab for up to one hour. Manually block the user in the GitLab Admin Area to -immediately block all access. - -NOTE: **Note**: -GitLab Enterprise Edition Starter supports a -[configurable sync time](ldap-ee.md#adjusting-ldap-user-sync-schedule), -with a default of one hour. - -## Git password authentication - -LDAP-enabled users can always authenticate with Git using their GitLab username -or email and LDAP password, even if password authentication for Git is disabled -in the application settings. - -## Google Secure LDAP **(CORE ONLY)** - -> Introduced in GitLab 11.9. - -[Google Cloud Identity](https://cloud.google.com/identity/) provides a Secure -LDAP service that can be configured with GitLab for authentication and group sync. -See [Google Secure LDAP](google_secure_ldap.md) for detailed configuration instructions. - -## Configuration - -NOTE: **Note**: -In GitLab Enterprise Edition Starter, you can configure multiple LDAP servers -to connect to one GitLab server. - -For a complete guide on configuring LDAP with: - -- GitLab Community Edition, see - [How to configure LDAP with GitLab CE](how_to_configure_ldap_gitlab_ce/index.md). -- Enterprise Editions, see - [How to configure LDAP with GitLab EE](how_to_configure_ldap_gitlab_ee/index.md). **(STARTER ONLY)** - -To enable LDAP integration you need to add your LDAP server settings in -`/etc/gitlab/gitlab.rb` or `/home/git/gitlab/config/gitlab.yml` for Omnibus -GitLab and installations from source respectively. - -There is a Rake task to check LDAP configuration. After configuring LDAP -using the documentation below, see [LDAP check Rake task](../raketasks/check.md#ldap-check) -for information on the LDAP check Rake task. - -Prior to version 7.4, GitLab used a different syntax for configuring -LDAP integration. The old LDAP integration syntax still works but may be -removed in a future version. If your `gitlab.rb` or `gitlab.yml` file contains -LDAP settings in both the old syntax and the new syntax, only the __old__ -syntax will be used by GitLab. - -The configuration inside `gitlab_rails['ldap_servers']` below is sensitive to -incorrect indentation. Be sure to retain the indentation given in the example. -Copy/paste can sometimes cause problems. - -NOTE: **Note:** -The `encryption` value `ssl` corresponds to 'Simple TLS' in the LDAP -library. `tls` corresponds to StartTLS, not to be confused with regular TLS. -Normally, if you specify `ssl` it will be on port 636, while `tls` (StartTLS) -would be on port 389. `plain` also operates on port 389. - -NOTE: **Note:** -LDAP users must have an email address set, regardless of whether it is used to log in. - -**Omnibus configuration** - -```ruby -gitlab_rails['ldap_enabled'] = true -gitlab_rails['prevent_ldap_sign_in'] = false -gitlab_rails['ldap_servers'] = YAML.load <<-EOS # remember to close this block with 'EOS' below -## -## 'main' is the GitLab 'provider ID' of this LDAP server -## -main: - ## - ## A human-friendly name for your LDAP server. It is OK to change the label later, - ## for instance if you find out it is too large to fit on the web page. - ## - ## Example: 'Paris' or 'Acme, Ltd.' - ## - label: 'LDAP' - - ## - ## Example: 'ldap.mydomain.com' - ## - host: '_your_ldap_server' - - ## - ## This port is an example, it is sometimes different but it is always an - ## integer and not a string. - ## - port: 389 # usually 636 for SSL - uid: 'sAMAccountName' # This should be the attribute, not the value that maps to uid. - - ## - ## Examples: 'america\momo' or 'CN=Gitlab Git,CN=Users,DC=mydomain,DC=com' - ## - bind_dn: '_the_full_dn_of_the_user_you_will_bind_with' - password: '_the_password_of_the_bind_user' - - ## - ## Encryption method. The "method" key is deprecated in favor of - ## "encryption". - ## - ## Examples: "start_tls" or "simple_tls" or "plain" - ## - ## Deprecated values: "tls" was replaced with "start_tls" and "ssl" was - ## replaced with "simple_tls". - ## - ## - encryption: 'plain' - - ## - ## Enables SSL certificate verification if encryption method is - ## "start_tls" or "simple_tls". Defaults to true since GitLab 10.0 for - ## security. This may break installations upon upgrade to 10.0, that did - ## not know their LDAP SSL certificates were not set up properly. - ## - verify_certificates: true - - # OpenSSL::SSL::SSLContext options. - tls_options: - # Specifies the path to a file containing a PEM-format CA certificate, - # e.g. if you need to use an internal CA. - # - # Example: '/etc/ca.pem' - # - ca_file: '' - - # Specifies the SSL version for OpenSSL to use, if the OpenSSL default - # is not appropriate. - # - # Example: 'TLSv1_1' - # - ssl_version: '' - - # Specific SSL ciphers to use in communication with LDAP servers. - # - # Example: 'ALL:!EXPORT:!LOW:!aNULL:!eNULL:!SSLv2' - ciphers: '' - - # Client certificate - # - # Example: - # cert: | - # -----BEGIN CERTIFICATE----- - # MIIDbDCCAlSgAwIBAgIGAWkJxLmKMA0GCSqGSIb3DQEBCwUAMHcxFDASBgNVBAoTC0dvb2dsZSBJ - # bmMuMRYwFAYDVQQHEw1Nb3VudGFpbiBWaWV3MRQwEgYDVQQDEwtMREFQIENsaWVudDEPMA0GA1UE - # CxMGR1N1aXRlMQswCQYDVQQGEwJVUzETMBEGA1UECBMKQ2FsaWZvcm5pYTAeFw0xOTAyMjAwNzE4 - # rntnF4d+0dd7zP3jrWkbdtoqjLDT/5D7NYRmVCD5vizV98FJ5//PIHbD1gL3a9b2MPAc6k7NV8tl - # ... - # 4SbuJPAiJxC1LQ0t39dR6oMCAMab3hXQqhL56LrR6cRBp6Mtlphv7alu9xb/x51y2x+g2zWtsf80 - # Jrv/vKMsIh/sAyuogb7hqMtp55ecnKxceg== - # -----END CERTIFICATE ----- - cert: '' - - # Client private key - # key: | - # -----BEGIN PRIVATE KEY----- - # MIIEvQIBADANBgkqhkiG9w0BAQEFAASCBKcwggSjAgEAAoIBAQC3DmJtLRmJGY4xU1QtI3yjvxO6 - # bNuyE4z1NF6Xn7VSbcAaQtavWQ6GZi5uukMo+W5DHVtEkgDwh92ySZMuJdJogFbNvJvHAayheCdN - # 7mCQ2UUT9jGXIbmksUn9QMeJVXTZjgJWJzPXToeUdinx9G7+lpVa62UATEd1gaI3oyL72WmpDy/C - # rntnF4d+0dd7zP3jrWkbdtoqjLDT/5D7NYRmVCD5vizV98FJ5//PIHbD1gL3a9b2MPAc6k7NV8tl - # ... - # +9IhSYX+XIg7BZOVDeYqlPfxRvQh8vy3qjt/KUihmEPioAjLaGiihs1Fk5ctLk9A2hIUyP+sEQv9 - # l6RG+a/mW+0rCWn8JAd464Ps9hE= - # -----END PRIVATE KEY----- - key: '' - - ## - ## Set a timeout, in seconds, for LDAP queries. This helps avoid blocking - ## a request if the LDAP server becomes unresponsive. - ## A value of 0 means there is no timeout. - ## - timeout: 10 - - ## - ## This setting specifies if LDAP server is Active Directory LDAP server. - ## For non AD servers it skips the AD specific queries. - ## If your LDAP server is not AD, set this to false. - ## - active_directory: true - - ## - ## If allow_username_or_email_login is enabled, GitLab will ignore everything - ## after the first '@' in the LDAP username submitted by the user on login. - ## - ## Example: - ## - the user enters 'jane.doe@example.com' and 'p@ssw0rd' as LDAP credentials; - ## - GitLab queries the LDAP server with 'jane.doe' and 'p@ssw0rd'. - ## - ## If you are using "uid: 'userPrincipalName'" on ActiveDirectory you need to - ## disable this setting, because the userPrincipalName contains an '@'. - ## - allow_username_or_email_login: false - - ## - ## To maintain tight control over the number of active users on your GitLab installation, - ## enable this setting to keep new users blocked until they have been cleared by the admin - ## (default: false). - ## - block_auto_created_users: false - - ## - ## Base where we can search for users - ## - ## Ex. 'ou=People,dc=gitlab,dc=example' or 'DC=mydomain,DC=com' - ## - ## - base: '' - - ## - ## Filter LDAP users - ## - ## Format: RFC 4515 https://tools.ietf.org/search/rfc4515 - ## Ex. (employeeType=developer) - ## - ## Note: GitLab does not support omniauth-ldap's custom filter syntax. - ## - ## Example for getting only specific users: - ## '(&(objectclass=user)(|(samaccountname=momo)(samaccountname=toto)))' - ## - user_filter: '' - - ## - ## LDAP attributes that GitLab will use to create an account for the LDAP user. - ## The specified attribute can either be the attribute name as a string (e.g. 'mail'), - ## or an array of attribute names to try in order (e.g. ['mail', 'email']). - ## Note that the user's LDAP login will always be the attribute specified as `uid` above. - ## - attributes: - ## - ## The username will be used in paths for the user's own projects - ## (like `gitlab.example.com/username/project`) and when mentioning - ## them in issues, merge request and comments (like `@username`). - ## If the attribute specified for `username` contains an email address, - ## the GitLab username will be the part of the email address before the '@'. - ## - username: ['uid', 'userid', 'sAMAccountName'] - email: ['mail', 'email', 'userPrincipalName'] - - ## - ## If no full name could be found at the attribute specified for `name`, - ## the full name is determined using the attributes specified for - ## `first_name` and `last_name`. - ## - name: 'cn' - first_name: 'givenName' - last_name: 'sn' - - ## - ## If lowercase_usernames is enabled, GitLab will lower case the username. - ## - lowercase_usernames: false - - ## - ## EE only - ## - - ## Base where we can search for groups - ## - ## Ex. ou=groups,dc=gitlab,dc=example - ## - group_base: '' - - ## The CN of a group containing GitLab administrators - ## - ## Ex. administrators - ## - ## Note: Not `cn=administrators` or the full DN - ## - admin_group: '' - - ## An array of CNs of groups containing users that should be considered external - ## - ## Ex. ['interns', 'contractors'] - ## - ## Note: Not `cn=interns` or the full DN - ## - external_groups: [] - - ## - ## The LDAP attribute containing a user's public SSH key - ## - ## Example: sshPublicKey - ## - sync_ssh_keys: false - -## GitLab EE only: add more LDAP servers -## Choose an ID made of a-z and 0-9 . This ID will be stored in the database -## so that GitLab can remember which LDAP server a user belongs to. -#uswest2: -# label: -# host: -# .... -EOS -``` - -**Source configuration** - -Use the same format as `gitlab_rails['ldap_servers']` for the contents under -`servers:` in the example below: - -```yaml -production: - # snip... - ldap: - enabled: false - prevent_ldap_sign_in: false - servers: - ## - ## 'main' is the GitLab 'provider ID' of this LDAP server - ## - main: - ## - ## A human-friendly name for your LDAP server. It is OK to change the label later, - ## for instance if you find out it is too large to fit on the web page. - ## - ## Example: 'Paris' or 'Acme, Ltd.' - label: 'LDAP' - ## snip... -``` - -## Using an LDAP filter to limit access to your GitLab server - -If you want to limit all GitLab access to a subset of the LDAP users on your -LDAP server, the first step should be to narrow the configured `base`. However, -it is sometimes necessary to filter users further. In this case, you can set up -an LDAP user filter. The filter must comply with -[RFC 4515](https://tools.ietf.org/search/rfc4515). - -**Omnibus configuration** - -```ruby -gitlab_rails['ldap_servers'] = YAML.load <<-EOS -main: - # snip... - user_filter: '(employeeType=developer)' -EOS -``` - -**Source configuration** - -```yaml -production: - ldap: - servers: - main: - # snip... - user_filter: '(employeeType=developer)' -``` - -Tip: If you want to limit access to the nested members of an Active Directory -group, you can use the following syntax: - -```plaintext -(memberOf:1.2.840.113556.1.4.1941:=CN=My Group,DC=Example,DC=com) -``` - -Find more information about this "LDAP_MATCHING_RULE_IN_CHAIN" filter at -<https://docs.microsoft.com/en-us/windows/win32/adsi/search-filter-syntax>. Support for -nested members in the user filter should not be confused with -[group sync nested groups support](ldap-ee.md#supported-ldap-group-typesattributes). **(STARTER ONLY)** - -Please note that GitLab does not support the custom filter syntax used by -OmniAuth LDAP. - -### Escaping special characters - -The `user_filter` DN can contain special characters. For example: - -- A comma: - - ```plaintext - OU=GitLab, Inc,DC=gitlab,DC=com - ``` - -- Open and close brackets: - - ```plaintext - OU=Gitlab (Inc),DC=gitlab,DC=com - ``` - - These characters must be escaped as documented in - [RFC 4515](https://tools.ietf.org/search/rfc4515). - -- Escape commas with `\2C`. For example: - - ```plaintext - OU=GitLab\2C Inc,DC=gitlab,DC=com - ``` - -- Escape open and close brackets with `\28` and `\29`, respectively. For example: - - ```plaintext - OU=Gitlab \28Inc\29,DC=gitlab,DC=com - ``` - -## Enabling LDAP sign-in for existing GitLab users - -When a user signs in to GitLab with LDAP for the first time, and their LDAP -email address is the primary email address of an existing GitLab user, then -the LDAP DN will be associated with the existing user. If the LDAP email -attribute is not found in GitLab's database, a new user is created. - -In other words, if an existing GitLab user wants to enable LDAP sign-in for -themselves, they should check that their GitLab email address matches their -LDAP email address, and then sign into GitLab via their LDAP credentials. - -## Enabling LDAP username lowercase - -Some LDAP servers, depending on their configurations, can return uppercase usernames. -This can lead to several confusing issues such as creating links or namespaces with uppercase names. - -GitLab can automatically lowercase usernames provided by the LDAP server by enabling -the configuration option `lowercase_usernames`. By default, this configuration option is `false`. - -**Omnibus configuration** - -1. Edit `/etc/gitlab/gitlab.rb`: - - ```ruby - gitlab_rails['ldap_servers'] = YAML.load <<-EOS - main: - # snip... - lowercase_usernames: true - EOS - ``` - -1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. - -**Source configuration** - -1. Edit `config/gitlab.yaml`: - - ```yaml - production: - ldap: - servers: - main: - # snip... - lowercase_usernames: true - ``` - -1. [Restart GitLab](../restart_gitlab.md#installations-from-source) for the changes to take effect. - -## Disable LDAP web sign in - -It can be useful to prevent using LDAP credentials through the web UI when -an alternative such as SAML is preferred. This allows LDAP to be used for group -sync, while also allowing your SAML identity provider to handle additional -checks like custom 2FA. - -When LDAP web sign in is disabled, users will not see a **LDAP** tab on the sign in page. -This does not disable [using LDAP credentials for Git access](#git-password-authentication). - -**Omnibus configuration** - -1. Edit `/etc/gitlab/gitlab.rb`: - - ```ruby - gitlab_rails['prevent_ldap_sign_in'] = true - ``` - -1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. - -**Source configuration** - -1. Edit `config/gitlab.yaml`: - - ```yaml - production: - ldap: - prevent_ldap_sign_in: true - ``` - -1. [Restart GitLab](../restart_gitlab.md#installations-from-source) for the changes to take effect. - -## Encryption - -### TLS Server Authentication - -There are two encryption methods, `simple_tls` and `start_tls`. - -For either encryption method, if setting `verify_certificates: false`, TLS -encryption is established with the LDAP server before any LDAP-protocol data is -exchanged but no validation of the LDAP server's SSL certificate is performed. - ->**Note**: Before GitLab 9.5, `verify_certificates: false` is the default if -unspecified. - -## Limitations - -### TLS Client Authentication - -Not implemented by `Net::LDAP`. -You should disable anonymous LDAP authentication and enable simple or SASL -authentication. The TLS client authentication setting in your LDAP server cannot -be mandatory and clients cannot be authenticated with the TLS protocol. - -## Troubleshooting - -Please see our [administrator guide to troubleshooting LDAP](ldap-troubleshooting.md). +This document was moved to [another location](ldap/index.md). diff --git a/doc/administration/auth/ldap/google_secure_ldap.md b/doc/administration/auth/ldap/google_secure_ldap.md new file mode 100644 index 00000000000..2271ce93b6f --- /dev/null +++ b/doc/administration/auth/ldap/google_secure_ldap.md @@ -0,0 +1,222 @@ +--- +type: reference +stage: Manage +group: Access +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + +# Google Secure LDAP **(CORE ONLY)** + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/46391) in GitLab 11.9. + +[Google Cloud Identity](https://cloud.google.com/identity/) provides a Secure +LDAP service that can be configured with GitLab for authentication and group sync. + +Secure LDAP requires a slightly different configuration than standard LDAP servers. +The steps below cover: + +- Configuring the Secure LDAP Client in the Google Admin console. +- Required GitLab configuration. + +## Configuring Google LDAP client + +1. Navigate to <https://admin.google.com/Dashboard> and sign in as a GSuite domain administrator. + +1. Go to **Apps > LDAP > Add Client**. + +1. Provide an `LDAP client name` and an optional `Description`. Any descriptive + values are acceptable. For example, the name could be 'GitLab' and the + description could be 'GitLab LDAP Client'. Click the **Continue** button. + +  + +1. Set **Access Permission** according to your needs. You must choose either + 'Entire domain (GitLab)' or 'Selected organizational units' for both 'Verify user + credentials' and 'Read user information'. Select 'Add LDAP Client' + + TIP: **Tip:** If you plan to use GitLab [LDAP Group Sync](index.md#group-sync-starter-only) + , turn on 'Read group information'. + +  + +1. Download the generated certificate. This is required for GitLab to + communicate with the Google Secure LDAP service. Save the downloaded certificates + for later use. After downloading, click the **Continue to Client Details** button. + +1. Expand the **Service Status** section and turn the LDAP client 'ON for everyone'. + After selecting 'Save', click on the 'Service Status' bar again to collapse + and return to the rest of the settings. + +1. Expand the **Authentication** section and choose 'Generate New Credentials'. + Copy/note these credentials for later use. After selecting 'Close', click + on the 'Authentication' bar again to collapse and return to the rest of the settings. + +Now the Google Secure LDAP Client configuration is finished. The screenshot below +shows an example of the final settings. Continue on to configure GitLab. + + + +## Configuring GitLab + +Edit GitLab configuration, inserting the access credentials and certificate +obtained earlier. + +The following are the configuration keys that need to be modified using the +values obtained during the LDAP client configuration earlier: + +- `bind_dn`: The access credentials username +- `password`: The access credentials password +- `cert`: The `.crt` file text from the downloaded certificate bundle +- `key`: The `.key` file text from the downloaded certificate bundle + +**For Omnibus installations** + +1. Edit `/etc/gitlab/gitlab.rb`: + + ```ruby + gitlab_rails['ldap_enabled'] = true + gitlab_rails['ldap_servers'] = YAML.load <<-EOS # remember to close this block with 'EOS' below + main: # 'main' is the GitLab 'provider ID' of this LDAP server + label: 'Google Secure LDAP' + + host: 'ldap.google.com' + port: 636 + uid: 'uid' + bind_dn: 'DizzyHorse' + password: 'd6V5H8nhMUW9AuDP25abXeLd' + encryption: 'simple_tls' + verify_certificates: true + + tls_options: + cert: | + -----BEGIN CERTIFICATE----- + MIIDbDCCAlSgAwIBAgIGAWlzxiIfMA0GCSqGSIb3DQEBCwUAMHcxFDASBgNVBAoTC0dvb2dsZSBJ + bmMuMRYwFAYDVQQHEw1Nb3VudGFpbiBWaWV3MRQwEgYDVQQDEwtMREFQIENsaWVudDEPMA0GA1UE + CxMGR1N1aXRlMQswCQYDVQQGEwJVUzETMBEGA1UECBMKQ2FsaWZvcm5pYTAeFw0xOTAzMTIyMTE5 + MThaFw0yMjAzMTEyMTE5MThaMHcxFDASBgNVBAoTC0dvb2dsZSBJbmMuMRYwFAYDVQQHEw1Nb3Vu + dGFpbiBWaWV3MRQwEgYDVQQDEwtMREFQIENsaWVudDEPMA0GA1UECxMGR1N1aXRlMQswCQYDVQQG + EwJVUzETMBEGA1UECBMKQ2FsaWZvcm5pYTCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEB + ALOTy4aC38dyjESk6N8fRsKk8DN23ZX/GaNFL5OUmmA1KWzrvVC881OzNdtGm3vNOIxr9clteEG/ + tQwsmsJvQT5U+GkBt+tGKF/zm7zueHUYqTP7Pg5pxAnAei90qkIRFi17ulObyRHPYv1BbCt8pxNB + 4fG/gAXkFbCNxwh1eiQXXRTfruasCZ4/mHfX7MVm8JmWU9uAVIOLW+DSWOFhrDQduJdGBXJOyC2r + Gqoeg9+tkBmNH/jjxpnEkFW8q7io9DdOUqqNgoidA1h9vpKTs3084sy2DOgUvKN9uXWx14uxIyYU + Y1DnDy0wczcsuRt7l+EgtCEgpsLiLJQbKW+JS1UCAwEAATANBgkqhkiG9w0BAQsFAAOCAQEAf60J + yazhbHkDKIH2gFxfm7QLhhnqsmafvl4WP7JqZt0u0KdnvbDPfokdkM87yfbKJU1MTI86M36wEC+1 + P6bzklKz7kXbzAD4GggksAzxsEE64OWHC+Y64Tkxq2NiZTw/76POkcg9StiIXjG0ZcebHub9+Ux/ + rTncip92nDuvgEM7lbPFKRIS/YMhLCk09B/U0F6XLsf1yYjyf5miUTDikPkov23b/YGfpc8kh6hq + 1kqdi6a1cYPP34eAhtRhMqcZU9qezpJF6s9EeN/3YFfKzLODFSsVToBRAdZgGHzj//SAtLyQTD4n + KCSvK1UmaMxNaZyTHg8JnMf0ZuRpv26iSg== + -----END CERTIFICATE----- + + key: | + -----BEGIN PRIVATE KEY----- + MIIEvQIBADANBgkqhkiG9w0BAQEFAASCBKcwggSjAgEAAoIBAQCzk8uGgt/HcoxEpOjfH0bCpPAz + dt2V/xmjRS+TlJpgNSls671QvPNTszXbRpt7zTiMa/XJbXhBv7UMLJrCb0E+VPhpAbfrRihf85u8 + 7nh1GKkz+z4OacQJwHovdKpCERYte7pTm8kRz2L9QWwrfKcTQeHxv4AF5BWwjccIdXokF10U367m + rAmeP5h31+zFZvCZllPbgFSDi1vg0ljhYaw0HbiXRgVyTsgtqxqqHoPfrZAZjR/448aZxJBVvKu4 + qPQ3TlKqjYKInQNYfb6Sk7N9POLMtgzoFLyjfbl1sdeLsSMmFGNQ5w8tMHM3LLkbe5fhILQhIKbC + 4iyUGylviUtVAgMBAAECggEAIPb0CQy0RJoX+q/lGbRVmnyJpYDf+115WNnl+mrwjdGkeZyqw4v0 + BPzkWYzUFP1esJRO6buBNFybQRFdFW0z5lvVv/zzRKq71aVUBPInxaMRyHuJ8D5lIL8nDtgVOwyE + 7DOGyDtURUMzMjdUwoTe7K+O6QBU4X/1pVPZYgmissYSMmt68LiP8k0p601F4+r5xOi/QEy44aVp + aOJZBUOisKB8BmUXZqmQ4Cy05vU9Xi1rLyzkn9s7fxnZ+JO6Sd1r0Thm1mE0yuPgxkDBh/b4f3/2 + GsQNKKKCiij/6TfkjnBi8ZvWR44LnKpu760g/K7psVNrKwqJG6C/8RAcgISWQQKBgQDop7BaKGhK + 1QMJJ/vnlyYFTucfGLn6bM//pzTys5Gop0tpcfX/Hf6a6Dd+zBhmC3tBmhr80XOX/PiyAIbc0lOI + 31rafZuD/oVx5mlIySWX35EqS14LXmdVs/5vOhsInNgNiE+EPFf1L9YZgG/zA7OUBmqtTeYIPDVC + 7ViJcydItQKBgQDFmK0H0IA6W4opGQo+zQKhefooqZ+RDk9IIZMPOAtnvOM7y3rSVrfsSjzYVuMS + w/RP/vs7rwhaZejnCZ8/7uIqwg4sdUBRzZYR3PRNFeheW+BPZvb+2keRCGzOs7xkbF1mu54qtYTa + HZGZj1OsD83AoMwVLcdLDgO1kw32dkS8IQKBgFRdgoifAHqqVah7VFB9se7Y1tyi5cXWsXI+Wufr + j9U9nQ4GojK52LqpnH4hWnOelDqMvF6TQTyLIk/B+yWWK26Ft/dk9wDdSdystd8L+dLh4k0Y+Whb + +lLMq2YABw+PeJUnqdYE38xsZVHoDjBsVjFGRmbDybeQxauYT7PACy3FAoGBAK2+k9bdNQMbXp7I + j8OszHVkJdz/WXlY1cmdDAxDwXOUGVKIlxTAf7TbiijILZ5gg0Cb+hj+zR9/oI0WXtr+mAv02jWp + W8cSOLS4TnBBpTLjIpdu+BwbnvYeLF6MmEjNKEufCXKQbaLEgTQ/XNlchBSuzwSIXkbWqdhM1+gx + EjtBAoGARAdMIiDMPWIIZg3nNnFebbmtBP0qiBsYohQZ+6i/8s/vautEHBEN6Q0brIU/goo+nTHc + t9VaOkzjCmAJSLPUanuBC8pdYgLu5J20NXUZLD9AE/2bBT3OpezKcdYeI2jqoc1qlWHlNtVtdqQ2 + AcZSFJQjdg5BTyvdEDhaYUKGdRw= + -----END PRIVATE KEY----- + EOS + ``` + +1. Save the file and [reconfigure](../../restart_gitlab.md#omnibus-gitlab-reconfigure) GitLab for the changes to take effect. + +--- + +**For installations from source** + +1. Edit `config/gitlab.yml`: + + ```yaml + ldap: + enabled: true + servers: + main: # 'main' is the GitLab 'provider ID' of this LDAP server + label: 'Google Secure LDAP' + + host: 'ldap.google.com' + port: 636 + uid: 'uid' + bind_dn: 'DizzyHorse' + password: 'd6V5H8nhMUW9AuDP25abXeLd' + encryption: 'simple_tls' + verify_certificates: true + + tls_options: + cert: | + -----BEGIN CERTIFICATE----- + MIIDbDCCAlSgAwIBAgIGAWlzxiIfMA0GCSqGSIb3DQEBCwUAMHcxFDASBgNVBAoTC0dvb2dsZSBJ + bmMuMRYwFAYDVQQHEw1Nb3VudGFpbiBWaWV3MRQwEgYDVQQDEwtMREFQIENsaWVudDEPMA0GA1UE + CxMGR1N1aXRlMQswCQYDVQQGEwJVUzETMBEGA1UECBMKQ2FsaWZvcm5pYTAeFw0xOTAzMTIyMTE5 + MThaFw0yMjAzMTEyMTE5MThaMHcxFDASBgNVBAoTC0dvb2dsZSBJbmMuMRYwFAYDVQQHEw1Nb3Vu + dGFpbiBWaWV3MRQwEgYDVQQDEwtMREFQIENsaWVudDEPMA0GA1UECxMGR1N1aXRlMQswCQYDVQQG + EwJVUzETMBEGA1UECBMKQ2FsaWZvcm5pYTCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEB + ALOTy4aC38dyjESk6N8fRsKk8DN23ZX/GaNFL5OUmmA1KWzrvVC881OzNdtGm3vNOIxr9clteEG/ + tQwsmsJvQT5U+GkBt+tGKF/zm7zueHUYqTP7Pg5pxAnAei90qkIRFi17ulObyRHPYv1BbCt8pxNB + 4fG/gAXkFbCNxwh1eiQXXRTfruasCZ4/mHfX7MVm8JmWU9uAVIOLW+DSWOFhrDQduJdGBXJOyC2r + Gqoeg9+tkBmNH/jjxpnEkFW8q7io9DdOUqqNgoidA1h9vpKTs3084sy2DOgUvKN9uXWx14uxIyYU + Y1DnDy0wczcsuRt7l+EgtCEgpsLiLJQbKW+JS1UCAwEAATANBgkqhkiG9w0BAQsFAAOCAQEAf60J + yazhbHkDKIH2gFxfm7QLhhnqsmafvl4WP7JqZt0u0KdnvbDPfokdkM87yfbKJU1MTI86M36wEC+1 + P6bzklKz7kXbzAD4GggksAzxsEE64OWHC+Y64Tkxq2NiZTw/76POkcg9StiIXjG0ZcebHub9+Ux/ + rTncip92nDuvgEM7lbPFKRIS/YMhLCk09B/U0F6XLsf1yYjyf5miUTDikPkov23b/YGfpc8kh6hq + 1kqdi6a1cYPP34eAhtRhMqcZU9qezpJF6s9EeN/3YFfKzLODFSsVToBRAdZgGHzj//SAtLyQTD4n + KCSvK1UmaMxNaZyTHg8JnMf0ZuRpv26iSg== + -----END CERTIFICATE----- + + key: | + -----BEGIN PRIVATE KEY----- + MIIEvQIBADANBgkqhkiG9w0BAQEFAASCBKcwggSjAgEAAoIBAQCzk8uGgt/HcoxEpOjfH0bCpPAz + dt2V/xmjRS+TlJpgNSls671QvPNTszXbRpt7zTiMa/XJbXhBv7UMLJrCb0E+VPhpAbfrRihf85u8 + 7nh1GKkz+z4OacQJwHovdKpCERYte7pTm8kRz2L9QWwrfKcTQeHxv4AF5BWwjccIdXokF10U367m + rAmeP5h31+zFZvCZllPbgFSDi1vg0ljhYaw0HbiXRgVyTsgtqxqqHoPfrZAZjR/448aZxJBVvKu4 + qPQ3TlKqjYKInQNYfb6Sk7N9POLMtgzoFLyjfbl1sdeLsSMmFGNQ5w8tMHM3LLkbe5fhILQhIKbC + 4iyUGylviUtVAgMBAAECggEAIPb0CQy0RJoX+q/lGbRVmnyJpYDf+115WNnl+mrwjdGkeZyqw4v0 + BPzkWYzUFP1esJRO6buBNFybQRFdFW0z5lvVv/zzRKq71aVUBPInxaMRyHuJ8D5lIL8nDtgVOwyE + 7DOGyDtURUMzMjdUwoTe7K+O6QBU4X/1pVPZYgmissYSMmt68LiP8k0p601F4+r5xOi/QEy44aVp + aOJZBUOisKB8BmUXZqmQ4Cy05vU9Xi1rLyzkn9s7fxnZ+JO6Sd1r0Thm1mE0yuPgxkDBh/b4f3/2 + GsQNKKKCiij/6TfkjnBi8ZvWR44LnKpu760g/K7psVNrKwqJG6C/8RAcgISWQQKBgQDop7BaKGhK + 1QMJJ/vnlyYFTucfGLn6bM//pzTys5Gop0tpcfX/Hf6a6Dd+zBhmC3tBmhr80XOX/PiyAIbc0lOI + 31rafZuD/oVx5mlIySWX35EqS14LXmdVs/5vOhsInNgNiE+EPFf1L9YZgG/zA7OUBmqtTeYIPDVC + 7ViJcydItQKBgQDFmK0H0IA6W4opGQo+zQKhefooqZ+RDk9IIZMPOAtnvOM7y3rSVrfsSjzYVuMS + w/RP/vs7rwhaZejnCZ8/7uIqwg4sdUBRzZYR3PRNFeheW+BPZvb+2keRCGzOs7xkbF1mu54qtYTa + HZGZj1OsD83AoMwVLcdLDgO1kw32dkS8IQKBgFRdgoifAHqqVah7VFB9se7Y1tyi5cXWsXI+Wufr + j9U9nQ4GojK52LqpnH4hWnOelDqMvF6TQTyLIk/B+yWWK26Ft/dk9wDdSdystd8L+dLh4k0Y+Whb + +lLMq2YABw+PeJUnqdYE38xsZVHoDjBsVjFGRmbDybeQxauYT7PACy3FAoGBAK2+k9bdNQMbXp7I + j8OszHVkJdz/WXlY1cmdDAxDwXOUGVKIlxTAf7TbiijILZ5gg0Cb+hj+zR9/oI0WXtr+mAv02jWp + W8cSOLS4TnBBpTLjIpdu+BwbnvYeLF6MmEjNKEufCXKQbaLEgTQ/XNlchBSuzwSIXkbWqdhM1+gx + EjtBAoGARAdMIiDMPWIIZg3nNnFebbmtBP0qiBsYohQZ+6i/8s/vautEHBEN6Q0brIU/goo+nTHc + t9VaOkzjCmAJSLPUanuBC8pdYgLu5J20NXUZLD9AE/2bBT3OpezKcdYeI2jqoc1qlWHlNtVtdqQ2 + AcZSFJQjdg5BTyvdEDhaYUKGdRw= + -----END PRIVATE KEY----- + ``` + +1. Save the file and [restart](../../restart_gitlab.md#installations-from-source) GitLab for the changes to take effect. + +<!-- ## Troubleshooting + +Include any troubleshooting steps that you can foresee. If you know beforehand what issues +one might have when setting this up, or when something is changed, or on upgrading, it's +important to describe those, too. Think of things that may go wrong and include them here. +This is important to minimize requests for support, and to avoid doc comments with +questions that you know someone might ask. + +Each scenario can be a third-level heading, e.g. `### Getting error message X`. +If you have none to add when creating a doc, leave this section in place +but commented out to help encourage others to add to it in the future. --> diff --git a/doc/administration/auth/img/google_secure_ldap_add_step_1.png b/doc/administration/auth/ldap/img/google_secure_ldap_add_step_1.png Binary files differindex bee9c602a14..bee9c602a14 100644 --- a/doc/administration/auth/img/google_secure_ldap_add_step_1.png +++ b/doc/administration/auth/ldap/img/google_secure_ldap_add_step_1.png diff --git a/doc/administration/auth/img/google_secure_ldap_add_step_2.png b/doc/administration/auth/ldap/img/google_secure_ldap_add_step_2.png Binary files differindex b127410cb8c..b127410cb8c 100644 --- a/doc/administration/auth/img/google_secure_ldap_add_step_2.png +++ b/doc/administration/auth/ldap/img/google_secure_ldap_add_step_2.png diff --git a/doc/administration/auth/img/google_secure_ldap_client_settings.png b/doc/administration/auth/ldap/img/google_secure_ldap_client_settings.png Binary files differindex 868e6645f56..868e6645f56 100644 --- a/doc/administration/auth/img/google_secure_ldap_client_settings.png +++ b/doc/administration/auth/ldap/img/google_secure_ldap_client_settings.png diff --git a/doc/administration/auth/ldap/img/multi_login.gif b/doc/administration/auth/ldap/img/multi_login.gif Binary files differnew file mode 100644 index 00000000000..5aee6090793 --- /dev/null +++ b/doc/administration/auth/ldap/img/multi_login.gif diff --git a/doc/administration/auth/ldap/index.md b/doc/administration/auth/ldap/index.md new file mode 100644 index 00000000000..4a7a972596f --- /dev/null +++ b/doc/administration/auth/ldap/index.md @@ -0,0 +1,756 @@ +--- +type: reference +stage: Manage +group: Access +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + +# General LDAP Setup + +GitLab integrates with LDAP to support user authentication. + +This integration works with most LDAP-compliant directory servers, including: + +- Microsoft Active Directory +- Apple Open Directory +- Open LDAP +- 389 Server + +GitLab Enterprise Editions (EE) include enhanced integration, +including group membership syncing as well as multiple LDAP servers support. + +NOTE: **Note:** +[Microsoft Active Directory Trusts](https://docs.microsoft.com/en-us/previous-versions/windows/it-pro/windows-server-2008-R2-and-2008/cc771568(v=ws.10)) are not supported. + +## Overview + +[LDAP](https://en.wikipedia.org/wiki/Lightweight_Directory_Access_Protocol) +stands for **Lightweight Directory Access Protocol**, which is a standard +application protocol for accessing and maintaining distributed directory +information services over an Internet Protocol (IP) network. + +## Security **(CORE ONLY)** + +GitLab assumes that LDAP users: + +- Are not able to change their LDAP `mail`, `email`, or `userPrincipalName` attributes. + An LDAP user who is allowed to change their email on the LDAP server can potentially + [take over any account](#enabling-ldap-sign-in-for-existing-gitlab-users-core-only) + on your GitLab server. +- Have unique email addresses, otherwise it is possible for LDAP users with the same + email address to share the same GitLab account. + +We recommend against using LDAP integration if your LDAP users are +allowed to change their 'mail', 'email' or 'userPrincipalName' attribute on +the LDAP server or share email addresses. + +### User deletion **(CORE ONLY)** + +If a user is deleted from the LDAP server, they will be blocked in GitLab as +well. Users will be immediately blocked from logging in. However, there is an +LDAP check cache time of one hour (see note) which means users that +are already logged in or are using Git over SSH will still be able to access +GitLab for up to one hour. Manually block the user in the GitLab Admin Area to +immediately block all access. + +NOTE: **Note**: +GitLab Enterprise Edition Starter supports a +[configurable sync time](#adjusting-ldap-user-sync-schedule-starter-only). + +## Git password authentication **(CORE ONLY)** + +LDAP-enabled users can always authenticate with Git using their GitLab username +or email and LDAP password, even if password authentication for Git is disabled +in the application settings. + +## Enabling LDAP sign-in for existing GitLab users **(CORE ONLY)** + +When a user signs in to GitLab with LDAP for the first time, and their LDAP +email address is the primary email address of an existing GitLab user, then +the LDAP DN will be associated with the existing user. If the LDAP email +attribute is not found in GitLab's database, a new user is created. + +In other words, if an existing GitLab user wants to enable LDAP sign-in for +themselves, they should check that their GitLab email address matches their +LDAP email address, and then sign into GitLab via their LDAP credentials. + +## Google Secure LDAP **(CORE ONLY)** + +> Introduced in GitLab 11.9. + +[Google Cloud Identity](https://cloud.google.com/identity/) provides a Secure +LDAP service that can be configured with GitLab for authentication and group sync. +See [Google Secure LDAP](google_secure_ldap.md) for detailed configuration instructions. + +## Configuration **(CORE ONLY)** + +To enable LDAP integration you need to add your LDAP server settings in +`/etc/gitlab/gitlab.rb` or `/home/git/gitlab/config/gitlab.yml` for Omnibus +GitLab and installations from source respectively. + +There is a Rake task to check LDAP configuration. After configuring LDAP +using the documentation below, see [LDAP check Rake task](../../raketasks/check.md#ldap-check) +for information on the LDAP check Rake task. + +NOTE: **Note:** +The `encryption` value `simple_tls` corresponds to 'Simple TLS' in the LDAP +library. `start_tls` corresponds to StartTLS, not to be confused with regular TLS. +Normally, if you specify `simple_tls` it will be on port 636, while `start_tls` (StartTLS) +would be on port 389. `plain` also operates on port 389. Removed values: `tls` was replaced with `start_tls` and `ssl` was replaced with `simple_tls`. + +NOTE: **Note:** +LDAP users must have an email address set, regardless of whether it is used to log in. + +### Example Configurations **(CORE ONLY)** + +**Omnibus Configuration** + +```ruby +gitlab_rails['ldap_enabled'] = true +gitlab_rails['prevent_ldap_sign_in'] = false +gitlab_rails['ldap_servers'] = { +'main' => { + 'label' => 'LDAP', + 'host' => 'ldap.mydomain.com', + 'port' => 389, + 'uid' => 'sAMAccountName', + 'encryption' => 'simple_tls', + 'verify_certificates' => true, + 'bind_dn' => '_the_full_dn_of_the_user_you_will_bind_with', + 'password' => '_the_password_of_the_bind_user', + 'encryption' => 'plain', + 'verify_certificates' => true, + 'tls_options' => { + 'ca_file' => '', + 'ssl_version' => '', + 'ciphers' => '', + 'cert' => '', + 'key' => '' + }, + 'timeout' => 10, + 'active_directory' => true, + 'allow_username_or_email_login' => false, + 'block_auto_created_users' => false, + 'base' => 'dc=example,dc=com', + 'user_filter' => '', + 'attributes' => { + 'username' => ['uid', 'userid', 'sAMAccountName'], + 'email' => ['mail', 'email', 'userPrincipalName'], + 'name' => 'cn', + 'first_name' => 'givenName', + 'last_name' => 'sn' + }, + 'lowercase_usernames' => false, + + # EE Only + 'group_base' => '', + 'admin_group' => '', + 'external_groups' => [], + 'sync_ssh_keys' => false + } +} +``` + +**Source Configuration** + +```yaml +production: + # snip... + ldap: + enabled: false + prevent_ldap_sign_in: false + servers: + main: + label: 'LDAP' + ... +``` + +### Basic Configuration Settings **(CORE ONLY)** + +| Setting | Description | Required | Examples | +| ------- | ----------- | -------- | -------- | +| `label` | A human-friendly name for your LDAP server. It will be displayed on your login page. | yes | `'Paris'` or `'Acme, Ltd.'` | +| `host` | IP address or domain name of your LDAP server. | yes | `'ldap.mydomain.com'` | +| `port` | The port to connect with on your LDAP server. Always an integer, not a string. | yes | `389` or `636` (for SSL) | +| `uid` | LDAP attribute for username. Should be the attribute, not the value that maps to the `uid`. | yes | `'sAMAccountName'`, `'uid'`, `'userPrincipalName'` | +| `bind_dn` | The full DN of the user you will bind with. | no | `'america\momo'` or `'CN=Gitlab,OU=Users,DC=domain,DC=com'` | +| `password` | The password of the bind user. | no | `'your_great_password'` | +| `encryption` | Encryption method. The `method` key is deprecated in favor of `encryption`. | yes | `'start_tls'` or `'simple_tls'` or `'plain'` | +| `verify_certificates` | Enables SSL certificate verification if encryption method is `start_tls` or `simple_tls`. Defaults to true. | no | boolean | +| `timeout` | Set a timeout, in seconds, for LDAP queries. This helps avoid blocking a request if the LDAP server becomes unresponsive. A value of 0 means there is no timeout. | no | `10` or `30` | +| `active_directory` | This setting specifies if LDAP server is Active Directory LDAP server. For non-AD servers it skips the AD specific queries. If your LDAP server is not AD, set this to false. | no | boolean | +| `allow_username_or_email_login` | If enabled, GitLab will ignore everything after the first `@` in the LDAP username submitted by the user on login. If you are using `uid: 'userPrincipalName'` on ActiveDirectory you need to disable this setting, because the userPrincipalName contains an `@`. | no | boolean | +| `block_auto_created_users` | To maintain tight control over the number of active users on your GitLab installation, enable this setting to keep new users blocked until they have been cleared by the admin (default: false). | no | boolean | +| `base` | Base where we can search for users. | yes | `'ou=people,dc=gitlab,dc=example'` or `'DC=mydomain,DC=com'` | +| `user_filter` | Filter LDAP users. Format: [RFC 4515](https://tools.ietf.org/search/rfc4515) Note: GitLab does not support `omniauth-ldap`'s custom filter syntax. | no | `'(employeeType=developer)'` or `'(&(objectclass=user)(|(samaccountname=momo)(samaccountname=toto)))'` | +| `lowercase_usernames` | If lowercase_usernames is enabled, GitLab will lower case the username. | no | boolean | + +### SSL Configuration Settings **(CORE ONLY)** + +| Setting | Description | Required | Examples | +| ------- | ----------- | -------- | -------- | +| `ca_file` | Specifies the path to a file containing a PEM-format CA certificate, e.g. if you need to use an internal CA. | no | `'/etc/ca.pem'` | +| `ssl_version` | Specifies the SSL version for OpenSSL to use, if the OpenSSL default is not appropriate. | no | `'TLSv1_1'` | +| `ciphers` | Specific SSL ciphers to use in communication with LDAP servers. | no | `'ALL:!EXPORT:!LOW:!aNULL:!eNULL:!SSLv2'` | +| `cert` | Client certificate | no | `'-----BEGIN CERTIFICATE----- <REDACTED> -----END CERTIFICATE -----'` | +| `key` | Client private key | no | `'-----BEGIN PRIVATE KEY----- <REDACTED> -----END PRIVATE KEY -----'` | + +### Attribute Configuration Settings **(CORE ONLY)** + +LDAP attributes that GitLab will use to create an account for the LDAP user. The specified attribute can either be the attribute name as a string (e.g. `'mail'`), or an array of attribute names to try in order (e.g. `['mail', 'email']`). Note that the user's LDAP login will always be the attribute specified as `uid` above. + +| Setting | Description | Required | Examples | +| ------- | ----------- | -------- | -------- | +| `username` | The username will be used in paths for the user's own projects (like `gitlab.example.com/username/project`) and when mentioning them in issues, merge request and comments (like `@username`). If the attribute specified for `username` contains an email address, the GitLab username will be the part of the email address before the `@`. | no | `['uid', 'userid', 'sAMAccountName']` | +| `email` | LDAP attribute for user email. | no | `['mail', 'email', 'userPrincipalName']` | +| `name` | LDAP attribute for user display name. If no full name could be found at the attribute specified for `name`, the full name is determined using the attributes specified for `first_name` and `last_name`. | no | `'cn'` or `'displayName'` | +| `first_name` | LDAP attribute for user first name. | no | `'givenName'` | +| `last_name` | LDAP attribute for user last name. | no | `'sn'` | + +### LDAP Sync Configuration Settings **(STARTER ONLY)** + +| Setting | Description | Required | Examples | +| ------- | ----------- | -------- | -------- | +| `group_base` | Base used to search for groups. | no | `'ou=groups,dc=gitlab,dc=example'` | +| `admin_group` | The CN of a group containing GitLab administrators. Note: Not `cn=administrators` or the full DN. | no | `'administrators'` | +| `external_groups` | An array of CNs of groups containing users that should be considered external. Note: Not `cn=interns` or the full DN. | no | `['interns', 'contractors']` | +| `sync_ssh_keys` | The LDAP attribute containing a user's public SSH key. | no | `'sshPublicKey'` or false if not set | + +### Set up LDAP user filter **(CORE ONLY)** + +If you want to limit all GitLab access to a subset of the LDAP users on your +LDAP server, the first step should be to narrow the configured `base`. However, +it is sometimes necessary to filter users further. In this case, you can set up +an LDAP user filter. The filter must comply with +[RFC 4515](https://tools.ietf.org/search/rfc4515). + +**Omnibus configuration** + +```ruby +gitlab_rails['ldap_servers'] = { +'main' => { + # snip... + 'user_filter' => '(employeeType=developer)' + } +} +``` + +**Source configuration** + +```yaml +production: + ldap: + servers: + main: + # snip... + user_filter: '(employeeType=developer)' +``` + +If you want to limit access to the nested members of an Active Directory +group, you can use the following syntax: + +```plaintext +(memberOf:1.2.840.113556.1.4.1941:=CN=My Group,DC=Example,DC=com) +``` + +For more information about this "LDAP_MATCHING_RULE_IN_CHAIN" filter, see the following +[Microsoft Search Filter Syntax](https://docs.microsoft.com/en-us/windows/win32/adsi/search-filter-syntax) document. +Support for nested members in the user filter should not be confused with +[group sync nested groups support](#supported-ldap-group-typesattributes). **(STARTER ONLY)** + +Please note that GitLab does not support the custom filter syntax used by +OmniAuth LDAP. + +#### Escaping special characters **(CORE ONLY)** + +The `user_filter` DN can contain special characters. For example: + +- A comma: + + ```plaintext + OU=GitLab, Inc,DC=gitlab,DC=com + ``` + +- Open and close brackets: + + ```plaintext + OU=Gitlab (Inc),DC=gitlab,DC=com + ``` + + These characters must be escaped as documented in + [RFC 4515](https://tools.ietf.org/search/rfc4515). + +- Escape commas with `\2C`. For example: + + ```plaintext + OU=GitLab\2C Inc,DC=gitlab,DC=com + ``` + +- Escape open and close brackets with `\28` and `\29`, respectively. For example: + + ```plaintext + OU=Gitlab \28Inc\29,DC=gitlab,DC=com + ``` + +### Enabling LDAP username lowercase **(CORE ONLY)** + +Some LDAP servers, depending on their configurations, can return uppercase usernames. +This can lead to several confusing issues such as creating links or namespaces with uppercase names. + +GitLab can automatically lowercase usernames provided by the LDAP server by enabling +the configuration option `lowercase_usernames`. By default, this configuration option is `false`. + +**Omnibus configuration** + +1. Edit `/etc/gitlab/gitlab.rb`: + + ```ruby + gitlab_rails['ldap_servers'] = { + 'main' => { + # snip... + 'lowercase_usernames' => true + } + } + ``` + +1. [Reconfigure GitLab](../../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. + +**Source configuration** + +1. Edit `config/gitlab.yaml`: + + ```yaml + production: + ldap: + servers: + main: + # snip... + lowercase_usernames: true + ``` + +1. [Restart GitLab](../../restart_gitlab.md#installations-from-source) for the changes to take effect. + +### Disable LDAP web sign in **(CORE ONLY)** + +It can be useful to prevent using LDAP credentials through the web UI when +an alternative such as SAML is preferred. This allows LDAP to be used for group +sync, while also allowing your SAML identity provider to handle additional +checks like custom 2FA. + +When LDAP web sign in is disabled, users will not see a **LDAP** tab on the sign in page. +This does not disable [using LDAP credentials for Git access](#git-password-authentication-core-only). + +**Omnibus configuration** + +1. Edit `/etc/gitlab/gitlab.rb`: + + ```ruby + gitlab_rails['prevent_ldap_sign_in'] = true + ``` + +1. [Reconfigure GitLab](../../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. + +**Source configuration** + +1. Edit `config/gitlab.yaml`: + + ```yaml + production: + ldap: + prevent_ldap_sign_in: true + ``` + +1. [Restart GitLab](../../restart_gitlab.md#installations-from-source) for the changes to take effect. + +## Encryption **(CORE ONLY)** + +### TLS Server Authentication + +There are two encryption methods, `simple_tls` and `start_tls`. + +For either encryption method, if setting `verify_certificates: false`, TLS +encryption is established with the LDAP server before any LDAP-protocol data is +exchanged but no validation of the LDAP server's SSL certificate is performed. + +### Limitations + +#### TLS Client Authentication + +Not implemented by `Net::LDAP`. + +You should disable anonymous LDAP authentication and enable simple or SASL +authentication. The TLS client authentication setting in your LDAP server cannot +be mandatory and clients cannot be authenticated with the TLS protocol. + +## Multiple LDAP servers **(STARTER ONLY)** + +With GitLab Enterprise Edition Starter, you can configure multiple LDAP servers +that your GitLab instance will connect to. + +To add another LDAP server: + +1. Duplicate the settings under [the main configuration](#configuration-core-only). +1. Edit them to match the additional LDAP server. + +Be sure to choose a different provider ID made of letters a-z and numbers 0-9. +This ID will be stored in the database so that GitLab can remember which LDAP +server a user belongs to. + + + +Based on the example illustrated on the image above, +our `gitlab.rb` configuration would look like: + +```ruby +gitlab_rails['ldap_enabled'] = true +gitlab_rails['ldap_servers'] = { +'main' => { + 'label' => 'GitLab AD', + 'host' => 'ad.example.org', + 'port' => 636, + ... + }, + +'secondary' => { + 'label' => 'GitLab Secondary AD', + 'host' => 'ad-secondary.example.net', + 'port' => 636, + ... + }, + +'tertiary' => { + 'label' => 'GitLab Tertiary AD', + 'host' => 'ad-tertiary.example.net', + 'port' => 636, + ... + } + +} +``` + +NOTE: **Note:** +Any number of LDAP servers can be configured. However, make sure to use a unique naming convention for the `label` section of each entry as this will be the display name of the tab shown on the sign-in page. + +## User sync **(STARTER ONLY)** + +Once per day, GitLab runs a worker to check and update GitLab +users against LDAP. + +The process executes the following access checks: + +- Ensure the user is still present in LDAP. +- If the LDAP server is Active Directory, ensure the user is active (not + blocked/disabled state). This will only be checked if + `active_directory: true` is set in the LDAP configuration. + +NOTE: **Note:** +In Active Directory, a user is marked as disabled/blocked if the user +account control attribute (`userAccountControl:1.2.840.113556.1.4.803`) +has bit 2 set. See <https://ctovswild.com/2009/09/03/bitmask-searches-in-ldap/> +for more information. + +The user will be set to `ldap_blocked` state in GitLab if the above conditions +fail. This means the user will not be able to login or push/pull code. + +The process will also update the following user information: + +- Email address. +- If `sync_ssh_keys` is set, SSH public keys. +- If Kerberos is enabled, Kerberos identity. + +NOTE: **Note:** +The LDAP sync process updates existing users while new users are created on first sign in. + +### Adjusting LDAP user sync schedule **(STARTER ONLY)** + +NOTE: **Note:** +These are cron formatted values. You can use a crontab generator to create +these values, for example <http://www.crontabgenerator.com/>. + +By default, GitLab will run a worker once per day at 01:30 a.m. server time to +check and update GitLab users against LDAP. + +You can manually configure LDAP user sync times by setting the +following configuration values. The example below shows how to set LDAP user +sync to run once every 12 hours at the top of the hour. + +**Omnibus installations** + +1. Edit `/etc/gitlab/gitlab.rb`: + + ```ruby + gitlab_rails['ldap_sync_worker_cron'] = "0 */12 * * *" + ``` + +1. [Reconfigure GitLab](../../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. + +**Source installations** + +1. Edit `config/gitlab.yaml`: + + ```yaml + cron_jobs: + ldap_sync_worker_cron: + "0 */12 * * *" + ``` + +1. [Restart GitLab](../../restart_gitlab.md#installations-from-source) for the changes to take effect. + +## Group Sync **(STARTER ONLY)** + +If your LDAP supports the `memberof` property, when the user signs in for the +first time GitLab will trigger a sync for groups the user should be a member of. +That way they don't need to wait for the hourly sync to be granted +access to their groups and projects. + +A group sync process will run every hour on the hour, and `group_base` must be set +in LDAP configuration for LDAP synchronizations based on group CN to work. This allows +GitLab group membership to be automatically updated based on LDAP group members. + +The `group_base` configuration should be a base LDAP 'container', such as an +'organization' or 'organizational unit', that contains LDAP groups that should +be available to GitLab. For example, `group_base` could be +`ou=groups,dc=example,dc=com`. In the config file it will look like the +following. + +**Omnibus configuration** + +1. Edit `/etc/gitlab/gitlab.rb`: + + ```ruby + gitlab_rails['ldap_servers'] = { + 'main' => { + # snip... + 'group_base' => 'ou=groups,dc=example,dc=com', + } + } + ``` + +1. [Apply your changes to GitLab](../../restart_gitlab.md#omnibus-gitlab-reconfigure). + +**Source configuration** + +1. Edit `/home/git/gitlab/config/gitlab.yml`: + + ```yaml + production: + ldap: + servers: + main: + # snip... + group_base: ou=groups,dc=example,dc=com + ``` + +1. [Restart GitLab](../../restart_gitlab.md#installations-from-source) for the changes to take effect. + +To take advantage of group sync, group owners or maintainers will need to [create one +or more LDAP group links](#adding-group-links-starter-only). + +### Adding group links **(STARTER ONLY)** + +For information on adding group links via CNs and filters, refer to [the GitLab groups documentation](../../../user/group/index.md#manage-group-memberships-via-ldap). + +### Administrator sync **(STARTER ONLY)** + +As an extension of group sync, you can automatically manage your global GitLab +administrators. Specify a group CN for `admin_group` and all members of the +LDAP group will be given administrator privileges. The configuration will look +like the following. + +NOTE: **Note:** +Administrators will not be synced unless `group_base` is also +specified alongside `admin_group`. Also, only specify the CN of the admin +group, as opposed to the full DN. + +**Omnibus configuration** + +1. Edit `/etc/gitlab/gitlab.rb`: + + ```ruby + gitlab_rails['ldap_servers'] = { + 'main' => { + # snip... + 'group_base' => 'ou=groups,dc=example,dc=com', + 'admin_group' => 'my_admin_group', + } + } + ``` + +1. [Apply your changes to GitLab](../../restart_gitlab.md#omnibus-gitlab-reconfigure). + +**Source configuration** + +1. Edit `/home/git/gitlab/config/gitlab.yml`: + + ```yaml + production: + ldap: + servers: + main: + # snip... + group_base: ou=groups,dc=example,dc=com + admin_group: my_admin_group + ``` + +1. [Restart GitLab](../../restart_gitlab.md#installations-from-source) for the changes to take effect. + +### Global group memberships lock **(STARTER ONLY)** + +"Lock memberships to LDAP synchronization" setting allows instance administrators +to lock down user abilities to invite new members to a group. + +When enabled, the following applies: + +- Only administrator can manage memberships of any group including access levels. +- Users are not allowed to share project with other groups or invite members to + a project created in a group. + +### Adjusting LDAP group sync schedule **(STARTER ONLY)** + +NOTE: **Note:** +These are cron formatted values. You can use a crontab generator to create +these values, for example [Crontab Generator](http://www.crontabgenerator.com/). + +By default, GitLab runs a group sync process every hour, on the hour. + +CAUTION: **Important:** +It's recommended that you do not start the sync process too frequently as this +could lead to multiple syncs running concurrently. This is primarily a concern +for installations with a large number of LDAP users. Please review the +[LDAP group sync benchmark metrics](#benchmarks) to see how +your installation compares before proceeding. + +You can manually configure LDAP group sync times by setting the +following configuration values. The example below shows how to set group +sync to run once every 2 hours at the top of the hour. + +**Omnibus installations** + +1. Edit `/etc/gitlab/gitlab.rb`: + + ```ruby + gitlab_rails['ldap_group_sync_worker_cron'] = "0 */2 * * * *" + ``` + +1. [Reconfigure GitLab](../../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. + +**Source installations** + +1. Edit `config/gitlab.yaml`: + + ```yaml + cron_jobs: + ldap_group_sync_worker_cron: + "*/30 * * * *" + ``` + +1. [Restart GitLab](../../restart_gitlab.md#installations-from-source) for the changes to take effect. + +### External groups **(STARTER ONLY)** + +Using the `external_groups` setting will allow you to mark all users belonging +to these groups as [external users](../../../user/permissions.md#external-users-core-only). +Group membership is checked periodically through the `LdapGroupSync` background +task. + +**Omnibus configuration** + +1. Edit `/etc/gitlab/gitlab.rb`: + + ```ruby + gitlab_rails['ldap_servers'] = { + 'main' => { + # snip... + 'external_groups' => ['interns', 'contractors'], + } + } + ``` + +1. [Apply your changes to GitLab](../../restart_gitlab.md#omnibus-gitlab-reconfigure). + +**Source configuration** + +1. Edit `config/gitlab.yaml`: + + ```yaml + production: + ldap: + servers: + main: + # snip... + external_groups: ['interns', 'contractors'] + ``` + +1. [Restart GitLab](../../restart_gitlab.md#installations-from-source) for the changes to take effect. + +### Group sync technical details + +There is a lot going on with group sync 'under the hood'. This section +outlines what LDAP queries are executed and what behavior you can expect +from group sync. + +Group member access will be downgraded from a higher level if their LDAP group +membership changes. For example, if a user has 'Owner' rights in a group and the +next group sync reveals they should only have 'Developer' privileges, their +access will be adjusted accordingly. The only exception is if the user is the +*last* owner in a group. Groups need at least one owner to fulfill +administrative duties. + +#### Supported LDAP group types/attributes + +GitLab supports LDAP groups that use member attributes: + +- `member` +- `submember` +- `uniquemember` +- `memberof` +- `memberuid`. + +This means group sync supports, at least, LDAP groups with the following object classes: +`groupOfNames`, `posixGroup`, and `groupOfUniqueNames`. + +Other object classes should work fine as long as members +are defined as one of the mentioned attributes. This also means GitLab supports +Microsoft Active Directory, Apple Open Directory, Open LDAP, and 389 Server. +Other LDAP servers should work, too. + +Active Directory also supports nested groups. Group sync will recursively +resolve membership if `active_directory: true` is set in the configuration file. + +NOTE: **Note:** +Nested group memberships are resolved only if the nested group +is found within the configured `group_base`. For example, if GitLab sees a +nested group with DN `cn=nested_group,ou=special_groups,dc=example,dc=com` but +the configured `group_base` is `ou=groups,dc=example,dc=com`, `cn=nested_group` +is ignored. + +#### Queries + +- Each LDAP group is queried a maximum of one time with base `group_base` and + filter `(cn=<cn_from_group_link>)`. +- If the LDAP group has the `memberuid` attribute, GitLab will execute another + LDAP query per member to obtain each user's full DN. These queries are + executed with base `base`, scope 'base object', and a filter depending on + whether `user_filter` is set. Filter may be `(uid=<uid_from_group>)` or a + joining of `user_filter`. + +#### Benchmarks + +Group sync was written to be as performant as possible. Data is cached, database +queries are optimized, and LDAP queries are minimized. The last benchmark run +revealed the following metrics: + +For 20000 LDAP users, 11000 LDAP groups and 1000 GitLab groups with 10 +LDAP group links each: + +- Initial sync (no existing members assigned in GitLab) took 1.8 hours +- Subsequent syncs (checking membership, no writes) took 15 minutes + +These metrics are meant to provide a baseline and performance may vary based on +any number of factors. This was a pretty extreme benchmark and most instances will +not have near this many users or groups. Disk speed, database performance, +network and LDAP server response time will affect these metrics. + +## Troubleshooting + +Please see our [administrator guide to troubleshooting LDAP](ldap-troubleshooting.md). diff --git a/doc/administration/auth/ldap/ldap-troubleshooting.md b/doc/administration/auth/ldap/ldap-troubleshooting.md new file mode 100644 index 00000000000..909802b5dec --- /dev/null +++ b/doc/administration/auth/ldap/ldap-troubleshooting.md @@ -0,0 +1,678 @@ +--- +type: reference +stage: Manage +group: Access +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + +# LDAP Troubleshooting for Administrators + +## Common Problems & Workflows + +### Connection + +#### Connection refused + +If you are getting `Connection Refused` errors when trying to connect to the +LDAP server please double-check the LDAP `port` and `encryption` settings used by +GitLab. Common combinations are `encryption: 'plain'` and `port: 389`, OR +`encryption: 'simple_tls'` and `port: 636`. + +#### Connection times out + +If GitLab cannot reach your LDAP endpoint, you will see a message like this: + +```plaintext +Could not authenticate you from Ldapmain because "Connection timed out - user specified timeout". +``` + +If your configured LDAP provider and/or endpoint is offline or otherwise +unreachable by GitLab, no LDAP user will be able to authenticate and log in. +GitLab does not cache or store credentials for LDAP users to provide authentication +during an LDAP outage. + +Contact your LDAP provider or administrator if you are seeing this error. + +#### Referral error + +If you see `LDAP search error: Referral` in the logs, or when troubleshooting +LDAP Group Sync, this error may indicate a configuration problem. The LDAP +configuration `/etc/gitlab/gitlab.rb` (Omnibus) or `config/gitlab.yml` (source) +is in YAML format and is sensitive to indentation. Check that `group_base` and +`admin_group` configuration keys are indented 2 spaces past the server +identifier. The default identifier is `main` and an example snippet looks like +the following: + +```yaml +main: # 'main' is the GitLab 'provider ID' of this LDAP server + label: 'LDAP' + host: 'ldap.example.com' + ... + group_base: 'cn=my_group,ou=groups,dc=example,dc=com' + admin_group: 'my_admin_group' +``` + +#### Query LDAP **(STARTER ONLY)** + +The following allows you to perform a search in LDAP using the rails console. +Depending on what you're trying to do, it may make more sense to query [a +user](#query-a-user-in-ldap) or [a group](#query-a-group-in-ldap-starter-only) directly, or +even [use `ldapsearch`](#ldapsearch) instead. + +```ruby +adapter = Gitlab::Auth::Ldap::Adapter.new('ldapmain') +options = { + # :base is required + # use .base or .group_base + base: adapter.config.group_base, + + # :filter is optional + # 'cn' looks for all "cn"s under :base + # '*' is the search string - here, it's a wildcard + filter: Net::Ldap::Filter.eq('cn', '*'), + + # :attributes is optional + # the attributes we want to get returned + attributes: %w(dn cn memberuid member submember uniquemember memberof) +} +adapter.ldap_search(options) +``` + +For examples of how this is run, +[review the `Adapter` module](https://gitlab.com/gitlab-org/gitlab/blob/master/ee/lib/ee/gitlab/auth/ldap/adapter.rb). + +### User logins + +#### No users are found + +If [you've confirmed](#ldap-check) that a connection to LDAP can be +established but GitLab doesn't show you LDAP users in the output, one of the +following is most likely true: + +- The `bind_dn` user doesn't have enough permissions to traverse the user tree. +- The user(s) don't fall under the [configured `base`](index.md#configuration-core-only). +- The [configured `user_filter`](index.md#set-up-ldap-user-filter-core-only) blocks access to the user(s). + +In this case, you con confirm which of the above is true using +[ldapsearch](#ldapsearch) with the existing LDAP configuration in your +`/etc/gitlab/gitlab.rb`. + +#### User(s) cannot login + +A user can have trouble logging in for any number of reasons. To get started, +here are some questions to ask yourself: + +- Does the user fall under the [configured `base`](index.md#configuration-core-only) in + LDAP? The user must fall under this `base` to login. +- Does the user pass through the [configured `user_filter`](index.md#set-up-ldap-user-filter-core-only)? + If one is not configured, this question can be ignored. If it is, then the + user must also pass through this filter to be allowed to login. + - Refer to our docs on [debugging the `user_filter`](#debug-ldap-user-filter). + +If the above are both okay, the next place to look for the problem is +the logs themselves while reproducing the issue. + +- Ask the user to login and let it fail. +- [Look through the output](#gitlab-logs) for any errors or other + messages about the login. You may see one of the other error messages on + this page, in which case that section can help resolve the issue. + +If the logs don't lead to the root of the problem, use the +[rails console](#rails-console) to [query this user](#query-a-user-in-ldap) +to see if GitLab can read this user on the LDAP server. + +It can also be helpful to +[debug a user sync](#sync-all-users-starter-only) to +investigate further. + +#### Invalid credentials on login + +If that the login credentials used are accurate on LDAP, ensure the following +are true for the user in question: + +- Make sure the user you are binding with has enough permissions to read the user's + tree and traverse it. +- Check that the `user_filter` is not blocking otherwise valid users. +- Run [an LDAP check command](#ldap-check) to make sure that the LDAP settings + are correct and [GitLab can see your users](#no-users-are-found). + +#### Email has already been taken + +A user tries to login with the correct LDAP credentials, is denied access, +and the [production.log](../../logs.md#productionlog) shows an error that looks like this: + +```plaintext +(LDAP) Error saving user <USER DN> (email@example.com): ["Email has already been taken"] +``` + +This error is referring to the email address in LDAP, `email@example.com`. Email +addresses must be unique in GitLab and LDAP links to a user's primary email (as opposed +to any of their possibly-numerous secondary emails). Another user (or even the +same user) has the email `email@example.com` set as a secondary email, which +is throwing this error. + +We can check where this conflicting email address is coming from using the +[rails console](#rails-console). Once in the console, run the following: + +```ruby +# This searches for an email among the primary AND secondary emails +user = User.find_by_any_email('email@example.com') +user.username +``` + +This will show you which user has this email address. One of two steps will +have to be taken here: + +- To create a new GitLab user/username for this user when logging in with LDAP, + remove the secondary email to remove the conflict. +- To use an existing GitLab user/username for this user to use with LDAP, + remove this email as a secondary email and make it a primary one so GitLab + will associate this profile to the LDAP identity. + +The user can do either of these steps [in their +profile](../../../user/profile/index.md#user-profile) or an admin can do it. + +#### Debug LDAP user filter + +[`ldapsearch`](#ldapsearch) allows you to test your configured +[user filter](index.md#set-up-ldap-user-filter-core-only) +to confirm that it returns the users you expect it to return. + +```shell +ldapsearch -H ldaps://$host:$port -D "$bind_dn" -y bind_dn_password.txt -b "$base" "$user_filter" sAMAccountName +``` + +- Variables beginning with a `$` refer to a variable from the LDAP section of + your configuration file. +- Replace `ldaps://` with `ldap://` if you are using the plain authentication method. + Port `389` is the default `ldap://` port and `636` is the default `ldaps://` + port. +- We are assuming the password for the `bind_dn` user is in `bind_dn_password.txt`. + +#### Sync all users **(STARTER ONLY)** + +The output from a manual [user sync](index.md#user-sync-starter-only) can show you what happens when +GitLab tries to sync its users against LDAP. Enter the [rails console](#rails-console) +and then run: + +```ruby +Rails.logger.level = Logger::DEBUG + +LdapSyncWorker.new.perform +``` + +Next, [learn how to read the +output](#example-console-output-after-a-user-sync-starter-only). + +##### Example console output after a user sync **(STARTER ONLY)** + +The output from a [manual user sync](#sync-all-users-starter-only) will be very verbose, and a +single user's successful sync can look like this: + +```shell +Syncing user John, email@example.com + Identity Load (0.9ms) SELECT "identities".* FROM "identities" WHERE "identities"."user_id" = 20 AND (provider LIKE 'ldap%') LIMIT 1 +Instantiating Gitlab::Auth::Ldap::Person with LDIF: +dn: cn=John Smith,ou=people,dc=example,dc=com +cn: John Smith +mail: email@example.com +memberof: cn=admin_staff,ou=people,dc=example,dc=com +uid: John + + UserSyncedAttributesMetadata Load (0.9ms) SELECT "user_synced_attributes_metadata".* FROM "user_synced_attributes_metadata" WHERE "user_synced_attributes_metadata"."user_id" = 20 LIMIT 1 + (0.3ms) BEGIN + Namespace Load (1.0ms) SELECT "namespaces".* FROM "namespaces" WHERE "namespaces"."owner_id" = 20 AND "namespaces"."type" IS NULL LIMIT 1 + Route Load (0.8ms) SELECT "routes".* FROM "routes" WHERE "routes"."source_id" = 27 AND "routes"."source_type" = 'Namespace' LIMIT 1 + Ci::Runner Load (1.1ms) SELECT "ci_runners".* FROM "ci_runners" INNER JOIN "ci_runner_namespaces" ON "ci_runners"."id" = "ci_runner_namespaces"."runner_id" WHERE "ci_runner_namespaces"."namespace_id" = 27 + (0.7ms) COMMIT + (0.4ms) BEGIN + Route Load (0.8ms) SELECT "routes".* FROM "routes" WHERE (LOWER("routes"."path") = LOWER('John')) + Namespace Load (1.0ms) SELECT "namespaces".* FROM "namespaces" WHERE "namespaces"."id" = 27 LIMIT 1 + Route Exists (0.9ms) SELECT 1 AS one FROM "routes" WHERE LOWER("routes"."path") = LOWER('John') AND "routes"."id" != 50 LIMIT 1 + User Update (1.1ms) UPDATE "users" SET "updated_at" = '2019-10-17 14:40:59.751685', "last_credential_check_at" = '2019-10-17 14:40:59.738714' WHERE "users"."id" = 20 +``` + +There's a lot here, so let's go over what could be helpful when debugging. + +First, GitLab will look for all users that have previously +logged in with LDAP and iterate on them. Each user's sync will start with +the following line that contains the user's username and email, as they +exist in GitLab now: + +```shell +Syncing user John, email@example.com +``` + +If you don't find a particular user's GitLab email in the output, then that +user hasn't logged in with LDAP yet. + +Next, GitLab searches its `identities` table for the existing +link between this user and the configured LDAP provider(s): + +```sql + Identity Load (0.9ms) SELECT "identities".* FROM "identities" WHERE "identities"."user_id" = 20 AND (provider LIKE 'ldap%') LIMIT 1 +``` + +The identity object will have the DN that GitLab will use to look for the user +in LDAP. If the DN isn't found, the email is used instead. We can see that +this user is found in LDAP: + +```shell +Instantiating Gitlab::Auth::Ldap::Person with LDIF: +dn: cn=John Smith,ou=people,dc=example,dc=com +cn: John Smith +mail: email@example.com +memberof: cn=admin_staff,ou=people,dc=example,dc=com +uid: John +``` + +If the user wasn't found in LDAP with either the DN or email, you may see the +following message instead: + +```shell +LDAP search error: No Such Object +``` + +...in which case the user will be blocked: + +```shell + User Update (0.4ms) UPDATE "users" SET "state" = $1, "updated_at" = $2 WHERE "users"."id" = $3 [["state", "ldap_blocked"], ["updated_at", "2019-10-18 15:46:22.902177"], ["id", 20]] +``` + +Once the user is found in LDAP the rest of the output will update the GitLab +database with any changes. + +#### Query a user in LDAP + +This will test that GitLab can reach out to LDAP and read a particular user. +It can expose potential errors connecting to and/or querying LDAP +that may seem to fail silently in the GitLab UI. + +```ruby +Rails.logger.level = Logger::DEBUG + +adapter = Gitlab::Auth::Ldap::Adapter.new('ldapmain') # If `main` is the LDAP provider +Gitlab::Auth::Ldap::Person.find_by_uid('<uid>', adapter) +``` + +### Group memberships **(STARTER ONLY)** + +#### Membership(s) not granted **(STARTER ONLY)** + +Sometimes you may think a particular user should be added to a GitLab group via +LDAP group sync, but for some reason it's not happening. There are several +things to check to debug the situation. + +- Ensure LDAP configuration has a `group_base` specified. + [This configuration](index.md#group-sync-starter-only) is required for group sync to work properly. +- Ensure the correct [LDAP group link is added to the GitLab + group](index.md#adding-group-links-starter-only). +- Check that the user has an LDAP identity: + 1. Sign in to GitLab as an administrator user. + 1. Navigate to **Admin area -> Users**. + 1. Search for the user + 1. Open the user, by clicking on their name. Do not click 'Edit'. + 1. Navigate to the **Identities** tab. There should be an LDAP identity with + an LDAP DN as the 'Identifier'. If not, this user hasn't logged in with + LDAP yet and must do so first. +- You've waited an hour or [the configured + interval](index.md#adjusting-ldap-group-sync-schedule-starter-only) for the group to + sync. To speed up the process, either go to the GitLab group **Settings -> + Members** and press **Sync now** (sync one group) or [run the group sync Rake + task](../../raketasks/ldap.md#run-a-group-sync-starter-only) (sync all groups). + +If all of the above looks good, jump in to a little more advanced debugging in +the rails console. + +1. Enter the [rails console](#rails-console). +1. Choose a GitLab group to test with. This group should have an LDAP group link + already configured. +1. [Enable debug logging, find the above GitLab group, and sync it with LDAP](#sync-one-group-starter-only). +1. Look through the output of the sync. See [example log + output](#example-console-output-after-a-group-sync-starter-only) + for how to read the output. +1. If you still aren't able to see why the user isn't being added, [query the + LDAP group directly](#query-a-group-in-ldap-starter-only) to see what members are listed. +1. Is the user's DN or UID in one of the lists from the above output? One of the DNs or + UIDs here should match the 'Identifier' from the LDAP identity checked earlier. If it doesn't, + the user does not appear to be in the LDAP group. + +#### Admin privileges not granted + +When [Administrator sync](index.md#administrator-sync-starter-only) has been configured +but the configured users aren't granted the correct admin privileges, confirm +the following are true: + +- A [`group_base` is also configured](index.md#group-sync-starter-only). +- The configured `admin_group` in the `gitlab.rb` is a CN, rather than a DN or an array. +- This CN falls under the scope of the configured `group_base`. +- The members of the `admin_group` have already logged into GitLab with their LDAP + credentials. GitLab will only grant this admin access to the users whose + accounts are already connected to LDAP. + +If all the above are true and the users are still not getting access, [run a manual +group sync](#sync-all-groups-starter-only) in the rails console and [look through the +output](#example-console-output-after-a-group-sync-starter-only) to see what happens when +GitLab syncs the `admin_group`. + +#### Sync all groups **(STARTER ONLY)** + +NOTE: **NOTE:** +To sync all groups manually when debugging is unnecessary, [use the Rake +task](../../raketasks/ldap.md#run-a-group-sync-starter-only) instead. + +The output from a manual [group sync](index.md#group-sync-starter-only) can show you what happens +when GitLab syncs its LDAP group memberships against LDAP. + +```ruby +Rails.logger.level = Logger::DEBUG + +LdapAllGroupsSyncWorker.new.perform +``` + +Next, [learn how to read the +output](#example-console-output-after-a-group-sync-starter-only). + +##### Example console output after a group sync **(STARTER ONLY)** + +Like the output from the user sync, the output from the [manual group +sync](#sync-all-groups-starter-only) will also be very verbose. However, it contains lots +of helpful information. + +Indicates the point where syncing actually begins: + +```shell +Started syncing 'ldapmain' provider for 'my_group' group +``` + +The following entry shows an array of all user DNs GitLab sees in the LDAP server. +Note that these are the users for a single LDAP group, not a GitLab group. If +you have multiple LDAP groups linked to this GitLab group, you will see multiple +log entries like this - one for each LDAP group. If you don't see an LDAP user +DN in this log entry, LDAP is not returning the user when we do the lookup. +Verify the user is actually in the LDAP group. + +```shell +Members in 'ldap_group_1' LDAP group: ["uid=john0,ou=people,dc=example,dc=com", +"uid=mary0,ou=people,dc=example,dc=com", "uid=john1,ou=people,dc=example,dc=com", +"uid=mary1,ou=people,dc=example,dc=com", "uid=john2,ou=people,dc=example,dc=com", +"uid=mary2,ou=people,dc=example,dc=com", "uid=john3,ou=people,dc=example,dc=com", +"uid=mary3,ou=people,dc=example,dc=com", "uid=john4,ou=people,dc=example,dc=com", +"uid=mary4,ou=people,dc=example,dc=com"] +``` + +Shortly after each of the above entries, you will see a hash of resolved member +access levels. This hash represents all user DNs GitLab thinks should have +access to this group, and at which access level (role). This hash is additive, +and more DNs may be added, or existing entries modified, based on additional +LDAP group lookups. The very last occurrence of this entry should indicate +exactly which users GitLab believes should be added to the group. + +NOTE: **Note:** +10 is 'Guest', 20 is 'Reporter', 30 is 'Developer', 40 is 'Maintainer' +and 50 is 'Owner'. + +```shell +Resolved 'my_group' group member access: {"uid=john0,ou=people,dc=example,dc=com"=>30, +"uid=mary0,ou=people,dc=example,dc=com"=>30, "uid=john1,ou=people,dc=example,dc=com"=>30, +"uid=mary1,ou=people,dc=example,dc=com"=>30, "uid=john2,ou=people,dc=example,dc=com"=>30, +"uid=mary2,ou=people,dc=example,dc=com"=>30, "uid=john3,ou=people,dc=example,dc=com"=>30, +"uid=mary3,ou=people,dc=example,dc=com"=>30, "uid=john4,ou=people,dc=example,dc=com"=>30, +"uid=mary4,ou=people,dc=example,dc=com"=>30} +``` + +It's not uncommon to see warnings like the following. These indicate that GitLab +would have added the user to a group, but the user could not be found in GitLab. +Usually this is not a cause for concern. + +If you think a particular user should already exist in GitLab, but you're seeing +this entry, it could be due to a mismatched DN stored in GitLab. See +[User DN and/or email have changed](#user-dn-orand-email-have-changed) to update the user's LDAP identity. + +```shell +User with DN `uid=john0,ou=people,dc=example,dc=com` should have access +to 'my_group' group but there is no user in GitLab with that +identity. Membership will be updated once the user signs in for +the first time. +``` + +Finally, the following entry says syncing has finished for this group: + +```shell +Finished syncing all providers for 'my_group' group +``` + +Once all the configured group links have been synchronized, GitLab will look +for any Administrators or External users to sync: + +```shell +Syncing admin users for 'ldapmain' provider +``` + +The output will look similar to what happens with a single group, and then +this line will indicate the sync is finished: + +```shell +Finished syncing admin users for 'ldapmain' provider +``` + +If [admin sync](index.md#administrator-sync-starter-only) is not configured, you'll see a message +stating as such: + +```shell +No `admin_group` configured for 'ldapmain' provider. Skipping +``` + +#### Sync one group **(STARTER ONLY)** + +[Syncing all groups](#sync-all-groups-starter-only) can produce a lot of noise in the output, which can be +distracting when you're only interested in troubleshooting the memberships of +a single GitLab group. In that case, here's how you can just sync this group +and see its debug output: + +```ruby +Rails.logger.level = Logger::DEBUG + +# Find the GitLab group. +# If the output is `nil`, the group could not be found. +# If a bunch of group attributes are in the output, your group was found successfully. +group = Group.find_by(name: 'my_gitlab_group') + +# Sync this group against LDAP +EE::Gitlab::Auth::Ldap::Sync::Group.execute_all_providers(group) +``` + +The output will be similar to +[that you'd get from syncing all groups](#example-console-output-after-a-group-sync-starter-only). + +#### Query a group in LDAP **(STARTER ONLY)** + +When you'd like to confirm that GitLab can read a LDAP group and see all its members, +you can run the following: + +```ruby +# Find the adapter and the group itself +adapter = Gitlab::Auth::Ldap::Adapter.new('ldapmain') # If `main` is the LDAP provider +ldap_group = EE::Gitlab::Auth::Ldap::Group.find_by_cn('group_cn_here', adapter) + +# Find the members of the LDAP group +ldap_group.member_dns +ldap_group.member_uids +``` + +### User DN or/and email have changed + +When an LDAP user is created in GitLab, their LDAP DN is stored for later reference. + +If GitLab cannot find a user by their DN, it will fall back +to finding the user by their email. If the lookup is successful, GitLab will +update the stored DN to the new value so both values will now match what's in +LDAP. + +If the email has changed and the DN has not, GitLab will find the user with +the DN and update its own record of the user's email to match the one in LDAP. + +However, if the primary email _and_ the DN change in LDAP, then GitLab will +have no way of identifying the correct LDAP record of the user and, as a +result, the user will be blocked. To rectify this, the user's existing +profile will have to be updated with at least one of the new values (primary +email or DN) so the LDAP record can be found. + +The following script will update the emails for all provided users so they +won't be blocked or unable to access their accounts. + +>**NOTE**: The following script will require that any new accounts with the new +email address are removed first. This is because emails have to be unique in GitLab. + +Go to the [rails console](#rails-console) and then run: + +```ruby +# Each entry will have to include the old username and the new email +emails = { + 'ORIGINAL_USERNAME' => 'NEW_EMAIL_ADDRESS', + ... +} + +emails.each do |username, email| + user = User.find_by_username(username) + user.email = email + user.skip_reconfirmation! + user.save! +end +``` + +You can then [run a UserSync](#sync-all-users-starter-only) **(STARTER ONLY)** to sync the latest DN +for each of these users. + +## Debugging Tools + +### LDAP check + +The [Rake task to check LDAP](../../raketasks/ldap.md#check) is a valuable tool +to help determine whether GitLab can successfully establish a connection to +LDAP and can get so far as to even read users. + +If a connection can't be established, it is likely either because of a problem +with your configuration or a firewall blocking the connection. + +- Ensure you don't have a firewall blocking the +connection, and that the LDAP server is accessible to the GitLab host. +- Look for an error message in the Rake check output, which may lead to your LDAP configuration to +confirm that the configuration values (specifically `host`, `port`, `bind_dn`, and +`password`) are correct. +- Look for [errors](#connection) in [the logs](#gitlab-logs) to further debug connection failures. + +If GitLab can successfully connect to LDAP but doesn't return any +users, [see what to do when no users are found](#no-users-are-found). + +### GitLab logs + +If a user account is blocked or unblocked due to the LDAP configuration, a +message will be [logged to `application.log`](../../logs.md#applicationlog). + +If there is an unexpected error during an LDAP lookup (configuration error, +timeout), the login is rejected and a message will be [logged to +`production.log`](../../logs.md#productionlog). + +### ldapsearch + +`ldapsearch` is a utility that will allow you to query your LDAP server. You can +use it to test your LDAP settings and ensure that the settings you're using +will get you the results you expect. + +When using `ldapsearch`, be sure to use the same settings you've already +specified in your `gitlab.rb` configuration so you can confirm what happens +when those exact settings are used. + +Running this command on the GitLab host will also help confirm that there's no +obstruction between the GitLab host and LDAP. + +For example, consider the following GitLab configuration: + +```shell +gitlab_rails['ldap_servers'] = YAML.load <<-'EOS' # remember to close this block with 'EOS' below + main: # 'main' is the GitLab 'provider ID' of this LDAP server + label: 'LDAP' + host: '127.0.0.1' + port: 389 + uid: 'uid' + encryption: 'plain' + bind_dn: 'cn=admin,dc=ldap-testing,dc=example,dc=com' + password: 'Password1' + active_directory: true + allow_username_or_email_login: false + block_auto_created_users: false + base: 'dc=ldap-testing,dc=example,dc=com' + user_filter: '' + attributes: + username: ['uid', 'userid', 'sAMAccountName'] + email: ['mail', 'email', 'userPrincipalName'] + name: 'cn' + first_name: 'givenName' + last_name: 'sn' + group_base: 'ou=groups,dc=ldap-testing,dc=example,dc=com' + admin_group: 'gitlab_admin' +EOS +``` + +You would run the following `ldapsearch` to find the `bind_dn` user: + +```shell +ldapsearch -D "cn=admin,dc=ldap-testing,dc=example,dc=com" \ + -w Password1 \ + -p 389 \ + -h 127.0.0.1 \ + -b "dc=ldap-testing,dc=example,dc=com" +``` + +Note that the `bind_dn`, `password`, `port`, `host`, and `base` are all +identical to what's configured in the `gitlab.rb`. + +Please see [the official +`ldapsearch` documentation](https://linux.die.net/man/1/ldapsearch) for more. + +### Using **AdFind** (Windows) + +You can use the [`AdFind`](https://social.technet.microsoft.com/wiki/contents/articles/7535.adfind-command-examples.aspx) utility (on Windows based systems) to test that your LDAP server is accessible and authentication is working correctly. This is a freeware utility built by [Joe Richards](http://www.joeware.net/freetools/tools/adfind/index.htm). + +**Return all objects** + +You can use the filter `objectclass=*` to return all directory objects. + +```shell +adfind -h ad.example.org:636 -ssl -u "CN=GitLabSRV,CN=Users,DC=GitLab,DC=org" -up Password1 -b "OU=GitLab INT,DC=GitLab,DC=org" -f (objectClass=*) +``` + +**Return single object using filter** + +You can also retrieve a single object by **specifying** the object name or full **DN**. In this example we specify the object name only `CN=Leroy Fox`. + +```shell +adfind -h ad.example.org:636 -ssl -u "CN=GitLabSRV,CN=Users,DC=GitLab,DC=org" -up Password1 -b "OU=GitLab INT,DC=GitLab,DC=org" -f (&(objectcategory=person)(CN=Leroy Fox))” +``` + +### Rails console + +CAUTION: **CAUTION:** +Please note that it is very easy to create, read, modify, and destroy data on the +rails console, so please be sure to run commands exactly as listed. + +The rails console is a valuable tool to help debug LDAP problems. It allows you to +directly interact with the application by running commands and seeing how GitLab +responds to them. + +Please refer to [this guide](../../troubleshooting/debug.md#starting-a-rails-console-session) +for instructions on how to use the rails console. + +#### Enable debug output + +This will provide debug output that will be useful to see +what GitLab is doing and with what. This value is not persisted, and will only +be enabled for this session in the rails console. + +To enable debug output in the rails console, [enter the rails +console](#rails-console) and run: + +```ruby +Rails.logger.level = Logger::DEBUG +``` diff --git a/doc/administration/auth/oidc.md b/doc/administration/auth/oidc.md index 7a808636e94..794733f860c 100644 --- a/doc/administration/auth/oidc.md +++ b/doc/administration/auth/oidc.md @@ -1,5 +1,8 @@ --- type: reference +stage: Manage +group: Access +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers --- # OpenID Connect OmniAuth provider @@ -96,8 +99,8 @@ The OpenID Connect will provide you with a client details and secret for you to - `mtls` - Mutual TLS or X.509 certificate validation - Any other value will POST the client ID and secret in the request body - If not specified, defaults to `basic`. - - `<uid_field>` (optional) is the field name from the `user_info` details that will be used as `uid` value. For example, `preferred_username`. - If this value is not provided or the field with the configured value is missing from the `user_info` details, the `uid` will use the `sub` field. + - `<uid_field>` (optional) is the field name from the `user_info.raw_attributes` details that will be used as `uid` value. For example, `preferred_username`. + If this value is not provided or the field with the configured value is missing from the `user_info.raw_attributes` details, the `uid` will use the `sub` field. - `send_scope_to_token_endpoint` is `true` by default. In other words, the `scope` parameter is normally included in requests to the token endpoint. However, if your OpenID Connect provider does not accept the `scope` parameter in such requests, set this to `false`. - `client_options` are the OpenID Connect client-specific options. Specifically: diff --git a/doc/administration/auth/okta.md b/doc/administration/auth/okta.md index 4fd37b51f24..f7ab60ab56b 100644 --- a/doc/administration/auth/okta.md +++ b/doc/administration/auth/okta.md @@ -1,5 +1,8 @@ --- type: reference +stage: Manage +group: Access +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers --- # Okta SSO provider @@ -17,7 +20,7 @@ The following documentation enables Okta as a SAML provider. 1. Now, very important, add a logo (you can choose it from <https://about.gitlab.com/press/>). You'll have to crop and resize it. -1. Next, you'll need the to fill in the SAML general config. Here's an example +1. Next, you'll need the to fill in the SAML general configuration. Here's an example image.  @@ -25,14 +28,14 @@ The following documentation enables Okta as a SAML provider. 1. The last part of the configuration is the feedback section where you can just say you're a customer and creating an app for internal use. 1. When you have your app you'll have a few tabs on the top of the app's - profile. Click on the SAML 2.0 config instructions button which should + profile. Click on the SAML 2.0 configuration instructions button which should look like the following:  1. On the screen that comes up take note of the **Identity Provider Single Sign-On URL** which you'll use for the - `idp_sso_target_url` on your GitLab config file. + `idp_sso_target_url` on your GitLab configuration file. 1. **Before you leave Okta make sure you add your user and groups if any.** diff --git a/doc/administration/auth/smartcard.md b/doc/administration/auth/smartcard.md index 7131fd7571f..9ad1e0641f6 100644 --- a/doc/administration/auth/smartcard.md +++ b/doc/administration/auth/smartcard.md @@ -8,7 +8,7 @@ GitLab supports authentication using smartcards. ## Existing password authentication -> [Introduced](https://gitlab.com/gitlab-org/gitlab/issues/33669) in GitLab 12.6. +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/33669) in GitLab 12.6. By default, existing users can continue to log in with a username and password when smartcard authentication is enabled. @@ -25,7 +25,7 @@ GitLab supports two authentication methods: ### Authentication against a local database with X.509 certificates -> [Introduced](https://gitlab.com/gitlab-org/gitlab/issues/726) in [GitLab Premium](https://about.gitlab.com/pricing/) 11.6 as an experimental feature. +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/726) in [GitLab Premium](https://about.gitlab.com/pricing/) 11.6 as an experimental feature. CAUTION: **Caution:** Smartcard authentication against local databases may change or be removed completely in future @@ -52,7 +52,7 @@ Certificate: ### Authentication against a local database with X.509 certificates and SAN extension -> [Introduced](https://gitlab.com/gitlab-org/gitlab/issues/8605) in [GitLab Premium](https://about.gitlab.com/pricing/) 12.3. +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/8605) in [GitLab Premium](https://about.gitlab.com/pricing/) 12.3. Smartcards with X.509 certificates using SAN extensions can be used to authenticate with GitLab. @@ -67,7 +67,7 @@ database with GitLab, in: - GitLab 12.4 and later, at least one of the `subjectAltName` (SAN) extensions need to define the user identity (`email`) within the GitLab instance (`URI`). `URI`: needs to match `Gitlab.config.host.gitlab`. -- From [GitLab 12.5](https://gitlab.com/gitlab-org/gitlab/issues/33907), +- From [GitLab 12.5](https://gitlab.com/gitlab-org/gitlab/-/issues/33907), if your certificate contains only **one** SAN email entry, you don't need to add or modify it to match the `email` with the `URI`. @@ -95,7 +95,7 @@ Certificate: ### Authentication against an LDAP server -> [Introduced](https://gitlab.com/gitlab-org/gitlab/issues/7693) in [GitLab Premium](https://about.gitlab.com/pricing/) 11.8 as an experimental feature. Smartcard authentication against an LDAP server may change or be removed completely in future releases. +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/7693) in [GitLab Premium](https://about.gitlab.com/pricing/) 11.8 as an experimental feature. Smartcard authentication against an LDAP server may change or be removed completely in future releases. GitLab implements a standard way of certificate matching following [RFC4523](https://tools.ietf.org/html/rfc4523). It uses the @@ -159,7 +159,7 @@ attribute. As a prerequisite, you must use an LDAP server that: ``` For example, the following is an example server context in an NGINX - configuration file (eg. in `/etc/nginx/sites-available/gitlab-ssl`): + configuration file (such as in `/etc/nginx/sites-available/gitlab-ssl`): ```plaintext server { diff --git a/doc/administration/compliance.md b/doc/administration/compliance.md index 8cd10f5ea4e..47e4b6c068c 100644 --- a/doc/administration/compliance.md +++ b/doc/administration/compliance.md @@ -12,8 +12,8 @@ GitLab’s [security features](../security/README.md) may also help you meet rel |**[Email all users of a project, group, or entire server](../user/admin_area/settings/terms.md)**<br>An admin can email groups of users based on project or group membership, or email everyone using the GitLab instance. This is great for scheduled maintenance or upgrades.|Starter+|| |**[Omnibus package supports log forwarding](https://docs.gitlab.com/omnibus/settings/logs.html#udp-log-forwarding)**<br>Forward your logs to a central system.|Starter+|| |**[Lock project membership to group](../user/group/index.md#member-lock-starter)**<br>Group owners can prevent new members from being added to projects within a group.|Starter+|✓| -|**[LDAP group sync](auth/ldap-ee.md#group-sync)**<br>GitLab Enterprise Edition gives admins the ability to automatically sync groups and manage SSH keys, permissions, and authentication, so you can focus on building your product, not configuring your tools.|Starter+|| -|**[LDAP group sync filters](auth/ldap-ee.md#group-sync)**<br>GitLab Enterprise Edition Premium gives more flexibility to synchronize with LDAP based on filters, meaning you can leverage LDAP attributes to map GitLab permissions.|Premium+|| +|**[LDAP group sync](auth/ldap/index.md#group-sync-starter-only)**<br>GitLab Enterprise Edition gives admins the ability to automatically sync groups and manage SSH keys, permissions, and authentication, so you can focus on building your product, not configuring your tools.|Starter+|| +|**[LDAP group sync filters](auth/ldap/index.md#group-sync-starter-only)**<br>GitLab Enterprise Edition Premium gives more flexibility to synchronize with LDAP based on filters, meaning you can leverage LDAP attributes to map GitLab permissions.|Premium+|| |**[Audit logs](audit_events.md)**<br>To maintain the integrity of your code, GitLab Enterprise Edition Premium gives admins the ability to view any modifications made within the GitLab server in an advanced audit log system, so you can control, analyze, and track every change.|Premium+|| |**[Auditor users](auditor_users.md)**<br>Auditor users are users who are given read-only access to all projects, groups, and other resources on the GitLab instance.|Premium+|| |**[Credentials inventory](../user/admin_area/credentials_inventory.md)**<br>With a credentials inventory, GitLab administrators can keep track of the credentials used by all of the users in their GitLab instance. |Ultimate|| diff --git a/doc/administration/environment_variables.md b/doc/administration/environment_variables.md index 37d7194af53..ecf9464eb91 100644 --- a/doc/administration/environment_variables.md +++ b/doc/administration/environment_variables.md @@ -1,3 +1,10 @@ +--- +stage: Verify +group: Continuous Integration +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +type: reference +--- + # Environment Variables GitLab exposes certain environment variables which can be used to override @@ -63,6 +70,7 @@ to the naming scheme `GITLAB_#{name in 1_settings.rb in upper case}`. To set environment variables, follow [these instructions](https://docs.gitlab.com/omnibus/settings/environment-variables.html). -It's possible to preconfigure the GitLab docker image by adding the environment +It's possible to preconfigure the GitLab Docker image by adding the environment variable `GITLAB_OMNIBUS_CONFIG` to the `docker run` command. -For more information see the ['preconfigure-docker-container' section in the Omnibus documentation](https://docs.gitlab.com/omnibus/docker/#preconfigure-docker-container). +For more information see the [Pre-configure Docker container](https://docs.gitlab.com/omnibus/docker/#pre-configure-docker-container) +section in the Omnibus documentation. diff --git a/doc/administration/external_database.md b/doc/administration/external_database.md index 47509828c20..7424b4e1e83 100644 --- a/doc/administration/external_database.md +++ b/doc/administration/external_database.md @@ -1,41 +1,7 @@ -# Configure GitLab using an external PostgreSQL service - -If you're hosting GitLab on a cloud provider, you can optionally use a -managed service for PostgreSQL. For example, AWS offers a managed Relational -Database Service (RDS) that runs PostgreSQL. - -Alternatively, you may opt to manage your own PostgreSQL instance or cluster -separate from the Omnibus GitLab package. - -If you use a cloud-managed service, or provide your own PostgreSQL instance: - -1. Set up PostgreSQL according to the - [database requirements document](../install/requirements.md#database). -1. Set up a `gitlab` username with a password of your choice. The `gitlab` user - needs privileges to create the `gitlabhq_production` database. -1. If you are using a cloud-managed service, you may need to grant additional - roles to your `gitlab` user: - - Amazon RDS requires the [`rds_superuser`](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Appendix.PostgreSQL.CommonDBATasks.html#Appendix.PostgreSQL.CommonDBATasks.Roles) role. - - Azure Database for PostgreSQL requires the [`azure_pg_admin`](https://docs.microsoft.com/en-us/azure/postgresql/howto-create-users#how-to-create-additional-admin-users-in-azure-database-for-postgresql) role. +--- +redirect_to: 'postgresql/external.md' +--- -1. Configure the GitLab application servers with the appropriate connection details - for your external PostgreSQL service in your `/etc/gitlab/gitlab.rb` file: - - ```ruby - # Disable the bundled Omnibus provided PostgreSQL - postgresql['enable'] = false - - # PostgreSQL connection details - gitlab_rails['db_adapter'] = 'postgresql' - gitlab_rails['db_encoding'] = 'unicode' - gitlab_rails['db_host'] = '10.1.0.5' # IP/hostname of database server - gitlab_rails['db_password'] = 'DB password' - ``` - - For more information on GitLab HA setups, refer to [configuring GitLab for HA](high_availability/gitlab.md). - -1. Reconfigure for the changes to take effect: +# Configure GitLab using an external PostgreSQL service - ```shell - sudo gitlab-ctl reconfigure - ``` +This content has been moved to a [new location](postgresql/external.md). diff --git a/doc/administration/external_pipeline_validation.md b/doc/administration/external_pipeline_validation.md index 489d1924803..cb1d35f24d5 100644 --- a/doc/administration/external_pipeline_validation.md +++ b/doc/administration/external_pipeline_validation.md @@ -1,3 +1,10 @@ +--- +stage: Verify +group: Continuous Integration +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +type: reference, howto +--- + # External Pipeline Validation You can use an external service for validating a pipeline before it's created. diff --git a/doc/administration/feature_flags.md b/doc/administration/feature_flags.md index 59cd5497032..9e87eed4508 100644 --- a/doc/administration/feature_flags.md +++ b/doc/administration/feature_flags.md @@ -1,4 +1,7 @@ --- +stage: Release +group: Progressive Delivery +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers type: reference description: "GitLab administrator: enable and disable GitLab features deployed behind feature flags" --- @@ -29,7 +32,7 @@ them. It can be done by GitLab administrators with access to GitLab Rails console. If you used a certain feature and identified a bug, a misbehavior, or an -error, it's very important that you [**provide feedback**](https://gitlab.com/gitlab-org/gitlab/issues/new?issue[title]=Docs%20-%20feature%20flag%20feedback%3A%20Feature%20Name&issue[description]=Describe%20the%20problem%20you%27ve%20encountered.%0A%0A%3C!--%20Don%27t%20edit%20below%20this%20line%20--%3E%0A%0A%2Flabel%20~%22docs%5C-comments%22%20) to GitLab as soon +error, it's very important that you [**provide feedback**](https://gitlab.com/gitlab-org/gitlab/-/issues/new?issue[title]=Docs%20-%20feature%20flag%20feedback%3A%20Feature%20Name&issue[description]=Describe%20the%20problem%20you%27ve%20encountered.%0A%0A%3C!--%20Don%27t%20edit%20below%20this%20line%20--%3E%0A%0A%2Flabel%20~%22docs%5C-comments%22%20) to GitLab as soon as possible so we can improve or fix it while behind a flag. When you upgrade GitLab to an earlier version, the feature flag status may change. @@ -94,6 +97,18 @@ Example, to disable Evidence Collection: Feature.disable(:release_evidence_collection) ``` +Some feature flags can be enabled or disabled on a per project basis: + +```ruby +Feature.enable(:<feature flag>, Project.find(<project id>)) +``` + +For example, to enable the [`:release_evidence_collection`](../ci/junit_test_reports.md#enabling-the-feature) feature flag for project `1234`: + +```ruby +Feature.enable(:release_evidence_collection, Project.find(1234)) +``` + When the feature is ready, GitLab will remove the feature flag, the option for enabling and disabling it will no longer exist, and the feature will become available in all instances. diff --git a/doc/administration/geo/disaster_recovery/background_verification.md b/doc/administration/geo/disaster_recovery/background_verification.md index 5b24c222f06..88218d24b2f 100644 --- a/doc/administration/geo/disaster_recovery/background_verification.md +++ b/doc/administration/geo/disaster_recovery/background_verification.md @@ -1,3 +1,10 @@ +--- +stage: Enablement +group: Geo +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +type: howto +--- + # Automatic background verification **(PREMIUM ONLY)** NOTE: **Note:** diff --git a/doc/administration/geo/disaster_recovery/bring_primary_back.md b/doc/administration/geo/disaster_recovery/bring_primary_back.md index 43089237a75..b19e55595e7 100644 --- a/doc/administration/geo/disaster_recovery/bring_primary_back.md +++ b/doc/administration/geo/disaster_recovery/bring_primary_back.md @@ -1,3 +1,10 @@ +--- +stage: Enablement +group: Geo +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +type: howto +--- + # Bring a demoted primary node back online **(PREMIUM ONLY)** After a failover, it is possible to fail back to the demoted **primary** node to diff --git a/doc/administration/geo/disaster_recovery/index.md b/doc/administration/geo/disaster_recovery/index.md index 9f7afad353b..f690ec63cf9 100644 --- a/doc/administration/geo/disaster_recovery/index.md +++ b/doc/administration/geo/disaster_recovery/index.md @@ -1,3 +1,10 @@ +--- +stage: Enablement +group: Geo +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +type: howto +--- + # Disaster Recovery (Geo) **(PREMIUM ONLY)** Geo replicates your database, your Git repositories, and few other assets. @@ -51,7 +58,7 @@ must disable the **primary** node. NOTE: **Note:** (**CentOS only**) In CentOS 6 or older, there is no easy way to prevent GitLab from being - started if the machine reboots isn't available (see [Omnibus GitLab issue #3058](https://gitlab.com/gitlab-org/omnibus-gitlab/issues/3058)). + started if the machine reboots isn't available (see [Omnibus GitLab issue #3058](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/3058)). It may be safest to uninstall the GitLab package completely: ```shell @@ -60,7 +67,7 @@ must disable the **primary** node. NOTE: **Note:** (**Ubuntu 14.04 LTS**) If you are using an older version of Ubuntu - or any other distro based on the Upstart init system, you can prevent GitLab + or any other distribution based on the Upstart init system, you can prevent GitLab from starting if the machine reboots by doing the following: ```shell @@ -121,14 +128,20 @@ Note the following when promoting a secondary: gitlab-ctl promote-to-primary-node ``` + If you have already run the [preflight checks](planned_failover.md#preflight-checks), you can skip them with: + + ```shell + gitlab-ctl promote-to-primary-node --skip-preflight-check + ``` + 1. Verify you can connect to the newly promoted **primary** node using the URL used previously for the **secondary** node. 1. If successful, the **secondary** node has now been promoted to the **primary** node. -#### Promoting a **secondary** node with HA +#### Promoting a **secondary** node with multiple servers The `gitlab-ctl promote-to-primary-node` command cannot be used yet in -conjunction with High Availability or with multiple machines, as it can only +conjunction with multiple servers, as it can only perform changes on a **secondary** with only a single machine. Instead, you must do this manually. @@ -178,6 +191,27 @@ required: set the database to read-write: - Amazon RDS - [Promoting a Read Replica to Be a Standalone DB Instance](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_ReadRepl.html#USER_ReadRepl.Promote) - Azure Database for PostgreSQL - [Stop replication](https://docs.microsoft.com/en-us/azure/postgresql/howto-read-replicas-portal#stop-replication) + - Other external PostgreSQL databases - save the script below in you secondary node, for example + `/tmp/geo_promote.sh`, and modify the connection parameters to match your + environment. Then, execute it to promote the replica: + + ```shell + #!/bin/bash + + PG_SUPERUSER=postgres + + # The path to your pg_ctl binary. You may need to adjust this path to match + # your PostgreSQL installation + PG_CTL_BINARY=/usr/lib/postgresql/10/bin/pg_ctl + + # The path to your PostgreSQL data directory. You may need to adjust this + # path to match your PostgreSQL installation. You can also run + # `SHOW data_directory;` from PostgreSQL to find your data directory + PG_DATA_DIRECTORY=/etc/postgresql/10/main + + # Promote the PostgreSQL database and allow read/write operations + sudo -u $PG_SUPERUSER $PG_CTL_BINARY -D $PG_DATA_DIRECTORY promote + ``` 1. Edit `/etc/gitlab/gitlab.rb` on every node in the **secondary** site to reflect its new status as **primary** by removing any lines that enabled the diff --git a/doc/administration/geo/disaster_recovery/planned_failover.md b/doc/administration/geo/disaster_recovery/planned_failover.md index 00ca1e4b8b0..0ce1325a537 100644 --- a/doc/administration/geo/disaster_recovery/planned_failover.md +++ b/doc/administration/geo/disaster_recovery/planned_failover.md @@ -1,3 +1,10 @@ +--- +stage: Enablement +group: Geo +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +type: howto +--- + # Disaster recovery for planned failover **(PREMIUM ONLY)** The primary use-case of Disaster Recovery is to ensure business continuity in @@ -126,7 +133,7 @@ will take to finish syncing. An example message would be: ## Prevent updates to the **primary** node -Until a [read-only mode](https://gitlab.com/gitlab-org/gitlab-foss/issues/19739) is implemented, updates must be prevented +Until a [read-only mode](https://gitlab.com/gitlab-org/gitlab/-/issues/14609) is implemented, updates must be prevented from happening manually. Note that your **secondary** node still needs read-only access to the **primary** node during the maintenance window. diff --git a/doc/administration/geo/replication/configuration.md b/doc/administration/geo/replication/configuration.md index 86a8e5b28d1..3d08ed81700 100644 --- a/doc/administration/geo/replication/configuration.md +++ b/doc/administration/geo/replication/configuration.md @@ -1,3 +1,10 @@ +--- +stage: Enablement +group: Geo +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +type: howto +--- + # Geo configuration **(PREMIUM ONLY)** ## Configuring a new **secondary** node @@ -25,7 +32,7 @@ Any change that requires access to the **Admin Area** needs to be done in the GitLab stores a number of secret values in the `/etc/gitlab/gitlab-secrets.json` file which *must* be the same on all nodes. Until there is -a means of automatically replicating these between nodes (see [issue #3789](https://gitlab.com/gitlab-org/gitlab/issues/3789)), +a means of automatically replicating these between nodes (see [issue #3789](https://gitlab.com/gitlab-org/gitlab/-/issues/3789)), they must be manually replicated to the **secondary** node. 1. SSH into the **primary** node, and execute the command below: diff --git a/doc/administration/geo/replication/database.md b/doc/administration/geo/replication/database.md index 62bd0e6ac19..3f2d46ba457 100644 --- a/doc/administration/geo/replication/database.md +++ b/doc/administration/geo/replication/database.md @@ -1,3 +1,10 @@ +--- +stage: Enablement +group: Geo +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +type: howto +--- + # Geo database replication **(PREMIUM ONLY)** NOTE: **Note:** @@ -23,10 +30,6 @@ The GitLab **primary** node where the write operations happen will connect to the **primary** database server, and **secondary** nodes will connect to their own database servers (which are also read-only). -NOTE: **Note:** -In database documentation, you may see "**primary**" being referenced as "master" -and "**secondary**" as either "slave" or "standby" server (read-only). - We recommend using [PostgreSQL replication slots](https://medium.com/@tk512/replication-slots-in-postgresql-b4b03d277c75) to ensure that the **primary** node retains all the data necessary for the **secondary** nodes to recover. See below for more details. @@ -43,7 +46,7 @@ The following guide assumes that: CAUTION: **Warning:** Geo works with streaming replication. Logical replication is not supported at this time. -There is an [issue where support is being discussed](https://gitlab.com/gitlab-org/gitlab/issues/7420). +There is an [issue where support is being discussed](https://gitlab.com/gitlab-org/gitlab/-/issues/7420). ### Step 1. Configure the **primary** server @@ -317,7 +320,7 @@ There is an [issue where support is being discussed](https://gitlab.com/gitlab-o to the private key, which is **only** present on the **primary** node. 1. Test that the `gitlab-psql` user can connect to the **primary** node's database - (the default Omnibus database name is gitlabhq_production): + (the default Omnibus database name is `gitlabhq_production`): ```shell sudo \ @@ -461,7 +464,7 @@ The replication process is now complete. PostgreSQL connections. We recommend using PgBouncer if you use GitLab in a high-availability configuration with a cluster of nodes supporting a Geo **primary** node and another cluster of nodes supporting a Geo **secondary** node. For more -information, see [High Availability with Omnibus GitLab](../../high_availability/database.md#high-availability-with-omnibus-gitlab-premium-only). +information, see [High Availability with Omnibus GitLab](../../postgresql/replication_and_failover.md). For a Geo **secondary** node to work properly with PgBouncer in front of the database, it will need a separate read-only user to make [PostgreSQL FDW queries](https://www.postgresql.org/docs/11/postgres-fdw.html) diff --git a/doc/administration/geo/replication/datatypes.md b/doc/administration/geo/replication/datatypes.md index 17031b11f51..f50da27d82f 100644 --- a/doc/administration/geo/replication/datatypes.md +++ b/doc/administration/geo/replication/datatypes.md @@ -1,3 +1,10 @@ +--- +stage: Enablement +group: Geo +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +type: howto +--- + # Geo data types support A Geo data type is a specific class of data that is required by one or more GitLab features to @@ -129,11 +136,11 @@ successfully, you must replicate their data using some other means. | Application data in PostgreSQL | **Yes** | **Yes** | | | Project repository | **Yes** | **Yes** | | | Project wiki repository | **Yes** | **Yes** | | -| Project designs repository | **Yes** | [No](https://gitlab.com/gitlab-org/gitlab/issues/32467) | | +| Project designs repository | **Yes** | [No](https://gitlab.com/gitlab-org/gitlab/-/issues/32467) | | | Uploads | **Yes** | [No](https://gitlab.com/groups/gitlab-org/-/epics/1817) | Verified only on transfer, or manually (*1*) | -| LFS objects | **Yes** | [No](https://gitlab.com/gitlab-org/gitlab/issues/8922) | Verified only on transfer, or manually (*1*). Unavailable for new LFS objects in 11.11.x and 12.0.x (*2*). | -| CI job artifacts (other than traces) | **Yes** | [No](https://gitlab.com/gitlab-org/gitlab/issues/8923) | Verified only manually (*1*) | -| Archived traces | **Yes** | [No](https://gitlab.com/gitlab-org/gitlab/issues/8923) | Verified only on transfer, or manually (*1*) | +| LFS objects | **Yes** | [No](https://gitlab.com/gitlab-org/gitlab/-/issues/8922) | Verified only on transfer, or manually (*1*). Unavailable for new LFS objects in 11.11.x and 12.0.x (*2*). | +| CI job artifacts (other than traces) | **Yes** | [No](https://gitlab.com/gitlab-org/gitlab/-/issues/8923) | Verified only manually (*1*) | +| Archived traces | **Yes** | [No](https://gitlab.com/gitlab-org/gitlab/-/issues/8923) | Verified only on transfer, or manually (*1*) | | Personal snippets | **Yes** | **Yes** | | | [Versioned snippets](../../../user/snippets.md#versioned-snippets) | [No](https://gitlab.com/groups/gitlab-org/-/epics/2809) | [No](https://gitlab.com/groups/gitlab-org/-/epics/2810) | | | Project snippets | **Yes** | **Yes** | | @@ -147,11 +154,16 @@ successfully, you must replicate their data using some other means. | [Conan Repository](../../../user/packages/conan_repository/index.md) | [No](https://gitlab.com/groups/gitlab-org/-/epics/2346) | No | | | [NuGet Repository](../../../user/packages/nuget_repository/index.md) | [No](https://gitlab.com/groups/gitlab-org/-/epics/2346) | No | | | [PyPi Repository](../../../user/packages/pypi_repository/index.md) | [No](https://gitlab.com/groups/gitlab-org/-/epics/2554) | No | | -| [External merge request diffs](../../merge_request_diffs.md) | [No](https://gitlab.com/gitlab-org/gitlab/issues/33817) | No | | +| [Composer Repository](../../../user/packages/composer_repository/index.md) | [No](https://gitlab.com/groups/gitlab-org/-/epics/3096) | No | | +| [External merge request diffs](../../merge_request_diffs.md) | [No](https://gitlab.com/gitlab-org/gitlab/-/issues/33817) | No | | +| [Terraform State](../../terraform_state.md) | [No](https://gitlab.com/groups/gitlab-org/-/epics/3112)(*3*) | No | | +| [Vulnerability Export](../../../user/application_security/security_dashboard/#export-vulnerabilities-1) | [No](https://gitlab.com/groups/gitlab-org/-/epics/3111)(*3*) | No | | | | Content in object storage | **Yes** | No | | - (*1*): The integrity can be verified manually using [Integrity Check Rake Task](../../raketasks/check.md) on both nodes and comparing the output between them. - (*2*): GitLab versions 11.11.x and 12.0.x are affected by [a bug that prevents any new - LFS objects from replicating](https://gitlab.com/gitlab-org/gitlab/issues/32696). + LFS objects from replicating](https://gitlab.com/gitlab-org/gitlab/-/issues/32696). +- (*3*): If you are using Object Storage, the replication can be performed by the + Object Storage provider if supported. Please see [Geo with Object Storage](object_storage.md) diff --git a/doc/administration/geo/replication/docker_registry.md b/doc/administration/geo/replication/docker_registry.md index e9c69dd32e4..bea6528dc9b 100644 --- a/doc/administration/geo/replication/docker_registry.md +++ b/doc/administration/geo/replication/docker_registry.md @@ -1,3 +1,10 @@ +--- +stage: Enablement +group: Geo +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +type: howto +--- + # Docker Registry for a secondary node **(PREMIUM ONLY)** You can set up a [Docker Registry](https://docs.docker.com/registry/) on your @@ -5,7 +12,7 @@ You can set up a [Docker Registry](https://docs.docker.com/registry/) on your ## Storage support -Docker Registry currently supports a few types of storages. If you choose a +Docker Registry currently supports a few types of storage. If you choose a distributed storage (`azure`, `gcs`, `s3`, `swift`, or `oss`) for your Docker Registry on the **primary** node, you can use the same storage for a **secondary** Docker Registry as well. For more information, read the @@ -16,7 +23,7 @@ integrated [Container Registry](../../packages/container_registry.md#container-r ## Replicating Docker Registry You can enable a storage-agnostic replication so it -can be used for cloud or local storages. Whenever a new image is pushed to the +can be used for cloud or local storage. Whenever a new image is pushed to the **primary** node, each **secondary** node will pull it to its own container repository. @@ -104,7 +111,7 @@ generate a short-lived JWT that is pull-only-capable to access the ```ruby gitlab_rails['geo_registry_replication_enabled'] = true - gitlab_rails['geo_registry_replication_primary_api_url'] = 'http://primary.example.com:4567/' # Primary registry address, it will be used by the secondary node to directly communicate to primary registry + gitlab_rails['geo_registry_replication_primary_api_url'] = 'https://primary.example.com:5050/' # Primary registry address, it will be used by the secondary node to directly communicate to primary registry ``` 1. Reconfigure the **secondary** node for the change to take effect: diff --git a/doc/administration/geo/replication/external_database.md b/doc/administration/geo/replication/external_database.md index ae3069a0e40..491b3278ead 100644 --- a/doc/administration/geo/replication/external_database.md +++ b/doc/administration/geo/replication/external_database.md @@ -1,3 +1,10 @@ +--- +stage: Enablement +group: Geo +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +type: howto +--- + # Geo with external PostgreSQL instances **(PREMIUM ONLY)** This document is relevant if you are using a PostgreSQL instance that is *not @@ -62,7 +69,7 @@ Once your read-only replica is set up, you can skip to [configure you secondary #### Manually configure the primary database for replication -The [geo_primary_role](https://docs.gitlab.com/omnibus/roles/#gitlab-geo-roles) +The [`geo_primary_role`](https://docs.gitlab.com/omnibus/roles/#gitlab-geo-roles) configures the **primary** node's database to be replicated by making changes to `pg_hba.conf` and `postgresql.conf`. Make the following configuration changes manually to your external database configuration and ensure that you restart PostgreSQL @@ -123,7 +130,7 @@ hot_standby = on ### Configure **secondary** application nodes to use the external read-replica With Omnibus, the -[geo_secondary_role](https://docs.gitlab.com/omnibus/roles/#gitlab-geo-roles) +[`geo_secondary_role`](https://docs.gitlab.com/omnibus/roles/#gitlab-geo-roles) has three main functions: 1. Configure the replica database. diff --git a/doc/administration/geo/replication/faq.md b/doc/administration/geo/replication/faq.md index 2405e2cbfd2..522ad32c352 100644 --- a/doc/administration/geo/replication/faq.md +++ b/doc/administration/geo/replication/faq.md @@ -1,3 +1,10 @@ +--- +stage: Enablement +group: Geo +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +type: howto +--- + # Geo Frequently Asked Questions **(PREMIUM ONLY)** ## What are the minimum requirements to run Geo? diff --git a/doc/administration/geo/replication/geo_validation_tests.md b/doc/administration/geo/replication/geo_validation_tests.md index a8b0bdeb7da..7b186d15fae 100644 --- a/doc/administration/geo/replication/geo_validation_tests.md +++ b/doc/administration/geo/replication/geo_validation_tests.md @@ -1,3 +1,10 @@ +--- +stage: Enablement +group: Geo +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +type: howto +--- + # Geo validation tests The Geo team performs manual testing and validation on common deployment configurations to ensure @@ -26,8 +33,8 @@ The following are GitLab upgrade validation tests we performed. configuration. - Outcome: Upgrade test was successful. - Follow up issues: - - [Investigate Geo end-to-end test failures](https://gitlab.com/gitlab-org/gitlab/issues/201823). - - [Add more logging to Geo end-to-end tests](https://gitlab.com/gitlab-org/gitlab/issues/201830). + - [Investigate Geo end-to-end test failures](https://gitlab.com/gitlab-org/gitlab/-/issues/201823). + - [Add more logging to Geo end-to-end tests](https://gitlab.com/gitlab-org/gitlab/-/issues/201830). - [Excess service restarts during zero-downtime upgrade](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/5047). [Upgrade Geo multi-server installation](https://gitlab.com/gitlab-org/gitlab/-/issues/199836): @@ -43,7 +50,7 @@ The following are GitLab upgrade validation tests we performed. configuration. - Outcome: Upgrade test was successful. - Follow up issues: - - [Investigate why HTTP push spec failed on primary node](https://gitlab.com/gitlab-org/gitlab/issues/199825). + - [Investigate why HTTP push spec failed on primary node](https://gitlab.com/gitlab-org/gitlab/-/issues/199825). - [Investigate if documentation should be modified to include refresh foreign tables task](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/5041). ### October 2019 @@ -89,7 +96,7 @@ The following are PostgreSQL upgrade validation tests we performed. ### September 2019 -[Test and validate PostgreSQL 10.0 upgrade for Geo](https://gitlab.com/gitlab-org/gitlab/issues/12092): +[Test and validate PostgreSQL 10.0 upgrade for Geo](https://gitlab.com/gitlab-org/gitlab/-/issues/12092): - Description: With the 12.0 release, GitLab required an upgrade to PostgreSQL 10.0. We tested various upgrade scenarios from GitLab 11.11.5 through to GitLab 12.1.8. diff --git a/doc/administration/geo/replication/index.md b/doc/administration/geo/replication/index.md index 87bd7b69515..5b4b476bfa8 100644 --- a/doc/administration/geo/replication/index.md +++ b/doc/administration/geo/replication/index.md @@ -1,3 +1,10 @@ +--- +stage: Enablement +group: Geo +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +type: howto +--- + # Replication (Geo) **(PREMIUM ONLY)** > - Introduced in GitLab Enterprise Edition 8.9. @@ -191,7 +198,7 @@ If you installed GitLab using the Omnibus packages (highly recommended): 1. [Set up the database replication](database.md) (`primary (read-write) <-> secondary (read-only)` topology). 1. [Configure fast lookup of authorized SSH keys in the database](../../operations/fast_ssh_key_lookup.md). This step is required and needs to be done on **both** the **primary** and **secondary** nodes. 1. [Configure GitLab](configuration.md) to set the **primary** and **secondary** nodes. -1. Optional: [Configure a secondary LDAP server](../../auth/ldap.md) for the **secondary** node. See [notes on LDAP](#ldap). +1. Optional: [Configure a secondary LDAP server](../../auth/ldap/index.md) for the **secondary** node. See [notes on LDAP](#ldap). 1. [Follow the "Using a Geo Server" guide](using_a_geo_server.md). ## Post-installation documentation @@ -243,15 +250,15 @@ For more information on removing a Geo node, see [Removing **secondary** Geo nod CAUTION: **Caution:** This list of limitations only reflects the latest version of GitLab. If you are using an older version, extra limitations may be in place. -- Pushing directly to a **secondary** node redirects (for HTTP) or proxies (for SSH) the request to the **primary** node instead of [handling it directly](https://gitlab.com/gitlab-org/gitlab/issues/1381), except when using Git over HTTP with credentials embedded within the URI. For example, `https://user:password@secondary.tld`. +- Pushing directly to a **secondary** node redirects (for HTTP) or proxies (for SSH) the request to the **primary** node instead of [handling it directly](https://gitlab.com/gitlab-org/gitlab/-/issues/1381), except when using Git over HTTP with credentials embedded within the URI. For example, `https://user:password@secondary.tld`. - Cloning, pulling, or pushing repositories that exist on the **primary** node but not on the **secondary** nodes where [selective synchronization](configuration.md#selective-synchronization) does not include the project is not supported over SSH [but support is planned](https://gitlab.com/groups/gitlab-org/-/epics/2562). HTTP(S) is supported. -- The **primary** node has to be online for OAuth login to happen. Existing sessions and Git are not affected. Support for the **secondary** node to use an OAuth provider independent from the primary is [being planned](https://gitlab.com/gitlab-org/gitlab/issues/208465). -- The installation takes multiple manual steps that together can take about an hour depending on circumstances. We are working on improving this experience. See [Omnibus GitLab issue #2978](https://gitlab.com/gitlab-org/omnibus-gitlab/issues/2978) for details. +- The **primary** node has to be online for OAuth login to happen. Existing sessions and Git are not affected. Support for the **secondary** node to use an OAuth provider independent from the primary is [being planned](https://gitlab.com/gitlab-org/gitlab/-/issues/208465). +- The installation takes multiple manual steps that together can take about an hour depending on circumstances. We are working on improving this experience. See [Omnibus GitLab issue #2978](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/2978) for details. - Real-time updates of issues/merge requests (for example, via long polling) doesn't work on the **secondary** node. - [Selective synchronization](configuration.md#selective-synchronization) applies only to files and repositories. Other datasets are replicated to the **secondary** node in full, making it inappropriate for use as an access control mechanism. - Object pools for forked project deduplication work only on the **primary** node, and are duplicated on the **secondary** node. - [External merge request diffs](../../merge_request_diffs.md) will not be replicated if they are on-disk, and viewing merge requests will fail. However, external MR diffs in object storage **are** supported. The default configuration (in-database) does work. -- GitLab Runners cannot register with a **secondary** node. Support for this is [planned for the future](https://gitlab.com/gitlab-org/gitlab/issues/3294). +- GitLab Runners cannot register with a **secondary** node. Support for this is [planned for the future](https://gitlab.com/gitlab-org/gitlab/-/issues/3294). ### Limitations on replication/verification diff --git a/doc/administration/geo/replication/location_aware_git_url.md b/doc/administration/geo/replication/location_aware_git_url.md index f1f1edd4a9b..49c83ee1718 100644 --- a/doc/administration/geo/replication/location_aware_git_url.md +++ b/doc/administration/geo/replication/location_aware_git_url.md @@ -1,3 +1,10 @@ +--- +stage: Enablement +group: Geo +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +type: howto +--- + # Location-aware Git remote URL with AWS Route53 **(PREMIUM ONLY)** You can provide GitLab users with a single remote URL that automatically uses @@ -15,7 +22,7 @@ NOTE: **Note** You can also use a load balancer to distribute web UI or API traffic to [multiple Geo **secondary** nodes](../../../user/admin_area/geo_nodes.md#multiple-secondary-nodes-behind-a-load-balancer). Importantly, the **primary** node cannot yet be included. See the feature request -[Support putting the **primary** behind a Geo node load balancer](https://gitlab.com/gitlab-org/gitlab/issues/10888). +[Support putting the **primary** behind a Geo node load balancer](https://gitlab.com/gitlab-org/gitlab/-/issues/10888). ## Prerequisites diff --git a/doc/administration/geo/replication/multiple_servers.md b/doc/administration/geo/replication/multiple_servers.md index 9322c4cc417..2747aaeb105 100644 --- a/doc/administration/geo/replication/multiple_servers.md +++ b/doc/administration/geo/replication/multiple_servers.md @@ -1,3 +1,10 @@ +--- +stage: Enablement +group: Geo +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +type: howto +--- + # Geo for multiple servers **(PREMIUM ONLY)** This document describes a minimal reference architecture for running Geo @@ -38,7 +45,7 @@ Because of the additional complexity involved in setting up this configuration for PostgreSQL and Redis, it is not covered by this Geo multi-server documentation. For more information about setting up a multi-server PostgreSQL cluster and Redis cluster using the omnibus package see the multi-server documentation for -[PostgreSQL](../../high_availability/database.md) and +[PostgreSQL](../../postgresql/replication_and_failover.md) and [Redis](../../high_availability/redis.md), respectively. NOTE: **Note:** @@ -87,7 +94,7 @@ NOTE: **Note:** PostgreSQL and Redis should have already been disabled on the application servers, and connections from the application servers to those services on the backend servers configured, during normal GitLab multi-server set up. See multi-server configuration documentation for -[PostgreSQL](../../high_availability/database.md#configuring-the-application-nodes) +[PostgreSQL](../../postgresql/replication_and_failover.md#configuring-the-application-nodes) and [Redis](../../high_availability/redis.md#example-configuration-for-the-gitlab-application). ### Step 2: Configure the **primary** database diff --git a/doc/administration/geo/replication/object_storage.md b/doc/administration/geo/replication/object_storage.md index ffd44282b23..3cc0ade414e 100644 --- a/doc/administration/geo/replication/object_storage.md +++ b/doc/administration/geo/replication/object_storage.md @@ -1,3 +1,10 @@ +--- +stage: Enablement +group: Geo +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +type: howto +--- + # Geo with Object storage **(PREMIUM ONLY)** Geo can be used in combination with Object Storage (AWS S3, or other compatible object storage). @@ -16,7 +23,7 @@ To have: ## Enabling GitLab managed object storage replication -> [Introduced](https://gitlab.com/gitlab-org/gitlab/issues/10586) in GitLab 12.4. +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/10586) in GitLab 12.4. CAUTION: **Caution:** This is a [**beta** feature](https://about.gitlab.com/handbook/product/#beta) and is not ready yet for production use at any scale. diff --git a/doc/administration/geo/replication/remove_geo_node.md b/doc/administration/geo/replication/remove_geo_node.md index c04c7aec858..539132776b3 100644 --- a/doc/administration/geo/replication/remove_geo_node.md +++ b/doc/administration/geo/replication/remove_geo_node.md @@ -1,3 +1,10 @@ +--- +stage: Enablement +group: Geo +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +type: howto +--- + # Removing secondary Geo nodes **(PREMIUM ONLY)** **Secondary** nodes can be removed from the Geo cluster using the Geo admin page of the **primary** node. To remove a **secondary** node: diff --git a/doc/administration/geo/replication/security_review.md b/doc/administration/geo/replication/security_review.md index 0ac8157220a..f5edf79c6e4 100644 --- a/doc/administration/geo/replication/security_review.md +++ b/doc/administration/geo/replication/security_review.md @@ -1,3 +1,10 @@ +--- +stage: Enablement +group: Geo +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +type: howto +--- + # Geo security review (Q&A) **(PREMIUM ONLY)** The following security review of the Geo feature set focuses on security aspects of @@ -22,7 +29,7 @@ from [owasp.org](https://owasp.org/). etc) and repository + wiki data. In a typical configuration, this will happen across the public Internet, and be TLS-encrypted. - PostgreSQL replication is TLS-encrypted. -- See also: [only TLSv1.2 should be supported](https://gitlab.com/gitlab-org/omnibus-gitlab/issues/2948) +- See also: [only TLSv1.2 should be supported](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/2948) ### How can the data be classified into categories according to its sensitivity? @@ -72,7 +79,7 @@ from [owasp.org](https://owasp.org/). - Nothing Geo-specific. Any user where `admin: true` is set in the database is considered an admin with super-user privileges. -- See also: [more granular access control](https://gitlab.com/gitlab-org/gitlab-foss/issues/32730) +- See also: [more granular access control](https://gitlab.com/gitlab-org/gitlab/-/issues/18242) (not Geo-specific). - Much of Geo’s integration (database replication, for instance) must be configured with the application, typically by system administrators. diff --git a/doc/administration/geo/replication/troubleshooting.md b/doc/administration/geo/replication/troubleshooting.md index 293414a6e5e..b03a2dae971 100644 --- a/doc/administration/geo/replication/troubleshooting.md +++ b/doc/administration/geo/replication/troubleshooting.md @@ -1,3 +1,10 @@ +--- +stage: Enablement +group: Geo +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +type: howto +--- + # Geo Troubleshooting **(PREMIUM ONLY)** Setting up Geo requires careful attention to details and sometimes it's easy to @@ -332,7 +339,7 @@ some of these queries will never be able to complete due to being canceled every time. These long-running queries are -[planned to be removed in the future](https://gitlab.com/gitlab-org/gitlab/issues/34269), +[planned to be removed in the future](https://gitlab.com/gitlab-org/gitlab/-/issues/34269), but as a workaround, we recommend enabling [hot_standby_feedback](https://www.postgresql.org/docs/10/hot-standby.html#HOT-STANDBY-CONFLICT). This increases the likelihood of bloat on the **primary** node as it prevents @@ -353,7 +360,7 @@ sudo gitlab-ctl reconfigure ``` To help us resolve this problem, consider commenting on -[the issue](https://gitlab.com/gitlab-org/gitlab/issues/4489). +[the issue](https://gitlab.com/gitlab-org/gitlab/-/issues/4489). ### Message: `LOG: invalid CIDR mask in address` @@ -405,7 +412,7 @@ long enough to accommodate a full clone of your largest repositories. If new LFS objects are never replicated to secondary Geo nodes, check the version of GitLab you are running. GitLab versions 11.11.x or 12.0.x are affected by -[a bug that results in new LFS objects not being replicated to Geo secondary nodes](https://gitlab.com/gitlab-org/gitlab/issues/32696). +[a bug that results in new LFS objects not being replicated to Geo secondary nodes](https://gitlab.com/gitlab-org/gitlab/-/issues/32696). To resolve the issue, upgrade to GitLab 12.1 or newer. @@ -535,7 +542,7 @@ or `gitlab-ctl promote-to-primary-node`, either: ```ruby Rails.application.load_tasks; nil - Gitlab::Geo.expire_cache_keys!([:primary_node, :current_node]) + Gitlab::Geo.expire_cache! Rake::Task['geo:set_secondary_as_primary'].invoke ``` @@ -568,7 +575,7 @@ is displayed if you attempt to run this command on a primary node. ### Message: `sudo: gitlab-pg-ctl: command not found` When -[promoting a **secondary** node with HA](../disaster_recovery/index.md#promoting-a-secondary-node-with-ha), +[promoting a **secondary** node with multiple servers](../disaster_recovery/index.md#promoting-a-secondary-node-with-multiple-servers), you need to run the `gitlab-pg-ctl` command to promote the PostgreSQL read-replica database. diff --git a/doc/administration/geo/replication/tuning.md b/doc/administration/geo/replication/tuning.md index 972bf002935..63a8f81eecb 100644 --- a/doc/administration/geo/replication/tuning.md +++ b/doc/administration/geo/replication/tuning.md @@ -1,3 +1,10 @@ +--- +stage: Enablement +group: Geo +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +type: howto +--- + # Tuning Geo **(PREMIUM ONLY)** ## Changing the sync capacity values diff --git a/doc/administration/geo/replication/updating_the_geo_nodes.md b/doc/administration/geo/replication/updating_the_geo_nodes.md index fa1576e19eb..6c2778ad0fe 100644 --- a/doc/administration/geo/replication/updating_the_geo_nodes.md +++ b/doc/administration/geo/replication/updating_the_geo_nodes.md @@ -1,3 +1,10 @@ +--- +stage: Enablement +group: Geo +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +type: howto +--- + # Updating the Geo nodes **(PREMIUM ONLY)** Updating Geo nodes involves performing: diff --git a/doc/administration/geo/replication/using_a_geo_server.md b/doc/administration/geo/replication/using_a_geo_server.md index 2fec2b2b59c..3f2895f1c71 100644 --- a/doc/administration/geo/replication/using_a_geo_server.md +++ b/doc/administration/geo/replication/using_a_geo_server.md @@ -1,3 +1,10 @@ +--- +stage: Enablement +group: Geo +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +type: howto +--- + <!-- Please update EE::GitLab::GeoGitAccess::GEO_SERVER_DOCS_URL if this file is moved) --> # Using a Geo Server **(PREMIUM ONLY)** diff --git a/doc/administration/geo/replication/version_specific_updates.md b/doc/administration/geo/replication/version_specific_updates.md index db8bbddec3b..777de715a8c 100644 --- a/doc/administration/geo/replication/version_specific_updates.md +++ b/doc/administration/geo/replication/version_specific_updates.md @@ -1,3 +1,10 @@ +--- +stage: Enablement +group: Geo +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +type: howto +--- + # Version specific update instructions Check this document if it includes instructions for the version you are updating. @@ -16,7 +23,7 @@ The issue is fixed in GitLab 12.9.4. Please upgrade to GitLab 12.9.4 or later. DANGER: **Danger:** Only upgrade to GitLab 12.7.5 or later. Do not upgrade to versions 12.7.0 through 12.7.4 because there is [an initialization order -bug](https://gitlab.com/gitlab-org/gitlab/issues/199672) that causes Geo +bug](https://gitlab.com/gitlab-org/gitlab/-/issues/199672) that causes Geo **secondaries** to set the incorrect database connection pool size. [The fix](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/24021) was shipped in 12.7.5. @@ -56,14 +63,14 @@ sudo touch /etc/gitlab/disable-postgresql-upgrade CAUTION: **Warning:** This version is affected by [a bug that results in new LFS objects not being replicated to -Geo secondary nodes](https://gitlab.com/gitlab-org/gitlab/issues/32696). The issue is fixed +Geo secondary nodes](https://gitlab.com/gitlab-org/gitlab/-/issues/32696). The issue is fixed in GitLab 12.1. Please upgrade to GitLab 12.1 or later. ## Updating to GitLab 11.11 CAUTION: **Warning:** This version is affected by [a bug that results in new LFS objects not being replicated to -Geo secondary nodes](https://gitlab.com/gitlab-org/gitlab/issues/32696). The issue is fixed +Geo secondary nodes](https://gitlab.com/gitlab-org/gitlab/-/issues/32696). The issue is fixed in GitLab 12.1. Please upgrade to GitLab 12.1 or later. ## Updating to GitLab 10.8 diff --git a/doc/administration/git_annex.md b/doc/administration/git_annex.md index 0d44ed9312c..c4c38f8e683 100644 --- a/doc/administration/git_annex.md +++ b/doc/administration/git_annex.md @@ -5,7 +5,7 @@ disqus_identifier: 'https://docs.gitlab.com/ee/workflow/git_annex.html' # Git annex > **Warning:** GitLab has [completely -removed](https://gitlab.com/gitlab-org/gitlab/issues/1648) in GitLab 9.0 (2017/03/22). +removed](https://gitlab.com/gitlab-org/gitlab/-/issues/1648) in GitLab 9.0 (2017/03/22). Read through the [migration guide from git-annex to Git LFS](../topics/git/lfs/migrate_from_git_annex_to_git_lfs.md). The biggest limitation of Git, compared to some older centralized version diff --git a/doc/administration/git_protocol.md b/doc/administration/git_protocol.md index a8e785f9344..e1600d972bd 100644 --- a/doc/administration/git_protocol.md +++ b/doc/administration/git_protocol.md @@ -4,9 +4,9 @@ description: "Set and configure Git protocol v2" # Configuring Git Protocol v2 -> - [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/issues/46555) in GitLab 11.4. -> - [Temporarily disabled](https://gitlab.com/gitlab-org/gitlab-foss/issues/55769) in GitLab 11.5.8, 11.6.6, 11.7.1, and 11.8+. -> - [Re-enabled](https://gitlab.com/gitlab-org/gitlab/issues/27828) in GitLab 12.8. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/46555) in GitLab 11.4. +> - [Temporarily disabled](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/55769) in GitLab 11.5.8, 11.6.6, 11.7.1, and 11.8+. +> - [Re-enabled](https://gitlab.com/gitlab-org/gitlab/-/issues/27828) in GitLab 12.8. Git protocol v2 improves the v1 wire protocol in several ways and is enabled by default in GitLab for HTTP requests. In order to enable SSH, @@ -24,7 +24,7 @@ From the server side, if we want to configure SSH we need to set the `sshd` server to accept the `GIT_PROTOCOL` environment. In installations using [GitLab Helm Charts](https://docs.gitlab.com/charts/) -and [All-in-one docker image](https://docs.gitlab.com/omnibus/docker/), the SSH +and [All-in-one Docker image](https://docs.gitlab.com/omnibus/docker/), the SSH service is already configured to accept the `GIT_PROTOCOL` environment and users need not do anything more. diff --git a/doc/administration/gitaly/index.md b/doc/administration/gitaly/index.md index 14b0a6bd450..1469ed64004 100644 --- a/doc/administration/gitaly/index.md +++ b/doc/administration/gitaly/index.md @@ -1,14 +1,25 @@ +--- +stage: Create +group: Gitaly +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +type: reference +--- + # Gitaly -[Gitaly](https://gitlab.com/gitlab-org/gitaly) is the service that -provides high-level RPC access to Git repositories. Without it, no other -components can read or write Git data. GitLab components that access Git -repositories (GitLab Rails, GitLab Shell, GitLab Workhorse, etc.) act as clients -to Gitaly. End users do not have direct access to Gitaly. +[Gitaly](https://gitlab.com/gitlab-org/gitaly) is the service that provides high-level RPC access to +Git repositories. Without it, no GitLab components can read or write Git data. + +In the Gitaly documentation: -On this page, *Gitaly server* refers to a standalone node that only runs Gitaly -and *Gitaly client* is a GitLab Rails app node that runs all other processes -except Gitaly. +- **Gitaly server** refers to any node that runs Gitaly itself. +- **Gitaly client** refers to any node that runs a process that makes requests of the + Gitaly server. Processes include, but are not limited to: + - [GitLab Rails application](https://gitlab.com/gitlab-org/gitlab). + - [GitLab Shell](https://gitlab.com/gitlab-org/gitlab-shell). + - [GitLab Workhorse](https://gitlab.com/gitlab-org/gitlab-workhorse). + +GitLab end users do not have direct access to Gitaly. CAUTION: **Caution:** From GitLab 13.0, using NFS for Git repositories is deprecated. In GitLab 14.0, @@ -17,19 +28,20 @@ support for NFS for Git repositories is scheduled to be removed. Upgrade to ## Architecture -Here's a high-level architecture overview of how Gitaly is used. +The following is a high-level architecture overview of how Gitaly is used.  -## Configuring Gitaly +## Configure Gitaly The Gitaly service itself is configured via a [TOML configuration file](reference.md). -If you want to change any of its settings: +To change Gitaly settings: **For Omnibus GitLab** -1. Edit `/etc/gitlab/gitlab.rb` and add or change the [Gitaly settings](https://gitlab.com/gitlab-org/omnibus-gitlab/blob/1dd07197c7e5ae23626aad5a4a070a800b670380/files/gitlab-config-template/gitlab.rb.template#L1622-1676). +1. Edit `/etc/gitlab/gitlab.rb` and add or change the + [Gitaly settings](https://gitlab.com/gitlab-org/omnibus-gitlab/blob/1dd07197c7e5ae23626aad5a4a070a800b670380/files/gitlab-config-template/gitlab.rb.template#L1622-1676). 1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). **For installations from source** @@ -37,92 +49,121 @@ If you want to change any of its settings: 1. Edit `/home/git/gitaly/config.toml` and add or change the [Gitaly settings](https://gitlab.com/gitlab-org/gitaly/blob/master/config.toml.example). 1. Save the file and [restart GitLab](../restart_gitlab.md#installations-from-source). -## Running Gitaly on its own server +## Run Gitaly on its own server + +By default, Gitaly is run on the same server as Gitaly clients and is +[configured as above](#configure-gitaly). Single-server installations are best served by +this default configuration used by: -This is an optional way to deploy Gitaly which can benefit GitLab -installations that are larger than a single machine. Most -installations will be better served with the default configuration -used by Omnibus and the GitLab source installation guide. -Follow transition to Gitaly on its own server, [Gitaly servers will need to be upgraded before other servers in your cluster](https://docs.gitlab.com/omnibus/update/#upgrading-gitaly-servers). +- [Omnibus GitLab](https://docs.gitlab.com/omnibus/). +- The GitLab [source installation guide](../../install/installation.md). -Starting with GitLab 11.4, Gitaly is able to serve all Git requests without -requiring a shared NFS mount for Git repository data. -Between 11.4 and 11.8 the exception was the -[Elasticsearch indexer](https://gitlab.com/gitlab-org/gitlab-elasticsearch-indexer). -But since 11.8 the indexer uses Gitaly for data access as well. NFS can still -be leveraged for redundancy on block level of the Git data. But only has to -be mounted on the Gitaly server. +However, Gitaly can be deployed to its own server, which can benefit GitLab installations that span +multiple machines. -From GitLab v11.8 to v12.2, it is possible to use Elasticsearch in conjunction with -a Gitaly setup that isn't utilising NFS. In order to use Elasticsearch in this -scenario, the [new repository indexer](../../integration/elasticsearch.md#elasticsearch-repository-indexer) -needs to be enabled in your GitLab configuration. [Since GitLab v12.3](https://gitlab.com/gitlab-org/gitlab/issues/6481), -the new indexer becomes the default and no configuration is required. +NOTE: **Note:** +When configured to run on their own servers, Gitaly servers +[must be upgraded](https://docs.gitlab.com/omnibus/update/#upgrading-gitaly-servers) before Gitaly +clients in your cluster. + +The process for setting up Gitaly on its own server is: + +1. [Install Gitaly](#install-gitaly). +1. [Configure authentication](#configure-authentication). +1. [Configure Gitaly servers](#configure-gitaly-servers). +1. [Configure Gitaly clients](#configure-gitaly-clients). + +When running Gitaly on its own server, note the following regarding GitLab versions: + +- From GitLab 11.4, Gitaly was able to serve all Git requests without requiring a shared NFS mount + for Git repository data, except for the + [Elasticsearch indexer](https://gitlab.com/gitlab-org/gitlab-elasticsearch-indexer). +- From GitLab 11.8, the Elasticsearch indexer uses Gitaly for data access as well. NFS can still be + leveraged for redundancy on block-level Git data, but only has to be mounted on the Gitaly + servers. +- From GitLab 11.8 to 12.2, it is possible to use Elasticsearch in a Gitaly setup that doesn't use + NFS. In order to use Elasticsearch in these versions, the + [repository indexer](../../integration/elasticsearch.md#elasticsearch-repository-indexer) + must be enabled in your GitLab configuration. +- [Since GitLab 12.3](https://gitlab.com/gitlab-org/gitlab/-/issues/6481), the new indexer is + the default and no configuration is required. ### Network architecture -The following list depicts what the network architecture of Gitaly is: +The following list depicts the network architecture of Gitaly: - GitLab Rails shards repositories into [repository storages](../repository_storage_paths.md). -- `/config/gitlab.yml` contains a map from storage names to - `(Gitaly address, Gitaly token)` pairs. -- the `storage name` -\> `(Gitaly address, Gitaly token)` map in - `/config/gitlab.yml` is the single source of truth for the Gitaly network - topology. +- `/config/gitlab.yml` contains a map from storage names to `(Gitaly address, Gitaly token)` pairs. +- The `storage name` -\> `(Gitaly address, Gitaly token)` map in `/config/gitlab.yml` is the single + source of truth for the Gitaly network topology. - A `(Gitaly address, Gitaly token)` corresponds to a Gitaly server. - A Gitaly server hosts one or more storages. -- A GitLab server can use one or more Gitaly servers. -- Gitaly addresses must be specified in such a way that they resolve - correctly for ALL Gitaly clients. -- Gitaly clients are: Puma/Unicorn, Sidekiq, GitLab Workhorse, - GitLab Shell, Elasticsearch Indexer, and Gitaly itself. +- A Gitaly client can use one or more Gitaly servers. +- Gitaly addresses must be specified in such a way that they resolve correctly for **all** Gitaly + clients. +- Gitaly clients are: + - Puma or Unicorn. + - Sidekiq. + - GitLab Workhorse. + - GitLab Shell. + - Elasticsearch indexer. + - Gitaly itself. - A Gitaly server must be able to make RPC calls **to itself** via its own `(Gitaly address, Gitaly token)` pair as specified in `/config/gitlab.yml`. -- Gitaly servers must not be exposed to the public internet as Gitaly's network - traffic is unencrypted by default. The use of firewall is highly recommended - to restrict access to the Gitaly server. Another option is to - [use TLS](#tls-support). -- Authentication is done through a static token which is shared among the Gitaly - and GitLab Rails nodes. +- Authentication is done through a static token which is shared among the Gitaly and GitLab Rails + nodes. + +DANGER: **Danger:** +Gitaly servers must not be exposed to the public internet as Gitaly's network traffic is unencrypted +by default. The use of firewall is highly recommended to restrict access to the Gitaly server. +Another option is to [use TLS](#tls-support). + +In the following sections, we describe how to configure two Gitaly servers with secret token +`abc123secret`: -Below we describe how to configure two Gitaly servers one at -`gitaly1.internal` and the other at `gitaly2.internal` -with secret token `abc123secret`. We assume -your GitLab installation has three repository storages: `default`, -`storage1` and `storage2`. You can use as little as just one server with one -repository storage if desired. +- `gitaly1.internal`. +- `gitaly2.internal`. -Note: **Note:** The token referred to throughout the Gitaly documentation is -just an arbitrary password selected by the administrator. It is unrelated to -tokens created for the GitLab API or other similar web API tokens. +We assume your GitLab installation has three repository storages: -### 1. Installation +- `default`. +- `storage1`. +- `storage2`. -First install Gitaly on each Gitaly server using either -Omnibus GitLab or install it from source: +You can use as few as one server with one repository storage if desired. -- For Omnibus GitLab: [Download/install](https://about.gitlab.com/install/) the Omnibus GitLab - package you want using **steps 1 and 2** from the GitLab downloads page but - **_do not_** provide the `EXTERNAL_URL=` value. -- From source: [Install Gitaly](../../install/installation.md#install-gitaly). +NOTE: **Note:** +The token referred to throughout the Gitaly documentation is just an arbitrary password selected by +the administrator. It is unrelated to tokens created for the GitLab API or other similar web API +tokens. + +### Install Gitaly + +Install Gitaly on each Gitaly server using either Omnibus GitLab or install it from source: + +- For Omnibus GitLab, [download and install](https://about.gitlab.com/install/) the Omnibus GitLab + package you want but **do not** provide the `EXTERNAL_URL=` value. +- To install from source, follow the steps at + [Install Gitaly](../../install/installation.md#install-gitaly). -### 2. Authentication +### Configure authentication -Gitaly and GitLab use two shared secrets for authentication, one to authenticate gRPC requests -to Gitaly, and a second for authentication callbacks from GitLab-Shell to the GitLab internal API. +Gitaly and GitLab use two shared secrets for authentication: + +- One to authenticate gRPC requests to Gitaly. +- A second for authentication callbacks from GitLab Shell to the GitLab internal API. **For Omnibus GitLab** To configure the Gitaly token: -1. On the client server, edit `/etc/gitlab/gitlab.rb`: +1. On the Gitaly clients, edit `/etc/gitlab/gitlab.rb`: ```ruby gitlab_rails['gitaly_token'] = 'abc123secret' ``` 1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). - 1. On the Gitaly server, edit `/etc/gitlab/gitlab.rb`: ```ruby @@ -131,22 +172,24 @@ To configure the Gitaly token: 1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). -There are two ways to configure the GitLab-Shell token: +There are two ways to configure the GitLab Shell token. -1. Copy `/etc/gitlab/gitlab-secrets.json` from the client server to same path on the Gitaly server. -1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). +Method 1: + +1. Copy `/etc/gitlab/gitlab-secrets.json` from the Gitaly client to same path on the Gitaly servers + (and any other Gitaly clients). +1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) on Gitaly servers. -**OR** +Method 2: -1. On the client server, edit `/etc/gitlab/gitlab.rb`: +1. On the Gitaly clients, edit `/etc/gitlab/gitlab.rb`: ```ruby gitlab_shell['secret_token'] = 'shellsecret' ``` 1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). - -1. On the Gitaly server, edit `/etc/gitlab/gitlab.rb`: +1. On the Gitaly servers, edit `/etc/gitlab/gitlab.rb`: ```ruby gitlab_shell['secret_token'] = 'shellsecret' @@ -156,9 +199,9 @@ There are two ways to configure the GitLab-Shell token: **For installations from source** -1. Copy `/home/git/gitlab/.gitlab_shell_secret` from the client server to the same path on the Gitaly -server. -1. On the client server, edit `/home/git/gitlab/config/gitlab.yml`: +1. Copy `/home/git/gitlab/.gitlab_shell_secret` from the Gitaly client to the same path on the + Gitaly servers (and any other Gitaly clients). +1. On the Gitaly clients, edit `/home/git/gitlab/config/gitlab.yml`: ```yaml gitlab: @@ -167,17 +210,22 @@ server. ``` 1. Save the file and [restart GitLab](../restart_gitlab.md#installations-from-source). +1. On the Gitaly servers, edit `/home/git/gitaly/config.toml`: + + ```toml + [auth] + token = 'abc123secret' + ``` -### 3. Gitaly server configuration +1. Save the file and [restart GitLab](../restart_gitlab.md#installations-from-source). -Next, on the Gitaly servers, you need to configure storage paths, enable -the network listener and configure the token. +### Configure Gitaly servers -NOTE: **Note:** If you want to reduce the risk of downtime when you enable -authentication you can temporarily disable enforcement, see [the -documentation on configuring Gitaly -authentication](https://gitlab.com/gitlab-org/gitaly/blob/master/doc/configuration/README.md#authentication) -. +On the Gitaly servers, you must configure storage paths and enable the network listener. + +If you want to reduce the risk of downtime when you enable authentication, you can temporarily +disable enforcement. For more information, see the documentation on configuring +[Gitaly authentication](https://gitlab.com/gitlab-org/gitaly/blob/master/doc/configuration/README.md#authentication). **For Omnibus GitLab** @@ -199,6 +247,7 @@ authentication](https://gitlab.com/gitlab-org/gitaly/blob/master/doc/configurati sidekiq['enable'] = false gitlab_workhorse['enable'] = false grafana['enable'] = false + gitlab_exporter['enable'] = false # If you run a separate monitoring node you can disable these services alertmanager['enable'] = false @@ -211,7 +260,6 @@ authentication](https://gitlab.com/gitlab-org/gitaly/blob/master/doc/configurati # prometheus['monitor_kubernetes'] = false # If you don't want to run monitoring services uncomment the following (not recommended) - # gitlab_exporter['enable'] = false # node_exporter['enable'] = false # Prevent database connections during 'gitlab-ctl reconfigure' @@ -221,7 +269,7 @@ authentication](https://gitlab.com/gitlab-org/gitaly/blob/master/doc/configurati # Configure the gitlab-shell API callback URL. Without this, `git push` will # fail. This can be your 'front door' GitLab URL or an internal load # balancer. - # Don't forget to copy `/etc/gitlab/gitlab-secrets.json` from web server to Gitaly server. + # Don't forget to copy `/etc/gitlab/gitlab-secrets.json` from Gitaly client to Gitaly server. gitlab_rails['internal_api_url'] = 'https://gitlab.example.com' # Make Gitaly accept connections on all network interfaces. You must use @@ -230,7 +278,7 @@ authentication](https://gitlab.com/gitlab-org/gitaly/blob/master/doc/configurati gitaly['listen_addr'] = "0.0.0.0:8075" ``` -1. Append the following to `/etc/gitlab/gitlab.rb` for each respective server: +1. Append the following to `/etc/gitlab/gitlab.rb` for each respective Gitaly server: <!-- updates to following example must also be made at @@ -262,27 +310,24 @@ authentication](https://gitlab.com/gitlab-org/gitaly/blob/master/doc/configurati 1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). 1. Run `sudo /opt/gitlab/embedded/service/gitlab-shell/bin/check -config /opt/gitlab/embedded/service/gitlab-shell/config.yml` -to confirm that Gitaly can perform callbacks to the internal API. + to confirm that Gitaly can perform callbacks to the GitLab internal API. **For installations from source** -1. On the client node(s), edit `/home/git/gitaly/config.toml`: +1. Edit `/home/git/gitaly/config.toml`: ```toml listen_addr = '0.0.0.0:8075' internal_socket_dir = '/var/opt/gitlab/gitaly' - [auth] - token = 'abc123secret' - [logging] format = 'json' level = 'info' dir = '/var/log/gitaly' ``` -1. Append the following to `/home/git/gitaly/config.toml` for each respective server: +1. Append the following to `/home/git/gitaly/config.toml` for each respective Gitaly server: On `gitaly1.internal`: @@ -304,34 +349,51 @@ to confirm that Gitaly can perform callbacks to the internal API. path = '/srv/gitlab/git-data/repositories' ``` -1. On each Gitaly server, edit `/home/git/gitlab-shell/config.yml`: +1. Edit `/home/git/gitlab-shell/config.yml`: ```yaml gitlab_url: https://gitlab.example.com ``` -1. Save the file and [restart GitLab](../restart_gitlab.md#installations-from-source). +1. Save the files and [restart GitLab](../restart_gitlab.md#installations-from-source). 1. Run `sudo -u git /home/git/gitlab-shell/bin/check -config /home/git/gitlab-shell/config.yml` -to confirm that Gitaly can perform callbacks to the internal API. + to confirm that Gitaly can perform callbacks to the GitLab internal API. + +### Configure Gitaly clients -### 4. Converting clients to use the Gitaly server +As the final step, you must update Gitaly clients to switch from using local Gitaly service to use +the Gitaly servers you just configured. -As the final step, you need to update the client machines to switch from using -their local Gitaly service to the new Gitaly server you just configured. This -is a risky step because if there is any sort of network, firewall, or name -resolution problem preventing your GitLab server from reaching the Gitaly server, -then all Gitaly requests will fail. +This can be risky because anything that prevents your Gitaly clients from reaching the Gitaly +servers will cause all Gitaly requests to fail. For example, any sort of network, firewall, or name +resolution problems. -Additionally, you need to -[disable Rugged if previously manually enabled](../high_availability/nfs.md#improving-nfs-performance-with-gitlab). +Additionally, you must [disable Rugged](../high_availability/nfs.md#improving-nfs-performance-with-gitlab) +if previously enabled manually. -We assume that your `gitaly1.internal` Gitaly server can be reached at -`gitaly1.internal:8075` from your GitLab server, and that Gitaly server -can read and write to `/mnt/gitlab/default` and `/mnt/gitlab/storage1`. +Gitaly makes the following assumptions: -We assume also that your `gitaly2.internal` Gitaly server can be reached at -`gitaly2.internal:8075` from your GitLab server, and that Gitaly server -can read and write to `/mnt/gitlab/storage2`. +- Your `gitaly1.internal` Gitaly server can be reached at `gitaly1.internal:8075` from your Gitaly + clients, and that Gitaly server can read and write to `/mnt/gitlab/default` and + `/mnt/gitlab/storage1`. +- Your `gitaly2.internal` Gitaly server can be reached at `gitaly2.internal:8075` from your Gitaly + clients, and that Gitaly server can read and write to `/mnt/gitlab/storage2`. +- Your `gitaly1.internal` and `gitaly2.internal` Gitaly servers can reach each other. + +Note you can't a use mixed setup, with at least one of your Gitaly servers configured as a local +server with the `path` setting provided. This is because other Gitaly instances can't communicate +with it. The following setup is _incorrect_, because: + +- You must replace `path` with `gitaly_address` containing a proper value. +- The address must be reachable from the other two addresses provided. + +```ruby +git_data_dirs({ + 'default' => { 'gitaly_address' => 'tcp://gitaly1.internal:8075' }, + 'storage1' => { 'path' => '/var/opt/gitlab/git-data' }, + 'storage2' => { 'gitaly_address' => 'tcp://gitaly2.internal:8075' }, +}) +``` **For Omnibus GitLab** @@ -346,7 +408,8 @@ can read and write to `/mnt/gitlab/storage2`. ``` 1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). -1. Run `sudo gitlab-rake gitlab:gitaly:check` to confirm the client can connect to Gitaly. +1. Run `sudo gitlab-rake gitlab:gitaly:check` to confirm the Gitaly client can connect to Gitaly + servers. 1. Tail the logs to see the requests: ```shell @@ -373,39 +436,38 @@ can read and write to `/mnt/gitlab/storage2`. ``` NOTE: **Note:** - `/some/dummy/path` should be set to a local folder that exists, however no - data will be stored in this folder. This will no longer be necessary after - [this issue](https://gitlab.com/gitlab-org/gitaly/issues/1282) is resolved. + `/some/dummy/path` should be set to a local folder that exists, however no data will be stored in + this folder. This will no longer be necessary after + [this issue](https://gitlab.com/gitlab-org/gitaly/-/issues/1282) is resolved. 1. Save the file and [restart GitLab](../restart_gitlab.md#installations-from-source). -1. Run `sudo -u git -H bundle exec rake gitlab:gitaly:check RAILS_ENV=production` to -confirm the client can connect to Gitaly. +1. Run `sudo -u git -H bundle exec rake gitlab:gitaly:check RAILS_ENV=production` to confirm the + Gitaly client can connect to Gitaly servers. 1. Tail the logs to see the requests: ```shell tail -f /home/git/gitlab/log/gitaly.log ``` -When you tail the Gitaly logs on your Gitaly server you should see requests -coming in. One sure way to trigger a Gitaly request is to clone a repository -from your GitLab server over HTTP. +When you tail the Gitaly logs on your Gitaly server, you should see requests coming in. One sure way +to trigger a Gitaly request is to clone a repository from GitLab over HTTP or HTTPS. DANGER: **Danger:** -If you have [Server hooks](../server_hooks.md) configured, -either per repository or globally, you must move these to the Gitaly node. -If you have multiple Gitaly nodes, copy your server hook(s) to all nodes. +If you have [server hooks](../server_hooks.md) configured, either per repository or globally, you +must move these to the Gitaly servers. If you have multiple Gitaly servers, copy your server hooks +to all Gitaly servers. ### Disabling the Gitaly service in a cluster environment If you are running Gitaly [as a remote -service](#running-gitaly-on-its-own-server) you may want to disable +service](#run-gitaly-on-its-own-server) you may want to disable the local Gitaly service that runs on your GitLab server by default. Disabling Gitaly only makes sense when you run GitLab in a custom cluster configuration, where different services run on different machines. Disabling Gitaly on all machines in the cluster is not a valid configuration. -To disable Gitaly on a client node: +To disable Gitaly on a GitLab server: **For Omnibus GitLab** @@ -436,13 +498,16 @@ with a Gitaly instance that listens for secure connections you will need to use scheme in the `gitaly_address` of the corresponding storage entry in the GitLab configuration. You will need to bring your own certificates as this isn't provided automatically. -The certificate, or its certificate authority, must be installed on all Gitaly -nodes (including the Gitaly node using the certificate) and on all client nodes +The certificate corresponding to each Gitaly server will need to be installed +on that Gitaly server. + +Additionally the certificate, or its certificate authority, must be installed on all Gitaly servers +(including the Gitaly server using the certificate) and on all Gitaly clients that communicate with it following the procedure described in -[GitLab custom certificate configuration](https://docs.gitlab.com/omnibus/settings/ssl.html#install-custom-public-certificates). +[GitLab custom certificate configuration](https://docs.gitlab.com/omnibus/settings/ssl.html#install-custom-public-certificates) (and repeated below). NOTE: **Note** -The self-signed certificate must specify the address you use to access the +The certificate must specify the address you use to access the Gitaly server. If you are addressing the Gitaly server by a hostname, you can either use the Common Name field for this, or add it as a Subject Alternative Name. If you are addressing the Gitaly server by its IP address, you must add it @@ -459,7 +524,14 @@ To configure Gitaly with TLS: **For Omnibus GitLab** -1. On the client node(s), edit `/etc/gitlab/gitlab.rb` as follows: +1. Create certificates for Gitaly servers. +1. On the Gitaly clients, copy the certificates, or their certificate authority, into the `/etc/gitlab/trusted-certs`: + + ```shell + sudo cp cert.pem /etc/gitlab/trusted-certs/ + ``` + +1. On the Gitaly clients, edit `git_data_dirs` in `/etc/gitlab/gitlab.rb` as follows: ```ruby git_data_dirs({ @@ -467,19 +539,10 @@ To configure Gitaly with TLS: 'storage1' => { 'gitaly_address' => 'tls://gitaly1.internal:9999' }, 'storage2' => { 'gitaly_address' => 'tls://gitaly2.internal:9999' }, }) - - gitlab_rails['gitaly_token'] = 'abc123secret' - gitlab_shell['secret_token'] = 'shellsecret' ``` -1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) on client node(s). -1. On the client node(s), copy the cert into the `/etc/gitlab/trusted-certs`: - - ```shell - sudo cp cert.pem /etc/gitlab/trusted-certs/ - ``` - -1. On the Gitaly server, create the `/etc/gitlab/ssl` directory and copy your key and certificate there: +1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). +1. On the Gitaly servers, create the `/etc/gitlab/ssl` directory and copy your key and certificate there: ```shell sudo mkdir -p /etc/gitlab/ssl @@ -488,14 +551,14 @@ To configure Gitaly with TLS: sudo chmod 644 key.pem cert.pem ``` -1. Copy the cert to `/etc/gitlab/trusted-certs` so Gitaly will trust the cert when -calling into itself: +1. Copy all Gitaly server certificates, or their certificate authority, to `/etc/gitlab/trusted-certs` so Gitaly server will trust the certificate when +calling into itself or other Gitaly servers: - ```shell - sudo cp /etc/gitlab/ssl/cert.pem /etc/gitlab/trusted-certs/ - ``` + ```shell + sudo cp cert1.pem cert2.pem /etc/gitlab/trusted-certs/ + ``` -1. On the Gitaly server node(s), edit `/etc/gitlab/gitlab.rb` and add: +1. Edit `/etc/gitlab/gitlab.rb` and add: <!-- updates to following example must also be made at @@ -508,23 +571,23 @@ calling into itself: gitaly['key_path'] = "/etc/gitlab/ssl/key.pem" ``` -1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) on Gitaly server node(s). +1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). 1. (Optional) After [verifying that all Gitaly traffic is being served over TLS](#observe-type-of-gitaly-connections), you can improve security by disabling non-TLS connections by commenting out or deleting `gitaly['listen_addr']` in `/etc/gitlab/gitlab.rb`, saving the file, - and [reconfiguring GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) - on Gitaly server node(s). + and [reconfiguring GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). **For installations from source** -1. On the client node(s), add the cert to the system trusted certs: +1. Create certificates for Gitaly servers. +1. On the Gitaly clients, copy the certificates into the system trusted certificates: ```shell sudo cp cert.pem /usr/local/share/ca-certificates/gitaly.crt sudo update-ca-certificates ``` -1. On the client node(s), edit `/home/git/gitlab/config/gitlab.yml` as follows: +1. On the Gitaly clients, edit `storages` in `/home/git/gitlab/config/gitlab.yml` as follows: ```yaml gitlab: @@ -539,28 +602,21 @@ calling into itself: storage2: gitaly_address: tls://gitaly2.internal:9999 path: /some/dummy/path - - gitaly: - token: 'abc123secret' ``` NOTE: **Note:** `/some/dummy/path` should be set to a local folder that exists, however no data will be stored in this folder. This will no longer be necessary after - [this issue](https://gitlab.com/gitlab-org/gitaly/issues/1282) is resolved. + [this issue](https://gitlab.com/gitlab-org/gitaly/-/issues/1282) is resolved. -1. Save the file and[restart GitLab](../restart_gitlab.md#installations-from-source) -on client node(s). -1. Copy `/home/git/gitlab/.gitlab_shell_secret` from the client server to the same -path on the Gitaly server. -1. On the Gitaly server, create or edit `/etc/default/gitlab` and add: +1. Save the file and [restart GitLab](../restart_gitlab.md#installations-from-source). +1. On the Gitaly servers, create or edit `/etc/default/gitlab` and add: ```shell export SSL_CERT_DIR=/etc/gitlab/ssl ``` -1. Save the file. -1. Create the `/etc/gitlab/ssl` directory and copy your key and certificate there: +1. On the Gitaly servers, create the `/etc/gitlab/ssl` directory and copy your key and certificate there: ```shell sudo mkdir -p /etc/gitlab/ssl @@ -569,15 +625,14 @@ path on the Gitaly server. sudo chmod 644 key.pem cert.pem ``` -1. On the Gitaly server, add the cert to the system trusted certs so Gitaly will trust it -when calling into itself: +1. Copy all Gitaly server certificates, or their certificate authority, to the system trusted certificates so Gitaly server will trust the certificate when calling into itself or other Gitaly servers. ```shell sudo cp cert.pem /usr/local/share/ca-certificates/gitaly.crt sudo update-ca-certificates ``` -1. On the Gitaly server node(s), edit `/home/git/gitaly/config.toml` and add: +1. Edit `/home/git/gitaly/config.toml` and add: ```toml tls_listen_addr = '0.0.0.0:9999' @@ -587,12 +642,11 @@ when calling into itself: key_path = '/etc/gitlab/ssl/key.pem' ``` -1. Save the file and [restart GitLab](../restart_gitlab.md#installations-from-source) on Gitaly server node(s). +1. Save the file and [restart GitLab](../restart_gitlab.md#installations-from-source). 1. (Optional) After [verifying that all Gitaly traffic is being served over TLS](#observe-type-of-gitaly-connections), you can improve security by disabling non-TLS connections by commenting out or deleting `listen_addr` in `/home/git/gitaly/config.toml`, saving the file, - and [restarting GitLab](../restart_gitlab.md#installations-from-source) - on Gitaly server node(s). + and [restarting GitLab](../restart_gitlab.md#installations-from-source). ### Observe type of Gitaly connections @@ -673,8 +727,8 @@ gitaly['concurrency'] = [ This will limit the number of in-flight RPC calls for the given RPC's. The limit is applied per repository. In the example above, each on the -Gitaly server can have at most 20 simultaneous PostUploadPack calls in -flight, and the same for SSHUploadPack. If another request comes in for +Gitaly server can have at most 20 simultaneous `PostUploadPack` calls in +flight, and the same for `SSHUploadPack`. If another request comes in for a repository that has used up its 20 slots, that request will get queued. @@ -686,7 +740,7 @@ reporting about the concurrency limiter. In Prometheus, look for the `gitaly_rate_limiting_seconds` metrics. The name of the Prometheus metric is not quite right because this is a -concurrency limiter, not a rate limiter. If a client makes 1000 requests +concurrency limiter, not a rate limiter. If a Gitaly client makes 1000 requests in a row in a very short timespan, the concurrency will not exceed 1, and this mechanism (the concurrency limiter) will do nothing. @@ -697,7 +751,7 @@ downtime, or causes outages, or both. If you are careful, though, you *can* rotate Gitaly credentials without a service interruption. This procedure also works if you are running GitLab on a single server. -In that case, "Gitaly servers" and "Gitaly clients" refers to the same +In that case, "Gitaly server" and "Gitaly client" refers to the same machine. ### 1. Monitor current authentication behavior @@ -723,7 +777,7 @@ The only non-zero number should have `enforced="true",status="ok"`. If you have other non-zero numbers, something is wrong in your configuration. -The 'status="ok"' number reflects your current request rate. In the example +The `status="ok"` number reflects your current request rate. In the example above, Gitaly is handling about 4000 requests per second. Now you have established that you can monitor the Gitaly authentication @@ -894,9 +948,9 @@ which is still under development as of December 2019. ## Troubleshooting Gitaly -### Checking versions when using standalone Gitaly nodes +### Checking versions when using standalone Gitaly servers -When using standalone Gitaly nodes, you must make sure they are the same version +When using standalone Gitaly servers, you must make sure they are the same version as GitLab to ensure full compatibility. Check **Admin Area > Gitaly Servers** on your GitLab instance and confirm all Gitaly Servers are `Up to date`. @@ -931,8 +985,8 @@ gitaly-debug -h remote: GitLab: 401 Unauthorized ``` -You will need to sync your `gitlab-secrets.json` file with your GitLab -app nodes. +You will need to sync your `gitlab-secrets.json` file with your Gitaly clients (GitLab +app nodes). ### Client side gRPC logs @@ -975,11 +1029,11 @@ If you're running Gitaly on its own server and notice that users can successfully clone and fetch repositories (via both SSH and HTTPS), but can't push to them or make changes to the repository in the web UI without getting a `401 Unauthorized` message, then it's possible Gitaly is failing to authenticate -with the other nodes due to having the [wrong secrets file](#3-gitaly-server-configuration). +with the Gitaly client due to having the [wrong secrets file](#configure-gitaly-servers). Confirm the following are all true: -- When any user performs a `git push` to any repository on this Gitaly node, it +- When any user performs a `git push` to any repository on this Gitaly server, it fails with the following error (note the `401 Unauthorized`): ```shell @@ -993,7 +1047,8 @@ Confirm the following are all true: UI, it immediately fails with a red `401 Unauthorized` banner. - Creating a new project and [initializing it with a README](../../gitlab-basics/create-project.md#blank-projects) successfully creates the project but doesn't create the README. -- When [tailing the logs](https://docs.gitlab.com/omnibus/settings/logs.html#tail-logs-in-a-console-on-the-server) on an app node and reproducing the error, you get `401` errors +- When [tailing the logs](https://docs.gitlab.com/omnibus/settings/logs.html#tail-logs-in-a-console-on-the-server) + on a Gitaly client and reproducing the error, you get `401` errors when reaching the `/api/v4/internal/allowed` endpoint: ```shell @@ -1055,14 +1110,16 @@ Confirm the following are all true: [IP] - - [18/Jul/2019:00:30:14 +0000] "POST /api/v4/internal/allowed HTTP/1.1" 401 30 "" "Ruby" ``` -To fix this problem, confirm that your [`gitlab-secrets.json` file](#3-gitaly-server-configuration) -on the Gitaly node matches the one on all other nodes. If it doesn't match, -update the secrets file on the Gitaly node to match the others, then -[reconfigure the node](../restart_gitlab.md#omnibus-gitlab-reconfigure). +To fix this problem, confirm that your [`gitlab-secrets.json` file](#configure-gitaly-servers) +on the Gitaly server matches the one on Gitaly client. If it doesn't match, +update the secrets file on the Gitaly server to match the Gitaly client, then +[reconfigure](../restart_gitlab.md#omnibus-gitlab-reconfigure). ### Command line tools cannot connect to Gitaly -If you are having trouble connecting to a Gitaly node with command line (CLI) tools, and certain actions result in a `14: Connect Failed` error message, it means that gRPC cannot reach your Gitaly node. +If you are having trouble connecting to a Gitaly server with command line (CLI) tools, +and certain actions result in a `14: Connect Failed` error message, +it means that gRPC cannot reach your Gitaly server. Verify that you can reach Gitaly via TCP: @@ -1070,18 +1127,22 @@ Verify that you can reach Gitaly via TCP: sudo gitlab-rake gitlab:tcp_check[GITALY_SERVER_IP,GITALY_LISTEN_PORT] ``` -If the TCP connection fails, check your network settings and your firewall rules. If the TCP connection succeeds, your networking and firewall rules are correct. +If the TCP connection fails, check your network settings and your firewall rules. +If the TCP connection succeeds, your networking and firewall rules are correct. -If you use proxy servers in your command line environment, such as Bash, these can interfere with your gRPC traffic. +If you use proxy servers in your command line environment, such as Bash, these +can interfere with your gRPC traffic. -If you use Bash or a compatible command line environment, run the following commands to determine whether you have proxy servers configured: +If you use Bash or a compatible command line environment, run the following commands +to determine whether you have proxy servers configured: ```shell echo $http_proxy echo $https_proxy ``` -If either of these variables have a value, your Gitaly CLI connections may be getting routed through a proxy which cannot connect to Gitaly. +If either of these variables have a value, your Gitaly CLI connections may be +getting routed through a proxy which cannot connect to Gitaly. To remove the proxy setting, run the following commands (depending on which variables had values): @@ -1092,20 +1153,21 @@ unset https_proxy ### Gitaly not listening on new address after reconfiguring -When updating the `gitaly['listen_addr']` or `gitaly['prometheus_listen_addr']` values, Gitaly may continue to listen on the old address after a `sudo gitlab-ctl reconfigure`. +When updating the `gitaly['listen_addr']` or `gitaly['prometheus_listen_addr']` +values, Gitaly may continue to listen on the old address after a `sudo gitlab-ctl reconfigure`. -When this occurs, performing a `sudo gitlab-ctl restart` will resolve the issue. This will no longer be necessary after [this issue](https://gitlab.com/gitlab-org/gitaly/issues/2521) is resolved. +When this occurs, performing a `sudo gitlab-ctl restart` will resolve the issue. This will no longer be necessary after [this issue](https://gitlab.com/gitlab-org/gitaly/-/issues/2521) is resolved. -### Permission denied errors appearing in Gitaly logs when accessing repositories from a standalone Gitaly node +### Permission denied errors appearing in Gitaly logs when accessing repositories from a standalone Gitaly server If this error occurs even though file permissions are correct, it's likely that -the Gitaly node is experiencing +the Gitaly server is experiencing [clock drift](https://en.wikipedia.org/wiki/Clock_drift). -Please ensure that the GitLab and Gitaly nodes are synchronized and use an NTP time +Please ensure that the Gitaly clients and servers are synchronized and use an NTP time server to keep them synchronized if possible. ### Praefect -Praefect is an experimental daemon that allows for replication of the Git data. -It can be setup with omnibus, [as explained here](./praefect.md). +Praefect is a router and transaction manager for Gitaly, and a required +component for running a Gitaly Cluster. For more information see [Gitaly Cluster](praefect.md). diff --git a/doc/administration/gitaly/praefect.md b/doc/administration/gitaly/praefect.md index 0ea83f0e090..3d4e606205e 100644 --- a/doc/administration/gitaly/praefect.md +++ b/doc/administration/gitaly/praefect.md @@ -1,4 +1,7 @@ --- +stage: Create +group: Gitaly +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers type: reference --- @@ -89,7 +92,7 @@ to install GitLab](https://about.gitlab.com/install/). Provision a PostgreSQL server (PostgreSQL 11 or newer). Configuration through the Omnibus GitLab distribution is not yet supported. Follow this -[issue](https://gitlab.com/gitlab-org/gitaly/issues/2476) for updates. +[issue](https://gitlab.com/gitlab-org/gitaly/-/issues/2476) for updates. Prepare all your new nodes by [installing GitLab](https://about.gitlab.com/install/). @@ -319,37 +322,6 @@ application server, or a Gitaly node. } ``` -1. Enable the database replication queue: - - ```ruby - praefect['postgres_queue_enabled'] = true - ``` - - In the next release, database replication queue will be enabled by default. - See [issue #2615](https://gitlab.com/gitlab-org/gitaly/-/issues/2615). - -1. Enable automatic failover by editing `/etc/gitlab/gitlab.rb`: - - ```ruby - praefect['failover_enabled'] = true - praefect['failover_election_strategy'] = 'sql' - ``` - - When automatic failover is enabled, Praefect checks the health of internal - Gitaly nodes. If the primary has a certain amount of health checks fail, it - will promote one of the secondaries to be primary, and demote the primary to - be a secondary. - - NOTE: **Note:** Database leader election will be [enabled by default in the - future](https://gitlab.com/gitlab-org/gitaly/-/issues/2682). - - Caution, **automatic failover** favors availability over consistency and will - cause data loss if changes have not been replicated to the newly elected - primary. In the next release, leader election will [prefer to promote up to - date replicas](https://gitlab.com/gitlab-org/gitaly/-/issues/2642), and it - will be an option to favor consistency by marking [out-of-date repositories - read-only](https://gitlab.com/gitlab-org/gitaly/-/issues/2630). - 1. Save the changes to `/etc/gitlab/gitlab.rb` and [reconfigure Praefect](../restart_gitlab.md#omnibus-gitlab-reconfigure): @@ -409,7 +381,7 @@ Particular attention should be shown to: `gitaly-2`, and `gitaly-3` as Gitaly storage names. For more information on Gitaly server configuration, see our [Gitaly -documentation](index.md#3-gitaly-server-configuration). +documentation](index.md#configure-gitaly-servers). 1. SSH into the **Gitaly** node and login as root: @@ -424,16 +396,18 @@ documentation](index.md#3-gitaly-server-configuration). postgresql['enable'] = false redis['enable'] = false nginx['enable'] = false - prometheus['enable'] = false grafana['enable'] = false puma['enable'] = false sidekiq['enable'] = false gitlab_workhorse['enable'] = false prometheus_monitoring['enable'] = false - # Enable only the Praefect service + # Enable only the Gitaly service gitaly['enable'] = true + # Enable Prometheus if needed + prometheus['enable'] = true + # Prevent database connections during 'gitlab-ctl reconfigure' gitlab_rails['rake_cache_clear'] = false gitlab_rails['auto_migrate'] = false @@ -682,8 +656,8 @@ Particular attention should be shown to: Repository > Repository storage** to make the newly configured Praefect cluster the storage location for new Git repositories. - - Deselect the **default** storage location - - Select the **praefect** storage location + - The default option is unchecked. + - The Praefect option is checked.  @@ -744,7 +718,7 @@ Praefect regularly checks the health of each backend Gitaly node. This information can be used to automatically failover to a new primary node if the current primary node is found to be unhealthy. -- **PostgreSQL (recommended):** Enabled by setting +- **PostgreSQL (recommended):** Enabled by default, and equivalent to: `praefect['failover_election_strategy'] = sql`. This configuration option will allow multiple Praefect nodes to coordinate via the PostgreSQL database to elect a primary Gitaly node. This configuration @@ -755,24 +729,21 @@ current primary node is found to be unhealthy. reconfigured in `/etc/gitlab/gitlab.rb` on the Praefect node. Modify the `praefect['virtual_storages']` field by moving the `primary = true` to promote a different Gitaly node to primary. In the steps above, `gitaly-1` was set to - the primary. -- **Memory:** Enabled by setting `praefect['failover_enabled'] = true` in - `/etc/gitlab/gitlab.rb` on the Praefect node. If a sufficient number of health + the primary. Requires `praefect['failover_enabled'] = false` in the configuration. +- **Memory:** Enabled by setting `praefect['failover_election_strategy'] = 'local'` + in `/etc/gitlab/gitlab.rb` on the Praefect node. If a sufficient number of health checks fail for the current primary backend Gitaly node, and new primary will be elected. **Do not use with multiple Praefect nodes!** Using with multiple Praefect nodes is likely to result in a split brain. -NOTE: **Note:**: Praefect does not yet account for replication lag on -the secondaries during the election process, so data loss can occur -during a failover. Follow issue -[#2642](https://gitlab.com/gitlab-org/gitaly/-/issues/2642) for updates. - It is likely that we will implement support for Consul, and a cloud native strategy in the future. ## Identifying Impact of a Primary Node Failure -When a primary Gitaly node fails, there is a chance of data loss. Data loss can occur if there were outstanding replication jobs the secondaries did not manage to process before the failure. The Praefect `dataloss` sub-command helps identify these cases by counting the number of dead replication jobs for each repository within a given time frame. +When a primary Gitaly node fails, there is a chance of data loss. Data loss can occur if there were outstanding replication jobs the secondaries did not manage to process before the failure. The `dataloss` Praefect sub-command helps identify these cases by counting the number of dead replication jobs for each repository. This command must be executed on a Praefect node. + +A time frame to search can be specified with `-from` and `-to`: ```shell sudo /opt/gitlab/embedded/bin/praefect -config /var/opt/gitlab/praefect/config.toml dataloss -from <rfc3339-time> -to <rfc3339-time> @@ -784,9 +755,7 @@ If the time frame is not specified, dead replication jobs from the last six hour sudo /opt/gitlab/embedded/bin/praefect -config /var/opt/gitlab/praefect/config.toml dataloss Failed replication jobs between [2020-01-02 00:00:00 +0000 UTC, 2020-01-02 06:00:00 +0000 UTC): -example/repository-1: 1 jobs -example/repository-2: 4 jobs -example/repository-3: 2 jobs +@hashed/fa/53/fa539965395b8382145f8370b34eab249cf610d2d6f2943c95b9b9d08a63d4a3.git: 2 jobs ``` To specify a time frame in UTC, run: @@ -797,7 +766,8 @@ sudo /opt/gitlab/embedded/bin/praefect -config /var/opt/gitlab/praefect/config.t ### Checking repository checksums -To check a project's checksums across all nodes, the Praefect replicas Rake task can be used: +To check a project's repository checksums across on all Gitaly nodes, the +replicas Rake task can be run on the main GitLab node: ```shell sudo gitlab-rake "gitlab:praefect:replicas[project_id]" diff --git a/doc/administration/gitaly/reference.md b/doc/administration/gitaly/reference.md index e718d8953ca..52fd6fa6900 100644 --- a/doc/administration/gitaly/reference.md +++ b/doc/administration/gitaly/reference.md @@ -1,4 +1,7 @@ --- +stage: Create +group: Gitaly +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers type: reference --- @@ -12,7 +15,7 @@ The configuration file is passed as an argument to the `gitaly` executable. This is usually done by either Omnibus GitLab or your [init](https://en.wikipedia.org/wiki/Init) script. -An [example config file](https://gitlab.com/gitlab-org/gitaly/blob/master/config.toml.example) +An [example configuration file](https://gitlab.com/gitlab-org/gitaly/blob/master/config.toml.example) can be found in the Gitaly project. ## Format @@ -157,7 +160,7 @@ sum(rate(gitaly_catfile_cache_total{type="hit"}[5m])) / sum(rate(gitaly_catfile_ A Gitaly process uses one or more `gitaly-ruby` helper processes to execute RPC's implemented in Ruby instead of Go. The `[gitaly-ruby]` -section of the config file contains settings for these helper processes. +section of the configuration file contains settings for these helper processes. These processes are known to occasionally suffer from memory leaks. Gitaly restarts its `gitaly-ruby` helpers when their memory exceeds the @@ -190,7 +193,7 @@ For historical reasons the Git hooks that allow GitLab to validate and react to Git pushes. Because Gitaly "owns" Git pushes, GitLab Shell must therefore be installed alongside Gitaly. This will be [simplified in the -future](https://gitlab.com/gitlab-org/gitaly/issues/1226). +future](https://gitlab.com/gitlab-org/gitaly/-/issues/1226). | Name | Type | Required | Description | | ---- | ---- | -------- | ----------- | diff --git a/doc/administration/high_availability/consul.md b/doc/administration/high_availability/consul.md index 1f22c46a0ad..a87c1f1027f 100644 --- a/doc/administration/high_availability/consul.md +++ b/doc/administration/high_availability/consul.md @@ -7,7 +7,7 @@ type: reference As part of its High Availability stack, GitLab Premium includes a bundled version of [Consul](https://www.consul.io/) that can be managed through `/etc/gitlab/gitlab.rb`. Consul is a service networking solution. When it comes to [GitLab Architecture](../../development/architecture.md), Consul utilization is supported for configuring: 1. [Monitoring in Scaled and Highly Available environments](monitoring_node.md) -1. [PostgreSQL High Availability with Omnibus](database.md#high-availability-with-omnibus-gitlab-premium-only) +1. [PostgreSQL High Availability with Omnibus](../postgresql/replication_and_failover.md) A Consul cluster consists of multiple server agents, as well as client agents that run on other nodes which need to talk to the Consul cluster. @@ -27,7 +27,7 @@ When installing the GitLab package, do not supply `EXTERNAL_URL` value. On each Consul node perform the following: -1. Make sure you collect [`CONSUL_SERVER_NODES`](database.md#consul-information), which are the IP addresses or DNS records of the Consul server nodes, for the next step, before executing the next step. +1. Make sure you collect [`CONSUL_SERVER_NODES`](../postgresql/replication_and_failover.md#consul-information), which are the IP addresses or DNS records of the Consul server nodes, for the next step, before executing the next step. 1. Edit `/etc/gitlab/gitlab.rb` replacing values noted in the `# START user configuration` section: diff --git a/doc/administration/high_availability/database.md b/doc/administration/high_availability/database.md index 6f1873af993..75183436046 100644 --- a/doc/administration/high_availability/database.md +++ b/doc/administration/high_availability/database.md @@ -4,1100 +4,17 @@ type: reference # Configuring PostgreSQL for Scaling and High Availability -In this section, you'll be guided through configuring a PostgreSQL database -to be used with GitLab in a highly available environment. +In this section, you'll be guided through configuring a PostgreSQL database to +be used with GitLab in one of our [Scalable and Highly Available Setups](../reference_architectures/index.md). ## Provide your own PostgreSQL instance **(CORE ONLY)** -If you're hosting GitLab on a cloud provider, you can optionally use a -managed service for PostgreSQL. For example, AWS offers a managed Relational -Database Service (RDS) that runs PostgreSQL. +This content has been moved to a [new location](../postgresql/external.md). -If you use a cloud-managed service, or provide your own PostgreSQL: +## Standalone PostgreSQL using Omnibus GitLab **(CORE ONLY)** -1. Set up PostgreSQL according to the - [database requirements document](../../install/requirements.md#database). -1. Set up a `gitlab` username with a password of your choice. The `gitlab` user - needs privileges to create the `gitlabhq_production` database. -1. Configure the GitLab application servers with the appropriate details. - This step is covered in [Configuring GitLab for HA](gitlab.md). +This content has been moved to a [new location](../postgresql/standalone.md). -## PostgreSQL in a Scaled and Highly Available Environment +## PostgreSQL replication and failover with Omnibus GitLab **(PREMIUM ONLY)** -This section is relevant for [Scalable and Highly Available Setups](../reference_architectures/index.md). - -### Provide your own PostgreSQL instance **(CORE ONLY)** - -If you want to use your own deployed PostgreSQL instance(s), -see [Provide your own PostgreSQL instance](#provide-your-own-postgresql-instance-core-only) -for more details. However, you can use the Omnibus GitLab package to easily -deploy the bundled PostgreSQL. - -### Standalone PostgreSQL using Omnibus GitLab **(CORE ONLY)** - -1. SSH into the PostgreSQL server. -1. [Download/install](https://about.gitlab.com/install/) the Omnibus GitLab - package you want using **steps 1 and 2** from the GitLab downloads page. - - Do not complete any other steps on the download page. -1. Generate a password hash for PostgreSQL. This assumes you will use the default - username of `gitlab` (recommended). The command will request a password - and confirmation. Use the value that is output by this command in the next - step as the value of `POSTGRESQL_PASSWORD_HASH`. - - ```shell - sudo gitlab-ctl pg-password-md5 gitlab - ``` - -1. Edit `/etc/gitlab/gitlab.rb` and add the contents below, updating placeholder - values appropriately. - - - `POSTGRESQL_PASSWORD_HASH` - The value output from the previous step - - `APPLICATION_SERVER_IP_BLOCKS` - A space delimited list of IP subnets or IP - addresses of the GitLab application servers that will connect to the - database. Example: `%w(123.123.123.123/32 123.123.123.234/32)` - - ```ruby - # Disable all components except PostgreSQL - roles ['postgres_role'] - repmgr['enable'] = false - consul['enable'] = false - prometheus['enable'] = false - alertmanager['enable'] = false - pgbouncer_exporter['enable'] = false - redis_exporter['enable'] = false - gitlab_exporter['enable'] = false - - postgresql['listen_address'] = '0.0.0.0' - postgresql['port'] = 5432 - - # Replace POSTGRESQL_PASSWORD_HASH with a generated md5 value - postgresql['sql_user_password'] = 'POSTGRESQL_PASSWORD_HASH' - - # Replace XXX.XXX.XXX.XXX/YY with Network Address - # ???? - postgresql['trust_auth_cidr_addresses'] = %w(APPLICATION_SERVER_IP_BLOCKS) - - # Disable automatic database migrations - gitlab_rails['auto_migrate'] = false - ``` - - NOTE: **Note:** The role `postgres_role` was introduced with GitLab 10.3 - -1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. -1. Note the PostgreSQL node's IP address or hostname, port, and - plain text password. These will be necessary when configuring the GitLab - application servers later. -1. [Enable monitoring](#enable-monitoring) - -Advanced configuration options are supported and can be added if -needed. - -### High Availability with Omnibus GitLab **(PREMIUM ONLY)** - -> Important notes: -> -> - This document will focus only on configuration supported with [GitLab Premium](https://about.gitlab.com/pricing/), using the Omnibus GitLab package. -> - If you are a Community Edition or Starter user, consider using a cloud hosted solution. -> - This document will not cover installations from source. -> -> - If HA setup is not what you were looking for, see the [database configuration document](https://docs.gitlab.com/omnibus/settings/database.html) -> for the Omnibus GitLab packages. -> -> Please read this document fully before attempting to configure PostgreSQL HA -> for GitLab. -> -> This configuration is GA in EE 10.2. - -The recommended configuration for a PostgreSQL HA requires: - -- A minimum of three database nodes - - Each node will run the following services: - - `PostgreSQL` - The database itself - - `repmgrd` - A service to monitor, and handle failover in case of a failure - - `Consul` agent - Used for service discovery, to alert other nodes when failover occurs -- A minimum of three `Consul` server nodes -- A minimum of one `pgbouncer` service node, but it's recommended to have one per database node - - An internal load balancer (TCP) is required when there is more than one `pgbouncer` service node - -You also need to take into consideration the underlying network topology, -making sure you have redundant connectivity between all Database and GitLab instances, -otherwise the networks will become a single point of failure. - -#### Architecture - - - -Database nodes run two services with PostgreSQL: - -- Repmgrd. Monitors the cluster and handles failover when issues with the master occur. The failover consists of: - - Selecting a new master for the cluster. - - Promoting the new node to master. - - Instructing remaining servers to follow the new master node. - - On failure, the old master node is automatically evicted from the cluster, and should be rejoined manually once recovered. -- Consul. Monitors the status of each node in the database cluster and tracks its health in a service definition on the Consul cluster. - -Alongside each PgBouncer, there is a Consul agent that watches the status of the PostgreSQL service. If that status changes, Consul runs a script which updates the configuration and reloads PgBouncer - -##### Connection flow - -Each service in the package comes with a set of [default ports](https://docs.gitlab.com/omnibus/package-information/defaults.html#ports). You may need to make specific firewall rules for the connections listed below: - -- Application servers connect to either PgBouncer directly via its [default port](https://docs.gitlab.com/omnibus/package-information/defaults.html#pgbouncer) or via a configured Internal Load Balancer (TCP) that serves multiple PgBouncers. -- PgBouncer connects to the primary database servers [PostgreSQL default port](https://docs.gitlab.com/omnibus/package-information/defaults.html#postgresql) -- Repmgr connects to the database servers [PostgreSQL default port](https://docs.gitlab.com/omnibus/package-information/defaults.html#postgresql) -- PostgreSQL secondaries connect to the primary database servers [PostgreSQL default port](https://docs.gitlab.com/omnibus/package-information/defaults.html#postgresql) -- Consul servers and agents connect to each others [Consul default ports](https://docs.gitlab.com/omnibus/package-information/defaults.html#consul) - -#### Required information - -Before proceeding with configuration, you will need to collect all the necessary -information. - -##### Network information - -PostgreSQL does not listen on any network interface by default. It needs to know -which IP address to listen on in order to be accessible to other services. -Similarly, PostgreSQL access is controlled based on the network source. - -This is why you will need: - -- IP address of each nodes network interface. This can be set to `0.0.0.0` to - listen on all interfaces. It cannot be set to the loopback address `127.0.0.1`. -- Network Address. This can be in subnet (i.e. `192.168.0.0/255.255.255.0`) - or CIDR (i.e. `192.168.0.0/24`) form. - -##### User information - -Various services require different configuration to secure -the communication as well as information required for running the service. -Bellow you will find details on each service and the minimum required -information you need to provide. - -##### Consul information - -When using default setup, minimum configuration requires: - -- `CONSUL_USERNAME`. Defaults to `gitlab-consul` -- `CONSUL_DATABASE_PASSWORD`. Password for the database user. -- `CONSUL_PASSWORD_HASH`. This is a hash generated out of Consul username/password pair. - Can be generated with: - - ```shell - sudo gitlab-ctl pg-password-md5 CONSUL_USERNAME - ``` - -- `CONSUL_SERVER_NODES`. The IP addresses or DNS records of the Consul server nodes. - -Few notes on the service itself: - -- The service runs under a system account, by default `gitlab-consul`. - - If you are using a different username, you will have to specify it. We - will refer to it with `CONSUL_USERNAME`, -- There will be a database user created with read only access to the repmgr - database -- Passwords will be stored in the following locations: - - `/etc/gitlab/gitlab.rb`: hashed - - `/var/opt/gitlab/pgbouncer/pg_auth`: hashed - - `/var/opt/gitlab/consul/.pgpass`: plaintext - -##### PostgreSQL information - -When configuring PostgreSQL, we will set `max_wal_senders` to one more than -the number of database nodes in the cluster. -This is used to prevent replication from using up all of the -available database connections. - -In this document we are assuming 3 database nodes, which makes this configuration: - -```ruby -postgresql['max_wal_senders'] = 4 -``` - -As previously mentioned, you'll have to prepare the network subnets that will -be allowed to authenticate with the database. -You'll also need to supply the IP addresses or DNS records of Consul -server nodes. - -We will need the following password information for the application's database user: - -- `POSTGRESQL_USERNAME`. Defaults to `gitlab` -- `POSTGRESQL_USER_PASSWORD`. The password for the database user -- `POSTGRESQL_PASSWORD_HASH`. This is a hash generated out of the username/password pair. - Can be generated with: - - ```shell - sudo gitlab-ctl pg-password-md5 POSTGRESQL_USERNAME - ``` - -##### PgBouncer information - -When using default setup, minimum configuration requires: - -- `PGBOUNCER_USERNAME`. Defaults to `pgbouncer` -- `PGBOUNCER_PASSWORD`. This is a password for PgBouncer service. -- `PGBOUNCER_PASSWORD_HASH`. This is a hash generated out of PgBouncer username/password pair. - Can be generated with: - - ```shell - sudo gitlab-ctl pg-password-md5 PGBOUNCER_USERNAME - ``` - -- `PGBOUNCER_NODE`, is the IP address or a FQDN of the node running PgBouncer. - -Few notes on the service itself: - -- The service runs as the same system account as the database - - In the package, this is by default `gitlab-psql` -- If you use a non-default user account for PgBouncer service (by default `pgbouncer`), you will have to specify this username. We will refer to this requirement with `PGBOUNCER_USERNAME`. -- The service will have a regular database user account generated for it - - This defaults to `repmgr` -- Passwords will be stored in the following locations: - - `/etc/gitlab/gitlab.rb`: hashed, and in plain text - - `/var/opt/gitlab/pgbouncer/pg_auth`: hashed - -##### Repmgr information - -When using default setup, you will only have to prepare the network subnets that will -be allowed to authenticate with the service. - -Few notes on the service itself: - -- The service runs under the same system account as the database - - In the package, this is by default `gitlab-psql` -- The service will have a superuser database user account generated for it - - This defaults to `gitlab_repmgr` - -#### Installing Omnibus GitLab - -First, make sure to [download/install](https://about.gitlab.com/install/) -Omnibus GitLab **on each node**. - -Make sure you install the necessary dependencies from step 1, -add GitLab package repository from step 2. -When installing the GitLab package, do not supply `EXTERNAL_URL` value. - -#### Configuring the Database nodes - -1. Make sure to [configure the Consul nodes](consul.md). -1. Make sure you collect [`CONSUL_SERVER_NODES`](#consul-information), [`PGBOUNCER_PASSWORD_HASH`](#pgbouncer-information), [`POSTGRESQL_PASSWORD_HASH`](#postgresql-information), the [number of db nodes](#postgresql-information), and the [network address](#network-information) before executing the next step. - -1. On the master database node, edit `/etc/gitlab/gitlab.rb` replacing values noted in the `# START user configuration` section: - - ```ruby - # Disable all components except PostgreSQL and Repmgr and Consul - roles ['postgres_role'] - - # PostgreSQL configuration - postgresql['listen_address'] = '0.0.0.0' - postgresql['hot_standby'] = 'on' - postgresql['wal_level'] = 'replica' - postgresql['shared_preload_libraries'] = 'repmgr_funcs' - - # Disable automatic database migrations - gitlab_rails['auto_migrate'] = false - - # Configure the Consul agent - consul['services'] = %w(postgresql) - - # START user configuration - # Please set the real values as explained in Required Information section - # - # Replace PGBOUNCER_PASSWORD_HASH with a generated md5 value - postgresql['pgbouncer_user_password'] = 'PGBOUNCER_PASSWORD_HASH' - # Replace POSTGRESQL_PASSWORD_HASH with a generated md5 value - postgresql['sql_user_password'] = 'POSTGRESQL_PASSWORD_HASH' - # Replace X with value of number of db nodes + 1 - postgresql['max_wal_senders'] = X - postgresql['max_replication_slots'] = X - - # Replace XXX.XXX.XXX.XXX/YY with Network Address - postgresql['trust_auth_cidr_addresses'] = %w(XXX.XXX.XXX.XXX/YY) - repmgr['trust_auth_cidr_addresses'] = %w(127.0.0.1/32 XXX.XXX.XXX.XXX/YY) - - # Replace placeholders: - # - # Y.Y.Y.Y consul1.gitlab.example.com Z.Z.Z.Z - # with the addresses gathered for CONSUL_SERVER_NODES - consul['configuration'] = { - retry_join: %w(Y.Y.Y.Y consul1.gitlab.example.com Z.Z.Z.Z) - } - # - # END user configuration - ``` - - > `postgres_role` was introduced with GitLab 10.3 - -1. On secondary nodes, add all the configuration specified above for primary node - to `/etc/gitlab/gitlab.rb`. In addition, append the following configuration - to inform `gitlab-ctl` that they are standby nodes initially and it need not - attempt to register them as primary node - - ```ruby - # HA setting to specify if a node should attempt to be master on initialization - repmgr['master_on_initialization'] = false - ``` - -1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. -1. [Enable Monitoring](#enable-monitoring) - -> Please note: -> -> - If you want your database to listen on a specific interface, change the config: -> `postgresql['listen_address'] = '0.0.0.0'`. -> - If your PgBouncer service runs under a different user account, -> you also need to specify: `postgresql['pgbouncer_user'] = PGBOUNCER_USERNAME` in -> your configuration. - -##### Database nodes post-configuration - -###### Primary node - -Select one node as a primary node. - -1. Open a database prompt: - - ```shell - gitlab-psql -d gitlabhq_production - ``` - -1. Enable the `pg_trgm` extension: - - ```shell - CREATE EXTENSION pg_trgm; - ``` - -1. Exit the database prompt by typing `\q` and Enter. - -1. Verify the cluster is initialized with one node: - - ```shell - gitlab-ctl repmgr cluster show - ``` - - The output should be similar to the following: - - ```plaintext - Role | Name | Upstream | Connection String - ----------+----------|----------|---------------------------------------- - * master | HOSTNAME | | host=HOSTNAME user=gitlab_repmgr dbname=gitlab_repmgr - ``` - -1. Note down the hostname or IP address in the connection string: `host=HOSTNAME`. We will - refer to the hostname in the next section as `MASTER_NODE_NAME`. If the value - is not an IP address, it will need to be a resolvable name (via DNS or - `/etc/hosts`) - -###### Secondary nodes - -1. Set up the repmgr standby: - - ```shell - gitlab-ctl repmgr standby setup MASTER_NODE_NAME - ``` - - Do note that this will remove the existing data on the node. The command - has a wait time. - - The output should be similar to the following: - - ```console - # gitlab-ctl repmgr standby setup MASTER_NODE_NAME - Doing this will delete the entire contents of /var/opt/gitlab/postgresql/data - If this is not what you want, hit Ctrl-C now to exit - To skip waiting, rerun with the -w option - Sleeping for 30 seconds - Stopping the database - Removing the data - Cloning the data - Starting the database - Registering the node with the cluster - ok: run: repmgrd: (pid 19068) 0s - ``` - -1. Verify the node now appears in the cluster: - - ```shell - gitlab-ctl repmgr cluster show - ``` - - The output should be similar to the following: - - ```plaintext - Role | Name | Upstream | Connection String - ----------+---------|-----------|------------------------------------------------ - * master | MASTER | | host=MASTER_NODE_NAME user=gitlab_repmgr dbname=gitlab_repmgr - standby | STANDBY | MASTER | host=STANDBY_HOSTNAME user=gitlab_repmgr dbname=gitlab_repmgr - ``` - -Repeat the above steps on all secondary nodes. - -##### Database checkpoint - -Before moving on, make sure the databases are configured correctly. Run the -following command on the **primary** node to verify that replication is working -properly: - -```shell -gitlab-ctl repmgr cluster show -``` - -The output should be similar to: - -```plaintext -Role | Name | Upstream | Connection String -----------+--------------|--------------|-------------------------------------------------------------------- -* master | MASTER | | host=MASTER port=5432 user=gitlab_repmgr dbname=gitlab_repmgr - standby | STANDBY | MASTER | host=STANDBY port=5432 user=gitlab_repmgr dbname=gitlab_repmgr -``` - -If the 'Role' column for any node says "FAILED", check the -[Troubleshooting section](#troubleshooting) before proceeding. - -Also, check that the check master command works successfully on each node: - -```shell -su - gitlab-consul -gitlab-ctl repmgr-check-master || echo 'This node is a standby repmgr node' -``` - -This command relies on exit codes to tell Consul whether a particular node is a master -or secondary. The most important thing here is that this command does not produce errors. -If there are errors it's most likely due to incorrect `gitlab-consul` database user permissions. -Check the [Troubleshooting section](#troubleshooting) before proceeding. - -#### Configuring the PgBouncer node - -See our [documentation for PgBouncer](pgbouncer.md) for information on running PgBouncer as part of an HA setup. - -#### Configuring the Application nodes - -These will be the nodes running the `gitlab-rails` service. You may have other -attributes set, but the following need to be set. - -1. Edit `/etc/gitlab/gitlab.rb`: - - ```ruby - # Disable PostgreSQL on the application node - postgresql['enable'] = false - - gitlab_rails['db_host'] = 'PGBOUNCER_NODE' or 'INTERNAL_LOAD_BALANCER' - gitlab_rails['db_port'] = 6432 - gitlab_rails['db_password'] = 'POSTGRESQL_USER_PASSWORD' - gitlab_rails['auto_migrate'] = false - ``` - -1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. - -##### Application node post-configuration - -Ensure that all migrations ran: - -```shell -gitlab-rake gitlab:db:configure -``` - -> **Note**: If you encounter a `rake aborted!` error stating that PgBouncer is failing to connect to -PostgreSQL it may be that your PgBouncer node's IP address is missing from -PostgreSQL's `trust_auth_cidr_addresses` in `gitlab.rb` on your database nodes. See -[PgBouncer error `ERROR: pgbouncer cannot connect to server`](#pgbouncer-error-error-pgbouncer-cannot-connect-to-server) -in the Troubleshooting section before proceeding. - -##### Ensure GitLab is running - -At this point, your GitLab instance should be up and running. Verify you are -able to login, and create issues and merge requests. If you have troubles check -the [Troubleshooting section](#troubleshooting). - -#### Example configuration - -Here we'll show you some fully expanded example configurations. - -##### Example recommended setup - -This example uses 3 Consul servers, 3 PgBouncer servers (with associated internal load balancer), -3 PostgreSQL servers, and 1 application node. - -We start with all servers on the same 10.6.0.0/16 private network range, they -can connect to each freely other on those addresses. - -Here is a list and description of each machine and the assigned IP: - -- `10.6.0.11`: Consul 1 -- `10.6.0.12`: Consul 2 -- `10.6.0.13`: Consul 3 -- `10.6.0.20`: Internal Load Balancer -- `10.6.0.21`: PgBouncer 1 -- `10.6.0.22`: PgBouncer 2 -- `10.6.0.23`: PgBouncer 3 -- `10.6.0.31`: PostgreSQL master -- `10.6.0.32`: PostgreSQL secondary -- `10.6.0.33`: PostgreSQL secondary -- `10.6.0.41`: GitLab application - -All passwords are set to `toomanysecrets`, please do not use this password or derived hashes and the `external_url` for GitLab is `http://gitlab.example.com`. - -Please note that after the initial configuration, if a failover occurs, the PostgresSQL master will change to one of the available secondaries until it is failed back. - -##### Example recommended setup for Consul servers - -On each server edit `/etc/gitlab/gitlab.rb`: - -```ruby -# Disable all components except Consul -roles ['consul_role'] - -consul['configuration'] = { - server: true, - retry_join: %w(10.6.0.11 10.6.0.12 10.6.0.13) -} -consul['monitoring_service_discovery'] = true -``` - -[Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. - -##### Example recommended setup for PgBouncer servers - -On each server edit `/etc/gitlab/gitlab.rb`: - -```ruby -# Disable all components except Pgbouncer and Consul agent -roles ['pgbouncer_role'] - -# Configure PgBouncer -pgbouncer['admin_users'] = %w(pgbouncer gitlab-consul) - -pgbouncer['users'] = { - 'gitlab-consul': { - password: '5e0e3263571e3704ad655076301d6ebe' - }, - 'pgbouncer': { - password: '771a8625958a529132abe6f1a4acb19c' - } -} - -consul['watchers'] = %w(postgresql) -consul['enable'] = true -consul['configuration'] = { - retry_join: %w(10.6.0.11 10.6.0.12 10.6.0.13) -} -consul['monitoring_service_discovery'] = true -``` - -[Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. - -##### Internal load balancer setup - -An internal load balancer (TCP) is then required to be setup to serve each PgBouncer node (in this example on the IP of `10.6.0.20`). An example of how to do this can be found in the [PgBouncer Configure Internal Load Balancer](pgbouncer.md#configure-the-internal-load-balancer) section. - -##### Example recommended setup for PostgreSQL servers - -###### Primary node - -On primary node edit `/etc/gitlab/gitlab.rb`: - -```ruby -# Disable all components except PostgreSQL and Repmgr and Consul -roles ['postgres_role'] - -# PostgreSQL configuration -postgresql['listen_address'] = '0.0.0.0' -postgresql['hot_standby'] = 'on' -postgresql['wal_level'] = 'replica' -postgresql['shared_preload_libraries'] = 'repmgr_funcs' - -# Disable automatic database migrations -gitlab_rails['auto_migrate'] = false - -postgresql['pgbouncer_user_password'] = '771a8625958a529132abe6f1a4acb19c' -postgresql['sql_user_password'] = '450409b85a0223a214b5fb1484f34d0f' -postgresql['max_wal_senders'] = 4 - -postgresql['trust_auth_cidr_addresses'] = %w(10.6.0.0/16) -repmgr['trust_auth_cidr_addresses'] = %w(10.6.0.0/16) - -# Configure the Consul agent -consul['services'] = %w(postgresql) -consul['enable'] = true -consul['configuration'] = { - retry_join: %w(10.6.0.11 10.6.0.12 10.6.0.13) -} -consul['monitoring_service_discovery'] = true -``` - -[Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. - -###### Secondary nodes - -On secondary nodes, edit `/etc/gitlab/gitlab.rb` and add all the configuration -added to primary node, noted above. In addition, append the following -configuration: - -```ruby -# HA setting to specify if a node should attempt to be master on initialization -repmgr['master_on_initialization'] = false -``` - -[Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. - -##### Example recommended setup for application server - -On the server edit `/etc/gitlab/gitlab.rb`: - -```ruby -external_url 'http://gitlab.example.com' - -gitlab_rails['db_host'] = '10.6.0.20' # Internal Load Balancer for PgBouncer nodes -gitlab_rails['db_port'] = 6432 -gitlab_rails['db_password'] = 'toomanysecrets' -gitlab_rails['auto_migrate'] = false - -postgresql['enable'] = false -pgbouncer['enable'] = false -consul['enable'] = true - -# Configure Consul agent -consul['watchers'] = %w(postgresql) - -pgbouncer['users'] = { - 'gitlab-consul': { - password: '5e0e3263571e3704ad655076301d6ebe' - }, - 'pgbouncer': { - password: '771a8625958a529132abe6f1a4acb19c' - } -} - -consul['configuration'] = { - retry_join: %w(10.6.0.11 10.6.0.12 10.6.0.13) -} -``` - -[Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. - -##### Example recommended setup manual steps - -After deploying the configuration follow these steps: - -1. On `10.6.0.31`, our primary database - - Enable the `pg_trgm` extension - - ```shell - gitlab-psql -d gitlabhq_production - ``` - - ```shell - CREATE EXTENSION pg_trgm; - ``` - -1. On `10.6.0.32`, our first standby database - - Make this node a standby of the primary - - ```shell - gitlab-ctl repmgr standby setup 10.6.0.21 - ``` - -1. On `10.6.0.33`, our second standby database - - Make this node a standby of the primary - - ```shell - gitlab-ctl repmgr standby setup 10.6.0.21 - ``` - -1. On `10.6.0.41`, our application server - - Set `gitlab-consul` user's PgBouncer password to `toomanysecrets` - - ```shell - gitlab-ctl write-pgpass --host 127.0.0.1 --database pgbouncer --user pgbouncer --hostuser gitlab-consul - ``` - - Run database migrations - - ```shell - gitlab-rake gitlab:db:configure - ``` - -#### Example minimal setup - -This example uses 3 PostgreSQL servers, and 1 application node (with PgBouncer setup alongside). - -It differs from the [recommended setup](#example-recommended-setup) by moving the Consul servers into the same servers we use for PostgreSQL. -The trade-off is between reducing server counts, against the increased operational complexity of needing to deal with PostgreSQL [failover](#failover-procedure) and [restore](#restore-procedure) procedures in addition to [Consul outage recovery](consul.md#outage-recovery) on the same set of machines. - -In this example we start with all servers on the same 10.6.0.0/16 private network range, they can connect to each freely other on those addresses. - -Here is a list and description of each machine and the assigned IP: - -- `10.6.0.21`: PostgreSQL master -- `10.6.0.22`: PostgreSQL secondary -- `10.6.0.23`: PostgreSQL secondary -- `10.6.0.31`: GitLab application - -All passwords are set to `toomanysecrets`, please do not use this password or derived hashes. - -The `external_url` for GitLab is `http://gitlab.example.com` - -Please note that after the initial configuration, if a failover occurs, the PostgresSQL master will change to one of the available secondaries until it is failed back. - -##### Example minimal configuration for database servers - -##### Primary node - -On primary database node edit `/etc/gitlab/gitlab.rb`: - -```ruby -# Disable all components except PostgreSQL, Repmgr, and Consul -roles ['postgres_role'] - -# PostgreSQL configuration -postgresql['listen_address'] = '0.0.0.0' -postgresql['hot_standby'] = 'on' -postgresql['wal_level'] = 'replica' -postgresql['shared_preload_libraries'] = 'repmgr_funcs' - -# Disable automatic database migrations -gitlab_rails['auto_migrate'] = false - -# Configure the Consul agent -consul['services'] = %w(postgresql) - -postgresql['pgbouncer_user_password'] = '771a8625958a529132abe6f1a4acb19c' -postgresql['sql_user_password'] = '450409b85a0223a214b5fb1484f34d0f' -postgresql['max_wal_senders'] = 4 - -postgresql['trust_auth_cidr_addresses'] = %w(10.6.0.0/16) -repmgr['trust_auth_cidr_addresses'] = %w(10.6.0.0/16) - -consul['configuration'] = { - server: true, - retry_join: %w(10.6.0.21 10.6.0.22 10.6.0.23) -} -``` - -[Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. - -###### Secondary nodes - -On secondary nodes, edit `/etc/gitlab/gitlab.rb` and add all the information added -to primary node, noted above. In addition, append the following configuration - -```ruby -# HA setting to specify if a node should attempt to be master on initialization -repmgr['master_on_initialization'] = false -``` - -##### Example minimal configuration for application server - -On the server edit `/etc/gitlab/gitlab.rb`: - -```ruby -external_url 'http://gitlab.example.com' - -gitlab_rails['db_host'] = '127.0.0.1' -gitlab_rails['db_port'] = 6432 -gitlab_rails['db_password'] = 'toomanysecrets' -gitlab_rails['auto_migrate'] = false - -postgresql['enable'] = false -pgbouncer['enable'] = true -consul['enable'] = true - -# Configure PgBouncer -pgbouncer['admin_users'] = %w(pgbouncer gitlab-consul) - -# Configure Consul agent -consul['watchers'] = %w(postgresql) - -pgbouncer['users'] = { - 'gitlab-consul': { - password: '5e0e3263571e3704ad655076301d6ebe' - }, - 'pgbouncer': { - password: '771a8625958a529132abe6f1a4acb19c' - } -} - -consul['configuration'] = { - retry_join: %w(10.6.0.21 10.6.0.22 10.6.0.23) -} -``` - -[Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. - -##### Example minimal setup manual steps - -The manual steps for this configuration are the same as for the [example recommended setup](#example-recommended-setup-manual-steps). - -#### Failover procedure - -By default, if the master database fails, `repmgrd` should promote one of the -standby nodes to master automatically, and Consul will update PgBouncer with -the new master. - -If you need to failover manually, you have two options: - -**Shutdown the current master database** - -Run: - -```shell -gitlab-ctl stop postgresql -``` - -The automated failover process will see this and failover to one of the -standby nodes. - -**Or perform a manual failover** - -1. Ensure the old master node is not still active. -1. Login to the server that should become the new master and run: - - ```shell - gitlab-ctl repmgr standby promote - ``` - -1. If there are any other standby servers in the cluster, have them follow - the new master server: - - ```shell - gitlab-ctl repmgr standby follow NEW_MASTER - ``` - -#### Restore procedure - -If a node fails, it can be removed from the cluster, or added back as a standby -after it has been restored to service. - -##### Remove a standby from the cluster - - From any other node in the cluster, run: - - ```shell - gitlab-ctl repmgr standby unregister --node=X - ``` - - where X is the value of node in `repmgr.conf` on the old server. - - To find this, you can use: - - ```shell - awk -F = '$1 == "node" { print $2 }' /var/opt/gitlab/postgresql/repmgr.conf - ``` - - It will output something like: - - ```plaintext - 959789412 - ``` - - Then you will use this ID to unregister the node: - - ```shell - gitlab-ctl repmgr standby unregister --node=959789412 - ``` - -##### Add a node as a standby server - - From the stnadby node, run: - - ```shell - gitlab-ctl repmgr standby follow NEW_MASTER - gitlab-ctl restart repmgrd - ``` - - CAUTION: **Warning:** When the server is brought back online, and before - you switch it to a standby node, repmgr will report that there are two masters. - If there are any clients that are still attempting to write to the old master, - this will cause a split, and the old master will need to be resynced from - scratch by performing a `gitlab-ctl repmgr standby setup NEW_MASTER`. - -##### Add a failed master back into the cluster as a standby node - - Once `repmgrd` and PostgreSQL are runnning, the node will need to follow the new - as a standby node. - - ```shell - gitlab-ctl repmgr standby follow NEW_MASTER - ``` - - Once the node is following the new master as a standby, the node needs to be - [unregistered from the cluster on the new master node](#remove-a-standby-from-the-cluster). - - Once the old master node has been unregistered from the cluster, it will need - to be setup as a new standby: - - ```shell - gitlab-ctl repmgr standby setup NEW_MASTER - ``` - - Failure to unregister and readd the old master node can lead to subsequent failovers - not working. - -#### Alternate configurations - -##### Database authorization - -By default, we give any host on the database network the permission to perform -repmgr operations using PostgreSQL's `trust` method. If you do not want this -level of trust, there are alternatives. - -You can trust only the specific nodes that will be database clusters, or you -can require md5 authentication. - -##### Trust specific addresses - -If you know the IP address, or FQDN of all database and PgBouncer nodes in the -cluster, you can trust only those nodes. - -In `/etc/gitlab/gitlab.rb` on all of the database nodes, set -`repmgr['trust_auth_cidr_addresses']` to an array of strings containing all of -the addresses. - -If setting to a node's FQDN, they must have a corresponding PTR record in DNS. -If setting to a node's IP address, specify it as `XXX.XXX.XXX.XXX/32`. - -For example: - -```ruby -repmgr['trust_auth_cidr_addresses'] = %w(192.168.1.44/32 db2.example.com) -``` - -##### MD5 Authentication - -If you are running on an untrusted network, repmgr can use md5 authentication -with a [`.pgpass` file](https://www.postgresql.org/docs/11/libpq-pgpass.html) -to authenticate. - -You can specify by IP address, FQDN, or by subnet, using the same format as in -the previous section: - -1. On the current master node, create a password for the `gitlab` and - `gitlab_repmgr` user: - - ```shell - gitlab-psql -d template1 - template1=# \password gitlab_repmgr - Enter password: **** - Confirm password: **** - template1=# \password gitlab - ``` - -1. On each database node: - - 1. Edit `/etc/gitlab/gitlab.rb`: - 1. Ensure `repmgr['trust_auth_cidr_addresses']` is **not** set - 1. Set `postgresql['md5_auth_cidr_addresses']` to the desired value - 1. Set `postgresql['sql_replication_user'] = 'gitlab_repmgr'` - 1. Reconfigure with `gitlab-ctl reconfigure` - 1. Restart PostgreSQL with `gitlab-ctl restart postgresql` - - 1. Create a `.pgpass` file. Enter the `gitlab_repmgr` password twice to - when asked: - - ```shell - gitlab-ctl write-pgpass --user gitlab_repmgr --hostuser gitlab-psql --database '*' - ``` - -1. On each PgBouncer node, edit `/etc/gitlab/gitlab.rb`: - 1. Ensure `gitlab_rails['db_password']` is set to the plaintext password for - the `gitlab` database user - 1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect - -## Enable Monitoring - -> [Introduced](https://gitlab.com/gitlab-org/omnibus-gitlab/issues/3786) in GitLab 12.0. - -If you enable Monitoring, it must be enabled on **all** database servers. - -1. Create/edit `/etc/gitlab/gitlab.rb` and add the following configuration: - - ```ruby - # Enable service discovery for Prometheus - consul['monitoring_service_discovery'] = true - - # Set the network addresses that the exporters will listen on - node_exporter['listen_address'] = '0.0.0.0:9100' - postgres_exporter['listen_address'] = '0.0.0.0:9187' - ``` - -1. Run `sudo gitlab-ctl reconfigure` to compile the configuration. - -## Troubleshooting - -### Consul and PostgreSQL changes not taking effect - -Due to the potential impacts, `gitlab-ctl reconfigure` only reloads Consul and PostgreSQL, it will not restart the services. However, not all changes can be activated by reloading. - -To restart either service, run `gitlab-ctl restart SERVICE` - -For PostgreSQL, it is usually safe to restart the master node by default. Automatic failover defaults to a 1 minute timeout. Provided the database returns before then, nothing else needs to be done. To be safe, you can stop `repmgrd` on the standby nodes first with `gitlab-ctl stop repmgrd`, then start afterwards with `gitlab-ctl start repmgrd`. - -On the Consul server nodes, it is important to restart the Consul service in a controlled fashion. Read our [Consul documentation](consul.md#restarting-the-server-cluster) for instructions on how to restart the service. - -### `gitlab-ctl repmgr-check-master` command produces errors - -If this command displays errors about database permissions it is likely that something failed during -install, resulting in the `gitlab-consul` database user getting incorrect permissions. Follow these -steps to fix the problem: - -1. On the master database node, connect to the database prompt - `gitlab-psql -d template1` -1. Delete the `gitlab-consul` user - `DROP USER "gitlab-consul";` -1. Exit the database prompt - `\q` -1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) and the user will be re-added with the proper permissions. -1. Change to the `gitlab-consul` user - `su - gitlab-consul` -1. Try the check command again - `gitlab-ctl repmgr-check-master`. - -Now there should not be errors. If errors still occur then there is another problem. - -### PgBouncer error `ERROR: pgbouncer cannot connect to server` - -You may get this error when running `gitlab-rake gitlab:db:configure` or you -may see the error in the PgBouncer log file. - -```plaintext -PG::ConnectionBad: ERROR: pgbouncer cannot connect to server -``` - -The problem may be that your PgBouncer node's IP address is not included in the -`trust_auth_cidr_addresses` setting in `/etc/gitlab/gitlab.rb` on the database nodes. - -You can confirm that this is the issue by checking the PostgreSQL log on the master -database node. If you see the following error then `trust_auth_cidr_addresses` -is the problem. - -```plaintext -2018-03-29_13:59:12.11776 FATAL: no pg_hba.conf entry for host "123.123.123.123", user "pgbouncer", database "gitlabhq_production", SSL off -``` - -To fix the problem, add the IP address to `/etc/gitlab/gitlab.rb`. - -```ruby -postgresql['trust_auth_cidr_addresses'] = %w(123.123.123.123/32 <other_cidrs>) -``` - -[Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. - -### Issues with other components - -If you're running into an issue with a component not outlined here, be sure to check the troubleshooting section of their specific documentation page. - -- [Consul](consul.md#troubleshooting) -- [PostgreSQL](https://docs.gitlab.com/omnibus/settings/database.html#troubleshooting) -- [GitLab application](gitlab.md#troubleshooting) - -## Configure using Omnibus - -**Note**: We recommend that you follow the instructions here for a full [PostgreSQL cluster](#high-availability-with-omnibus-gitlab-premium-only). -If you are reading this section due to an old bookmark, you can find that old documentation [in the repository](https://gitlab.com/gitlab-org/gitlab/blob/v10.1.4/doc/administration/high_availability/database.md#configure-using-omnibus). - -Read more on high-availability configuration: - -1. [Configure Redis](redis.md) -1. [Configure NFS](nfs.md) -1. [Configure the GitLab application servers](gitlab.md) -1. [Configure the load balancers](load_balancer.md) -1. [Manage the bundled Consul cluster](consul.md) +This content has been moved to a [new location](../postgresql/replication_and_failover.md). diff --git a/doc/administration/high_availability/gitaly.md b/doc/administration/high_availability/gitaly.md index 2e6bcabeb06..230f5baf33a 100644 --- a/doc/administration/high_availability/gitaly.md +++ b/doc/administration/high_availability/gitaly.md @@ -1,4 +1,7 @@ --- +stage: Create +group: Gitaly +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers type: reference --- @@ -11,7 +14,7 @@ This document is relevant for [scalable and highly available setups](../referenc ## Running Gitaly on its own server -See [Running Gitaly on its own server](../gitaly/index.md#running-gitaly-on-its-own-server) +See [Run Gitaly on its own server](../gitaly/index.md#run-gitaly-on-its-own-server) in Gitaly documentation. Continue configuration of other components by going back to the @@ -19,9 +22,9 @@ Continue configuration of other components by going back to the ## Enable Monitoring -> [Introduced](https://gitlab.com/gitlab-org/omnibus-gitlab/issues/3786) in GitLab 12.0. +> [Introduced](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/3786) in GitLab 12.0. -1. Make sure to collect [`CONSUL_SERVER_NODES`](database.md#consul-information), which are the IP addresses or DNS records of the Consul server nodes, for the next step. Note they are presented as `Y.Y.Y.Y consul1.gitlab.example.com Z.Z.Z.Z` +1. Make sure to collect [`CONSUL_SERVER_NODES`](../postgresql/replication_and_failover.md#consul-information), which are the IP addresses or DNS records of the Consul server nodes, for the next step. Note they are presented as `Y.Y.Y.Y consul1.gitlab.example.com Z.Z.Z.Z` 1. Create/edit `/etc/gitlab/gitlab.rb` and add the following configuration: diff --git a/doc/administration/high_availability/gitlab.md b/doc/administration/high_availability/gitlab.md index cdafdbc4954..67a84f99bea 100644 --- a/doc/administration/high_availability/gitlab.md +++ b/doc/administration/high_availability/gitlab.md @@ -138,11 +138,11 @@ migrations performed. ## Enable Monitoring -> [Introduced](https://gitlab.com/gitlab-org/omnibus-gitlab/issues/3786) in GitLab 12.0. +> [Introduced](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/3786) in GitLab 12.0. If you enable Monitoring, it must be enabled on **all** GitLab servers. -1. Make sure to collect [`CONSUL_SERVER_NODES`](database.md#consul-information), which are the IP addresses or DNS records of the Consul server nodes, for the next step. Note they are presented as `Y.Y.Y.Y consul1.gitlab.example.com Z.Z.Z.Z` +1. Make sure to collect [`CONSUL_SERVER_NODES`](../postgresql/replication_and_failover.md#consul-information), which are the IP addresses or DNS records of the Consul server nodes, for the next step. Note they are presented as `Y.Y.Y.Y consul1.gitlab.example.com Z.Z.Z.Z` 1. Create/edit `/etc/gitlab/gitlab.rb` and add the following configuration: @@ -178,7 +178,7 @@ If you enable Monitoring, it must be enabled on **all** GitLab servers. running `sudo gitlab-ctl reconfigure`, it can take an extended period of time for Unicorn to complete reloading after receiving a `HUP`. For more information, see the - [issue](https://gitlab.com/gitlab-org/omnibus-gitlab/issues/4401). + [issue](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/4401). ## Troubleshooting @@ -203,7 +203,7 @@ for more details. Read more on high-availability configuration: -1. [Configure the database](database.md) +1. [Configure the database](../postgresql/replication_and_failover.md) 1. [Configure Redis](redis.md) 1. [Configure NFS](nfs.md) 1. [Configure the load balancers](load_balancer.md) diff --git a/doc/administration/high_availability/load_balancer.md b/doc/administration/high_availability/load_balancer.md index beeb57a0e32..75703327140 100644 --- a/doc/administration/high_availability/load_balancer.md +++ b/doc/administration/high_availability/load_balancer.md @@ -2,9 +2,9 @@ type: reference --- -# Load Balancer for GitLab HA +# Load Balancer for multi-node GitLab -In an active/active GitLab configuration, you will need a load balancer to route +In an multi-node GitLab configuration, you will need a load balancer to route traffic to the application servers. The specifics on which load balancer to use or the exact configuration is beyond the scope of GitLab documentation. We hope that if you're managing HA systems like GitLab you have a load balancer of @@ -14,7 +14,7 @@ you need to use with GitLab. ## SSL -How will you handle SSL in your HA environment? There are several different +How will you handle SSL in your multi-node environment? There are several different options: - Each application node terminates SSL @@ -109,11 +109,15 @@ Configure DNS for an alternate SSH hostname such as `altssh.gitlab.example.com`. | ------- | ------------ | -------- | | 443 | 22 | TCP | +## Readiness check + +It is strongly recommend that multi-node deployments configure load balancers to utilize the [readiness check](../../user/admin_area/monitoring/health_check.md#readiness) to ensure a node is ready to accept traffic, before routing traffic to it. This is especially important when utilizing Puma, as there is a brief period during a restart where Puma will not accept requests. + --- Read more on high-availability configuration: -1. [Configure the database](database.md) +1. [Configure the database](../postgresql/replication_and_failover.md) 1. [Configure Redis](redis.md) 1. [Configure NFS](nfs.md) 1. [Configure the GitLab application servers](gitlab.md) diff --git a/doc/administration/high_availability/monitoring_node.md b/doc/administration/high_availability/monitoring_node.md index 409a4dfecb9..653a0b32ad7 100644 --- a/doc/administration/high_availability/monitoring_node.md +++ b/doc/administration/high_availability/monitoring_node.md @@ -4,7 +4,7 @@ type: reference # Configuring a Monitoring node for Scaling and High Availability -> [Introduced](https://gitlab.com/gitlab-org/omnibus-gitlab/issues/3786) in GitLab 12.0. +> [Introduced](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/3786) in GitLab 12.0. You can configure a Prometheus node to monitor GitLab. @@ -22,7 +22,7 @@ Omnibus: package you want using **steps 1 and 2** from the GitLab downloads page. - Do not complete any other steps on the download page. -1. Make sure to collect [`CONSUL_SERVER_NODES`](database.md#consul-information), which are the IP addresses or DNS records of the Consul server nodes, for the next step. Note they are presented as `Y.Y.Y.Y consul1.gitlab.example.com Z.Z.Z.Z` +1. Make sure to collect [`CONSUL_SERVER_NODES`](../postgresql/replication_and_failover.md#consul-information), which are the IP addresses or DNS records of the Consul server nodes, for the next step. Note they are presented as `Y.Y.Y.Y consul1.gitlab.example.com Z.Z.Z.Z` 1. Edit `/etc/gitlab/gitlab.rb` and add the contents: diff --git a/doc/administration/high_availability/nfs.md b/doc/administration/high_availability/nfs.md index d2b8cf65b35..6511f9bd85d 100644 --- a/doc/administration/high_availability/nfs.md +++ b/doc/administration/high_availability/nfs.md @@ -59,7 +59,7 @@ management between systems: We recommend that all NFS users disable the NFS server delegation feature. This is to avoid a [Linux kernel bug](https://bugzilla.redhat.com/show_bug.cgi?id=1552203) which causes NFS clients to slow precipitously due to -[excessive network traffic from numerous `TEST_STATEID` NFS messages](https://gitlab.com/gitlab-org/gitlab-foss/issues/52017). +[excessive network traffic from numerous `TEST_STATEID` NFS messages](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/52017). To disable NFS server delegation, do the following: @@ -146,7 +146,7 @@ Additionally, this configuration is specifically warned against in the >to the NFS server can cause data corruption problems. For supported database architecture, please see our documentation on -[Configuring a Database for GitLab HA](database.md). +[Configuring a Database for GitLab HA](../postgresql/replication_and_failover.md). ## NFS Client mount options @@ -164,7 +164,7 @@ Note there are several options that you should consider using: | Setting | Description | | ------- | ----------- | -| `vers=4.1` |NFS v4.1 should be used instead of v4.0 because there is a Linux [NFS client bug in v4.0](https://gitlab.com/gitlab-org/gitaly/issues/1339) that can cause significant problems due to stale data. +| `vers=4.1` |NFS v4.1 should be used instead of v4.0 because there is a Linux [NFS client bug in v4.0](https://gitlab.com/gitlab-org/gitaly/-/issues/1339) that can cause significant problems due to stale data. | `nofail` | Don't halt boot process waiting for this mount to become available | `lookupcache=positive` | Tells the NFS client to honor `positive` cache results but invalidates any `negative` cache results. Negative cache results cause problems with Git. Specifically, a `git push` can fail to register uniformly across all NFS clients. The negative cache causes the clients to 'remember' that the files did not exist previously. | `hard` | Instead of `soft`. [Further details](#soft-mount-option). @@ -194,7 +194,7 @@ use the `hard` option, because (from the man page): Other vendors make similar recommendations, including [SAP](http://wiki.scn.sap.com/wiki/x/PARnFQ) and NetApp's -[knowledge base](https://kb.netapp.com/app/answers/answer_view/a_id/1004893/~/hard-mount-vs-soft-mount-), +[knowledge base](https://kb.netapp.com/Advice_and_Troubleshooting/Data_Storage_Software/ONTAP_OS/What_are_the_differences_between_hard_mount_and_soft_mount), they highlight that if the NFS client driver caches data, `soft` means there is no certainty if writes by GitLab are actually on disk. @@ -284,7 +284,7 @@ are empty before attempting a restore. Read more about the Read more on high-availability configuration: -1. [Configure the database](database.md) +1. [Configure the database](../postgresql/replication_and_failover.md) 1. [Configure Redis](redis.md) 1. [Configure the GitLab application servers](gitlab.md) 1. [Configure the load balancers](load_balancer.md) diff --git a/doc/administration/high_availability/pgbouncer.md b/doc/administration/high_availability/pgbouncer.md index 4c672f49e26..15e4da5b1f7 100644 --- a/doc/administration/high_availability/pgbouncer.md +++ b/doc/administration/high_availability/pgbouncer.md @@ -12,124 +12,7 @@ In a HA setup, it's recommended to run a PgBouncer node separately for each data ### Running PgBouncer as part of an HA GitLab installation -1. Make sure you collect [`CONSUL_SERVER_NODES`](database.md#consul-information), [`CONSUL_PASSWORD_HASH`](database.md#consul-information), and [`PGBOUNCER_PASSWORD_HASH`](database.md#pgbouncer-information) before executing the next step. - -1. One each node, edit the `/etc/gitlab/gitlab.rb` config file and replace values noted in the `# START user configuration` section as below: - - ```ruby - # Disable all components except PgBouncer and Consul agent - roles ['pgbouncer_role'] - - # Configure PgBouncer - pgbouncer['admin_users'] = %w(pgbouncer gitlab-consul) - - # Configure Consul agent - consul['watchers'] = %w(postgresql) - - # START user configuration - # Please set the real values as explained in Required Information section - # Replace CONSUL_PASSWORD_HASH with with a generated md5 value - # Replace PGBOUNCER_PASSWORD_HASH with with a generated md5 value - pgbouncer['users'] = { - 'gitlab-consul': { - password: 'CONSUL_PASSWORD_HASH' - }, - 'pgbouncer': { - password: 'PGBOUNCER_PASSWORD_HASH' - } - } - # Replace placeholders: - # - # Y.Y.Y.Y consul1.gitlab.example.com Z.Z.Z.Z - # with the addresses gathered for CONSUL_SERVER_NODES - consul['configuration'] = { - retry_join: %w(Y.Y.Y.Y consul1.gitlab.example.com Z.Z.Z.Z) - } - # - # END user configuration - ``` - - NOTE: **Note:** - `pgbouncer_role` was introduced with GitLab 10.3. - -1. Run `gitlab-ctl reconfigure` - -1. Create a `.pgpass` file so Consul is able to - reload PgBouncer. Enter the `PGBOUNCER_PASSWORD` twice when asked: - - ```shell - gitlab-ctl write-pgpass --host 127.0.0.1 --database pgbouncer --user pgbouncer --hostuser gitlab-consul - ``` - -#### PgBouncer Checkpoint - -1. Ensure each node is talking to the current master: - - ```shell - gitlab-ctl pgb-console # You will be prompted for PGBOUNCER_PASSWORD - ``` - - If there is an error `psql: ERROR: Auth failed` after typing in the - password, ensure you previously generated the MD5 password hashes with the correct - format. The correct format is to concatenate the password and the username: - `PASSWORDUSERNAME`. For example, `Sup3rS3cr3tpgbouncer` would be the text - needed to generate an MD5 password hash for the `pgbouncer` user. - -1. Once the console prompt is available, run the following queries: - - ```shell - show databases ; show clients ; - ``` - - The output should be similar to the following: - - ```plaintext - name | host | port | database | force_user | pool_size | reserve_pool | pool_mode | max_connections | current_connections - ---------------------+-------------+------+---------------------+------------+-----------+--------------+-----------+-----------------+--------------------- - gitlabhq_production | MASTER_HOST | 5432 | gitlabhq_production | | 20 | 0 | | 0 | 0 - pgbouncer | | 6432 | pgbouncer | pgbouncer | 2 | 0 | statement | 0 | 0 - (2 rows) - - type | user | database | state | addr | port | local_addr | local_port | connect_time | request_time | ptr | link | remote_pid | tls - ------+-----------+---------------------+---------+----------------+-------+------------+------------+---------------------+---------------------+-----------+------+------------+----- - C | pgbouncer | pgbouncer | active | 127.0.0.1 | 56846 | 127.0.0.1 | 6432 | 2017-08-21 18:09:59 | 2017-08-21 18:10:48 | 0x22b3880 | | 0 | - (2 rows) - ``` - -#### Configure the internal load balancer - -If you're running more than one PgBouncer node as recommended, then at this time you'll need to set up a TCP internal load balancer to serve each correctly. This can be done with any reputable TCP load balancer. - -As an example here's how you could do it with [HAProxy](https://www.haproxy.org/): - -```plaintext -global - log /dev/log local0 - log localhost local1 notice - log stdout format raw local0 - -defaults - log global - default-server inter 10s fall 3 rise 2 - balance leastconn - -frontend internal-pgbouncer-tcp-in - bind *:6432 - mode tcp - option tcplog - - default_backend pgbouncer - -backend pgbouncer - mode tcp - option tcp-check - - server pgbouncer1 <ip>:6432 check - server pgbouncer2 <ip>:6432 check - server pgbouncer3 <ip>:6432 check -``` - -Refer to your preferred Load Balancer's documentation for further guidance. +This content has been moved to a [new location](../postgresql/replication_and_failover.md#configuring-the-pgbouncer-node). ### Running PgBouncer as part of a non-HA GitLab installation @@ -179,7 +62,7 @@ Refer to your preferred Load Balancer's documentation for further guidance. ## Enable Monitoring -> [Introduced](https://gitlab.com/gitlab-org/omnibus-gitlab/issues/3786) in GitLab 12.0. +> [Introduced](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/3786) in GitLab 12.0. If you enable Monitoring, it must be enabled on **all** PgBouncer servers. diff --git a/doc/administration/high_availability/redis.md b/doc/administration/high_availability/redis.md index d52c80aec0d..bad50f7ca74 100644 --- a/doc/administration/high_availability/redis.md +++ b/doc/administration/high_availability/redis.md @@ -746,11 +746,11 @@ gitlab_rails['redis_sentinels'] = [ ## Enable Monitoring -> [Introduced](https://gitlab.com/gitlab-org/omnibus-gitlab/issues/3786) in GitLab 12.0. +> [Introduced](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/3786) in GitLab 12.0. If you enable Monitoring, it must be enabled on **all** Redis servers. -1. Make sure to collect [`CONSUL_SERVER_NODES`](database.md#consul-information), which are the IP addresses or DNS records of the Consul server nodes, for the next step. Note they are presented as `Y.Y.Y.Y consul1.gitlab.example.com Z.Z.Z.Z` +1. Make sure to collect [`CONSUL_SERVER_NODES`](../postgresql/replication_and_failover.md#consul-information), which are the IP addresses or DNS records of the Consul server nodes, for the next step. Note they are presented as `Y.Y.Y.Y consul1.gitlab.example.com Z.Z.Z.Z` 1. Create/edit `/etc/gitlab/gitlab.rb` and add the following configuration: @@ -1002,7 +1002,7 @@ Changes to Redis HA over time. Read more on High Availability: 1. [High Availability Overview](README.md) -1. [Configure the database](database.md) +1. [Configure the database](../postgresql/replication_and_failover.md) 1. [Configure NFS](nfs.md) 1. [Configure the GitLab application servers](gitlab.md) 1. [Configure the load balancers](load_balancer.md) diff --git a/doc/administration/housekeeping.md b/doc/administration/housekeeping.md index 4a2e2b9aac9..4110f8b7646 100644 --- a/doc/administration/housekeeping.md +++ b/doc/administration/housekeeping.md @@ -1,6 +1,6 @@ # Housekeeping -> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/issues/3041) in GitLab 8.4. +> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/3041) in GitLab 8.4. ## Automatic housekeeping diff --git a/doc/administration/incoming_email.md b/doc/administration/incoming_email.md index fcd69464b13..e078a7c1098 100644 --- a/doc/administration/incoming_email.md +++ b/doc/administration/incoming_email.md @@ -1,3 +1,9 @@ +--- +stage: Plan +group: Project Management +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + # Incoming email GitLab has several features based on receiving incoming emails: @@ -92,13 +98,13 @@ authenticate solely based on access to an email domain such as `*.hooli.com.` Alternatively, use a dedicated domain for GitLab email communications such as `hooli-gitlab.com`. -See GitLab issue [#30366](https://gitlab.com/gitlab-org/gitlab-foss/issues/30366) +See GitLab issue [#30366](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/30366) for a real-world example of this exploit. ### Omnibus package installations 1. Find the `incoming_email` section in `/etc/gitlab/gitlab.rb`, enable the feature - and fill in the details for your specific IMAP server and email account (see [examples](#config-examples) below). + and fill in the details for your specific IMAP server and email account (see [examples](#configuration-examples) below). 1. Reconfigure GitLab for the changes to take effect: @@ -124,7 +130,7 @@ Reply by email should now be working. ``` 1. Find the `incoming_email` section in `config/gitlab.yml`, enable the feature - and fill in the details for your specific IMAP server and email account (see [examples](#config-examples) below). + and fill in the details for your specific IMAP server and email account (see [examples](#configuration-examples) below). 1. Enable `mail_room` in the init script at `/etc/default/gitlab`: @@ -147,7 +153,7 @@ Reply by email should now be working. Reply by email should now be working. -### Config examples +### Configuration examples #### Postfix diff --git a/doc/administration/index.md b/doc/administration/index.md index fb827ae8573..fa415e5f78d 100644 --- a/doc/administration/index.md +++ b/doc/administration/index.md @@ -109,7 +109,7 @@ Learn how to install, configure, update, and maintain your GitLab instance. - [Sign-up restrictions](../user/admin_area/settings/sign_up_restrictions.md): block email addresses of specific domains, or whitelist only specific domains. - [Access restrictions](../user/admin_area/settings/visibility_and_access_controls.md#enabled-git-access-protocols): Define which Git access protocols can be used to talk to GitLab (SSH, HTTP, HTTPS). - [Authentication and Authorization](auth/README.md): Configure external authentication with LDAP, SAML, CAS, and additional providers. - - [Sync LDAP](auth/ldap-ee.md) **(STARTER ONLY)** + - [Sync LDAP](auth/ldap/index.md) **(STARTER ONLY)** - [Kerberos authentication](../integration/kerberos.md) **(STARTER ONLY)** - See also other [authentication](../topics/authentication/index.md#gitlab-administrators) topics (for example, enforcing 2FA). - [Email users](../tools/email.md): Email GitLab users from within GitLab. **(STARTER ONLY)** @@ -157,7 +157,7 @@ Learn how to install, configure, update, and maintain your GitLab instance. - [External Pipeline Validation](external_pipeline_validation.md): Enable, disable and configure external pipeline validation. - [Job artifacts](job_artifacts.md): Enable, disable, and configure job artifacts (a set of files and directories which are outputted by a job when it completes successfully). - [Job logs](job_logs.md): Information about the job logs. -- [Register Shared and specific Runners](../ci/runners/README.md#registering-a-shared-runner): Learn how to register and configure Shared and specific Runners to your own instance. +- [Register Runners](../ci/runners/README.md#types-of-runners): Learn how to register and configure Runners. - [Shared Runners pipelines quota](../user/admin_area/settings/continuous_integration.md#shared-runners-pipeline-minutes-quota-starter-only): Limit the usage of pipeline minutes for Shared Runners. **(STARTER ONLY)** - [Enable/disable Auto DevOps](../topics/autodevops/index.md#enablingdisabling-auto-devops): Enable or disable Auto DevOps for your instance. diff --git a/doc/administration/instance_limits.md b/doc/administration/instance_limits.md index d0e8f079cb2..3d2f7380494 100644 --- a/doc/administration/instance_limits.md +++ b/doc/administration/instance_limits.md @@ -10,7 +10,7 @@ performance, data, or could even exhaust the allocated resources for the applica ## Number of comments per issue, merge request or commit -> [Introduced](https://gitlab.com/gitlab-org/gitlab/issues/22388) in GitLab 12.4. +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/22388) in GitLab 12.4. There's a limit to the number of comments that can be submitted on an issue, merge request, or commit. When the limit is reached, system notes can still be @@ -21,7 +21,7 @@ will fail. ## Size of comments and descriptions of issues, merge requests, and epics -> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/issues/61974) in GitLab 12.2. +> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/61974) in GitLab 12.2. There is a limit to the size of comments and descriptions of issues, merge requests, and epics. Attempting to add a body of text larger than the limit will result in an error, and the @@ -43,7 +43,7 @@ When the number exceeds the limit the page displays an alert and links to a pagi ## Number of pipelines per Git push -> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/issues/51401) in GitLab 11.10. +> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/51401) in GitLab 11.10. The number of pipelines that can be created in a single push is 4. This is to prevent the accidental creation of pipelines when `git push --all` @@ -53,9 +53,9 @@ Read more in the [CI documentation](../ci/yaml/README.md#processing-git-pushes). ## Retention of activity history -> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/issues/21164) in GitLab 8.12. +> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/21164) in GitLab 8.12. -Activity history for projects and individuals' profiles was limited to one year until [GitLab 11.4](https://gitlab.com/gitlab-org/gitlab-foss/issues/52246) when it was extended to two years, and in [GitLab 12.4](https://gitlab.com/gitlab-org/gitlab/issues/33840) to three years. +Activity history for projects and individuals' profiles was limited to one year until [GitLab 11.4](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/52246) when it was extended to two years, and in [GitLab 12.4](https://gitlab.com/gitlab-org/gitlab/-/issues/33840) to three years. ## Number of embedded metrics @@ -87,7 +87,7 @@ NOTE: **Note:** Set the limit to `0` to disable it. ## Incoming emails from auto-responders -> [Introduced](https://gitlab.com/gitlab-org/gitlab/issues/30327) in GitLab 12.4. +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/30327) in GitLab 12.4. GitLab ignores all incoming emails sent from auto-responders by looking for the `X-Autoreply` header. Such emails don't create comments on issues or merge requests. @@ -126,7 +126,7 @@ NOTE: **Note:** Set the limit to `0` to disable it. ### Number of jobs in active pipelines -> [Introduced](https://gitlab.com/gitlab-org/gitlab/issues/32823) in GitLab 12.6. +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/32823) in GitLab 12.6. The total number of jobs in active pipelines can be limited per project. This limit is checked each time a new pipeline is created. An active pipeline is any pipeline in one of the following states: @@ -156,7 +156,7 @@ NOTE: **Note:** Set the limit to `0` to disable it. ### Number of CI/CD subscriptions to a project -> [Introduced](https://gitlab.com/gitlab-org/gitlab/issues/9045) in GitLab 12.9. +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/9045) in GitLab 12.9. The total number of subscriptions can be limited per project. This limit is checked each time a new subscription is created. @@ -199,6 +199,24 @@ To set this limit on a self-managed installation, run the following in the Plan.default.limits.update!(ci_pipeline_schedules: 100) ``` +### Number of instance level variables + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/216097) in GitLab 13.1. + +The total number of instance level CI/CD variables is limited at the instance level. +This limit is checked each time a new instance level variable is created. If a new variable +would cause the total number of variables to exceed the limit, the new variable will not be created. + +On self-managed instances this limit is defined for the `default` plan. By default, +this limit is set to `25`. + +To update this limit to a new value on a self-managed installation, run the following in the +[GitLab Rails console](troubleshooting/debug.md#starting-a-rails-console-session): + +```ruby +Plan.default.limits.update!(ci_instance_level_variables: 30) +``` + ## Instance monitoring and metrics ### Incident Management inbound alert limits @@ -235,7 +253,7 @@ Kubernetes won't be shown. Reports that go over the 20 MB limit won't be loaded. Affected reports: -- [Merge Request security reports](../user/project/merge_requests/index.md#security-reports-ultimate) +- [Merge Request security reports](../user/project/merge_requests/testing_and_reports_in_merge_requests.md#security-reports-ultimate) - [CI/CD parameter `artifacts:expose_as`](../ci/yaml/README.md#artifactsexpose_as) - [JUnit test reports](../ci/junit_test_reports.md) @@ -243,7 +261,7 @@ Reports that go over the 20 MB limit won't be loaded. Affected reports: ### Maximum field length -> [Introduced](https://gitlab.com/gitlab-org/gitlab/issues/201826) in GitLab 12.8. +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/201826) in GitLab 12.8. You can set a limit on the content of text fields indexed for Global Search. Setting a maximum helps to reduce the load of the indexing processes. If any @@ -270,7 +288,7 @@ See the [documentation on Snippets settings](snippets/index.md). ### Webhooks and Project Services -> [Introduced](https://gitlab.com/gitlab-org/gitlab/issues/31009) in GitLab 12.4. +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/31009) in GitLab 12.4. Total number of changes (branches or tags) in a single push. If changes are more than the specified limit, hooks won't be executed. @@ -282,7 +300,7 @@ More information can be found in these docs: ### Activities -> [Introduced](https://gitlab.com/gitlab-org/gitlab/issues/31007) in GitLab 12.4. +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/31007) in GitLab 12.4. Total number of changes (branches or tags) in a single push to determine whether individual push events or bulk push event will be created. diff --git a/doc/administration/integration/plantuml.md b/doc/administration/integration/plantuml.md index 3372d05bd6b..cd25fbf351b 100644 --- a/doc/administration/integration/plantuml.md +++ b/doc/administration/integration/plantuml.md @@ -105,6 +105,21 @@ To activate the changes, run the following command: sudo gitlab-ctl reconfigure ``` +### Security + +PlantUML has features that allows fetching network resources. + +```plaintext +@startuml +start + ' ... + !include http://localhost/ +stop; +@enduml +``` + +**If you self-host the PlantUML server, network controls should be put in place to isolate it.** + ## GitLab You need to enable PlantUML integration from Settings under Admin Area. To do diff --git a/doc/administration/job_artifacts.md b/doc/administration/job_artifacts.md index fbfad46ef65..0dbda67d39b 100644 --- a/doc/administration/job_artifacts.md +++ b/doc/administration/job_artifacts.md @@ -1,3 +1,10 @@ +--- +stage: Verify +group: Continuous Integration +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +type: reference, howto +--- + # Jobs artifacts administration > - Introduced in GitLab 8.2 and GitLab Runner 0.7.0. @@ -40,6 +47,9 @@ GitLab Runner can upload an archive containing the job artifacts to GitLab. By d this is done when the job succeeds, but can also be done on failure, or always, via the [`artifacts:when`](../ci/yaml/README.md#artifactswhen) parameter. +Most artifacts are compressed by GitLab Runner before being sent to the coordinator. The exception to this is +[reports artifacts](../ci/pipelines/job_artifacts.md#artifactsreports), which are compressed after uploading. + ### Using local storage To change the location where the artifacts are stored locally, follow the steps @@ -119,7 +129,7 @@ The connection settings match those provided by [Fog](https://github.com/fog), a | `aws_signature_version` | AWS signature version to use. 2 or 4 are valid options. Digital Ocean Spaces and other providers may need 2. | 4 | | `enable_signature_v4_streaming` | Set to true to enable HTTP chunked transfers with [AWS v4 signatures](https://docs.aws.amazon.com/AmazonS3/latest/API/sigv4-streaming.html). Oracle Cloud S3 needs this to be false | true | | `region` | AWS region | us-east-1 | -| `host` | S3 compatible host for when not using AWS, e.g. `localhost` or `storage.example.com` | s3.amazonaws.com | +| `host` | S3 compatible host for when not using AWS, for example `localhost` or `storage.example.com` | s3.amazonaws.com | | `endpoint` | Can be used when configuring an S3 compatible service such as [MinIO](https://min.io), by entering a URL such as `http://127.0.0.1:9000` | (optional) | | `path_style` | Set to true to use `host/bucket_name/object` style paths instead of `bucket_name.host/object`. Leave as false for AWS S3 | false | | `use_iam_profile` | Set to true to use IAM profile instead of access keys | false @@ -129,7 +139,7 @@ The connection settings match those provided by [Fog](https://github.com/fog), a _The artifacts are stored by default in `/var/opt/gitlab/gitlab-rails/shared/artifacts`._ -1. Edit `/etc/gitlab/gitlab.rb` and add the following lines by replacing with +1. Edit `/etc/gitlab/gitlab.rb` and add the following lines, substituting the values you want: ```ruby @@ -144,7 +154,7 @@ _The artifacts are stored by default in } ``` - NOTE: For GitLab 9.4+, if you are using AWS IAM profiles, be sure to omit the + NOTE: For GitLab 9.4+, if you're using AWS IAM profiles, be sure to omit the AWS access key and secret access key/value pairs. For example: ```ruby @@ -164,7 +174,7 @@ _The artifacts are stored by default in CAUTION: **CAUTION:** JUnit test report artifact (`junit.xml.gz`) migration -[is not supported](https://gitlab.com/gitlab-org/gitlab/issues/27698) +[is not supported](https://gitlab.com/gitlab-org/gitlab/-/issues/27698) by the `gitlab:artifacts:migrate` script. **In installations from source:** @@ -197,7 +207,7 @@ _The artifacts are stored by default in CAUTION: **CAUTION:** JUnit test report artifact (`junit.xml.gz`) migration -[is not supported](https://gitlab.com/gitlab-org/gitlab/issues/27698) +[is not supported](https://gitlab.com/gitlab-org/gitlab/-/issues/27698) by the `gitlab:artifacts:migrate` script. ### OpenStack compatible connection settings @@ -209,8 +219,8 @@ The connection settings match those provided by [Fog](https://github.com/fog), a | `provider` | Always `OpenStack` for compatible hosts | OpenStack | | `openstack_username` | OpenStack username | | | `openstack_api_key` | OpenStack API key | | -| `openstack_temp_url_key` | OpenStack key for generating temporary urls | | -| `openstack_auth_url` | OpenStack authentication endpont | | +| `openstack_temp_url_key` | OpenStack key for generating temporary URLs | | +| `openstack_auth_url` | OpenStack authentication endpoint | | | `openstack_region` | OpenStack region | | | `openstack_tenant_id` | OpenStack tenant ID | @@ -219,7 +229,7 @@ The connection settings match those provided by [Fog](https://github.com/fog), a _The uploads are stored by default in `/var/opt/gitlab/gitlab-rails/shared/artifacts`._ -1. Edit `/etc/gitlab/gitlab.rb` and add the following lines by replacing with +1. Edit `/etc/gitlab/gitlab.rb` and add the following lines, substituting the values you want: ```ruby @@ -296,17 +306,20 @@ In order to migrate back to local storage: ## Expiring artifacts -If an expiry date is used for the artifacts, they are marked for deletion -right after that date passes. Artifacts are cleaned up by the -`expire_build_artifacts_worker` cron job which is run by Sidekiq every hour at -50 minutes (`50 * * * *`). +If [`artifacts:expire_in`](../ci/yaml/README.md#artifactsexpire_in) is used to set +an expiry for the artifacts, they are marked for deletion right after that date passes. +Otherwise, they will expire per the [default artifacts expiration setting](../user/admin_area/settings/continuous_integration.md). + +Artifacts are cleaned up by the `expire_build_artifacts_worker` cron job which Sidekiq +runs every hour at 50 minutes (`50 * * * *`). To change the default schedule on which the artifacts are expired, follow the steps below. **In Omnibus installations:** -1. Edit `/etc/gitlab/gitlab.rb` and comment out or add the following line +1. Edit `/etc/gitlab/gitlab.rb` and add the following line (or uncomment it if it already exists and is commented out), substituting + your schedule in cron syntax: ```ruby gitlab_rails['expire_build_artifacts_worker_cron'] = "50 * * * *" @@ -326,12 +339,15 @@ steps below. 1. Save the file and [restart GitLab](restart_gitlab.md#installations-from-source) for the changes to take effect. +If the `expire` directive is not set explicitly in your pipeline, artifacts will expire per the +default artifacts expiration setting, which you can find in the [CI/CD Admin settings](../user/admin_area/settings/continuous_integration.md). + ## Validation for dependencies > Introduced in GitLab 10.3. To disable [the dependencies validation](../ci/yaml/README.md#when-a-dependent-job-will-fail), -you can flip the feature flag from a Rails console. +you can enable the `ci_disable_validates_dependencies` feature flag from a Rails console. **In Omnibus installations:** @@ -341,10 +357,10 @@ you can flip the feature flag from a Rails console. sudo gitlab-rails console ``` -1. Flip the switch and disable it: +1. Enable the feature flag to disable the validation: ```ruby - Feature.enable('ci_disable_validates_dependencies') + Feature.enable(:ci_disable_validates_dependencies) ``` **In installations from source:** @@ -356,15 +372,15 @@ you can flip the feature flag from a Rails console. sudo -u git -H bundle exec rails console -e production ``` -1. Flip the switch and disable it: +1. Enable the feature flag to disable the validation: ```ruby - Feature.enable('ci_disable_validates_dependencies') + Feature.enable(:ci_disable_validates_dependencies) ``` ## Set the maximum file size of the artifacts -Provided the artifacts are enabled, you can change the maximum file size of the +If artifacts are enabled, you can change the maximum file size of the artifacts through the [Admin Area settings](../user/admin_area/settings/continuous_integration.md#maximum-artifacts-size-core-only). ## Storage statistics @@ -404,6 +420,8 @@ In these and other cases, you'll need to identify the projects most responsible for disk space usage, figure out what types of artifacts are using the most space, and in some cases, manually delete job artifacts to reclaim disk space. +One possible first step is to [clean up _orphaned_ artifact files](../raketasks/cleanup.md#remove-orphan-artifact-files). + #### List projects by total size of job artifacts stored List the top 20 projects, sorted by the total size of job artifacts stored, by @@ -488,7 +506,7 @@ highly recommend running them only under the guidance of a Support Engineer, or running them in a test environment with a backup of the instance ready to be restored, just in case. -If you need to manually remove ALL job artifacts associated with multiple jobs, +If you need to manually remove **all** job artifacts associated with multiple jobs, **including job logs**, this can be done from the Rails console (`sudo gitlab-rails console`): 1. Select jobs to be deleted: diff --git a/doc/administration/job_logs.md b/doc/administration/job_logs.md index 5c03ff1f4b2..d3484536a76 100644 --- a/doc/administration/job_logs.md +++ b/doc/administration/job_logs.md @@ -1,6 +1,13 @@ +--- +stage: Verify +group: Runner +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +type: reference +--- + # Job logs -> [Renamed from job traces to job logs](https://gitlab.com/gitlab-org/gitlab/issues/29121) in GitLab 12.5. +> [Renamed from job traces to job logs](https://gitlab.com/gitlab-org/gitlab/-/issues/29121) in GitLab 12.5. Job logs are sent by GitLab Runner while it's processing a job. You can see logs in job pages, pipelines, email notifications, etc. @@ -64,10 +71,19 @@ There isn't a way to automatically expire old job logs, but it's safe to remove them if they're taking up too much space. If you remove the logs manually, the job output in the UI will be empty. +For example, to delete all job logs older than 60 days, run the following from a shell in your GitLab instance: + +DANGER: **Warning:** +This command will permanently delete the log files and is irreversible. + +```shell +find /var/opt/gitlab/gitlab-rails/shared/artifacts -name "job.log" -mtime +60 -delete +``` + ## New incremental logging architecture > - [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/18169) in GitLab 10.4. -> - [Announced as generally available](https://gitlab.com/gitlab-org/gitlab-foss/issues/46097) in GitLab 11.0. +> - [Announced as generally available](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/46097) in GitLab 11.0. NOTE: **Note:** This feature is off by default. See below for how to [enable or disable](#enabling-incremental-logging) it. @@ -112,13 +128,13 @@ sudo -u git -H bin/rails console -e production **To check if incremental logging (trace) is enabled:** ```ruby -Feature.enabled?('ci_enable_live_trace') +Feature.enabled?(:ci_enable_live_trace) ``` **To enable incremental logging (trace):** ```ruby -Feature.enable('ci_enable_live_trace') +Feature.enable(:ci_enable_live_trace) ``` NOTE: **Note:** diff --git a/doc/administration/lfs/index.md b/doc/administration/lfs/index.md index e2b982448ef..dd0e25b05f1 100644 --- a/doc/administration/lfs/index.md +++ b/doc/administration/lfs/index.md @@ -110,7 +110,7 @@ Here is a configuration example with GCS. | `provider` | The provider name | `Google` | | `google_project` | GCP project name | `gcp-project-12345` | | `google_client_email` | The email address of the service account | `foo@gcp-project-12345.iam.gserviceaccount.com` | -| `google_json_key_location` | The json key path | `/path/to/gcp-project-12345-abcde.json` | +| `google_json_key_location` | The JSON key path | `/path/to/gcp-project-12345-abcde.json` | NOTE: **Note:** The service account must have permission to access the bucket. diff --git a/doc/administration/libravatar.md b/doc/administration/libravatar.md index e1c38b3409f..beecd9e4783 100644 --- a/doc/administration/libravatar.md +++ b/doc/administration/libravatar.md @@ -37,7 +37,7 @@ the configuration options as follows: ### Your own Libravatar server -If you are [running your own libravatar service](https://wiki.libravatar.org/running_your_own/), +If you are [running your own Libravatar service](https://wiki.libravatar.org/running_your_own/), the URL will be different in the configuration, but you must provide the same placeholders so GitLab can parse the URL correctly. diff --git a/doc/administration/logs.md b/doc/administration/logs.md index 57b12897979..7d7053a26db 100644 --- a/doc/administration/logs.md +++ b/doc/administration/logs.md @@ -46,6 +46,8 @@ Line breaks have been added to this example for legibility: "gitaly_duration_s":0.16, "redis_calls":115, "redis_duration_s":0.13, + "redis_read_bytes":1507378, + "redis_write_bytes":2920, "correlation_id":"O1SdybnnIq7", "cpu_s":17.50, "db_duration_s":0.08, @@ -56,23 +58,60 @@ Line breaks have been added to this example for legibility: This example was a GET request for a specific issue. Each line also contains performance data, with times in -milliseconds: +seconds: 1. `duration_s`: total time taken to retrieve the request 1. `queue_duration_s`: total time that the request was queued inside GitLab Workhorse 1. `view_duration_s`: total time taken inside the Rails views 1. `db_duration_s`: total time to retrieve data from PostgreSQL -1. `redis_duration_s`: total time to retrieve data from Redis 1. `cpu_s`: total time spent on CPU 1. `gitaly_duration_s`: total time taken by Gitaly calls 1. `gitaly_calls`: total number of calls made to Gitaly 1. `redis_calls`: total number of calls made to Redis +1. `redis_duration_s`: total time to retrieve data from Redis +1. `redis_read_bytes`: total bytes read from Redis +1. `redis_write_bytes`: total bytes written to Redis +1. `redis_<instance>_calls`: total number of calls made to a Redis instance +1. `redis_<instance>_duration_s`: total time to retrieve data from a Redis instance +1. `redis_<instance>_read_bytes`: total bytes read from a Redis instance +1. `redis_<instance>_write_bytes`: total bytes written to a Redis instance User clone and fetch activity using HTTP transport appears in this log as `action: git_upload_pack`. In addition, the log contains the originating IP address, (`remote_ip`),the user's ID (`user_id`), and username (`username`). +Some endpoints such as `/search` may make requests to Elasticsearch if using +[Advanced Global Search](../user/search/advanced_global_search.md). These will +additionally log `elasticsearch_calls` and `elasticsearch_call_duration_s`, +which correspond to: + +1. `elasticsearch_calls`: total number of calls to Elasticsearch +1. `elasticsearch_duration_s`: total time taken by Elasticsearch calls + +ActionCable connection and subscription events are also logged to this file and they follow the same +format above. The `method`, `path`, and `format` fields are not applicable, and are always empty. +The ActionCable connection or channel class is used as the `controller`. + +```json +{ + "method":{}, + "path":{}, + "format":{}, + "controller":"IssuesChannel", + "action":"subscribe", + "status":200, + "time":"2020-05-14T19:46:22.008Z", + "params":[{"key":"project_path","value":"gitlab/gitlab-foss"},{"key":"iid","value":"1"}], + "remote_ip":"127.0.0.1", + "user_id":1, + "username":"admin", + "ua":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:76.0) Gecko/20100101 Firefox/76.0", + "correlation_id":"jSOIEynHCUa", + "duration_s":0.32566 +} +``` + NOTE: **Note:** Starting with GitLab 12.5, if an error occurs, an `exception` field is included with `class`, `message`, and `backtrace`. Previous versions included an `error` field instead of @@ -218,7 +257,7 @@ October 07, 2014 11:25: Project "project133" was removed ## `application_json.log` -> [Introduced](https://gitlab.com/gitlab-org/gitlab/issues/22812) in GitLab 12.7. +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/22812) in GitLab 12.7. This file lives in `/var/log/gitlab/gitlab-rails/application_json.log` for Omnibus GitLab packages or in `/home/git/gitlab/log/application_json.log` for @@ -340,7 +379,7 @@ only. For example: ## `audit_json.log` NOTE: **Note:** -Most log entries only exist in [GitLab Starter](https://about.gitlab.com/pricing), however a few exist in GitLab Core. +Most log entries only exist in [GitLab Starter](https://about.gitlab.com/pricing/), however a few exist in GitLab Core. This file lives in `/var/log/gitlab/gitlab-rails/audit_json.log` for Omnibus GitLab packages or in `/home/git/gitlab/log/audit_json.log` for @@ -593,6 +632,16 @@ installations from source. It logs the progress of the import process. +## `exporter.log` + +> Introduced in GitLab 13.1. + +This file lives in `/var/log/gitlab/gitlab-rails/exporter.log` for +Omnibus GitLab packages or in `/home/git/gitlab/log/exporter.log` for +installations from source. + +It logs the progress of the export process. + ## `auth.log` > Introduced in GitLab 12.0. @@ -608,12 +657,12 @@ This log records: - [Protected paths](../user/admin_area/settings/protected_paths.md) abusive requests. NOTE: **Note:** -From [%12.1](https://gitlab.com/gitlab-org/gitlab-foss/issues/62756), user ID and username are also +From [%12.3](https://gitlab.com/gitlab-org/gitlab/-/issues/29239), user ID and username are also recorded on this log, if available. ## `graphql_json.log` -> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/issues/59587) in GitLab 12.0. +> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/59587) in GitLab 12.0. This file lives in `/var/log/gitlab/gitlab-rails/graphql_json.log` for Omnibus GitLab packages or in `/home/git/gitlab/log/graphql_json.log` for @@ -724,17 +773,6 @@ Each line contains a JSON line that can be ingested by Elasticsearch. For exampl "severity": "ERROR", "time": "2019-12-17T11:49:29.485Z", "correlation_id": "AbDVUrrTvM1", - "extra.server": { - "os": { - "name": "Darwin", - "version": "Darwin Kernel Version 19.2.0", - "build": "19.2.0", - }, - "runtime": { - "name": "ruby", - "version": "ruby 2.6.5p114 (2019-10-01 revision 67812) [x86_64-darwin18]" - } - }, "extra.project_id": 55, "extra.relation_key": "milestones", "extra.relation_index": 1, @@ -756,7 +794,7 @@ Omnibus GitLab packages or in `/home/git/gitlab/log/service_measurement.log` for installations from source. It contain only a single structured log with measurements for each service execution. -It will contain measurement such as: number of sql calls, execution_time, gc_stats, memory usage, etc... +It will contain measurement such as: number of SQL calls, `execution_time`, `gc_stats`, memory usage, etc... For example: diff --git a/doc/administration/merge_request_diffs.md b/doc/administration/merge_request_diffs.md index 795933e2772..da9ca42ed9e 100644 --- a/doc/administration/merge_request_diffs.md +++ b/doc/administration/merge_request_diffs.md @@ -1,6 +1,6 @@ # Merge request diffs storage **(CORE ONLY)** -> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/issues/52568) in GitLab 11.8. +> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/52568) in GitLab 11.8. Merge request diffs are size-limited copies of diffs associated with merge requests. When viewing a merge request, diffs are sourced from these copies diff --git a/doc/administration/monitoring/github_imports.md b/doc/administration/monitoring/github_imports.md index 6d5915930b2..21cc4c708a8 100644 --- a/doc/administration/monitoring/github_imports.md +++ b/doc/administration/monitoring/github_imports.md @@ -1,3 +1,9 @@ +--- +stage: Monitor +group: APM +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + # Monitoring GitHub imports >**Note:** diff --git a/doc/administration/monitoring/gitlab_self_monitoring_project/index.md b/doc/administration/monitoring/gitlab_self_monitoring_project/index.md index 091da46f316..d05fb803c5c 100644 --- a/doc/administration/monitoring/gitlab_self_monitoring_project/index.md +++ b/doc/administration/monitoring/gitlab_self_monitoring_project/index.md @@ -1,7 +1,13 @@ +--- +stage: Monitor +group: APM +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + # GitLab self monitoring project -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/issues/32351) in GitLab 12.7, behind a disabled feature flag (`self_monitoring_project`). -> - The feature flag was removed and the Self Monitoring Project was [made generally available](https://gitlab.com/gitlab-org/gitlab/issues/198511) in GitLab 12.8. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/32351) in GitLab 12.7, behind a disabled feature flag (`self_monitoring_project`). +> - The feature flag was removed and the Self Monitoring Project was [made generally available](https://gitlab.com/gitlab-org/gitlab/-/issues/198511) in GitLab 12.8. GitLab has been adding the ability for administrators to see insights into the health of their GitLab instance. In order to surface this experience in a native way, similar to how @@ -15,10 +21,12 @@ All administrators at the time of creation of the project and group will be adde as maintainers of the group and project, and as an admin, you'll be able to add new members to the group in order to give them maintainer access to the project. -This project is used to self monitor your GitLab instance. Metrics are not yet -fully integrated, and the dashboard does not aggregate any data on Omnibus installations. GitLab plans -to provide integrated self-monitoring metrics in a future release. You can -currently use the project to configure your own [custom metrics](../../../user/project/integrations/prometheus.md#adding-custom-metrics) using +This project is used to self monitor your GitLab instance. The metrics dashboard +of the project shows some basic resource usage charts, such as CPU and memory usage +of each server in [Omnibus GitLab](https://docs.gitlab.com/omnibus/) installations. + +You can also use the project to configure your own +[custom metrics](../../../user/project/integrations/prometheus.md#adding-custom-metrics) using metrics exposed by the [GitLab exporter](../prometheus/gitlab_metrics.md#metrics-available). ## Creating the self monitoring project @@ -53,7 +61,7 @@ you should ## Taking action on Prometheus alerts **(ULTIMATE)** You can [add a webhook](../../../user/project/integrations/prometheus.md#external-prometheus-instances) -to the Prometheus config in order for GitLab to receive notifications of any alerts. +to the Prometheus configuration in order for GitLab to receive notifications of any alerts. Once the webhook is setup, you can [take action on incoming alerts](../../../user/project/integrations/prometheus.md#taking-action-on-incidents-ultimate). @@ -69,7 +77,7 @@ You can add custom metrics in the self monitoring project by: ### Getting error message in logs: `Could not create instance administrators group. Errors: ["You don’t have permission to create groups."]` -There is [a bug](https://gitlab.com/gitlab-org/gitlab/issues/208676) which causes +There is [a bug](https://gitlab.com/gitlab-org/gitlab/-/issues/208676) which causes project creation to fail with the following error (which appears in the log file) when the first admin user is an [external user](../../../user/permissions.md#external-users-core-only): diff --git a/doc/administration/monitoring/index.md b/doc/administration/monitoring/index.md index 1d7c52a198f..a54c25450c6 100644 --- a/doc/administration/monitoring/index.md +++ b/doc/administration/monitoring/index.md @@ -1,3 +1,9 @@ +--- +stage: Monitor +group: APM +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + # Monitoring GitLab Explore our features to monitor your GitLab instance: diff --git a/doc/administration/monitoring/ip_whitelist.md b/doc/administration/monitoring/ip_whitelist.md index 1b0ea1744d9..862a9368be8 100644 --- a/doc/administration/monitoring/ip_whitelist.md +++ b/doc/administration/monitoring/ip_whitelist.md @@ -1,3 +1,9 @@ +--- +stage: Monitor +group: APM +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + # IP whitelist > Introduced in GitLab 9.4. diff --git a/doc/administration/monitoring/performance/gitlab_configuration.md b/doc/administration/monitoring/performance/gitlab_configuration.md index 14119a5d8f3..d09dabab40d 100644 --- a/doc/administration/monitoring/performance/gitlab_configuration.md +++ b/doc/administration/monitoring/performance/gitlab_configuration.md @@ -1,3 +1,9 @@ +--- +stage: Monitor +group: APM +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + # GitLab Configuration GitLab Performance Monitoring is disabled by default. To enable it and change any of its diff --git a/doc/administration/monitoring/performance/grafana_configuration.md b/doc/administration/monitoring/performance/grafana_configuration.md index 3438a564d2f..4dbe3aed84e 100644 --- a/doc/administration/monitoring/performance/grafana_configuration.md +++ b/doc/administration/monitoring/performance/grafana_configuration.md @@ -1,3 +1,9 @@ +--- +stage: Monitor +group: APM +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + # Grafana Configuration [Grafana](https://grafana.com/) is a tool that allows you to visualize time @@ -57,7 +63,7 @@ repository for more information on this process. ## Integration with GitLab UI -> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/issues/61005) in GitLab 12.1. +> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/61005) in GitLab 12.1. If you have set up Grafana, you can enable a link to access it easily from the sidebar: diff --git a/doc/administration/monitoring/performance/img/request_profiling_token.png b/doc/administration/monitoring/performance/img/request_profiling_token.png Binary files differdeleted file mode 100644 index ee819fcb437..00000000000 --- a/doc/administration/monitoring/performance/img/request_profiling_token.png +++ /dev/null diff --git a/doc/administration/monitoring/performance/index.md b/doc/administration/monitoring/performance/index.md index 02070287611..6f22327e499 100644 --- a/doc/administration/monitoring/performance/index.md +++ b/doc/administration/monitoring/performance/index.md @@ -1,3 +1,9 @@ +--- +stage: Monitor +group: APM +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + # GitLab Performance Monitoring GitLab comes with its own application performance measuring system as of GitLab diff --git a/doc/administration/monitoring/performance/performance_bar.md b/doc/administration/monitoring/performance/performance_bar.md index 89246bc9782..c681dedbca8 100644 --- a/doc/administration/monitoring/performance/performance_bar.md +++ b/doc/administration/monitoring/performance/performance_bar.md @@ -1,3 +1,9 @@ +--- +stage: Monitor +group: APM +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + # Performance Bar A Performance Bar can be displayed, to dig into the performance of a page. When @@ -18,7 +24,7 @@ It allows you to see (from left to right):  - total load timings of the page; click through for details of these calls. Values in the following order: - Backend - Time that the actual base page took to load - - [First Contentful Paint](https://developers.google.com/web/tools/lighthouse/audits/first-contentful-paint) - Time until something was visible to the user + - [First Contentful Paint](hhttps://web.dev/first-contentful-paint/) - Time until something was visible to the user - [DomContentLoaded](https://developers.google.com/web/fundamentals/performance/critical-rendering-path/measure-crp) Event - Number of Requests that the page loaded  diff --git a/doc/administration/monitoring/performance/request_profiling.md b/doc/administration/monitoring/performance/request_profiling.md index c32edb60f9d..a3b29493d84 100644 --- a/doc/administration/monitoring/performance/request_profiling.md +++ b/doc/administration/monitoring/performance/request_profiling.md @@ -1,17 +1,39 @@ +--- +stage: Monitor +group: APM +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + # Request Profiling -## Procedure +To profile a request: + +1. Sign in to GitLab as a user with Administrator or Maintainer [permissions](../../../user/permissions.md). +1. In the navigation bar, click **{admin}** **Admin area**. +1. Navigate to **{monitor}** **Monitoring > Requests Profiles**. +1. In the **Requests Profiles** section, copy the token. +1. Pass the headers `X-Profile-Token: <token>` and `X-Profile-Mode: <mode>`(where + `<mode>` can be `execution` or `memory`) to the request you want to profile. When + passing headers, you can use: + + - Browser extensions such as the + [ModHeader](https://chrome.google.com/webstore/detail/modheader/idgpnmonknjnojddfkpgkljpfnnfcklj) + Chrome extension. + - `curl`. For example: + + ```shell + curl --header 'X-Profile-Token: <token>' --header 'X-Profile-Mode: <mode>' "https://gitlab.example.com/group/project" + ``` + + NOTE: **Note:** + Profiled requests can take longer than usual. + +After the request completes, you can view the profiling output from the +**{monitor}** **Monitoring > Requests Profiles** administration page: -1. Grab the profiling token from **Monitoring > Requests Profiles** admin page - (highlighted in a blue in the image below). -  -1. Pass the header `X-Profile-Token: <token>` and `X-Profile-Mode: <mode>`(where `<mode>` can be `execution` or `memory`) to the request you want to profile. You can use: - - Browser extensions. For example, [ModHeader](https://chrome.google.com/webstore/detail/modheader/idgpnmonknjnojddfkpgkljpfnnfcklj) Chrome extension. - - `curl`. For example, `curl --header 'X-Profile-Token: <token>' --header 'X-Profile-Mode: <mode>' https://gitlab.example.com/group/project`. -1. Once request is finished (which will take a little longer than usual), you can - view the profiling output from **Monitoring > Requests Profiles** admin page. -  + -## Cleaning up +## Cleaning up profiled requests -Profiling output will be cleared out every day via a Sidekiq worker. +The output from profiled requests is cleared out once each day through a +Sidekiq worker. diff --git a/doc/administration/monitoring/prometheus/gitlab_metrics.md b/doc/administration/monitoring/prometheus/gitlab_metrics.md index f725db9a039..f3084b1732e 100644 --- a/doc/administration/monitoring/prometheus/gitlab_metrics.md +++ b/doc/administration/monitoring/prometheus/gitlab_metrics.md @@ -6,30 +6,29 @@ info: To determine the technical writer assigned to the Stage/Group associated w # GitLab Prometheus metrics ->**Note:** -Available since [Omnibus GitLab 9.3](https://gitlab.com/gitlab-org/gitlab-foss/issues/29118). For -installations from source you'll have to configure it yourself. - To enable the GitLab Prometheus metrics: -1. Log into GitLab as an administrator. -1. Navigate to **Admin Area > Settings > Metrics and profiling**. +1. Log into GitLab as a user with [administrator permissions](../../../user/permissions.md). +1. Navigate to **{admin}** **Admin Area > Settings > Metrics and profiling**. 1. Find the **Metrics - Prometheus** section, and click **Enable Prometheus Metrics**. 1. [Restart GitLab](../../restart_gitlab.md#omnibus-gitlab-restart) for the changes to take effect. +NOTE: **Note:** +For installations from source you'll have to configure it yourself. + ## Collecting the metrics GitLab monitors its own internal service metrics, and makes them available at the -`/-/metrics` endpoint. Unlike other [Prometheus](https://prometheus.io) exporters, in order to access -it, the client IP needs to be [included in a whitelist](../ip_whitelist.md). +`/-/metrics` endpoint. Unlike other [Prometheus](https://prometheus.io) exporters, to access +it, the client IP address needs to be [explicitly allowed](../ip_whitelist.md). -For Omnibus and Chart installations, these metrics are automatically enabled -and collected as of [GitLab -9.4](https://gitlab.com/gitlab-org/omnibus-gitlab/-/merge_requests/1702). For -source installations or earlier versions, these metrics will need to be enabled +For [Omnibus GitLab](https://docs.gitlab.com/omnibus/) and Chart installations, +these metrics are enabled and collected as of +[GitLab 9.4](https://gitlab.com/gitlab-org/omnibus-gitlab/-/merge_requests/1702). +For source installations or earlier versions, these metrics must be enabled manually and collected by a Prometheus server. -See also [Sidekiq metrics](#sidekiq-metrics) for how to enable and view metrics from Sidekiq nodes. +For enabling and viewing metrics from Sidekiq nodes, see [Sidekiq metrics](#sidekiq-metrics). ## Metrics available @@ -43,32 +42,33 @@ The following metrics are available: | `gitlab_cache_operation_duration_seconds` | Histogram | 10.2 | Cache access time | | | `gitlab_cache_operations_total` | Counter | 12.2 | Cache operations by controller/action | `controller`, `action`, `operation` | | `gitlab_ci_pipeline_creation_duration_seconds` | Histogram | 13.0 | Time in seconds it takes to create a CI/CD pipeline | | +| `gitlab_ci_pipeline_size_builds` | Histogram | 13.1 | Total number of builds within a pipeline grouped by a pipeline source | `source` | | `job_waiter_started_total` | Counter | 12.9 | Number of batches of jobs started where a web request is waiting for the jobs to complete | `worker` | | `job_waiter_timeouts_total` | Counter | 12.9 | Number of batches of jobs that timed out where a web request is waiting for the jobs to complete | `worker` | | `gitlab_database_transaction_seconds` | Histogram | 12.1 | Time spent in database transactions, in seconds | | | `gitlab_method_call_duration_seconds` | Histogram | 10.2 | Method calls real duration | `controller`, `action`, `module`, `method` | | `gitlab_page_out_of_bounds` | Counter | 12.8 | Counter for the PageLimiter pagination limit being hit | `controller`, `action`, `bot` | | `gitlab_rails_queue_duration_seconds` | Histogram | 9.4 | Measures latency between GitLab Workhorse forwarding a request to Rails | | -| `gitlab_sql_duration_seconds` | Histogram | 10.2 | SQL execution time, excluding SCHEMA operations and BEGIN / COMMIT | | -| `gitlab_transaction_allocated_memory_bytes` | Histogram | 10.2 | Allocated memory for all transactions (gitlab_transaction_* metrics) | | +| `gitlab_sql_duration_seconds` | Histogram | 10.2 | SQL execution time, excluding `SCHEMA` operations and `BEGIN` / `COMMIT` | | +| `gitlab_transaction_allocated_memory_bytes` | Histogram | 10.2 | Allocated memory for all transactions (`gitlab_transaction_*` metrics) | | | `gitlab_transaction_cache_<key>_count_total` | Counter | 10.2 | Counter for total Rails cache calls (per key) | | | `gitlab_transaction_cache_<key>_duration_total` | Counter | 10.2 | Counter for total time (seconds) spent in Rails cache calls (per key) | | | `gitlab_transaction_cache_count_total` | Counter | 10.2 | Counter for total Rails cache calls (aggregate) | | | `gitlab_transaction_cache_duration_total` | Counter | 10.2 | Counter for total time (seconds) spent in Rails cache calls (aggregate) | | | `gitlab_transaction_cache_read_hit_count_total` | Counter | 10.2 | Counter for cache hits for Rails cache calls | `controller`, `action` | | `gitlab_transaction_cache_read_miss_count_total` | Counter | 10.2 | Counter for cache misses for Rails cache calls | `controller`, `action` | -| `gitlab_transaction_duration_seconds` | Histogram | 10.2 | Duration for all transactions (gitlab_transaction_* metrics) | `controller`, `action` | +| `gitlab_transaction_duration_seconds` | Histogram | 10.2 | Duration for all transactions (`gitlab_transaction_*` metrics) | `controller`, `action` | | `gitlab_transaction_event_build_found_total` | Counter | 9.4 | Counter for build found for API /jobs/request | | | `gitlab_transaction_event_build_invalid_total` | Counter | 9.4 | Counter for build invalid due to concurrency conflict for API /jobs/request | | | `gitlab_transaction_event_build_not_found_cached_total` | Counter | 9.4 | Counter for cached response of build not found for API /jobs/request | | | `gitlab_transaction_event_build_not_found_total` | Counter | 9.4 | Counter for build not found for API /jobs/request | | | `gitlab_transaction_event_change_default_branch_total` | Counter | 9.4 | Counter when default branch is changed for any repository | | | `gitlab_transaction_event_create_repository_total` | Counter | 9.4 | Counter when any repository is created | | -| `gitlab_transaction_event_etag_caching_cache_hit_total` | Counter | 9.4 | Counter for etag cache hit. | `endpoint` | -| `gitlab_transaction_event_etag_caching_header_missing_total` | Counter | 9.4 | Counter for etag cache miss - header missing | `endpoint` | -| `gitlab_transaction_event_etag_caching_key_not_found_total` | Counter | 9.4 | Counter for etag cache miss - key not found | `endpoint` | -| `gitlab_transaction_event_etag_caching_middleware_used_total` | Counter | 9.4 | Counter for etag middleware accessed | `endpoint` | -| `gitlab_transaction_event_etag_caching_resource_changed_total` | Counter | 9.4 | Counter for etag cache miss - resource changed | `endpoint` | +| `gitlab_transaction_event_etag_caching_cache_hit_total` | Counter | 9.4 | Counter for ETag cache hit. | `endpoint` | +| `gitlab_transaction_event_etag_caching_header_missing_total` | Counter | 9.4 | Counter for ETag cache miss - header missing | `endpoint` | +| `gitlab_transaction_event_etag_caching_key_not_found_total` | Counter | 9.4 | Counter for ETag cache miss - key not found | `endpoint` | +| `gitlab_transaction_event_etag_caching_middleware_used_total` | Counter | 9.4 | Counter for ETag middleware accessed | `endpoint` | +| `gitlab_transaction_event_etag_caching_resource_changed_total` | Counter | 9.4 | Counter for ETag cache miss - resource changed | `endpoint` | | `gitlab_transaction_event_fork_repository_total` | Counter | 9.4 | Counter for repository forks (RepositoryForkWorker). Only incremented when source repository exists | | | `gitlab_transaction_event_import_repository_total` | Counter | 9.4 | Counter for repository imports (RepositoryImportWorker) | | | `gitlab_transaction_event_push_branch_total` | Counter | 9.4 | Counter for all branch pushes | | @@ -92,6 +92,13 @@ The following metrics are available: | `gitlab_view_rendering_duration_seconds` | Histogram | 10.2 | Duration for views (histogram) | `controller`, `action`, `view` | | `http_requests_total` | Counter | 9.4 | Rack request count | `method` | | `http_request_duration_seconds` | Histogram | 9.4 | HTTP response time from rack middleware | `method`, `status` | +| `gitlab_transaction_db_count_total` | Counter | 13.1 | Counter for total number of sql calls | `controller`, `action` | +| `gitlab_transaction_db_write_count_total` | Counter | 13.1 | Counter for total number of write sql calls | `controller`, `action` | +| `gitlab_transaction_db_cached_count_total` | Counter | 13.1 | Counter for total number of cached sql calls | `controller`, `action` | +| `http_redis_requests_duration_seconds` | Histogram | 13.1 | Redis requests duration during web transactions | `controller`, `action` | +| `http_redis_requests_total` | Counter | 13.1 | Redis requests count during web transactions | `controller`, `action` | +| `http_elasticsearch_requests_duration_seconds` **(STARTER)** | Histogram | 13.1 | Elasticsearch requests duration during web transactions | `controller`, `action` | +| `http_elasticsearch_requests_total` **(STARTER)** | Counter | 13.1 | Elasticsearch requests count during web transactions | `controller`, `action` | | `pipelines_created_total` | Counter | 9.4 | Counter of pipelines created | | | `rack_uncaught_errors_total` | Counter | 9.4 | Rack connections handling uncaught errors count | | | `user_session_logins_total` | Counter | 9.4 | Counter of how many users have logged in | | @@ -99,7 +106,7 @@ The following metrics are available: | `failed_login_captcha_total` | Gauge | 11.0 | Counter of failed CAPTCHA attempts during login | | | `successful_login_captcha_total` | Gauge | 11.0 | Counter of successful CAPTCHA attempts during login | | | `auto_devops_pipelines_completed_total` | Counter | 12.7 | Counter of completed Auto DevOps pipelines, labeled by status | | -| `gitlab_metrics_dashboard_processing_time_ms` | Summary | 12.10 | Metrics dashboard processing time in milliseconds | service, stages | +| `gitlab_metrics_dashboard_processing_time_ms` | Summary | 12.10 | Metrics dashboard processing time in milliseconds | service, stages | ## Metrics controlled by a feature flag @@ -119,13 +126,17 @@ configuration option in `gitlab.yml`. These metrics are served from the | Metric | Type | Since | Description | Labels | |:---------------------------------------------- |:------- |:----- |:----------- |:------ | -| `sidekiq_jobs_cpu_seconds` | Histogram | 12.4 | Seconds of cpu time to run Sidekiq job | `queue`, `boundary`, `external_dependencies`, `feature_category`, `job_status`, `urgency` | +| `sidekiq_jobs_cpu_seconds` | Histogram | 12.4 | Seconds of CPU time to run Sidekiq job | `queue`, `boundary`, `external_dependencies`, `feature_category`, `job_status`, `urgency` | | `sidekiq_jobs_completion_seconds` | Histogram | 12.2 | Seconds to complete Sidekiq job | `queue`, `boundary`, `external_dependencies`, `feature_category`, `job_status`, `urgency` | | `sidekiq_jobs_db_seconds` | Histogram | 12.9 | Seconds of DB time to run Sidekiq job | `queue`, `boundary`, `external_dependencies`, `feature_category`, `job_status`, `urgency` | | `sidekiq_jobs_gitaly_seconds` | Histogram | 12.9 | Seconds of Gitaly time to run Sidekiq job | `queue`, `boundary`, `external_dependencies`, `feature_category`, `job_status`, `urgency` | +| `sidekiq_redis_requests_duration_seconds` | Histogram | 13.1 | Duration in seconds that a Sidekiq job spent querying a Redis server | `queue`, `boundary`, `external_dependencies`, `feature_category`, `job_status`, `urgency` | +| `sidekiq_elasticsearch_requests_duration_seconds` | Histogram | 13.1 | Duration in seconds that a Sidekiq job spent in requests to an Elasticsearch server | `queue`, `boundary`, `external_dependencies`, `feature_category`, `job_status`, `urgency` | | `sidekiq_jobs_queue_duration_seconds` | Histogram | 12.5 | Duration in seconds that a Sidekiq job was queued before being executed | `queue`, `boundary`, `external_dependencies`, `feature_category`, `urgency` | | `sidekiq_jobs_failed_total` | Counter | 12.2 | Sidekiq jobs failed | `queue`, `boundary`, `external_dependencies`, `feature_category`, `urgency` | | `sidekiq_jobs_retried_total` | Counter | 12.2 | Sidekiq jobs retried | `queue`, `boundary`, `external_dependencies`, `feature_category`, `urgency` | +| `sidekiq_redis_requests_total` | Counter | 13.1 | Redis requests during a Sidekiq job execution | `queue`, `boundary`, `external_dependencies`, `feature_category`, `job_status`, `urgency` | +| `sidekiq_elasticsearch_requests_total` | Counter | 13.1 | Elasticsearch requests during a Sidekiq job execution | `queue`, `boundary`, `external_dependencies`, `feature_category`, `job_status`, `urgency` | | `sidekiq_running_jobs` | Gauge | 12.2 | Number of Sidekiq jobs running | `queue`, `boundary`, `external_dependencies`, `feature_category`, `urgency` | | `sidekiq_concurrency` | Gauge | 12.5 | Maximum number of Sidekiq jobs | | | `geo_db_replication_lag_seconds` | Gauge | 10.2 | Database replication lag (seconds) | `url` | @@ -172,7 +183,29 @@ The following metrics are available: | Metric | Type | Since | Description | |:--------------------------------- |:--------- |:------------------------------------------------------------- |:-------------------------------------- | -| `db_load_balancing_hosts` | Gauge | [12.3](https://gitlab.com/gitlab-org/gitlab/issues/13630) | Current number of load balancing hosts | +| `db_load_balancing_hosts` | Gauge | [12.3](https://gitlab.com/gitlab-org/gitlab/-/issues/13630) | Current number of load balancing hosts | + +## Connection pool metrics + +These metrics record the status of the database [connection pools](https://api.rubyonrails.org/classes/ActiveRecord/ConnectionAdapters/ConnectionPool.html). + +They all have these labels: + +1. `class` - the Ruby class being recorded. + 1. `ActiveRecord::Base` is the main database connection. + 1. `Geo::TrackingBase` is the connection to the Geo tracking database, if + enabled. +1. `host` - the host name used to connect to the database. +1. `port` - the port used to connect to the database. + +| Metric | Type | Since | Description | +|:----------------------------------------------|:------|:------|:--------------------------------------------------| +| `gitlab_database_connection_pool_size` | Gauge | 13.0 | Total connection pool capacity | +| `gitlab_database_connection_pool_connections` | Gauge | 13.0 | Current connections in the pool | +| `gitlab_database_connection_pool_busy` | Gauge | 13.0 | Connections in use where the owner is still alive | +| `gitlab_database_connection_pool_dead` | Gauge | 13.0 | Connections in use where the owner is not alive | +| `gitlab_database_connection_pool_idle` | Gauge | 13.0 | Connections not in use | +| `gitlab_database_connection_pool_waiting` | Gauge | 13.0 | Threads currently waiting on this queue | ## Ruby metrics @@ -205,31 +238,28 @@ Unicorn specific metrics, when Unicorn is used. When Puma is used instead of Unicorn, the following metrics are available: -| Metric | Type | Since | Description | -|:---------------------------------------------- |:------- |:----- |:----------- | -| `puma_workers` | Gauge | 12.0 | Total number of workers | -| `puma_running_workers` | Gauge | 12.0 | Number of booted workers | -| `puma_stale_workers` | Gauge | 12.0 | Number of old workers | -| `puma_running` | Gauge | 12.0 | Number of running threads | -| `puma_queued_connections` | Gauge | 12.0 | Number of connections in that worker's "to do" set waiting for a worker thread | -| `puma_active_connections` | Gauge | 12.0 | Number of threads processing a request | -| `puma_pool_capacity` | Gauge | 12.0 | Number of requests the worker is capable of taking right now | -| `puma_max_threads` | Gauge | 12.0 | Maximum number of worker threads | -| `puma_idle_threads` | Gauge | 12.0 | Number of spawned threads which are not processing a request | -| `puma_killer_terminations_total` | Gauge | 12.0 | Number of workers terminated by PumaWorkerKiller | +| Metric | Type | Since | Description | +|:--------------------------------- |:------- |:----- |:----------- | +| `puma_workers` | Gauge | 12.0 | Total number of workers | +| `puma_running_workers` | Gauge | 12.0 | Number of booted workers | +| `puma_stale_workers` | Gauge | 12.0 | Number of old workers | +| `puma_running` | Gauge | 12.0 | Number of running threads | +| `puma_queued_connections` | Gauge | 12.0 | Number of connections in that worker's "to do" set waiting for a worker thread | +| `puma_active_connections` | Gauge | 12.0 | Number of threads processing a request | +| `puma_pool_capacity` | Gauge | 12.0 | Number of requests the worker is capable of taking right now | +| `puma_max_threads` | Gauge | 12.0 | Maximum number of worker threads | +| `puma_idle_threads` | Gauge | 12.0 | Number of spawned threads which are not processing a request | +| `puma_killer_terminations_total` | Gauge | 12.0 | Number of workers terminated by PumaWorkerKiller | ## Metrics shared directory GitLab's Prometheus client requires a directory to store metrics data shared between multi-process services. Those files are shared among all instances running under Unicorn server. -The directory needs to be accessible to all running Unicorn's processes otherwise -metrics will not function correctly. - -For best performance its advisable that this directory will be located in `tmpfs`. - -Its location is configured using environment variable `prometheus_multiproc_dir`. +The directory must be accessible to all running Unicorn's processes, or +metrics won't function correctly. -If GitLab is installed using Omnibus and `tmpfs` is available then metrics -directory will be automatically configured. +This directory's location is configured using environment variable `prometheus_multiproc_dir`. +For best performance, create this directory in `tmpfs`. -[← Back to the main Prometheus page](index.md) +If GitLab is installed using [Omnibus GitLab](https://docs.gitlab.com/omnibus/) +and `tmpfs` is available, then the metrics directory will be configured for you. diff --git a/doc/administration/monitoring/prometheus/index.md b/doc/administration/monitoring/prometheus/index.md index cb93aca6e4e..1e233b890a2 100644 --- a/doc/administration/monitoring/prometheus/index.md +++ b/doc/administration/monitoring/prometheus/index.md @@ -84,7 +84,7 @@ To change the address/port that Prometheus listens on: 1. Save the file and [reconfigure GitLab](../../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect -### Adding custom scrape configs +### Adding custom scrape configurations You can configure additional scrape targets for the Omnibus GitLab-bundled Prometheus by editing `prometheus['scrape_configs']` in `/etc/gitlab/gitlab.rb` diff --git a/doc/administration/monitoring/prometheus/node_exporter.md b/doc/administration/monitoring/prometheus/node_exporter.md index 357303ee4e1..07f6b8b8e1e 100644 --- a/doc/administration/monitoring/prometheus/node_exporter.md +++ b/doc/administration/monitoring/prometheus/node_exporter.md @@ -6,27 +6,24 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Node exporter ->**Note:** -Available since Omnibus GitLab 8.16. For installations from source you'll -have to install and configure it yourself. - -The [node exporter](https://github.com/prometheus/node_exporter) allows you to measure +The [node exporter](https://github.com/prometheus/node_exporter) enables you to measure various machine resources such as memory, disk and CPU utilization. +NOTE: **Note:** +For installations from source you'll have to install and configure it yourself. + To enable the node exporter: -1. [Enable Prometheus](index.md#configuring-prometheus) -1. Edit `/etc/gitlab/gitlab.rb` -1. Add or find and uncomment the following line, making sure it's set to `true`: +1. [Enable Prometheus](index.md#configuring-prometheus). +1. Edit `/etc/gitlab/gitlab.rb`. +1. Add (or find and uncomment) the following line, making sure it's set to `true`: ```ruby node_exporter['enable'] = true ``` -1. Save the file and [reconfigure GitLab](../../restart_gitlab.md#omnibus-gitlab-reconfigure) - for the changes to take effect - -Prometheus will now automatically begin collecting performance data from -the node exporter exposed under `localhost:9100`. +1. Save the file, and [reconfigure GitLab](../../restart_gitlab.md#omnibus-gitlab-reconfigure) + for the changes to take effect. -[← Back to the main Prometheus page](index.md) +Prometheus will now begin collecting performance data from the node exporter +exposed at `localhost:9100`. diff --git a/doc/administration/monitoring/prometheus/pgbouncer_exporter.md b/doc/administration/monitoring/prometheus/pgbouncer_exporter.md index 92ba2d9bb52..62d0bf684b6 100644 --- a/doc/administration/monitoring/prometheus/pgbouncer_exporter.md +++ b/doc/administration/monitoring/prometheus/pgbouncer_exporter.md @@ -6,29 +6,30 @@ info: To determine the technical writer assigned to the Stage/Group associated w # PgBouncer exporter ->**Note:** -Available since [Omnibus GitLab 11.0](https://gitlab.com/gitlab-org/omnibus-gitlab/-/merge_requests/2493). -For installations from source you'll have to install and configure it yourself. +> Introduced in [Omnibus GitLab 11.0](https://gitlab.com/gitlab-org/omnibus-gitlab/-/merge_requests/2493). + +The [PgBouncer exporter](https://github.com/prometheus-community/pgbouncer_exporter) enables +you to measure various [PgBouncer](https://www.pgbouncer.org/) metrics. -The [PgBouncer exporter](https://github.com/stanhu/pgbouncer_exporter) allows you to measure various PgBouncer metrics. +NOTE: **Note:** +For installations from source you'll have to install and configure it yourself. To enable the PgBouncer exporter: -1. [Enable Prometheus](index.md#configuring-prometheus) -1. Edit `/etc/gitlab/gitlab.rb` -1. Add or find and uncomment the following line, making sure it's set to `true`: +1. [Enable Prometheus](index.md#configuring-prometheus). +1. Edit `/etc/gitlab/gitlab.rb`. +1. Add (or find and uncomment) the following line, making sure it's set to `true`: ```ruby pgbouncer_exporter['enable'] = true ``` -1. Save the file and [reconfigure GitLab](../../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to - take effect. +1. Save the file and [reconfigure GitLab](../../restart_gitlab.md#omnibus-gitlab-reconfigure) + for the changes to take effect. -Prometheus will now automatically begin collecting performance data from -the PgBouncer exporter exposed under `localhost:9188`. +Prometheus will now begin collecting performance data from the PgBouncer exporter +exposed at `localhost:9188`. -The PgBouncer exporter will also be enabled by default if the [`pgbouncer_role`](https://docs.gitlab.com/omnibus/roles/#postgres-roles) +The PgBouncer exporter will also be enabled by default if the +[`pgbouncer_role`](https://docs.gitlab.com/omnibus/roles/#postgresql-roles) role is enabled. - -[← Back to the main Prometheus page](index.md) diff --git a/doc/administration/monitoring/prometheus/postgres_exporter.md b/doc/administration/monitoring/prometheus/postgres_exporter.md index 77ca502b21d..e3fff45fce3 100644 --- a/doc/administration/monitoring/prometheus/postgres_exporter.md +++ b/doc/administration/monitoring/prometheus/postgres_exporter.md @@ -6,12 +6,11 @@ info: To determine the technical writer assigned to the Stage/Group associated w # PostgreSQL Server Exporter ->**Note:** -Available since [Omnibus GitLab 8.17](https://gitlab.com/gitlab-org/omnibus-gitlab/-/merge_requests/1131). -For installations from source you will have to install and configure it yourself. - The [PostgreSQL Server Exporter](https://github.com/wrouesnel/postgres_exporter) allows you to export various PostgreSQL metrics. +NOTE: **Note:** +For installations from source you will have to install and configure it yourself. + To enable the PostgreSQL Server Exporter: 1. [Enable Prometheus](index.md#configuring-prometheus). @@ -21,10 +20,10 @@ To enable the PostgreSQL Server Exporter: postgres_exporter['enable'] = true ``` -NOTE: **Note:** -If PostgreSQL Server Exporter is configured on a separate node, make sure that the local -address is [listed in `trust_auth_cidr_addresses`](../../high_availability/database.md#network-information) or the -exporter will not be able to connect to the database. + NOTE: **Note:** + If PostgreSQL Server Exporter is configured on a separate node, make sure that the local + address is [listed in `trust_auth_cidr_addresses`](../../postgresql/replication_and_failover.md#network-information) or the + exporter will not be able to connect to the database. 1. Save the file and [reconfigure GitLab](../../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. @@ -34,31 +33,44 @@ the PostgreSQL Server Exporter exposed under `localhost:9187`. ## Advanced configuration -In most cases, PostgreSQL Server Exporter will work with the defaults and you should not -need to change anything. - -To further customize the PostgreSQL Server Exporter, use the following configuration options: +In most cases, PostgreSQL Server Exporter works with the defaults and you should not +need to change anything. To further customize the PostgreSQL Server Exporter, +use the following configuration options: 1. Edit `/etc/gitlab/gitlab.rb`: ```ruby - postgres_exporter['dbname'] = 'pgbouncer' # The name of the database to connect to. - postgres_exporter['user'] = 'gitlab-psql' # The user to sign in as. - postgres_exporter['password'] = '' # The user's password. - postgres_exporter['host'] = 'localhost' # The host to connect to. Values that start with '/' are for unix domain sockets (default is 'localhost'). - postgres_exporter['port'] = 5432 # The port to bind to (default is '5432'). - postgres_exporter['sslmode'] = 'require' # Whether or not to use SSL. Valid options are: - # 'disable' (no SSL), - # 'require' (always use SSL and skip verification, this is the default value), - # 'verify-ca' (always use SSL and verify that the certificate presented by the server was signed by a trusted CA), - # 'verify-full' (always use SSL and verify that the certification presented by the server was signed by a trusted CA and the server host name matches the one in the certificate). - postgres_exporter['fallback_application_name'] = '' # An application_name to fall back to if one isn't provided. - postgres_exporter['connect_timeout'] = '' # Maximum wait for connection, in seconds. Zero or not specified means wait indefinitely. - postgres_exporter['sslcert'] = 'ssl.crt' # Cert file location. The file must contain PEM encoded data. - postgres_exporter['sslkey'] = 'ssl.key' # Key file location. The file must contain PEM encoded data. - postgres_exporter['sslrootcert'] = 'ssl-root.crt' # The location of the root certificate file. The file must contain PEM encoded data. + # The name of the database to connect to. + postgres_exporter['dbname'] = 'pgbouncer' + # The user to sign in as. + postgres_exporter['user'] = 'gitlab-psql' + # The user's password. + postgres_exporter['password'] = '' + # The host to connect to. Values that start with '/' are for unix domain sockets + # (default is 'localhost'). + postgres_exporter['host'] = 'localhost' + # The port to bind to (default is '5432'). + postgres_exporter['port'] = 5432 + # Whether or not to use SSL. Valid options are: + # 'disable' (no SSL), + # 'require' (always use SSL and skip verification, this is the default value), + # 'verify-ca' (always use SSL and verify that the certificate presented by + # the server was signed by a trusted CA), + # 'verify-full' (always use SSL and verify that the certification presented + # by the server was signed by a trusted CA and the server host name matches + # the one in the certificate). + postgres_exporter['sslmode'] = 'require' + # An application_name to fall back to if one isn't provided. + postgres_exporter['fallback_application_name'] = '' + # Maximum wait for connection, in seconds. Zero or not specified means wait indefinitely. + postgres_exporter['connect_timeout'] = '' + # Cert file location. The file must contain PEM encoded data. + postgres_exporter['sslcert'] = 'ssl.crt' + # Key file location. The file must contain PEM encoded data. + postgres_exporter['sslkey'] = 'ssl.key' + # The location of the root certificate file. The file must contain PEM encoded data. + postgres_exporter['sslrootcert'] = 'ssl-root.crt' ``` -1. Save the file and [reconfigure GitLab](../../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. - -[← Back to the main Prometheus page](index.md) +1. Save the file and [reconfigure GitLab](../../restart_gitlab.md#omnibus-gitlab-reconfigure) + for the changes to take effect. diff --git a/doc/administration/monitoring/prometheus/redis_exporter.md b/doc/administration/monitoring/prometheus/redis_exporter.md index bef87400f5a..b7c66959349 100644 --- a/doc/administration/monitoring/prometheus/redis_exporter.md +++ b/doc/administration/monitoring/prometheus/redis_exporter.md @@ -6,19 +6,18 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Redis exporter ->**Note:** -Available since [Omnibus GitLab 8.17](https://gitlab.com/gitlab-org/omnibus-gitlab/-/merge_requests/1118). -For installations from source you'll have to install and configure it yourself. - -The [Redis exporter](https://github.com/oliver006/redis_exporter) allows you to measure -various [Redis](https://redis.io) metrics. For more information on what's exported, +The [Redis exporter](https://github.com/oliver006/redis_exporter) enables you to measure +various [Redis](https://redis.io) metrics. For more information on what is exported, [read the upstream documentation](https://github.com/oliver006/redis_exporter/blob/master/README.md#whats-exported). +NOTE: **Note:** +For installations from source you'll have to install and configure it yourself. + To enable the Redis exporter: -1. [Enable Prometheus](index.md#configuring-prometheus) -1. Edit `/etc/gitlab/gitlab.rb` -1. Add or find and uncomment the following line, making sure it's set to `true`: +1. [Enable Prometheus](index.md#configuring-prometheus). +1. Edit `/etc/gitlab/gitlab.rb`. +1. Add (or find and uncomment) the following line, making sure it's set to `true`: ```ruby redis_exporter['enable'] = true @@ -27,7 +26,5 @@ To enable the Redis exporter: 1. Save the file and [reconfigure GitLab](../../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. -Prometheus will now automatically begin collecting performance data from -the Redis exporter exposed under `localhost:9121`. - -[← Back to the main Prometheus page](index.md) +Prometheus will now begin collecting performance data from +the Redis exporter exposed at `localhost:9121`. diff --git a/doc/administration/object_storage.md b/doc/administration/object_storage.md index c4f9b672923..1dea2de73f6 100644 --- a/doc/administration/object_storage.md +++ b/doc/administration/object_storage.md @@ -66,7 +66,7 @@ One risk of using a single bucket would be that if your organisation decided to migrate GitLab to the Helm deployment in the future. GitLab would run, but the situation with backups might not be realised until the organisation had a critical requirement for the backups to work. -### S3 API compatability issues +### S3 API compatibility issues Not all S3 providers [are fully compatible](../raketasks/backup_restore.md#other-s3-providers) with the Fog library that GitLab uses. Symptoms include: @@ -79,7 +79,7 @@ with the Fog library that GitLab uses. Symptoms include: If you're working to add more GitLab servers for [scaling or fault tolerance](reference_architectures/index.md) and one of your requirements is [GitLab Pages](../user/project/pages/index.md) this currently requires -NFS. There is [work in progress](https://gitlab.com/gitlab-org/gitlab-pages/issues/196) +NFS. There is [work in progress](https://gitlab.com/gitlab-org/gitlab-pages/-/issues/196) to remove this dependency. In the future, GitLab Pages may use [object storage](https://gitlab.com/gitlab-org/gitlab/-/issues/208135). @@ -141,10 +141,88 @@ Using the default GitLab settings, some object storage back-ends such as and [Alibaba](https://gitlab.com/gitlab-org/charts/gitlab/-/issues/1564) might generate `ETag mismatch` errors. +If you are seeing this ETag mismatch error with Amazon Web Services S3, +it's likely this is due to [encryption settings on your bucket](https://docs.aws.amazon.com/AmazonS3/latest/API/RESTCommonResponseHeaders.html). +See the section on [using Amazon instance profiles](#using-amazon-instance-profiles) on how to fix this issue. + When using GitLab direct upload, the [workaround for MinIO](https://gitlab.com/gitlab-org/charts/gitlab/-/issues/1564#note_244497658) is to use the `--compat` parameter on the server. -We are working on a fix to GitLab component Workhorse, and also -a workaround, in the mean time, to -[allow ETag verification to be disabled](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/18175). +We are working on a fix to the [GitLab Workhorse +component](https://gitlab.com/gitlab-org/gitlab-workhorse/-/issues/222). + +### Using Amazon instance profiles + +Instead of supplying AWS access and secret keys in object storage +configuration, GitLab can be configured to use IAM roles to set up an +[Amazon instance profile](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_use_switch-role-ec2.html). +When this is used, GitLab will fetch temporary credentials each time an +S3 bucket is accessed, so no hard-coded values are needed in the +configuration. + +#### Encrypted S3 buckets + +> Introduced in [GitLab 13.1](https://gitlab.com/gitlab-org/gitlab-workhorse/-/merge_requests/466) only for instance profiles. + +When configured to use an instance profile, GitLab Workhorse +will properly upload files to S3 buckets that have [SSE-S3 or SSE-KMS +encryption enabled by default](https://docs.aws.amazon.com/kms/latest/developerguide/services-s3.html). +Note that customer master keys (CMKs) and SSE-C encryption are not yet +supported since this requires supplying keys to the GitLab +configuration. + +Without instance profiles enabled (or prior to GitLab 13.1), GitLab +Workhorse will upload files to S3 using pre-signed URLs that do not have +a `Content-MD5` HTTP header computed for them. To ensure data is not +corrupted, Workhorse checks that the MD5 hash of the data sent equals +the ETag header returned from the S3 server. When encryption is enabled, +this is not the case, which causes Workhorse to report an `ETag +mismatch` error during an upload. + +With instance profiles enabled, GitLab Workhorse uses an AWS S3 client +that properly computes and sends the `Content-MD5` header to the server, +which eliminates the need for comparing ETag headers. If the data is +corrupted in transit, the S3 server will reject the file. + +#### IAM Permissions + +To set up an instance profile, create an Amazon Identity Access and +Management (IAM) role with the necessary permissions. The following +example is a role for an S3 bucket named `test-bucket`: + +```json +{ + "Version": "2012-10-17", + "Statement": [ + { + "Sid": "VisualEditor0", + "Effect": "Allow", + "Action": [ + "s3:PutObject", + "s3:GetObject", + "s3:AbortMultipartUpload", + "s3:DeleteObject" + ], + "Resource": "arn:aws:s3:::test-bucket/*" + } + ] +} +``` + +Associate this role with your GitLab instance, and then configure GitLab +to use it via the `use_iam_profile` configuration option. For example, +when configuring uploads to use object storage, see the `AWS IAM profiles` +section in [S3 compatible connection settings](uploads.md#s3-compatible-connection-settings). + +#### Disabling the feature + +The Workhorse S3 client is only enabled when the `use_iam_profile` +configuration flag is `true`. + +To disable this feature, ask a GitLab administrator with [Rails console access](feature_flags.md#how-to-enable-and-disable-features-behind-flags) to run the +following command: + +```ruby +Feature.disable(:use_workhorse_s3_client) +``` diff --git a/doc/administration/operations/extra_sidekiq_processes.md b/doc/administration/operations/extra_sidekiq_processes.md index 8f54b82c325..155680354da 100644 --- a/doc/administration/operations/extra_sidekiq_processes.md +++ b/doc/administration/operations/extra_sidekiq_processes.md @@ -106,14 +106,14 @@ you list: ## Queue selector (experimental) -> [Introduced](https://gitlab.com/gitlab-com/gl-infra/scalability/issues/45) in [GitLab Starter](https://about.gitlab.com/pricing/) 12.8. +> [Introduced](https://gitlab.com/gitlab-com/gl-infra/scalability/-/issues/45) in [GitLab Starter](https://about.gitlab.com/pricing/) 12.8. CAUTION: **Caution:** As this is marked as **experimental**, it is subject to change at any time, including **breaking backwards compatibility**. This is so that we can react to changes we need for our GitLab.com deployment. We have a tracking issue open to [remove the experimental -designation](https://gitlab.com/gitlab-com/gl-infra/scalability/issues/147) +designation](https://gitlab.com/gitlab-com/gl-infra/scalability/-/issues/147) from this feature; please comment there if you are interested in using this in your own deployment. @@ -126,6 +126,8 @@ in a more general way using the following components: ### Available attributes +- [Introduced](https://gitlab.com/gitlab-com/gl-infra/scalability/-/issues/261) in GitLab 13.1, `tags`. + From the [list of all available attributes](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/workers/all_queues.yml), `experimental_queue_selector` allows selecting of queues by the @@ -144,14 +146,21 @@ following attributes: - `name` - the queue name. The other attributes are typically more useful as they are more general, but this is available in case a particular queue needs to be selected. -- `resource_boundary` - if the worker is bound by `cpu`, `memory`, or +- `resource_boundary` - if the queue is bound by `cpu`, `memory`, or `unknown`. For example, the `project_export` queue is memory bound as it has to load data in memory before saving it for export. +- `tags` - short-lived annotations for queues. These are expected to frequently + change from release to release, and may be removed entirely. `has_external_dependencies` is a boolean attribute: only the exact string `true` is considered true, and everything else is considered false. +`tags` is a set, which means that `=` checks for intersecting sets, and +`!=` checks for disjoint sets. For example, `tags=a,b` selects queues +that have tags `a`, `b`, or both. `tags!=a,b` selects queues that have +neither of those tags. + ### Available operators `experimental_queue_selector` supports the following operators, listed diff --git a/doc/administration/operations/fast_ssh_key_lookup.md b/doc/administration/operations/fast_ssh_key_lookup.md index 6759c3f265d..9f67c927128 100644 --- a/doc/administration/operations/fast_ssh_key_lookup.md +++ b/doc/administration/operations/fast_ssh_key_lookup.md @@ -1,7 +1,7 @@ # Fast lookup of authorized SSH keys in the database -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/issues/1631) in [GitLab Starter](https://about.gitlab.com/pricing/) 9.3. -> - [Available in](https://gitlab.com/gitlab-org/gitlab/issues/3953) GitLab Community Edition 10.4. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/1631) in [GitLab Starter](https://about.gitlab.com/pricing/) 9.3. +> - [Available in](https://gitlab.com/gitlab-org/gitlab/-/issues/3953) GitLab Community Edition 10.4. NOTE: **Note:** This document describes a drop-in replacement for the `authorized_keys` file. For normal (non-deploy key) users, consider using @@ -125,7 +125,7 @@ the database. The following instructions can be used to build OpenSSH 7.5: ```shell sudo su - cd /tmp - curl --remote-name https://mirrors.evowise.com/pub/OpenBSD/OpenSSH/portable/openssh-7.5p1.tar.gz + curl --remote-name https://cdn.openbsd.org/pub/OpenBSD/OpenSSH/portable/openssh-7.5p1.tar.gz tar xzvf openssh-7.5p1.tar.gz yum install rpm-build gcc make wget openssl-devel krb5-devel pam-devel libX11-devel xmkmf libXt-devel ``` diff --git a/doc/administration/operations/filesystem_benchmarking.md b/doc/administration/operations/filesystem_benchmarking.md index 019909e2e89..c5c5a8b4313 100644 --- a/doc/administration/operations/filesystem_benchmarking.md +++ b/doc/administration/operations/filesystem_benchmarking.md @@ -14,7 +14,7 @@ I/O. The information on this page can be used for either scenario. ### Benchmarking with `fio` We recommend using -[fio](https://fio.readthedocs.io/en/latest/fio_doc.html) to test I/O +[Fio](https://fio.readthedocs.io/en/latest/fio_doc.html) to test I/O performance. This test should be run both on the NFS server and on the application nodes that talk to the NFS server. diff --git a/doc/administration/operations/puma.md b/doc/administration/operations/puma.md index af559cf00e9..af28335ef91 100644 --- a/doc/administration/operations/puma.md +++ b/doc/administration/operations/puma.md @@ -10,7 +10,8 @@ Unicorn unless explicitly specified not to. ## Why switch to Puma? Puma has a multi-thread architecture which uses less memory than a multi-process -application server like Unicorn. +application server like Unicorn. On GitLab.com, we saw a 40% reduction in memory +consumption. Most Rails applications requests normally include a proportion of I/O wait time. During I/O wait time MRI Ruby will release the GVL (Global VM Lock) to other threads. @@ -18,9 +19,15 @@ Multi-threaded Puma can therefore still serve more requests than a single proces ## Configuring Puma to replace Unicorn -If you are currently running Unicorn and would like to switch to Puma, server configuration -will _not_ carry over automatically. For details on matching Unicorn configuration settings with -the Puma equivalent, where applicable, see [Converting Unicorn settings to Puma](https://docs.gitlab.com/omnibus/settings/puma.html#converting-unicorn-settings-to-puma). +Beginning with GitLab 13.0, Puma is the default application server. We plan to remove support for +Unicorn in GitLab 14.0. + +When switching to Puma, Unicorn server configuration +will _not_ carry over automatically, due to differences between the two application servers. For Omnibus-based +deployments, see [Configuring Puma Settings](https://docs.gitlab.com/omnibus/settings/puma.html#configuring-puma-settings). +For Helm based deployments, see the [Webservice Chart documentation](https://docs.gitlab.com/charts/charts/gitlab/webservice/index.html). + +Additionally we strongly recommend that multi-node deployments [configure their load balancers to utilize the readiness check](../high_availability/load_balancer.md#readiness-check) due to a difference between Unicorn and Puma in how they handle connections during a restart of the service. ## Performance caveat when using Puma with Rugged diff --git a/doc/administration/operations/ssh_certificates.md b/doc/administration/operations/ssh_certificates.md index b652f282b7b..c81eb15955d 100644 --- a/doc/administration/operations/ssh_certificates.md +++ b/doc/administration/operations/ssh_certificates.md @@ -158,7 +158,7 @@ Users can still bypass SSH certificate authentication by manually uploading an SSH public key to their profile, relying on the `~/.ssh/authorized_keys` fallback to authenticate it. There's currently no feature to prevent this, [but there's an open request for -adding it](https://gitlab.com/gitlab-org/gitlab-foss/issues/49218). +adding it](https://gitlab.com/gitlab-org/gitlab/-/issues/23260). Such a restriction can currently be hacked in by e.g. providing a custom `AuthorizedKeysCommand` which checks if the discovered key-ID diff --git a/doc/administration/operations/unicorn.md b/doc/administration/operations/unicorn.md index 50481482f4c..eabf99eb08c 100644 --- a/doc/administration/operations/unicorn.md +++ b/doc/administration/operations/unicorn.md @@ -40,9 +40,9 @@ master process has PID 56227 below. [2015-06-05T10:58:08.708824 #62538] INFO -- : worker=10 ready ``` -### Tunables +### Tunable options -The main tunables for Unicorn are the number of worker processes and the +The main tunable options for Unicorn are the number of worker processes and the request timeout after which the Unicorn master terminates a worker process. See the [Omnibus GitLab Unicorn settings documentation](https://gitlab.com/gitlab-org/omnibus-gitlab/blob/master/doc/settings/unicorn.md) diff --git a/doc/administration/packages/container_registry.md b/doc/administration/packages/container_registry.md index 503e5fb4a2a..8f55345a9a8 100644 --- a/doc/administration/packages/container_registry.md +++ b/doc/administration/packages/container_registry.md @@ -1,3 +1,9 @@ +--- +stage: Package +group: Package +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + # GitLab Container Registry administration > - [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/4040) in GitLab 8.8. @@ -92,7 +98,7 @@ auth: ``` CAUTION: **Caution:** -If `auth` is not set up, users will be able to pull docker images without authentication. +If `auth` is not set up, users will be able to pull Docker images without authentication. ## Container Registry domain configuration @@ -116,7 +122,7 @@ expose the Registry on a port so that you can reuse the existing GitLab TLS certificate. Assuming that the GitLab domain is `https://gitlab.example.com` and the port the -Registry is exposed to the outside world is `4567`, here is what you need to set +Registry is exposed to the outside world is `5050`, here is what you need to set in `gitlab.rb` or `gitlab.yml` if you are using Omnibus GitLab or installed GitLab from source respectively. @@ -130,7 +136,7 @@ otherwise you will run into conflicts. path to the existing TLS certificate and key used by GitLab: ```ruby - registry_external_url 'https://gitlab.example.com:4567' + registry_external_url 'https://gitlab.example.com:5050' ``` Note how the `registry_external_url` is listening on HTTPS under the @@ -151,7 +157,7 @@ otherwise you will run into conflicts. 1. Validate using: ```shell - openssl s_client -showcerts -servername gitlab.example.com -connect gitlab.example.com:443 > cacert.pem + openssl s_client -showcerts -servername gitlab.example.com -connect gitlab.example.com:5050 > cacert.pem ``` NOTE: **Note:** @@ -166,7 +172,7 @@ If your certificate provider provides the CA Bundle certificates, append them to registry: enabled: true host: gitlab.example.com - port: 4567 + port: 5050 ``` 1. Save the file and [restart GitLab](../restart_gitlab.md#installations-from-source) for the changes to take effect. @@ -176,7 +182,7 @@ Users should now be able to login to the Container Registry with their GitLab credentials using: ```shell -docker login gitlab.example.com:4567 +docker login gitlab.example.com:5050 ``` ### Configure Container Registry under its own domain @@ -370,7 +376,7 @@ The different supported drivers are: | swift | OpenStack Swift Object Storage | | oss | Aliyun OSS | -Read more about the individual driver's config options in the +Read more about the individual driver's configuration options in the [Docker Registry docs](https://docs.docker.com/registry/configuration/#storage). [Read more about using object storage with GitLab](../object_storage.md). @@ -407,8 +413,8 @@ NOTE: **Note:** **Installations from source** -Configuring the storage driver is done in your registry config YML file created -when you [deployed your docker registry](https://docs.docker.com/registry/deploying/). +Configuring the storage driver is done in your registry configuration YML file created +when you [deployed your Docker registry](https://docs.docker.com/registry/deploying/). `s3` storage driver example: @@ -429,6 +435,16 @@ storage: NOTE: **Note:** `your-s3-bucket` should only be the name of a bucket that exists, and can't include subdirectories. +**Migrate without downtime** + +To migrate the data to AWS S3 without downtime: + +1. To reduce the amount of data to be migrated, run the [garbage collection tool without downtime](#performing-garbage-collection-without-downtime). Part of this process sets the registry to `read-only`. +1. Copy the data to your AWS S3 bucket, for example with [AWS CLI's `cp`](https://docs.aws.amazon.com/cli/latest/reference/s3/cp.html) command. +1. Configure your registry to use the S3 bucket for storage. +1. Put the registry back to `read-write`. +1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. + ### Disable redirect for storage driver By default, users accessing a registry configured with a remote backend are redirected to the default backend for the storage driver. For example, registries can be configured using the `s3` storage driver, which redirects requests to a remote S3 bucket to alleviate load on the GitLab server. @@ -604,7 +620,7 @@ You can use GitLab as an auth endpoint with an external container registry. You can configure the Container Registry to send webhook notifications in response to events happening within the registry. -Read more about the Container Registry notifications config options in the +Read more about the Container Registry notifications configuration options in the [Docker Registry notifications documentation](https://docs.docker.com/registry/notifications/). NOTE: **Note:** @@ -635,8 +651,8 @@ To configure a notification endpoint in Omnibus: **Installations from source** -Configuring the notification endpoint is done in your registry config YML file created -when you [deployed your docker registry](https://docs.docker.com/registry/deploying/). +Configuring the notification endpoint is done in your registry configuration YML file created +when you [deployed your Docker registry](https://docs.docker.com/registry/deploying/). Example: @@ -704,6 +720,8 @@ no longer directly accessible via the `:latest` tag. ### Recycling unused tags +> [Introduced](https://gitlab.com/gitlab-org/omnibus-gitlab/-/merge_requests/987) in Omnibus GitLab 8.12. + There are a couple of considerations you need to note before running the built-in command: @@ -759,6 +777,8 @@ that you have backed up all registry data. ### Performing garbage collection without downtime +> [Introduced](https://gitlab.com/gitlab-org/omnibus-gitlab/-/merge_requests/764) in GitLab 8.8. + You can perform a garbage collection without stopping the Container Registry by setting it into a read-only mode and by not using the built-in command. During this time, you will be able to pull from the Container Registry, but you will not be able to @@ -873,9 +893,9 @@ thus the error above. While GitLab doesn't support using self-signed certificates with Container Registry out of the box, it is possible to make it work by -[instructing the docker-daemon to trust the self-signed certificates](https://docs.docker.com/registry/insecure/#use-self-signed-certificates), -mounting the docker-daemon and setting `privileged = false` in the Runner's -`config.toml`. Setting `privileged = true` takes precedence over the docker-daemon: +[instructing the Docker daemon to trust the self-signed certificates](https://docs.docker.com/registry/insecure/#use-self-signed-certificates), +mounting the Docker daemon and setting `privileged = false` in the Runner's +`config.toml`. Setting `privileged = true` takes precedence over the Docker daemon: ```toml [runners.docker] @@ -884,7 +904,7 @@ mounting the docker-daemon and setting `privileged = false` in the Runner's volumes = ["/var/run/docker.sock:/var/run/docker.sock", "/cache"] ``` -Additional information about this: [issue 18239](https://gitlab.com/gitlab-org/gitlab-foss/issues/18239). +Additional information about this: [issue 18239](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/18239). ### `unauthorized: authentication required` when pushing large images @@ -955,7 +975,7 @@ Start with a value between `25000000` (25MB) and `50000000` (50MB). ### Supporting older Docker clients -As of GitLab 11.9, we began shipping version 2.7.1 of the Docker container registry, which disables the schema1 manifest by default. If you are still using older Docker clients (1.9 or older), you may experience an error pushing images. See [omnibus-4145](https://gitlab.com/gitlab-org/omnibus-gitlab/issues/4145) for more details. +As of GitLab 11.9, we began shipping version 2.7.1 of the Docker container registry, which disables the schema1 manifest by default. If you are still using older Docker clients (1.9 or older), you may experience an error pushing images. See [omnibus-4145](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/4145) for more details. You can add a configuration option for backwards compatibility. @@ -1002,7 +1022,7 @@ there is likely an issue with the headers forwarded to the registry by NGINX. Th NGINX configurations should handle this, but it might occur in custom setups where the SSL is offloaded to a third party reverse proxy. -This problem was discussed in a [docker project issue](https://github.com/docker/distribution/issues/970) +This problem was discussed in a [Docker project issue](https://github.com/docker/distribution/issues/970) and a simple solution would be to enable relative URLs in the Registry. **For Omnibus installations** @@ -1062,7 +1082,7 @@ A user attempted to enable an S3-backed Registry. The `docker login` step went fine. However, when pushing an image, the output showed: ```plaintext -The push refers to a repository [s3-testing.myregistry.com:4567/root/docker-test/docker-image] +The push refers to a repository [s3-testing.myregistry.com:5050/root/docker-test/docker-image] dc5e59c14160: Pushing [==================================================>] 14.85 kB 03c20c1a019a: Pushing [==================================================>] 2.048 kB a08f14ef632e: Pushing [==================================================>] 2.048 kB @@ -1154,8 +1174,8 @@ Now that we have mitmproxy and Docker running, we can attempt to login and push a container image. You may need to run as root to do this. For example: ```shell -docker login s3-testing.myregistry.com:4567 -docker push s3-testing.myregistry.com:4567/root/docker-test/docker-image +docker login s3-testing.myregistry.com:5050 +docker push s3-testing.myregistry.com:5050/root/docker-test/docker-image ``` In the example above, we see the following trace on the mitmproxy window: diff --git a/doc/administration/packages/dependency_proxy.md b/doc/administration/packages/dependency_proxy.md index ec2020c26de..565a4521c2a 100644 --- a/doc/administration/packages/dependency_proxy.md +++ b/doc/administration/packages/dependency_proxy.md @@ -1,6 +1,12 @@ +--- +stage: Package +group: Package +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + # GitLab Dependency Proxy administration **(PREMIUM ONLY)** -> [Introduced](https://gitlab.com/gitlab-org/gitlab/issues/7934) in [GitLab Premium](https://about.gitlab.com/pricing/) 11.11. +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/7934) in [GitLab Premium](https://about.gitlab.com/pricing/) 11.11. GitLab can be utilized as a dependency proxy for a variety of common package managers. diff --git a/doc/administration/packages/index.md b/doc/administration/packages/index.md index f826741d66f..5088dd86a86 100644 --- a/doc/administration/packages/index.md +++ b/doc/administration/packages/index.md @@ -1,3 +1,9 @@ +--- +stage: Package +group: Package +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + # GitLab Package Registry administration **(PREMIUM ONLY)** GitLab Packages allows organizations to utilize GitLab as a private repository @@ -9,10 +15,12 @@ The Packages feature allows GitLab to act as a repository for the following: | Software repository | Description | Available in GitLab version | | ------------------- | ----------- | --------------------------- | | [PyPi Repository](../../user/packages/pypi_repository/index.md) | The GitLab PyPi Repository enables every project in GitLab to have its own space to store [PyPi](https://pypi.org/) packages. | 12.10+ | +| [Composer Repository](../../user/packages/composer_repository/index.md) | The GitLab Composer Repository enables every project in GitLab to have its own space to store [Composer](https://getcomposer.org/) packages. | 13.1+ | | [NuGet Repository](../../user/packages/nuget_repository/index.md) | The GitLab NuGet Repository enables every project in GitLab to have its own space to store [NuGet](https://www.nuget.org/) packages. | 12.8+ | | [Conan Repository](../../user/packages/conan_repository/index.md) | The GitLab Conan Repository enables every project in GitLab to have its own space to store [Conan](https://conan.io/) packages. | 12.4+ | | [Maven Repository](../../user/packages/maven_repository/index.md) | The GitLab Maven Repository enables every project in GitLab to have its own space to store [Maven](https://maven.apache.org/) packages. | 11.3+ | | [NPM Registry](../../user/packages/npm_registry/index.md) | The GitLab NPM Registry enables every project in GitLab to have its own space to store [NPM](https://www.npmjs.com/) packages. | 11.7+ | +| [Go Proxy](../../user/packages/go_proxy/index.md) | The Go proxy for GitLab enables every project in GitLab to be fetched with the [Go proxy protocol](https://proxy.golang.org/). | 13.1+ | Don't you see your package management system supported yet? Please consider contributing diff --git a/doc/administration/pages/index.md b/doc/administration/pages/index.md index 21d13be47bd..a7a3a86de8e 100644 --- a/doc/administration/pages/index.md +++ b/doc/administration/pages/index.md @@ -1,4 +1,7 @@ --- +stage: Release +group: Release Management +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers description: 'Learn how to administer GitLab Pages.' --- @@ -6,9 +9,9 @@ description: 'Learn how to administer GitLab Pages.' > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/80) in GitLab EE 8.3. > - Custom CNAMEs with TLS support were [introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/173) in GitLab EE 8.5. -> - GitLab Pages [was ported](https://gitlab.com/gitlab-org/gitlab-foss/issues/14605) to Community Edition in GitLab 8.17. +> - GitLab Pages [was ported](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/14605) to Community Edition in GitLab 8.17. > - Support for subgroup project's websites was -> [introduced](https://gitlab.com/gitlab-org/gitlab-foss/issues/30548) in GitLab 11.8. +> [introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/30548) in GitLab 11.8. GitLab Pages allows for hosting of static sites. It must be configured by an administrator. Separate [user documentation](../../user/project/pages/index.md) is available. @@ -75,7 +78,7 @@ among other things. Follow [these instructions](https://publicsuffix.org/submit/) to submit your GitLab Pages subdomain. For instance, if your domain is `example.io`, you should request that `example.io` is added to the Public Suffix List. GitLab.com -added `gitlab.io` [in 2016](https://gitlab.com/gitlab-com/infrastructure/issues/230). +added `gitlab.io` [in 2016](https://gitlab.com/gitlab-com/infrastructure/-/issues/230). ### DNS configuration @@ -98,7 +101,8 @@ You should not use the GitLab domain to serve user pages. For more information s ## Configuration Depending on your needs, you can set up GitLab Pages in 4 different ways. -The following options are listed from the easiest setup to the most + +The following examples are listed from the easiest setup to the most advanced one. The absolute minimum requirement is to set up the wildcard DNS since that is needed in all configurations. @@ -175,6 +179,64 @@ NOTE: **Note:** `inplace_chroot` option might not work with the other features, such as [Pages Access Control](#access-control). The [GitLab Pages README](https://gitlab.com/gitlab-org/gitlab-pages#caveats) has more information about caveats and workarounds. +### Global settings + +Below is a table of all configuration settings known to Pages in Omnibus GitLab, +and what they do. These options can be adjusted in `/etc/gitlab/gitlab.rb`, +and will take effect after you [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). +Most of these settings don't need to be configured manually unless you need more granular +control over how the Pages daemon runs and serves content in your environment. + +| Setting | Description | +| ------- | ----------- | +| `pages_external_url` | The URL where GitLab Pages is accessible, including protocol (HTTP / HTTPS). If `https://` is used, you must also set `gitlab_pages['ssl_certificate']` and `gitlab_pages['ssl_certificate_key']`. +| **gitlab_pages[]** | | +| `access_control` | Whether to enable [access control](index.md#access-control). +| `api_secret_key` | Full path to file with secret key used to authenticate with the GitLab API. Auto-generated when left unset. +| `artifacts_server` | Enable viewing [artifacts](../job_artifacts.md) in GitLab Pages. +| `artifacts_server_timeout` | Timeout (in seconds) for a proxied request to the artifacts server. +| `artifacts_server_url` | API URL to proxy artifact requests to. Defaults to GitLab `external URL` + `/api/v4`, for example `https://gitlab.com/api/v4`. +| `auth_redirect_uri` | Callback URL for authenticating with GitLab. Defaults to project's subdomain of `pages_external_url` + `/auth`. +| `auth_secret` | Secret key for signing authentication requests. Leave blank to pull automatically from GitLab during OAuth registration. +| `dir` | Working directory for config and secrets files. +| `enable` | Enable or disable GitLab Pages on the current system. +| `external_http` | Configure Pages to bind to one or more secondary IP addresses, serving HTTP requests. Multiple addresses can be given as an array, along with exact ports, for example `['1.2.3.4', '1.2.3.5:8063']`. Sets value for `listen_http`. +| `external_https` | Configure Pages to bind to one or more secondary IP addresses, serving HTTPS requests. Multiple addresses can be given as an array, along with exact ports, for example `['1.2.3.4', '1.2.3.5:8063']`. Sets value for `listen_https`. +| `gitlab_client_http_timeout` | GitLab API HTTP client connection timeout in seconds (default: 10s). +| `gitlab_client_jwt_expiry` | JWT Token expiry time in seconds (default: 30s). +| `gitlab_id` | The OAuth application public ID. Leave blank to automatically fill when Pages authenticates with GitLab. +| `gitlab_secret` | The OAuth application secret. Leave blank to automatically fill when Pages authenticates with GitLab. +| `gitlab_server` | Server to use for authentication when access control is enabled; defaults to GitLab `external_url`. +| `headers` | Specify any additional http headers that should be sent to the client with each response. +| `inplace_chroot` | On [systems that don't support bind-mounts](index.md#additional-configuration-for-docker-container), this instructs GitLab Pages to chroot into its `pages_path` directory. Some caveats exist when using inplace chroot; refer to the GitLab Pages [README](https://gitlab.com/gitlab-org/gitlab-pages/blob/master/README.md#caveats) for more information. +| `insecure_ciphers` | Use default list of cipher suites, may contain insecure ones like 3DES and RC4. +| `internal_gitlab_server` | Internal GitLab server address used exclusively for API requests. Useful if you want to send that traffic over an internal load balancer. Defaults to GitLab `external_url`. +| `listen_proxy` | The addresses to listen on for reverse-proxy requests. Pages will bind to these addresses' network socket and receives incoming requests from it. Sets the value of `proxy_pass` in `$nginx-dir/conf/gitlab-pages.conf`. +| `log_directory` | Absolute path to a log directory. +| `log_format` | The log output format: 'text' or 'json'. +| `log_verbose` | Verbose logging, true/false. +| `max_connections` | Limit on the number of concurrent connections to the HTTP, HTTPS or proxy listeners. +| `metrics_address` | The address to listen on for metrics requests. +| `redirect_http` | Redirect pages from HTTP to HTTPS, true/false. +| `sentry_dsn` | The address for sending Sentry crash reporting to. +| `sentry_enabled` | Enable reporting and logging with Sentry, true/false. +| `sentry_environment` | The environment for Sentry crash reporting. +| `status_uri` | The URL path for a status page, for example, `/@status`. +| `tls_max_version` | Specifies the maximum SSL/TLS version ("ssl3", "tls1.0", "tls1.1" or "tls1.2"). +| `tls_min_version` | Specifies the minimum SSL/TLS version ("ssl3", "tls1.0", "tls1.1" or "tls1.2"). +| `use_http2` | Enable HTTP2 support. +| **gitlab_pages['env'][]** | | +| `http_proxy` | Configure GitLab Pages to use an HTTP Proxy to mediate traffic between Pages and GitLab. Sets an environment variable `http_proxy` when starting Pages daemon. +| **gitlab_rails[]** | | +| `pages_domain_verification_cron_worker` | Schedule for verifying custom GitLab Pages domains. +| `pages_domain_ssl_renewal_cron_worker` | Schedule for obtaining and renewing SSL certificates through Let's Encrypt for GitLab Pages domains. +| `pages_domain_removal_cron_worker` | Schedule for removing unverified custom GitLab Pages domains. +| `pages_path` | The directory on disk where pages are stored, defaults to `GITLAB-RAILS/shared/pages`. +| **pages_nginx[]** | | +| `enable` | Include a virtual host `server{}` block for Pages inside NGINX. Needed for NGINX to proxy traffic back to the Pages daemon. Set to `false` if the Pages daemon should directly receive all requests, for example, when using [custom domains](index.md#custom-domains). + +--- + ## Advanced configuration In addition to the wildcard domains, you can also have the option to configure @@ -261,7 +323,7 @@ This setting is enabled by default. ### Let's Encrypt integration -> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/issues/28996) in GitLab 12.1. +> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/28996) in GitLab 12.1. [GitLab Pages' Let's Encrypt integration](../../user/project/pages/custom_domains_ssl_tls_certification/lets_encrypt_integration.md) allows users to add Let's Encrypt SSL certificates for GitLab Pages @@ -278,7 +340,7 @@ To enable it, you'll need to: ### Access control -> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/issues/33422) in GitLab 11.5. +> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/33422) in GitLab 11.5. GitLab Pages access control can be configured per-project, and allows access to a Pages site to be controlled based on a user's membership to that project. @@ -305,9 +367,13 @@ Pages access control is disabled by default. To enable it: 1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). 1. Users can now configure it in their [projects' settings](../../user/project/pages/pages_access_control.md). +NOTE: **Important:** +For multi-node setups, in order for this setting to be effective, it has to be applied +to all the App nodes as well as the Sidekiq nodes. + #### Disabling public access to all Pages websites -> [Introduced](https://gitlab.com/gitlab-org/gitlab/issues/32095) in GitLab 12.7. +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/32095) in GitLab 12.7. You can enforce [Access Control](#access-control) for all GitLab Pages websites hosted on your GitLab instance. By doing so, only logged-in users will have access to them. @@ -324,7 +390,7 @@ To do that: CAUTION: **Warning:** This action will not make all currently public web-sites private until they redeployed. This issue among others will be resolved by -[changing GitLab Pages configuration mechanism](https://gitlab.com/gitlab-org/gitlab-pages/issues/282). +[changing GitLab Pages configuration mechanism](https://gitlab.com/gitlab-org/gitlab-pages/-/issues/282). ### Running behind a proxy @@ -335,7 +401,7 @@ pages: 1. Configure in `/etc/gitlab/gitlab.rb`: ```ruby - gitlab_pages['http_proxy'] = 'http://example:8080' + gitlab_pages['env']['http_proxy'] = 'http://example:8080' ``` 1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. @@ -353,7 +419,7 @@ For installation from source this can be fixed by installing the custom Certific Authority (CA) in the system certificate store. For Omnibus, normally this would be fixed by [installing a custom CA in Omnibus GitLab](https://docs.gitlab.com/omnibus/settings/ssl.html#install-custom-public-certificates) -but a [bug](https://gitlab.com/gitlab-org/gitlab/issues/25411) is currently preventing +but a [bug](https://gitlab.com/gitlab-org/gitlab/-/issues/25411) is currently preventing that method from working. Use the following workaround: 1. Append your GitLab server TLS/SSL certificate to `/opt/gitlab/embedded/ssl/certs/cacert.pem` where `gitlab-domain-example.com` is your GitLab application URL @@ -435,7 +501,7 @@ The default is 100MB. ### Override maximum pages size per project or group **(PREMIUM ONLY)** -> [Introduced](https://gitlab.com/gitlab-org/gitlab/issues/16610) in GitLab 12.7. +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/16610) in GitLab 12.7. To override the global maximum pages size for a specific project: diff --git a/doc/administration/pages/source.md b/doc/administration/pages/source.md index 1bb3b86b419..d5b49bdf839 100644 --- a/doc/administration/pages/source.md +++ b/doc/administration/pages/source.md @@ -1,3 +1,9 @@ +--- +stage: Release +group: Release Management +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + # GitLab Pages administration for source installations >**Note:** @@ -266,7 +272,7 @@ world. Custom domains are supported, but no TLS. sudo ln -sf /etc/nginx/sites-{available,enabled}/gitlab-pages.conf ``` -1. Edit all GitLab related configs in `/etc/nginx/site-available/` and replace +1. Edit all GitLab related configurations in `/etc/nginx/site-available/` and replace `0.0.0.0` with `192.0.2.1`, where `192.0.2.1` the primary IP where GitLab listens to. 1. Restart NGINX @@ -335,7 +341,7 @@ world. Custom domains and TLS are supported. sudo ln -sf /etc/nginx/sites-{available,enabled}/gitlab-pages-ssl.conf ``` -1. Edit all GitLab related configs in `/etc/nginx/site-available/` and replace +1. Edit all GitLab related configurations in `/etc/nginx/site-available/` and replace `0.0.0.0` with `192.0.2.1`, where `192.0.2.1` the primary IP where GitLab listens to. 1. Restart NGINX @@ -346,7 +352,7 @@ world. Custom domains and TLS are supported. >**Note:** The following information applies only for installations from source. -Be extra careful when setting up the domain name in the NGINX config. You must +Be extra careful when setting up the domain name in the NGINX configuration. You must not remove the backslashes. If your GitLab Pages domain is `example.io`, replace: @@ -370,7 +376,7 @@ server_name ~^.*\.pages\.example\.io$; ## Access control -> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/issues/33422) in GitLab 11.5. +> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/33422) in GitLab 11.5. GitLab Pages access control can be configured per-project, and allows access to a Pages site to be controlled based on a user's membership to that project. diff --git a/doc/administration/postgresql/external.md b/doc/administration/postgresql/external.md new file mode 100644 index 00000000000..6e2bbc0aae1 --- /dev/null +++ b/doc/administration/postgresql/external.md @@ -0,0 +1,41 @@ +# Configure GitLab using an external PostgreSQL service + +If you're hosting GitLab on a cloud provider, you can optionally use a +managed service for PostgreSQL. For example, AWS offers a managed Relational +Database Service (RDS) that runs PostgreSQL. + +Alternatively, you may opt to manage your own PostgreSQL instance or cluster +separate from the Omnibus GitLab package. + +If you use a cloud-managed service, or provide your own PostgreSQL instance: + +1. Set up PostgreSQL according to the + [database requirements document](../../install/requirements.md#database). +1. Set up a `gitlab` username with a password of your choice. The `gitlab` user + needs privileges to create the `gitlabhq_production` database. +1. If you are using a cloud-managed service, you may need to grant additional + roles to your `gitlab` user: + - Amazon RDS requires the [`rds_superuser`](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Appendix.PostgreSQL.CommonDBATasks.html#Appendix.PostgreSQL.CommonDBATasks.Roles) role. + - Azure Database for PostgreSQL requires the [`azure_pg_admin`](https://docs.microsoft.com/en-us/azure/postgresql/howto-create-users#how-to-create-additional-admin-users-in-azure-database-for-postgresql) role. + +1. Configure the GitLab application servers with the appropriate connection details + for your external PostgreSQL service in your `/etc/gitlab/gitlab.rb` file: + + ```ruby + # Disable the bundled Omnibus provided PostgreSQL + postgresql['enable'] = false + + # PostgreSQL connection details + gitlab_rails['db_adapter'] = 'postgresql' + gitlab_rails['db_encoding'] = 'unicode' + gitlab_rails['db_host'] = '10.1.0.5' # IP/hostname of database server + gitlab_rails['db_password'] = 'DB password' + ``` + + For more information on GitLab HA setups, refer to [configuring GitLab for HA](../high_availability/gitlab.md). + +1. Reconfigure for the changes to take effect: + + ```shell + sudo gitlab-ctl reconfigure + ``` diff --git a/doc/administration/high_availability/img/pg_ha_architecture.png b/doc/administration/postgresql/img/pg_ha_architecture.png Binary files differindex ef870f652ae..ef870f652ae 100644 --- a/doc/administration/high_availability/img/pg_ha_architecture.png +++ b/doc/administration/postgresql/img/pg_ha_architecture.png diff --git a/doc/administration/postgresql/replication_and_failover.md b/doc/administration/postgresql/replication_and_failover.md new file mode 100644 index 00000000000..aa95b983d20 --- /dev/null +++ b/doc/administration/postgresql/replication_and_failover.md @@ -0,0 +1,1129 @@ +# PostgreSQL replication and failover with Omnibus GitLab **(PREMIUM ONLY)** + +> Important notes: +> +> - This document will focus only on configuration supported with [GitLab Premium](https://about.gitlab.com/pricing/), using the Omnibus GitLab package. +> - If you are a Community Edition or Starter user, consider using a cloud hosted solution. +> - This document will not cover installations from source. +> +> - If a setup with replication and failover is not what you were looking for, see the [database configuration document](https://docs.gitlab.com/omnibus/settings/database.html) +> for the Omnibus GitLab packages. +> +> Please read this document fully before attempting to configure PostgreSQL with +> replication and failover for GitLab. + +## Architecture + +The Omnibus GitLab recommended configuration for a PostgreSQL cluster with +replication and failover requires: + +- A minimum of three database nodes. +- A minimum of three `Consul` server nodes. +- A minimum of one `pgbouncer` service node, but it's recommended to have one + per database node. + - An internal load balancer (TCP) is required when there is more than one + `pgbouncer` service node. + + + +You also need to take into consideration the underlying network topology, making +sure you have redundant connectivity between all Database and GitLab instances +to avoid the network becoming a single point of failure. + +### Database node + +Each database node runs three services: + +`PostgreSQL` - The database itself. + +`repmgrd` - Communicates with other repmgrd services in the cluster and handles +failover when issues with the master server occurs. The failover procedure +consists of: + +- Selecting a new master for the cluster. +- Promoting the new node to master. +- Instructing remaining servers to follow the new master node. +- The old master node is automatically evicted from the cluster and should be + rejoined manually once recovered. + +`Consul` agent - Monitors the status of each node in the database cluster and +tracks its health in a service definition on the Consul cluster. + +### Consul server node + +The Consul server node runs the Consul server service. + +### PgBouncer node + +Each PgBouncer node runs two services: + +`PgBouncer` - The database connection pooler itself. + +`Consul` agent - Watches the status of the PostgreSQL service definition on the +Consul cluster. If that status changes, Consul runs a script which updates the +PgBouncer configuration to point to the new PostgreSQL master node and reloads +the PgBouncer service. + +### Connection flow + +Each service in the package comes with a set of [default ports](https://docs.gitlab.com/omnibus/package-information/defaults.html#ports). You may need to make specific firewall rules for the connections listed below: + +- Application servers connect to either PgBouncer directly via its [default port](https://docs.gitlab.com/omnibus/package-information/defaults.html#pgbouncer) or via a configured Internal Load Balancer (TCP) that serves multiple PgBouncers. +- PgBouncer connects to the primary database servers [PostgreSQL default port](https://docs.gitlab.com/omnibus/package-information/defaults.html#postgresql) +- Repmgr connects to the database servers [PostgreSQL default port](https://docs.gitlab.com/omnibus/package-information/defaults.html#postgresql) +- PostgreSQL secondaries connect to the primary database servers [PostgreSQL default port](https://docs.gitlab.com/omnibus/package-information/defaults.html#postgresql) +- Consul servers and agents connect to each others [Consul default ports](https://docs.gitlab.com/omnibus/package-information/defaults.html#consul) + +## Setting it up + +### Required information + +Before proceeding with configuration, you will need to collect all the necessary +information. + +#### Network information + +PostgreSQL does not listen on any network interface by default. It needs to know +which IP address to listen on in order to be accessible to other services. +Similarly, PostgreSQL access is controlled based on the network source. + +This is why you will need: + +- IP address of each nodes network interface. This can be set to `0.0.0.0` to + listen on all interfaces. It cannot be set to the loopback address `127.0.0.1`. +- Network Address. This can be in subnet (i.e. `192.168.0.0/255.255.255.0`) + or CIDR (i.e. `192.168.0.0/24`) form. + +#### Consul information + +When using default setup, minimum configuration requires: + +- `CONSUL_USERNAME`. Defaults to `gitlab-consul` +- `CONSUL_DATABASE_PASSWORD`. Password for the database user. +- `CONSUL_PASSWORD_HASH`. This is a hash generated out of Consul username/password pair. + Can be generated with: + + ```shell + sudo gitlab-ctl pg-password-md5 CONSUL_USERNAME + ``` + +- `CONSUL_SERVER_NODES`. The IP addresses or DNS records of the Consul server nodes. + +Few notes on the service itself: + +- The service runs under a system account, by default `gitlab-consul`. + - If you are using a different username, you will have to specify it. We + will refer to it with `CONSUL_USERNAME`, +- There will be a database user created with read only access to the repmgr + database +- Passwords will be stored in the following locations: + - `/etc/gitlab/gitlab.rb`: hashed + - `/var/opt/gitlab/pgbouncer/pg_auth`: hashed + - `/var/opt/gitlab/consul/.pgpass`: plaintext + +#### PostgreSQL information + +When configuring PostgreSQL, we will set `max_wal_senders` to one more than +the number of database nodes in the cluster. +This is used to prevent replication from using up all of the +available database connections. + +In this document we are assuming 3 database nodes, which makes this configuration: + +```ruby +postgresql['max_wal_senders'] = 4 +``` + +As previously mentioned, you'll have to prepare the network subnets that will +be allowed to authenticate with the database. +You'll also need to supply the IP addresses or DNS records of Consul +server nodes. + +We will need the following password information for the application's database user: + +- `POSTGRESQL_USERNAME`. Defaults to `gitlab` +- `POSTGRESQL_USER_PASSWORD`. The password for the database user +- `POSTGRESQL_PASSWORD_HASH`. This is a hash generated out of the username/password pair. + Can be generated with: + + ```shell + sudo gitlab-ctl pg-password-md5 POSTGRESQL_USERNAME + ``` + +#### PgBouncer information + +When using default setup, minimum configuration requires: + +- `PGBOUNCER_USERNAME`. Defaults to `pgbouncer` +- `PGBOUNCER_PASSWORD`. This is a password for PgBouncer service. +- `PGBOUNCER_PASSWORD_HASH`. This is a hash generated out of PgBouncer username/password pair. + Can be generated with: + + ```shell + sudo gitlab-ctl pg-password-md5 PGBOUNCER_USERNAME + ``` + +- `PGBOUNCER_NODE`, is the IP address or a FQDN of the node running PgBouncer. + +Few notes on the service itself: + +- The service runs as the same system account as the database + - In the package, this is by default `gitlab-psql` +- If you use a non-default user account for PgBouncer service (by default `pgbouncer`), you will have to specify this username. We will refer to this requirement with `PGBOUNCER_USERNAME`. +- The service will have a regular database user account generated for it + - This defaults to `repmgr` +- Passwords will be stored in the following locations: + - `/etc/gitlab/gitlab.rb`: hashed, and in plain text + - `/var/opt/gitlab/pgbouncer/pg_auth`: hashed + +#### Repmgr information + +When using default setup, you will only have to prepare the network subnets that will +be allowed to authenticate with the service. + +Few notes on the service itself: + +- The service runs under the same system account as the database + - In the package, this is by default `gitlab-psql` +- The service will have a superuser database user account generated for it + - This defaults to `gitlab_repmgr` + +### Installing Omnibus GitLab + +First, make sure to [download/install](https://about.gitlab.com/install/) +Omnibus GitLab **on each node**. + +Make sure you install the necessary dependencies from step 1, +add GitLab package repository from step 2. +When installing the GitLab package, do not supply `EXTERNAL_URL` value. + +### Configuring the Database nodes + +1. Make sure to [configure the Consul nodes](../high_availability/consul.md). +1. Make sure you collect [`CONSUL_SERVER_NODES`](#consul-information), [`PGBOUNCER_PASSWORD_HASH`](#pgbouncer-information), [`POSTGRESQL_PASSWORD_HASH`](#postgresql-information), the [number of db nodes](#postgresql-information), and the [network address](#network-information) before executing the next step. + +1. On the master database node, edit `/etc/gitlab/gitlab.rb` replacing values noted in the `# START user configuration` section: + + ```ruby + # Disable all components except PostgreSQL and Repmgr and Consul + roles ['postgres_role'] + + # PostgreSQL configuration + postgresql['listen_address'] = '0.0.0.0' + postgresql['hot_standby'] = 'on' + postgresql['wal_level'] = 'replica' + postgresql['shared_preload_libraries'] = 'repmgr_funcs' + + # Disable automatic database migrations + gitlab_rails['auto_migrate'] = false + + # Configure the Consul agent + consul['services'] = %w(postgresql) + + # START user configuration + # Please set the real values as explained in Required Information section + # + # Replace PGBOUNCER_PASSWORD_HASH with a generated md5 value + postgresql['pgbouncer_user_password'] = 'PGBOUNCER_PASSWORD_HASH' + # Replace POSTGRESQL_PASSWORD_HASH with a generated md5 value + postgresql['sql_user_password'] = 'POSTGRESQL_PASSWORD_HASH' + # Replace X with value of number of db nodes + 1 + postgresql['max_wal_senders'] = X + postgresql['max_replication_slots'] = X + + # Replace XXX.XXX.XXX.XXX/YY with Network Address + postgresql['trust_auth_cidr_addresses'] = %w(XXX.XXX.XXX.XXX/YY) + repmgr['trust_auth_cidr_addresses'] = %w(127.0.0.1/32 XXX.XXX.XXX.XXX/YY) + + # Replace placeholders: + # + # Y.Y.Y.Y consul1.gitlab.example.com Z.Z.Z.Z + # with the addresses gathered for CONSUL_SERVER_NODES + consul['configuration'] = { + retry_join: %w(Y.Y.Y.Y consul1.gitlab.example.com Z.Z.Z.Z) + } + # + # END user configuration + ``` + + > `postgres_role` was introduced with GitLab 10.3 + +1. On secondary nodes, add all the configuration specified above for primary node + to `/etc/gitlab/gitlab.rb`. In addition, append the following configuration + to inform `gitlab-ctl` that they are standby nodes initially and it need not + attempt to register them as primary node + + ```ruby + # Specify if a node should attempt to be master on initialization + repmgr['master_on_initialization'] = false + ``` + +1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. +1. [Enable Monitoring](#enable-monitoring) + +> Please note: +> +> - If you want your database to listen on a specific interface, change the configuration: +> `postgresql['listen_address'] = '0.0.0.0'`. +> - If your PgBouncer service runs under a different user account, +> you also need to specify: `postgresql['pgbouncer_user'] = PGBOUNCER_USERNAME` in +> your configuration. + +#### Enable Monitoring + +> [Introduced](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/3786) in GitLab 12.0. + +If you enable Monitoring, it must be enabled on **all** database servers. + +1. Create/edit `/etc/gitlab/gitlab.rb` and add the following configuration: + + ```ruby + # Enable service discovery for Prometheus + consul['monitoring_service_discovery'] = true + + # Set the network addresses that the exporters will listen on + node_exporter['listen_address'] = '0.0.0.0:9100' + postgres_exporter['listen_address'] = '0.0.0.0:9187' + ``` + +1. Run `sudo gitlab-ctl reconfigure` to compile the configuration. + +#### Database nodes post-configuration + +##### Primary node + +Select one node as a primary node. + +1. Open a database prompt: + + ```shell + gitlab-psql -d gitlabhq_production + ``` + +1. Enable the `pg_trgm` extension: + + ```shell + CREATE EXTENSION pg_trgm; + ``` + +1. Exit the database prompt by typing `\q` and Enter. + +1. Verify the cluster is initialized with one node: + + ```shell + gitlab-ctl repmgr cluster show + ``` + + The output should be similar to the following: + + ```plaintext + Role | Name | Upstream | Connection String + ----------+----------|----------|---------------------------------------- + * master | HOSTNAME | | host=HOSTNAME user=gitlab_repmgr dbname=gitlab_repmgr + ``` + +1. Note down the hostname or IP address in the connection string: `host=HOSTNAME`. We will + refer to the hostname in the next section as `MASTER_NODE_NAME`. If the value + is not an IP address, it will need to be a resolvable name (via DNS or + `/etc/hosts`) + +##### Secondary nodes + +1. Set up the repmgr standby: + + ```shell + gitlab-ctl repmgr standby setup MASTER_NODE_NAME + ``` + + Do note that this will remove the existing data on the node. The command + has a wait time. + + The output should be similar to the following: + + ```console + # gitlab-ctl repmgr standby setup MASTER_NODE_NAME + Doing this will delete the entire contents of /var/opt/gitlab/postgresql/data + If this is not what you want, hit Ctrl-C now to exit + To skip waiting, rerun with the -w option + Sleeping for 30 seconds + Stopping the database + Removing the data + Cloning the data + Starting the database + Registering the node with the cluster + ok: run: repmgrd: (pid 19068) 0s + ``` + +1. Verify the node now appears in the cluster: + + ```shell + gitlab-ctl repmgr cluster show + ``` + + The output should be similar to the following: + + ```plaintext + Role | Name | Upstream | Connection String + ----------+---------|-----------|------------------------------------------------ + * master | MASTER | | host=MASTER_NODE_NAME user=gitlab_repmgr dbname=gitlab_repmgr + standby | STANDBY | MASTER | host=STANDBY_HOSTNAME user=gitlab_repmgr dbname=gitlab_repmgr + ``` + +Repeat the above steps on all secondary nodes. + +#### Database checkpoint + +Before moving on, make sure the databases are configured correctly. Run the +following command on the **primary** node to verify that replication is working +properly: + +```shell +gitlab-ctl repmgr cluster show +``` + +The output should be similar to: + +```plaintext +Role | Name | Upstream | Connection String +----------+--------------|--------------|-------------------------------------------------------------------- +* master | MASTER | | host=MASTER port=5432 user=gitlab_repmgr dbname=gitlab_repmgr + standby | STANDBY | MASTER | host=STANDBY port=5432 user=gitlab_repmgr dbname=gitlab_repmgr +``` + +If the 'Role' column for any node says "FAILED", check the +[Troubleshooting section](#troubleshooting) before proceeding. + +Also, check that the check master command works successfully on each node: + +```shell +su - gitlab-consul +gitlab-ctl repmgr-check-master || echo 'This node is a standby repmgr node' +``` + +This command relies on exit codes to tell Consul whether a particular node is a master +or secondary. The most important thing here is that this command does not produce errors. +If there are errors it's most likely due to incorrect `gitlab-consul` database user permissions. +Check the [Troubleshooting section](#troubleshooting) before proceeding. + +### Configuring the PgBouncer node + +1. Make sure you collect [`CONSUL_SERVER_NODES`](#consul-information), [`CONSUL_PASSWORD_HASH`](#consul-information), and [`PGBOUNCER_PASSWORD_HASH`](#pgbouncer-information) before executing the next step. + +1. One each node, edit the `/etc/gitlab/gitlab.rb` config file and replace values noted in the `# START user configuration` section as below: + + ```ruby + # Disable all components except PgBouncer and Consul agent + roles ['pgbouncer_role'] + + # Configure PgBouncer + pgbouncer['admin_users'] = %w(pgbouncer gitlab-consul) + + # Configure Consul agent + consul['watchers'] = %w(postgresql) + + # START user configuration + # Please set the real values as explained in Required Information section + # Replace CONSUL_PASSWORD_HASH with with a generated md5 value + # Replace PGBOUNCER_PASSWORD_HASH with with a generated md5 value + pgbouncer['users'] = { + 'gitlab-consul': { + password: 'CONSUL_PASSWORD_HASH' + }, + 'pgbouncer': { + password: 'PGBOUNCER_PASSWORD_HASH' + } + } + # Replace placeholders: + # + # Y.Y.Y.Y consul1.gitlab.example.com Z.Z.Z.Z + # with the addresses gathered for CONSUL_SERVER_NODES + consul['configuration'] = { + retry_join: %w(Y.Y.Y.Y consul1.gitlab.example.com Z.Z.Z.Z) + } + # + # END user configuration + ``` + + NOTE: **Note:** + `pgbouncer_role` was introduced with GitLab 10.3. + +1. Run `gitlab-ctl reconfigure` + +1. Create a `.pgpass` file so Consul is able to + reload PgBouncer. Enter the `PGBOUNCER_PASSWORD` twice when asked: + + ```shell + gitlab-ctl write-pgpass --host 127.0.0.1 --database pgbouncer --user pgbouncer --hostuser gitlab-consul + ``` + +1. [Enable monitoring](../high_availability/pgbouncer.md#enable-monitoring) + +#### PgBouncer Checkpoint + +1. Ensure each node is talking to the current master: + + ```shell + gitlab-ctl pgb-console # You will be prompted for PGBOUNCER_PASSWORD + ``` + + If there is an error `psql: ERROR: Auth failed` after typing in the + password, ensure you previously generated the MD5 password hashes with the correct + format. The correct format is to concatenate the password and the username: + `PASSWORDUSERNAME`. For example, `Sup3rS3cr3tpgbouncer` would be the text + needed to generate an MD5 password hash for the `pgbouncer` user. + +1. Once the console prompt is available, run the following queries: + + ```shell + show databases ; show clients ; + ``` + + The output should be similar to the following: + + ```plaintext + name | host | port | database | force_user | pool_size | reserve_pool | pool_mode | max_connections | current_connections + ---------------------+-------------+------+---------------------+------------+-----------+--------------+-----------+-----------------+--------------------- + gitlabhq_production | MASTER_HOST | 5432 | gitlabhq_production | | 20 | 0 | | 0 | 0 + pgbouncer | | 6432 | pgbouncer | pgbouncer | 2 | 0 | statement | 0 | 0 + (2 rows) + + type | user | database | state | addr | port | local_addr | local_port | connect_time | request_time | ptr | link | remote_pid | tls + ------+-----------+---------------------+---------+----------------+-------+------------+------------+---------------------+---------------------+-----------+------+------------+----- + C | pgbouncer | pgbouncer | active | 127.0.0.1 | 56846 | 127.0.0.1 | 6432 | 2017-08-21 18:09:59 | 2017-08-21 18:10:48 | 0x22b3880 | | 0 | + (2 rows) + ``` + +#### Configure the internal load balancer + +If you're running more than one PgBouncer node as recommended, then at this time you'll need to set up a TCP internal load balancer to serve each correctly. This can be done with any reputable TCP load balancer. + +As an example here's how you could do it with [HAProxy](https://www.haproxy.org/): + +```plaintext +global + log /dev/log local0 + log localhost local1 notice + log stdout format raw local0 + +defaults + log global + default-server inter 10s fall 3 rise 2 + balance leastconn + +frontend internal-pgbouncer-tcp-in + bind *:6432 + mode tcp + option tcplog + + default_backend pgbouncer + +backend pgbouncer + mode tcp + option tcp-check + + server pgbouncer1 <ip>:6432 check + server pgbouncer2 <ip>:6432 check + server pgbouncer3 <ip>:6432 check +``` + +Refer to your preferred Load Balancer's documentation for further guidance. + +### Configuring the Application nodes + +These will be the nodes running the `gitlab-rails` service. You may have other +attributes set, but the following need to be set. + +1. Edit `/etc/gitlab/gitlab.rb`: + + ```ruby + # Disable PostgreSQL on the application node + postgresql['enable'] = false + + gitlab_rails['db_host'] = 'PGBOUNCER_NODE' or 'INTERNAL_LOAD_BALANCER' + gitlab_rails['db_port'] = 6432 + gitlab_rails['db_password'] = 'POSTGRESQL_USER_PASSWORD' + gitlab_rails['auto_migrate'] = false + ``` + +1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. + +#### Application node post-configuration + +Ensure that all migrations ran: + +```shell +gitlab-rake gitlab:db:configure +``` + +> **Note**: If you encounter a `rake aborted!` error stating that PgBouncer is failing to connect to +PostgreSQL it may be that your PgBouncer node's IP address is missing from +PostgreSQL's `trust_auth_cidr_addresses` in `gitlab.rb` on your database nodes. See +[PgBouncer error `ERROR: pgbouncer cannot connect to server`](#pgbouncer-error-error-pgbouncer-cannot-connect-to-server) +in the Troubleshooting section before proceeding. + +### Ensure GitLab is running + +At this point, your GitLab instance should be up and running. Verify you are +able to login, and create issues and merge requests. If you have troubles check +the [Troubleshooting section](#troubleshooting). + +## Example configuration + +Here we'll show you some fully expanded example configurations. + +### Example recommended setup + +This example uses 3 Consul servers, 3 PgBouncer servers (with associated internal load balancer), +3 PostgreSQL servers, and 1 application node. + +We start with all servers on the same 10.6.0.0/16 private network range, they +can connect to each freely other on those addresses. + +Here is a list and description of each machine and the assigned IP: + +- `10.6.0.11`: Consul 1 +- `10.6.0.12`: Consul 2 +- `10.6.0.13`: Consul 3 +- `10.6.0.20`: Internal Load Balancer +- `10.6.0.21`: PgBouncer 1 +- `10.6.0.22`: PgBouncer 2 +- `10.6.0.23`: PgBouncer 3 +- `10.6.0.31`: PostgreSQL master +- `10.6.0.32`: PostgreSQL secondary +- `10.6.0.33`: PostgreSQL secondary +- `10.6.0.41`: GitLab application + +All passwords are set to `toomanysecrets`, please do not use this password or derived hashes and the `external_url` for GitLab is `http://gitlab.example.com`. + +Please note that after the initial configuration, if a failover occurs, the PostgresSQL master will change to one of the available secondaries until it is failed back. + +#### Example recommended setup for Consul servers + +On each server edit `/etc/gitlab/gitlab.rb`: + +```ruby +# Disable all components except Consul +roles ['consul_role'] + +consul['configuration'] = { + server: true, + retry_join: %w(10.6.0.11 10.6.0.12 10.6.0.13) +} +consul['monitoring_service_discovery'] = true +``` + +[Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. + +#### Example recommended setup for PgBouncer servers + +On each server edit `/etc/gitlab/gitlab.rb`: + +```ruby +# Disable all components except Pgbouncer and Consul agent +roles ['pgbouncer_role'] + +# Configure PgBouncer +pgbouncer['admin_users'] = %w(pgbouncer gitlab-consul) + +pgbouncer['users'] = { + 'gitlab-consul': { + password: '5e0e3263571e3704ad655076301d6ebe' + }, + 'pgbouncer': { + password: '771a8625958a529132abe6f1a4acb19c' + } +} + +consul['watchers'] = %w(postgresql) +consul['enable'] = true +consul['configuration'] = { + retry_join: %w(10.6.0.11 10.6.0.12 10.6.0.13) +} +consul['monitoring_service_discovery'] = true +``` + +[Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. + +#### Internal load balancer setup + +An internal load balancer (TCP) is then required to be setup to serve each PgBouncer node (in this example on the IP of `10.6.0.20`). An example of how to do this can be found in the [PgBouncer Configure Internal Load Balancer](#configure-the-internal-load-balancer) section. + +#### Example recommended setup for PostgreSQL servers + +##### Primary node + +On primary node edit `/etc/gitlab/gitlab.rb`: + +```ruby +# Disable all components except PostgreSQL and Repmgr and Consul +roles ['postgres_role'] + +# PostgreSQL configuration +postgresql['listen_address'] = '0.0.0.0' +postgresql['hot_standby'] = 'on' +postgresql['wal_level'] = 'replica' +postgresql['shared_preload_libraries'] = 'repmgr_funcs' + +# Disable automatic database migrations +gitlab_rails['auto_migrate'] = false + +postgresql['pgbouncer_user_password'] = '771a8625958a529132abe6f1a4acb19c' +postgresql['sql_user_password'] = '450409b85a0223a214b5fb1484f34d0f' +postgresql['max_wal_senders'] = 4 + +postgresql['trust_auth_cidr_addresses'] = %w(10.6.0.0/16) +repmgr['trust_auth_cidr_addresses'] = %w(10.6.0.0/16) + +# Configure the Consul agent +consul['services'] = %w(postgresql) +consul['enable'] = true +consul['configuration'] = { + retry_join: %w(10.6.0.11 10.6.0.12 10.6.0.13) +} +consul['monitoring_service_discovery'] = true +``` + +[Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. + +##### Secondary nodes + +On secondary nodes, edit `/etc/gitlab/gitlab.rb` and add all the configuration +added to primary node, noted above. In addition, append the following +configuration: + +```ruby +# Specify if a node should attempt to be master on initialization +repmgr['master_on_initialization'] = false +``` + +[Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. + +###### Example recommended setup for application server + +On the server edit `/etc/gitlab/gitlab.rb`: + +```ruby +external_url 'http://gitlab.example.com' + +gitlab_rails['db_host'] = '10.6.0.20' # Internal Load Balancer for PgBouncer nodes +gitlab_rails['db_port'] = 6432 +gitlab_rails['db_password'] = 'toomanysecrets' +gitlab_rails['auto_migrate'] = false + +postgresql['enable'] = false +pgbouncer['enable'] = false +consul['enable'] = true + +# Configure Consul agent +consul['watchers'] = %w(postgresql) + +pgbouncer['users'] = { + 'gitlab-consul': { + password: '5e0e3263571e3704ad655076301d6ebe' + }, + 'pgbouncer': { + password: '771a8625958a529132abe6f1a4acb19c' + } +} + +consul['configuration'] = { + retry_join: %w(10.6.0.11 10.6.0.12 10.6.0.13) +} +``` + +[Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. + +#### Example recommended setup manual steps + +After deploying the configuration follow these steps: + +1. On `10.6.0.31`, our primary database + + Enable the `pg_trgm` extension + + ```shell + gitlab-psql -d gitlabhq_production + ``` + + ```shell + CREATE EXTENSION pg_trgm; + ``` + +1. On `10.6.0.32`, our first standby database + + Make this node a standby of the primary + + ```shell + gitlab-ctl repmgr standby setup 10.6.0.21 + ``` + +1. On `10.6.0.33`, our second standby database + + Make this node a standby of the primary + + ```shell + gitlab-ctl repmgr standby setup 10.6.0.21 + ``` + +1. On `10.6.0.41`, our application server + + Set `gitlab-consul` user's PgBouncer password to `toomanysecrets` + + ```shell + gitlab-ctl write-pgpass --host 127.0.0.1 --database pgbouncer --user pgbouncer --hostuser gitlab-consul + ``` + + Run database migrations + + ```shell + gitlab-rake gitlab:db:configure + ``` + +### Example minimal setup + +This example uses 3 PostgreSQL servers, and 1 application node (with PgBouncer setup alongside). + +It differs from the [recommended setup](#example-recommended-setup) by moving the Consul servers into the same servers we use for PostgreSQL. +The trade-off is between reducing server counts, against the increased operational complexity of needing to deal with PostgreSQL [failover](#failover-procedure) and [restore](#restore-procedure) procedures in addition to [Consul outage recovery](../high_availability/consul.md#outage-recovery) on the same set of machines. + +In this example we start with all servers on the same 10.6.0.0/16 private network range, they can connect to each freely other on those addresses. + +Here is a list and description of each machine and the assigned IP: + +- `10.6.0.21`: PostgreSQL master +- `10.6.0.22`: PostgreSQL secondary +- `10.6.0.23`: PostgreSQL secondary +- `10.6.0.31`: GitLab application + +All passwords are set to `toomanysecrets`, please do not use this password or derived hashes. + +The `external_url` for GitLab is `http://gitlab.example.com` + +Please note that after the initial configuration, if a failover occurs, the PostgresSQL master will change to one of the available secondaries until it is failed back. + +#### Example minimal configuration for database servers + +##### Primary node + +On primary database node edit `/etc/gitlab/gitlab.rb`: + +```ruby +# Disable all components except PostgreSQL, Repmgr, and Consul +roles ['postgres_role'] + +# PostgreSQL configuration +postgresql['listen_address'] = '0.0.0.0' +postgresql['hot_standby'] = 'on' +postgresql['wal_level'] = 'replica' +postgresql['shared_preload_libraries'] = 'repmgr_funcs' + +# Disable automatic database migrations +gitlab_rails['auto_migrate'] = false + +# Configure the Consul agent +consul['services'] = %w(postgresql) + +postgresql['pgbouncer_user_password'] = '771a8625958a529132abe6f1a4acb19c' +postgresql['sql_user_password'] = '450409b85a0223a214b5fb1484f34d0f' +postgresql['max_wal_senders'] = 4 + +postgresql['trust_auth_cidr_addresses'] = %w(10.6.0.0/16) +repmgr['trust_auth_cidr_addresses'] = %w(10.6.0.0/16) + +consul['configuration'] = { + server: true, + retry_join: %w(10.6.0.21 10.6.0.22 10.6.0.23) +} +``` + +[Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. + +##### Secondary nodes + +On secondary nodes, edit `/etc/gitlab/gitlab.rb` and add all the information added +to primary node, noted above. In addition, append the following configuration + +```ruby +# Specify if a node should attempt to be master on initialization +repmgr['master_on_initialization'] = false +``` + +#### Example minimal configuration for application server + +On the server edit `/etc/gitlab/gitlab.rb`: + +```ruby +external_url 'http://gitlab.example.com' + +gitlab_rails['db_host'] = '127.0.0.1' +gitlab_rails['db_port'] = 6432 +gitlab_rails['db_password'] = 'toomanysecrets' +gitlab_rails['auto_migrate'] = false + +postgresql['enable'] = false +pgbouncer['enable'] = true +consul['enable'] = true + +# Configure PgBouncer +pgbouncer['admin_users'] = %w(pgbouncer gitlab-consul) + +# Configure Consul agent +consul['watchers'] = %w(postgresql) + +pgbouncer['users'] = { + 'gitlab-consul': { + password: '5e0e3263571e3704ad655076301d6ebe' + }, + 'pgbouncer': { + password: '771a8625958a529132abe6f1a4acb19c' + } +} + +consul['configuration'] = { + retry_join: %w(10.6.0.21 10.6.0.22 10.6.0.23) +} +``` + +[Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. + +#### Example minimal setup manual steps + +The manual steps for this configuration are the same as for the [example recommended setup](#example-recommended-setup-manual-steps). + +### Failover procedure + +By default, if the master database fails, `repmgrd` should promote one of the +standby nodes to master automatically, and Consul will update PgBouncer with +the new master. + +If you need to failover manually, you have two options: + +**Shutdown the current master database** + +Run: + +```shell +gitlab-ctl stop postgresql +``` + +The automated failover process will see this and failover to one of the +standby nodes. + +**Or perform a manual failover** + +1. Ensure the old master node is not still active. +1. Login to the server that should become the new master and run: + + ```shell + gitlab-ctl repmgr standby promote + ``` + +1. If there are any other standby servers in the cluster, have them follow + the new master server: + + ```shell + gitlab-ctl repmgr standby follow NEW_MASTER + ``` + +### Restore procedure + +If a node fails, it can be removed from the cluster, or added back as a standby +after it has been restored to service. + +#### Remove a standby from the cluster + + From any other node in the cluster, run: + + ```shell + gitlab-ctl repmgr standby unregister --node=X + ``` + + where X is the value of node in `repmgr.conf` on the old server. + + To find this, you can use: + + ```shell + awk -F = '$1 == "node" { print $2 }' /var/opt/gitlab/postgresql/repmgr.conf + ``` + + It will output something like: + + ```plaintext + 959789412 + ``` + + Then you will use this ID to unregister the node: + + ```shell + gitlab-ctl repmgr standby unregister --node=959789412 + ``` + +#### Add a node as a standby server + + From the standby node, run: + + ```shell + gitlab-ctl repmgr standby follow NEW_MASTER + gitlab-ctl restart repmgrd + ``` + + CAUTION: **Warning:** When the server is brought back online, and before + you switch it to a standby node, repmgr will report that there are two masters. + If there are any clients that are still attempting to write to the old master, + this will cause a split, and the old master will need to be resynced from + scratch by performing a `gitlab-ctl repmgr standby setup NEW_MASTER`. + +#### Add a failed master back into the cluster as a standby node + + Once `repmgrd` and PostgreSQL are running, the node will need to follow the new + as a standby node. + + ```shell + gitlab-ctl repmgr standby follow NEW_MASTER + ``` + + Once the node is following the new master as a standby, the node needs to be + [unregistered from the cluster on the new master node](#remove-a-standby-from-the-cluster). + + Once the old master node has been unregistered from the cluster, it will need + to be setup as a new standby: + + ```shell + gitlab-ctl repmgr standby setup NEW_MASTER + ``` + + Failure to unregister and read the old master node can lead to subsequent failovers + not working. + +### Alternate configurations + +#### Database authorization + +By default, we give any host on the database network the permission to perform +repmgr operations using PostgreSQL's `trust` method. If you do not want this +level of trust, there are alternatives. + +You can trust only the specific nodes that will be database clusters, or you +can require md5 authentication. + +#### Trust specific addresses + +If you know the IP address, or FQDN of all database and PgBouncer nodes in the +cluster, you can trust only those nodes. + +In `/etc/gitlab/gitlab.rb` on all of the database nodes, set +`repmgr['trust_auth_cidr_addresses']` to an array of strings containing all of +the addresses. + +If setting to a node's FQDN, they must have a corresponding PTR record in DNS. +If setting to a node's IP address, specify it as `XXX.XXX.XXX.XXX/32`. + +For example: + +```ruby +repmgr['trust_auth_cidr_addresses'] = %w(192.168.1.44/32 db2.example.com) +``` + +#### MD5 Authentication + +If you are running on an untrusted network, repmgr can use md5 authentication +with a [`.pgpass` file](https://www.postgresql.org/docs/11/libpq-pgpass.html) +to authenticate. + +You can specify by IP address, FQDN, or by subnet, using the same format as in +the previous section: + +1. On the current master node, create a password for the `gitlab` and + `gitlab_repmgr` user: + + ```shell + gitlab-psql -d template1 + template1=# \password gitlab_repmgr + Enter password: **** + Confirm password: **** + template1=# \password gitlab + ``` + +1. On each database node: + + 1. Edit `/etc/gitlab/gitlab.rb`: + 1. Ensure `repmgr['trust_auth_cidr_addresses']` is **not** set + 1. Set `postgresql['md5_auth_cidr_addresses']` to the desired value + 1. Set `postgresql['sql_replication_user'] = 'gitlab_repmgr'` + 1. Reconfigure with `gitlab-ctl reconfigure` + 1. Restart PostgreSQL with `gitlab-ctl restart postgresql` + + 1. Create a `.pgpass` file. Enter the `gitlab_repmgr` password twice to + when asked: + + ```shell + gitlab-ctl write-pgpass --user gitlab_repmgr --hostuser gitlab-psql --database '*' + ``` + +1. On each PgBouncer node, edit `/etc/gitlab/gitlab.rb`: + 1. Ensure `gitlab_rails['db_password']` is set to the plaintext password for + the `gitlab` database user + 1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect + +## Troubleshooting + +### Consul and PostgreSQL changes not taking effect + +Due to the potential impacts, `gitlab-ctl reconfigure` only reloads Consul and PostgreSQL, it will not restart the services. However, not all changes can be activated by reloading. + +To restart either service, run `gitlab-ctl restart SERVICE` + +For PostgreSQL, it is usually safe to restart the master node by default. Automatic failover defaults to a 1 minute timeout. Provided the database returns before then, nothing else needs to be done. To be safe, you can stop `repmgrd` on the standby nodes first with `gitlab-ctl stop repmgrd`, then start afterwards with `gitlab-ctl start repmgrd`. + +On the Consul server nodes, it is important to restart the Consul service in a controlled fashion. Read our [Consul documentation](../high_availability/consul.md#restarting-the-server-cluster) for instructions on how to restart the service. + +### `gitlab-ctl repmgr-check-master` command produces errors + +If this command displays errors about database permissions it is likely that something failed during +install, resulting in the `gitlab-consul` database user getting incorrect permissions. Follow these +steps to fix the problem: + +1. On the master database node, connect to the database prompt - `gitlab-psql -d template1` +1. Delete the `gitlab-consul` user - `DROP USER "gitlab-consul";` +1. Exit the database prompt - `\q` +1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) and the user will be re-added with the proper permissions. +1. Change to the `gitlab-consul` user - `su - gitlab-consul` +1. Try the check command again - `gitlab-ctl repmgr-check-master`. + +Now there should not be errors. If errors still occur then there is another problem. + +### PgBouncer error `ERROR: pgbouncer cannot connect to server` + +You may get this error when running `gitlab-rake gitlab:db:configure` or you +may see the error in the PgBouncer log file. + +```plaintext +PG::ConnectionBad: ERROR: pgbouncer cannot connect to server +``` + +The problem may be that your PgBouncer node's IP address is not included in the +`trust_auth_cidr_addresses` setting in `/etc/gitlab/gitlab.rb` on the database nodes. + +You can confirm that this is the issue by checking the PostgreSQL log on the master +database node. If you see the following error then `trust_auth_cidr_addresses` +is the problem. + +```plaintext +2018-03-29_13:59:12.11776 FATAL: no pg_hba.conf entry for host "123.123.123.123", user "pgbouncer", database "gitlabhq_production", SSL off +``` + +To fix the problem, add the IP address to `/etc/gitlab/gitlab.rb`. + +```ruby +postgresql['trust_auth_cidr_addresses'] = %w(123.123.123.123/32 <other_cidrs>) +``` + +[Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. + +### Issues with other components + +If you're running into an issue with a component not outlined here, be sure to check the troubleshooting section of their specific documentation page. + +- [Consul](../high_availability/consul.md#troubleshooting) +- [PostgreSQL](https://docs.gitlab.com/omnibus/settings/database.html#troubleshooting) +- [GitLab application](../high_availability/gitlab.md#troubleshooting) diff --git a/doc/administration/postgresql/standalone.md b/doc/administration/postgresql/standalone.md new file mode 100644 index 00000000000..3e7826ce009 --- /dev/null +++ b/doc/administration/postgresql/standalone.md @@ -0,0 +1,65 @@ +# Standalone PostgreSQL using Omnibus GitLab **(CORE ONLY)** + +If you wish to have your database service hosted separately from your GitLab +application server(s), you can do this using the PostgreSQL binaries packaged +together with Omnibus GitLab. This is recommended as part of our +[reference architecture for up to 2,000 users](../reference_architectures/2k_users.md). + +## Setting it up + +1. SSH into the PostgreSQL server. +1. [Download/install](https://about.gitlab.com/install/) the Omnibus GitLab + package you want using **steps 1 and 2** from the GitLab downloads page. + - Do not complete any other steps on the download page. +1. Generate a password hash for PostgreSQL. This assumes you will use the default + username of `gitlab` (recommended). The command will request a password + and confirmation. Use the value that is output by this command in the next + step as the value of `POSTGRESQL_PASSWORD_HASH`. + + ```shell + sudo gitlab-ctl pg-password-md5 gitlab + ``` + +1. Edit `/etc/gitlab/gitlab.rb` and add the contents below, updating placeholder + values appropriately. + + - `POSTGRESQL_PASSWORD_HASH` - The value output from the previous step + - `APPLICATION_SERVER_IP_BLOCKS` - A space delimited list of IP subnets or IP + addresses of the GitLab application servers that will connect to the + database. Example: `%w(123.123.123.123/32 123.123.123.234/32)` + + ```ruby + # Disable all components except PostgreSQL + roles ['postgres_role'] + repmgr['enable'] = false + consul['enable'] = false + prometheus['enable'] = false + alertmanager['enable'] = false + pgbouncer_exporter['enable'] = false + redis_exporter['enable'] = false + gitlab_exporter['enable'] = false + + postgresql['listen_address'] = '0.0.0.0' + postgresql['port'] = 5432 + + # Replace POSTGRESQL_PASSWORD_HASH with a generated md5 value + postgresql['sql_user_password'] = 'POSTGRESQL_PASSWORD_HASH' + + # Replace XXX.XXX.XXX.XXX/YY with Network Address + # ???? + postgresql['trust_auth_cidr_addresses'] = %w(APPLICATION_SERVER_IP_BLOCKS) + + # Disable automatic database migrations + gitlab_rails['auto_migrate'] = false + ``` + + NOTE: **Note:** The role `postgres_role` was introduced with GitLab 10.3 + +1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. +1. Note the PostgreSQL node's IP address or hostname, port, and + plain text password. These will be necessary when configuring the GitLab + application servers later. +1. [Enable monitoring](replication_and_failover.md#enable-monitoring) + +Advanced configuration options are supported and can be added if +needed. diff --git a/doc/administration/raketasks/github_import.md b/doc/administration/raketasks/github_import.md index 83a3d2c8884..a46a2b34687 100644 --- a/doc/administration/raketasks/github_import.md +++ b/doc/administration/raketasks/github_import.md @@ -12,6 +12,11 @@ Bear in mind that the syntax is very specific. Remove any spaces within the argu before/after the brackets. Also, some shells (for example, `zsh`) can interpret the open/close brackets (`[]`) separately. You may need to either escape the brackets or use double quotes. +## Caveats + +If the GitHub [rate limit](https://developer.github.com/v3/#rate-limiting) is reached while importing, +the importing process will wait (`sleep()`) until it can continue importing. + ## Importing multiple projects To import a project from the list of your GitHub projects available: diff --git a/doc/administration/raketasks/ldap.md b/doc/administration/raketasks/ldap.md index 2871a9235a3..d168e3d568c 100644 --- a/doc/administration/raketasks/ldap.md +++ b/doc/administration/raketasks/ldap.md @@ -32,13 +32,13 @@ rake gitlab:ldap:check[50] > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/14735) in [GitLab Starter](https://about.gitlab.com/pricing/) 12.2. -The following task will run a [group sync](../auth/ldap-ee.md#group-sync) immediately. This is valuable +The following task will run a [group sync](../auth/ldap/index.md#group-sync-starter-only) immediately. This is valuable when you'd like to update all configured group memberships against LDAP without waiting for the next scheduled group sync to be run. NOTE: **NOTE:** If you'd like to change the frequency at which a group sync is performed, -[adjust the cron schedule](../auth/ldap-ee.md#adjusting-ldap-group-sync-schedule) +[adjust the cron schedule](../auth/ldap/index.md#adjusting-ldap-group-sync-schedule-starter-only) instead. **Omnibus Installation** diff --git a/doc/administration/raketasks/maintenance.md b/doc/administration/raketasks/maintenance.md index eee68c0da0a..78094f00a43 100644 --- a/doc/administration/raketasks/maintenance.md +++ b/doc/administration/raketasks/maintenance.md @@ -51,6 +51,40 @@ Hooks: /home/git/gitlab-shell/hooks/ Git: /usr/bin/git ``` +## Show GitLab license information **(STARTER ONLY)** + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/20501) in GitLab Starter 12.6. + +This command shows information about your [GitLab license](../../user/admin_area/license.md) and +how many seats are used. It is only available on GitLab Enterprise +installations: a license cannot be installed into GitLab Community Edition. + +These may be useful when raising tickets with Support, or for programmatically +checking your license parameters. + +**Omnibus Installation** + +```shell +sudo gitlab-rake gitlab:license:info +``` + +**Source Installation** + +```shell +bundle exec rake gitlab:license:info RAILS_ENV=production +``` + +Example output: + +```plaintext +Today's Date: 2020-02-29 +Current User Count: 30 +Max Historical Count: 30 +Max Users in License: 40 +License valid from: 2019-11-29 to 2020-11-28 +Email associated with license: user@example.com +``` + ## Check GitLab configuration The `gitlab:check` Rake task runs the following Rake tasks: @@ -62,7 +96,7 @@ The `gitlab:check` Rake task runs the following Rake tasks: It will check that each component was set up according to the installation guide and suggest fixes for issues found. This command must be run from your application server and will not work correctly on -component servers like [Gitaly](../gitaly/index.md#running-gitaly-on-its-own-server). +component servers like [Gitaly](../gitaly/index.md#run-gitaly-on-its-own-server). You may also have a look at our troubleshooting guides for: @@ -265,6 +299,20 @@ database: gitlabhq_production up migration_id migration_name ``` +## Run incomplete database migrations + +Database migrations can be stuck in an incomplete state. That is, they'll have a `down` +status in the output of the `sudo gitlab-rake db:migrate:status` command. + +To complete these migrations, use the following Rake task: + +```shell +sudo gitlab-rake db:migrate +``` + +After the command completes, run `sudo gitlab-rake db:migrate:status` to check if all +migrations are completed (have an `up` status). + ## Import common metrics Sometimes you may need to re-import the common metrics that power the Metrics dashboards. diff --git a/doc/administration/raketasks/praefect.md b/doc/administration/raketasks/praefect.md index c3dadb6bc30..c48e23df77a 100644 --- a/doc/administration/raketasks/praefect.md +++ b/doc/administration/raketasks/praefect.md @@ -7,7 +7,7 @@ Prints out checksums of the repository of a given project_id on the primary as well as secondary internal Gitaly nodes. NOTE: **Note:** -This only is relevant and works for projects that have been created on a praefect storage. See the [Praefect Documentation](../gitaly/praefect.md) for configuring Praefect. +This only is relevant and works for projects that have been created on a Praefect storage. See the [Praefect Documentation](../gitaly/praefect.md) for configuring Praefect. **Omnibus Installation** diff --git a/doc/administration/raketasks/project_import_export.md b/doc/administration/raketasks/project_import_export.md index 2ab8b13e29e..2e65e889c90 100644 --- a/doc/administration/raketasks/project_import_export.md +++ b/doc/administration/raketasks/project_import_export.md @@ -1,6 +1,6 @@ # Project import/export administration **(CORE ONLY)** -> - [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/issues/3050) in GitLab 8.9. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/3050) in GitLab 8.9. > - From GitLab 11.3, import/export can use object storage automatically. GitLab provides Rake tasks relating to project import and export. For more information, see: diff --git a/doc/administration/raketasks/storage.md b/doc/administration/raketasks/storage.md index 30f50c24138..74fd2c2ebb6 100644 --- a/doc/administration/raketasks/storage.md +++ b/doc/administration/raketasks/storage.md @@ -104,7 +104,7 @@ You can monitor the progress in the **{admin}** **Admin Area > Monitoring > Back There is a specific queue you can watch to see how long it will take to finish: `hashed_storage:hashed_storage_project_migrate`. -After it reaches zero, you can confirm every project has been migrated by running the commands bellow. +After it reaches zero, you can confirm every project has been migrated by running the commands below. If you find it necessary, you can run this migration script again to schedule missing projects. Any error or warning will be logged in Sidekiq's log file. diff --git a/doc/administration/raketasks/uploads/migrate.md b/doc/administration/raketasks/uploads/migrate.md index d58b802b024..b164c4e744d 100644 --- a/doc/administration/raketasks/uploads/migrate.md +++ b/doc/administration/raketasks/uploads/migrate.md @@ -128,7 +128,7 @@ CAUTION: **Warning:** **Extended downtime is required** so no new files are created in object storage during the migration. A configuration setting will be added soon to allow migrating from object storage to local files with only a brief moment of downtime for configuration changes. -To follow progress, see the [relevant issue](https://gitlab.com/gitlab-org/gitlab/issues/30979). +To follow progress, see the [relevant issue](https://gitlab.com/gitlab-org/gitlab/-/issues/30979). ### All-in-one Rake task diff --git a/doc/administration/raketasks/uploads/sanitize.md b/doc/administration/raketasks/uploads/sanitize.md index 7c760b95c33..9586ab2c6f4 100644 --- a/doc/administration/raketasks/uploads/sanitize.md +++ b/doc/administration/raketasks/uploads/sanitize.md @@ -65,5 +65,5 @@ The output is written into an `exif.log` file because it will probably be long. If sanitization fails for an upload, an error message should be in the output of the Rake task. Typical reasons include that the file is missing in the storage or it's not a valid image. -[Report](https://gitlab.com/gitlab-org/gitlab/issues/new) any issues and use the prefix 'EXIF' in +[Report](https://gitlab.com/gitlab-org/gitlab/-/issues/new) any issues and use the prefix 'EXIF' in the issue title with the error output and (if possible) the image. diff --git a/doc/administration/reference_architectures/10k_users.md b/doc/administration/reference_architectures/10k_users.md index 7f31f336251..a15fcf722a5 100644 --- a/doc/administration/reference_architectures/10k_users.md +++ b/doc/administration/reference_architectures/10k_users.md @@ -8,8 +8,8 @@ For a full list of reference architectures, see > - **High Availability:** True > - **Test RPS rates:** API: 200 RPS, Web: 20 RPS, Git: 20 RPS -| Service | Nodes | Configuration ([8](#footnotes)) | GCP | AWS ([9](#footnotes)) | Azure([9](#footnotes)) | -|--------------------------------------------------------------|-------|---------------------------------|----------------|-----------------------|------------------------| +| Service | Nodes | Configuration ([8](#footnotes)) | GCP | AWS | Azure | +|--------------------------------------------------------------|-------|---------------------------------|----------------|-----------------------|----------------| | GitLab Rails ([1](#footnotes)) | 3 | 32 vCPU, 28.8GB Memory | n1-highcpu-32 | c5.9xlarge | F32s v2 | | PostgreSQL | 3 | 4 vCPU, 15GB Memory | n1-standard-4 | m5.xlarge | D4s v3 | | PgBouncer | 3 | 2 vCPU, 1.8GB Memory | n1-highcpu-2 | c5.large | F2s v2 | @@ -56,7 +56,7 @@ For a full list of reference architectures, see 1. NFS can be used as an alternative for both repository data (replacing Gitaly) and object storage but this isn't typically recommended for performance reasons. Note however it is required for - [GitLab Pages](https://gitlab.com/gitlab-org/gitlab-pages/issues/196). + [GitLab Pages](https://gitlab.com/gitlab-org/gitlab-pages/-/issues/196). 1. Our architectures have been tested and validated with [HAProxy](https://www.haproxy.org/) as the load balancer. Although other load balancers with similar feature sets @@ -74,6 +74,3 @@ For a full list of reference architectures, see or higher, are required for your CPU or Node counts accordingly. For more information, a [Sysbench](https://github.com/akopytov/sysbench) benchmark of the CPU can be found [here](https://gitlab.com/gitlab-org/quality/performance/-/wikis/Reference-Architectures/GCP-CPU-Benchmarks). - -1. AWS-equivalent and Azure-equivalent configurations are rough suggestions - and may change in the future. They have not yet been tested and validated. diff --git a/doc/administration/reference_architectures/1k_users.md b/doc/administration/reference_architectures/1k_users.md index b6aaffa9488..34805a8ac68 100644 --- a/doc/administration/reference_architectures/1k_users.md +++ b/doc/administration/reference_architectures/1k_users.md @@ -7,11 +7,18 @@ For a full list of reference architectures, see > - **Supported users (approximate):** 1,000 > - **High Availability:** False -| Users | Configuration([8](#footnotes)) | GCP | AWS([9](#footnotes)) | Azure([9](#footnotes)) | -|-------|--------------------------------|---------------|----------------------|------------------------| -| 100 | 2 vCPU, 7.2GB Memory | n1-standard-2 | m5.large | D2s v3 | -| 500 | 4 vCPU, 15GB Memory | n1-standard-4 | m5.xlarge | D4s v3 | -| 1000 | 8 vCPU, 30GB Memory | n1-standard-8 | m5.2xlarge | D8s v3 | +| Users | Configuration([8](#footnotes)) | GCP | AWS | Azure | +|-------|------------------------------------|----------------|---------------------|------------------------| +| 500 | 4 vCPU, 3.6GB Memory | `n1-highcpu-4` | `c5.xlarge` | F4s v2 | +| 1000 | 8 vCPU, 7.2GB Memory | `n1-highcpu-8` | `c5.2xlarge` | F8s v2 | + +In addition to the above, we recommend having at least +2GB of swap on your server, even if you currently have +enough available RAM. Having swap will help reduce the chance of errors occurring +if your available memory changes. We also recommend +configuring the kernel's swappiness setting +to a low value like `10` to make the most of your RAM while still having the swap +available when needed. For situations where you need to serve up to 1,000 users, a single-node solution with [frequent backups](index.md#automated-backups-core-only) is appropriate @@ -25,7 +32,7 @@ requirements, this is the ideal solution. NOTE: **Note:** You can also optionally configure GitLab to use an -[external PostgreSQL service](../external_database.md) or an +[external PostgreSQL service](../postgresql/external.md) or an [external object storage service](../high_availability/object_storage.md) for added performance and reliability at a reduced complexity cost. @@ -59,7 +66,7 @@ added performance and reliability at a reduced complexity cost. 1. NFS can be used as an alternative for both repository data (replacing Gitaly) and object storage but this isn't typically recommended for performance reasons. Note however it is required for - [GitLab Pages](https://gitlab.com/gitlab-org/gitlab-pages/issues/196). + [GitLab Pages](https://gitlab.com/gitlab-org/gitlab-pages/-/issues/196). 1. Our architectures have been tested and validated with [HAProxy](https://www.haproxy.org/) as the load balancer. Although other load balancers with similar feature sets @@ -77,6 +84,3 @@ added performance and reliability at a reduced complexity cost. or higher, are required for your CPU or Node counts accordingly. For more information, a [Sysbench](https://github.com/akopytov/sysbench) benchmark of the CPU can be found [here](https://gitlab.com/gitlab-org/quality/performance/-/wikis/Reference-Architectures/GCP-CPU-Benchmarks). - -1. AWS-equivalent and Azure-equivalent configurations are rough suggestions - and may change in the future. They have not yet been tested and validated. diff --git a/doc/administration/reference_architectures/25k_users.md b/doc/administration/reference_architectures/25k_users.md index 2ee692d635c..d851fa124c6 100644 --- a/doc/administration/reference_architectures/25k_users.md +++ b/doc/administration/reference_architectures/25k_users.md @@ -8,23 +8,23 @@ For a full list of reference architectures, see > - **High Availability:** True > - **Test RPS rates:** API: 500 RPS, Web: 50 RPS, Git: 50 RPS -| Service | Nodes | Configuration ([8](#footnotes)) | GCP | AWS ([9](#footnotes)) | Azure([9](#footnotes)) | -|--------------------------------------------------------------|-------|---------------------------------|----------------|-----------------------|------------------------| -| GitLab Rails ([1](#footnotes)) | 5 | 32 vCPU, 28.8GB Memory | n1-highcpu-32 | c5.9xlarge | F32s v2 | -| PostgreSQL | 3 | 8 vCPU, 30GB Memory | n1-standard-8 | m5.2xlarge | D8s v3 | -| PgBouncer | 3 | 2 vCPU, 1.8GB Memory | n1-highcpu-2 | c5.large | F2s v2 | -| Gitaly ([2](#footnotes)) ([5](#footnotes)) ([7](#footnotes)) | X | 32 vCPU, 120GB Memory | n1-standard-32 | m5.8xlarge | D32s v3 | -| Redis ([3](#footnotes)) - Cache | 3 | 4 vCPU, 15GB Memory | n1-standard-4 | m5.xlarge | D4s v3 | -| Redis ([3](#footnotes)) - Queues / Shared State | 3 | 4 vCPU, 15GB Memory | n1-standard-4 | m5.xlarge | D4s v3 | -| Redis Sentinel ([3](#footnotes)) - Cache | 3 | 1 vCPU, 1.7GB Memory | g1-small | t2.small | B1MS | -| Redis Sentinel ([3](#footnotes)) - Queues / Shared State | 3 | 1 vCPU, 1.7GB Memory | g1-small | t2.small | B1MS | -| Consul | 3 | 2 vCPU, 1.8GB Memory | n1-highcpu-2 | c5.large | F2s v2 | -| Sidekiq | 4 | 4 vCPU, 15GB Memory | n1-standard-4 | m5.xlarge | D4s v3 | +| Service | Nodes | Configuration ([8](#footnotes)) | GCP | AWS | Azure | +|--------------------------------------------------------------|-------|---------------------------------|------------------|-----------------------|----------------| +| GitLab Rails ([1](#footnotes)) | 5 | 32 vCPU, 28.8GB Memory | `n1-highcpu-32` | `c5.9xlarge` | F32s v2 | +| PostgreSQL | 3 | 8 vCPU, 30GB Memory | `n1-standard-8` | `m5.2xlarge` | D8s v3 | +| PgBouncer | 3 | 2 vCPU, 1.8GB Memory | `n1-highcpu-2` | `c5.large` | F2s v2 | +| Gitaly ([2](#footnotes)) ([5](#footnotes)) ([7](#footnotes)) | X | 32 vCPU, 120GB Memory | `n1-standard-32` | `m5.8xlarge` | D32s v3 | +| Redis ([3](#footnotes)) - Cache | 3 | 4 vCPU, 15GB Memory | `n1-standard-4` | `m5.xlarge` | D4s v3 | +| Redis ([3](#footnotes)) - Queues / Shared State | 3 | 4 vCPU, 15GB Memory | `n1-standard-4` | `m5.xlarge` | D4s v3 | +| Redis Sentinel ([3](#footnotes)) - Cache | 3 | 1 vCPU, 1.7GB Memory | `g1-small` | `t2.small` | B1MS | +| Redis Sentinel ([3](#footnotes)) - Queues / Shared State | 3 | 1 vCPU, 1.7GB Memory | `g1-small` | `t2.small` | B1MS | +| Consul | 3 | 2 vCPU, 1.8GB Memory | `n1-highcpu-2` | `c5.large` | F2s v2 | +| Sidekiq | 4 | 4 vCPU, 15GB Memory | `n1-standard-4` | `m5.xlarge` | D4s v3 | | Object Storage ([4](#footnotes)) | - | - | - | - | - | -| NFS Server ([5](#footnotes)) ([7](#footnotes)) | 1 | 4 vCPU, 3.6GB Memory | n1-highcpu-4 | c5.xlarge | F4s v2 | -| Monitoring node | 1 | 4 vCPU, 3.6GB Memory | n1-highcpu-4 | c5.xlarge | F4s v2 | -| External load balancing node ([6](#footnotes)) | 1 | 2 vCPU, 1.8GB Memory | n1-highcpu-2 | c5.large | F2s v2 | -| Internal load balancing node ([6](#footnotes)) | 1 | 4 vCPU, 3.6GB Memory | n1-highcpu-4 | c5.xlarge | F4s v2 | +| NFS Server ([5](#footnotes)) ([7](#footnotes)) | 1 | 4 vCPU, 3.6GB Memory | `n1-highcpu-4` | `c5.xlarge` | F4s v2 | +| Monitoring node | 1 | 4 vCPU, 3.6GB Memory | `n1-highcpu-4` | `c5.xlarge` | F4s v2 | +| External load balancing node ([6](#footnotes)) | 1 | 4 vCPU, 3.6GB Memory | `n1-highcpu-4` | `c5.xlarge` | F4s v2 | +| Internal load balancing node ([6](#footnotes)) | 1 | 4 vCPU, 3.6GB Memory | `n1-highcpu-4` | `c5.xlarge` | F4s v2 | ## Footnotes @@ -56,7 +56,7 @@ For a full list of reference architectures, see 1. NFS can be used as an alternative for both repository data (replacing Gitaly) and object storage but this isn't typically recommended for performance reasons. Note however it is required for - [GitLab Pages](https://gitlab.com/gitlab-org/gitlab-pages/issues/196). + [GitLab Pages](https://gitlab.com/gitlab-org/gitlab-pages/-/issues/196). 1. Our architectures have been tested and validated with [HAProxy](https://www.haproxy.org/) as the load balancer. Although other load balancers with similar feature sets @@ -74,6 +74,3 @@ For a full list of reference architectures, see or higher, are required for your CPU or Node counts accordingly. For more information, a [Sysbench](https://github.com/akopytov/sysbench) benchmark of the CPU can be found [here](https://gitlab.com/gitlab-org/quality/performance/-/wikis/Reference-Architectures/GCP-CPU-Benchmarks). - -1. AWS-equivalent and Azure-equivalent configurations are rough suggestions - and may change in the future. They have not yet been tested and validated. diff --git a/doc/administration/reference_architectures/2k_users.md b/doc/administration/reference_architectures/2k_users.md index 874e00e6722..0a3ade1acf1 100644 --- a/doc/administration/reference_architectures/2k_users.md +++ b/doc/administration/reference_architectures/2k_users.md @@ -1,3 +1,7 @@ +--- +reading_time: true +--- + # Reference architecture: up to 2,000 users This page describes GitLab reference architecture for up to 2,000 users. @@ -6,85 +10,864 @@ For a full list of reference architectures, see > - **Supported users (approximate):** 2,000 > - **High Availability:** False -> - **Test RPS rates:** API: 40 RPS, Web: 4 RPS, Git: 4 RPS - -| Service | Nodes | Configuration ([8](#footnotes)) | GCP | AWS ([9](#footnotes)) | Azure([9](#footnotes)) | -|--------------------------------------------------------------|-------|---------------------------------|---------------|-----------------------|----------------| -| External load balancing node ([6](#footnotes)) | 1 | 2 vCPU, 1.8GB Memory | n1-highcpu-2 | c5.large | F2s v2 | -| Object Storage ([4](#footnotes)) | - | - | - | - | - | -| NFS Server ([5](#footnotes)) ([7](#footnotes)) | 1 | 4 vCPU, 3.6GB Memory | n1-highcpu-4 | c5.xlarge | F4s v2 | -| PostgreSQL | 1 | 2 vCPU, 7.5GB Memory | n1-standard-2 | m5.large | D2s v3 | -| Redis ([3](#footnotes)) | 1 | 1 vCPU, 3.75GB Memory | n1-standard-1 | m5.large | D2s v3 | -| Gitaly ([5](#footnotes)) ([7](#footnotes)) | X ([2](#footnotes)) | 4 vCPU, 15GB Memory | n1-standard-4 | m5.xlarge | D4s v3 | -| GitLab Rails ([1](#footnotes)) | 2 | 8 vCPU, 7.2GB Memory | n1-highcpu-8 | c5.2xlarge | F8s v2 | -| Monitoring node | 1 | 2 vCPU, 1.8GB Memory | n1-highcpu-2 | c5.large | F2s v2 | - -## Setup instructions - -1. [Configure the external load balancing node](../high_availability/load_balancer.md) - that will handle the load balancing of the two GitLab application services nodes. -1. [Configure the Object Storage](../object_storage.md) ([4](#footnotes)) used for shared data objects. -1. (Optional) [Configure NFS](../high_availability/nfs.md) to have - shared disk storage service as an alternative to Gitaly and/or - [Object Storage](../object_storage.md) (although not recommended). - NFS is required for GitLab Pages, you can skip this step if you're not using that feature. -1. [Configure PostgreSQL](../high_availability/load_balancer.md), the database for GitLab. -1. [Configure Redis](../high_availability/redis.md). -1. [Configure Gitaly](../gitaly/index.md#running-gitaly-on-its-own-server), - which is used to provide access to the Git repositories. -1. [Configure the main GitLab Rails application](../high_availability/gitlab.md) - to run Puma/Unicorn, Workhorse, GitLab Shell, and to serve all - frontend requests (UI, API, Git over HTTP/SSH). -1. [Configure Prometheus](../high_availability/monitoring_node.md) to monitor your GitLab environment. - -## Footnotes - -1. In our architectures we run each GitLab Rails node using the Puma webserver - and have its number of workers set to 90% of available CPUs along with four threads. For - nodes that are running Rails with other components the worker value should be reduced - accordingly where we've found 50% achieves a good balance but this is dependent - on workload. - -1. Gitaly node requirements are dependent on customer data, specifically the number of - projects and their sizes. We recommend two nodes as an absolute minimum for HA environments - and at least four nodes should be used when supporting 50,000 or more users. - We also recommend that each Gitaly node should store no more than 5TB of data - and have the number of [`gitaly-ruby` workers](../gitaly/index.md#gitaly-ruby) - set to 20% of available CPUs. Additional nodes should be considered in conjunction - with a review of expected data size and spread based on the recommendations above. - -1. Recommended Redis setup differs depending on the size of the architecture. - For smaller architectures (less than 3,000 users) a single instance should suffice. - For medium sized installs (3,000 - 5,000) we suggest one Redis cluster for all - classes and that Redis Sentinel is hosted alongside Consul. - For larger architectures (10,000 users or more) we suggest running a separate - [Redis Cluster](../high_availability/redis.md#running-multiple-redis-clusters) for the Cache class - and another for the Queues and Shared State classes respectively. We also recommend - that you run the Redis Sentinel clusters separately for each Redis Cluster. - -1. For data objects such as LFS, Uploads, Artifacts, etc. We recommend an [Object Storage service](../object_storage.md) - over NFS where possible, due to better performance and availability. - -1. NFS can be used as an alternative for both repository data (replacing Gitaly) and - object storage but this isn't typically recommended for performance reasons. Note however it is required for - [GitLab Pages](https://gitlab.com/gitlab-org/gitlab-pages/issues/196). - -1. Our architectures have been tested and validated with [HAProxy](https://www.haproxy.org/) - as the load balancer. Although other load balancers with similar feature sets - could also be used, those load balancers have not been validated. - -1. We strongly recommend that any Gitaly or NFS nodes be set up with SSD disks over - HDD with a throughput of at least 8,000 IOPS for read operations and 2,000 IOPS for write - as these components have heavy I/O. These IOPS values are recommended only as a starter - as with time they may be adjusted higher or lower depending on the scale of your - environment's workload. If you're running the environment on a Cloud provider - you may need to refer to their documentation on how configure IOPS correctly. - -1. The architectures were built and tested with the [Intel Xeon E5 v3 (Haswell)](https://cloud.google.com/compute/docs/cpu-platforms) - CPU platform on GCP. On different hardware you may find that adjustments, either lower - or higher, are required for your CPU or Node counts accordingly. For more information, a - [Sysbench](https://github.com/akopytov/sysbench) benchmark of the CPU can be found - [here](https://gitlab.com/gitlab-org/quality/performance/-/wikis/Reference-Architectures/GCP-CPU-Benchmarks). - -1. AWS-equivalent and Azure-equivalent configurations are rough suggestions - and may change in the future. They have not yet been tested and validated. +> - **Test requests per second (RPS) rates:** API: 40 RPS, Web: 4 RPS, Git: 4 RPS + +| Service | Nodes | Configuration | GCP | AWS | Azure | +|--------------------------------------------------------------|-----------|---------------------------------|---------------|-----------------------|----------------| +| Load balancer | 1 | 2 vCPU, 1.8GB memory | n1-highcpu-2 | c5.large | F2s v2 | +| Object storage | n/a | n/a | n/a | n/a | n/a | +| NFS server (optional, not recommended) | 1 | 4 vCPU, 3.6GB memory | n1-highcpu-4 | c5.xlarge | F4s v2 | +| PostgreSQL | 1 | 2 vCPU, 7.5GB memory | n1-standard-2 | m5.large | D2s v3 | +| Redis | 1 | 1 vCPU, 3.75GB memory | n1-standard-1 | m5.large | D2s v3 | +| Gitaly | 1 | 4 vCPU, 15GB memory | n1-standard-4 | m5.xlarge | D4s v3 | +| GitLab Rails | 2 | 8 vCPU, 7.2GB memory | n1-highcpu-8 | c5.2xlarge | F8s v2 | +| Monitoring node | 1 | 2 vCPU, 1.8GB memory | n1-highcpu-2 | c5.large | F2s v2 | + +The Google Cloud Platform (GCP) architectures were built and tested using the +[Intel Xeon E5 v3 (Haswell)](https://cloud.google.com/compute/docs/cpu-platforms) +CPU platform. On different hardware you may find that adjustments, either lower +or higher, are required for your CPU or node counts. For more information, see +our [Sysbench](https://github.com/akopytov/sysbench)-based +[CPU benchmark](https://gitlab.com/gitlab-org/quality/performance/-/wikis/Reference-Architectures/GCP-CPU-Benchmarks). + +AWS-equivalent and Azure-equivalent configurations are rough suggestions that +may change in the future, and haven't been tested or validated. + +Due to better performance and availability, for data objects (such as LFS, +uploads, or artifacts), using an [object storage service](#configure-the-object-storage) +is recommended instead of using NFS. Using an object storage service also +doesn't require you to provision and maintain a node. + +## Setup components + +To set up GitLab and its components to accommodate up to 2,000 users: + +1. [Configure the external load balancing node](#configure-the-load-balancer) + to handle the load balancing of the two GitLab application services nodes. +1. [Configure the object storage](#configure-the-object-storage) used for + shared data objects. +1. [Configure NFS](#configure-nfs-optional) (optional, and not recommended) + to have shared disk storage service as an alternative to Gitaly or object + storage. You can skip this step if you're not using GitLab Pages (which + requires NFS). +1. [Configure PostgreSQL](#configure-postgresql), the database for GitLab. +1. [Configure Redis](#configure-redis). +1. [Configure Gitaly](#configure-gitaly), which provides access to the Git + repositories. +1. [Configure the main GitLab Rails application](#configure-gitlab-rails) + to run Puma/Unicorn, Workhorse, GitLab Shell, and to serve all frontend + requests (which include UI, API, and Git over HTTP/SSH). +1. [Configure Prometheus](#configure-prometheus) to monitor your GitLab + environment. + +## Configure the load balancer + +NOTE: **Note:** +This architecture has been tested and validated with [HAProxy](https://www.haproxy.org/). +Although you can use a load balancer with a similar set of features, GitLab +hasn't validated other load balancers. + +In an active/active GitLab configuration, you'll need a load balancer to route +traffic to the application servers. The specifics for which load balancer to +use or its exact configuration is out of scope for the GitLab documentation. +If you're managing multi-node systems (including GitLab) you'll probably +already have a load balancer of choice. Some examples including HAProxy +(open-source), F5 Big-IP LTM, and Citrix Net Scaler. This documentation +includes the ports and protocols for use with GitLab. + +The next question is how you will handle SSL in your environment. There are +several different options: + +- [The application node terminates SSL](#application-node-terminates-ssl). +- [The load balancer terminates SSL without backend SSL](#load-balancer-terminates-ssl-without-backend-ssl) + and communication is not secure between the load balancer and the application node. +- [The load balancer terminates SSL with backend SSL](#load-balancer-terminates-ssl-with-backend-ssl) + and communication is *secure* between the load balancer and the application node. + +### Application node terminates SSL + +Configure your load balancer to pass connections on port 443 as `TCP` instead +of `HTTP(S)`. This will pass the connection unaltered to the application node's +NGINX service, which has the SSL certificate and listens to port 443. + +For details about managing SSL certificates and configuring NGINX, see the +[NGINX HTTPS documentation](https://docs.gitlab.com/omnibus/settings/nginx.html#enable-https). + +### Load balancer terminates SSL without backend SSL + +Configure your load balancer to use the `HTTP(S)` protocol instead of `TCP`. +The load balancer will be responsible for both managing SSL certificates and +terminating SSL. + +Due to communication between the load balancer and GitLab not being secure, +you'll need to complete some additional configuration. For details, see the +[NGINX proxied SSL documentation](https://docs.gitlab.com/omnibus/settings/nginx.html#supporting-proxied-ssl). + +### Load balancer terminates SSL with backend SSL + +Configure your load balancers (or single balancer, if you have only one) to use +the `HTTP(S)` protocol rather than `TCP`. The load balancers will be +responsible for the managing SSL certificates for end users. + +Traffic will be secure between the load balancers and NGINX in this scenario, +and there's no need to add a configuration for proxied SSL. However, you'll +need to add a configuration to GitLab to configure SSL certificates. For +details about managing SSL certificates and configuring NGINX, see the +[NGINX HTTPS documentation](https://docs.gitlab.com/omnibus/settings/nginx.html#enable-https). + +### Ports + +The basic load balancer ports you should use are described in the following +table: + +| Port | Backend Port | Protocol | +| ------- | ------------ | ------------------------ | +| 80 | 80 | HTTP (*1*) | +| 443 | 443 | TCP or HTTPS (*1*) (*2*) | +| 22 | 22 | TCP | + +- (*1*): [Web terminal](../../ci/environments/index.md#web-terminals) support + requires your load balancer to correctly handle WebSocket connections. + When using HTTP or HTTPS proxying, your load balancer must be configured + to pass through the `Connection` and `Upgrade` hop-by-hop headers. For + details, see the [web terminal](../integration/terminal.md) integration guide. +- (*2*): When using the HTTPS protocol for port 443, you'll need to add an SSL + certificate to the load balancers. If you need to terminate SSL at the + GitLab application server, use the TCP protocol. + +If you're using GitLab Pages with custom domain support you will need some +additional port configurations. GitLab Pages requires a separate virtual IP +address. Configure DNS to point the `pages_external_url` from +`/etc/gitlab/gitlab.rb` to the new virtual IP address. For more information, +see the [GitLab Pages documentation](../pages/index.md). + +| Port | Backend Port | Protocol | +| ------- | ------------- | --------- | +| 80 | Varies (*1*) | HTTP | +| 443 | Varies (*1*) | TCP (*2*) | + +- (*1*): The backend port for GitLab Pages depends on the + `gitlab_pages['external_http']` and `gitlab_pages['external_https']` + settings. For details, see the [GitLab Pages documentation](../pages/index.md). +- (*2*): Port 443 for GitLab Pages must use the TCP protocol. Users can + configure custom domains with custom SSL, which wouldn't be possible if SSL + was terminated at the load balancer. + +#### Alternate SSH Port + +Some organizations have policies against opening SSH port 22. In this case, +it may be helpful to configure an alternate SSH hostname that instead allows +users to use SSH over port 443. An alternate SSH hostname requires a new +virtual IP address compared to the previously described GitLab HTTP +configuration. + +Configure DNS for an alternate SSH hostname, such as `altssh.gitlab.example.com`: + +| LB Port | Backend Port | Protocol | +| ------- | ------------ | -------- | +| 443 | 22 | TCP | + +<div align="right"> + <a type="button" class="btn btn-default" href="#setup-components"> + Back to setup components <i class="fa fa-angle-double-up" aria-hidden="true"></i> + </a> +</div> + +## Configure the object storage + +GitLab supports using an object storage service for holding several types of +data, and is recommended over [NFS](#configure-nfs-optional). In general, +object storage services are better for larger environments, as object storage +is typically much more performant, reliable, and scalable. + +Object storage options that GitLab has either tested or is aware of customers +using, includes: + +- SaaS/Cloud solutions (such as [Amazon S3](https://aws.amazon.com/s3/) or + [Google Cloud Storage](https://cloud.google.com/storage)). +- On-premises hardware and appliances, from various storage vendors. +- MinIO ([Deployment guide](https://docs.gitlab.com/charts/advanced/external-object-storage/minio.html)). + +To configure GitLab to use object storage, refer to the following guides based +on the features you intend to use: + +1. [Object storage for backups](../../raketasks/backup_restore.md#uploading-backups-to-a-remote-cloud-storage). +1. [Object storage for job artifacts](../job_artifacts.md#using-object-storage) + including [incremental logging](../job_logs.md#new-incremental-logging-architecture). +1. [Object storage for LFS objects](../lfs/index.md#storing-lfs-objects-in-remote-object-storage). +1. [Object storage for uploads](../uploads.md#using-object-storage-core-only). +1. [Object storage for merge request diffs](../merge_request_diffs.md#using-object-storage). +1. [Object storage for Container Registry](../packages/container_registry.md#container-registry-storage-driver) (optional feature). +1. [Object storage for Mattermost](https://docs.mattermost.com/administration/config-settings.html#file-storage) (optional feature). +1. [Object storage for packages](../packages/index.md#using-object-storage) (optional feature). **(PREMIUM ONLY)** +1. [Object storage for Dependency Proxy](../packages/dependency_proxy.md#using-object-storage) (optional feature). **(PREMIUM ONLY)** +1. [Object storage for Pseudonymizer](../pseudonymizer.md#configuration) (optional feature). **(ULTIMATE ONLY)** +1. [Object storage for autoscale Runner caching](https://docs.gitlab.com/runner/configuration/autoscale.html#distributed-runners-caching) (optional, for improved performance). +1. [Object storage for Terraform state files](../terraform_state.md#using-object-storage-core-only). + +Using separate buckets for each data type is the recommended approach for GitLab. + +A limitation of our configuration is that each use of object storage is +separately configured. We have an [issue](https://gitlab.com/gitlab-org/gitlab/-/issues/23345) +for improving this, which would allow for one bucket with separate folders. + +Using a single bucket when GitLab is deployed with the Helm chart causes +restoring from a backup to +[not function properly](https://docs.gitlab.com/charts/advanced/external-object-storage/#lfs-artifacts-uploads-packages-external-diffs-pseudonymizer). +Although you may not be using a Helm deployment right now, if you migrate +GitLab to a Helm deployment later, GitLab would still work, but you may not +realize backups aren't working correctly until a critical requirement for +functioning backups is encountered. + +<div align="right"> + <a type="button" class="btn btn-default" href="#setup-components"> + Back to setup components <i class="fa fa-angle-double-up" aria-hidden="true"></i> + </a> +</div> + +## Configure NFS (optional) + +For improved performance, [object storage](#configure-the-object-storage), +along with [Gitaly](#configure-gitaly), are recommended over using NFS whenever +possible. However, if you intend to use GitLab Pages, +[you must use NFS](troubleshooting.md#gitlab-pages-requires-nfs). + +For information about configuring NFS, see the [NFS documentation page](../high_availability/nfs.md). + +<div align="right"> + <a type="button" class="btn btn-default" href="#setup-components"> + Back to setup components <i class="fa fa-angle-double-up" aria-hidden="true"></i> + </a> +</div> + +## Configure PostgreSQL + +In this section, you'll be guided through configuring an external PostgreSQL database +to be used with GitLab. + +### Provide your own PostgreSQL instance + +If you're hosting GitLab on a cloud provider, you can optionally use a +managed service for PostgreSQL. For example, AWS offers a managed relational +database service (RDS) that runs PostgreSQL. + +If you use a cloud-managed service, or provide your own PostgreSQL: + +1. Set up PostgreSQL according to the + [database requirements document](../../install/requirements.md#database). +1. Create a `gitlab` username with a password of your choice. The `gitlab` user + needs privileges to create the `gitlabhq_production` database. +1. Configure the GitLab application servers with the appropriate details. + This step is covered in [Configuring the GitLab Rails application](#configure-gitlab-rails). + +### Standalone PostgreSQL using Omnibus GitLab + +1. SSH into the PostgreSQL server. +1. [Download/install](https://about.gitlab.com/install/) the Omnibus GitLab + package you want using **steps 1 and 2** from the GitLab downloads page. + - Do not complete any other steps on the download page. +1. Generate a password hash for PostgreSQL. This assumes you will use the default + username of `gitlab` (recommended). The command will request a password + and confirmation. Use the value that is output by this command in the next + step as the value of `POSTGRESQL_PASSWORD_HASH`. + + ```shell + sudo gitlab-ctl pg-password-md5 gitlab + ``` + +1. Edit `/etc/gitlab/gitlab.rb` and add the contents below, updating placeholder + values appropriately. + + - `POSTGRESQL_PASSWORD_HASH` - The value output from the previous step + - `APPLICATION_SERVER_IP_BLOCKS` - A space delimited list of IP subnets or IP + addresses of the GitLab application servers that will connect to the + database. Example: `%w(123.123.123.123/32 123.123.123.234/32)` + + ```ruby + # Disable all components except PostgreSQL + roles ['postgres_role'] + repmgr['enable'] = false + consul['enable'] = false + prometheus['enable'] = false + alertmanager['enable'] = false + pgbouncer_exporter['enable'] = false + redis_exporter['enable'] = false + gitlab_exporter['enable'] = false + + # Set the network addresses that the exporters used for monitoring will listen on + node_exporter['listen_address'] = '0.0.0.0:9100' + postgres_exporter['listen_address'] = '0.0.0.0:9187' + postgres_exporter['dbname'] = 'gitlabhq_production' + postgres_exporter['password'] = 'POSTGRESQL_PASSWORD_HASH' + + # Set the PostgreSQL address and port + postgresql['listen_address'] = '0.0.0.0' + postgresql['port'] = 5432 + + # Replace POSTGRESQL_PASSWORD_HASH with a generated md5 value + postgresql['sql_user_password'] = 'POSTGRESQL_PASSWORD_HASH' + + # Replace APPLICATION_SERVER_IP_BLOCK with the CIDR address of the application node + postgresql['trust_auth_cidr_addresses'] = %w(127.0.0.1/32 APPLICATION_SERVER_IP_BLOCK) + + # Disable automatic database migrations + gitlab_rails['auto_migrate'] = false + ``` + +1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. +1. Note the PostgreSQL node's IP address or hostname, port, and + plain text password. These will be necessary when configuring the [GitLab + application server](#configure-gitlab-rails) later. + +Advanced [configuration options](https://docs.gitlab.com/omnibus/settings/database.html) +are supported and can be added if needed. + +<div align="right"> + <a type="button" class="btn btn-default" href="#setup-components"> + Back to setup components <i class="fa fa-angle-double-up" aria-hidden="true"></i> + </a> +</div> + +## Configure Redis + +In this section, you'll be guided through configuring an external Redis instance +to be used with GitLab. + +### Provide your own Redis instance + +Redis version 5.0 or higher is required, as this is what ships with +Omnibus GitLab packages starting with GitLab 13.0. Older Redis versions +do not support an optional count argument to SPOP which is now required for +[Merge Trains](../../ci/merge_request_pipelines/pipelines_for_merged_results/merge_trains/index.md). + +In addition, GitLab makes use of certain commands like `UNLINK` and `USAGE` which +were introduced only in Redis 4. + +Managed Redis from cloud providers such as AWS ElastiCache will work. If these +services support high availability, be sure it is not the Redis Cluster type. + +Note the Redis node's IP address or hostname, port, and password (if required). +These will be necessary when configuring the +[GitLab application servers](#configure-gitlab-rails) later. + +### Standalone Redis using Omnibus GitLab + +The Omnibus GitLab package can be used to configure a standalone Redis server. +The steps below are the minimum necessary to configure a Redis server with +Omnibus: + +1. SSH into the Redis server. +1. [Download/install](https://about.gitlab.com/install/) the Omnibus GitLab + package you want using **steps 1 and 2** from the GitLab downloads page. + - Do not complete any other steps on the download page. + +1. Edit `/etc/gitlab/gitlab.rb` and add the contents: + + ```ruby + ## Enable Redis + redis['enable'] = true + + ## Disable all other services + sidekiq['enable'] = false + gitlab_workhorse['enable'] = false + puma['enable'] = false + unicorn['enable'] = false + postgresql['enable'] = false + nginx['enable'] = false + prometheus['enable'] = false + alertmanager['enable'] = false + pgbouncer_exporter['enable'] = false + gitlab_exporter['enable'] = false + gitaly['enable'] = false + grafana['enable'] = false + + redis['bind'] = '0.0.0.0' + redis['port'] = 6379 + redis['password'] = 'SECRET_PASSWORD_HERE' + + gitlab_rails['enable'] = false + + # Set the network addresses that the exporters used for monitoring will listen on + node_exporter['listen_address'] = '0.0.0.0:9100' + redis_exporter['listen_address'] = '0.0.0.0:9121' + redis_exporter['flags'] = { + 'redis.addr' => 'redis://0.0.0.0:6379', + 'redis.password' => 'SECRET_PASSWORD_HERE', + } + ``` + +1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. +1. Note the Redis node's IP address or hostname, port, and + Redis password. These will be necessary when [configuring the GitLab + application servers](#configure-gitlab-rails) later. + +Advanced [configuration options](https://docs.gitlab.com/omnibus/settings/redis.html) +are supported and can be added if needed. + +<div align="right"> + <a type="button" class="btn btn-default" href="#setup-components"> + Back to setup components <i class="fa fa-angle-double-up" aria-hidden="true"></i> + </a> +</div> + +## Configure Gitaly + +Deploying Gitaly in its own server can benefit GitLab installations that are +larger than a single machine. Gitaly node requirements are dependent on data, +specifically the number of projects and their sizes. It's recommended that each +Gitaly node store no more than 5TB of data. Your 2K setup may require one or more +nodes depending on your repository storage requirements. + +We strongly recommend that all Gitaly nodes should be set up with SSD disks with a throughput of at least +8,000 IOPS for read operations and 2,000 IOPS for write, as Gitaly has heavy I/O. +These IOPS values are recommended only as a starter as with time they may be +adjusted higher or lower depending on the scale of your environment's workload. +If you're running the environment on a Cloud provider +you may need to refer to their documentation on how configure IOPS correctly. + +Some things to note: + +- The GitLab Rails application shards repositories into [repository storages](../repository_storage_paths.md). +- A Gitaly server can host one or more storages. +- A GitLab server can use one or more Gitaly servers. +- Gitaly addresses must be specified in such a way that they resolve + correctly for ALL Gitaly clients. +- Gitaly servers must not be exposed to the public internet, as Gitaly's network + traffic is unencrypted by default. The use of a firewall is highly recommended + to restrict access to the Gitaly server. Another option is to + [use TLS](#gitaly-tls-support). + +TIP: **Tip:** +For more information about Gitaly's history and network architecture see the +[standalone Gitaly documentation](../gitaly/index.md). + +Note: **Note:** The token referred to throughout the Gitaly documentation is +just an arbitrary password selected by the administrator. It is unrelated to +tokens created for the GitLab API or other similar web API tokens. + +Below we describe how to configure one Gitaly server `gitaly1.internal` with +secret token `gitalysecret`. We assume your GitLab installation has two +repository storages: `default` and `storage1`. + +To configure the Gitaly server: + +1. [Download/Install](https://about.gitlab.com/install/) the Omnibus GitLab + package you want using **steps 1 and 2** from the GitLab downloads page but + **without** providing the `EXTERNAL_URL` value. +1. Edit `/etc/gitlab/gitlab.rb` to configure storage paths, enable + the network listener and configure the token: + + <!-- + updates to following example must also be made at + https://gitlab.com/gitlab-org/charts/gitlab/blob/master/doc/advanced/external-gitaly/external-omnibus-gitaly.md#configure-omnibus-gitlab + --> + + ```ruby + # /etc/gitlab/gitlab.rb + + # Gitaly and GitLab use two shared secrets for authentication, one to authenticate gRPC requests + # to Gitaly, and a second for authentication callbacks from GitLab-Shell to the GitLab internal API. + # The following two values must be the same as their respective values + # of the GitLab Rails application setup + gitaly['auth_token'] = 'gitlaysecret' + gitlab_shell['secret_token'] = 'shellsecret' + + # Avoid running unnecessary services on the Gitaly server + postgresql['enable'] = false + redis['enable'] = false + nginx['enable'] = false + puma['enable'] = false + unicorn['enable'] = false + sidekiq['enable'] = false + gitlab_workhorse['enable'] = false + grafana['enable'] = false + + # If you run a seperate monitoring node you can disable these services + alertmanager['enable'] = false + prometheus['enable'] = false + + # Prevent database connections during 'gitlab-ctl reconfigure' + gitlab_rails['rake_cache_clear'] = false + gitlab_rails['auto_migrate'] = false + + # Configure the gitlab-shell API callback URL. Without this, `git push` will + # fail. This can be your 'front door' GitLab URL or an internal load + # balancer. + # Don't forget to copy `/etc/gitlab/gitlab-secrets.json` from web server to Gitaly server. + gitlab_rails['internal_api_url'] = 'https://gitlab.example.com' + + # Make Gitaly accept connections on all network interfaces. You must use + # firewalls to restrict access to this address/port. + # Comment out following line if you only want to support TLS connections + gitaly['listen_addr'] = "0.0.0.0:8075" + gitaly['prometheus_listen_addr'] = "0.0.0.0:9236" + + # Set the network addresses that the exporters used for monitoring will listen on + node_exporter['listen_address'] = '0.0.0.0:9100' + ``` + +1. Append the following to `/etc/gitlab/gitlab.rb` on `gitaly1.internal`: + + ```ruby + git_data_dirs({ + 'default' => { + 'path' => '/var/opt/gitlab/git-data' + }, + 'storage1' => { + 'path' => '/mnt/gitlab/git-data' + }, + }) + ``` + + <!-- + updates to following example must also be made at + https://gitlab.com/gitlab-org/charts/gitlab/blob/master/doc/advanced/external-gitaly/external-omnibus-gitaly.md#configure-omnibus-gitlab + --> + +1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). +1. Confirm that Gitaly can perform callbacks to the internal API: + + ```shell + sudo /opt/gitlab/embedded/service/gitlab-shell/bin/check -config /opt/gitlab/embedded/service/gitlab-shell/config.yml + ``` + +### Gitaly TLS support + +Gitaly supports TLS encryption. To be able to communicate +with a Gitaly instance that listens for secure connections you will need to use `tls://` URL +scheme in the `gitaly_address` of the corresponding storage entry in the GitLab configuration. + +You will need to bring your own certificates as this isn't provided automatically. +The certificate, or its certificate authority, must be installed on all Gitaly +nodes (including the Gitaly node using the certificate) and on all client nodes +that communicate with it following the procedure described in +[GitLab custom certificate configuration](https://docs.gitlab.com/omnibus/settings/ssl.html#install-custom-public-certificates). + +NOTE: **Note** +The self-signed certificate must specify the address you use to access the +Gitaly server. If you are addressing the Gitaly server by a hostname, you can +either use the Common Name field for this, or add it as a Subject Alternative +Name. If you are addressing the Gitaly server by its IP address, you must add it +as a Subject Alternative Name to the certificate. +[gRPC does not support using an IP address as Common Name in a certificate](https://github.com/grpc/grpc/issues/2691). + +NOTE: **Note:** +It is possible to configure Gitaly servers with both an +unencrypted listening address `listen_addr` and an encrypted listening +address `tls_listen_addr` at the same time. This allows you to do a +gradual transition from unencrypted to encrypted traffic, if necessary. + +To configure Gitaly with TLS: + +1. Create the `/etc/gitlab/ssl` directory and copy your key and certificate there: + + ```shell + sudo mkdir -p /etc/gitlab/ssl + sudo chmod 755 /etc/gitlab/ssl + sudo cp key.pem cert.pem /etc/gitlab/ssl/ + sudo chmod 644 key.pem cert.pem + ``` + +1. Copy the cert to `/etc/gitlab/trusted-certs` so Gitaly will trust the cert when + calling into itself: + + ```shell + sudo cp /etc/gitlab/ssl/cert.pem /etc/gitlab/trusted-certs/ + ``` + +1. Edit `/etc/gitlab/gitlab.rb` and add: + + <!-- + updates to following example must also be made at + https://gitlab.com/gitlab-org/charts/gitlab/blob/master/doc/advanced/external-gitaly/external-omnibus-gitaly.md#configure-omnibus-gitlab + --> + + ```ruby + gitaly['tls_listen_addr'] = "0.0.0.0:9999" + gitaly['certificate_path'] = "/etc/gitlab/ssl/cert.pem" + gitaly['key_path'] = "/etc/gitlab/ssl/key.pem" + ``` + +1. Delete `gitaly['listen_addr']` to allow only encrypted connections. +1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). + +<div align="right"> + <a type="button" class="btn btn-default" href="#setup-components"> + Back to setup components <i class="fa fa-angle-double-up" aria-hidden="true"></i> + </a> +</div> + +## Configure GitLab Rails + +NOTE: **Note:** +In our architectures we run each GitLab Rails node using the Puma webserver +and have its number of workers set to 90% of available CPUs along with four threads. For +nodes that are running Rails with other components the worker value should be reduced +accordingly where we've found 50% achieves a good balance but this is dependent +on workload. + +This section describes how to configure the GitLab application (Rails) component. +On each node perform the following: + +1. If you're [using NFS](#configure-nfs-optional): + + 1. If necessary, install the NFS client utility packages using the following + commands: + + ```shell + # Ubuntu/Debian + apt-get install nfs-common + + # CentOS/Red Hat + yum install nfs-utils nfs-utils-lib + ``` + + 1. Specify the necessary NFS mounts in `/etc/fstab`. + The exact contents of `/etc/fstab` will depend on how you chose + to configure your NFS server. See the [NFS documentation](../high_availability/nfs.md) + for examples and the various options. + + 1. Create the shared directories. These may be different depending on your NFS + mount locations. + + ```shell + mkdir -p /var/opt/gitlab/.ssh /var/opt/gitlab/gitlab-rails/uploads /var/opt/gitlab/gitlab-rails/shared /var/opt/gitlab/gitlab-ci/builds /var/opt/gitlab/git-data + ``` + +1. Download/install Omnibus GitLab using **steps 1 and 2** from + [GitLab downloads](https://about.gitlab.com/install/). Do not complete other + steps on the download page. +1. Create/edit `/etc/gitlab/gitlab.rb` and use the following configuration. + To maintain uniformity of links across nodes, the `external_url` + on the application server should point to the external URL that users will use + to access GitLab. This would be the URL of the [load balancer](#configure-the-load-balancer) + which will route traffic to the GitLab application server: + + ```ruby + external_url 'https://gitlab.example.com' + + # Gitaly and GitLab use two shared secrets for authentication, one to authenticate gRPC requests + # to Gitaly, and a second for authentication callbacks from GitLab-Shell to the GitLab internal API. + # The following two values must be the same as their respective values + # of the Gitaly setup + gitlab_rails['gitaly_token'] = 'gitalyecret' + gitlab_shell['secret_token'] = 'shellsecret' + + git_data_dirs({ + 'default' => { 'gitaly_address' => 'tcp://gitaly1.internal:8075' }, + 'storage1' => { 'gitaly_address' => 'tcp://gitaly1.internal:8075' }, + 'storage2' => { 'gitaly_address' => 'tcp://gitaly2.internal:8075' }, + }) + + ## Disable components that will not be on the GitLab application server + roles ['application_role'] + gitaly['enable'] = false + nginx['enable'] = true + + ## PostgreSQL connection details + gitlab_rails['db_adapter'] = 'postgresql' + gitlab_rails['db_encoding'] = 'unicode' + gitlab_rails['db_host'] = '10.1.0.5' # IP/hostname of database server + gitlab_rails['db_password'] = 'DB password' + + ## Redis connection details + gitlab_rails['redis_port'] = '6379' + gitlab_rails['redis_host'] = '10.1.0.6' # IP/hostname of Redis server + gitlab_rails['redis_password'] = 'Redis Password' + + # Set the network addresses that the exporters used for monitoring will listen on + node_exporter['listen_address'] = '0.0.0.0:9100' + gitlab_workhorse['prometheus_listen_addr'] = '0.0.0.0:9229' + sidekiq['listen_address'] = "0.0.0.0" + puma['listen'] = '0.0.0.0' + + # Add the monitoring node's IP address to the monitoring whitelist and allow it to + # scrape the NGINX metrics. Replace placeholder `monitoring.gitlab.example.com` with + # the address and/or subnets gathered from the monitoring node + gitlab_rails['monitoring_whitelist'] = ['<MONITOR NODE IP>/32', '127.0.0.0/8'] + nginx['status']['options']['allow'] = ['<MONITOR NODE IP>/32', '127.0.0.0/8'] + + ## Uncomment and edit the following options if you have set up NFS + ## + ## Prevent GitLab from starting if NFS data mounts are not available + ## + #high_availability['mountpoint'] = '/var/opt/gitlab/git-data' + ## + ## Ensure UIDs and GIDs match between servers for permissions via NFS + ## + #user['uid'] = 9000 + #user['gid'] = 9000 + #web_server['uid'] = 9001 + #web_server['gid'] = 9001 + #registry['uid'] = 9002 + #registry['gid'] = 9002 + ``` + +1. If you're using [Gitaly with TLS support](#gitaly-tls-support), make sure the + `git_data_dirs` entry is configured with `tls` instead of `tcp`: + + ```ruby + git_data_dirs({ + 'default' => { 'gitaly_address' => 'tls://gitaly1.internal:9999' }, + 'storage1' => { 'gitaly_address' => 'tls://gitaly1.internal:9999' }, + 'storage2' => { 'gitaly_address' => 'tls://gitaly2.internal:9999' }, + }) + ``` + + 1. Copy the cert into `/etc/gitlab/trusted-certs`: + + ```shell + sudo cp cert.pem /etc/gitlab/trusted-certs/ + ``` + +1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). +1. Run `sudo gitlab-rake gitlab:gitaly:check` to confirm the node can connect to Gitaly. +1. Tail the logs to see the requests: + + ```shell + sudo gitlab-ctl tail gitaly + ``` + +NOTE: **Note:** When you specify `https` in the `external_url`, as in the example +above, GitLab assumes you have SSL certificates in `/etc/gitlab/ssl/`. If +certificates are not present, NGINX will fail to start. See the +[NGINX documentation](https://docs.gitlab.com/omnibus/settings/nginx.html#enable-https) +for more information. + +<div align="right"> + <a type="button" class="btn btn-default" href="#setup-components"> + Back to setup components <i class="fa fa-angle-double-up" aria-hidden="true"></i> + </a> +</div> + +## Configure Prometheus + +The Omnibus GitLab package can be used to configure a standalone Monitoring node +running [Prometheus](../monitoring/prometheus/index.md) and +[Grafana](../monitoring/performance/grafana_configuration.md): + +1. SSH into the Monitoring node. +1. [Download/install](https://about.gitlab.com/install/) the Omnibus GitLab + package you want using **steps 1 and 2** from the GitLab downloads page. + Do not complete any other steps on the download page. +1. Edit `/etc/gitlab/gitlab.rb` and add the contents: + + ```ruby + external_url 'http://gitlab.example.com' + + # Enable Prometheus + prometheus['enable'] = true + prometheus['listen_address'] = '0.0.0.0:9090' + prometheus['monitor_kubernetes'] = false + + # Enable Login form + grafana['disable_login_form'] = false + + # Enable Grafana + grafana['enable'] = true + grafana['admin_password'] = 'toomanysecrets' + + # Disable all other services + gitlab_rails['auto_migrate'] = false + alertmanager['enable'] = false + gitaly['enable'] = false + gitlab_exporter['enable'] = false + gitlab_workhorse['enable'] = false + nginx['enable'] = true + postgres_exporter['enable'] = false + postgresql['enable'] = false + redis['enable'] = false + redis_exporter['enable'] = false + sidekiq['enable'] = false + puma['enable'] = false + unicorn['enable'] = false + node_exporter['enable'] = false + gitlab_exporter['enable'] = false + ``` + +1. Prometheus also needs some scrape configs to pull all the data from the various + nodes where we configured exporters. Assuming that your nodes' IPs are: + + ```plaintext + 1.1.1.1: postgres + 1.1.1.2: redis + 1.1.1.3: gitaly1 + 1.1.1.4: rails1 + 1.1.1.5: rails2 + ``` + + Add the following to `/etc/gitlab/gitlab.rb`: + + ```ruby + prometheus['scrape_configs'] = [ + { + 'job_name': 'postgres', + 'static_configs' => [ + 'targets' => ['1.1.1.1:9187'], + ], + }, + { + 'job_name': 'redis', + 'static_configs' => [ + 'targets' => ['1.1.1.2:9121'], + ], + }, + { + 'job_name': 'gitaly', + 'static_configs' => [ + 'targets' => ['1.1.1.3:9236'], + ], + }, + { + 'job_name': 'gitlab-nginx', + 'static_configs' => [ + 'targets' => ['1.1.1.4:8060', '1.1.1.5:8060'], + ], + }, + { + 'job_name': 'gitlab-workhorse', + 'static_configs' => [ + 'targets' => ['1.1.1.4:9229', '1.1.1.5:9229'], + ], + }, + { + 'job_name': 'gitlab-rails', + 'metrics_path': '/-/metrics', + 'static_configs' => [ + 'targets' => ['1.1.1.4:8080', '1.1.1.5:8080'], + ], + }, + { + 'job_name': 'gitlab-sidekiq', + 'static_configs' => [ + 'targets' => ['1.1.1.4:8082', '1.1.1.5:8082'], + ], + }, + { + 'job_name': 'node', + 'static_configs' => [ + 'targets' => ['1.1.1.1:9100', '1.1.1.2:9100', '1.1.1.3:9100', '1.1.1.4:9100', '1.1.1.5:9100'], + ], + }, + ] + ``` + +1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). +1. In the GitLab UI, set `admin/application_settings/metrics_and_profiling` > Metrics - Grafana to `/-/grafana` to +`http[s]://<MONITOR NODE>/-/grafana` + +<div align="right"> + <a type="button" class="btn btn-default" href="#setup-components"> + Back to setup components <i class="fa fa-angle-double-up" aria-hidden="true"></i> + </a> +</div> + +## Troubleshooting + +See the [troubleshooting documentation](troubleshooting.md). + +<div align="right"> + <a type="button" class="btn btn-default" href="#setup-components"> + Back to setup components <i class="fa fa-angle-double-up" aria-hidden="true"></i> + </a> +</div> diff --git a/doc/administration/reference_architectures/3k_users.md b/doc/administration/reference_architectures/3k_users.md index bd429fbc4b4..efeed3e9ffd 100644 --- a/doc/administration/reference_architectures/3k_users.md +++ b/doc/administration/reference_architectures/3k_users.md @@ -14,20 +14,20 @@ following the [2,000-user reference architecture](2k_users.md). > - **High Availability:** True > - **Test RPS rates:** API: 60 RPS, Web: 6 RPS, Git: 6 RPS -| Service | Nodes | Configuration ([8](#footnotes)) | GCP | AWS ([9](#footnotes)) | Azure([9](#footnotes)) | -|--------------------------------------------------------------|-------|---------------------------------|---------------|-----------------------|------------------------| -| GitLab Rails ([1](#footnotes)) | 3 | 8 vCPU, 7.2GB Memory | n1-highcpu-8 | c5.2xlarge | F8s v2 | -| PostgreSQL | 3 | 2 vCPU, 7.5GB Memory | n1-standard-2 | m5.large | D2s v3 | -| PgBouncer | 3 | 2 vCPU, 1.8GB Memory | n1-highcpu-2 | c5.large | F2s v2 | -| Gitaly ([2](#footnotes)) ([5](#footnotes)) ([7](#footnotes)) | X | 4 vCPU, 15GB Memory | n1-standard-4 | m5.xlarge | D4s v3 | -| Redis ([3](#footnotes)) | 3 | 2 vCPU, 7.5GB Memory | n1-standard-2 | m5.large | D2s v3 | -| Consul + Sentinel ([3](#footnotes)) | 3 | 2 vCPU, 1.8GB Memory | n1-highcpu-2 | c5.large | F2s v2 | -| Sidekiq | 4 | 2 vCPU, 7.5GB Memory | n1-standard-2 | m5.large | D2s v3 | +| Service | Nodes | Configuration ([8](#footnotes)) | GCP | AWS | Azure | +|--------------------------------------------------------------|-------|---------------------------------|---------------|-----------------------|----------------| +| GitLab Rails ([1](#footnotes)) | 3 | 8 vCPU, 7.2GB Memory | `n1-highcpu-8` | `c5.2xlarge` | F8s v2 | +| PostgreSQL | 3 | 2 vCPU, 7.5GB Memory | `n1-standard-2` | `m5.large` | D2s v3 | +| PgBouncer | 3 | 2 vCPU, 1.8GB Memory | `n1-highcpu-2` | `c5.large` | F2s v2 | +| Gitaly ([2](#footnotes)) ([5](#footnotes)) ([7](#footnotes)) | X | 4 vCPU, 15GB Memory | `n1-standard-4` | `m5.xlarge` | D4s v3 | +| Redis ([3](#footnotes)) | 3 | 2 vCPU, 7.5GB Memory | `n1-standard-2` | `m5.large` | D2s v3 | +| Consul + Sentinel ([3](#footnotes)) | 3 | 2 vCPU, 1.8GB Memory | `n1-highcpu-2` | `c5.large` | F2s v2 | +| Sidekiq | 4 | 2 vCPU, 7.5GB Memory | `n1-standard-2` | `m5.large` | D2s v3 | | Object Storage ([4](#footnotes)) | - | - | - | - | - | -| NFS Server ([5](#footnotes)) ([7](#footnotes)) | 1 | 4 vCPU, 3.6GB Memory | n1-highcpu-4 | c5.xlarge | F4s v2 | -| Monitoring node | 1 | 2 vCPU, 1.8GB Memory | n1-highcpu-2 | c5.large | F2s v2 | -| External load balancing node ([6](#footnotes)) | 1 | 2 vCPU, 1.8GB Memory | n1-highcpu-2 | c5.large | F2s v2 | -| Internal load balancing node ([6](#footnotes)) | 1 | 2 vCPU, 1.8GB Memory | n1-highcpu-2 | c5.large | F2s v2 | +| NFS Server ([5](#footnotes)) ([7](#footnotes)) | 1 | 4 vCPU, 3.6GB Memory | `n1-highcpu-4` | `c5.xlarge` | F4s v2 | +| Monitoring node | 1 | 2 vCPU, 1.8GB Memory | `n1-highcpu-2` | `c5.large` | F2s v2 | +| External load balancing node ([6](#footnotes)) | 1 | 2 vCPU, 1.8GB Memory | `n1-highcpu-2` | `c5.large` | F2s v2 | +| Internal load balancing node ([6](#footnotes)) | 1 | 2 vCPU, 1.8GB Memory | `n1-highcpu-2` | `c5.large` | F2s v2 | ## Footnotes @@ -59,7 +59,7 @@ following the [2,000-user reference architecture](2k_users.md). 1. NFS can be used as an alternative for both repository data (replacing Gitaly) and object storage but this isn't typically recommended for performance reasons. Note however it is required for - [GitLab Pages](https://gitlab.com/gitlab-org/gitlab-pages/issues/196). + [GitLab Pages](https://gitlab.com/gitlab-org/gitlab-pages/-/issues/196). 1. Our architectures have been tested and validated with [HAProxy](https://www.haproxy.org/) as the load balancer. Although other load balancers with similar feature sets @@ -77,6 +77,3 @@ following the [2,000-user reference architecture](2k_users.md). or higher, are required for your CPU or Node counts accordingly. For more information, a [Sysbench](https://github.com/akopytov/sysbench) benchmark of the CPU can be found [here](https://gitlab.com/gitlab-org/quality/performance/-/wikis/Reference-Architectures/GCP-CPU-Benchmarks). - -1. AWS-equivalent and Azure-equivalent configurations are rough suggestions - and may change in the future. They have not yet been tested and validated. diff --git a/doc/administration/reference_architectures/50k_users.md b/doc/administration/reference_architectures/50k_users.md index 67f773a021f..dd94f5470b4 100644 --- a/doc/administration/reference_architectures/50k_users.md +++ b/doc/administration/reference_architectures/50k_users.md @@ -8,23 +8,23 @@ For a full list of reference architectures, see > - **High Availability:** True > - **Test RPS rates:** API: 1000 RPS, Web: 100 RPS, Git: 100 RPS -| Service | Nodes | Configuration ([8](#footnotes)) | GCP | AWS ([9](#footnotes)) | Azure([9](#footnotes)) | -|--------------------------------------------------------------|-------|---------------------------------|----------------|-----------------------|------------------------| -| GitLab Rails ([1](#footnotes)) | 12 | 32 vCPU, 28.8GB Memory | n1-highcpu-32 | c5.9xlarge | F32s v2 | -| PostgreSQL | 3 | 16 vCPU, 60GB Memory | n1-standard-16 | m5.4xlarge | D16s v3 | -| PgBouncer | 3 | 2 vCPU, 1.8GB Memory | n1-highcpu-2 | c5.large | F2s v2 | -| Gitaly ([2](#footnotes)) ([5](#footnotes)) ([7](#footnotes)) | X | 64 vCPU, 240GB Memory | n1-standard-64 | m5.16xlarge | D64s v3 | -| Redis ([3](#footnotes)) - Cache | 3 | 4 vCPU, 15GB Memory | n1-standard-4 | m5.xlarge | D4s v3 | -| Redis ([3](#footnotes)) - Queues / Shared State | 3 | 4 vCPU, 15GB Memory | n1-standard-4 | m5.xlarge | D4s v3 | -| Redis Sentinel ([3](#footnotes)) - Cache | 3 | 1 vCPU, 1.7GB Memory | g1-small | t2.small | B1MS | -| Redis Sentinel ([3](#footnotes)) - Queues / Shared State | 3 | 1 vCPU, 1.7GB Memory | g1-small | t2.small | B1MS | -| Consul | 3 | 2 vCPU, 1.8GB Memory | n1-highcpu-2 | c5.large | F2s v2 | -| Sidekiq | 4 | 4 vCPU, 15GB Memory | n1-standard-4 | m5.xlarge | D4s v3 | -| NFS Server ([5](#footnotes)) ([7](#footnotes)) | 1 | 4 vCPU, 3.6GB Memory | n1-highcpu-4 | c5.xlarge | F4s v2 | +| Service | Nodes | Configuration ([8](#footnotes)) | GCP | AWS | Azure | +|--------------------------------------------------------------|-------|---------------------------------|----------------|-----------------------|----------------| +| GitLab Rails ([1](#footnotes)) | 12 | 32 vCPU, 28.8GB Memory | `n1-highcpu-32` | `c5.9xlarge` | F32s v2 | +| PostgreSQL | 3 | 16 vCPU, 60GB Memory | `n1-standard-16` | `m5.4xlarge` | D16s v3 | +| PgBouncer | 3 | 2 vCPU, 1.8GB Memory | `n1-highcpu-2` | `c5.large` | F2s v2 | +| Gitaly ([2](#footnotes)) ([5](#footnotes)) ([7](#footnotes)) | X | 64 vCPU, 240GB Memory | `n1-standard-64` | `m5.16xlarge` | D64s v3 | +| Redis ([3](#footnotes)) - Cache | 3 | 4 vCPU, 15GB Memory | `n1-standard-4` | `m5.xlarge` | D4s v3 | +| Redis ([3](#footnotes)) - Queues / Shared State | 3 | 4 vCPU, 15GB Memory | `n1-standard-4` | `m5.xlarge` | D4s v3 | +| Redis Sentinel ([3](#footnotes)) - Cache | 3 | 1 vCPU, 1.7GB Memory | `g1-small` | `t2.small` | B1MS | +| Redis Sentinel ([3](#footnotes)) - Queues / Shared State | 3 | 1 vCPU, 1.7GB Memory | `g1-small` | `t2.small` | B1MS | +| Consul | 3 | 2 vCPU, 1.8GB Memory | `n1-highcpu-2` | `c5.large` | F2s v2 | +| Sidekiq | 4 | 4 vCPU, 15GB Memory | `n1-standard-4` | `m5.xlarge` | D4s v3 | +| NFS Server ([5](#footnotes)) ([7](#footnotes)) | 1 | 4 vCPU, 3.6GB Memory | `n1-highcpu-4` | `c5.xlarge` | F4s v2 | | Object Storage ([4](#footnotes)) | - | - | - | - | - | -| Monitoring node | 1 | 4 vCPU, 3.6GB Memory | n1-highcpu-4 | c5.xlarge | F4s v2 | -| External load balancing node ([6](#footnotes)) | 1 | 2 vCPU, 1.8GB Memory | n1-highcpu-2 | c5.large | F2s v2 | -| Internal load balancing node ([6](#footnotes)) | 1 | 8 vCPU, 7.2GB Memory | n1-highcpu-8 | c5.2xlarge | F8s v2 | +| Monitoring node | 1 | 4 vCPU, 3.6GB Memory | `n1-highcpu-4` | `c5.xlarge` | F4s v2 | +| External load balancing node ([6](#footnotes)) | 1 | 8 vCPU, 7.2GB Memory | `n1-highcpu-8` | `c5.2xlarge` | F8s v2 | +| Internal load balancing node ([6](#footnotes)) | 1 | 8 vCPU, 7.2GB Memory | `n1-highcpu-8` | `c5.2xlarge` | F8s v2 | ## Footnotes @@ -56,7 +56,7 @@ For a full list of reference architectures, see 1. NFS can be used as an alternative for both repository data (replacing Gitaly) and object storage but this isn't typically recommended for performance reasons. Note however it is required for - [GitLab Pages](https://gitlab.com/gitlab-org/gitlab-pages/issues/196). + [GitLab Pages](https://gitlab.com/gitlab-org/gitlab-pages/-/issues/196). 1. Our architectures have been tested and validated with [HAProxy](https://www.haproxy.org/) as the load balancer. Although other load balancers with similar feature sets @@ -74,6 +74,3 @@ For a full list of reference architectures, see or higher, are required for your CPU or Node counts accordingly. For more information, a [Sysbench](https://github.com/akopytov/sysbench) benchmark of the CPU can be found [here](https://gitlab.com/gitlab-org/quality/performance/-/wikis/Reference-Architectures/GCP-CPU-Benchmarks). - -1. AWS-equivalent and Azure-equivalent configurations are rough suggestions - and may change in the future. They have not yet been tested and validated. diff --git a/doc/administration/reference_architectures/5k_users.md b/doc/administration/reference_architectures/5k_users.md index 41ef6f369c2..604572b083e 100644 --- a/doc/administration/reference_architectures/5k_users.md +++ b/doc/administration/reference_architectures/5k_users.md @@ -8,20 +8,20 @@ For a full list of reference architectures, see > - **High Availability:** True > - **Test RPS rates:** API: 100 RPS, Web: 10 RPS, Git: 10 RPS -| Service | Nodes | Configuration ([8](#footnotes)) | GCP | AWS ([9](#footnotes)) | Azure([9](#footnotes)) | -|--------------------------------------------------------------|-------|---------------------------------|---------------|-----------------------|------------------------| -| GitLab Rails ([1](#footnotes)) | 3 | 16 vCPU, 14.4GB Memory | n1-highcpu-16 | c5.4xlarge | F16s v2 | -| PostgreSQL | 3 | 2 vCPU, 7.5GB Memory | n1-standard-2 | m5.large | D2s v3 | -| PgBouncer | 3 | 2 vCPU, 1.8GB Memory | n1-highcpu-2 | c5.large | F2s v2 | -| Gitaly ([2](#footnotes)) ([5](#footnotes)) ([7](#footnotes)) | X | 8 vCPU, 30GB Memory | n1-standard-8 | m5.2xlarge | D8s v3 | -| Redis ([3](#footnotes)) | 3 | 2 vCPU, 7.5GB Memory | n1-standard-2 | m5.large | D2s v3 | -| Consul + Sentinel ([3](#footnotes)) | 3 | 2 vCPU, 1.8GB Memory | n1-highcpu-2 | c5.large | F2s v2 | -| Sidekiq | 4 | 2 vCPU, 7.5GB Memory | n1-standard-2 | m5.large | D2s v3 | +| Service | Nodes | Configuration ([8](#footnotes)) | GCP | AWS | Azure | +|--------------------------------------------------------------|-------|---------------------------------|---------------|-----------------------|----------------| +| GitLab Rails ([1](#footnotes)) | 3 | 16 vCPU, 14.4GB Memory | `n1-highcpu-16` | `c5.4xlarge` | F16s v2 | +| PostgreSQL | 3 | 2 vCPU, 7.5GB Memory | `n1-standard-2` | `m5.large` | D2s v3 | +| PgBouncer | 3 | 2 vCPU, 1.8GB Memory | `n1-highcpu-2` | `c5.large` | F2s v2 | +| Gitaly ([2](#footnotes)) ([5](#footnotes)) ([7](#footnotes)) | X | 8 vCPU, 30GB Memory | `n1-standard-8` | `m5.2xlarge` | D8s v3 | +| Redis ([3](#footnotes)) | 3 | 2 vCPU, 7.5GB Memory | `n1-standard-2` | `m5.large` | D2s v3 | +| Consul + Sentinel ([3](#footnotes)) | 3 | 2 vCPU, 1.8GB Memory | `n1-highcpu-2` | `c5.large` | F2s v2 | +| Sidekiq | 4 | 2 vCPU, 7.5GB Memory | `n1-standard-2` | `m5.large` | D2s v3 | | Object Storage ([4](#footnotes)) | - | - | - | - | - | -| NFS Server ([5](#footnotes)) ([7](#footnotes)) | 1 | 4 vCPU, 3.6GB Memory | n1-highcpu-4 | c5.xlarge | F4s v2 | -| Monitoring node | 1 | 2 vCPU, 1.8GB Memory | n1-highcpu-2 | c5.large | F2s v2 | -| External load balancing node ([6](#footnotes)) | 1 | 2 vCPU, 1.8GB Memory | n1-highcpu-2 | c5.large | F2s v2 | -| Internal load balancing node ([6](#footnotes)) | 1 | 2 vCPU, 1.8GB Memory | n1-highcpu-2 | c5.large | F2s v2 | +| NFS Server ([5](#footnotes)) ([7](#footnotes)) | 1 | 4 vCPU, 3.6GB Memory | `n1-highcpu-4` | `c5.xlarge` | F4s v2 | +| Monitoring node | 1 | 2 vCPU, 1.8GB Memory | `n1-highcpu-2` | `c5.large` | F2s v2 | +| External load balancing node ([6](#footnotes)) | 1 | 2 vCPU, 1.8GB Memory | `n1-highcpu-2` | `c5.large` | F2s v2 | +| Internal load balancing node ([6](#footnotes)) | 1 | 2 vCPU, 1.8GB Memory | `n1-highcpu-2` | `c5.large` | F2s v2 | ## Footnotes @@ -53,7 +53,7 @@ For a full list of reference architectures, see 1. NFS can be used as an alternative for both repository data (replacing Gitaly) and object storage but this isn't typically recommended for performance reasons. Note however it is required for - [GitLab Pages](https://gitlab.com/gitlab-org/gitlab-pages/issues/196). + [GitLab Pages](https://gitlab.com/gitlab-org/gitlab-pages/-/issues/196). 1. Our architectures have been tested and validated with [HAProxy](https://www.haproxy.org/) as the load balancer. Although other load balancers with similar feature sets @@ -71,6 +71,3 @@ For a full list of reference architectures, see or higher, are required for your CPU or Node counts accordingly. For more information, a [Sysbench](https://github.com/akopytov/sysbench) benchmark of the CPU can be found [here](https://gitlab.com/gitlab-org/quality/performance/-/wikis/Reference-Architectures/GCP-CPU-Benchmarks). - -1. AWS-equivalent and Azure-equivalent configurations are rough suggestions - and may change in the future. They have not yet been tested and validated. diff --git a/doc/administration/reference_architectures/index.md b/doc/administration/reference_architectures/index.md index 26244368234..623d7f3f776 100644 --- a/doc/administration/reference_architectures/index.md +++ b/doc/administration/reference_architectures/index.md @@ -126,8 +126,8 @@ As long as at least one of each component is online and capable of handling the By adding automatic failover for database systems, you can enable higher uptime with additional database nodes. This extends the default database with cluster management and failover policies. -[PgBouncer](../../development/architecture.md#pgbouncer) in conjunction with -[Repmgr](../high_availability/database.md) is recommended. +[PgBouncer in conjunction with Repmgr](../postgresql/replication_and_failover.md) +is recommended. ### Instance level replication with GitLab Geo **(PREMIUM ONLY)** @@ -141,6 +141,9 @@ that can also be promoted in case of disaster. ## Configure GitLab to scale +NOTE: **Note:** +From GitLab 13.0, using NFS for Git repositories is deprecated. In GitLab 14.0, support for NFS for Git repositories is scheduled to be removed. Upgrade to [Gitaly Cluster](../gitaly/praefect.md) as soon as possible. + The following components are the ones you need to configure in order to scale GitLab. They are listed in the order you'll typically configure them if they are required by your [reference architecture](#reference-architectures) of choice. @@ -160,14 +163,33 @@ column. | [Consul](../../development/architecture.md#consul) ([3](#footnotes)) | Service discovery and health checks/failover | [Consul configuration](../high_availability/consul.md) **(PREMIUM ONLY)** | Yes | | [PostgreSQL](../../development/architecture.md#postgresql) | Database | [PostgreSQL configuration](https://docs.gitlab.com/omnibus/settings/database.html) | Yes | | [PgBouncer](../../development/architecture.md#pgbouncer) | Database connection pooler | [PgBouncer configuration](../high_availability/pgbouncer.md#running-pgbouncer-as-part-of-a-non-ha-gitlab-installation) **(PREMIUM ONLY)** | Yes | -| Repmgr | PostgreSQL cluster management and failover | [PostgreSQL and Repmgr configuration](../high_availability/database.md) | Yes | +| Repmgr | PostgreSQL cluster management and failover | [PostgreSQL and Repmgr configuration](../postgresql/replication_and_failover.md) | Yes | | [Redis](../../development/architecture.md#redis) ([3](#footnotes)) | Key/value store for fast data lookup and caching | [Redis configuration](../high_availability/redis.md) | Yes | | Redis Sentinel | Redis | [Redis Sentinel configuration](../high_availability/redis.md) | Yes | -| [Gitaly](../../development/architecture.md#gitaly) ([2](#footnotes)) ([7](#footnotes)) ([10](#footnotes)) | Provides access to Git repositories | [Gitaly configuration](../gitaly/index.md#running-gitaly-on-its-own-server) | Yes | +| [Gitaly](../../development/architecture.md#gitaly) ([2](#footnotes)) ([7](#footnotes)) | Provides access to Git repositories | [Gitaly configuration](../gitaly/index.md#run-gitaly-on-its-own-server) | Yes | | [Sidekiq](../../development/architecture.md#sidekiq) | Asynchronous/background jobs | [Sidekiq configuration](../high_availability/sidekiq.md) | Yes | | [GitLab application services](../../development/architecture.md#unicorn)([1](#footnotes)) | Puma/Unicorn, Workhorse, GitLab Shell - serves front-end requests (UI, API, Git over HTTP/SSH) | [GitLab app scaling configuration](../high_availability/gitlab.md) | Yes | | [Prometheus](../../development/architecture.md#prometheus) and [Grafana](../../development/architecture.md#grafana) | GitLab environment monitoring | [Monitoring node for scaling](../high_availability/monitoring_node.md) | Yes | +### Configuring select components with Cloud Native Helm + +We also provide [Helm charts](https://docs.gitlab.com/charts/) as a Cloud Native installation +method for GitLab. For the reference architectures, select components can be set up in this +way as an alternative if so desired. + +For these kind of setups we support using the charts in an [advanced configuration](https://docs.gitlab.com/charts/#advanced-configuration) +where stateful backend components, such as the database or Gitaly, are run externally - either +via Omnibus or reputable third party services. Note that we don't currently support running the +stateful components via Helm _at large scales_. + +When designing these environments you should refer to the respective [Reference Architecture](#available-reference-architectures) +above for guidance on sizing. Components run via Helm would be similarly scaled to their Omnibus +specs, only translated into Kubernetes resources. + +For example, if you were to set up a 50k installation with the Rails nodes being run in Helm, +then the same amount of resources as given for Omnibus should be given to the Kubernetes +cluster with the Rails nodes broken down into a number of smaller Pods across that cluster. + ## Footnotes 1. In our architectures we run each GitLab Rails node using the Puma webserver @@ -177,9 +199,7 @@ column. on workload. 1. Gitaly node requirements are dependent on customer data, specifically the number of - projects and their sizes. We recommend two nodes as an absolute minimum, - and at least four nodes should be used when supporting 50,000 or more users. - We also recommend that each Gitaly node should store no more than 5TB of data + projects and their sizes. We recommend that each Gitaly node should store no more than 5TB of data and have the number of [`gitaly-ruby` workers](../gitaly/index.md#gitaly-ruby) set to 20% of available CPUs. Additional nodes should be considered in conjunction with a review of expected data size and spread based on the recommendations above. @@ -198,7 +218,7 @@ column. 1. NFS can be used as an alternative for object storage but this isn't typically recommended for performance reasons. Note however it is required for [GitLab - Pages](https://gitlab.com/gitlab-org/gitlab-pages/issues/196). + Pages](https://gitlab.com/gitlab-org/gitlab-pages/-/issues/196). 1. Our architectures have been tested and validated with [HAProxy](https://www.haproxy.org/) as the load balancer. Although other load balancers with similar feature sets @@ -216,10 +236,3 @@ column. or higher, are required for your CPU or Node counts accordingly. For more information, a [Sysbench](https://github.com/akopytov/sysbench) benchmark of the CPU can be found [here](https://gitlab.com/gitlab-org/quality/performance/-/wikis/Reference-Architectures/GCP-CPU-Benchmarks). - -1. AWS-equivalent and Azure-equivalent configurations are rough suggestions - and may change in the future. They have not yet been tested and validated. - -1. From GitLab 13.0, using NFS for Git repositories is deprecated. In GitLab - 14.0, support for NFS for Git repositories is scheduled to be removed. - Upgrade to [Gitaly Cluster](../gitaly/praefect.md) as soon as possible. diff --git a/doc/administration/reference_architectures/troubleshooting.md b/doc/administration/reference_architectures/troubleshooting.md new file mode 100644 index 00000000000..15e377fe183 --- /dev/null +++ b/doc/administration/reference_architectures/troubleshooting.md @@ -0,0 +1,329 @@ +# Troubleshooting a reference architecture set up + +This page serves as the troubleshooting documentation if you followed one of +the [reference architectures](index.md#reference-architectures). + +## Troubleshooting object storage + +### S3 API compatibility issues + +Not all S3 providers [are fully compatible](../../raketasks/backup_restore.md#other-s3-providers) +with the Fog library that GitLab uses. Symptoms include: + +```plaintext +411 Length Required +``` + +### GitLab Pages requires NFS + +If you intend to use [GitLab Pages](../../user/project/pages/index.md), this currently requires +[NFS](../high_availability/nfs.md). There is [work in progress](https://gitlab.com/gitlab-org/gitlab-pages/issues/196) +to remove this dependency. In the future, GitLab Pages may use +[object storage](https://gitlab.com/gitlab-org/gitlab/-/issues/208135). + +The dependency on disk storage also prevents Pages being deployed using the +[GitLab Helm chart](https://gitlab.com/gitlab-org/charts/gitlab/-/issues/37). + +### Incremental logging is required for CI to use object storage + +If you configure GitLab to use object storage for CI logs and artifacts, +[you must also enable incremental logging](../job_logs.md#new-incremental-logging-architecture). + +### Proxy Download + +A number of the use cases for object storage allow client traffic to be redirected to the +object storage back end, like when Git clients request large files via LFS or when +downloading CI artifacts and logs. + +When the files are stored on local block storage or NFS, GitLab has to act as a proxy. +With object storage, the default behavior is for GitLab to redirect to the object +storage device rather than proxy the request. + +The `proxy_download` setting controls this behavior: the default is generally `false`. +Verify this in the documentation for each use case. Set it to `true` to make +GitLab proxy the files rather than redirect. + +When not proxying files, GitLab returns an +[HTTP 302 redirect with a pre-signed, time-limited object storage URL](https://gitlab.com/gitlab-org/gitlab/-/issues/32117#note_218532298). +This can result in some of the following problems: + +- If GitLab is using non-secure HTTP to access the object storage, clients may generate +`https->http` downgrade errors and refuse to process the redirect. The solution to this +is for GitLab to use HTTPS. LFS, for example, will generate this error: + + ```plaintext + LFS: lfsapi/client: refusing insecure redirect, https->http + ``` + +- Clients will need to trust the certificate authority that issued the object storage +certificate, or may return common TLS errors such as: + + ```plaintext + x509: certificate signed by unknown authority + ``` + +- Clients will need network access to the object storage. Errors that might result +if this access is not in place include: + + ```plaintext + Received status code 403 from server: Forbidden + ``` + +### ETag mismatch + +Using the default GitLab settings, some object storage back-ends such as +[MinIO](https://gitlab.com/gitlab-org/gitlab/-/issues/23188) +and [Alibaba](https://gitlab.com/gitlab-org/charts/gitlab/-/issues/1564) +might generate `ETag mismatch` errors. + +When using GitLab direct upload, the +[workaround for MinIO](https://gitlab.com/gitlab-org/charts/gitlab/-/issues/1564#note_244497658) +is to use the `--compat` parameter on the server. + +We are working on a fix to GitLab component Workhorse, and also +a workaround, in the mean time, to +[allow ETag verification to be disabled](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/18175). + +## Troubleshooting Redis + +If the application node cannot connect to the Redis node, check your firewall rules and +make sure Redis can accept TCP connections under port `6379`. + +## Troubleshooting Gitaly + +### Checking versions when using standalone Gitaly nodes + +When using standalone Gitaly nodes, you must make sure they are the same version +as GitLab to ensure full compatibility. Check **Admin Area > Gitaly Servers** on +your GitLab instance and confirm all Gitaly Servers are `Up to date`. + + + +### `gitaly-debug` + +The `gitaly-debug` command provides "production debugging" tools for Gitaly and Git +performance. It is intended to help production engineers and support +engineers investigate Gitaly performance problems. + +If you're using GitLab 11.6 or newer, this tool should be installed on +your GitLab / Gitaly server already at `/opt/gitlab/embedded/bin/gitaly-debug`. +If you're investigating an older GitLab version you can compile this +tool offline and copy the executable to your server: + +```shell +git clone https://gitlab.com/gitlab-org/gitaly.git +cd cmd/gitaly-debug +GOOS=linux GOARCH=amd64 go build -o gitaly-debug +``` + +To see the help page of `gitaly-debug` for a list of supported sub-commands, run: + +```shell +gitaly-debug -h +``` + +### Commits, pushes, and clones return a 401 + +```plaintext +remote: GitLab: 401 Unauthorized +``` + +You will need to sync your `gitlab-secrets.json` file with your GitLab +app nodes. + +### Client side gRPC logs + +Gitaly uses the [gRPC](https://grpc.io/) RPC framework. The Ruby gRPC +client has its own log file which may contain useful information when +you are seeing Gitaly errors. You can control the log level of the +gRPC client with the `GRPC_LOG_LEVEL` environment variable. The +default level is `WARN`. + +You can run a gRPC trace with: + +```shell +sudo GRPC_TRACE=all GRPC_VERBOSITY=DEBUG gitlab-rake gitlab:gitaly:check +``` + +### Observing `gitaly-ruby` traffic + +[`gitaly-ruby`](../gitaly/index.md#gitaly-ruby) is an internal implementation detail of Gitaly, +so, there's not that much visibility into what goes on inside +`gitaly-ruby` processes. + +If you have Prometheus set up to scrape your Gitaly process, you can see +request rates and error codes for individual RPCs in `gitaly-ruby` by +querying `grpc_client_handled_total`. Strictly speaking, this metric does +not differentiate between `gitaly-ruby` and other RPCs, but in practice +(as of GitLab 11.9), all gRPC calls made by Gitaly itself are internal +calls from the main Gitaly process to one of its `gitaly-ruby` sidecars. + +Assuming your `grpc_client_handled_total` counter only observes Gitaly, +the following query shows you RPCs are (most likely) internally +implemented as calls to `gitaly-ruby`: + +```prometheus +sum(rate(grpc_client_handled_total[5m])) by (grpc_method) > 0 +``` + +### Repository changes fail with a `401 Unauthorized` error + +If you're running Gitaly on its own server and notice that users can +successfully clone and fetch repositories (via both SSH and HTTPS), but can't +push to them or make changes to the repository in the web UI without getting a +`401 Unauthorized` message, then it's possible Gitaly is failing to authenticate +with the other nodes due to having the wrong secrets file. + +Confirm the following are all true: + +- When any user performs a `git push` to any repository on this Gitaly node, it + fails with the following error (note the `401 Unauthorized`): + + ```shell + remote: GitLab: 401 Unauthorized + To <REMOTE_URL> + ! [remote rejected] branch-name -> branch-name (pre-receive hook declined) + error: failed to push some refs to '<REMOTE_URL>' + ``` + +- When any user adds or modifies a file from the repository using the GitLab + UI, it immediately fails with a red `401 Unauthorized` banner. +- Creating a new project and [initializing it with a README](../../gitlab-basics/create-project.md#blank-projects) + successfully creates the project but doesn't create the README. +- When [tailing the logs](https://docs.gitlab.com/omnibus/settings/logs.html#tail-logs-in-a-console-on-the-server) on an app node and reproducing the error, you get `401` errors + when reaching the `/api/v4/internal/allowed` endpoint: + + ```shell + # api_json.log + { + "time": "2019-07-18T00:30:14.967Z", + "severity": "INFO", + "duration": 0.57, + "db": 0, + "view": 0.57, + "status": 401, + "method": "POST", + "path": "\/api\/v4\/internal\/allowed", + "params": [ + { + "key": "action", + "value": "git-receive-pack" + }, + { + "key": "changes", + "value": "REDACTED" + }, + { + "key": "gl_repository", + "value": "REDACTED" + }, + { + "key": "project", + "value": "\/path\/to\/project.git" + }, + { + "key": "protocol", + "value": "web" + }, + { + "key": "env", + "value": "{\"GIT_ALTERNATE_OBJECT_DIRECTORIES\":[],\"GIT_ALTERNATE_OBJECT_DIRECTORIES_RELATIVE\":[],\"GIT_OBJECT_DIRECTORY\":null,\"GIT_OBJECT_DIRECTORY_RELATIVE\":null}" + }, + { + "key": "user_id", + "value": "2" + }, + { + "key": "secret_token", + "value": "[FILTERED]" + } + ], + "host": "gitlab.example.com", + "ip": "REDACTED", + "ua": "Ruby", + "route": "\/api\/:version\/internal\/allowed", + "queue_duration": 4.24, + "gitaly_calls": 0, + "gitaly_duration": 0, + "correlation_id": "XPUZqTukaP3" + } + + # nginx_access.log + [IP] - - [18/Jul/2019:00:30:14 +0000] "POST /api/v4/internal/allowed HTTP/1.1" 401 30 "" "Ruby" + ``` + +To fix this problem, confirm that your `gitlab-secrets.json` file +on the Gitaly node matches the one on all other nodes. If it doesn't match, +update the secrets file on the Gitaly node to match the others, then +[reconfigure the node](../restart_gitlab.md#omnibus-gitlab-reconfigure). + +### Command line tools cannot connect to Gitaly + +If you are having trouble connecting to a Gitaly node with command line (CLI) tools, and certain actions result in a `14: Connect Failed` error message, it means that gRPC cannot reach your Gitaly node. + +Verify that you can reach Gitaly via TCP: + +```shell +sudo gitlab-rake gitlab:tcp_check[GITALY_SERVER_IP,GITALY_LISTEN_PORT] +``` + +If the TCP connection fails, check your network settings and your firewall rules. If the TCP connection succeeds, your networking and firewall rules are correct. + +If you use proxy servers in your command line environment, such as Bash, these can interfere with your gRPC traffic. + +If you use Bash or a compatible command line environment, run the following commands to determine whether you have proxy servers configured: + +```shell +echo $http_proxy +echo $https_proxy +``` + +If either of these variables have a value, your Gitaly CLI connections may be getting routed through a proxy which cannot connect to Gitaly. + +To remove the proxy setting, run the following commands (depending on which variables had values): + +```shell +unset http_proxy +unset https_proxy +``` + +### Gitaly not listening on new address after reconfiguring + +When updating the `gitaly['listen_addr']` or `gitaly['prometheus_listen_addr']` values, Gitaly may continue to listen on the old address after a `sudo gitlab-ctl reconfigure`. + +When this occurs, performing a `sudo gitlab-ctl restart` will resolve the issue. This will no longer be necessary after [this issue](https://gitlab.com/gitlab-org/gitaly/issues/2521) is resolved. + +### Permission denied errors appearing in Gitaly logs when accessing repositories from a standalone Gitaly node + +If this error occurs even though file permissions are correct, it's likely that +the Gitaly node is experiencing +[clock drift](https://en.wikipedia.org/wiki/Clock_drift). + +Please ensure that the GitLab and Gitaly nodes are synchronized and use an NTP time +server to keep them synchronized if possible. + +## Troubleshooting the GitLab Rails application + +- `mount: wrong fs type, bad option, bad superblock on` + +You have not installed the necessary NFS client utilities. See step 1 above. + +- `mount: mount point /var/opt/gitlab/... does not exist` + +This particular directory does not exist on the NFS server. Ensure +the share is exported and exists on the NFS server and try to remount. + +## Troubleshooting Monitoring + +If the monitoring node is not receiving any data, check that the exporters are +capturing data. + +```shell +curl http[s]://localhost:<EXPORTER LISTENING PORT>/metric +``` + +or + +```shell +curl http[s]://localhost:<EXPORTER LISTENING PORT>/-/metric +``` diff --git a/doc/administration/repository_storage_paths.md b/doc/administration/repository_storage_paths.md index 283401dafff..87f901becf5 100644 --- a/doc/administration/repository_storage_paths.md +++ b/doc/administration/repository_storage_paths.md @@ -10,7 +10,7 @@ storage shards) to distribute the storage load between several mount points. > - You must have at least one storage path called `default`. > - The paths are defined in key-value pairs. The key is an arbitrary name you > can pick to name the file path. -> - The target directories and any of its subpaths must not be a symlink. +> - The target directories and any of its sub-paths must not be a symlink. > - No target directory may be a sub-directory of another; no nesting. Example: this is OK: @@ -57,7 +57,7 @@ storage2: Now that you've read that big fat warning above, let's edit the configuration files and add the full paths of the alternative repository storage paths. In -the example below, we add two more mountpoints that are named `nfs_1` and `nfs_2` +the example below, we add two more mount points that are named `nfs_1` and `nfs_2` respectively. NOTE: **Note:** This example uses NFS. We do not recommend using EFS for storage as it may impact GitLab's performance. See the [relevant documentation](high_availability/nfs.md#avoid-using-awss-elastic-file-system-efs) for more details. diff --git a/doc/administration/repository_storage_types.md b/doc/administration/repository_storage_types.md index a95178c01e2..825da08b66e 100644 --- a/doc/administration/repository_storage_types.md +++ b/doc/administration/repository_storage_types.md @@ -1,6 +1,6 @@ # Repository storage types **(CORE ONLY)** -> - [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/issues/28283) in GitLab 10.0. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/28283) in GitLab 10.0. > - Hashed storage became the default for new installations in GitLab 12.0 > - Hashed storage is enabled by default for new and renamed projects in GitLab 13.0. @@ -108,7 +108,7 @@ The output includes the project ID and the project name: ### Hashed object pools -> [Introduced](https://gitlab.com/gitlab-org/gitaly/issues/1606) in GitLab 12.1. +> [Introduced](https://gitlab.com/gitlab-org/gitaly/-/issues/1606) in GitLab 12.1. DANGER: **Danger:** Do not run `git prune` or `git gc` in pool repositories! This can diff --git a/doc/administration/server_hooks.md b/doc/administration/server_hooks.md index 0b8c66805ae..6df0f187a42 100644 --- a/doc/administration/server_hooks.md +++ b/doc/administration/server_hooks.md @@ -1,4 +1,7 @@ --- +stage: Create +group: Gitaly +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers type: reference, howto disqus_identifier: 'https://docs.gitlab.com/ee/administration/custom_hooks.html' --- @@ -7,7 +10,7 @@ disqus_identifier: 'https://docs.gitlab.com/ee/administration/custom_hooks.html' > **Notes:** > -> - Server hooks were [introduced](https://gitlab.com/gitlab-org/gitlab/issues/196051) in GitLab 12.8 replacing Custom Hooks. +> - Server hooks were [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/196051) in GitLab 12.8 replacing Custom Hooks. > - Server hooks must be configured on the filesystem of the GitLab server. Only GitLab server administrators will be able to complete these tasks. Please explore [webhooks](../user/project/integrations/webhooks.md) and [GitLab CI/CD](../ci/README.md) as an option if you do not have filesystem access. For a user-configurable Git hook interface, see [Push Rules](../push_rules/push_rules.md), available in GitLab Starter **(STARTER)**. > - Server hooks won't be replicated to secondary nodes if you use [GitLab Geo](geo/replication/index.md). @@ -65,9 +68,18 @@ Follow the steps below to properly set up a server hook for all repositories: `/home/git/gitlab-shell/hooks`. For Omnibus installs the path is usually `/opt/gitlab/embedded/service/gitlab-shell/hooks`. To look in a different directory for the global custom hooks, - set `custom_hooks_dir` in the GitLab Shell config. For - Omnibus installations, this can be set in `gitlab.rb`; and in source - installations, this can be set in `gitlab-shell/config.yml`. + set `custom_hooks_dir` in the Gitaly config. For Omnibus installations, this is set + in `gitlab.rb`. For source installations, the configuration location depends on the + GitLab version. For: + + - GitLab 13.0 and earlier, this is set in `gitlab-shell/config.yml`. + - GitLab 13.1 and later, this is set in `gitaly/config.toml` under the `[hooks]` + section. + + NOTE: **Note:** + The `custom_hooks_dir` value in `gitlab-shell/config.yml` is still honored in GitLab + 13.1 and later if the value in `gitaly/config.toml` is blank or non-existent. + 1. Create a new directory in this location. Depending on your hook, it will be either a `pre-receive.d`, `post-receive.d`, or `update.d` directory. 1. Inside this new directory, add your hook. Hooks can be diff --git a/doc/administration/snippets/index.md b/doc/administration/snippets/index.md index e6bbfa8cf00..cf3d8bec1a6 100644 --- a/doc/administration/snippets/index.md +++ b/doc/administration/snippets/index.md @@ -8,7 +8,7 @@ Adjust the snippets' settings of your GitLab instance. ## Snippets content size limit -> [Introduced](https://gitlab.com/gitlab-org/gitlab/issues/31133) in GitLab 12.6. +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/31133) in GitLab 12.6. You can set a content size max limit in snippets. This limit can prevent abuses of the feature. The default content size limit is **52428800 Bytes** (50MB). diff --git a/doc/administration/timezone.md b/doc/administration/timezone.md index d6112c45141..3cfee8630b7 100644 --- a/doc/administration/timezone.md +++ b/doc/administration/timezone.md @@ -15,7 +15,7 @@ To see all available time zones, run `bundle exec rake time:zones:all`. For Omnibus installations, run `gitlab-rake time:zones:all`. NOTE: **Note:** -Currently, this Rake task does not list timezones in TZInfo format required by Omnibus GitLab during a reconfigure: [#58672](https://gitlab.com/gitlab-org/gitlab-foss/issues/58672). +Currently, this Rake task does not list timezones in TZInfo format required by Omnibus GitLab during a reconfigure: [#27209](https://gitlab.com/gitlab-org/gitlab/-/issues/27209). ## Changing time zone in Omnibus installations diff --git a/doc/administration/troubleshooting/debug.md b/doc/administration/troubleshooting/debug.md index 1e1b2ad8378..55fd6462bc3 100644 --- a/doc/administration/troubleshooting/debug.md +++ b/doc/administration/troubleshooting/debug.md @@ -244,7 +244,7 @@ separate Rails process to debug the issue: For example: ```ruby - app.get 'https://gitlab.com/gitlab-org/gitlab-foss/issues/1?private_token=123456' + app.get 'https://gitlab.com/gitlab-org/gitlab-foss/-/issues/1?private_token=123456' ``` 1. In a new window, run `top`. It should show this Ruby process using 100% CPU. Write down the PID. diff --git a/doc/administration/troubleshooting/elasticsearch.md b/doc/administration/troubleshooting/elasticsearch.md index a39fe4ba8c3..12b82e4bc48 100644 --- a/doc/administration/troubleshooting/elasticsearch.md +++ b/doc/administration/troubleshooting/elasticsearch.md @@ -189,7 +189,7 @@ Moving past that, it is best to attempt the same search using the [Elasticsearch If the results: -- Sync up, then there is not a technical "issue" per se. Instead, it might be a problem +- Sync up, then there is not a technical "issue." Instead, it might be a problem with the Elasticsearch filters we are using. This can be complicated, so it is best to escalate to GitLab support to check these and guide you on the potential on whether or not a feature request is needed. @@ -330,10 +330,10 @@ feel free to update that page with issues you encounter and solutions. Setting up Elasticsearch isn't too bad, but it can be a bit finicky and time consuming. -The easiest method is to spin up a docker container with the required version and +The easiest method is to spin up a Docker container with the required version and bind ports 9200/9300 so it can be used. -The following is an example of running a docker container of Elasticsearch v7.2.0: +The following is an example of running a Docker container of Elasticsearch v7.2.0: ```shell docker pull docker.elastic.co/elasticsearch/elasticsearch:7.2.0 @@ -342,7 +342,7 @@ docker run -p 9200:9200 -p 9300:9300 -e "discovery.type=single-node" docker.elas From here, you can: -- Grab the IP of the docker container (use `docker inspect <container_id>`) +- Grab the IP of the Docker container (use `docker inspect <container_id>`) - Use `<IP.add.re.ss:9200>` to communicate with it. This is a quick method to test out Elasticsearch, but by no means is this a diff --git a/doc/administration/troubleshooting/gitlab_rails_cheat_sheet.md b/doc/administration/troubleshooting/gitlab_rails_cheat_sheet.md index 2cbc994fb4c..33af356b37d 100644 --- a/doc/administration/troubleshooting/gitlab_rails_cheat_sheet.md +++ b/doc/administration/troubleshooting/gitlab_rails_cheat_sheet.md @@ -263,7 +263,7 @@ p.import_state.mark_as_failed("Failed manually through console.") In a specific situation, an imported repository needed to be renamed. The Support Team was informed of a backup restore that failed on a single repository, which created -the project with an empty repository. The project was successfully restored to a dev +the project with an empty repository. The project was successfully restored to a development instance, then exported, and imported into a new project under a different name. The Support Team was able to transfer the incorrectly named imported project into the @@ -302,7 +302,7 @@ you will see two pushes with the same "from" SHA: ```ruby p = Project.find_with_namespace('u/p') -p.events.code_push.last(100).each do |e| +p.events.pushed_action.last(100).each do |e| printf "%-20.20s %8s...%8s (%s)\n", e.data[:ref], e.data[:before], e.data[:after], e.author.try(:username) end ``` @@ -311,7 +311,7 @@ GitLab 9.5 and above: ```ruby p = Project.find_by_full_path('u/p') -p.events.code_push.last(100).each do |e| +p.events.pushed_action.last(100).each do |e| printf "%-20.20s %8s...%8s (%s)\n", e.push_event_payload[:ref], e.push_event_payload[:commit_from], e.push_event_payload[:commit_to], e.author.try(:username) end ``` @@ -380,39 +380,6 @@ user = User.find_by_username '' user.skip_reconfirmation! ``` -### Get an admin token - -```ruby -# Get the first admin's first access token (no longer works on 11.9+. see: https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/22743) -User.where(admin:true).first.personal_access_tokens.first.token - -# Get the first admin's private token (no longer works on 10.2+) -User.where(admin:true).private_token -``` - -### Create personal access token - -```ruby -personal_access_token = User.find(123).personal_access_tokens.create( - name: 'apitoken', - impersonation: false, - scopes: [:api] -) - -puts personal_access_token.token -``` - -You might also want to manually set the token string: - -```ruby -User.find(123).personal_access_tokens.create( - name: 'apitoken', - token_digest: Gitlab::CryptoHelper.sha256('some-token-string-here'), - impersonation: false, - scopes: [:api] -) -``` - ### Active users & Historical users ```ruby @@ -518,7 +485,7 @@ group.project_creation_level=0 ### Remove redirecting routes -See <https://gitlab.com/gitlab-org/gitlab-foss/issues/41758#note_54828133>. +See <https://gitlab.com/gitlab-org/gitlab-foss/-/issues/41758#note_54828133>. ```ruby path = 'foo' @@ -576,7 +543,7 @@ This section has been moved to the [job artifacts troubleshooting documentation] ### Find reason failure (for when build trace is empty) (Introduced in 10.3.0) -See <https://gitlab.com/gitlab-org/gitlab-foss/issues/41111>. +See <https://gitlab.com/gitlab-org/gitlab-foss/-/issues/41111>. ```ruby build = Ci::Build.find(78420) @@ -620,10 +587,26 @@ Gitlab::CurrentSettings.current_application_settings.runners_registration_token ## License -### See license plan name (since v9.3.0-ee) +### See current license information ```ruby +# License information (name, company, email address) +License.current.licensee + +# Plan: License.current.plan + +# Uploaded: +License.current.created_at + +# Started: +License.current.starts_at + +# Expires at: +License.current.expires_at + +# Is this a trial license? +License.current.trial? ``` ### Check if a project feature is available on the instance @@ -636,7 +619,7 @@ License.current.feature_available?(:jira_dev_panel_integration) ### Check if a project feature is available in a project -Features listed in <https://gitlab.com/gitlab-org/gitlab/blob/master/ee/app/models/license.rb>. +Features listed in [`license.rb`](https://gitlab.com/gitlab-org/gitlab/blob/master/ee/app/models/license.rb). ```ruby p = Project.find_by_full_path('<group>/<project>') diff --git a/doc/administration/troubleshooting/index.md b/doc/administration/troubleshooting/index.md new file mode 100644 index 00000000000..50f192b1983 --- /dev/null +++ b/doc/administration/troubleshooting/index.md @@ -0,0 +1,20 @@ +# Troubleshooting a GitLab installation + +Below are some resources to help you troubleshoot a GitLab installation +in case something goes wrong: + +- [Debugging tips](debug.md) +- [Diagnostics tools](diagnostics_tools.md) +- [Elasticsearch](elasticsearch.md) +- [GitLab Rails console cheat sheet](gitlab_rails_cheat_sheet.md) +- [Group SAML and SCIM troubleshooting](group_saml_scim.md) **(SILVER ONLY)** +- [Kubernetes cheat sheet](kubernetes_cheat_sheet.md) +- [Linux cheat sheet](linux_cheat_sheet.md) +- [Parsing GitLab logs with `jq`](log_parsing.md) +- [Navigating GitLab via Rails console](navigating_gitlab_via_rails_console.md) +- [PostgreSQL](postgresql.md) +- [Sidekiq](sidekiq.md) +- [SSL](ssl.md) + +If you need a testing environment to troubleshoot, see the +[apps for a testing environment](test_environments.md). diff --git a/doc/administration/troubleshooting/kubernetes_cheat_sheet.md b/doc/administration/troubleshooting/kubernetes_cheat_sheet.md index cab073b9924..01532032b49 100644 --- a/doc/administration/troubleshooting/kubernetes_cheat_sheet.md +++ b/doc/administration/troubleshooting/kubernetes_cheat_sheet.md @@ -85,7 +85,7 @@ and they will assist you with any issues you are having. ## GitLab-specific Kubernetes information - Minimal config that can be used to test a Kubernetes Helm chart can be found - [here](https://gitlab.com/gitlab-org/charts/gitlab/issues/620). + [here](https://gitlab.com/gitlab-org/charts/gitlab/-/issues/620). - Tailing logs of a separate pod. An example for a Webservice pod: @@ -186,7 +186,7 @@ and they will assist you with any issues you are having. helm upgrade <release name> <chart path> -f gitlab.yaml ``` - After <https://gitlab.com/gitlab-org/charts/gitlab/issues/780> is fixed, it should + After <https://gitlab.com/gitlab-org/charts/gitlab/-/issues/780> is fixed, it should be possible to use [Updating GitLab using the Helm Chart](https://docs.gitlab.com/charts/index.html#updating-gitlab-using-the-helm-chart) for upgrades. diff --git a/doc/administration/troubleshooting/postgresql.md b/doc/administration/troubleshooting/postgresql.md index 65a6bffca44..e5a4dffb3cc 100644 --- a/doc/administration/troubleshooting/postgresql.md +++ b/doc/administration/troubleshooting/postgresql.md @@ -31,7 +31,7 @@ This section is for links to information elsewhere in the GitLab documentation. - Destructively reseeding the GitLab database. - Guidance around updating packaged PostgreSQL, including how to stop it happening automatically. -- [More about external PostgreSQL](../external_database.md) +- [More about external PostgreSQL](../postgresql/external.md) - [Running Geo with external PostgreSQL](../geo/replication/external_database.md) @@ -45,8 +45,8 @@ This section is for links to information elsewhere in the GitLab documentation. - Managing Omnibus PostgreSQL versions [from the development docs](https://docs.gitlab.com/omnibus/development/managing-postgresql-versions.html) -- [PostgreSQL scaling](../high_availability/database.md) - - including [troubleshooting](../high_availability/database.md#troubleshooting) `gitlab-ctl repmgr-check-master` and PgBouncer errors +- [PostgreSQL scaling](../postgresql/replication_and_failover.md) + - including [troubleshooting](../postgresql/replication_and_failover.md#troubleshooting) `gitlab-ctl repmgr-check-master` and PgBouncer errors - [Developer database documentation](../../development/README.md#database-guides) - some of which is absolutely not for production use. Including: - understanding EXPLAIN plans @@ -55,8 +55,8 @@ This section is for links to information elsewhere in the GitLab documentation. - [GitLab database requirements](../../install/requirements.md#database) including - Support for MySQL was removed in GitLab 12.1; [migrate to PostgreSQL](../../update/mysql_to_postgresql.md) - - required extension pg_trgm - - required extension postgres_fdw for Geo + - required extension `pg_trgm` + - required extension `postgres_fdw` for Geo - Errors like this in the `production/sidekiq` log; see: [Set default_transaction_isolation into read committed](https://docs.gitlab.com/omnibus/settings/database.html#set-default_transaction_isolation-into-read-committed): @@ -95,15 +95,15 @@ This section is for links to information elsewhere in the GitLab documentation. References: -- [Issue #1 Deadlocks with GitLab 12.1, PostgreSQL 10.7](https://gitlab.com/gitlab-org/gitlab/issues/30528) +- [Issue #1 Deadlocks with GitLab 12.1, PostgreSQL 10.7](https://gitlab.com/gitlab-org/gitlab/-/issues/30528) - [Customer ticket (internal) GitLab 12.1.6](https://gitlab.zendesk.com/agent/tickets/134307) and [Google doc (internal)](https://docs.google.com/document/d/19xw2d_D1ChLiU-MO1QzWab-4-QXgsIUcN5e_04WTKy4) -- [Issue #2 deadlocks can occur if an instance is flooded with pushes](https://gitlab.com/gitlab-org/gitlab/issues/33650). Provided for context about how GitLab code can have this sort of unanticipated effect in unusual situations. +- [Issue #2 deadlocks can occur if an instance is flooded with pushes](https://gitlab.com/gitlab-org/gitlab/-/issues/33650). Provided for context about how GitLab code can have this sort of unanticipated effect in unusual situations. ```plaintext ERROR: deadlock detected ``` -Three applicable timeouts are identified in the issue [#1](https://gitlab.com/gitlab-org/gitlab/issues/30528); our recommended settings are as follows: +Three applicable timeouts are identified in the issue [#1](https://gitlab.com/gitlab-org/gitlab/-/issues/30528); our recommended settings are as follows: ```ini deadlock_timeout = 5s @@ -111,20 +111,20 @@ statement_timeout = 15s idle_in_transaction_session_timeout = 60s ``` -Quoting from issue [#1](https://gitlab.com/gitlab-org/gitlab/issues/30528): +Quoting from issue [#1](https://gitlab.com/gitlab-org/gitlab/-/issues/30528): > "If a deadlock is hit, and we resolve it through aborting the transaction after a short period, then the retry mechanisms we already have will make the deadlocked piece of work try again, and it's unlikely we'll deadlock multiple times in a row." TIP: **Tip:** In support, our general approach to reconfiguring timeouts (applies also to the HTTP stack as well) is that it's acceptable to do it temporarily as a workaround. If it makes GitLab usable for the customer, then it buys time to understand the problem more completely, implement a hot fix, or make some other change that addresses the root cause. Generally, the timeouts should be put back to reasonable defaults once the root cause is resolved. -In this case, the guidance we had from development was to drop deadlock_timeout and/or statement_timeout but to leave the third setting at 60s. Setting idle_in_transaction protects the database from sessions potentially hanging for days. There's more discussion in [the issue relating to introducing this timeout on GitLab.com](https://gitlab.com/gitlab-com/gl-infra/production/issues/1053). +In this case, the guidance we had from development was to drop deadlock_timeout and/or statement_timeout but to leave the third setting at 60s. Setting idle_in_transaction protects the database from sessions potentially hanging for days. There's more discussion in [the issue relating to introducing this timeout on GitLab.com](https://gitlab.com/gitlab-com/gl-infra/production/-/issues/1053). PostgresSQL defaults: - `statement_timeout = 0` (never) - `idle_in_transaction_session_timeout = 0` (never) -Comments in issue [#1](https://gitlab.com/gitlab-org/gitlab/issues/30528) indicate that these should both be set to at least a number of minutes for all Omnibus installations (so they don't hang indefinitely). However, 15s for statement_timeout is very short, and will only be effective if the underlying infrastructure is very performant. +Comments in issue [#1](https://gitlab.com/gitlab-org/gitlab/-/issues/30528) indicate that these should both be set to at least a number of minutes for all Omnibus installations (so they don't hang indefinitely). However, 15s for statement_timeout is very short, and will only be effective if the underlying infrastructure is very performant. See current settings with: diff --git a/doc/administration/troubleshooting/sidekiq.md b/doc/administration/troubleshooting/sidekiq.md index ca21c038267..5109a3baff2 100644 --- a/doc/administration/troubleshooting/sidekiq.md +++ b/doc/administration/troubleshooting/sidekiq.md @@ -29,9 +29,18 @@ Example: gitlab_rails['env'] = {"SIDEKIQ_LOG_ARGUMENTS" => "1"} ``` -Please note: It is not recommend to enable this setting in production because some -Sidekiq jobs (such as sending a password reset email) take secret arguments (for -example the password reset token). +This does not log all job arguments. To avoid logging sensitive +information (for instance, password reset tokens), it logs numeric +arguments for all workers, with overrides for some specific workers +where their arguments are not sensitive. + +Example log output: + +```json +{"severity":"INFO","time":"2020-06-08T14:37:37.892Z","class":"AdminEmailsWorker","args":["[FILTERED]","[FILTERED]","[FILTERED]"],"retry":3,"queue":"admin_emails","backtrace":true,"jid":"9e35e2674ac7b12d123e13cc","created_at":"2020-06-08T14:37:37.373Z","meta.user":"root","meta.caller_id":"Admin::EmailsController#create","correlation_id":"37D3lArJmT1","uber-trace-id":"2d942cc98cc1b561:6dc94409cfdd4d77:9fbe19bdee865293:1","enqueued_at":"2020-06-08T14:37:37.410Z","pid":65011,"message":"AdminEmailsWorker JID-9e35e2674ac7b12d123e13cc: done: 0.48085 sec","job_status":"done","scheduling_latency_s":0.001012,"redis_calls":9,"redis_duration_s":0.004608,"redis_read_bytes":696,"redis_write_bytes":6141,"duration_s":0.48085,"cpu_s":0.308849,"completed_at":"2020-06-08T14:37:37.892Z","db_duration_s":0.010742} +{"severity":"INFO","time":"2020-06-08T14:37:37.894Z","class":"ActiveJob::QueueAdapters::SidekiqAdapter::JobWrapper","wrapped":"ActionMailer::DeliveryJob","queue":"mailers","args":["[FILTERED]"],"retry":3,"backtrace":true,"jid":"e47a4f6793d475378432e3c8","created_at":"2020-06-08T14:37:37.884Z","meta.user":"root","meta.caller_id":"AdminEmailsWorker","correlation_id":"37D3lArJmT1","uber-trace-id":"2d942cc98cc1b561:29344de0f966446d:5c3b0e0e1bef987b:1","enqueued_at":"2020-06-08T14:37:37.885Z","pid":65011,"message":"ActiveJob::QueueAdapters::SidekiqAdapter::JobWrapper JID-e47a4f6793d475378432e3c8: start","job_status":"start","scheduling_latency_s":0.009473} +{"severity":"INFO","time":"2020-06-08T14:39:50.648Z","class":"NewIssueWorker","args":["455","1"],"retry":3,"queue":"new_issue","backtrace":true,"jid":"a24af71f96fd129ec47f5d1e","created_at":"2020-06-08T14:39:50.643Z","meta.user":"root","meta.project":"h5bp/html5-boilerplate","meta.root_namespace":"h5bp","meta.caller_id":"Projects::IssuesController#create","correlation_id":"f9UCZHqhuP7","uber-trace-id":"28f65730f99f55a3:a5d2b62dec38dffc:48ddd092707fa1b7:1","enqueued_at":"2020-06-08T14:39:50.646Z","pid":65011,"message":"NewIssueWorker JID-a24af71f96fd129ec47f5d1e: start","job_status":"start","scheduling_latency_s":0.001144} +``` When using [Sidekiq JSON logging](../logs.md#sidekiqlog), arguments logs are limited to a maximum size of 10 kilobytes of text; @@ -86,13 +95,13 @@ sudo apt-get install linux-tools-common linux-tools-generic linux-tools-`uname - sudo yum install perf ``` -Run perf against the Sidekiq PID: +Run `perf` against the Sidekiq PID: ```shell sudo perf record -p <sidekiq_pid> ``` -Let this run for 30-60 seconds and then press Ctrl-C. Then view the perf report: +Let this run for 30-60 seconds and then press Ctrl-C. Then view the `perf` report: ```shell $ sudo perf report @@ -105,13 +114,13 @@ Samples: 348K of event 'cycles', Event count (approx.): 280908431073 0.10% ruby libc-2.12.so [.] _int_free ``` -Above you see sample output from a perf report. It shows that 97% of the CPU is +Above you see sample output from a `perf` report. It shows that 97% of the CPU is being spent inside Nokogiri and `xmlXPathNodeSetMergeAndClear`. For something this obvious you should then go investigate what job in GitLab would use Nokogiri and XPath. Combine with `TTIN` or `gdb` output to show the corresponding Ruby code where this is happening. -## The GNU Project Debugger (gdb) +## The GNU Project Debugger (`gdb`) `gdb` can be another effective tool for debugging Sidekiq. It gives you a little more interactive way to look at each thread and see what's causing problems. diff --git a/doc/administration/troubleshooting/test_environments.md b/doc/administration/troubleshooting/test_environments.md index e9db5f64446..80ccd15aa22 100644 --- a/doc/administration/troubleshooting/test_environments.md +++ b/doc/administration/troubleshooting/test_environments.md @@ -16,7 +16,7 @@ are only available internally at GitLab. ## Docker -The following were tested on docker containers running in the cloud. Support Engineers, +The following were tested on Docker containers running in the cloud. Support Engineers, please see [these docs](https://gitlab.com/gitlab-com/dev-resources/tree/master/dev-resources#running-docker-containers) on how to run Docker containers on `dev-resources`. Other setups haven't been tested, but contributions are welcome. diff --git a/doc/administration/uploads.md b/doc/administration/uploads.md index 0a300084342..620f349912c 100644 --- a/doc/administration/uploads.md +++ b/doc/administration/uploads.md @@ -168,8 +168,8 @@ The connection settings match those provided by [Fog](https://github.com/fog), a | `provider` | Always `OpenStack` for compatible hosts | `OpenStack` | | `openstack_username` | OpenStack username | | | `openstack_api_key` | OpenStack API key | | -| `openstack_temp_url_key` | OpenStack key for generating temporary urls | | -| `openstack_auth_url` | OpenStack authentication endpont | | +| `openstack_temp_url_key` | OpenStack key for generating temporary URLs | | +| `openstack_auth_url` | OpenStack authentication endpoint | | | `openstack_region` | OpenStack region | | | `openstack_tenant` | OpenStack tenant ID | |
