diff options
Diffstat (limited to 'doc/administration')
127 files changed, 3305 insertions, 2597 deletions
diff --git a/doc/administration/audit_events.md b/doc/administration/audit_events.md index 26b4434de77..e42982e6524 100644 --- a/doc/administration/audit_events.md +++ b/doc/administration/audit_events.md @@ -1,6 +1,12 @@ +--- +stage: Monitor +group: APM +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + # Audit Events **(STARTER)** -GitLab offers a way to view the changes made within the GitLab server for owners and administrators on a [paid plan][ee]. +GitLab offers a way to view the changes made within the GitLab server for owners and administrators on a [paid plan](https://about.gitlab.com/pricing/). GitLab system administrators can also take advantage of the logs located on the filesystem. See [the logs system documentation](logs.md) for more details. @@ -28,10 +34,16 @@ There are two kinds of events logged: - Instance events scoped to the whole GitLab instance, used by your Compliance team to perform formal audits. +### Impersonation data **(PREMIUM)** + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/issues/536) in [GitLab Premium](https://about.gitlab.com/pricing/) 13.0. + +Impersonation is where an administrator uses credentials to perform an action as a different user. + ### Group events **(STARTER)** NOTE: **Note:** -You need Owner [permissions] to view the group Audit Events page. +You need Owner [permissions](../user/permissions.md) to view the group Audit Events page. To view a group's audit events, navigate to **Group > Settings > Audit Events**. From there, you can see the following actions: @@ -40,14 +52,15 @@ From there, you can see the following actions: - Group repository size limit changed - Group created or deleted - Group changed visibility -- User was added to group and with which [permissions] +- User was added to group and with which [permissions](../user/permissions.md) - User sign-in via [Group SAML](../user/group/saml_sso/index.md) - Permissions changes of a user assigned to a group - Removed user from group +- Project imported in to group - Project added to group and with which visibility level - Project removed from group - [Project shared with group](../user/project/members/share_project_with_groups.md) - and with which [permissions] + and with which [permissions](../user/permissions.md) - Removal of a previously shared group with a project - LFS enabled or disabled - Shared runners minutes limit changed @@ -61,7 +74,7 @@ Group events can also be accessed via the [Group Audit Events API](../api/audit_ ### Project events **(STARTER)** NOTE: **Note:** -You need Maintainer [permissions] or higher to view the project Audit Events page. +You need Maintainer [permissions](../user/permissions.md) or higher to view the project Audit Events page. To view a project's audit events, navigate to **Project > Settings > Audit Events**. From there, you can see the following actions: @@ -69,7 +82,7 @@ From there, you can see the following actions: - Added or removed deploy keys - Project created, deleted, renamed, moved(transferred), changed path - Project changed visibility level -- User was added to project and with which [permissions] +- User was added to project and with which [permissions](../user/permissions.md) - Permission changes of a user assigned to a project - User was removed from project - Project export was downloaded @@ -86,7 +99,7 @@ From there, you can see the following actions: ### Instance events **(PREMIUM ONLY)** -> [Introduced][ee-2336] in [GitLab Premium][ee] 9.3. +> [Introduced](https://gitlab.com/gitlab-org/gitlab/issues/2336) in [GitLab Premium](https://about.gitlab.com/pricing/) 9.3. Server-wide audit logging introduces the ability to observe user actions across the entire instance of your GitLab server, making it easy to understand who @@ -157,7 +170,3 @@ the steps bellow. ```ruby Feature.enable(:repository_push_audit_event) ``` - -[ee-2336]: https://gitlab.com/gitlab-org/gitlab/issues/2336 -[ee]: https://about.gitlab.com/pricing/ -[permissions]: ../user/permissions.md diff --git a/doc/administration/auditor_users.md b/doc/administration/auditor_users.md index 2bb229d4f68..ace210183b2 100644 --- a/doc/administration/auditor_users.md +++ b/doc/administration/auditor_users.md @@ -1,6 +1,6 @@ # Auditor users **(PREMIUM ONLY)** ->[Introduced][ee-998] in [GitLab Premium][eep] 8.17. +>[Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/998) in [GitLab Premium](https://about.gitlab.com/pricing/) 8.17. Auditor users are given read-only access to all projects, groups, and other resources on the GitLab instance. @@ -12,7 +12,7 @@ snippets, etc.), and read-only access to **all** other resources, except the Admin Area. To put another way, they are just regular users (who can be added to projects, create personal snippets, create milestones on their groups, etc.) who also happen to have read-only access to all projects on the system that -they haven't been explicitly [given access][permissions] to. +they haven't been explicitly [given access](../user/permissions.md) to. The Auditor role is _not_ a read-only version of the Admin role. Auditor users will not be able to access the project/group settings pages, or the Admin Area. @@ -25,7 +25,7 @@ To sum up, assuming you have logged-in as an Auditor user: - For a project the Auditor owns, the Auditor should have full access to everything. - For a project the Auditor has been added to as a member, the Auditor should - have the same access as the [permissions] they were given to. For example, if + have the same access as the [permissions](../user/permissions.md) they were given to. For example, if they were added as a Developer, they could then push commits or comment on issues. - The Auditor cannot view the Admin Area, or perform any admin actions. @@ -82,7 +82,3 @@ instance, with the following permissions/restrictions: - Cannot create/modify files from the Web UI - Cannot merge a merge request - Cannot create project snippets - -[ee-998]: https://gitlab.com/gitlab-org/gitlab/-/merge_requests/998 -[eep]: https://about.gitlab.com/pricing/ -[permissions]: ../user/permissions.md diff --git a/doc/administration/auth/crowd.md b/doc/administration/auth/crowd.md index 6c2e4edac31..71938d4fd2b 100644 --- a/doc/administration/auth/crowd.md +++ b/doc/administration/auth/crowd.md @@ -66,14 +66,11 @@ Authenticate to GitLab using the Atlassian Crowd OmniAuth provider. 1. Change `YOUR_APP_NAME` to the application name from Crowd applications page. 1. Change `YOUR_APP_PASSWORD` to the application password you've set. 1. Save the configuration file. -1. [Reconfigure][] or [restart][] for the changes to take effect if you +1. [Reconfigure](../restart_gitlab.md#omnibus-gitlab-reconfigure) or [restart](../restart_gitlab.md#installations-from-source) for the changes to take effect if you installed GitLab via Omnibus or from source respectively. On the sign in page there should now be a Crowd tab in the sign in form. -[reconfigure]: ../restart_gitlab.md#omnibus-gitlab-reconfigure -[restart]: ../restart_gitlab.md#installations-from-source - ## Troubleshooting If you see an error message like the one below when you sign in after Crowd authentication is configured, you may want to consult the Crowd administrator for the Crowd log file to know the exact cause: diff --git a/doc/administration/auth/google_secure_ldap.md b/doc/administration/auth/google_secure_ldap.md index cb7901ea5b4..b643dd2f7b9 100644 --- a/doc/administration/auth/google_secure_ldap.md +++ b/doc/administration/auth/google_secure_ldap.md @@ -133,7 +133,7 @@ values obtained during the LDAP client configuration earlier: EOS ``` -1. Save the file and [reconfigure] GitLab for the changes to take effect. +1. Save the file and [reconfigure](../restart_gitlab.md#omnibus-gitlab-reconfigure) GitLab for the changes to take effect. --- @@ -204,10 +204,7 @@ values obtained during the LDAP client configuration earlier: -----END PRIVATE KEY----- ``` -1. Save the file and [restart] GitLab for the changes to take effect. - -[reconfigure]: ../restart_gitlab.md#omnibus-gitlab-reconfigure -[restart]: ../restart_gitlab.md#installations-from-source +1. Save the file and [restart](../restart_gitlab.md#installations-from-source) GitLab for the changes to take effect. <!-- ## Troubleshooting diff --git a/doc/administration/auth/how_to_configure_ldap_gitlab_ce/index.md b/doc/administration/auth/how_to_configure_ldap_gitlab_ce/index.md index 49bf786061e..01da4528eab 100644 --- a/doc/administration/auth/how_to_configure_ldap_gitlab_ce/index.md +++ b/doc/administration/auth/how_to_configure_ldap_gitlab_ce/index.md @@ -45,10 +45,7 @@ Configuring organizational units (**OU**s) is an important part of setting up Ac | GitLab **OU** Design | GitLab AD Structure | | :----------------------------: | :------------------------------: | -| ![GitLab OU Design][gitlab_ou] | ![GitLab AD Structure][ldap_ou] | - -[gitlab_ou]: img/gitlab_ou.png -[ldap_ou]: img/ldap_ou.gif +|  |  | Using PowerShell you can output the **OU** structure as a table (_all names are examples only_): diff --git a/doc/administration/auth/how_to_configure_ldap_gitlab_ee/index.md b/doc/administration/auth/how_to_configure_ldap_gitlab_ee/index.md index 46bc079971d..d7b32c8a92a 100644 --- a/doc/administration/auth/how_to_configure_ldap_gitlab_ee/index.md +++ b/doc/administration/auth/how_to_configure_ldap_gitlab_ee/index.md @@ -8,7 +8,7 @@ This article expands on [How to Configure LDAP with GitLab CE](../how_to_configu ## GitLab Enterprise Edition - LDAP features -[GitLab Enterprise Edition (EE)](https://about.gitlab.com/pricing/) has a number of advantages when it comes to integrating with Active Directory (LDAP): +[GitLab Enterprise Edition (EE)](https://about.gitlab.com/pricing/) has several advantages when it comes to integrating with Active Directory (LDAP): - [Administrator Sync](../ldap-ee.md#administrator-sync): As an extension of group sync, you can automatically manage your global GitLab administrators. Specify a group CN for `admin_group` and all members of the LDAP group will be given administrator privileges. - [Group Sync](#group-sync): This allows GitLab group membership to be automatically updated based on LDAP group members. @@ -16,7 +16,7 @@ This article expands on [How to Configure LDAP with GitLab CE](../how_to_configu - Daily user synchronization: Once a day, GitLab will run a synchronization to check and update GitLab users against LDAP. This process updates all user details automatically. -On the following section, you'll find a description for each of these features. Read through [LDAP GitLab EE docs](../ldap-ee.md) for complementary information. +In the following section, you'll find a description of each of these features. Read through [LDAP GitLab EE docs](../ldap-ee.md) for complementary information.  @@ -28,7 +28,7 @@ Group syncing allows AD (LDAP) groups to be mapped to GitLab groups. This provid #### Creating group links - example -As an example, let's suppose we have a "UKGov" GitLab group, which deals with confidential government information. Therefore, it is important that users of this group are given the correct permissions to projects contained within the group. Granular group permissions can be applied based on the AD group. +As an example, let's suppose we have a "UKGov" GitLab group, which deals with confidential government information. Therefore, users of this group must be given the correct permissions to projects contained within the group. Granular group permissions can be applied based on the AD group. **UK Developers** of our "UKGov" group are given **"developer"** permissions. diff --git a/doc/administration/auth/jwt.md b/doc/administration/auth/jwt.md index 15eee50c771..5a773485842 100644 --- a/doc/administration/auth/jwt.md +++ b/doc/administration/auth/jwt.md @@ -64,7 +64,7 @@ JWT will provide you with a secret key for you to use. 1. Change `YOUR_APP_SECRET` to the client secret and set `auth_url` to your redirect URL. 1. Save the configuration file. -1. [Reconfigure][] or [restart GitLab][] for the changes to take effect if you +1. [Reconfigure](../restart_gitlab.md#omnibus-gitlab-reconfigure) or [restart GitLab](../restart_gitlab.md#installations-from-source) for the changes to take effect if you installed GitLab via Omnibus or from source respectively. On the sign in page there should now be a JWT icon below the regular sign in form. @@ -72,9 +72,6 @@ Click the icon to begin the authentication process. JWT will ask the user to sign in and authorize the GitLab application. If everything goes well, the user will be redirected to GitLab and will be signed in. -[reconfigure]: ../restart_gitlab.md#omnibus-gitlab-reconfigure -[restart GitLab]: ../restart_gitlab.md#installations-from-source - <!-- ## Troubleshooting Include any troubleshooting steps that you can foresee. If you know beforehand what issues diff --git a/doc/administration/auth/ldap-ee.md b/doc/administration/auth/ldap-ee.md index 3bc30c4c01c..9e193dba08c 100644 --- a/doc/administration/auth/ldap-ee.md +++ b/doc/administration/auth/ldap-ee.md @@ -119,6 +119,9 @@ following. To take advantage of group sync, group owners or maintainers will need to [create one or more LDAP group links](#adding-group-links). +NOTE: **Note:** +If an LDAP user is a group member when LDAP Synchronization is added, and they are not part of the LDAP group, they will be removed from the group. + ### Adding group links Once [group sync has been configured](#group-sync) on the instance, one or more LDAP diff --git a/doc/administration/auth/ldap-troubleshooting.md b/doc/administration/auth/ldap-troubleshooting.md index 612f782e841..77c9c30d140 100644 --- a/doc/administration/auth/ldap-troubleshooting.md +++ b/doc/administration/auth/ldap-troubleshooting.md @@ -312,7 +312,7 @@ things to check to debug the situation. interval](ldap-ee.md#adjusting-ldap-group-sync-schedule) for the group to sync. To speed up the process, either go to the GitLab group **Settings -> Members** and press **Sync now** (sync one group) or [run the group sync Rake - task](../raketasks/ldap.md#run-a-group-sync) (sync all groups). + task](../raketasks/ldap.md#run-a-group-sync-starter-only) (sync all groups). If all of the above looks good, jump in to a little more advanced debugging in the rails console. @@ -352,7 +352,7 @@ GitLab syncs the `admin_group`. NOTE: **NOTE:** To sync all groups manually when debugging is unnecessary, [use the Rake -task](../raketasks/ldap.md#run-a-group-sync) instead. +task](../raketasks/ldap.md#run-a-group-sync-starter-only) instead. The output from a manual [group sync](ldap-ee.md#group-sync) can show you what happens when GitLab syncs its LDAP group memberships against LDAP. diff --git a/doc/administration/auth/oidc.md b/doc/administration/auth/oidc.md index 76e83c8e0ad..7a808636e94 100644 --- a/doc/administration/auth/oidc.md +++ b/doc/administration/auth/oidc.md @@ -94,7 +94,7 @@ The OpenID Connect will provide you with a client details and secret for you to - `basic` - HTTP Basic Authentication - `jwt_bearer` - JWT based authentication (private key and client secret signing) - `mtls` - Mutual TLS or X.509 certificate validation - - Any other value will POST the client id and secret in the request body + - Any other value will POST the client ID and secret in the request body - If not specified, defaults to `basic`. - `<uid_field>` (optional) is the field name from the `user_info` details that will be used as `uid` value. For example, `preferred_username`. If this value is not provided or the field with the configured value is missing from the `user_info` details, the `uid` will use the `sub` field. diff --git a/doc/administration/auth/smartcard.md b/doc/administration/auth/smartcard.md index 8986a9866e7..7131fd7571f 100644 --- a/doc/administration/auth/smartcard.md +++ b/doc/administration/auth/smartcard.md @@ -37,7 +37,7 @@ To use a smartcard with an X.509 certificate to authenticate against a local database with GitLab, `CN` and `emailAddress` must be defined in the certificate. For example: -```text +```plaintext Certificate: Data: Version: 1 (0x0) @@ -73,7 +73,7 @@ database with GitLab, in: For example: -```text +```plaintext Certificate: Data: Version: 1 (0x0) diff --git a/doc/administration/availability/index.md b/doc/administration/availability/index.md index a0d4ea7919f..748373c8941 100644 --- a/doc/administration/availability/index.md +++ b/doc/administration/availability/index.md @@ -1,139 +1,5 @@ --- -type: reference, concepts +redirect_to: ../reference_architectures/index.md --- -# Availability - -GitLab offers high availability options for organizations that require -the fault tolerance and redundancy necessary to maintain high-uptime operations. - -Please consult our [scaling documentation](../scaling) if you want to resolve -performance bottlenecks you encounter in individual GitLab components without -incurring the additional complexity costs associated with maintaining a -highly-available architecture. - -On this page, we present examples of self-managed instances which demonstrate -how GitLab can be scaled out and made highly available. These examples progress -from simple to complex as scaling or highly-available components are added. - -For larger setups serving 2,000 or more users, we provide -[reference architectures](../scaling/index.md#reference-architectures) based on GitLab's -experience with GitLab.com and internal scale testing that aim to achieve the -right balance of scalability and availability. - -For detailed insight into how GitLab scales and configures GitLab.com, you can -watch [this 1 hour Q&A](https://www.youtube.com/watch?v=uCU8jdYzpac) -with [John Northrup](https://gitlab.com/northrup), and live questions coming -in from some of our customers. - -GitLab offers a number of options to manage availability and resiliency. Below are the options to consider with trade-offs. - -| Event | GitLab Feature | Recovery Point Objective (RPO) | Recovery Time Objective (RTO) | Cost | -| ----- | -------------- | --- | --- | ---- | -| Availability Zone failure | "GitLab HA" | No loss | No loss | 2x Git storage, multiple nodes balanced across AZ's | -| Region failure | "GitLab Disaster Recovery" | 5-10 minutes | 30 minutes | 2x primary cost | -| All failures | Backup/Restore | Last backup | Hours to Days | Cost of storing the backups | - -## High availability - -### Omnibus installation with automatic database failover - -By adding automatic failover for database systems, we can enable higher uptime with an additional layer of complexity. - -- For PostgreSQL, we provide repmgr for server cluster management and failover - and a combination of [PgBouncer](../high_availability/pgbouncer.md) and [Consul](../high_availability/consul.md) for - database client cutover. -- For Redis, we use [Redis Sentinel](../high_availability/redis.md) for server failover and client cutover. - -You can also optionally run [additional Sidekiq processes on dedicated hardware](../high_availability/sidekiq.md) -and configure individual Sidekiq processes to -[process specific background job queues](../operations/extra_sidekiq_processes.md) -if you need to scale out background job processing. - -### GitLab Geo - -GitLab Geo allows you to replicate your GitLab instance to other geographical locations as a read-only fully operational instance that can also be promoted in case of disaster. - -This configuration is supported in [GitLab Premium and Ultimate](https://about.gitlab.com/pricing/). - -References: - -- [Geo Documentation](../geo/replication/index.md) -- [GitLab Geo with a highly available configuration](../geo/replication/high_availability.md) - -## GitLab components and configuration instructions - -The GitLab application depends on the following [components](../../development/architecture.md#component-diagram). -It can also depend on several third party services depending on -your environment setup. Here we'll detail both in the order in which -you would typically configure them along with our recommendations for -their use and configuration. - -### Third party services - -Here's some details of several third party services a typical environment -will depend on. The services can be provided by numerous applications -or providers and further advice can be given on how best to select. -These should be configured first, before the [GitLab components](#gitlab-components). - -| Component | Description | Configuration instructions | -|--------------------------------------------------------|---------------------------------------------------------------------------------------------------------------------|---------------------------------------------------------| -| [Load Balancer(s)](../high_availability/load_balancer.md)[^6] | Handles load balancing for the GitLab nodes where required | [Load balancer HA configuration](../high_availability/load_balancer.md) | -| [Cloud Object Storage service](../high_availability/object_storage.md)[^4] | Recommended store for shared data objects | [Cloud Object Storage configuration](../high_availability/object_storage.md) | -| [NFS](../high_availability/nfs.md)[^5] [^7] | Shared disk storage service. Can be used as an alternative for Gitaly or Object Storage. Required for GitLab Pages | [NFS configuration](../high_availability/nfs.md) | - -### GitLab components - -Next are all of the components provided directly by GitLab. As mentioned -earlier, they are presented in the typical order you would configure -them. - -| Component | Description | Configuration instructions | -|---------------------------------------------------------------------------------------------------------------------|---------------------------------------------------------------------|---------------------------------------------------------------| -| [Consul](../../development/architecture.md#consul)[^3] | Service discovery and health checks/failover | [Consul HA configuration](../high_availability/consul.md) **(PREMIUM ONLY)** | -| [PostgreSQL](../../development/architecture.md#postgresql) | Database | [Database HA configuration](../high_availability/database.md) | -| [PgBouncer](../../development/architecture.md#pgbouncer) | Database Pool Manager | [PgBouncer HA configuration](../high_availability/pgbouncer.md) **(PREMIUM ONLY)** | -| [Redis](../../development/architecture.md#redis)[^3] with Redis Sentinel | Key/Value store for shared data with HA watcher service | [Redis HA configuration](../high_availability/redis.md) | -| [Gitaly](../../development/architecture.md#gitaly)[^2] [^5] [^7] | Recommended high-level storage for Git repository data | [Gitaly HA configuration](../high_availability/gitaly.md) | -| [Sidekiq](../../development/architecture.md#sidekiq) | Asynchronous/Background jobs | [Sidekiq configuration](../high_availability/sidekiq.md) | -| [GitLab application nodes](../../development/architecture.md#unicorn)[^1] | (Unicorn / Puma, Workhorse) - Web-requests (UI, API, Git over HTTP) | [GitLab app HA/scaling configuration](../high_availability/gitlab.md) | -| [Prometheus](../../development/architecture.md#prometheus) and [Grafana](../../development/architecture.md#grafana) | GitLab environment monitoring | [Monitoring node for scaling/HA](../high_availability/monitoring_node.md) | - -In some cases, components can be combined on the same nodes to reduce complexity as well. - -[^1]: In our architectures we run each GitLab Rails node using the Puma webserver - and have its number of workers set to 90% of available CPUs along with 4 threads. - -[^2]: Gitaly node requirements are dependent on customer data, specifically the number of - projects and their sizes. We recommend 2 nodes as an absolute minimum for HA environments - and at least 4 nodes should be used when supporting 50,000 or more users. - We also recommend that each Gitaly node should store no more than 5TB of data - and have the number of [`gitaly-ruby` workers](../gitaly/index.md#gitaly-ruby) - set to 20% of available CPUs. Additional nodes should be considered in conjunction - with a review of expected data size and spread based on the recommendations above. - -[^3]: Recommended Redis setup differs depending on the size of the architecture. - For smaller architectures (up to 5,000 users) we suggest one Redis cluster for all - classes and that Redis Sentinel is hosted alongside Consul. - For larger architectures (10,000 users or more) we suggest running a separate - [Redis Cluster](../high_availability/redis.md#running-multiple-redis-clusters) for the Cache class - and another for the Queues and Shared State classes respectively. We also recommend - that you run the Redis Sentinel clusters separately as well for each Redis Cluster. - -[^4]: For data objects such as LFS, Uploads, Artifacts, etc. We recommend a [Cloud Object Storage service](../object_storage.md) - over NFS where possible, due to better performance and availability. - -[^5]: NFS can be used as an alternative for both repository data (replacing Gitaly) and - object storage but this isn't typically recommended for performance reasons. Note however it is required for - [GitLab Pages](https://gitlab.com/gitlab-org/gitlab-pages/issues/196). - -[^6]: Our architectures have been tested and validated with [HAProxy](https://www.haproxy.org/) - as the load balancer. However other reputable load balancers with similar feature sets - should also work instead but be aware these aren't validated. - -[^7]: We strongly recommend that any Gitaly and / or NFS nodes are set up with SSD disks over - HDD with a throughput of at least 8,000 IOPS for read operations and 2,000 IOPS for write - as these components have heavy I/O. These IOPS values are recommended only as a starter - as with time they may be adjusted higher or lower depending on the scale of your - environment's workload. If you're running the environment on a Cloud provider - you may need to refer to their documentation on how configure IOPS correctly. +This document was moved to [another location](../reference_architectures/index.md). diff --git a/doc/administration/database_load_balancing.md b/doc/administration/database_load_balancing.md index b6ce6883dd6..0f566fcc114 100644 --- a/doc/administration/database_load_balancing.md +++ b/doc/administration/database_load_balancing.md @@ -1,6 +1,6 @@ # Database Load Balancing **(PREMIUM ONLY)** -> [Introduced][ee-1283] in [GitLab Premium][eep] 9.0. +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/1283) in [GitLab Premium](https://about.gitlab.com/pricing/) 9.0. Distribute read-only queries among multiple database servers. @@ -11,7 +11,7 @@ multiple computing resources. Load balancing aims to optimize resource use, maximize throughput, minimize response time, and avoid overload of any single resource. Using multiple components with load balancing instead of a single component may increase reliability and availability through redundancy. -[_Wikipedia article_][wikipedia] +[_Wikipedia article_](https://en.wikipedia.org/wiki/Load_balancing_(computing)) When database load balancing is enabled in GitLab, the load is balanced using a simple round-robin algorithm, without any external dependencies such as Redis. @@ -26,9 +26,9 @@ sent to the primary (unless necessary), the primary (`db3`) hardly has any load. ## Requirements -For load balancing to work you will need at least PostgreSQL 9.2 or newer, -[**MySQL is not supported**][db-req]. You also need to make sure that you have -at least 1 secondary in [hot standby][hot-standby] mode. +For load balancing to work you will need at least PostgreSQL 11 or newer, +[**MySQL is not supported**](../install/requirements.md#database). You also need to make sure that you have +at least 1 secondary in [hot standby](https://www.postgresql.org/docs/11/hot-standby.html) mode. Load balancing also requires that the configured hosts **always** point to the primary, even after a database failover. Furthermore, the additional hosts to @@ -78,7 +78,7 @@ the following. This will balance the load between `host1.example.com` and gitlab_rails['db_load_balancing'] = { 'hosts' => ['host1.example.com', 'host2.example.com'] } ``` -1. Save the file and [reconfigure GitLab][] for the changes to take effect. +1. Save the file and [reconfigure GitLab](restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. --- @@ -97,11 +97,11 @@ the following. This will balance the load between `host1.example.com` and - host2.example.com ``` -1. Save the file and [restart GitLab][] for the changes to take effect. +1. Save the file and [restart GitLab](restart_gitlab.md#installations-from-source) for the changes to take effect. ## Service Discovery -> [Introduced][ee-5883] in [GitLab Premium][eep] 11.0. +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/5883) in [GitLab Premium](https://about.gitlab.com/pricing/) 11.0. Service discovery allows GitLab to automatically retrieve a list of secondary databases to use, instead of having to manually specify these in the @@ -161,12 +161,16 @@ When the list of hosts is updated, it might take a while for the old connections to be terminated. The `disconnect_timeout` setting can be used to enforce an upper limit on the time it will take to terminate all old database connections. -Some nameservers (like [Consul][consul-udp]) can return a truncated list of hosts when +Some nameservers (like [Consul](https://www.consul.io/docs/agent/dns.html#udp-based-dns-queries)) can return a truncated list of hosts when queried over UDP. To overcome this issue, you can use TCP for querying by setting `use_tcp` to `true`. ### Forking +NOTE: **Note:** +Starting with GitLab 13.0, Puma is the default web server used in GitLab +all-in-one package based installations as well as GitLab Helm chart deployments. + If you use an application server that forks, such as Unicorn, you _have to_ update your Unicorn configuration to start service discovery _after_ a fork. Failure to do so will lead to service discovery only running in the parent @@ -239,14 +243,14 @@ For example: ## Handling Stale Reads -> [Introduced][ee-3526] in [GitLab Premium][eep] 10.3. +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/3526) in [GitLab Premium](https://about.gitlab.com/pricing/) 10.3. To prevent reading from an outdated secondary the load balancer will check if it is in sync with the primary. If the data is determined to be recent enough the secondary can be used, otherwise it will be ignored. To reduce the overhead of these checks we only perform these checks at certain intervals. -There are three configuration options that influence this behaviour: +There are three configuration options that influence this behavior: | Option | Description | Default | |------------------------------|----------------------------------------------------------------------------------------------------------------|------------| @@ -270,14 +274,3 @@ production: max_replication_lag_time: 30 replica_check_interval: 30 ``` - -[hot-standby]: https://www.postgresql.org/docs/9.6/hot-standby.html -[ee-1283]: https://gitlab.com/gitlab-org/gitlab/-/merge_requests/1283 -[eep]: https://about.gitlab.com/pricing/ -[reconfigure gitlab]: restart_gitlab.md#omnibus-gitlab-reconfigure "How to reconfigure Omnibus GitLab" -[restart gitlab]: restart_gitlab.md#installations-from-source "How to restart GitLab" -[wikipedia]: https://en.wikipedia.org/wiki/Load_balancing_(computing) -[db-req]: ../install/requirements.md#database -[ee-3526]: https://gitlab.com/gitlab-org/gitlab/-/merge_requests/3526 -[ee-5883]: https://gitlab.com/gitlab-org/gitlab/-/merge_requests/5883 -[consul-udp]: https://www.consul.io/docs/agent/dns.html#udp-based-dns-queries diff --git a/doc/administration/external_database.md b/doc/administration/external_database.md index ec2d30c82d1..47509828c20 100644 --- a/doc/administration/external_database.md +++ b/doc/administration/external_database.md @@ -5,7 +5,7 @@ managed service for PostgreSQL. For example, AWS offers a managed Relational Database Service (RDS) that runs PostgreSQL. Alternatively, you may opt to manage your own PostgreSQL instance or cluster -separate from the GitLab Omnibus package. +separate from the Omnibus GitLab package. If you use a cloud-managed service, or provide your own PostgreSQL instance: @@ -13,5 +13,29 @@ If you use a cloud-managed service, or provide your own PostgreSQL instance: [database requirements document](../install/requirements.md#database). 1. Set up a `gitlab` username with a password of your choice. The `gitlab` user needs privileges to create the `gitlabhq_production` database. -1. Configure the GitLab application servers with the appropriate details. - This step is covered in [Configuring GitLab for HA](high_availability/gitlab.md). +1. If you are using a cloud-managed service, you may need to grant additional + roles to your `gitlab` user: + - Amazon RDS requires the [`rds_superuser`](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Appendix.PostgreSQL.CommonDBATasks.html#Appendix.PostgreSQL.CommonDBATasks.Roles) role. + - Azure Database for PostgreSQL requires the [`azure_pg_admin`](https://docs.microsoft.com/en-us/azure/postgresql/howto-create-users#how-to-create-additional-admin-users-in-azure-database-for-postgresql) role. + +1. Configure the GitLab application servers with the appropriate connection details + for your external PostgreSQL service in your `/etc/gitlab/gitlab.rb` file: + + ```ruby + # Disable the bundled Omnibus provided PostgreSQL + postgresql['enable'] = false + + # PostgreSQL connection details + gitlab_rails['db_adapter'] = 'postgresql' + gitlab_rails['db_encoding'] = 'unicode' + gitlab_rails['db_host'] = '10.1.0.5' # IP/hostname of database server + gitlab_rails['db_password'] = 'DB password' + ``` + + For more information on GitLab HA setups, refer to [configuring GitLab for HA](high_availability/gitlab.md). + +1. Reconfigure for the changes to take effect: + + ```shell + sudo gitlab-ctl reconfigure + ``` diff --git a/doc/administration/external_pipeline_validation.md b/doc/administration/external_pipeline_validation.md index 19d4de3b705..489d1924803 100644 --- a/doc/administration/external_pipeline_validation.md +++ b/doc/administration/external_pipeline_validation.md @@ -20,7 +20,7 @@ Response Code Legend: ## Configuration -Set the `EXTERNAL_VALIDATION_SERVICE_URL` to the external service url. +Set the `EXTERNAL_VALIDATION_SERVICE_URL` to the external service URL. ## Payload Schema diff --git a/doc/administration/feature_flags.md b/doc/administration/feature_flags.md new file mode 100644 index 00000000000..59cd5497032 --- /dev/null +++ b/doc/administration/feature_flags.md @@ -0,0 +1,99 @@ +--- +type: reference +description: "GitLab administrator: enable and disable GitLab features deployed behind feature flags" +--- + +# Enable and disable GitLab features deployed behind feature flags **(CORE ONLY)** + +GitLab adopted [feature flags strategies](../development/feature_flags/index.md) +to deploy features in an early stage of development so that they can be +incrementally rolled out. + +Before making them permanently available, features can be deployed behind +flags for a [number of reasons](../development/feature_flags/process.md#when-to-use-feature-flags), such as: + +- To test the feature. +- To get feedback from users and customers while in an early stage of the development of the feature. +- To evaluate users adoption. +- To evaluate how it impacts GitLab's performance. +- To build it in smaller pieces throughout releases. + +Features behind flags can be gradually rolled out, typically: + +1. The feature starts disabled by default. +1. The feature becomes enabled by default. +1. The feature flag is removed. + +These features can be enabled and disabled to allow or disallow users to use +them. It can be done by GitLab administrators with access to GitLab Rails +console. + +If you used a certain feature and identified a bug, a misbehavior, or an +error, it's very important that you [**provide feedback**](https://gitlab.com/gitlab-org/gitlab/issues/new?issue[title]=Docs%20-%20feature%20flag%20feedback%3A%20Feature%20Name&issue[description]=Describe%20the%20problem%20you%27ve%20encountered.%0A%0A%3C!--%20Don%27t%20edit%20below%20this%20line%20--%3E%0A%0A%2Flabel%20~%22docs%5C-comments%22%20) to GitLab as soon +as possible so we can improve or fix it while behind a flag. When you upgrade +GitLab to an earlier version, the feature flag status may change. + +NOTE: **Note:** +Mind that features deployed behind feature flags may not be ready for +production use. However, disabling features behind flags that were deployed +enabled by default may also present a risk. If they're enabled, we recommend +you leave them as-is. + +## How to enable and disable features behind flags + +Each feature has its own flag that should be used to enable and disable it. +The documentation of each feature behind a flag includes a section informing +the status of the flag and the command to enable or disable it. + +### Start the GitLab Rails console + +The first thing you need to enable or disable a feature behind a flag is to +start a session on GitLab Rails console. + +For Omnibus installations: + +```shell +sudo gitlab-rails console +``` + +For installations from the source: + +```shell +sudo -u git -H bundle exec rails console -e production +``` + +For details, see [starting a Rails console session](troubleshooting/debug.md#starting-a-rails-console-session). + +### Enable or disable the feature + +Once the Rails console session has started, run the `Feature.enable` or +`Feature.disable` commands accordingly. The specific flag can be found +in the feature's documentation itself. + +To enable a feature, run: + +```ruby +Feature.enable(:<feature flag>) +``` + +Example, to enable Evidence Collection: + +```ruby +Feature.enable(:release_evidence_collection) +``` + +To disable a feature, run: + +```ruby +Feature.disable(:<feature flag>) +``` + +Example, to disable Evidence Collection: + +```ruby +Feature.disable(:release_evidence_collection) +``` + +When the feature is ready, GitLab will remove the feature flag, the option for +enabling and disabling it will no longer exist, and the feature will become +available in all instances. diff --git a/doc/administration/file_hooks.md b/doc/administration/file_hooks.md index 21ade36a2a5..7903da675fd 100644 --- a/doc/administration/file_hooks.md +++ b/doc/administration/file_hooks.md @@ -14,12 +14,12 @@ ensure functionality is preserved across versions and covered by tests. NOTE: **Note:** File hooks must be configured on the filesystem of the GitLab server. Only GitLab server administrators will be able to complete these tasks. Explore -[system hooks] or [webhooks] as an option if you do not have filesystem access. +[system hooks](../system_hooks/system_hooks.md) or [webhooks](../user/project/integrations/webhooks.md) as an option if you do not have filesystem access. A file hook will run on each event so it's up to you to filter events or projects within a file hook code. You can have as many file hooks as you want. Each file hook will be triggered by GitLab asynchronously in case of an event. For a list of events -see the [system hooks] documentation. +see the [system hooks](../system_hooks/system_hooks.md) documentation. ## Setup @@ -35,8 +35,8 @@ Follow the steps below to set up a custom hook: `/home/git/gitlab/file_hooks/`. For Omnibus installs the path is usually `/opt/gitlab/embedded/service/gitlab-rails/file_hooks`. - For [highly available] configurations, your hook file should exist on each - application server. + For [configurations with multiple servers](reference_architectures/index.md), + your hook file should exist on each application server. 1. Inside the `file_hooks` directory, create a file with a name of your choice, without spaces or special characters. @@ -46,7 +46,7 @@ Follow the steps below to set up a custom hook: language type. For example, if the script is in Ruby the shebang will probably be `#!/usr/bin/env ruby`. 1. The data to the file hook will be provided as JSON on STDIN. It will be exactly - same as for [system hooks] + same as for [system hooks](../system_hooks/system_hooks.md). That's it! Assuming the file hook code is properly implemented, the hook will fire as appropriate. The file hooks file list is updated for each event, there is no @@ -110,7 +110,3 @@ Validating file hooks from /file_hooks directory * /home/git/gitlab/file_hooks/save_to_file.clj succeed (zero exit code) * /home/git/gitlab/file_hooks/save_to_file.rb failure (non-zero exit code) ``` - -[system hooks]: ../system_hooks/system_hooks.md -[webhooks]: ../user/project/integrations/webhooks.md -[highly available]: ./availability/index.md diff --git a/doc/administration/geo/disaster_recovery/background_verification.md b/doc/administration/geo/disaster_recovery/background_verification.md index 6d6aee08c95..5b24c222f06 100644 --- a/doc/administration/geo/disaster_recovery/background_verification.md +++ b/doc/administration/geo/disaster_recovery/background_verification.md @@ -54,14 +54,14 @@ Feature.enable('geo_repository_verification') Navigate to the **{admin}** **Admin Area >** **{location-dot}** **Geo** dashboard on the **primary** node and expand the **Verification information** tab for that node to view automatic checksumming status for repositories and wikis. Successes are shown in green, pending work -in grey, and failures in red. +in gray, and failures in red.  Navigate to the **{admin}** **Admin Area >** **{location-dot}** **Geo** dashboard on the **secondary** node and expand the **Verification information** tab for that node to view automatic verification status for repositories and wikis. As with checksumming, successes are shown in -green, pending work in grey, and failures in red. +green, pending work in gray, and failures in red.  diff --git a/doc/administration/geo/disaster_recovery/index.md b/doc/administration/geo/disaster_recovery/index.md index 6f417f955ac..9f7afad353b 100644 --- a/doc/administration/geo/disaster_recovery/index.md +++ b/doc/administration/geo/disaster_recovery/index.md @@ -167,6 +167,45 @@ do this manually. previously for the **secondary**. 1. Success! The **secondary** has now been promoted to **primary**. +#### Promoting a **secondary** node with an external PostgreSQL database + +The `gitlab-ctl promote-to-primary-node` command cannot be used in conjunction with +an external PostgreSQL database, as it can only perform changes on a **secondary** +node with GitLab and the database on the same machine. As a result, a manual process is +required: + +1. Promote the replica database associated with the **secondary** site. This will + set the database to read-write: + - Amazon RDS - [Promoting a Read Replica to Be a Standalone DB Instance](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_ReadRepl.html#USER_ReadRepl.Promote) + - Azure Database for PostgreSQL - [Stop replication](https://docs.microsoft.com/en-us/azure/postgresql/howto-read-replicas-portal#stop-replication) + +1. Edit `/etc/gitlab/gitlab.rb` on every node in the **secondary** site to + reflect its new status as **primary** by removing any lines that enabled the + `geo_secondary_role`: + + ```ruby + ## In GitLab 11.4 and earlier, remove this line. + geo_secondary_role['enable'] = true + + ## In GitLab 11.5 and later, remove this line. + roles ['geo_secondary_role'] + ``` + + After making these changes [Reconfigure GitLab](../../restart_gitlab.md#omnibus-gitlab-reconfigure) + on each node so the changes take effect. + +1. Promote the **secondary** to **primary**. SSH into a single secondary application + node and execute: + + ```shell + sudo gitlab-rake geo:set_secondary_as_primary + ``` + +1. Verify you can connect to the newly promoted **primary** site using the URL used + previously for the **secondary** site. + +Success! The **secondary** site has now been promoted to **primary**. + ### Step 4. (Optional) Updating the primary domain DNS record Updating the DNS records for the primary domain to point to the **secondary** node diff --git a/doc/administration/geo/disaster_recovery/planned_failover.md b/doc/administration/geo/disaster_recovery/planned_failover.md index 4b3b464b710..00ca1e4b8b0 100644 --- a/doc/administration/geo/disaster_recovery/planned_failover.md +++ b/doc/administration/geo/disaster_recovery/planned_failover.md @@ -95,7 +95,7 @@ ensure these processes are close to 100% as possible during active use. Navigate to the **{admin}** **Admin Area >** **{location-dot}** **Geo** dashboard on the **secondary** node to review status. Replicated objects (shown in green) should be close to 100%, and there should be no failures (shown in red). If a large proportion of -objects aren't yet replicated (shown in grey), consider giving the node more +objects aren't yet replicated (shown in gray), consider giving the node more time to complete  diff --git a/doc/administration/geo/replication/configuration.md b/doc/administration/geo/replication/configuration.md index 0b076e7ff3c..86a8e5b28d1 100644 --- a/doc/administration/geo/replication/configuration.md +++ b/doc/administration/geo/replication/configuration.md @@ -262,7 +262,7 @@ You can login to the **secondary** node with the same credentials you used for t **secondary** Geo node and if Geo is enabled. The initial replication, or 'backfill', will probably still be in progress. You -can monitor the synchronization process on each geo node from the **primary** +can monitor the synchronization process on each Geo node from the **primary** node's **Geo Nodes** dashboard in your browser.  @@ -314,19 +314,17 @@ It is important to note that selective synchronization: Selective synchronization restrictions are implemented on the **secondary** nodes, not the **primary** node. -### Git operations on unreplicated respositories +### Git operations on unreplicated repositories -> [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/2562) in GitLab 12.10. +> [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/2562) in GitLab 12.10 for HTTP(S) and in GitLab 13.0 for SSH. -Git clone, pull, and push operations over HTTP(S) are supported for repositories that +Git clone, pull, and push operations over HTTP(S) and SSH are supported for repositories that exist on the **primary** node but not on **secondary** nodes. This situation can occur when: - Selective synchronization does not include the project attached to the repository. - The repository is actively being replicated but has not completed yet. -SSH [support is planned](https://gitlab.com/groups/gitlab-org/-/epics/2562). - ## Upgrading Geo See the [updating the Geo nodes document](updating_the_geo_nodes.md). diff --git a/doc/administration/geo/replication/database.md b/doc/administration/geo/replication/database.md index ffdec5a83c7..62bd0e6ac19 100644 --- a/doc/administration/geo/replication/database.md +++ b/doc/administration/geo/replication/database.md @@ -33,9 +33,9 @@ recover. See below for more details. The following guide assumes that: -- You are using Omnibus and therefore you are using PostgreSQL 9.6 or later - which includes the [`pg_basebackup` tool](https://www.postgresql.org/docs/9.6/app-pgbasebackup.html) and improved - [Foreign Data Wrapper](https://www.postgresql.org/docs/9.6/postgres-fdw.html) support. +- You are using Omnibus and therefore you are using PostgreSQL 11 or later + which includes the [`pg_basebackup` tool](https://www.postgresql.org/docs/11/app-pgbasebackup.html) and improved + [Foreign Data Wrapper](https://www.postgresql.org/docs/11/postgres-fdw.html) support. - You have a **primary** node already set up (the GitLab server you are replicating from), running Omnibus' PostgreSQL (or equivalent version), and you have a new **secondary** server set up with the same versions of the OS, @@ -91,7 +91,7 @@ There is an [issue where support is being discussed](https://gitlab.com/gitlab-o # Fill with the hash generated by `gitlab-ctl pg-password-md5 gitlab` postgresql['sql_user_password'] = '<md5_hash_of_your_password>' - # Every node that runs Unicorn or Sidekiq needs to have the database + # Every node that runs Puma or Sidekiq needs to have the database # password specified as below. If you have a high-availability setup, this # must be present in all application nodes. gitlab_rails['db_password'] = '<your_password_here>' @@ -160,7 +160,7 @@ There is an [issue where support is being discussed](https://gitlab.com/gitlab-o `postgresql['md5_auth_cidr_addresses']` and `postgresql['listen_address']`. The `listen_address` option opens PostgreSQL up to network connections with the interface - corresponding to the given address. See [the PostgreSQL documentation](https://www.postgresql.org/docs/9.6/runtime-config-connection.html) + corresponding to the given address. See [the PostgreSQL documentation](https://www.postgresql.org/docs/11/runtime-config-connection.html) for more details. Depending on your network configuration, the suggested addresses may not @@ -213,7 +213,7 @@ There is an [issue where support is being discussed](https://gitlab.com/gitlab-o ``` You may also want to edit the `wal_keep_segments` and `max_wal_senders` to match your - database replication requirements. Consult the [PostgreSQL - Replication documentation](https://www.postgresql.org/docs/9.6/runtime-config-replication.html) + database replication requirements. Consult the [PostgreSQL - Replication documentation](https://www.postgresql.org/docs/11/runtime-config-replication.html) for more information. 1. Save the file and reconfigure GitLab for the database listen changes and @@ -273,7 +273,7 @@ There is an [issue where support is being discussed](https://gitlab.com/gitlab-o 1. Stop application server and Sidekiq ```shell - gitlab-ctl stop unicorn + gitlab-ctl stop puma gitlab-ctl stop sidekiq ``` @@ -442,7 +442,7 @@ data before running `pg_basebackup`. (e.g., you know the network path is secure, or you are using a site-to-site VPN). This is **not** safe over the public Internet! - You can read more details about each `sslmode` in the - [PostgreSQL documentation](https://www.postgresql.org/docs/9.6/libpq-ssl.html#LIBPQ-SSL-PROTECTION); + [PostgreSQL documentation](https://www.postgresql.org/docs/11/libpq-ssl.html#LIBPQ-SSL-PROTECTION); the instructions above are carefully written to ensure protection against both passive eavesdroppers and active "man-in-the-middle" attackers. - Change the `--slot-name` to the name of the replication slot @@ -461,10 +461,10 @@ The replication process is now complete. PostgreSQL connections. We recommend using PgBouncer if you use GitLab in a high-availability configuration with a cluster of nodes supporting a Geo **primary** node and another cluster of nodes supporting a Geo **secondary** node. For more -information, see [High Availability with GitLab Omnibus](../../high_availability/database.md#high-availability-with-gitlab-omnibus-premium-only). +information, see [High Availability with Omnibus GitLab](../../high_availability/database.md#high-availability-with-omnibus-gitlab-premium-only). For a Geo **secondary** node to work properly with PgBouncer in front of the database, -it will need a separate read-only user to make [PostgreSQL FDW queries](https://www.postgresql.org/docs/9.6/postgres-fdw.html) +it will need a separate read-only user to make [PostgreSQL FDW queries](https://www.postgresql.org/docs/11/postgres-fdw.html) work: 1. On the **primary** Geo database, enter the PostgreSQL on the console as an diff --git a/doc/administration/geo/replication/datatypes.md b/doc/administration/geo/replication/datatypes.md index 3431df3ed1f..17031b11f51 100644 --- a/doc/administration/geo/replication/datatypes.md +++ b/doc/administration/geo/replication/datatypes.md @@ -79,7 +79,7 @@ GitLab stores files and blobs such as Issue attachments or LFS objects into eith - A Storage Appliance that exposes an Object Storage-compatible API. When using the filesystem store instead of Object Storage, you need to use network mounted filesystems -to run GitLab when using more than one server (for example with a High Availability setup). +to run GitLab when using more than one server. With respect to replication and verification: @@ -135,6 +135,7 @@ successfully, you must replicate their data using some other means. | CI job artifacts (other than traces) | **Yes** | [No](https://gitlab.com/gitlab-org/gitlab/issues/8923) | Verified only manually (*1*) | | Archived traces | **Yes** | [No](https://gitlab.com/gitlab-org/gitlab/issues/8923) | Verified only on transfer, or manually (*1*) | | Personal snippets | **Yes** | **Yes** | | +| [Versioned snippets](../../../user/snippets.md#versioned-snippets) | [No](https://gitlab.com/groups/gitlab-org/-/epics/2809) | [No](https://gitlab.com/groups/gitlab-org/-/epics/2810) | | | Project snippets | **Yes** | **Yes** | | | Object pools for forked project deduplication | **Yes** | No | | | [Server-side Git Hooks](../../custom_hooks.md) | No | No | | @@ -145,7 +146,7 @@ successfully, you must replicate their data using some other means. | [Maven Repository](../../../user/packages/maven_repository/index.md) | [No](https://gitlab.com/groups/gitlab-org/-/epics/2346) | No | | | [Conan Repository](../../../user/packages/conan_repository/index.md) | [No](https://gitlab.com/groups/gitlab-org/-/epics/2346) | No | | | [NuGet Repository](../../../user/packages/nuget_repository/index.md) | [No](https://gitlab.com/groups/gitlab-org/-/epics/2346) | No | | -| [PyPi Repository](../../../user/packages/pypi_repository/index.md) | [No](https://gitlab.com/groups/gitlab-org/-/epics/2554) | No | | +| [PyPi Repository](../../../user/packages/pypi_repository/index.md) | [No](https://gitlab.com/groups/gitlab-org/-/epics/2554) | No | | | [External merge request diffs](../../merge_request_diffs.md) | [No](https://gitlab.com/gitlab-org/gitlab/issues/33817) | No | | | Content in object storage | **Yes** | No | | diff --git a/doc/administration/geo/replication/external_database.md b/doc/administration/geo/replication/external_database.md index e305718580e..ae3069a0e40 100644 --- a/doc/administration/geo/replication/external_database.md +++ b/doc/administration/geo/replication/external_database.md @@ -17,6 +17,19 @@ developed and tested. We aim to be compatible with most external sudo -i ``` +1. Edit `/etc/gitlab/gitlab.rb` and add a **unique** ID for your node (arbitrary value): + + ```ruby + # The unique identifier for the Geo node. + gitlab_rails['geo_node_name'] = '<node_name_here>' + ``` + +1. Reconfigure the **primary** node for the change to take effect: + + ```shell + gitlab-ctl reconfigure + ``` + 1. Execute the command below to define the node as **primary** node: ```shell @@ -38,7 +51,14 @@ Given you have a primary node set up on AWS EC2 that uses RDS. You can now just create a read-only replica in a different region and the replication process will be managed by AWS. Make sure you've set Network ACL, Subnet, and Security Group according to your needs, so the secondary application node can access the database. -Skip to the [Configure secondary application node](#configure-secondary-application-nodes-to-use-the-external-read-replica) section below. + +The following instructions detail how to create a read-only replica for common +cloud providers: + +- Amazon RDS - [Creating a Read Replica](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_ReadRepl.html#USER_ReadRepl.Create) +- Azure Database for PostgreSQL - [Create and manage read replicas in Azure Database for PostgreSQL](https://docs.microsoft.com/en-us/azure/postgresql/howto-read-replicas-portal) + +Once your read-only replica is set up, you can skip to [configure you secondary application node](#configure-secondary-application-nodes-to-use-the-external-read-replica). #### Manually configure the primary database for replication @@ -133,6 +153,10 @@ To configure the connection to the external read-replica database and enable Log gitlab_rails['db_username'] = 'gitlab' gitlab_rails['db_host'] = '<database_read_replica_host>' + + # Disable the bundled Omnibus PostgreSQL, since we are + # using an external PostgreSQL + postgresql['enable'] = false ``` 1. Save the file and [reconfigure GitLab](../../restart_gitlab.md#omnibus-gitlab-reconfigure) @@ -142,11 +166,17 @@ To configure the connection to the external read-replica database and enable Log **Secondary** nodes use a separate PostgreSQL installation as a tracking database to keep track of replication status and automatically recover from potential replication issues. Omnibus automatically configures a tracking database -when `roles ['geo_secondary_role']` is set. For high availability, -refer to [Geo High Availability](../../availability/index.md). +when `roles ['geo_secondary_role']` is set. If you want to run this database external to Omnibus, please follow the instructions below. -The tracking database requires an [FDW](https://www.postgresql.org/docs/9.6/postgres-fdw.html) +If you are using a cloud-managed service for the tracking database, you may need +to grant additional roles to your tracking database user (by default, this is +`gitlab_geo`): + +- Amazon RDS requires the [`rds_superuser`](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Appendix.PostgreSQL.CommonDBATasks.html#Appendix.PostgreSQL.CommonDBATasks.Roles) role. +- Azure Database for PostgreSQL requires the [`azure_pg_admin`](https://docs.microsoft.com/en-us/azure/postgresql/howto-create-users#how-to-create-additional-admin-users-in-azure-database-for-postgresql) role. + +The tracking database requires an [FDW](https://www.postgresql.org/docs/11/postgres-fdw.html) connection with the **secondary** replica database for improved performance. If you have an external database ready to be used as the tracking database, @@ -200,7 +230,7 @@ the tracking database on port 5432. gitlab-rake geo:db:migrate ``` -1. Configure the [PostgreSQL FDW](https://www.postgresql.org/docs/9.6/postgres-fdw.html) +1. Configure the [PostgreSQL FDW](https://www.postgresql.org/docs/11/postgres-fdw.html) connection and credentials: Save the script below in a file, ex. `/tmp/geo_fdw.sh` and modify the connection diff --git a/doc/administration/geo/replication/geo_validation_tests.md b/doc/administration/geo/replication/geo_validation_tests.md new file mode 100644 index 00000000000..a8b0bdeb7da --- /dev/null +++ b/doc/administration/geo/replication/geo_validation_tests.md @@ -0,0 +1,100 @@ +# Geo validation tests + +The Geo team performs manual testing and validation on common deployment configurations to ensure +that Geo works when upgrading between minor GitLab versions and major PostgreSQL database versions. + +This section contains a journal of recent validation tests and links to the relevant issues. + +## GitLab upgrades + +The following are GitLab upgrade validation tests we performed. + +### February 2020 + +[Upgrade Geo multi-server installation](https://gitlab.com/gitlab-org/gitlab/-/issues/201837): + +- Description: Tested upgrading from GitLab 12.7.5 to the latest GitLab 12.8 package in a multi-server + configuration. +- Outcome: Partial success because we did not run the looping pipeline during the demo to monitor + downtime. + +### January 2020 + +[Upgrade Geo multi-server installation](https://gitlab.com/gitlab-org/gitlab/-/issues/200085): + +- Description: Tested upgrading from GitLab 12.6.x to the latest GitLab 12.7 package in a multi-server + configuration. +- Outcome: Upgrade test was successful. +- Follow up issues: + - [Investigate Geo end-to-end test failures](https://gitlab.com/gitlab-org/gitlab/issues/201823). + - [Add more logging to Geo end-to-end tests](https://gitlab.com/gitlab-org/gitlab/issues/201830). + - [Excess service restarts during zero-downtime upgrade](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/5047). + +[Upgrade Geo multi-server installation](https://gitlab.com/gitlab-org/gitlab/-/issues/199836): + +- Description: Tested upgrading from GitLab 12.5.7 to GitLab 12.6.6 in a multi-server configuration. +- Outcome: Upgrade test was successful. +- Follow up issue: + [Update documentation for zero-downtime upgrades to ensure deploy node it not in use](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/5046). + +[Upgrade Geo multi-server installation](https://gitlab.com/gitlab-org/gitlab/-/issues/37044): + +- Description: Tested upgrading from GitLab 12.4.x to the latest GitLab 12.5 package in a multi-server + configuration. +- Outcome: Upgrade test was successful. +- Follow up issues: + - [Investigate why HTTP push spec failed on primary node](https://gitlab.com/gitlab-org/gitlab/issues/199825). + - [Investigate if documentation should be modified to include refresh foreign tables task](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/5041). + +### October 2019 + +[Upgrade Geo multi-server installation](https://gitlab.com/gitlab-org/gitlab/-/issues/35262): + +- Description: Tested upgrading from GitLab 12.3.5 to GitLab 12.4.1 in a multi-server configuration. +- Outcome: Upgrade test was successful. + +[Upgrade Geo multi-server installation](https://gitlab.com/gitlab-org/gitlab/-/issues/32437): + +- Description: Tested upgrading from GitLab 12.2.8 to GitLab 12.3.5. +- Outcome: Upgrade test was successful. + +[Upgrade Geo multi-server installation](https://gitlab.com/gitlab-org/gitlab/-/issues/32435): + +- Description: Tested upgrading from GitLab 12.1.9 to GitLab 12.2.8. +- Outcome: Partial success due to possible misconfiguration issues. + +## PostgreSQL upgrades + +The following are PostgreSQL upgrade validation tests we performed. + +### April 2020 + +[PostgreSQL 11 upgrade procedure for Geo installations](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/4975): + +- Description: Prior to making PostgreSQL 11 the default version of PostgreSQL in GitLab 12.10, we + tested upgrading to PostgreSQL 11 in Geo deployments on GitLab 12.9. +- Outcome: Partially successful. Issues were discovered in multi-server configurations with a separate + tracking database and concerns were raised about allowing automatic upgrades when Geo enabled. +- Follow up issues: + - [`replicate-geo-database` incorrectly tries to back up repositories](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/5241). + - [`pg-upgrade` fails to upgrade a standalone Geo tracking database](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/5242). + - [`revert-pg-upgrade` fails to downgrade the PostgreSQL data of a Geo secondary’s standalone tracking database](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/5243). + - [Timeout error on Geo secondary read-replica near the end of `gitlab-ctl pg-upgrade`](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/5235). + +[Verify Geo installation with PostgreSQL 11](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/4971): + +- Description: Prior to making PostgreSQL 11 the default version of PostgreSQL in GitLab 12.10, we + tested fresh installations of GitLab 12.9 with Geo installed with PostgreSQL 11. +- Outcome: Installation test was successful. + +### September 2019 + +[Test and validate PostgreSQL 10.0 upgrade for Geo](https://gitlab.com/gitlab-org/gitlab/issues/12092): + +- Description: With the 12.0 release, GitLab required an upgrade to PostgreSQL 10.0. We tested + various upgrade scenarios from GitLab 11.11.5 through to GitLab 12.1.8. +- Outcome: Multiple issues were found when upgrading and addressed in follow-up issues. +- Follow up issues: + - [`gitlab-ctl` reconfigure fails on Redis node in multi-server Geo setup](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/4706). + - [Geo multi-server upgrade from 12.0.9 to 12.1.9 does not upgrade PostgreSQL](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/4705). + - [Refresh foreign tables fails on app server in multi-server setup after upgrade to 12.1.9](https://gitlab.com/gitlab-org/gitlab/-/issues/32119). diff --git a/doc/administration/geo/replication/high_availability.md b/doc/administration/geo/replication/high_availability.md index 5099e73d5e8..214f15b7565 100644 --- a/doc/administration/geo/replication/high_availability.md +++ b/doc/administration/geo/replication/high_availability.md @@ -1,460 +1,5 @@ -# Geo High Availability **(PREMIUM ONLY)** +--- +redirect_to: 'multiple_servers.md' +--- -This document describes a minimal reference architecture for running Geo -in a high availability configuration. If your HA setup differs from the one -described, it is possible to adapt these instructions to your needs. - -## Architecture overview - - - -_[diagram source - GitLab employees only](https://docs.google.com/drawings/d/1z0VlizKiLNXVVVaERFwgsIOuEgjcUqDTWPdQYsE7Z4c/edit)_ - -The topology above assumes that the **primary** and **secondary** Geo clusters -are located in two separate locations, on their own virtual network -with private IP addresses. The network is configured such that all machines within -one geographic location can communicate with each other using their private IP addresses. -The IP addresses given are examples and may be different depending on the -network topology of your deployment. - -The only external way to access the two Geo deployments is by HTTPS at -`gitlab.us.example.com` and `gitlab.eu.example.com` in the example above. - -NOTE: **Note:** -The **primary** and **secondary** Geo deployments must be able to communicate to each other over HTTPS. - -## Redis and PostgreSQL High Availability - -Geo supports: - -- Redis and PostgreSQL on the **primary** node configured for high availability -- Redis on **secondary** nodes configured for high availability. - -NOTE: **Note:** -Support for PostgreSQL on **secondary** nodes in high availability configuration -[is planned](https://gitlab.com/groups/gitlab-org/-/epics/2536). - -Because of the additional complexity involved in setting up this configuration -for PostgreSQL and Redis, it is not covered by this Geo HA documentation. - -For more information about setting up a highly available PostgreSQL cluster and Redis cluster using the omnibus package see the high availability documentation for -[PostgreSQL](../../high_availability/database.md) and -[Redis](../../high_availability/redis.md), respectively. - -NOTE: **Note:** -It is possible to use cloud hosted services for PostgreSQL and Redis, but this is beyond the scope of this document. - -## Prerequisites: Two working GitLab HA clusters - -One cluster will serve as the **primary** node. Use the -[GitLab HA documentation](../../availability/index.md) to set this up. If -you already have a working GitLab instance that is in-use, it can be used as a -**primary**. - -The second cluster will serve as the **secondary** node. Again, use the -[GitLab HA documentation](../../availability/index.md) to set this up. -It's a good idea to log in and test it, however, note that its data will be -wiped out as part of the process of replicating from the **primary**. - -## Configure the GitLab cluster to be the **primary** node - -The following steps enable a GitLab cluster to serve as the **primary** node. - -### Step 1: Configure the **primary** frontend servers - -1. Edit `/etc/gitlab/gitlab.rb` and add the following: - - ```ruby - ## - ## Enable the Geo primary role - ## - roles ['geo_primary_role'] - - ## - ## The unique identifier for the Geo node. - ## - gitlab_rails['geo_node_name'] = '<node_name_here>' - - ## - ## Disable automatic migrations - ## - gitlab_rails['auto_migrate'] = false - ``` - -After making these changes, [reconfigure GitLab](../../restart_gitlab.md#omnibus-gitlab-reconfigure) so the changes take effect. - -NOTE: **Note:** PostgreSQL and Redis should have already been disabled on the -application servers, and connections from the application servers to those -services on the backend servers configured, during normal GitLab HA set up. See -high availability configuration documentation for -[PostgreSQL](../../high_availability/database.md#configuring-the-application-nodes) -and [Redis](../../high_availability/redis.md#example-configuration-for-the-gitlab-application). - -### Step 2: Configure the **primary** database - -1. Edit `/etc/gitlab/gitlab.rb` and add the following: - - ```ruby - ## - ## Configure the Geo primary role and the PostgreSQL role - ## - roles ['geo_primary_role', 'postgres_role'] - ``` - -## Configure a **secondary** node - -A **secondary** cluster is similar to any other GitLab HA cluster, with two -major differences: - -- The main PostgreSQL database is a read-only replica of the **primary** node's - PostgreSQL database. -- There is also a single PostgreSQL database for the **secondary** cluster, - called the "tracking database", which tracks the synchronization state of - various resources. - -Therefore, we will set up the HA components one-by-one, and include deviations -from the normal HA setup. However, we highly recommend first configuring a -brand-new cluster as if it were not part of a Geo setup so that it can be -tested and verified as a working cluster. And only then should it be modified -for use as a Geo **secondary**. This helps to separate problems that are related -and are not related to Geo setup. - -### Step 1: Configure the Redis and Gitaly services on the **secondary** node - -Configure the following services, again using the non-Geo high availability -documentation: - -- [Configuring Redis for GitLab HA](../../high_availability/redis.md) for high - availability. -- [Gitaly](../../high_availability/gitaly.md), which will store data that is - synchronized from the **primary** node. - -NOTE: **Note:** -[NFS](../../high_availability/nfs.md) can be used in place of Gitaly but is not -recommended. - -### Step 2: Configure the main read-only replica PostgreSQL database on the **secondary** node - -NOTE: **Note:** The following documentation assumes the database will be run on -a single node only. PostgreSQL HA on **secondary** nodes is -[not currently supported](https://gitlab.com/groups/gitlab-org/-/epics/2536). - -Configure the [**secondary** database](database.md) as a read-only replica of -the **primary** database. Use the following as a guide. - -1. Generate an MD5 hash of the desired password for the database user that the - GitLab application will use to access the read-replica database: - - Note that the username (`gitlab` by default) is incorporated into the hash. - - ```shell - gitlab-ctl pg-password-md5 gitlab - # Enter password: <your_password_here> - # Confirm password: <your_password_here> - # fca0b89a972d69f00eb3ec98a5838484 - ``` - - Use this hash to fill in `<md5_hash_of_your_password>` in the next step. - -1. Edit `/etc/gitlab/gitlab.rb` in the replica database machine, and add the - following: - - ```ruby - ## - ## Configure the Geo secondary role and the PostgreSQL role - ## - roles ['geo_secondary_role', 'postgres_role'] - - ## - ## Secondary address - ## - replace '<secondary_node_ip>' with the public or VPC address of your Geo secondary node - ## - replace '<tracking_database_ip>' with the public or VPC address of your Geo tracking database node - ## - postgresql['listen_address'] = '<secondary_node_ip>' - postgresql['md5_auth_cidr_addresses'] = ['<secondary_node_ip>/32', '<tracking_database_ip>/32'] - - ## - ## Database credentials password (defined previously in primary node) - ## - replicate same values here as defined in primary node - ## - postgresql['sql_user_password'] = '<md5_hash_of_your_password>' - gitlab_rails['db_password'] = '<your_password_here>' - - ## - ## When running the Geo tracking database on a separate machine, disable it - ## here and allow connections from the tracking database host. And ensure - ## the tracking database IP is in postgresql['md5_auth_cidr_addresses'] above. - ## - geo_postgresql['enable'] = false - - ## - ## Disable `geo_logcursor` service so Rails doesn't get configured here - ## - geo_logcursor['enable'] = false - ``` - -After making these changes, [reconfigure GitLab](../../restart_gitlab.md#omnibus-gitlab-reconfigure) so the changes take effect. - -If using an external PostgreSQL instance, refer also to -[Geo with external PostgreSQL instances](external_database.md). - -### Step 3: Configure the tracking database on the **secondary** node - -NOTE: **Note:** This documentation assumes the tracking database will be run on -only a single machine, rather than as a PostgreSQL cluster. - -Configure the tracking database. - -1. Generate an MD5 hash of the desired password for the database user that the - GitLab application will use to access the tracking database: - - Note that the username (`gitlab_geo` by default) is incorporated into the - hash. - - ```shell - gitlab-ctl pg-password-md5 gitlab_geo - # Enter password: <your_password_here> - # Confirm password: <your_password_here> - # fca0b89a972d69f00eb3ec98a5838484 - ``` - - Use this hash to fill in `<tracking_database_password_md5_hash>` in the next - step. - -1. Edit `/etc/gitlab/gitlab.rb` in the tracking database machine, and add the - following: - - ```ruby - ## - ## Enable the Geo secondary tracking database - ## - geo_postgresql['enable'] = true - geo_postgresql['listen_address'] = '<ip_address_of_this_host>' - geo_postgresql['sql_user_password'] = '<tracking_database_password_md5_hash>' - - ## - ## Configure FDW connection to the replica database - ## - geo_secondary['db_fdw'] = true - geo_postgresql['fdw_external_password'] = '<replica_database_password_plaintext>' - geo_postgresql['md5_auth_cidr_addresses'] = ['<replica_database_ip>/32'] - gitlab_rails['db_host'] = '<replica_database_ip>' - - # Prevent reconfigure from attempting to run migrations on the replica DB - gitlab_rails['auto_migrate'] = false - - ## - ## Disable all other services that aren't needed, since we don't have a role - ## that does this. - ## - alertmanager['enable'] = false - consul['enable'] = false - gitaly['enable'] = false - gitlab_exporter['enable'] = false - gitlab_workhorse['enable'] = false - nginx['enable'] = false - node_exporter['enable'] = false - pgbouncer_exporter['enable'] = false - postgresql['enable'] = false - prometheus['enable'] = false - redis['enable'] = false - redis_exporter['enable'] = false - repmgr['enable'] = false - sidekiq['enable'] = false - unicorn['enable'] = false - ``` - -After making these changes, [reconfigure GitLab](../../restart_gitlab.md#omnibus-gitlab-reconfigure) so the changes take effect. - -If using an external PostgreSQL instance, refer also to -[Geo with external PostgreSQL instances](external_database.md). - -### Step 4: Configure the frontend application servers on the **secondary** node - -In the architecture overview, there are two machines running the GitLab -application services. These services are enabled selectively in the -configuration. - -Configure the application servers following -[Configuring GitLab for HA](../../high_availability/gitlab.md), then make the -following modifications: - -1. Edit `/etc/gitlab/gitlab.rb` on each application server in the **secondary** - cluster, and add the following: - - ```ruby - ## - ## Enable the Geo secondary role - ## - roles ['geo_secondary_role', 'application_role'] - - ## - ## The unique identifier for the Geo node. - ## - gitlab_rails['geo_node_name'] = '<node_name_here>' - - ## - ## Disable automatic migrations - ## - gitlab_rails['auto_migrate'] = false - - ## - ## Configure the connection to the tracking DB. And disable application - ## servers from running tracking databases. - ## - geo_secondary['db_host'] = '<geo_tracking_db_host>' - geo_secondary['db_password'] = '<geo_tracking_db_password>' - geo_postgresql['enable'] = false - - ## - ## Configure connection to the streaming replica database, if you haven't - ## already - ## - gitlab_rails['db_host'] = '<replica_database_host>' - gitlab_rails['db_password'] = '<replica_database_password>' - - ## - ## Configure connection to Redis, if you haven't already - ## - gitlab_rails['redis_host'] = '<redis_host>' - gitlab_rails['redis_password'] = '<redis_password>' - - ## - ## If you are using custom users not managed by Omnibus, you need to specify - ## UIDs and GIDs like below, and ensure they match between servers in a - ## cluster to avoid permissions issues - ## - user['uid'] = 9000 - user['gid'] = 9000 - web_server['uid'] = 9001 - web_server['gid'] = 9001 - registry['uid'] = 9002 - registry['gid'] = 9002 - ``` - -NOTE: **Note:** -If you had set up PostgreSQL cluster using the omnibus package and you had set -up `postgresql['sql_user_password'] = 'md5 digest of secret'` setting, keep in -mind that `gitlab_rails['db_password']` and `geo_secondary['db_password']` -mentioned above contains the plaintext passwords. This is used to let the Rails -servers connect to the databases. - -NOTE: **Note:** -Make sure that current node IP is listed in `postgresql['md5_auth_cidr_addresses']` setting of your remote database. - -After making these changes [Reconfigure GitLab](../../restart_gitlab.md#omnibus-gitlab-reconfigure) so the changes take effect. - -On the secondary the following GitLab frontend services will be enabled: - -- `geo-logcursor` -- `gitlab-pages` -- `gitlab-workhorse` -- `logrotate` -- `nginx` -- `registry` -- `remote-syslog` -- `sidekiq` -- `unicorn` - -Verify these services by running `sudo gitlab-ctl status` on the frontend -application servers. - -### Step 5: Set up the LoadBalancer for the **secondary** node - -In this topology, a load balancer is required at each geographic location to -route traffic to the application servers. - -See [Load Balancer for GitLab HA](../../high_availability/load_balancer.md) for -more information. - -### Step 6: Configure the backend application servers on the **secondary** node - -The minimal reference architecture diagram above shows all application services -running together on the same machines. However, for high availability we -[strongly recommend running all services separately](../../availability/index.md). - -For example, a Sidekiq server could be configured similarly to the frontend -application servers above, with some changes to run only the `sidekiq` service: - -1. Edit `/etc/gitlab/gitlab.rb` on each Sidekiq server in the **secondary** - cluster, and add the following: - - ```ruby - ## - ## Enable the Geo secondary role - ## - roles ['geo_secondary_role'] - - ## - ## Enable the Sidekiq service - ## - sidekiq['enable'] = true - - ## - ## Ensure unnecessary services are disabled - ## - alertmanager['enable'] = false - consul['enable'] = false - geo_logcursor['enable'] = false - gitaly['enable'] = false - gitlab_exporter['enable'] = false - gitlab_workhorse['enable'] = false - nginx['enable'] = false - node_exporter['enable'] = false - pgbouncer_exporter['enable'] = false - postgresql['enable'] = false - prometheus['enable'] = false - redis['enable'] = false - redis_exporter['enable'] = false - repmgr['enable'] = false - unicorn['enable'] = false - - ## - ## The unique identifier for the Geo node. - ## - gitlab_rails['geo_node_name'] = '<node_name_here>' - - ## - ## Disable automatic migrations - ## - gitlab_rails['auto_migrate'] = false - - ## - ## Configure the connection to the tracking DB. And disable application - ## servers from running tracking databases. - ## - geo_secondary['db_host'] = '<geo_tracking_db_host>' - geo_secondary['db_password'] = '<geo_tracking_db_password>' - geo_postgresql['enable'] = false - - ## - ## Configure connection to the streaming replica database, if you haven't - ## already - ## - gitlab_rails['db_host'] = '<replica_database_host>' - gitlab_rails['db_password'] = '<replica_database_password>' - - ## - ## Configure connection to Redis, if you haven't already - ## - gitlab_rails['redis_host'] = '<redis_host>' - gitlab_rails['redis_password'] = '<redis_password>' - - ## - ## If you are using custom users not managed by Omnibus, you need to specify - ## UIDs and GIDs like below, and ensure they match between servers in a - ## cluster to avoid permissions issues - ## - user['uid'] = 9000 - user['gid'] = 9000 - web_server['uid'] = 9001 - web_server['gid'] = 9001 - registry['uid'] = 9002 - registry['gid'] = 9002 - ``` - - You can similarly configure a server to run only the `geo-logcursor` service - with `geo_logcursor['enable'] = true` and disabling Sidekiq with - `sidekiq['enable'] = false`. - - These servers do not need to be attached to the load balancer. +This document was moved to [another location](multiple_servers.md). diff --git a/doc/administration/geo/replication/index.md b/doc/administration/geo/replication/index.md index 7c661abef9a..87bd7b69515 100644 --- a/doc/administration/geo/replication/index.md +++ b/doc/administration/geo/replication/index.md @@ -2,7 +2,7 @@ > - Introduced in GitLab Enterprise Edition 8.9. > - Using Geo in combination with -> [High Availability](../../availability/index.md) +> [multi-server architectures](../../reference_architectures/index.md) > is considered **Generally Available** (GA) in > [GitLab Premium](https://about.gitlab.com/pricing/) 10.4. @@ -110,7 +110,7 @@ The following are required to run Geo: The following operating systems are known to ship with a current version of OpenSSH: - [CentOS](https://www.centos.org) 7.4+ - [Ubuntu](https://ubuntu.com) 16.04+ -- PostgreSQL 9.6+ with [FDW](https://www.postgresql.org/docs/9.6/postgres-fdw.html) support and [Streaming Replication](https://wiki.postgresql.org/wiki/Streaming_Replication) +- PostgreSQL 11+ with [FDW](https://www.postgresql.org/docs/11/postgres-fdw.html) support and [Streaming Replication](https://wiki.postgresql.org/wiki/Streaming_Replication) - Git 2.9+ - All nodes must run the same GitLab version. @@ -134,7 +134,7 @@ The following table lists basic ports that must be open between the **primary** See the full list of ports used by GitLab in [Package defaults](https://docs.gitlab.com/omnibus/package-information/defaults.html) NOTE: **Note:** -[Web terminal](../../../ci/environments.md#web-terminals) support requires your load balancer to correctly handle WebSocket connections. +[Web terminal](../../../ci/environments/index.md#web-terminals) support requires your load balancer to correctly handle WebSocket connections. When using HTTP or HTTPS proxying, your load balancer must be configured to pass through the `Connection` and `Upgrade` hop-by-hop headers. See the [web terminal](../../integration/terminal.md) integration guide for more details. NOTE: **Note:** @@ -206,9 +206,9 @@ For information on configuring Geo, see [Geo configuration](configuration.md). For information on how to update your Geo nodes to the latest GitLab version, see [Updating the Geo nodes](updating_the_geo_nodes.md). -### Configuring Geo high availability +### Configuring Geo for multiple servers -For information on configuring Geo for high availability, see [Geo High Availability](high_availability.md). +For information on configuring Geo for multiple servers, see [Geo for multiple servers](multiple_servers.md). ### Configuring Geo with Object Storage @@ -245,7 +245,7 @@ This list of limitations only reflects the latest version of GitLab. If you are - Pushing directly to a **secondary** node redirects (for HTTP) or proxies (for SSH) the request to the **primary** node instead of [handling it directly](https://gitlab.com/gitlab-org/gitlab/issues/1381), except when using Git over HTTP with credentials embedded within the URI. For example, `https://user:password@secondary.tld`. - Cloning, pulling, or pushing repositories that exist on the **primary** node but not on the **secondary** nodes where [selective synchronization](configuration.md#selective-synchronization) does not include the project is not supported over SSH [but support is planned](https://gitlab.com/groups/gitlab-org/-/epics/2562). HTTP(S) is supported. -- The **primary** node has to be online for OAuth login to happen. Existing sessions and Git are not affected. +- The **primary** node has to be online for OAuth login to happen. Existing sessions and Git are not affected. Support for the **secondary** node to use an OAuth provider independent from the primary is [being planned](https://gitlab.com/gitlab-org/gitlab/issues/208465). - The installation takes multiple manual steps that together can take about an hour depending on circumstances. We are working on improving this experience. See [Omnibus GitLab issue #2978](https://gitlab.com/gitlab-org/omnibus-gitlab/issues/2978) for details. - Real-time updates of issues/merge requests (for example, via long polling) doesn't work on the **secondary** node. - [Selective synchronization](configuration.md#selective-synchronization) applies only to files and repositories. Other datasets are replicated to the **secondary** node in full, making it inappropriate for use as an access control mechanism. diff --git a/doc/administration/geo/replication/multiple_servers.md b/doc/administration/geo/replication/multiple_servers.md new file mode 100644 index 00000000000..9322c4cc417 --- /dev/null +++ b/doc/administration/geo/replication/multiple_servers.md @@ -0,0 +1,459 @@ +# Geo for multiple servers **(PREMIUM ONLY)** + +This document describes a minimal reference architecture for running Geo +in a multi-server configuration. If your multi-server setup differs from the one +described, it is possible to adapt these instructions to your needs. + +## Architecture overview + + + +_[diagram source - GitLab employees only](https://docs.google.com/drawings/d/1z0VlizKiLNXVVVaERFwgsIOuEgjcUqDTWPdQYsE7Z4c/edit)_ + +The topology above assumes that the **primary** and **secondary** Geo clusters +are located in two separate locations, on their own virtual network +with private IP addresses. The network is configured such that all machines within +one geographic location can communicate with each other using their private IP addresses. +The IP addresses given are examples and may be different depending on the +network topology of your deployment. + +The only external way to access the two Geo deployments is by HTTPS at +`gitlab.us.example.com` and `gitlab.eu.example.com` in the example above. + +NOTE: **Note:** +The **primary** and **secondary** Geo deployments must be able to communicate to each other over HTTPS. + +## Redis and PostgreSQL for multiple servers + +Geo supports: + +- Redis and PostgreSQL on the **primary** node configured for multiple servers. +- Redis on **secondary** nodes configured for multiple servers. + +NOTE: **Note:** +Support for PostgreSQL on **secondary** nodes in multi-server configuration +[is planned](https://gitlab.com/groups/gitlab-org/-/epics/2536). + +Because of the additional complexity involved in setting up this configuration +for PostgreSQL and Redis, it is not covered by this Geo multi-server documentation. + +For more information about setting up a multi-server PostgreSQL cluster and Redis cluster using the omnibus package see the multi-server documentation for +[PostgreSQL](../../high_availability/database.md) and +[Redis](../../high_availability/redis.md), respectively. + +NOTE: **Note:** +It is possible to use cloud hosted services for PostgreSQL and Redis, but this is beyond the scope of this document. + +## Prerequisites: Two working GitLab multi-server clusters + +One cluster will serve as the **primary** node. Use the +[GitLab multi-server documentation](../../reference_architectures/index.md) to set this up. If +you already have a working GitLab instance that is in-use, it can be used as a +**primary**. + +The second cluster will serve as the **secondary** node. Again, use the +[GitLab multi-server documentation](../../reference_architectures/index.md) to set this up. +It's a good idea to log in and test it, however, note that its data will be +wiped out as part of the process of replicating from the **primary**. + +## Configure the GitLab cluster to be the **primary** node + +The following steps enable a GitLab cluster to serve as the **primary** node. + +### Step 1: Configure the **primary** frontend servers + +1. Edit `/etc/gitlab/gitlab.rb` and add the following: + + ```ruby + ## + ## Enable the Geo primary role + ## + roles ['geo_primary_role'] + + ## + ## The unique identifier for the Geo node. + ## + gitlab_rails['geo_node_name'] = '<node_name_here>' + + ## + ## Disable automatic migrations + ## + gitlab_rails['auto_migrate'] = false + ``` + +After making these changes, [reconfigure GitLab](../../restart_gitlab.md#omnibus-gitlab-reconfigure) so the changes take effect. + +NOTE: **Note:** PostgreSQL and Redis should have already been disabled on the +application servers, and connections from the application servers to those +services on the backend servers configured, during normal GitLab multi-server set up. See +multi-server configuration documentation for +[PostgreSQL](../../high_availability/database.md#configuring-the-application-nodes) +and [Redis](../../high_availability/redis.md#example-configuration-for-the-gitlab-application). + +### Step 2: Configure the **primary** database + +1. Edit `/etc/gitlab/gitlab.rb` and add the following: + + ```ruby + ## + ## Configure the Geo primary role and the PostgreSQL role + ## + roles ['geo_primary_role', 'postgres_role'] + ``` + +## Configure a **secondary** node + +A **secondary** cluster is similar to any other GitLab multi-server cluster, with two +major differences: + +- The main PostgreSQL database is a read-only replica of the **primary** node's + PostgreSQL database. +- There is also a single PostgreSQL database for the **secondary** cluster, + called the "tracking database", which tracks the synchronization state of + various resources. + +Therefore, we will set up the multi-server components one-by-one, and include deviations +from the normal multi-server setup. However, we highly recommend first configuring a +brand-new cluster as if it were not part of a Geo setup so that it can be +tested and verified as a working cluster. And only then should it be modified +for use as a Geo **secondary**. This helps to separate problems that are related +and are not related to Geo setup. + +### Step 1: Configure the Redis and Gitaly services on the **secondary** node + +Configure the following services, again using the non-Geo multi-server +documentation: + +- [Configuring Redis for GitLab](../../high_availability/redis.md) for multiple servers. +- [Gitaly](../../high_availability/gitaly.md), which will store data that is + synchronized from the **primary** node. + +NOTE: **Note:** +[NFS](../../high_availability/nfs.md) can be used in place of Gitaly but is not +recommended. + +### Step 2: Configure the main read-only replica PostgreSQL database on the **secondary** node + +NOTE: **Note:** The following documentation assumes the database will be run on +a single node only. Multi-server PostgreSQL on **secondary** nodes is +[not currently supported](https://gitlab.com/groups/gitlab-org/-/epics/2536). + +Configure the [**secondary** database](database.md) as a read-only replica of +the **primary** database. Use the following as a guide. + +1. Generate an MD5 hash of the desired password for the database user that the + GitLab application will use to access the read-replica database: + + Note that the username (`gitlab` by default) is incorporated into the hash. + + ```shell + gitlab-ctl pg-password-md5 gitlab + # Enter password: <your_password_here> + # Confirm password: <your_password_here> + # fca0b89a972d69f00eb3ec98a5838484 + ``` + + Use this hash to fill in `<md5_hash_of_your_password>` in the next step. + +1. Edit `/etc/gitlab/gitlab.rb` in the replica database machine, and add the + following: + + ```ruby + ## + ## Configure the Geo secondary role and the PostgreSQL role + ## + roles ['geo_secondary_role', 'postgres_role'] + + ## + ## Secondary address + ## - replace '<secondary_node_ip>' with the public or VPC address of your Geo secondary node + ## - replace '<tracking_database_ip>' with the public or VPC address of your Geo tracking database node + ## + postgresql['listen_address'] = '<secondary_node_ip>' + postgresql['md5_auth_cidr_addresses'] = ['<secondary_node_ip>/32', '<tracking_database_ip>/32'] + + ## + ## Database credentials password (defined previously in primary node) + ## - replicate same values here as defined in primary node + ## + postgresql['sql_user_password'] = '<md5_hash_of_your_password>' + gitlab_rails['db_password'] = '<your_password_here>' + + ## + ## When running the Geo tracking database on a separate machine, disable it + ## here and allow connections from the tracking database host. And ensure + ## the tracking database IP is in postgresql['md5_auth_cidr_addresses'] above. + ## + geo_postgresql['enable'] = false + + ## + ## Disable `geo_logcursor` service so Rails doesn't get configured here + ## + geo_logcursor['enable'] = false + ``` + +After making these changes, [reconfigure GitLab](../../restart_gitlab.md#omnibus-gitlab-reconfigure) so the changes take effect. + +If using an external PostgreSQL instance, refer also to +[Geo with external PostgreSQL instances](external_database.md). + +### Step 3: Configure the tracking database on the **secondary** node + +NOTE: **Note:** This documentation assumes the tracking database will be run on +only a single machine, rather than as a PostgreSQL cluster. + +Configure the tracking database. + +1. Generate an MD5 hash of the desired password for the database user that the + GitLab application will use to access the tracking database: + + Note that the username (`gitlab_geo` by default) is incorporated into the + hash. + + ```shell + gitlab-ctl pg-password-md5 gitlab_geo + # Enter password: <your_password_here> + # Confirm password: <your_password_here> + # fca0b89a972d69f00eb3ec98a5838484 + ``` + + Use this hash to fill in `<tracking_database_password_md5_hash>` in the next + step. + +1. Edit `/etc/gitlab/gitlab.rb` in the tracking database machine, and add the + following: + + ```ruby + ## + ## Enable the Geo secondary tracking database + ## + geo_postgresql['enable'] = true + geo_postgresql['listen_address'] = '<ip_address_of_this_host>' + geo_postgresql['sql_user_password'] = '<tracking_database_password_md5_hash>' + + ## + ## Configure FDW connection to the replica database + ## + geo_secondary['db_fdw'] = true + geo_postgresql['fdw_external_password'] = '<replica_database_password_plaintext>' + geo_postgresql['md5_auth_cidr_addresses'] = ['<replica_database_ip>/32'] + gitlab_rails['db_host'] = '<replica_database_ip>' + + # Prevent reconfigure from attempting to run migrations on the replica DB + gitlab_rails['auto_migrate'] = false + + ## + ## Disable all other services that aren't needed, since we don't have a role + ## that does this. + ## + alertmanager['enable'] = false + consul['enable'] = false + gitaly['enable'] = false + gitlab_exporter['enable'] = false + gitlab_workhorse['enable'] = false + nginx['enable'] = false + node_exporter['enable'] = false + pgbouncer_exporter['enable'] = false + postgresql['enable'] = false + prometheus['enable'] = false + redis['enable'] = false + redis_exporter['enable'] = false + repmgr['enable'] = false + sidekiq['enable'] = false + puma['enable'] = false + ``` + +After making these changes, [reconfigure GitLab](../../restart_gitlab.md#omnibus-gitlab-reconfigure) so the changes take effect. + +If using an external PostgreSQL instance, refer also to +[Geo with external PostgreSQL instances](external_database.md). + +### Step 4: Configure the frontend application servers on the **secondary** node + +In the architecture overview, there are two machines running the GitLab +application services. These services are enabled selectively in the +configuration. + +Configure the application servers following +[Configuring GitLab for multiple servers](../../high_availability/gitlab.md), then make the +following modifications: + +1. Edit `/etc/gitlab/gitlab.rb` on each application server in the **secondary** + cluster, and add the following: + + ```ruby + ## + ## Enable the Geo secondary role + ## + roles ['geo_secondary_role', 'application_role'] + + ## + ## The unique identifier for the Geo node. + ## + gitlab_rails['geo_node_name'] = '<node_name_here>' + + ## + ## Disable automatic migrations + ## + gitlab_rails['auto_migrate'] = false + + ## + ## Configure the connection to the tracking DB. And disable application + ## servers from running tracking databases. + ## + geo_secondary['db_host'] = '<geo_tracking_db_host>' + geo_secondary['db_password'] = '<geo_tracking_db_password>' + geo_postgresql['enable'] = false + + ## + ## Configure connection to the streaming replica database, if you haven't + ## already + ## + gitlab_rails['db_host'] = '<replica_database_host>' + gitlab_rails['db_password'] = '<replica_database_password>' + + ## + ## Configure connection to Redis, if you haven't already + ## + gitlab_rails['redis_host'] = '<redis_host>' + gitlab_rails['redis_password'] = '<redis_password>' + + ## + ## If you are using custom users not managed by Omnibus, you need to specify + ## UIDs and GIDs like below, and ensure they match between servers in a + ## cluster to avoid permissions issues + ## + user['uid'] = 9000 + user['gid'] = 9000 + web_server['uid'] = 9001 + web_server['gid'] = 9001 + registry['uid'] = 9002 + registry['gid'] = 9002 + ``` + +NOTE: **Note:** +If you had set up PostgreSQL cluster using the omnibus package and you had set +up `postgresql['sql_user_password'] = 'md5 digest of secret'` setting, keep in +mind that `gitlab_rails['db_password']` and `geo_secondary['db_password']` +mentioned above contains the plaintext passwords. This is used to let the Rails +servers connect to the databases. + +NOTE: **Note:** +Make sure that current node IP is listed in `postgresql['md5_auth_cidr_addresses']` setting of your remote database. + +After making these changes [Reconfigure GitLab](../../restart_gitlab.md#omnibus-gitlab-reconfigure) so the changes take effect. + +On the secondary the following GitLab frontend services will be enabled: + +- `geo-logcursor` +- `gitlab-pages` +- `gitlab-workhorse` +- `logrotate` +- `nginx` +- `registry` +- `remote-syslog` +- `sidekiq` +- `puma` + +Verify these services by running `sudo gitlab-ctl status` on the frontend +application servers. + +### Step 5: Set up the LoadBalancer for the **secondary** node + +In this topology, a load balancer is required at each geographic location to +route traffic to the application servers. + +See [Load Balancer for GitLab with multiple servers](../../high_availability/load_balancer.md) for +more information. + +### Step 6: Configure the backend application servers on the **secondary** node + +The minimal reference architecture diagram above shows all application services +running together on the same machines. However, for multiple servers we +[strongly recommend running all services separately](../../reference_architectures/index.md). + +For example, a Sidekiq server could be configured similarly to the frontend +application servers above, with some changes to run only the `sidekiq` service: + +1. Edit `/etc/gitlab/gitlab.rb` on each Sidekiq server in the **secondary** + cluster, and add the following: + + ```ruby + ## + ## Enable the Geo secondary role + ## + roles ['geo_secondary_role'] + + ## + ## Enable the Sidekiq service + ## + sidekiq['enable'] = true + + ## + ## Ensure unnecessary services are disabled + ## + alertmanager['enable'] = false + consul['enable'] = false + geo_logcursor['enable'] = false + gitaly['enable'] = false + gitlab_exporter['enable'] = false + gitlab_workhorse['enable'] = false + nginx['enable'] = false + node_exporter['enable'] = false + pgbouncer_exporter['enable'] = false + postgresql['enable'] = false + prometheus['enable'] = false + redis['enable'] = false + redis_exporter['enable'] = false + repmgr['enable'] = false + puma['enable'] = false + + ## + ## The unique identifier for the Geo node. + ## + gitlab_rails['geo_node_name'] = '<node_name_here>' + + ## + ## Disable automatic migrations + ## + gitlab_rails['auto_migrate'] = false + + ## + ## Configure the connection to the tracking DB. And disable application + ## servers from running tracking databases. + ## + geo_secondary['db_host'] = '<geo_tracking_db_host>' + geo_secondary['db_password'] = '<geo_tracking_db_password>' + geo_postgresql['enable'] = false + + ## + ## Configure connection to the streaming replica database, if you haven't + ## already + ## + gitlab_rails['db_host'] = '<replica_database_host>' + gitlab_rails['db_password'] = '<replica_database_password>' + + ## + ## Configure connection to Redis, if you haven't already + ## + gitlab_rails['redis_host'] = '<redis_host>' + gitlab_rails['redis_password'] = '<redis_password>' + + ## + ## If you are using custom users not managed by Omnibus, you need to specify + ## UIDs and GIDs like below, and ensure they match between servers in a + ## cluster to avoid permissions issues + ## + user['uid'] = 9000 + user['gid'] = 9000 + web_server['uid'] = 9001 + web_server['gid'] = 9001 + registry['uid'] = 9002 + registry['gid'] = 9002 + ``` + + You can similarly configure a server to run only the `geo-logcursor` service + with `geo_logcursor['enable'] = true` and disabling Sidekiq with + `sidekiq['enable'] = false`. + + These servers do not need to be attached to the load balancer. diff --git a/doc/administration/geo/replication/security_review.md b/doc/administration/geo/replication/security_review.md index 18fe1ad22cd..0ac8157220a 100644 --- a/doc/administration/geo/replication/security_review.md +++ b/doc/administration/geo/replication/security_review.md @@ -73,7 +73,7 @@ from [owasp.org](https://owasp.org/). - Nothing Geo-specific. Any user where `admin: true` is set in the database is considered an admin with super-user privileges. - See also: [more granular access control](https://gitlab.com/gitlab-org/gitlab-foss/issues/32730) - (not geo-specific) + (not Geo-specific). - Much of Geo’s integration (database replication, for instance) must be configured with the application, typically by system administrators. @@ -177,7 +177,7 @@ from [owasp.org](https://owasp.org/). ### What databases and application servers support the application? -- PostgreSQL >= 9.6, Redis, Sidekiq, Unicorn. +- PostgreSQL >= 11, Redis, Sidekiq, Puma. ### How will database connection strings, encryption keys, and other sensitive components be stored, accessed, and protected from unauthorized detection? diff --git a/doc/administration/geo/replication/troubleshooting.md b/doc/administration/geo/replication/troubleshooting.md index fae9705e935..293414a6e5e 100644 --- a/doc/administration/geo/replication/troubleshooting.md +++ b/doc/administration/geo/replication/troubleshooting.md @@ -497,6 +497,12 @@ to start again from scratch, there are a few steps that can help you: gitlab-ctl start ``` +1. Refresh Foreign Data Wrapper tables + + ```shell + gitlab-rake geo:db:refresh_foreign_tables + ``` + ## Fixing errors during a failover or when promoting a secondary to a primary node The following are possible errors that might be encountered during failover or @@ -538,6 +544,27 @@ or `gitlab-ctl promote-to-primary-node`, either: bug](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/22021) was fixed. +### Message: ``NoMethodError: undefined method `secondary?' for nil:NilClass`` + +When [promoting a **secondary** node](../disaster_recovery/index.md#step-3-promoting-a-secondary-node), +you might encounter the following error: + +```plaintext +sudo gitlab-rake geo:set_secondary_as_primary + +rake aborted! +NoMethodError: undefined method `secondary?' for nil:NilClass +/opt/gitlab/embedded/service/gitlab-rails/ee/lib/tasks/geo.rake:232:in `block (3 levels) in <top (required)>' +/opt/gitlab/embedded/service/gitlab-rails/ee/lib/tasks/geo.rake:221:in `block (2 levels) in <top (required)>' +/opt/gitlab/embedded/bin/bundle:23:in `load' +/opt/gitlab/embedded/bin/bundle:23:in `<main>' +Tasks: TOP => geo:set_secondary_as_primary +(See full trace by running task with --trace) +``` + +This command is intended to be executed on a secondary node only, and this error +is displayed if you attempt to run this command on a primary node. + ### Message: `sudo: gitlab-pg-ctl: command not found` When @@ -624,9 +651,9 @@ To check the configuration: ``` This password is normally set on the tracking database during - [Step 3: Configure the tracking database on the secondary node](high_availability.md#step-3-configure-the-tracking-database-on-the-secondary-node), + [Step 3: Configure the tracking database on the secondary node](multiple_servers.md#step-3-configure-the-tracking-database-on-the-secondary-node), and it is set on the app nodes during - [Step 4: Configure the frontend application servers on the secondary node](high_availability.md#step-4-configure-the-frontend-application-servers-on-the-secondary-node). + [Step 4: Configure the frontend application servers on the secondary node](multiple_servers.md#step-4-configure-the-frontend-application-servers-on-the-secondary-node). 1. Check whether any tables are present with the following statement: @@ -833,6 +860,8 @@ which Geo expects to have access to. It usually means, either: - An unsupported replication method was used (for example, logical replication). - The instructions to setup a [Geo database replication](database.md) were not followed correctly. +- Your database connection details are incorrect, that is you have specified the wrong + user in your `/etc/gitlab/gitlab.rb` file. A common source of confusion with **secondary** nodes is that it requires two separate PostgreSQL instances: @@ -854,7 +883,7 @@ Make sure you follow the [Geo database replication](database.md) instructions fo ### Geo database version (...) does not match latest migration (...) -If you are using GitLab Omnibus installation, something might have failed during upgrade. You can: +If you are using Omnibus GitLab installation, something might have failed during upgrade. You can: - Run `sudo gitlab-ctl reconfigure`. - Manually trigger the database migration by running: `sudo gitlab-rake geo:db:migrate` as root on the **secondary** node. diff --git a/doc/administration/geo/replication/updating_the_geo_nodes.md b/doc/administration/geo/replication/updating_the_geo_nodes.md index df66b1b36ec..fa1576e19eb 100644 --- a/doc/administration/geo/replication/updating_the_geo_nodes.md +++ b/doc/administration/geo/replication/updating_the_geo_nodes.md @@ -11,6 +11,7 @@ Updating Geo nodes involves performing: Depending on which version of Geo you are updating to/from, there may be different steps. +- [Updating to GitLab 12.9](version_specific_updates.md#updating-to-gitlab-129) - [Updating to GitLab 12.7](version_specific_updates.md#updating-to-gitlab-127) - [Updating to GitLab 12.2](version_specific_updates.md#updating-to-gitlab-122) - [Updating to GitLab 12.1](version_specific_updates.md#updating-to-gitlab-121) @@ -44,7 +45,7 @@ and all **secondary** nodes: Now that the update process is complete, you may want to check whether everything is working correctly: -1. Run the Geo raketask on all nodes, everything should be green: +1. Run the Geo Rake task on all nodes, everything should be green: ```shell sudo gitlab-rake gitlab:geo:check diff --git a/doc/administration/geo/replication/using_a_geo_server.md b/doc/administration/geo/replication/using_a_geo_server.md index 0f55272f667..2fec2b2b59c 100644 --- a/doc/administration/geo/replication/using_a_geo_server.md +++ b/doc/administration/geo/replication/using_a_geo_server.md @@ -1,4 +1,4 @@ -[//]: # (Please update EE::GitLab::GeoGitAccess::GEO_SERVER_DOCS_URL if this file is moved) +<!-- Please update EE::GitLab::GeoGitAccess::GEO_SERVER_DOCS_URL if this file is moved) --> # Using a Geo Server **(PREMIUM ONLY)** diff --git a/doc/administration/geo/replication/version_specific_updates.md b/doc/administration/geo/replication/version_specific_updates.md index 81868d19c7f..db8bbddec3b 100644 --- a/doc/administration/geo/replication/version_specific_updates.md +++ b/doc/administration/geo/replication/version_specific_updates.md @@ -30,7 +30,7 @@ GitLab 12.2 includes the following minor PostgreSQL updates: This update will occur even if major PostgreSQL updates are disabled. -Before [refreshing Foreign Data Wrapper during a Geo HA upgrade](https://docs.gitlab.com/omnibus/update/README.html#run-post-deployment-migrations-and-checks), +Before [refreshing Foreign Data Wrapper during a Geo upgrade](https://docs.gitlab.com/omnibus/update/README.html#run-post-deployment-migrations-and-checks), restart the Geo tracking database: ```shell @@ -100,8 +100,8 @@ authentication method. postgresql['sql_user_password'] = '<md5_hash_of_your_password>' # Every node that runs Unicorn or Sidekiq needs to have the database - # password specified as below. If you have a high-availability setup, this - # must be present in all application nodes. + # password specified as below. + # This must be present in all application nodes. gitlab_rails['db_password'] = '<your_password_here>' ``` @@ -125,8 +125,8 @@ authentication method. postgresql['sql_user_password'] = '<md5_hash_of_your_password>' # Every node that runs Unicorn or Sidekiq needs to have the database - # password specified as below. If you have a high-availability setup, this - # must be present in all application nodes. + # password specified as below. + # This must be present in all application nodes. gitlab_rails['db_password'] = '<your_password_here>' # Enable Foreign Data Wrapper diff --git a/doc/administration/git_annex.md b/doc/administration/git_annex.md index 4ac70e7fac2..0d44ed9312c 100644 --- a/doc/administration/git_annex.md +++ b/doc/administration/git_annex.md @@ -5,11 +5,11 @@ disqus_identifier: 'https://docs.gitlab.com/ee/workflow/git_annex.html' # Git annex > **Warning:** GitLab has [completely -removed][deprecate-annex-issue] in GitLab 9.0 (2017/03/22). -Read through the [migration guide from git-annex to Git LFS][guide]. +removed](https://gitlab.com/gitlab-org/gitlab/issues/1648) in GitLab 9.0 (2017/03/22). +Read through the [migration guide from git-annex to Git LFS](../topics/git/lfs/migrate_from_git_annex_to_git_lfs.md). The biggest limitation of Git, compared to some older centralized version -control systems, has been the maximum size of the repositories. +control systems has been the maximum size of the repositories. The general recommendation is to not have Git repositories larger than 1GB to preserve performance. Although GitLab has no limit (some repositories in GitLab @@ -21,10 +21,10 @@ larger organizations. Videos, photos, audio, compiled binaries, and many other types of files are too large. As a workaround, people keep artwork-in-progress in a Dropbox folder and only check in the final result. This results in using outdated files, not -having a complete history and increases the risk of losing work. +having a complete history, and increases the risk of losing work. This problem is solved in GitLab Enterprise Edition by integrating the -[git-annex] application. +[git-annex](https://git-annex.branchable.com/) application. `git-annex` allows managing large binaries with Git without checking the contents into Git. @@ -39,7 +39,7 @@ configuration options required to enable it. ### Requirements -`git-annex` needs to be installed both on the server and the client side. +`git-annex` needs to be installed both on the server and the client-side. For Debian-like systems (for example, Debian and Ubuntu) this can be achieved by running: @@ -64,7 +64,7 @@ The Omnibus package will internally set the correct options in all locations. gitlab_shell['git_annex_enabled'] = true ``` -1. Save the file and [reconfigure GitLab][] for the changes to take effect. +1. Save the file and [reconfigure GitLab](restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. ### Configuration for installations from source @@ -86,7 +86,7 @@ one is located in `config.yml` of GitLab Shell. git_annex_enabled: true ``` -1. Save the files and [restart GitLab][] for the changes to take effect. +1. Save the files and [restart GitLab](restart_gitlab.md#installations-from-source) for the changes to take effect. ## Using GitLab git-annex @@ -186,17 +186,17 @@ access files of projects you have access to (developer, maintainer, or owner rol ## How it works -Internally GitLab uses [GitLab Shell] to handle SSH access and this was a great +Internally GitLab uses [GitLab Shell](https://gitlab.com/gitlab-org/gitlab-shell) to handle SSH access and this was a great integration point for `git-annex`. There is a setting in GitLab Shell so you can disable GitLab Annex support if you want to. ## Troubleshooting tips -Differences in version of `git-annex` on the GitLab server and on local machines +Differences in the version of `git-annex` on the GitLab server and on local machines can cause `git-annex` to raise unpredicted warnings and errors. -Consult the [Annex upgrade page][annex-upgrade] for more information about +Consult the [Annex upgrade page](https://git-annex.branchable.com/upgrades/) for more information about the differences between versions. You can find out which version is installed on your server by navigating to <https://pkgs.org/download/git-annex> and searching for your distribution. @@ -208,7 +208,7 @@ on how to go around the warnings. This warning can appear on the initial `git annex sync --content` and is caused by differences in `git-annex-shell`. You can read more about it -[in this git-annex issue][issue]. +[in this git-annex issue](https://git-annex.branchable.com/forum/Error_from_git-annex-shell_on_creation_of_gcrypt_special_remote/). One important thing to note is that despite the warning, the `sync` succeeds and the files are pushed to the GitLab repository. @@ -231,12 +231,3 @@ pull origin ok push origin ``` - -[annex-upgrade]: https://git-annex.branchable.com/upgrades/ -[deprecate-annex-issue]: https://gitlab.com/gitlab-org/gitlab/issues/1648 -[git-annex]: https://git-annex.branchable.com/ "git-annex website" -[gitlab shell]: https://gitlab.com/gitlab-org/gitlab-shell "GitLab Shell repository" -[guide]: ../topics/git/lfs/migrate_from_git_annex_to_git_lfs.md -[issue]: https://git-annex.branchable.com/forum/Error_from_git-annex-shell_on_creation_of_gcrypt_special_remote/ "git-annex issue" -[reconfigure GitLab]: restart_gitlab.md#omnibus-gitlab-reconfigure -[restart GitLab]: restart_gitlab.md#installations-from-source diff --git a/doc/administration/gitaly/img/praefect_architecture_v12_10.png b/doc/administration/gitaly/img/praefect_architecture_v12_10.png Binary files differindex 7b8f1138b23..024a12b0a5d 100644 --- a/doc/administration/gitaly/img/praefect_architecture_v12_10.png +++ b/doc/administration/gitaly/img/praefect_architecture_v12_10.png diff --git a/doc/administration/gitaly/index.md b/doc/administration/gitaly/index.md index 20b9f2104de..14b0a6bd450 100644 --- a/doc/administration/gitaly/index.md +++ b/doc/administration/gitaly/index.md @@ -10,6 +10,11 @@ On this page, *Gitaly server* refers to a standalone node that only runs Gitaly and *Gitaly client* is a GitLab Rails app node that runs all other processes except Gitaly. +CAUTION: **Caution:** +From GitLab 13.0, using NFS for Git repositories is deprecated. In GitLab 14.0, +support for NFS for Git repositories is scheduled to be removed. Upgrade to +[Gitaly Cluster](praefect.md) as soon as possible. + ## Architecture Here's a high-level architecture overview of how Gitaly is used. @@ -69,7 +74,7 @@ The following list depicts what the network architecture of Gitaly is: - A GitLab server can use one or more Gitaly servers. - Gitaly addresses must be specified in such a way that they resolve correctly for ALL Gitaly clients. -- Gitaly clients are: Unicorn, Sidekiq, GitLab Workhorse, +- Gitaly clients are: Puma/Unicorn, Sidekiq, GitLab Workhorse, GitLab Shell, Elasticsearch Indexer, and Gitaly itself. - A Gitaly server must be able to make RPC calls **to itself** via its own `(Gitaly address, Gitaly token)` pair as specified in `/config/gitlab.yml`. @@ -101,13 +106,16 @@ Omnibus GitLab or install it from source: **_do not_** provide the `EXTERNAL_URL=` value. - From source: [Install Gitaly](../../install/installation.md#install-gitaly). -### 2. Client side token configuration +### 2. Authentication -Configure a token on the instance that runs the GitLab Rails application. +Gitaly and GitLab use two shared secrets for authentication, one to authenticate gRPC requests +to Gitaly, and a second for authentication callbacks from GitLab-Shell to the GitLab internal API. **For Omnibus GitLab** -1. On the client node(s), edit `/etc/gitlab/gitlab.rb`: +To configure the Gitaly token: + +1. On the client server, edit `/etc/gitlab/gitlab.rb`: ```ruby gitlab_rails['gitaly_token'] = 'abc123secret' @@ -115,9 +123,42 @@ Configure a token on the instance that runs the GitLab Rails application. 1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). +1. On the Gitaly server, edit `/etc/gitlab/gitlab.rb`: + + ```ruby + gitaly['auth_token'] = 'abc123secret' + ``` + +1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). + +There are two ways to configure the GitLab-Shell token: + +1. Copy `/etc/gitlab/gitlab-secrets.json` from the client server to same path on the Gitaly server. +1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). + +**OR** + +1. On the client server, edit `/etc/gitlab/gitlab.rb`: + + ```ruby + gitlab_shell['secret_token'] = 'shellsecret' + ``` + +1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). + +1. On the Gitaly server, edit `/etc/gitlab/gitlab.rb`: + + ```ruby + gitlab_shell['secret_token'] = 'shellsecret' + ``` + +1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). + **For installations from source** -1. On the client node(s), edit `/home/git/gitlab/config/gitlab.yml`: +1. Copy `/home/git/gitlab/.gitlab_shell_secret` from the client server to the same path on the Gitaly +server. +1. On the client server, edit `/home/git/gitlab/config/gitlab.yml`: ```yaml gitlab: @@ -138,12 +179,6 @@ documentation on configuring Gitaly authentication](https://gitlab.com/gitlab-org/gitaly/blob/master/doc/configuration/README.md#authentication) . -Gitaly must trigger some callbacks to GitLab via GitLab Shell. As a result, -the GitLab Shell secret must be the same between the other GitLab servers and -the Gitaly server. The easiest way to accomplish this is to copy `/etc/gitlab/gitlab-secrets.json` -from an existing GitLab server to the Gitaly server. Without this shared secret, -Git operations in GitLab will result in an API error. - **For Omnibus GitLab** 1. Edit `/etc/gitlab/gitlab.rb`: @@ -160,17 +195,17 @@ Git operations in GitLab will result in an API error. postgresql['enable'] = false redis['enable'] = false nginx['enable'] = false - unicorn['enable'] = false + puma['enable'] = false sidekiq['enable'] = false gitlab_workhorse['enable'] = false grafana['enable'] = false - # If you run a seperate monitoring node you can disable these services + # If you run a separate monitoring node you can disable these services alertmanager['enable'] = false prometheus['enable'] = false - # If you don't run a seperate monitoring node you can - # Enable Prometheus access & disable these extra services + # If you don't run a separate monitoring node you can + # enable Prometheus access & disable these extra services. # This makes Prometheus listen on all interfaces. You must use firewalls to restrict access to this address/port. # prometheus['listen_address'] = '0.0.0.0:9090' # prometheus['monitor_kubernetes'] = false @@ -189,10 +224,6 @@ Git operations in GitLab will result in an API error. # Don't forget to copy `/etc/gitlab/gitlab-secrets.json` from web server to Gitaly server. gitlab_rails['internal_api_url'] = 'https://gitlab.example.com' - # Authentication token to ensure only authorized servers can communicate with - # Gitaly server - gitaly['auth_token'] = 'abc123secret' - # Make Gitaly accept connections on all network interfaces. You must use # firewalls to restrict access to this address/port. # Comment out following line if you only want to support TLS connections @@ -230,6 +261,8 @@ Git operations in GitLab will result in an API error. ``` 1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). +1. Run `sudo /opt/gitlab/embedded/service/gitlab-shell/bin/check -config /opt/gitlab/embedded/service/gitlab-shell/config.yml` +to confirm that Gitaly can perform callbacks to the internal API. **For installations from source** @@ -271,7 +304,15 @@ Git operations in GitLab will result in an API error. path = '/srv/gitlab/git-data/repositories' ``` +1. On each Gitaly server, edit `/home/git/gitlab-shell/config.yml`: + + ```yaml + gitlab_url: https://gitlab.example.com + ``` + 1. Save the file and [restart GitLab](../restart_gitlab.md#installations-from-source). +1. Run `sudo -u git /home/git/gitlab-shell/bin/check -config /home/git/gitlab-shell/config.yml` +to confirm that Gitaly can perform callbacks to the internal API. ### 4. Converting clients to use the Gitaly server @@ -302,11 +343,10 @@ can read and write to `/mnt/gitlab/storage2`. 'storage1' => { 'gitaly_address' => 'tcp://gitaly1.internal:8075' }, 'storage2' => { 'gitaly_address' => 'tcp://gitaly2.internal:8075' }, }) - - gitlab_rails['gitaly_token'] = 'abc123secret' ``` 1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). +1. Run `sudo gitlab-rake gitlab:gitaly:check` to confirm the client can connect to Gitaly. 1. Tail the logs to see the requests: ```shell @@ -330,9 +370,6 @@ can read and write to `/mnt/gitlab/storage2`. storage2: gitaly_address: tcp://gitaly2.internal:8075 path: /some/dummy/path - - gitaly: - token: 'abc123secret' ``` NOTE: **Note:** @@ -341,6 +378,8 @@ can read and write to `/mnt/gitlab/storage2`. [this issue](https://gitlab.com/gitlab-org/gitaly/issues/1282) is resolved. 1. Save the file and [restart GitLab](../restart_gitlab.md#installations-from-source). +1. Run `sudo -u git -H bundle exec rake gitlab:gitaly:check RAILS_ENV=production` to +confirm the client can connect to Gitaly. 1. Tail the logs to see the requests: ```shell @@ -397,9 +436,9 @@ with a Gitaly instance that listens for secure connections you will need to use scheme in the `gitaly_address` of the corresponding storage entry in the GitLab configuration. You will need to bring your own certificates as this isn't provided automatically. -The certificate to be used needs to be installed on all Gitaly nodes, and the -certificate (or CA of certificate) on all -client nodes that communicate with it following the procedure described in +The certificate, or its certificate authority, must be installed on all Gitaly +nodes (including the Gitaly node using the certificate) and on all client nodes +that communicate with it following the procedure described in [GitLab custom certificate configuration](https://docs.gitlab.com/omnibus/settings/ssl.html#install-custom-public-certificates). NOTE: **Note** @@ -430,17 +469,32 @@ To configure Gitaly with TLS: }) gitlab_rails['gitaly_token'] = 'abc123secret' + gitlab_shell['secret_token'] = 'shellsecret' ``` 1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) on client node(s). +1. On the client node(s), copy the cert into the `/etc/gitlab/trusted-certs`: + + ```shell + sudo cp cert.pem /etc/gitlab/trusted-certs/ + ``` + 1. On the Gitaly server, create the `/etc/gitlab/ssl` directory and copy your key and certificate there: ```shell sudo mkdir -p /etc/gitlab/ssl sudo chmod 755 /etc/gitlab/ssl sudo cp key.pem cert.pem /etc/gitlab/ssl/ + sudo chmod 644 key.pem cert.pem ``` +1. Copy the cert to `/etc/gitlab/trusted-certs` so Gitaly will trust the cert when +calling into itself: + + ```shell + sudo cp /etc/gitlab/ssl/cert.pem /etc/gitlab/trusted-certs/ + ``` + 1. On the Gitaly server node(s), edit `/etc/gitlab/gitlab.rb` and add: <!-- @@ -463,6 +517,13 @@ To configure Gitaly with TLS: **For installations from source** +1. On the client node(s), add the cert to the system trusted certs: + + ```shell + sudo cp cert.pem /usr/local/share/ca-certificates/gitaly.crt + sudo update-ca-certificates + ``` + 1. On the client node(s), edit `/home/git/gitlab/config/gitlab.yml` as follows: ```yaml @@ -488,13 +549,32 @@ To configure Gitaly with TLS: data will be stored in this folder. This will no longer be necessary after [this issue](https://gitlab.com/gitlab-org/gitaly/issues/1282) is resolved. -1. Save the file and [restart GitLab](../restart_gitlab.md#installations-from-source) on client node(s). +1. Save the file and[restart GitLab](../restart_gitlab.md#installations-from-source) +on client node(s). +1. Copy `/home/git/gitlab/.gitlab_shell_secret` from the client server to the same +path on the Gitaly server. +1. On the Gitaly server, create or edit `/etc/default/gitlab` and add: + + ```shell + export SSL_CERT_DIR=/etc/gitlab/ssl + ``` + +1. Save the file. 1. Create the `/etc/gitlab/ssl` directory and copy your key and certificate there: ```shell sudo mkdir -p /etc/gitlab/ssl - sudo chmod 700 /etc/gitlab/ssl + sudo chmod 755 /etc/gitlab/ssl sudo cp key.pem cert.pem /etc/gitlab/ssl/ + sudo chmod 644 key.pem cert.pem + ``` + +1. On the Gitaly server, add the cert to the system trusted certs so Gitaly will trust it +when calling into itself: + + ```shell + sudo cp cert.pem /usr/local/share/ca-certificates/gitaly.crt + sudo update-ca-certificates ``` 1. On the Gitaly server node(s), edit `/home/git/gitaly/config.toml` and add: @@ -781,7 +861,7 @@ two checks. The result of both of these checks is cached. see if we can access filesystem underneath the Gitaly server directly. If so, use the Rugged patch. -To see if GitLab Rails can access the repo filesystem directly, we use +To see if GitLab Rails can access the repository filesystem directly, we use the following heuristic: - Gitaly ensures that the filesystem has a metadata file in its root diff --git a/doc/administration/gitaly/praefect.md b/doc/administration/gitaly/praefect.md index 19a69dc348c..0ea83f0e090 100644 --- a/doc/administration/gitaly/praefect.md +++ b/doc/administration/gitaly/praefect.md @@ -1,35 +1,70 @@ -# Praefect: High Availability +--- +type: reference +--- -NOTE: **Note:** Praefect is an experimental service, and data loss is likely. +# Gitaly Cluster -Praefect is an optional reverse-proxy for [Gitaly](../index.md) to manage a -cluster of Gitaly nodes for high availability. Initially, high availability -be implemented through asynchronous replication. If a Gitaly node becomes -unavailable, it will be possible to fail over to a warm Gitaly replica. +[Gitaly](index.md), the service that provides storage for Git repositories, can +be run in a clustered configuration to increase fault tolerance. In this +configuration, every Git repository is stored on every Gitaly node in the +cluster. Multiple clusters (or shards), can be configured. -The first minimal version will support: +NOTE: **Note:** +Gitaly Clusters can be created using [GitLab Core](https://about.gitlab.com/pricing/#self-managed) +and higher tiers. However, technical support is limited to GitLab Premium and Ultimate customers +only. Not available in GitLab.com. + +Praefect is a router and transaction manager for Gitaly, and a required +component for running a Gitaly Cluster. + + + +Using a Gitaly Cluster increase fault tolerance by: + +- Replicating write operations to warm standby Gitaly nodes. +- Detecting Gitaly node failures. +- Automatically routing Git requests to an available Gitaly node. + +The availability objectives for Gitaly clusters are: + +- **Recovery Point Objective (RPO):** Less than 1 minute. + + Writes are replicated asynchronously. Any writes that have not been replicated + to the newly promoted primary are lost. + + [Strong Consistency](https://gitlab.com/groups/gitlab-org/-/epics/1189) is + planned to improve this to "no loss". + +- **Recovery Time Objective (RTO):** Less than 10 seconds. + + Outages are detected by a health checks run by each Praefect node every + second. Failover requires ten consecutive failed health checks on each + Praefect node. + + [Faster outage detection](https://gitlab.com/gitlab-org/gitaly/-/issues/2608) + is planned to improve this to less than 1 second. + +The current version supports: - Eventual consistency of the secondary replicas. -- Automatic fail over from the primary to the secondary. +- Automatic failover from the primary to the secondary. - Reporting of possible data loss if replication queue is non empty. +- Marking the newly promoted primary read only if possible data loss is + detected. Follow the [HA Gitaly epic](https://gitlab.com/groups/gitlab-org/-/epics/1489) -for updates and roadmap. - -## Requirements for configuring Gitaly for High Availability +for improvements including +[horizontally distributing reads](https://gitlab.com/groups/gitlab-org/-/epics/2013). -NOTE: **Note:** this reference architecture is not highly available because -Praefect is a single point of failure. +## Requirements for configuring a Gitaly Cluster -The minimal [alpha](https://about.gitlab.com/handbook/product/#alpha-beta-ga) -reference architecture additionally requires: +The minimum recommended configuration for a Gitaly Cluster requires: -- 1 Praefect node -- 1 PostgreSQL server (PostgreSQL 9.6 or newer) +- 1 load balancer +- 1 PostgreSQL server (PostgreSQL 11 or newer) +- 3 Praefect nodes - 3 Gitaly nodes (1 primary, 2 secondary) - - See the [design document](https://gitlab.com/gitlab-org/gitaly/-/blob/master/doc/design_ha.md) for implementation details. @@ -43,6 +78,7 @@ package (highly recommended), follow the steps below: 1. [Configuring the Praefect database](#postgresql) 1. [Configuring the Praefect proxy/router](#praefect) 1. [Configuring each Gitaly node](#gitaly) (once for each Gitaly node) +1. [Configure the load balancer](#load-balancer) 1. [Updating the GitLab server configuration](#gitlab) 1. [Configure Grafana](#grafana) @@ -51,8 +87,8 @@ package (highly recommended), follow the steps below: Before beginning, you should already have a working GitLab instance. [Learn how to install GitLab](https://about.gitlab.com/install/). -Provision a PostgreSQL server (PostgreSQL 9.6 or newer). Configuration through -the GitLab Omnibus distribution is not yet supported. Follow this +Provision a PostgreSQL server (PostgreSQL 11 or newer). Configuration through +the Omnibus GitLab distribution is not yet supported. Follow this [issue](https://gitlab.com/gitlab-org/gitaly/issues/2476) for updates. Prepare all your new nodes by [installing @@ -64,6 +100,7 @@ GitLab](https://about.gitlab.com/install/). You will need the IP/host address for each node. +1. `LOAD_BALANCER_SERVER_ADDRESS`: the IP/hots address of the load balancer 1. `POSTGRESQL_SERVER_ADDRESS`: the IP/host address of the PostgreSQL server 1. `PRAEFECT_HOST`: the IP/host address of the Praefect server 1. `GITALY_HOST`: the IP/host address of each Gitaly server @@ -98,17 +135,19 @@ We will note in the instructions below where these secrets are required. ### PostgreSQL -NOTE: **Note:** don't reuse the GitLab application database for the Praefect -database. +NOTE: **Note:** do not store the GitLab application database and the Praefect +database on the same PostgreSQL server if using +[Geo](../geo/replication/index.md). The replication state is internal to each instance +of GitLab and should not be replicated. To complete this section you will need: - 1 Praefect node -- 1 PostgreSQL server (PostgreSQL 9.6 or newer) +- 1 PostgreSQL server (PostgreSQL 11 or newer) - An SQL user with permissions to create databases During this section, we will configure the PostgreSQL server, from the Praefect -node, using `psql` which is installed by GitLab Omnibus. +node, using `psql` which is installed by Omnibus GitLab. 1. SSH into the **Praefect** node and login as root: @@ -173,7 +212,7 @@ application server, or a Gitaly node. nginx['enable'] = false prometheus['enable'] = false grafana['enable'] = false - unicorn['enable'] = false + puma['enable'] = false sidekiq['enable'] = false gitlab_workhorse['enable'] = false gitaly['enable'] = false @@ -189,16 +228,12 @@ application server, or a Gitaly node. 1. Configure **Praefect** to listen on network interfaces by editing `/etc/gitlab/gitlab.rb`: - You will need to replace: - - - `PRAEFECT_HOST` with the IP address or hostname of the Praefect node - ```ruby - praefect['listen_addr'] = 'PRAEFECT_HOST:2305' + praefect['listen_addr'] = '0.0.0.0:2305' # Enable Prometheus metrics access to Praefect. You must use firewalls # to restrict access to this address/port. - praefect['prometheus_listen_addr'] = 'PRAEFECT_HOST:9652' + praefect['prometheus_listen_addr'] = '0.0.0.0:9652' ``` 1. Configure a strong `auth_token` for **Praefect** by editing @@ -245,9 +280,9 @@ application server, or a Gitaly node. 1. Configure the **Praefect** cluster to connect to each Gitaly node in the cluster by editing `/etc/gitlab/gitlab.rb`. - In the example below we have configured one cluster named `praefect`. This - cluster has three Gitaly nodes `gitaly-1`, `gitaly-2`, and `gitaly-3`, which - will be replicas of each other. + In the example below we have configured one virtual storage (or shard) named + `storage-1`. This cluster has three Gitaly nodes `gitaly-1`, `gitaly-2`, and + `gitaly-3`, which will be replicas of each other. Replace `PRAEFECT_INTERNAL_TOKEN` with a strong secret, which will be used by Praefect when communicating with Gitaly nodes in the cluster. This token is @@ -260,13 +295,13 @@ application server, or a Gitaly node. NOTE: **Note:** The `gitaly-1` node is currently denoted the primary. This can be used to manually fail from one node to another. This will be removed - in the future to allow for automatic failover. + in the [future](https://gitlab.com/gitlab-org/gitaly/-/issues/2634). ```ruby # Name of storage hash must match storage name in git_data_dirs on GitLab # server ('praefect') and in git_data_dirs on Gitaly nodes ('gitaly-1') praefect['virtual_storages'] = { - 'praefect' => { + 'storage-1' => { 'gitaly-1' => { 'address' => 'tcp://GITALY_HOST:8075', 'token' => 'PRAEFECT_INTERNAL_TOKEN', @@ -284,75 +319,63 @@ application server, or a Gitaly node. } ``` -1. Save the changes to `/etc/gitlab/gitlab.rb` and [reconfigure Praefect](../restart_gitlab.md#omnibus-gitlab-reconfigure): +1. Enable the database replication queue: - ```shell - gitlab-ctl reconfigure + ```ruby + praefect['postgres_queue_enabled'] = true ``` -1. Verify that Praefect can reach PostgreSQL: + In the next release, database replication queue will be enabled by default. + See [issue #2615](https://gitlab.com/gitlab-org/gitaly/-/issues/2615). - ```shell - sudo -u git /opt/gitlab/embedded/bin/praefect -config /var/opt/gitlab/praefect/config.toml sql-ping +1. Enable automatic failover by editing `/etc/gitlab/gitlab.rb`: + + ```ruby + praefect['failover_enabled'] = true + praefect['failover_election_strategy'] = 'sql' ``` - If the check fails, make sure you have followed the steps correctly. If you - edit `/etc/gitlab/gitlab.rb`, remember to run `sudo gitlab-ctl reconfigure` - again before trying the `sql-ping` command. + When automatic failover is enabled, Praefect checks the health of internal + Gitaly nodes. If the primary has a certain amount of health checks fail, it + will promote one of the secondaries to be primary, and demote the primary to + be a secondary. -#### Automatic failover + NOTE: **Note:** Database leader election will be [enabled by default in the + future](https://gitlab.com/gitlab-org/gitaly/-/issues/2682). -When automatic failover is enabled, Praefect will do automatic detection of the health of internal Gitaly nodes. If the -primary has a certain amount of health checks fail, it will decide to promote one of the secondaries to be primary, and -demote the primary to be a secondary. + Caution, **automatic failover** favors availability over consistency and will + cause data loss if changes have not been replicated to the newly elected + primary. In the next release, leader election will [prefer to promote up to + date replicas](https://gitlab.com/gitlab-org/gitaly/-/issues/2642), and it + will be an option to favor consistency by marking [out-of-date repositories + read-only](https://gitlab.com/gitlab-org/gitaly/-/issues/2630). -1. To enable automatic failover, edit `/etc/gitlab/gitlab.rb`: +1. Save the changes to `/etc/gitlab/gitlab.rb` and [reconfigure + Praefect](../restart_gitlab.md#omnibus-gitlab-reconfigure): - ```ruby - # failover_enabled turns on automatic failover - praefect['failover_enabled'] = true - praefect['virtual_storages'] = { - 'praefect' => { - 'gitaly-1' => { - 'address' => 'tcp://GITALY_HOST:8075', - 'token' => 'PRAEFECT_INTERNAL_TOKEN', - 'primary' => true - }, - 'gitaly-2' => { - 'address' => 'tcp://GITALY_HOST:8075', - 'token' => 'PRAEFECT_INTERNAL_TOKEN' - }, - 'gitaly-3' => { - 'address' => 'tcp://GITALY_HOST:8075', - 'token' => 'PRAEFECT_INTERNAL_TOKEN' - } - } - } + ```shell + gitlab-ctl reconfigure ``` -1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). +1. To ensure that Praefect [has updated its Prometheus listen + address](https://gitlab.com/gitlab-org/gitaly/-/issues/2734), [restart + Gitaly](../restart_gitlab.md#omnibus-gitlab-restart): -Below is the picture when Praefect starts up with the config.toml above: + ```shell + gitlab-ctl restart praefect + ``` -```mermaid -graph TD - A[Praefect] -->|Mutator RPC| B(internal_storage_0) - B --> |Replication|C[internal_storage_1] -``` +1. Verify that Praefect can reach PostgreSQL: -Let's say suddenly `internal_storage_0` goes down. Praefect will detect this and -automatically switch over to `internal_storage_1`, and `internal_storage_0` will serve as a secondary: + ```shell + sudo -u git /opt/gitlab/embedded/bin/praefect -config /var/opt/gitlab/praefect/config.toml sql-ping + ``` -```mermaid -graph TD - A[Praefect] -->|Mutator RPC| B(internal_storage_1) - B --> |Replication|C[internal_storage_0] -``` + If the check fails, make sure you have followed the steps correctly. If you + edit `/etc/gitlab/gitlab.rb`, remember to run `sudo gitlab-ctl reconfigure` + again before trying the `sql-ping` command. -NOTE: **Note:**: Currently this feature is supported for setups that only have 1 Praefect instance. Praefect instances running, -for example behind a load balancer, `failover_enabled` should be disabled. The reason is The reason is because there -is no coordination that currently happens across different Praefect instances, so there could be a situation where -two Praefect instances think two different Gitaly nodes are the primary. +**The steps above must be completed for each Praefect node!** ### Gitaly @@ -365,7 +388,7 @@ To complete this section you will need: These should be dedicated nodes, do not run other services on these nodes. Every Gitaly server assigned to the Praefect cluster needs to be configured. The -configuration is the same as a normal [standalone Gitaly server](../index.md), +configuration is the same as a normal [standalone Gitaly server](index.md), except: - the storage names are exposed to Praefect, not GitLab @@ -403,7 +426,7 @@ documentation](index.md#3-gitaly-server-configuration). nginx['enable'] = false prometheus['enable'] = false grafana['enable'] = false - unicorn['enable'] = false + puma['enable'] = false sidekiq['enable'] = false gitlab_workhorse['enable'] = false prometheus_monitoring['enable'] = false @@ -419,18 +442,14 @@ documentation](index.md#3-gitaly-server-configuration). 1. Configure **Gitaly** to listen on network interfaces by editing `/etc/gitlab/gitlab.rb`: - You will need to replace: - - - `GITALY_HOST` with the IP address or hostname of the Gitaly node - ```ruby # Make Gitaly accept connections on all network interfaces. # Use firewalls to restrict access to this address/port. - gitaly['listen_addr'] = 'GITALY_HOST:8075' + gitaly['listen_addr'] = '0.0.0.0:8075' # Enable Prometheus metrics access to Gitaly. You must use firewalls # to restrict access to this address/port. - gitaly['prometheus_listen_addr'] = 'GITALY_HOST:9236' + gitaly['prometheus_listen_addr'] = '0.0.0.0:9236' ``` 1. Configure a strong `auth_token` for **Gitaly** by editing @@ -445,7 +464,7 @@ documentation](index.md#3-gitaly-server-configuration). 1. Configure the GitLab Shell `secret_token`, and `internal_api_url` which are needed for `git push` operations. - If you have already configured [Gitaly on its own server](../index.md) + If you have already configured [Gitaly on its own server](index.md) ```ruby gitlab_shell['secret_token'] = 'GITLAB_SHELL_SECRET_TOKEN' @@ -458,12 +477,12 @@ documentation](index.md#3-gitaly-server-configuration). 1. Configure the storage location for Git data by setting `git_data_dirs` in `/etc/gitlab/gitlab.rb`. Each Gitaly node should have a unique storage name - (eg `gitaly-1`). + (such as `gitaly-1`). Instead of configuring `git_data_dirs` uniquely for each Gitaly node, it is often easier to have include the configuration for all Gitaly nodes on every Gitaly node. This is supported because the Praefect `virtual_storages` - configuration maps each storage name (eg `gitaly-1`) to a specific node, and + configuration maps each storage name (such as `gitaly-1`) to a specific node, and requests are routed accordingly. This means every Gitaly node in your fleet can share the same configuration. @@ -484,13 +503,16 @@ documentation](index.md#3-gitaly-server-configuration). }) ``` -1. Save the changes to `/etc/gitlab/gitlab.rb` and [reconfigure Gitaly](../restart_gitlab.md#omnibus-gitlab-reconfigure): +1. Save the changes to `/etc/gitlab/gitlab.rb` and [reconfigure + Gitaly](../restart_gitlab.md#omnibus-gitlab-reconfigure): ```shell gitlab-ctl reconfigure ``` -1. To ensure that Gitaly [has updated its Prometheus listen address](https://gitlab.com/gitlab-org/gitaly/-/issues/2521), [restart Gitaly](../restart_gitlab.md#omnibus-gitlab-restart): +1. To ensure that Gitaly [has updated its Prometheus listen + address](https://gitlab.com/gitlab-org/gitaly/-/issues/2734), [restart + Gitaly](../restart_gitlab.md#omnibus-gitlab-restart): ```shell gitlab-ctl restart gitaly @@ -508,37 +530,23 @@ config. sudo /opt/gitlab/embedded/bin/praefect -config /var/opt/gitlab/praefect/config.toml dial-nodes ``` -1. Enable automatic failover by editing `/etc/gitlab/gitlab.rb`: +### Load Balancer - ```ruby - praefect['failover_enabled'] = true - ``` - - When automatic failover is enabled, Praefect checks the health of internal - Gitaly nodes. If the primary has a certain amount of health checks fail, it - will promote one of the secondaries to be primary, and demote the primary to - be a secondary. +In a highly available Gitaly configuration, a load balancer is needed to route +internal traffic from the GitLab application to the Praefect nodes. The +specifics on which load balancer to use or the exact configuration is beyond the +scope of the GitLab documentation. - Manual failover is possible by updating `praefect['virtual_storages']` and - nominating a new primary node. +We hope that if you’re managing HA systems like GitLab, you have a load balancer +of choice already. Some examples include [HAProxy](https://www.haproxy.org/) +(open-source), [Google Internal Load Balancer](https://cloud.google.com/load-balancing/docs/internal/), +[AWS Elastic Load Balancer](https://aws.amazon.com/elasticloadbalancing/), F5 +Big-IP LTM, and Citrix Net Scaler. This documentation will outline what ports +and protocols you need configure. -1. By default, Praefect will nominate a primary Gitaly node for each - shard and store the state of the primary in local memory. This state - does not persist across restarts and will cause a split brain - if multiple Praefect nodes are used for redundancy. - - To avoid this limitation, enable the SQL election strategy: - - ```ruby - praefect['failover_election_strategy'] = 'sql' - ``` - -1. Save the changes to `/etc/gitlab/gitlab.rb` and [reconfigure - Praefect](../restart_gitlab.md#omnibus-gitlab-reconfigure): - - ```shell - gitlab-ctl reconfigure - ``` +| LB Port | Backend Port | Protocol | +|---------|--------------|----------| +| 2305 | 2305 | TCP | ### GitLab @@ -555,7 +563,7 @@ Particular attention should be shown to: - the storage name added to `git_data_dirs` in this section must match the storage name under `praefect['virtual_storages']` on the Praefect node. This was set in the [Praefect](#praefect) section of this guide. This document uses - `praefect` as the Praefect storage name. + `storage-1` as the Praefect storage name. 1. SSH into the **GitLab** node and login as root: @@ -563,12 +571,23 @@ Particular attention should be shown to: sudo -i ``` +1. Configure the `external_url` so that files could be served by GitLab + by proper endpoint access by editing `/etc/gitlab/gitlab.rb`: + + You will need to replace `GITLAB_SERVER_URL` with the real external facing + URL on which current GitLab instance is serving: + + ```ruby + external_url 'GITLAB_SERVER_URL' + ``` + 1. Add the Praefect cluster as a storage location by editing `/etc/gitlab/gitlab.rb`. You will need to replace: - - `PRAEFECT_HOST` with the IP address or hostname of the Praefect node + - `LOAD_BALANCER_SERVER_ADDRESS` with the IP address or hostname of the load + balancer. - `GITLAB_HOST` with the IP address or hostname of the GitLab server - `PRAEFECT_EXTERNAL_TOKEN` with the real secret @@ -577,18 +596,18 @@ Particular attention should be shown to: "default" => { "gitaly_address" => "tcp://GITLAB_HOST:8075" }, - "praefect" => { - "gitaly_address" => "tcp://PRAEFECT_HOST:2305", + "storage-1" => { + "gitaly_address" => "tcp://LOAD_BALANCER_SERVER_ADDRESS:2305", "gitaly_token" => 'PRAEFECT_EXTERNAL_TOKEN' } }) ``` -1. Allow Gitaly to listen on a tcp port by editing +1. Allow Gitaly to listen on a TCP port by editing `/etc/gitlab/gitlab.rb` ```ruby - gitaly['listen_addr'] = 'GITLAB_HOST:8075' + gitaly['listen_addr'] = '0.0.0.0:8075' ``` 1. Configure the `gitlab_shell['secret_token']` so that callbacks from Gitaly @@ -601,16 +620,6 @@ Particular attention should be shown to: gitlab_shell['secret_token'] = 'GITLAB_SHELL_SECRET_TOKEN' ``` -1. Configure the `external_url` so that files could be served by GitLab - by proper endpoint access by editing `/etc/gitlab/gitlab.rb`: - - You will need to replace `GITLAB_SERVER_URL` with the real external facing URL on which - current GitLab instance is serving: - - ```ruby - external_url 'GITLAB_SERVER_URL' - ``` - 1. Add Prometheus monitoring settings by editing `/etc/gitlab/gitlab.rb`. You will need to replace: @@ -624,7 +633,9 @@ Particular attention should be shown to: 'job_name' => 'praefect', 'static_configs' => [ 'targets' => [ - 'PRAEFECT_HOST:9652' # praefect + 'PRAEFECT_HOST:9652', # praefect-1 + 'PRAEFECT_HOST:9652', # praefect-2 + 'PRAEFECT_HOST:9652', # praefect-3 ] ] }, @@ -647,6 +658,14 @@ Particular attention should be shown to: gitlab-ctl reconfigure ``` +1. To ensure that Gitaly [has updated its Prometheus listen + address](https://gitlab.com/gitlab-org/gitaly/-/issues/2734), [restart + Gitaly](../restart_gitlab.md#omnibus-gitlab-restart): + + ```shell + gitlab-ctl restart gitaly + ``` + 1. Verify each `gitlab-shell` on each Gitaly instance can reach GitLab. On each Gitaly instance run: ```shell @@ -725,6 +744,13 @@ Praefect regularly checks the health of each backend Gitaly node. This information can be used to automatically failover to a new primary node if the current primary node is found to be unhealthy. +- **PostgreSQL (recommended):** Enabled by setting + `praefect['failover_election_strategy'] = sql`. This configuration + option will allow multiple Praefect nodes to coordinate via the + PostgreSQL database to elect a primary Gitaly node. This configuration + will cause Praefect nodes to elect a new primary, monitor its health, + and elect a new primary if the current one has not been reachable in + 10 seconds by a majority of the Praefect nodes. - **Manual:** Automatic failover is disabled. The primary node can be reconfigured in `/etc/gitlab/gitlab.rb` on the Praefect node. Modify the `praefect['virtual_storages']` field by moving the `primary = true` to promote @@ -735,13 +761,6 @@ current primary node is found to be unhealthy. checks fail for the current primary backend Gitaly node, and new primary will be elected. **Do not use with multiple Praefect nodes!** Using with multiple Praefect nodes is likely to result in a split brain. -- **PostgreSQL:** Enabled by setting - `praefect['failover_election_strategy'] = sql`. This configuration - option will allow multiple Praefect nodes to coordinate via the - PostgreSQL database to elect a primary Gitaly node. This configuration - will cause Praefect nodes to elect a new primary, monitor its health, - and elect a new primary if the current one has not been reachable in - 10 seconds by a majority of the Praefect nodes. NOTE: **Note:**: Praefect does not yet account for replication lag on the secondaries during the election process, so data loss can occur @@ -753,13 +772,13 @@ strategy in the future. ## Identifying Impact of a Primary Node Failure -When a primary Gitaly node fails, there is a chance of dataloss. Dataloss can occur if there were outstanding replication jobs the secondaries did not manage to process before the failure. The Praefect `dataloss` subcommand helps identify these cases by counting the number of dead replication jobs for each repository within a given timeframe. +When a primary Gitaly node fails, there is a chance of data loss. Data loss can occur if there were outstanding replication jobs the secondaries did not manage to process before the failure. The Praefect `dataloss` sub-command helps identify these cases by counting the number of dead replication jobs for each repository within a given time frame. ```shell sudo /opt/gitlab/embedded/bin/praefect -config /var/opt/gitlab/praefect/config.toml dataloss -from <rfc3339-time> -to <rfc3339-time> ``` -If the timeframe is not specified, dead replication jobs from the last six hours are counted: +If the time frame is not specified, dead replication jobs from the last six hours are counted: ```shell sudo /opt/gitlab/embedded/bin/praefect -config /var/opt/gitlab/praefect/config.toml dataloss @@ -770,19 +789,27 @@ example/repository-2: 4 jobs example/repository-3: 2 jobs ``` -To specify a timeframe in UTC, run: +To specify a time frame in UTC, run: ```shell sudo /opt/gitlab/embedded/bin/praefect -config /var/opt/gitlab/praefect/config.toml dataloss -from 2020-01-02T00:00:00+00:00 -to 2020-01-02T00:02:00+00:00 ``` +### Checking repository checksums + +To check a project's checksums across all nodes, the Praefect replicas Rake task can be used: + +```shell +sudo gitlab-rake "gitlab:praefect:replicas[project_id]" +``` + ## Backend Node Recovery When a Praefect backend node fails and is no longer able to replicate changes, the backend node will start to drift from the primary. If that node eventually recovers, it will need to be reconciled with the current primary. The primary node is considered the single source of truth for the -state of a shard. The Praefect `reconcile` subcommand allows for the manual +state of a shard. The Praefect `reconcile` sub-command allows for the manual reconciliation between a backend node and the current primary. Run the following command on the Praefect server after all placeholders diff --git a/doc/administration/gitaly/reference.md b/doc/administration/gitaly/reference.md index 6b6919247fe..e718d8953ca 100644 --- a/doc/administration/gitaly/reference.md +++ b/doc/administration/gitaly/reference.md @@ -149,7 +149,7 @@ before and after. If the hit ratio does not improve, the higher limit is probably not making a meaningful difference. Here is an example Prometheus query to see the hit rate: -```text +```plaintext sum(rate(gitaly_catfile_cache_total{type="hit"}[5m])) / sum(rate(gitaly_catfile_cache_total{type=~"(hit)|(miss)"}[5m])) ``` @@ -258,7 +258,7 @@ You can adjust the `concurrency` of each RPC endpoint. | ---- | ---- | -------- | ----------- | | `concurrency` | array | yes | An array of RPC endpoints. | | `rpc` | string | no | The name of the RPC endpoint (`/gitaly.RepositoryService/GarbageCollect`). | -| `max_per_repo` | integer | no | Concurrency per RPC per repo. | +| `max_per_repo` | integer | no | Concurrency per RPC per repository. | Example: diff --git a/doc/administration/high_availability/README.md b/doc/administration/high_availability/README.md index 55ec3b8d6c4..d36b029cbb3 100644 --- a/doc/administration/high_availability/README.md +++ b/doc/administration/high_availability/README.md @@ -1,7 +1,7 @@ --- -type: reference, concepts +redirect_to: ../reference_architectures/index.md --- -# High Availability +# Reference Architectures -This content has been moved to the [availability page](../availability/index.md). +This document was moved to [another location](../reference_architectures/index.md). diff --git a/doc/administration/high_availability/consul.md b/doc/administration/high_availability/consul.md index 6762a81f671..1f22c46a0ad 100644 --- a/doc/administration/high_availability/consul.md +++ b/doc/administration/high_availability/consul.md @@ -4,14 +4,17 @@ type: reference # Working with the bundled Consul service **(PREMIUM ONLY)** -As part of its High Availability stack, GitLab Premium includes a bundled version of [Consul](https://www.consul.io/) that can be managed through `/etc/gitlab/gitlab.rb`. +As part of its High Availability stack, GitLab Premium includes a bundled version of [Consul](https://www.consul.io/) that can be managed through `/etc/gitlab/gitlab.rb`. Consul is a service networking solution. When it comes to [GitLab Architecture](../../development/architecture.md), Consul utilization is supported for configuring: + +1. [Monitoring in Scaled and Highly Available environments](monitoring_node.md) +1. [PostgreSQL High Availability with Omnibus](database.md#high-availability-with-omnibus-gitlab-premium-only) A Consul cluster consists of multiple server agents, as well as client agents that run on other nodes which need to talk to the Consul cluster. ## Prerequisites First, make sure to [download/install](https://about.gitlab.com/install/) -GitLab Omnibus **on each node**. +Omnibus GitLab **on each node**. Choose an installation method, then make sure you complete steps: @@ -107,7 +110,7 @@ For larger clusters, it is possible to restart multiple agents at a time. See th Nodes running GitLab-bundled Consul should be: -- Members of a healthy cluster prior to upgrading the GitLab Omnibus package. +- Members of a healthy cluster prior to upgrading the Omnibus GitLab package. - Upgraded one node at a time. NOTE: **NOTE:** diff --git a/doc/administration/high_availability/database.md b/doc/administration/high_availability/database.md index f06870be93c..6f1873af993 100644 --- a/doc/administration/high_availability/database.md +++ b/doc/administration/high_availability/database.md @@ -24,16 +24,16 @@ If you use a cloud-managed service, or provide your own PostgreSQL: ## PostgreSQL in a Scaled and Highly Available Environment -This section is relevant for [Scalable and Highly Available Setups](../scaling/index.md). +This section is relevant for [Scalable and Highly Available Setups](../reference_architectures/index.md). ### Provide your own PostgreSQL instance **(CORE ONLY)** If you want to use your own deployed PostgreSQL instance(s), see [Provide your own PostgreSQL instance](#provide-your-own-postgresql-instance-core-only) -for more details. However, you can use the GitLab Omnibus package to easily +for more details. However, you can use the Omnibus GitLab package to easily deploy the bundled PostgreSQL. -### Standalone PostgreSQL using GitLab Omnibus **(CORE ONLY)** +### Standalone PostgreSQL using Omnibus GitLab **(CORE ONLY)** 1. SSH into the PostgreSQL server. 1. [Download/install](https://about.gitlab.com/install/) the Omnibus GitLab @@ -92,7 +92,7 @@ deploy the bundled PostgreSQL. Advanced configuration options are supported and can be added if needed. -### High Availability with GitLab Omnibus **(PREMIUM ONLY)** +### High Availability with Omnibus GitLab **(PREMIUM ONLY)** > Important notes: > @@ -125,7 +125,7 @@ otherwise the networks will become a single point of failure. #### Architecture - + Database nodes run two services with PostgreSQL: @@ -271,7 +271,7 @@ Few notes on the service itself: #### Installing Omnibus GitLab First, make sure to [download/install](https://about.gitlab.com/install/) -GitLab Omnibus **on each node**. +Omnibus GitLab **on each node**. Make sure you install the necessary dependencies from step 1, add GitLab package repository from step 2. @@ -969,7 +969,7 @@ repmgr['trust_auth_cidr_addresses'] = %w(192.168.1.44/32 db2.example.com) ##### MD5 Authentication If you are running on an untrusted network, repmgr can use md5 authentication -with a [`.pgpass` file](https://www.postgresql.org/docs/9.6/libpq-pgpass.html) +with a [`.pgpass` file](https://www.postgresql.org/docs/11/libpq-pgpass.html) to authenticate. You can specify by IP address, FQDN, or by subnet, using the same format as in @@ -1091,7 +1091,7 @@ If you're running into an issue with a component not outlined here, be sure to c ## Configure using Omnibus -**Note**: We recommend that you follow the instructions here for a full [PostgreSQL cluster](#high-availability-with-gitlab-omnibus-premium-only). +**Note**: We recommend that you follow the instructions here for a full [PostgreSQL cluster](#high-availability-with-omnibus-gitlab-premium-only). If you are reading this section due to an old bookmark, you can find that old documentation [in the repository](https://gitlab.com/gitlab-org/gitlab/blob/v10.1.4/doc/administration/high_availability/database.md#configure-using-omnibus). Read more on high-availability configuration: diff --git a/doc/administration/high_availability/gitaly.md b/doc/administration/high_availability/gitaly.md index 5d66d3c5c94..2e6bcabeb06 100644 --- a/doc/administration/high_availability/gitaly.md +++ b/doc/administration/high_availability/gitaly.md @@ -4,14 +4,10 @@ type: reference # Configuring Gitaly for Scaled and High Availability -Gitaly does not yet support full high availability. However, Gitaly is quite -stable and is in use on GitLab.com. Scaled and highly available GitLab environments -should consider using Gitaly on a separate node. +A [Gitaly Cluster](../gitaly/praefect.md) can be used to increase the fault +tolerance of Gitaly in high availability configurations. -See the [Gitaly HA Epic](https://gitlab.com/groups/gitlab-org/-/epics/289) to -track plans and progress toward high availability support. - -This document is relevant for [Scalable and Highly Available Setups](../scaling/index.md). +This document is relevant for [scalable and highly available setups](../reference_architectures/index.md). ## Running Gitaly on its own server @@ -19,7 +15,7 @@ See [Running Gitaly on its own server](../gitaly/index.md#running-gitaly-on-its- in Gitaly documentation. Continue configuration of other components by going back to the -[High Availability](../availability/index.md#gitlab-components-and-configuration-instructions) page. +[reference architecture](../reference_architectures/index.md#configure-gitlab-to-scale) page. ## Enable Monitoring diff --git a/doc/administration/high_availability/gitlab.md b/doc/administration/high_availability/gitlab.md index c9c425d366b..cdafdbc4954 100644 --- a/doc/administration/high_availability/gitlab.md +++ b/doc/administration/high_availability/gitlab.md @@ -2,7 +2,9 @@ type: reference --- -# Configuring GitLab for Scaling and High Availability +# Configuring GitLab application (Rails) + +This section describes how to configure the GitLab application (Rails) component. NOTE: **Note:** There is some additional configuration near the bottom for additional GitLab application servers. It's important to read and understand @@ -34,7 +36,7 @@ is recommended over [NFS](nfs.md) wherever possible for improved performance. mkdir -p /var/opt/gitlab/.ssh /var/opt/gitlab/gitlab-rails/uploads /var/opt/gitlab/gitlab-rails/shared /var/opt/gitlab/gitlab-ci/builds /var/opt/gitlab/git-data ``` -1. Download/install GitLab Omnibus using **steps 1 and 2** from +1. Download/install Omnibus GitLab using **steps 1 and 2** from [GitLab downloads](https://about.gitlab.com/install/). Do not complete other steps on the download page. 1. Create/edit `/etc/gitlab/gitlab.rb` and use the following configuration. @@ -79,8 +81,8 @@ is recommended over [NFS](nfs.md) wherever possible for improved performance. NOTE: **Note:** To maintain uniformity of links across HA clusters, the `external_url` on the first application server as well as the additional application - servers should point to the external url that users will use to access GitLab. - In a typical HA setup, this will be the url of the load balancer which will + servers should point to the external URL that users will use to access GitLab. + In a typical HA setup, this will be the URL of the load balancer which will route traffic to all GitLab application servers in the HA cluster. NOTE: **Note:** When you specify `https` in the `external_url`, as in the example @@ -131,6 +133,9 @@ need some extra configuration. 1. Run `sudo gitlab-ctl reconfigure` to compile the configuration. +NOTE: **Note:** You will need to restart the GitLab applications nodes after an update has occurred and database +migrations performed. + ## Enable Monitoring > [Introduced](https://gitlab.com/gitlab-org/omnibus-gitlab/issues/3786) in GitLab 12.0. @@ -157,7 +162,7 @@ If you enable Monitoring, it must be enabled on **all** GitLab servers. node_exporter['listen_address'] = '0.0.0.0:9100' gitlab_workhorse['prometheus_listen_addr'] = '0.0.0.0:9229' sidekiq['listen_address'] = "0.0.0.0" - unicorn['listen'] = '0.0.0.0' + puma['listen'] = '0.0.0.0' # Add the monitoring node's IP address to the monitoring whitelist and allow it to # scrape the NGINX metrics. Replace placeholder `monitoring.gitlab.example.com` with @@ -169,9 +174,11 @@ If you enable Monitoring, it must be enabled on **all** GitLab servers. 1. Run `sudo gitlab-ctl reconfigure` to compile the configuration. CAUTION: **Warning:** - After changing `unicorn['listen']` in `gitlab.rb`, and running `sudo gitlab-ctl reconfigure`, - it can take an extended period of time for Unicorn to complete reloading after receiving a `HUP`. - For more information, see the [issue](https://gitlab.com/gitlab-org/omnibus-gitlab/issues/4401). + If running Unicorn, after changing `unicorn['listen']` in `gitlab.rb`, and + running `sudo gitlab-ctl reconfigure`, it can take an extended period of time + for Unicorn to complete reloading after receiving a `HUP`. For more + information, see the + [issue](https://gitlab.com/gitlab-org/omnibus-gitlab/issues/4401). ## Troubleshooting diff --git a/doc/administration/high_availability/load_balancer.md b/doc/administration/high_availability/load_balancer.md index 43a7c442d8c..beeb57a0e32 100644 --- a/doc/administration/high_availability/load_balancer.md +++ b/doc/administration/high_availability/load_balancer.md @@ -66,7 +66,7 @@ for details on managing SSL certificates and configuring NGINX. | 443 | 443 | TCP or HTTPS (*1*) (*2*) | | 22 | 22 | TCP | -- (*1*): [Web terminal](../../ci/environments.md#web-terminals) support requires +- (*1*): [Web terminal](../../ci/environments/index.md#web-terminals) support requires your load balancer to correctly handle WebSocket connections. When using HTTP or HTTPS proxying, this means your load balancer must be configured to pass through the `Connection` and `Upgrade` hop-by-hop headers. See the diff --git a/doc/administration/high_availability/monitoring_node.md b/doc/administration/high_availability/monitoring_node.md index 79e583c5fcb..409a4dfecb9 100644 --- a/doc/administration/high_availability/monitoring_node.md +++ b/doc/administration/high_availability/monitoring_node.md @@ -8,10 +8,10 @@ type: reference You can configure a Prometheus node to monitor GitLab. -## Standalone Monitoring node using GitLab Omnibus +## Standalone Monitoring node using Omnibus GitLab -The GitLab Omnibus package can be used to configure a standalone Monitoring node running [Prometheus](../monitoring/prometheus/index.md) and [Grafana](../monitoring/performance/grafana_configuration.md). -The monitoring node is not highly available. See [Scaling and High Availability](../scaling/index.md) +The Omnibus GitLab package can be used to configure a standalone Monitoring node running [Prometheus](../monitoring/prometheus/index.md) and [Grafana](../monitoring/performance/grafana_configuration.md). +The monitoring node is not highly available. See [Scaling and High Availability](../reference_architectures/index.md) for an overview of GitLab scaling and high availability options. The steps below are the minimum necessary to configure a Monitoring node running Prometheus and Grafana with @@ -64,7 +64,7 @@ Omnibus: redis['enable'] = false redis_exporter['enable'] = false sidekiq['enable'] = false - unicorn['enable'] = false + puma['enable'] = false node_exporter['enable'] = false gitlab_exporter['enable'] = false ``` diff --git a/doc/administration/high_availability/nfs.md b/doc/administration/high_availability/nfs.md index 66f2986ab2a..d2b8cf65b35 100644 --- a/doc/administration/high_availability/nfs.md +++ b/doc/administration/high_availability/nfs.md @@ -7,14 +7,16 @@ type: reference You can view information and options set for each of the mounted NFS file systems by running `nfsstat -m` and `cat /etc/fstab`. +CAUTION: **Caution:** +From GitLab 13.0, using NFS for Git repositories is deprecated. In GitLab 14.0, +support for NFS for Git repositories is scheduled to be removed. Upgrade to +[Gitaly Cluster](../gitaly/praefect.md) as soon as possible. + NOTE: **Note:** Filesystem performance has a big impact on overall GitLab performance, especially for actions that read or write to Git repositories. See [Filesystem Performance Benchmarking](../operations/filesystem_benchmarking.md) for steps to test filesystem performance. -NOTE: **Note:** [Cloud Object Storage service](object_storage.md) with [Gitaly](gitaly.md) -is recommended over NFS wherever possible for improved performance. - ## NFS Server features ### Required features @@ -73,7 +75,7 @@ To disable NFS server delegation, do the following: #### Important notes The kernel bug may be fixed in -[more recent kernels with this commit](https://github.om/torvalds/linux/commit/95da1b3a5aded124dd1bda1e3cdb876184813140). +[more recent kernels with this commit](https://github.com/torvalds/linux/commit/95da1b3a5aded124dd1bda1e3cdb876184813140). Red Hat Enterprise 7 [shipped a kernel update](https://access.redhat.com/errata/RHSA-2019:2029) on August 6, 2019 that may also have resolved this problem. @@ -184,7 +186,7 @@ The NFS man page states: Read the [Linux man page](https://linux.die.net/man/5/nfs) to understand the difference, and if you do use `soft`, ensure that you've taken steps to mitigate the risks. -If you experience behaviour that might have been caused by +If you experience behavior that might have been caused by writes to disk on the NFS server not occurring, such as commits going missing, use the `hard` option, because (from the man page): @@ -271,7 +273,7 @@ following are the 4 locations need to be shared: Other GitLab directories should not be shared between nodes. They contain node-specific files and GitLab code that does not need to be shared. To ship -logs to a central location consider using remote syslog. GitLab Omnibus packages +logs to a central location consider using remote syslog. Omnibus GitLab packages provide configuration for [UDP log shipping](https://docs.gitlab.com/omnibus/settings/logs.html#udp-log-shipping-gitlab-enterprise-edition-only). Having multiple NFS mounts will require manually making sure the data directories diff --git a/doc/administration/high_availability/nfs_host_client_setup.md b/doc/administration/high_availability/nfs_host_client_setup.md index 6823c1d9abe..7519ebf028d 100644 --- a/doc/administration/high_availability/nfs_host_client_setup.md +++ b/doc/administration/high_availability/nfs_host_client_setup.md @@ -36,7 +36,7 @@ apt-get install nfs-kernel-server In this setup we will share the home directory on the host with the client. Edit the exports file as below to share the host's home directory with the client. If you have multiple clients running GitLab you must enter the client IP addresses in line in the `/etc/exports` file. -```text +```plaintext #/etc/exports for one client /home <client-ip-address>(rw,sync,no_root_squash,no_subtree_check) @@ -96,7 +96,7 @@ NFS version 4. NFSv3 also supports locking as long as Linux Kernel 2.6.5+ is use We recommend using version 4 and do not specifically test NFSv3. See [NFS documentation](nfs.md#nfs-client-mount-options) for guidance on mount options. -```text +```plaintext #/etc/fstab 10.0.0.1:/nfs/home /nfs/home nfs4 defaults,hard,vers=4.1,rsize=1048576,wsize=1048576,noatime,nofail,lookupcache=positive 0 2 ``` @@ -111,7 +111,7 @@ default file locations in `gitlab.rb` on the client allows you to have one main point and have all the required locations as subdirectories to use the NFS mount for `git-data`. -```text +```plaintext git_data_dirs({"default" => {"path" => "/nfs/home/var/opt/gitlab-data/git-data"}}) gitlab_rails['uploads_directory'] = '/nfs/home/var/opt/gitlab-data/uploads' gitlab_rails['shared_path'] = '/nfs/home/var/opt/gitlab-data/shared' diff --git a/doc/administration/high_availability/pgbouncer.md b/doc/administration/high_availability/pgbouncer.md index f0e76172a00..4c672f49e26 100644 --- a/doc/administration/high_availability/pgbouncer.md +++ b/doc/administration/high_availability/pgbouncer.md @@ -165,7 +165,7 @@ Refer to your preferred Load Balancer's documentation for further guidance. 1. Run `gitlab-ctl reconfigure` -1. On the node running Unicorn, make sure the following is set in `/etc/gitlab/gitlab.rb` +1. On the node running Puma, make sure the following is set in `/etc/gitlab/gitlab.rb` ```ruby gitlab_rails['db_host'] = 'PGBOUNCER_HOST' @@ -215,7 +215,7 @@ To start a session, run ```shell # gitlab-ctl pgb-console Password for user pgbouncer: -psql (9.6.8, server 1.7.2/bouncer) +psql (11.7, server 1.7.2/bouncer) Type "help" for help. pgbouncer=# diff --git a/doc/administration/high_availability/redis.md b/doc/administration/high_availability/redis.md index 1e19e7e6c01..d52c80aec0d 100644 --- a/doc/administration/high_availability/redis.md +++ b/doc/administration/high_availability/redis.md @@ -27,24 +27,24 @@ These will be necessary when configuring the GitLab application servers later. ## Redis in a Scaled and Highly Available Environment -This section is relevant for [Scalable and Highly Available Setups](../scaling/index.md). +This section is relevant for [scalable and highly available setups](../reference_architectures/index.md). ### Provide your own Redis instance **(CORE ONLY)** If you want to use your own deployed Redis instance(s), see [Provide your own Redis instance](#provide-your-own-redis-instance-core-only) -for more details. However, you can use the GitLab Omnibus package to easily +for more details. However, you can use the Omnibus GitLab package to easily deploy the bundled Redis. -### Standalone Redis using GitLab Omnibus **(CORE ONLY)** +### Standalone Redis using Omnibus GitLab **(CORE ONLY)** -The GitLab Omnibus package can be used to configure a standalone Redis server. +The Omnibus GitLab package can be used to configure a standalone Redis server. In this configuration Redis is not highly available, and represents a single point of failure. However, in a scaled environment the objective is to allow the environment to handle more users or to increase throughput. Redis itself is generally stable and can handle many requests so it is an acceptable -trade off to have only a single instance. See [High Availability](../availability/index.md) -for an overview of GitLab scaling and high availability options. +trade off to have only a single instance. See the [reference architectures](../reference_architectures/index.md) +page for an overview of GitLab scaling and high availability options. The steps below are the minimum necessary to configure a Redis server with Omnibus: @@ -63,7 +63,7 @@ Omnibus: ## Disable all other services sidekiq['enable'] = false gitlab_workhorse['enable'] = false - unicorn['enable'] = false + puma['enable'] = false postgresql['enable'] = false nginx['enable'] = false prometheus['enable'] = false @@ -89,16 +89,16 @@ Advanced configuration options are supported and can be added if needed. Continue configuration of other components by going back to the -[High Availability](../availability/index.md#gitlab-components-and-configuration-instructions) page. +[reference architectures](../reference_architectures/index.md#configure-gitlab-to-scale) page. -### High Availability with GitLab Omnibus **(PREMIUM ONLY)** +### High Availability with Omnibus GitLab **(PREMIUM ONLY)** > Experimental Redis Sentinel support was [introduced in GitLab 8.11](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/1877). Starting with 8.14, Redis Sentinel is no longer experimental. If you've used it with versions `< 8.14` before, please check the updated documentation here. -High Availability with [Redis](https://redis.io/) is possible using a **Master** x **Slave** +High Availability with [Redis](https://redis.io/) is possible using a **Master** x **Replica** topology with a [Redis Sentinel](https://redis.io/topics/sentinel) service to watch and automatically start the failover procedure. @@ -130,12 +130,12 @@ make sure you read this Overview section to better understand how the components are tied together. You need at least `3` independent machines: physical, or VMs running into -distinct physical machines. It is essential that all master and slaves Redis +distinct physical machines. It is essential that all master and replica Redis instances run in different machines. If you fail to provision the machines in that specific way, any issue with the shared environment can bring your entire setup down. -It is OK to run a Sentinel alongside of a master or slave Redis instance. +It is OK to run a Sentinel alongside of a master or replica Redis instance. There should be no more than one Sentinel on the same machine though. You also need to take into consideration the underlying network topology, @@ -156,16 +156,16 @@ components below. High Availability with Redis requires a few things: - Multiple Redis instances -- Run Redis in a **Master** x **Slave** topology +- Run Redis in a **Master** x **Replica** topology - Multiple Sentinel instances - Application support and visibility to all Sentinel and Redis instances Redis Sentinel can handle the most important tasks in an HA environment and that's to help keep servers online with minimal to no downtime. Redis Sentinel: -- Monitors **Master** and **Slaves** instances to see if they are available -- Promotes a **Slave** to **Master** when the **Master** fails -- Demotes a **Master** to **Slave** when the failed **Master** comes back online +- Monitors **Master** and **Replicas** instances to see if they are available +- Promotes a **Replica** to **Master** when the **Master** fails +- Demotes a **Master** to **Replica** when the failed **Master** comes back online (to prevent data-partitioning) - Can be queried by the application to always connect to the current **Master** server @@ -185,8 +185,8 @@ For a minimal setup, you will install the Omnibus GitLab package in `3` **independent** machines, both with **Redis** and **Sentinel**: - Redis Master + Sentinel -- Redis Slave + Sentinel -- Redis Slave + Sentinel +- Redis Replica + Sentinel +- Redis Replica + Sentinel If you are not sure or don't understand why and where the amount of nodes come from, read [Redis setup overview](#redis-setup-overview) and @@ -197,14 +197,14 @@ the Omnibus GitLab package in `5` **independent** machines, both with **Redis** and **Sentinel**: - Redis Master + Sentinel -- Redis Slave + Sentinel -- Redis Slave + Sentinel -- Redis Slave + Sentinel -- Redis Slave + Sentinel +- Redis Replica + Sentinel +- Redis Replica + Sentinel +- Redis Replica + Sentinel +- Redis Replica + Sentinel ### Redis setup overview -You must have at least `3` Redis servers: `1` Master, `2` Slaves, and they +You must have at least `3` Redis servers: `1` Master, `2` Replicas, and they need to each be on independent machines (see explanation above). You can have additional Redis nodes, that will help survive a situation @@ -221,7 +221,7 @@ nodes to be provisioned. See [Sentinel setup overview](#sentinel-setup-overview) documentation for more information. All Redis nodes should be configured the same way and with similar server specs, as -in a failover situation, any **Slave** can be promoted as the new **Master** by +in a failover situation, any **Replica** can be promoted as the new **Master** by the Sentinel servers. The replication requires authentication, so you need to define a password to @@ -241,9 +241,9 @@ need to be available and reachable, so that they can elect the Sentinel **leader who will take all the decisions to restore the service availability by: - Promoting a new **Master** -- Reconfiguring the other **Slaves** and make them point to the new **Master** +- Reconfiguring the other **Replicas** and make them point to the new **Master** - Announce the new **Master** to every other Sentinel peer -- Reconfigure the old **Master** and demote to **Slave** when it comes back online +- Reconfigure the old **Master** and demote to **Replica** when it comes back online You must have at least `3` Redis Sentinel servers, and they need to be each in an independent machine (that are believed to fail independently), @@ -280,18 +280,18 @@ the official documentation: already tried against the same master by a given Sentinel, is two times the failover timeout. -- The time needed for a slave replicating to a wrong master according +- The time needed for a replica replicating to a wrong master according to a Sentinel current configuration, to be forced to replicate with the right master, is exactly the failover timeout (counting since the moment a Sentinel detected the misconfiguration). - The time needed to cancel a failover that is already in progress but - did not produced any configuration change (SLAVEOF NO ONE yet not - acknowledged by the promoted slave). + did not produced any configuration change (REPLICAOF NO ONE yet not + acknowledged by the promoted replica). -- The maximum time a failover in progress waits for all the slaves to be - reconfigured as slaves of the new master. However even after this time - the slaves will be reconfigured by the Sentinels anyway, but not with +- The maximum time a failover in progress waits for all the replicas to be + reconfigured as replicas of the new master. However even after this time + the replicas will be reconfigured by the Sentinels anyway, but not with the exact parallel-syncs progression as specified. ### Available configuration setups @@ -306,12 +306,12 @@ Pick the one that suits your needs. documentation. - [Omnibus GitLab **Community Edition** (CE) package](https://about.gitlab.com/install/?version=ce): Redis is bundled, so you can use the package with only the Redis service enabled as described in steps - 1 and 2 of this document (works for both master and slave setups). To install + 1 and 2 of this document (works for both master and replica setups). To install and configure Sentinel, jump directly to the Sentinel section in the [Redis HA installation from source](redis_source.md#step-3-configuring-the-redis-sentinel-instances) documentation. - [Omnibus GitLab **Enterprise Edition** (EE) package](https://about.gitlab.com/install/?version=ee): Both Redis and Sentinel are bundled in the package, so you can use the EE package to set up the whole - Redis HA infrastructure (master, slave and Sentinel) which is described in + Redis HA infrastructure (master, replica and Sentinel) which is described in this document. - If you have installed GitLab using the Omnibus GitLab packages (CE or EE), but you want to use your own external Redis server, follow steps 1-3 in the @@ -328,9 +328,9 @@ This is the section where we install and set up the new Redis instances. > - We assume that you have installed GitLab and all HA components from scratch. If you > already have it installed and running, read how to > [switch from a single-machine installation to Redis HA](#switching-from-an-existing-single-machine-installation-to-redis-ha). -> - Redis nodes (both master and slaves) will need the same password defined in +> - Redis nodes (both master and replica) will need the same password defined in > `redis['password']`. At any time during a failover the Sentinels can -> reconfigure a node and change its status from master to slave and vice versa. +> reconfigure a node and change its status from master to replica and vice versa. ### Prerequisites @@ -392,9 +392,9 @@ The prerequisites for a HA Redis setup are the following: > `roles ['redis_sentinel_role', 'redis_master_role']`. Read more about high > availability roles at <https://docs.gitlab.com/omnibus/roles/>. -### Step 2. Configuring the slave Redis instances +### Step 2. Configuring the replica Redis instances -1. SSH into the **slave** Redis server. +1. SSH into the **replica** Redis server. 1. [Download/install](https://about.gitlab.com/install/) the Omnibus GitLab package you want using **steps 1 and 2** from the GitLab downloads page. - Make sure you select the correct Omnibus package, with the same version @@ -404,8 +404,8 @@ The prerequisites for a HA Redis setup are the following: 1. Edit `/etc/gitlab/gitlab.rb` and add the contents: ```ruby - # Specify server role as 'redis_slave_role' - roles ['redis_slave_role'] + # Specify server role as 'redis_replica_role' + roles ['redis_replica_role'] # IP address pointing to a local IP that the other machines can reach to. # You can also set bind to '0.0.0.0' which listen in all interfaces. @@ -435,10 +435,10 @@ The prerequisites for a HA Redis setup are the following: ``` 1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. -1. Go through the steps again for all the other slave nodes. +1. Go through the steps again for all the other replica nodes. > Note: You can specify multiple roles like sentinel and Redis as: -> `roles ['redis_sentinel_role', 'redis_slave_role']`. Read more about high +> `roles ['redis_sentinel_role', 'redis_replica_role']`. Read more about high > availability roles at <https://docs.gitlab.com/omnibus/roles/>. --- @@ -542,18 +542,18 @@ multiple machines with the Sentinel daemon. ## already tried against the same master by a given Sentinel, is two ## times the failover timeout. ## - ## - The time needed for a slave replicating to a wrong master according + ## - The time needed for a replica replicating to a wrong master according ## to a Sentinel current configuration, to be forced to replicate ## with the right master, is exactly the failover timeout (counting since ## the moment a Sentinel detected the misconfiguration). ## ## - The time needed to cancel a failover that is already in progress but - ## did not produced any configuration change (SLAVEOF NO ONE yet not - ## acknowledged by the promoted slave). + ## did not produced any configuration change (REPLICAOF NO ONE yet not + ## acknowledged by the promoted replica). ## - ## - The maximum time a failover in progress waits for all the slaves to be - ## reconfigured as slaves of the new master. However even after this time - ## the slaves will be reconfigured by the Sentinels anyway, but not with + ## - The maximum time a failover in progress waits for all the replica to be + ## reconfigured as replicas of the new master. However even after this time + ## the replicas will be reconfigured by the Sentinels anyway, but not with ## the exact parallel-syncs progression as specified. # sentinel['failover_timeout'] = 60000 ``` @@ -612,7 +612,7 @@ replicate from this machine first, before de-activating the Redis instance inside it. Your single-machine install will be the initial **Master**, and the `3` others -should be configured as **Slave** pointing to this machine. +should be configured as **Replica** pointing to this machine. After replication catches up, you will need to stop services in the single-machine install, to rotate the **Master** to one of the new nodes. @@ -627,7 +627,7 @@ redis['enable'] = false If you fail to replicate first, you may loose data (unprocessed background jobs). -## Example of a minimal configuration with 1 master, 2 slaves and 3 Sentinels +## Example of a minimal configuration with 1 master, 2 replicas and 3 Sentinels >**Note:** Redis Sentinel is bundled with Omnibus GitLab Enterprise Edition only. For @@ -649,8 +649,8 @@ discussed in [Redis setup overview](#redis-setup-overview) and Here is a list and description of each **machine** and the assigned **IP**: - `10.0.0.1`: Redis Master + Sentinel 1 -- `10.0.0.2`: Redis Slave 1 + Sentinel 2 -- `10.0.0.3`: Redis Slave 2 + Sentinel 3 +- `10.0.0.2`: Redis Replica 1 + Sentinel 2 +- `10.0.0.3`: Redis Replica 2 + Sentinel 3 - `10.0.0.4`: GitLab application Please note that after the initial configuration, if a failover is initiated @@ -684,12 +684,12 @@ sentinel['quorum'] = 2 [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. -### Example configuration for Redis slave 1 and Sentinel 2 +### Example configuration for Redis replica 1 and Sentinel 2 In `/etc/gitlab/gitlab.rb`: ```ruby -roles ['redis_sentinel_role', 'redis_slave_role'] +roles ['redis_sentinel_role', 'redis_replica_role'] redis['bind'] = '10.0.0.2' redis['port'] = 6379 redis['password'] = 'redis-password-goes-here' @@ -706,12 +706,12 @@ sentinel['quorum'] = 2 [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. -### Example configuration for Redis slave 2 and Sentinel 3 +### Example configuration for Redis replica 2 and Sentinel 3 In `/etc/gitlab/gitlab.rb`: ```ruby -roles ['redis_sentinel_role', 'redis_slave_role'] +roles ['redis_sentinel_role', 'redis_replica_role'] redis['bind'] = '10.0.0.3' redis['port'] = 6379 redis['password'] = 'redis-password-goes-here' @@ -847,11 +847,11 @@ mailroom['enable'] = false ------- -## Redis master/slave Role +## Redis master/replica Role redis_master_role['enable'] = true # enable only one of them -redis_slave_role['enable'] = true # enable only one of them +redis_replica_role['enable'] = true # enable only one of them -# When Redis Master or Slave role are enabled, the following services are +# When Redis Master or Replica role are enabled, the following services are # enabled/disabled. Note that if Redis and Sentinel roles are combined, both # services will be enabled. @@ -863,7 +863,7 @@ postgresql['enable'] = false gitlab_rails['enable'] = false mailroom['enable'] = false -# For Redis Slave role, also change this setting from default 'true' to 'false': +# For Redis Replica role, also change this setting from default 'true' to 'false': redis['master'] = false ``` @@ -894,13 +894,13 @@ You can check if everything is correct by connecting to each server using ``` When connected to a `master` Redis, you will see the number of connected -`slaves`, and a list of each with connection details: +`replicas`, and a list of each with connection details: ```plaintext # Replication role:master -connected_slaves:1 -slave0:ip=10.133.5.21,port=6379,state=online,offset=208037514,lag=1 +connected_replicas:1 +replica0:ip=10.133.5.21,port=6379,state=online,offset=208037514,lag=1 master_repl_offset:208037658 repl_backlog_active:1 repl_backlog_size:1048576 @@ -908,21 +908,21 @@ repl_backlog_first_byte_offset:206989083 repl_backlog_histlen:1048576 ``` -When it's a `slave`, you will see details of the master connection and if +When it's a `replica`, you will see details of the master connection and if its `up` or `down`: ```plaintext # Replication -role:slave +role:replica master_host:10.133.1.58 master_port:6379 master_link_status:up master_last_io_seconds_ago:1 master_sync_in_progress:0 -slave_repl_offset:208096498 -slave_priority:100 -slave_read_only:1 -connected_slaves:0 +replica_repl_offset:208096498 +replica_priority:100 +replica_read_only:1 +connected_replicas:0 master_repl_offset:0 repl_backlog_active:0 repl_backlog_size:1048576 diff --git a/doc/administration/high_availability/redis_source.md b/doc/administration/high_availability/redis_source.md index 95b1397bab4..97be480bc3b 100644 --- a/doc/administration/high_availability/redis_source.md +++ b/doc/administration/high_availability/redis_source.md @@ -40,7 +40,7 @@ This is the section where we install and set up the new Redis instances. - Since Redis 3.2, you must define a password to receive external connections (`requirepass`). - If you are using Redis with Sentinel, you will also need to define the same - password for the slave password definition (`masterauth`) in the same instance. + password for the replica password definition (`masterauth`) in the same instance. In addition, read the prerequisites as described in the [Omnibus Redis HA document](redis.md#prerequisites) since they provide some @@ -72,9 +72,9 @@ Assuming that the Redis master instance IP is `10.0.0.1`: 1. Restart the Redis service for the changes to take effect. -### Step 2. Configuring the slave Redis instances +### Step 2. Configuring the replica Redis instances -Assuming that the Redis slave instance IP is `10.0.0.2`: +Assuming that the Redis replica instance IP is `10.0.0.2`: 1. [Install Redis](../../install/installation.md#7-redis). 1. Edit `/etc/redis/redis.conf`: @@ -95,12 +95,12 @@ Assuming that the Redis slave instance IP is `10.0.0.2`: requirepass redis-password-goes-here masterauth redis-password-goes-here - ## Define `slaveof` pointing to the Redis master instance with IP and port. - slaveof 10.0.0.1 6379 + ## Define `replicaof` pointing to the Redis master instance with IP and port. + replicaof 10.0.0.1 6379 ``` 1. Restart the Redis service for the changes to take effect. -1. Go through the steps again for all the other slave nodes. +1. Go through the steps again for all the other replica nodes. ### Step 3. Configuring the Redis Sentinel instances @@ -131,7 +131,7 @@ master with IP `10.0.0.1` (some settings might overlap with the master): masterauth redis-password-goes-here ## Define with `sentinel auth-pass` the same shared password you have - ## defined for both Redis master and slaves instances. + ## defined for both Redis master and replicas instances. sentinel auth-pass gitlab-redis redis-password-goes-here ## Define with `sentinel monitor` the IP and port of the Redis @@ -149,18 +149,18 @@ master with IP `10.0.0.1` (some settings might overlap with the master): ## already tried against the same master by a given Sentinel, is two ## times the failover timeout. ## - ## * The time needed for a slave replicating to a wrong master according + ## * The time needed for a replica replicating to a wrong master according ## to a Sentinel current configuration, to be forced to replicate ## with the right master, is exactly the failover timeout (counting since ## the moment a Sentinel detected the misconfiguration). ## ## * The time needed to cancel a failover that is already in progress but - ## did not produced any configuration change (SLAVEOF NO ONE yet not - ## acknowledged by the promoted slave). + ## did not produced any configuration change (REPLICAOF NO ONE yet not + ## acknowledged by the promoted replica). ## - ## * The maximum time a failover in progress waits for all the slaves to be - ## reconfigured as slaves of the new master. However even after this time - ## the slaves will be reconfigured by the Sentinels anyway, but not with + ## * The maximum time a failover in progress waits for all the replicas to be + ## reconfigured as replicas of the new master. However even after this time + ## the replicas will be reconfigured by the Sentinels anyway, but not with ## the exact parallel-syncs progression as specified. sentinel failover_timeout 30000 ``` @@ -203,7 +203,7 @@ setup: 1. [Restart GitLab](../restart_gitlab.md#installations-from-source) for the changes to take effect. -## Example of minimal configuration with 1 master, 2 slaves and 3 Sentinels +## Example of minimal configuration with 1 master, 2 replicas and 3 Sentinels In this example we consider that all servers have an internal network interface with IPs in the `10.0.0.x` range, and that they can connect @@ -215,13 +215,13 @@ outside ([Internet](https://gitlab.com/gitlab-org/gitlab-foss/uploads/c4cc8cd353 For this example, **Sentinel 1** will be configured in the same machine as the **Redis Master**, **Sentinel 2** and **Sentinel 3** in the same machines as the -**Slave 1** and **Slave 2** respectively. +**Replica 1** and **Replica 2** respectively. Here is a list and description of each **machine** and the assigned **IP**: - `10.0.0.1`: Redis Master + Sentinel 1 -- `10.0.0.2`: Redis Slave 1 + Sentinel 2 -- `10.0.0.3`: Redis Slave 2 + Sentinel 3 +- `10.0.0.2`: Redis Replica 1 + Sentinel 2 +- `10.0.0.3`: Redis Replica 2 + Sentinel 3 - `10.0.0.4`: GitLab application Please note that after the initial configuration, if a failover is initiated @@ -257,7 +257,7 @@ or a failover promotes a different **Master** node. 1. Restart the Redis service for the changes to take effect. -### Example configuration for Redis slave 1 and Sentinel 2 +### Example configuration for Redis replica 1 and Sentinel 2 1. In `/etc/redis/redis.conf`: @@ -266,7 +266,7 @@ or a failover promotes a different **Master** node. port 6379 requirepass redis-password-goes-here masterauth redis-password-goes-here - slaveof 10.0.0.1 6379 + replicaof 10.0.0.1 6379 ``` 1. In `/etc/redis/sentinel.conf`: @@ -282,7 +282,7 @@ or a failover promotes a different **Master** node. 1. Restart the Redis service for the changes to take effect. -### Example configuration for Redis slave 2 and Sentinel 3 +### Example configuration for Redis replica 2 and Sentinel 3 1. In `/etc/redis/redis.conf`: @@ -291,7 +291,7 @@ or a failover promotes a different **Master** node. port 6379 requirepass redis-password-goes-here masterauth redis-password-goes-here - slaveof 10.0.0.1 6379 + replicaof 10.0.0.1 6379 ``` 1. In `/etc/redis/sentinel.conf`: diff --git a/doc/administration/high_availability/sidekiq.md b/doc/administration/high_availability/sidekiq.md index ed273c3b113..493929dcf3b 100644 --- a/doc/administration/high_availability/sidekiq.md +++ b/doc/administration/high_availability/sidekiq.md @@ -88,12 +88,15 @@ you want using steps 1 and 2 from the GitLab downloads page. postgresql['enable'] = false redis['enable'] = false redis_exporter['enable'] = false - unicorn['enable'] = false + puma['enable'] = false gitlab_exporter['enable'] = false ``` 1. Run `gitlab-ctl reconfigure`. +NOTE: **Note:** You will need to restart the Sidekiq nodes after an update has occurred and database +migrations performed. + ## Example configuration Here's what the ending `/etc/gitlab/gitlab.rb` would look like: @@ -116,7 +119,7 @@ postgres_exporter['enable'] = false postgresql['enable'] = false redis['enable'] = false redis_exporter['enable'] = false -unicorn['enable'] = false +puma['enable'] = false gitlab_exporter['enable'] = false ######################################## diff --git a/doc/administration/incoming_email.md b/doc/administration/incoming_email.md index dcc590bea9c..fcd69464b13 100644 --- a/doc/administration/incoming_email.md +++ b/doc/administration/incoming_email.md @@ -182,6 +182,9 @@ gitlab_rails['incoming_email_start_tls'] = false gitlab_rails['incoming_email_mailbox_name'] = "inbox" # The IDLE command timeout. gitlab_rails['incoming_email_idle_timeout'] = 60 + +# Whether to expunge (permanently remove) messages from the mailbox when they are deleted after delivery +gitlab_rails['incoming_email_expunge_deleted'] = true ``` Example for source installs: @@ -214,12 +217,18 @@ incoming_email: mailbox: "inbox" # The IDLE command timeout. idle_timeout: 60 + + # Whether to expunge (permanently remove) messages from the mailbox when they are deleted after delivery + expunge_deleted: true ``` #### Gmail Example configuration for Gmail/G Suite. Assumes mailbox `gitlab-incoming@gmail.com`. +NOTE: **Note:** +`incoming_email_email` cannot be a Gmail alias account. + Example for Omnibus installs: ```ruby @@ -249,6 +258,9 @@ gitlab_rails['incoming_email_start_tls'] = false gitlab_rails['incoming_email_mailbox_name'] = "inbox" # The IDLE command timeout. gitlab_rails['incoming_email_idle_timeout'] = 60 + +# Whether to expunge (permanently remove) messages from the mailbox when they are deleted after delivery +gitlab_rails['incoming_email_expunge_deleted'] = true ``` Example for source installs: @@ -281,6 +293,9 @@ incoming_email: mailbox: "inbox" # The IDLE command timeout. idle_timeout: 60 + + # Whether to expunge (permanently remove) messages from the mailbox when they are deleted after delivery + expunge_deleted: true ``` #### Microsoft Exchange Server diff --git a/doc/administration/index.md b/doc/administration/index.md index 1f4e23fce8a..fb827ae8573 100644 --- a/doc/administration/index.md +++ b/doc/administration/index.md @@ -34,8 +34,8 @@ Learn how to install, configure, update, and maintain your GitLab instance. - [Install](../install/README.md): Requirements, directory structures, and installation methods. - [Database load balancing](database_load_balancing.md): Distribute database queries among multiple database servers. **(STARTER ONLY)** - [Omnibus support for log forwarding](https://docs.gitlab.com/omnibus/settings/logs.html#udp-log-shipping-gitlab-enterprise-edition-only) **(STARTER ONLY)** -- [High Availability](availability/index.md): Configure multiple servers for scaling or high availability. - - [Installing GitLab HA on Amazon Web Services (AWS)](../install/aws/index.md): Set up GitLab High Availability on Amazon AWS. +- [Reference architectures](reference_architectures/index.md): Add additional resources to support more users. + - [Installing GitLab on Amazon Web Services (AWS)](../install/aws/index.md): Set up GitLab on Amazon AWS. - [Geo](geo/replication/index.md): Replicate your GitLab instance to other geographic locations as a read-only fully operational version. **(PREMIUM ONLY)** - [Disaster Recovery](geo/disaster_recovery/index.md): Quickly fail-over to a different site with minimal effort in a disaster situation. **(PREMIUM ONLY)** - [Pivotal Tile](../install/pivotal/index.md): Deploy GitLab as a preconfigured appliance using Ops Manager (BOSH) for Pivotal Cloud Foundry. **(PREMIUM ONLY)** @@ -64,7 +64,8 @@ Learn how to install, configure, update, and maintain your GitLab instance. - [External Classification Policy Authorization](../user/admin_area/settings/external_authorization.md) **(PREMIUM ONLY)** - [Upload a license](../user/admin_area/license.md): Upload a license to unlock features that are in paid tiers of GitLab. **(STARTER ONLY)** - [Admin Area](../user/admin_area/index.md): for self-managed instance-wide configuration and maintenance. -- [S/MIME Signing](smime_signing_email.md): how to sign all outgoing notification emails with S/MIME +- [S/MIME Signing](smime_signing_email.md): how to sign all outgoing notification emails with S/MIME. +- [Enabling and disabling features flags](feature_flags.md): how to enable and disable GitLab features deployed behind feature flags. #### Customizing GitLab's appearance @@ -76,9 +77,9 @@ Learn how to install, configure, update, and maintain your GitLab instance. ### Maintaining GitLab -- [Raketasks](../raketasks/README.md): Perform various tasks for maintenance, backups, automatic webhooks setup, and more. +- [Rake tasks](../raketasks/README.md): Perform various tasks for maintenance, backups, automatic webhooks setup, and more. - [Backup and restore](../raketasks/backup_restore.md): Backup and restore your GitLab instance. -- [Operations](operations/index.md): Keeping GitLab up and running (clean up Redis sessions, moving repositories, Sidekiq MemoryKiller, Unicorn). +- [Operations](operations/index.md): Keeping GitLab up and running (clean up Redis sessions, moving repositories, Sidekiq MemoryKiller, Puma). - [Restart GitLab](restart_gitlab.md): Learn how to restart GitLab and its components. - [Invalidate Markdown cache](invalidate_markdown_cache.md): Invalidate any cached Markdown. @@ -98,8 +99,8 @@ Learn how to install, configure, update, and maintain your GitLab instance. - [Mattermost](https://docs.gitlab.com/omnibus/gitlab-mattermost/): Integrate with [Mattermost](https://mattermost.com), an open source, private cloud workplace for web messaging. - [PlantUML](integration/plantuml.md): Create simple diagrams in AsciiDoc and Markdown documents - created in snippets, wikis, and repos. -- [Web terminals](integration/terminal.md): Provide terminal access to your applications deployed to Kubernetes from within GitLab's CI/CD [environments](../ci/environments.md#web-terminals). + created in snippets, wikis, and repositories. +- [Web terminals](integration/terminal.md): Provide terminal access to your applications deployed to Kubernetes from within GitLab's CI/CD [environments](../ci/environments/index.md#web-terminals). ## User settings and permissions @@ -183,8 +184,6 @@ Learn how to install, configure, update, and maintain your GitLab instance. - [GitLab Performance Monitoring](monitoring/performance/index.md): - [Enable Performance Monitoring](monitoring/performance/gitlab_configuration.md): Enable GitLab Performance Monitoring. - - [GitLab performance monitoring with InfluxDB](monitoring/performance/influxdb_configuration.md): Configure GitLab and InfluxDB for measuring performance metrics. - - [InfluxDB Schema](monitoring/performance/influxdb_schema.md): Measurements stored in InfluxDB. - [GitLab performance monitoring with Prometheus](monitoring/prometheus/index.md): Configure GitLab and Prometheus for measuring performance metrics. - [GitLab performance monitoring with Grafana](monitoring/performance/grafana_configuration.md): Configure GitLab to visualize time series metrics through graphs and dashboards. - [Request Profiling](monitoring/performance/request_profiling.md): Get a detailed profile on slow requests. diff --git a/doc/administration/instance_limits.md b/doc/administration/instance_limits.md index 57acdec4ea2..d0e8f079cb2 100644 --- a/doc/administration/instance_limits.md +++ b/doc/administration/instance_limits.md @@ -99,6 +99,29 @@ header. Such emails don't create comments on issues or merge requests. Sentry payloads sent to GitLab have a 1 MB maximum limit, both for security reasons and to limit memory consumption. +## Max offset allowed via REST API for offset-based pagination + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/34565) in GitLab 13.0. + +When using offset-based pagination in the REST API, there is a limit to the maximum +requested offset into the set of results. This limit is only applied to endpoints that +support keyset-based pagination. More information about pagination options can be +found in the [API docs section on pagination](../api/README.md#pagination). + +To set this limit on a self-managed installation, run the following in the +[GitLab Rails console](troubleshooting/debug.md#starting-a-rails-console-session): + +```ruby +# If limits don't exist for the default plan, you can create one with: +# Plan.default.create_limits! + +Plan.default.limits.update!(offset_pagination_limit: 10000) +``` + +- **Default offset pagination limit:** 50000 + +NOTE: **Note:** Set the limit to `0` to disable it. + ## CI/CD limits ### Number of jobs in active pipelines @@ -180,7 +203,7 @@ Plan.default.limits.update!(ci_pipeline_schedules: 100) ### Incident Management inbound alert limits -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/14932) in GitLab 12.5. +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/17859) in GitLab 12.5. Limiting inbound alerts for an incident reduces the number of alerts (issues) that can be created within a period of time, which can help prevent overloading @@ -192,10 +215,16 @@ alerts in the following ways: ### Prometheus Alert JSON payloads -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/14929) in GitLab 12.6. +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/19940) in GitLab 12.6. Prometheus alert payloads sent to the `notify.json` endpoint are limited to 1 MB in size. +### Generic Alert JSON payloads + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/16441) in GitLab 12.4. + +Alert payloads sent to the `notify.json` endpoint are limited to 1 MB in size. + ## Environment data on Deploy Boards [Deploy Boards](../user/project/deploy_boards.md) load information from Kubernetes about @@ -233,6 +262,10 @@ NOTE: **Note:** Set the limit to `0` to disable it. - [Length restrictions for file and directory names](../user/project/wiki/index.md#length-restrictions-for-file-and-directory-names). +## Snippets limits + +See the [documentation on Snippets settings](snippets/index.md). + ## Push Event Limits ### Webhooks and Project Services diff --git a/doc/administration/instance_review.md b/doc/administration/instance_review.md index bb64c7967b7..8ea4bff252e 100644 --- a/doc/administration/instance_review.md +++ b/doc/administration/instance_review.md @@ -1,6 +1,6 @@ # Instance Review **(CORE ONLY)** -> [Introduced][6995] in [GitLab Core][ee] 11.3. +> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/6995) in [GitLab Core](https://about.gitlab.com/pricing/) 11.3. If you are running a medium size instance of GitLab Core edition you are qualified for a free Instance Review. You can find the button in the User menu. @@ -11,6 +11,3 @@ When you click the button you will be redirected to a form with prefilled data o Once you submit the data to GitLab Inc. you can see the initial report. Additionally you will be contacted by our team for further review which should help you to improve your usage of GitLab. - -[6995]: https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/6995 -[ee]: https://about.gitlab.com/pricing/ diff --git a/doc/administration/integration/plantuml.md b/doc/administration/integration/plantuml.md index 4af787fd19f..3372d05bd6b 100644 --- a/doc/administration/integration/plantuml.md +++ b/doc/administration/integration/plantuml.md @@ -4,7 +4,7 @@ When [PlantUML](https://plantuml.com) integration is enabled and configured in GitLab we are able to create simple diagrams in AsciiDoc and Markdown documents -created in snippets, wikis, and repos. +created in snippets, wikis, and repositories. ## PlantUML Server @@ -70,7 +70,7 @@ sudo service tomcat8 restart Once the Tomcat service restarts the PlantUML service will be ready and listening for requests on port 8080: -```text +```plaintext http://localhost:8080/plantuml ``` @@ -115,10 +115,23 @@ that, login with an Admin account and do following: - Check **Enable PlantUML** checkbox. - Set the PlantUML instance as `https://gitlab.example.com/-/plantuml/`. +NOTE: **Note:** If you are using a PlantUML server running v1.2020.9 and +above (for example, [plantuml.com](https://plantuml.com)), set the `PLANTUML_ENCODING` +environment variable to enable the `deflate` compression. On Omnibus, +this can be done set in `/etc/gitlab.rb`: + +```ruby +gitlab_rails['env'] = { 'PLANTUML_ENCODING' => 'deflate' } +``` + +From GitLab 13.1 and later, PlantUML integration now +[requires a header prefix in the URL](https://github.com/plantuml/plantuml/issues/117#issuecomment-6235450160) +to distinguish different encoding types. + ## Creating Diagrams With PlantUML integration enabled and configured, we can start adding diagrams to -our AsciiDoc snippets, wikis and repos using delimited blocks: +our AsciiDoc snippets, wikis, and repositories using delimited blocks: - **Markdown** @@ -131,7 +144,7 @@ our AsciiDoc snippets, wikis and repos using delimited blocks: - **AsciiDoc** - ```text + ```plaintext [plantuml, format="png", id="myDiagram", width="200px"] ---- Bob->Alice : hello @@ -141,7 +154,7 @@ our AsciiDoc snippets, wikis and repos using delimited blocks: - **reStructuredText** - ```text + ```plaintext .. plantuml:: :caption: Caption with **bold** and *italic* @@ -169,10 +182,10 @@ diagram delimiters `@startuml`/`@enduml` as these are replaced by the AsciiDoc ` Some parameters can be added to the AsciiDoc block definition: -- *format*: Can be either `png` or `svg`. Note that `svg` is not supported by +- `format`: Can be either `png` or `svg`. Note that `svg` is not supported by all browsers so use with care. The default is `png`. -- *id*: A CSS id added to the diagram HTML tag. -- *width*: Width attribute added to the image tag. -- *height*: Height attribute added to the image tag. +- `id`: A CSS ID added to the diagram HTML tag. +- `width`: Width attribute added to the image tag. +- `height`: Height attribute added to the image tag. Markdown does not support any parameters and will always use PNG format. diff --git a/doc/administration/integration/terminal.md b/doc/administration/integration/terminal.md index 251e4ded8b4..bc5816ce120 100644 --- a/doc/administration/integration/terminal.md +++ b/doc/administration/integration/terminal.md @@ -8,7 +8,7 @@ Only project maintainers and owners can access web terminals. With the introduction of the [Kubernetes integration](../../user/project/clusters/index.md), GitLab gained the ability to store and use credentials for a Kubernetes cluster. One of the things it uses these credentials for is providing access to -[web terminals](../../ci/environments.md#web-terminals) for environments. +[web terminals](../../ci/environments/index.md#web-terminals) for environments. ## How it works diff --git a/doc/administration/issue_closing_pattern.md b/doc/administration/issue_closing_pattern.md index 7b815143597..579b957eb47 100644 --- a/doc/administration/issue_closing_pattern.md +++ b/doc/administration/issue_closing_pattern.md @@ -28,7 +28,7 @@ Because Rubular doesn't understand `%{issue_ref}`, you can replace this by expression of your liking: ```ruby - gitlab_rails['gitlab_issue_closing_pattern'] = "\b((?:[Cc]los(?:e[sd]|ing)|\b[Ff]ix(?:e[sd]|ing)?) +(?:(?:issues? +)?%{issue_ref}(?:(?:, *| +and +)?))+)" + gitlab_rails['gitlab_issue_closing_pattern'] = "\b((?:[Cc]los(?:e[sd]?|ing)|\b[Ff]ix(?:e[sd]|ing)?|\b[Rr]esolv(?:e[sd]?|ing)|\b[Ii]mplement(?:s|ed|ing)?)(:?) +(?:(?:issues? +)?%{issue_ref}(?:(?: *,? +and +| *,? *)?)|([A-Z][A-Z0-9_]+-\d+))+)" ``` 1. [Reconfigure](restart_gitlab.md#omnibus-gitlab-reconfigure) GitLab for the changes to take effect. @@ -39,7 +39,7 @@ Because Rubular doesn't understand `%{issue_ref}`, you can replace this by 1. Change the value of `issue_closing_pattern`: ```yaml - issue_closing_pattern: "\b((?:[Cc]los(?:e[sd]|ing)|\b[Ff]ix(?:e[sd]|ing)?) +(?:(?:issues? +)?%{issue_ref}(?:(?:, *| +and +)?))+)" + issue_closing_pattern: "\b((?:[Cc]los(?:e[sd]?|ing)|\b[Ff]ix(?:e[sd]|ing)?|\b[Rr]esolv(?:e[sd]?|ing)|\b[Ii]mplement(?:s|ed|ing)?)(:?) +(?:(?:issues? +)?%{issue_ref}(?:(?: *,? +and +| *,? *)?)|([A-Z][A-Z0-9_]+-\d+))+)" ``` 1. [Restart](restart_gitlab.md#installations-from-source) GitLab for the changes to take effect. diff --git a/doc/administration/job_artifacts.md b/doc/administration/job_artifacts.md index 3777f551996..fbfad46ef65 100644 --- a/doc/administration/job_artifacts.md +++ b/doc/administration/job_artifacts.md @@ -378,7 +378,7 @@ and [projects APIs](../api/projects.md). When GitLab receives an artifacts archive, an archive metadata file is also generated by [GitLab Workhorse](https://gitlab.com/gitlab-org/gitlab-workhorse). This metadata file describes all the entries that are located in the artifacts archive itself. -The metadata file is in a binary format, with additional GZIP compression. +The metadata file is in a binary format, with additional Gzip compression. GitLab does not extract the artifacts archive in order to save space, memory and disk I/O. It instead inspects the metadata file which contains all the @@ -450,13 +450,13 @@ If you need to manually remove job artifacts associated with multiple jobs while ```ruby project = Project.find_by_full_path('path/to/project') - builds_with_artifacts = project.builds.with_artifacts_archive + builds_with_artifacts = project.builds.with_downloadable_artifacts ``` To select all jobs with artifacts across the entire GitLab instance: ```ruby - builds_with_artifacts = Ci::Build.with_artifacts_archive + builds_with_artifacts = Ci::Build.with_downloadable_artifacts ``` 1. Delete job artifacts older than a specific date: diff --git a/doc/administration/job_logs.md b/doc/administration/job_logs.md index 6020d1d2850..5c03ff1f4b2 100644 --- a/doc/administration/job_logs.md +++ b/doc/administration/job_logs.md @@ -12,8 +12,8 @@ In the following table you can see the phases a log goes through: | Phase | State | Condition | Data flow | Stored path | | -------------- | ------------ | ----------------------- | -----------------------------------------| ----------- | -| 1: patching | log | When a job is running | GitLab Runner => Unicorn => file storage | `#{ROOT_PATH}/gitlab-ci/builds/#{YYYY_mm}/#{project_id}/#{job_id}.log` | -| 2: overwriting | log | When a job is finished | GitLab Runner => Unicorn => file storage | `#{ROOT_PATH}/gitlab-ci/builds/#{YYYY_mm}/#{project_id}/#{job_id}.log` | +| 1: patching | log | When a job is running | GitLab Runner => Puma => file storage | `#{ROOT_PATH}/gitlab-ci/builds/#{YYYY_mm}/#{project_id}/#{job_id}.log` | +| 2: overwriting | log | When a job is finished | GitLab Runner => Puma => file storage | `#{ROOT_PATH}/gitlab-ci/builds/#{YYYY_mm}/#{project_id}/#{job_id}.log` | | 3: archiving | archived log | After a job is finished | Sidekiq moves log to artifacts folder | `#{ROOT_PATH}/gitlab-rails/shared/artifacts/#{disk_hash}/#{YYYY_mm_dd}/#{job_id}/#{job_artifact_id}/job.log` | | 4: uploading | archived log | After a log is archived | Sidekiq moves archived log to [object storage](#uploading-logs-to-object-storage) (if configured) | `#{bucket_name}/#{disk_hash}/#{YYYY_mm_dd}/#{job_id}/#{job_artifact_id}/job.log` | diff --git a/doc/administration/lfs/index.md b/doc/administration/lfs/index.md index 71c1ae22305..e2b982448ef 100644 --- a/doc/administration/lfs/index.md +++ b/doc/administration/lfs/index.md @@ -2,7 +2,7 @@ disqus_identifier: 'https://docs.gitlab.com/ee/workflow/lfs/lfs_administration.html' --- -# GitLab Git LFS Administration +# GitLab Git Large File Storage (LFS) Administration Documentation on how to use Git LFS are under [Managing large binary files with Git LFS doc](../../topics/git/lfs/index.md). @@ -50,7 +50,7 @@ In `config/gitlab.yml`: ## Storing LFS objects in remote object storage -> [Introduced][ee-2760] in [GitLab Premium][eep] 10.0. Brought to GitLab Core in 10.7. +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/2760) in [GitLab Premium](https://about.gitlab.com/pricing/) 10.0. Brought to GitLab Core in 10.7. It is possible to store LFS objects in remote object storage which allows you to offload local hard disk R/W operations, and free up disk space significantly. @@ -129,7 +129,7 @@ Here is a configuration example with Rackspace Cloud Files. NOTE: **Note:** Regardless of whether the container has public access enabled or disabled, Fog will use the TempURL method to grant access to LFS objects. If you see errors in logs referencing -instantiating storage with a temp-url-key, ensure that you have set the key properly +instantiating storage with a `temp-url-key`, ensure that you have set the key properly on the Rackspace API and in `gitlab.rb`. You can verify the value of the key Rackspace has set by sending a GET request with token header to the service access endpoint URL and comparing the output of the returned headers. @@ -141,18 +141,23 @@ There are two ways to manually do the same thing as automatic uploading (describ **Option 1: Rake task** ```shell -rake gitlab:lfs:migrate +gitlab-rake gitlab:lfs:migrate ``` -**Option 2: rails console** +**Option 2: Rails console** + +Log into the Rails console: ```shell -$ sudo gitlab-rails console # Login to rails console +sudo gitlab-rails console +``` -> # Upload LFS files manually -> LfsObject.where(file_store: [nil, 1]).find_each do |lfs_object| -> lfs_object.file.migrate!(ObjectStorage::Store::REMOTE) if lfs_object.file.file.exists? -> end +Upload LFS files manually + +```ruby +LfsObject.where(file_store: [nil, 1]).find_each do |lfs_object| + lfs_object.file.migrate!(ObjectStorage::Store::REMOTE) if lfs_object.file.file.exists? +end ``` ### S3 for Omnibus installations @@ -177,7 +182,7 @@ On Omnibus installations, the settings are prefixed by `lfs_object_store_`: } ``` -1. Save the file and [reconfigure GitLab]s for the changes to take effect. +1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. 1. Migrate any existing local LFS objects to the object storage: ```shell @@ -213,7 +218,7 @@ For source installations the settings are nested under `lfs:` and then path_style: true ``` -1. Save the file and [restart GitLab][] for the changes to take effect. +1. Save the file and [restart GitLab](../restart_gitlab.md#installations-from-source) for the changes to take effect. 1. Migrate any existing local LFS objects to the object storage: ```shell @@ -244,19 +249,29 @@ If LFS integration is configured with Google Cloud Storage and background upload Sidekiq workers may encounter this error. This is because the uploading timed out with very large files. LFS files up to 6Gb can be uploaded without any extra steps, otherwise you need to use the following workaround. +Log into Rails console: + ```shell -$ sudo gitlab-rails console # Login to rails console - -> # Set up timeouts. 20 minutes is enough to upload 30GB LFS files. -> # These settings are only in effect for the same session, i.e. they are not effective for sidekiq workers. -> ::Google::Apis::ClientOptions.default.open_timeout_sec = 1200 -> ::Google::Apis::ClientOptions.default.read_timeout_sec = 1200 -> ::Google::Apis::ClientOptions.default.send_timeout_sec = 1200 - -> # Upload LFS files manually. This process does not use sidekiq at all. -> LfsObject.where(file_store: [nil, 1]).find_each do |lfs_object| -> lfs_object.file.migrate!(ObjectStorage::Store::REMOTE) if lfs_object.file.file.exists? -> end +sudo gitlab-rails console +``` + +Set up timeouts: + +- These settings are only in effect for the same session. For example, they are not effective for Sidekiq workers. +- 20 minutes (1200 sec) is enough to upload 30GB LFS files: + +```ruby +::Google::Apis::ClientOptions.default.open_timeout_sec = 1200 +::Google::Apis::ClientOptions.default.read_timeout_sec = 1200 +::Google::Apis::ClientOptions.default.send_timeout_sec = 1200 +``` + +Upload LFS files manually (this process does not use Sidekiq at all): + +```ruby +LfsObject.where(file_store: [nil, 1]).find_each do |lfs_object| + lfs_object.file.migrate!(ObjectStorage::Store::REMOTE) if lfs_object.file.file.exists? +end ``` See more information in [!19581](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/19581) @@ -268,8 +283,3 @@ See more information in [!19581](https://gitlab.com/gitlab-org/gitlab-foss/-/mer - Only compatible with the Git LFS client versions 1.1.0 and up, or 1.0.2. - The storage statistics currently count each LFS object multiple times for every project linking to it. - -[reconfigure gitlab]: ../restart_gitlab.md#omnibus-gitlab-reconfigure "How to reconfigure Omnibus GitLab" -[restart gitlab]: ../restart_gitlab.md#installations-from-source "How to restart GitLab" -[eep]: https://about.gitlab.com/pricing/ "GitLab Premium" -[ee-2760]: https://gitlab.com/gitlab-org/gitlab/-/merge_requests/2760 diff --git a/doc/administration/libravatar.md b/doc/administration/libravatar.md index c28e701dc25..e1c38b3409f 100644 --- a/doc/administration/libravatar.md +++ b/doc/administration/libravatar.md @@ -19,7 +19,7 @@ the configuration options as follows: ### For HTTP -```yml +```yaml gravatar: enabled: true # gravatar URLs: possible placeholders: %{hash} %{size} %{email} %{username} @@ -28,7 +28,7 @@ the configuration options as follows: ### For HTTPS -```yml +```yaml gravatar: enabled: true # gravatar URLs: possible placeholders: %{hash} %{size} %{email} %{username} diff --git a/doc/administration/logs.md b/doc/administration/logs.md index e4a3514a539..57b12897979 100644 --- a/doc/administration/logs.md +++ b/doc/administration/logs.md @@ -1,3 +1,9 @@ +--- +stage: Monitor +group: APM +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + # Log system GitLab has an advanced log system where everything is logged, so you @@ -333,6 +339,9 @@ only. For example: ## `audit_json.log` +NOTE: **Note:** +Most log entries only exist in [GitLab Starter](https://about.gitlab.com/pricing), however a few exist in GitLab Core. + This file lives in `/var/log/gitlab/gitlab-rails/audit_json.log` for Omnibus GitLab packages or in `/home/git/gitlab/log/audit_json.log` for installations from source. @@ -356,7 +365,11 @@ Changes to group or project settings are logged to this file. For example: } ``` -## `sidekiq.log` +## Sidekiq Logs + +For Omnibus installations, some Sidekiq logs reside in `/var/log/gitlab/sidekiq/current` and as follows. + +### `sidekiq.log` This file lives in `/var/log/gitlab/gitlab-rails/sidekiq.log` for Omnibus GitLab packages or in `/home/git/gitlab/log/sidekiq.log` for @@ -413,7 +426,7 @@ For source installations, edit the `gitlab.yml` and set the Sidekiq log_format: json ``` -## `sidekiq_client.log` +### `sidekiq_client.log` > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/26586) in GitLab 12.9. @@ -424,21 +437,91 @@ installations from source. This file contains logging information about jobs before they are start being processed by Sidekiq, for example before being enqueued. -This logfile follows the same structure as +This log file follows the same structure as [`sidekiq.log`](#sidekiqlog), so it will be structured as JSON if you've configured this for Sidekiq as mentioned above. ## `gitlab-shell.log` -This file lives in `/var/log/gitlab/gitaly/gitlab-shell.log` for -Omnibus GitLab packages or in `/home/git/gitaly/gitlab-shell.log` for -installations from source. +GitLab Shell is used by GitLab for executing Git commands and provide SSH access to Git repositories. + +### For GitLab versions 12.10 and up + +For GitLab version 12.10 and later, there are 2 `gitlab-shell.log` files. Information containing `git-{upload-pack,receive-pack}` requests lives in `/var/log/gitlab/gitlab-shell/gitlab-shell.log`. Information about hooks to GitLab Shell from Gitaly lives in `/var/log/gitlab/gitaly/gitlab-shell.log`. + +Example log entries for `/var/log/gitlab/gitlab-shell/gitlab-shell.log`: + +```json +{ + "duration_ms": 74.104, + "level": "info", + "method": "POST", + "msg": "Finished HTTP request", + "time": "2020-04-17T20:28:46Z", + "url": "http://127.0.0.1:8080/api/v4/internal/allowed" +} +{ + "command": "git-upload-pack", + "git_protocol": "", + "gl_project_path": "root/example", + "gl_repository": "project-1", + "level": "info", + "msg": "executing git command", + "time": "2020-04-17T20:28:46Z", + "user_id": "user-1", + "username": "root" +} +``` + +Example log entries for `/var/log/gitlab/gitaly/gitlab-shell.log`: + +```json +{ + "method": "POST", + "url": "http://127.0.0.1:8080/api/v4/internal/allowed", + "duration": 0.058012959, + "gitaly_embedded": true, + "pid": 16636, + "level": "info", + "msg": "finished HTTP request", + "time": "2020-04-17T20:29:08+00:00" +} +{ + "method": "POST", + "url": "http://127.0.0.1:8080/api/v4/internal/pre_receive", + "duration": 0.031022552, + "gitaly_embedded": true, + "pid": 16636, + "level": "info", + "msg": "finished HTTP request", + "time": "2020-04-17T20:29:08+00:00" +} +``` + +### For GitLab versions 12.5 through 12.9 + +For GitLab 12.5 to 12.9, this file lives in `/var/log/gitlab/gitaly/gitlab-shell.log` for Omnibus GitLab packages or in `/home/git/gitaly/gitlab-shell.log` for installations from source. + +Example log entries: + +```json +{ + "method": "POST", + "url": "http://127.0.0.1:8080/api/v4/internal/post_receive", + "duration": 0.031809164, + "gitaly_embedded": true, + "pid": 27056, + "level": "info", + "msg": "finished HTTP request", + "time": "2020-04-17T16:24:38+00:00" +} +``` + +### For GitLab 12.5 and earlier -NOTE: **Note:** For GitLab 12.5 and earlier, the file lives in `/var/log/gitlab/gitlab-shell/gitlab-shell.log`. -GitLab Shell is used by GitLab for executing Git commands and provide -SSH access to Git repositories. For example: +Example log entries: ```plaintext I, [2015-02-13T06:17:00.671315 #9291] INFO -- : Adding project root/example.git at </var/opt/gitlab/git-data/repositories/root/dcdcdcdcd.git>. @@ -447,15 +530,29 @@ I, [2015-02-13T06:17:00.679433 #9291] INFO -- : Moving existing hooks directory User clone/fetch activity using SSH transport appears in this log as `executing git command <gitaly-upload-pack...`. -## `current` +## Gitaly Logs -This file lives in `/var/log/gitlab/gitaly/current` and is produced by [runit](http://smarden.org/runit/). `runit` is packaged with Omnibus and a brief explanation of its purpose is available [in the omnibus documentation](https://docs.gitlab.com/omnibus/architecture/#runit). [Log files are rotated](http://smarden.org/runit/svlogd.8.html), renamed in unix timestamp format and `gzip`-compressed (e.g. `@1584057562.s`). +This file lives in `/var/log/gitlab/gitaly/current` and is produced by [runit](http://smarden.org/runit/). `runit` is packaged with Omnibus and a brief explanation of its purpose is available [in the omnibus documentation](https://docs.gitlab.com/omnibus/architecture/#runit). [Log files are rotated](http://smarden.org/runit/svlogd.8.html), renamed in Unix timestamp format and `gzip`-compressed (e.g. `@1584057562.s`). -## `unicorn_stderr.log` +### `grpc.log` -This file lives in `/var/log/gitlab/unicorn/unicorn_stderr.log` for -Omnibus GitLab packages or in `/home/git/gitlab/log/unicorn_stderr.log` for -installations from source. +This file lives in `/var/log/gitlab/gitlab-rails/grpc.log` for Omnibus GitLab packages. Native [gRPC](https://grpc.io/) logging used by Gitaly. + +## `puma_stderr.log` & `puma_stdout.log` + +This file lives in `/var/log/gitlab/puma/puma_stderr.log` and `/var/log/gitlab/puma/puma_stdout.log` for +Omnibus GitLab packages or in `/home/git/gitlab/log/puma_stderr.log` and `/home/git/gitlab/log/puma_stdout.log` +for installations from source. + +## `unicorn_stderr.log` & `unicorn_stdout.log` + +NOTE: **Note:** +Starting with GitLab 13.0, Puma is the default web server used in GitLab +all-in-one package based installations as well as GitLab Helm chart deployments. + +This file lives in `/var/log/gitlab/unicorn/unicorn_stderr.log` and `/var/log/gitlab/unicorn/unicorn_stdout.log` for +Omnibus GitLab packages or in `/home/git/gitlab/log/unicorn_stderr.log` and `/home/git/gitlab/log/unicorn_stdout.log` +for installations from source. Unicorn is a high-performance forking Web server which is used for serving the GitLab application. You can look at this log if, for @@ -484,7 +581,7 @@ This file lives in `/var/log/gitlab/gitlab-rails/repocheck.log` for Omnibus GitLab packages or in `/home/git/gitlab/log/repocheck.log` for installations from source. -It logs information whenever a [repository check is run][repocheck] on a project. +It logs information whenever a [repository check is run](repository_checks.md) on a project. ## `importer.log` @@ -494,6 +591,8 @@ This file lives in `/var/log/gitlab/gitlab-rails/importer.log` for Omnibus GitLab packages or in `/home/git/gitlab/log/importer.log` for installations from source. +It logs the progress of the import process. + ## `auth.log` > Introduced in GitLab 12.0. @@ -504,9 +603,9 @@ installations from source. This log records: -- Information whenever [Rack Attack] registers an abusive request. -- Requests over the [Rate Limit] on raw endpoints. -- [Protected paths] abusive requests. +- Information whenever [Rack Attack](../security/rack_attack.md) registers an abusive request. +- Requests over the [Rate Limit](../user/admin_area/settings/rate_limits_on_raw_endpoints.md) on raw endpoints. +- [Protected paths](../user/admin_area/settings/protected_paths.md) abusive requests. NOTE: **Note:** From [%12.1](https://gitlab.com/gitlab-org/gitlab-foss/issues/62756), user ID and username are also @@ -523,7 +622,7 @@ installations from source. GraphQL queries are recorded in that file. For example: ```json -{"query_string":"query IntrospectionQuery{__schema {queryType { name },mutationType { name }}}...(etc)","variables":{"a":1,"b":2},"complexity":181,"depth":1,"duration":7} +{"query_string":"query IntrospectionQuery{__schema {queryType { name },mutationType { name }}}...(etc)","variables":{"a":1,"b":2},"complexity":181,"depth":1,"duration_s":7} ``` ## `migrations.log` @@ -538,7 +637,7 @@ installations from source. > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/19186) in GitLab 12.6. -This file lives in `/var/log/gitlab/mail_room/mail_room_json.log` for +This file lives in `/var/log/gitlab/mailroom/mail_room_json.log` for Omnibus GitLab packages or in `/home/git/gitlab/log/mail_room_json.log` for installations from source. @@ -562,7 +661,7 @@ will be generated in `/var/log/gitlab/gitlab-rails/sidekiq_exporter.log` for Omnibus GitLab packages or in `/home/git/gitlab/log/sidekiq_exporter.log` for installations from source. -If Prometheus metrics and the Web Exporter are both enabled, Unicorn/Puma will +If Prometheus metrics and the Web Exporter are both enabled, Puma/Unicorn will start a Web server and listen to the defined port (default: `8083`). Access logs will be generated in `/var/log/gitlab/gitlab-rails/web_exporter.log` for Omnibus GitLab packages or in `/home/git/gitlab/log/web_exporter.log` for @@ -578,7 +677,7 @@ It's stored at: - `/var/log/gitlab/gitlab-rails/database_load_balancing.log` for Omnibus GitLab packages. - `/home/git/gitlab/log/database_load_balancing.log` for installations from source. -## `elasticsearch.log` +## `elasticsearch.log` **(STARTER ONLY)** > Introduced in GitLab 12.6. @@ -648,7 +747,24 @@ Each line contains a JSON line that can be ingested by Elasticsearch. For exampl } ``` -## `geo.log` +## `service_measurement.log` + +> Introduced in GitLab 13.0. + +This file lives in `/var/log/gitlab/gitlab-rails/service_measurement.log` for +Omnibus GitLab packages or in `/home/git/gitlab/log/service_measurement.log` for +installations from source. + +It contain only a single structured log with measurements for each service execution. +It will contain measurement such as: number of sql calls, execution_time, gc_stats, memory usage, etc... + +For example: + +```json +{ "severity":"INFO", "time":"2020-04-22T16:04:50.691Z","correlation_id":"04f1366e-57a1-45b8-88c1-b00b23dc3616","class":"Projects::ImportExport::ExportService","current_user":"John Doe","project_full_path":"group1/test-export","file_path":"/path/to/archive","gc_stats":{"count":{"before":127,"after":127,"diff":0},"heap_allocated_pages":{"before":10369,"after":10369,"diff":0},"heap_sorted_length":{"before":10369,"after":10369,"diff":0},"heap_allocatable_pages":{"before":0,"after":0,"diff":0},"heap_available_slots":{"before":4226409,"after":4226409,"diff":0},"heap_live_slots":{"before":2542709,"after":2641420,"diff":98711},"heap_free_slots":{"before":1683700,"after":1584989,"diff":-98711},"heap_final_slots":{"before":0,"after":0,"diff":0},"heap_marked_slots":{"before":2542704,"after":2542704,"diff":0},"heap_eden_pages":{"before":10369,"after":10369,"diff":0},"heap_tomb_pages":{"before":0,"after":0,"diff":0},"total_allocated_pages":{"before":10369,"after":10369,"diff":0},"total_freed_pages":{"before":0,"after":0,"diff":0},"total_allocated_objects":{"before":24896308,"after":24995019,"diff":98711},"total_freed_objects":{"before":22353599,"after":22353599,"diff":0},"malloc_increase_bytes":{"before":140032,"after":6650240,"diff":6510208},"malloc_increase_bytes_limit":{"before":25804104,"after":25804104,"diff":0},"minor_gc_count":{"before":94,"after":94,"diff":0},"major_gc_count":{"before":33,"after":33,"diff":0},"remembered_wb_unprotected_objects":{"before":34284,"after":34284,"diff":0},"remembered_wb_unprotected_objects_limit":{"before":68568,"after":68568,"diff":0},"old_objects":{"before":2404725,"after":2404725,"diff":0},"old_objects_limit":{"before":4809450,"after":4809450,"diff":0},"oldmalloc_increase_bytes":{"before":140032,"after":6650240,"diff":6510208},"oldmalloc_increase_bytes_limit":{"before":68537556,"after":68537556,"diff":0}},"time_to_finish":0.12298400001600385,"number_of_sql_calls":70,"memory_usage":"0.0 MiB","label":"process_48616"} +``` + +## `geo.log` **(PREMIUM ONLY)** > Introduced in 9.5. @@ -677,7 +793,9 @@ For Omnibus installations, NGINX logs reside in: - `/var/log/gitlab/nginx/gitlab_pages_access.log` contains a log of requests made to Pages static sites. - `/var/log/gitlab/nginx/gitlab_pages_error.log` contains a log of NGINX errors for Pages static sites. - `/var/log/gitlab/nginx/gitlab_registry_access.log` contains a log of requests made to the Container Registry. -- `/var/log/gitlab/nginx/gitlab_registry_error.log` contains a log of NGINX errors for the Container Regsitry. +- `/var/log/gitlab/nginx/gitlab_registry_error.log` contains a log of NGINX errors for the Container Registry. +- `/var/log/gitlab/nginx/gitlab_mattermost_access.log` contains a log of requests made to Mattermost. +- `/var/log/gitlab/nginx/gitlab_mattermost_error.log` contains a log of NGINX errors for Mattermost. Below is the default GitLab NGINX access log format: @@ -685,7 +803,51 @@ Below is the default GitLab NGINX access log format: $remote_addr - $remote_user [$time_local] "$request" $status $body_bytes_sent "$http_referer" "$http_user_agent" ``` -[repocheck]: repository_checks.md -[Rack Attack]: ../security/rack_attack.md -[Rate Limit]: ../user/admin_area/settings/rate_limits_on_raw_endpoints.md -[Protected paths]: ../user/admin_area/settings/protected_paths.md +## Pages Logs + +For Omnibus installations, Pages logs reside in `/var/log/gitlab/gitlab-pages/current`. + +For example: + +```json +{ + "level": "info", + "msg": "GitLab Pages Daemon", + "revision": "52b2899", + "time": "2020-04-22T17:53:12Z", + "version": "1.17.0" +} +{ + "level": "info", + "msg": "URL: https://gitlab.com/gitlab-org/gitlab-pages", + "time": "2020-04-22T17:53:12Z" +} +{ + "gid": 998, + "in-place": false, + "level": "info", + "msg": "running the daemon as unprivileged user", + "time": "2020-04-22T17:53:12Z", + "uid": 998 +} +``` + +## Workhorse Logs + +For Omnibus installations, Workhorse logs reside in `/var/log/gitlab/gitlab-workhorse/current`. + +## PostgreSQL Logs + +For Omnibus installations, PostgreSQL logs reside in `/var/log/gitlab/postgresql/current`. + +## Prometheus Logs + +For Omnibus installations, Prometheus logs reside in `/var/log/gitlab/prometheus/current`. + +## Redis Logs + +For Omnibus installations, Redis logs reside in `/var/log/gitlab/redis/current`. + +## Mattermost Logs + +For Omnibus installations, Mattermost logs reside in `/var/log/gitlab/mattermost/mattermost.log`. diff --git a/doc/administration/monitoring/gitlab_self_monitoring_project/index.md b/doc/administration/monitoring/gitlab_self_monitoring_project/index.md index 884bb44cfda..091da46f316 100644 --- a/doc/administration/monitoring/gitlab_self_monitoring_project/index.md +++ b/doc/administration/monitoring/gitlab_self_monitoring_project/index.md @@ -15,7 +15,11 @@ All administrators at the time of creation of the project and group will be adde as maintainers of the group and project, and as an admin, you'll be able to add new members to the group in order to give them maintainer access to the project. -This project will be used for self monitoring your GitLab instance. +This project is used to self monitor your GitLab instance. Metrics are not yet +fully integrated, and the dashboard does not aggregate any data on Omnibus installations. GitLab plans +to provide integrated self-monitoring metrics in a future release. You can +currently use the project to configure your own [custom metrics](../../../user/project/integrations/prometheus.md#adding-custom-metrics) using +metrics exposed by the [GitLab exporter](../prometheus/gitlab_metrics.md#metrics-available). ## Creating the self monitoring project @@ -42,7 +46,7 @@ The project will be automatically configured to connect to the instance is present (should be the case if GitLab was installed via Omnibus and you haven't disabled it). -If that's not the case or if you have an external Prometheus instance or an HA setup, +If that's not the case or if you have an external Prometheus instance or a customized setup, you should [configure it manually](../../../user/project/integrations/prometheus.md#manual-configuration-of-prometheus). @@ -70,7 +74,7 @@ project creation to fail with the following error (which appears in the log file when the first admin user is an [external user](../../../user/permissions.md#external-users-core-only): -```text +```plaintext Could not create instance administrators group. Errors: ["You don’t have permission to create groups."] ``` diff --git a/doc/administration/monitoring/performance/gitlab_configuration.md b/doc/administration/monitoring/performance/gitlab_configuration.md index e8a6c661464..14119a5d8f3 100644 --- a/doc/administration/monitoring/performance/gitlab_configuration.md +++ b/doc/administration/monitoring/performance/gitlab_configuration.md @@ -1,17 +1,9 @@ # GitLab Configuration -CAUTION: **InfluxDB is deprecated in favor of Prometheus:** -InfluxDB support is scheduled to be removed in GitLab 13.0. -You are advised to use [Prometheus](../prometheus/index.md) instead. - GitLab Performance Monitoring is disabled by default. To enable it and change any of its settings, navigate to **Admin Area > Settings > Metrics and profiling** (`/admin/application_settings/metrics_and_profiling`). -The minimum required settings you need to set are the InfluxDB host and port. -Make sure _Enable InfluxDB Metrics_ is checked and hit **Save** to save the -changes. -  Finally, a restart of all GitLab processes is required for the changes to take @@ -33,6 +25,4 @@ have been performed. Read more on: - [Introduction to GitLab Performance Monitoring](index.md) -- [InfluxDB Configuration](influxdb_configuration.md) -- [InfluxDB Schema](influxdb_schema.md) - [Grafana Install/Configuration](grafana_configuration.md) diff --git a/doc/administration/monitoring/performance/grafana_configuration.md b/doc/administration/monitoring/performance/grafana_configuration.md index 3c26e399c7f..3438a564d2f 100644 --- a/doc/administration/monitoring/performance/grafana_configuration.md +++ b/doc/administration/monitoring/performance/grafana_configuration.md @@ -1,21 +1,12 @@ # Grafana Configuration -CAUTION: **InfluxDB is deprecated in favor of Prometheus:** -InfluxDB support is scheduled to be removed in GitLab 13.0. -You are advised to use [Prometheus](../prometheus/index.md) instead. - [Grafana](https://grafana.com/) is a tool that allows you to visualize time -series metrics through graphs and dashboards. It supports several backend -data stores, including InfluxDB. GitLab writes performance data to InfluxDB +series metrics through graphs and dashboards. GitLab writes performance data to Prometheus and Grafana will allow you to query to display useful graphs. -For the easiest installation and configuration, install Grafana on the same -server as InfluxDB. For larger installations, you may want to split out these -services. - ## Installation -[GitLab Omnibus can help you install Grafana (recommended)](https://docs.gitlab.com/omnibus/settings/grafana.html) +[Omnibus GitLab can help you install Grafana (recommended)](https://docs.gitlab.com/omnibus/settings/grafana.html) or Grafana supplies package repositories (Yum/Apt) for easy installation. See [Grafana installation documentation](https://grafana.com/docs/grafana/latest/installation/) for detailed steps. @@ -33,49 +24,8 @@ in the top bar.  -Fill in the configuration details for the InfluxDB data source. Save and -Test Connection to ensure the configuration is correct. - -- **Name**: `InfluxDB` -- **Default**: Checked -- **Type**: `InfluxDB 0.9.x` (Even if you're using InfluxDB 0.10.x) -- For the URL, use `https://localhost:8086`, or provide the remote URL if you've installed InfluxDB - on a separate server -- **Access**: `proxy` -- **Database**: `gitlab` -- **User**: `admin` (Or the username configured when setting up InfluxDB) -- **Password**: The password configured when you set up InfluxDB -  -## Apply retention policies and create continuous queries - -If you intend to import the GitLab provided Grafana dashboards, you will need to -set up the right retention policies and continuous queries. The easiest way of -doing this is by using the [InfluxDB Management](https://gitlab.com/gitlab-org/influxdb-management) -repository. - -To use this repository you must first clone it: - -```shell -git clone https://gitlab.com/gitlab-org/influxdb-management.git -cd influxdb-management -``` - -Next you must install the required dependencies: - -```shell -gem install bundler -bundle install -``` - -Now you must configure the repository by first copying `.env.example` to `.env` -and then editing the `.env` file to contain the correct InfluxDB settings. Once -configured you can simply run `bundle exec rake` and the InfluxDB database will -be configured for you. - -For more information see the [InfluxDB Management README](https://gitlab.com/gitlab-org/influxdb-management/blob/master/README.md). - ## Import Dashboards You can now import a set of default dashboards that will give you a good @@ -164,5 +114,3 @@ Read more on: - [Introduction to GitLab Performance Monitoring](index.md) - [GitLab Configuration](gitlab_configuration.md) -- [InfluxDB Installation/Configuration](influxdb_configuration.md) -- [InfluxDB Schema](influxdb_schema.md) diff --git a/doc/administration/monitoring/performance/index.md b/doc/administration/monitoring/performance/index.md index 3305ea33f2e..02070287611 100644 --- a/doc/administration/monitoring/performance/index.md +++ b/doc/administration/monitoring/performance/index.md @@ -1,9 +1,5 @@ # GitLab Performance Monitoring -CAUTION: **InfluxDB is deprecated in favor of Prometheus:** -InfluxDB support is scheduled to be removed in GitLab 13.0. -You are advised to use [Prometheus](../prometheus/index.md) instead. - GitLab comes with its own application performance measuring system as of GitLab 8.4, simply called "GitLab Performance Monitoring". GitLab Performance Monitoring is available in both the Community and Enterprise editions. @@ -12,17 +8,11 @@ Apart from this introduction, you are advised to read through the following documents in order to understand and properly configure GitLab Performance Monitoring: - [GitLab Configuration](gitlab_configuration.md) -- [InfluxDB Install/Configuration](influxdb_configuration.md) -- [InfluxDB Schema](influxdb_schema.md) +- [Prometheus documentation](../prometheus/index.md) - [Grafana Install/Configuration](grafana_configuration.md) - [Performance bar](performance_bar.md) - [Request profiling](request_profiling.md) -NOTE: **Note:** -Omnibus GitLab 8.16 includes Prometheus as an additional tool to collect -metrics. It will eventually replace InfluxDB when their metrics collection is -on par. Read more in the [Prometheus documentation](../prometheus/index.md). - ## Introduction to GitLab Performance Monitoring GitLab Performance Monitoring makes it possible to measure a wide variety of statistics @@ -35,12 +25,6 @@ including (but not limited to): - System statistics such as the process' memory usage and open file descriptors. - Ruby garbage collection statistics. -Metrics data is written to [InfluxDB](https://www.influxdata.com/products/influxdb-overview/) -over [UDP](https://docs.influxdata.com/influxdb/v0.9/write_protocols/udp/). -Stored data can be visualized using [Grafana](https://grafana.com) or any other application that -supports reading data from InfluxDB. Alternatively data can be queried using the -InfluxDB CLI. - ## Metric Types Two types of metrics are collected: diff --git a/doc/administration/monitoring/performance/influxdb_configuration.md b/doc/administration/monitoring/performance/influxdb_configuration.md index 8478272897b..d793e014a18 100644 --- a/doc/administration/monitoring/performance/influxdb_configuration.md +++ b/doc/administration/monitoring/performance/influxdb_configuration.md @@ -1,191 +1,5 @@ -# InfluxDB Configuration - -CAUTION: **InfluxDB is deprecated in favor of Prometheus:** -InfluxDB support is scheduled to be removed in GitLab 13.0. -You are advised to use [Prometheus](../prometheus/index.md) instead. - -The default settings provided by [InfluxDB](https://www.influxdata.com/products/influxdb-overview/) -are not sufficient for a high traffic GitLab environment. The settings discussed in -this document are based on the settings GitLab uses for GitLab.com. Depending on -your own needs, you may need to further adjust them. - -If you are intending to run InfluxDB on the same server as GitLab, make sure -you have plenty of RAM since InfluxDB can use quite a bit depending on traffic. - -Unless you are going with a budget setup, it's advised to run it separately. - -## Requirements - -- InfluxDB 0.9.5 or newer -- A fairly modern version of Linux -- At least 4GB of RAM -- At least 10GB of storage for InfluxDB data - -Note that the RAM and storage requirements can differ greatly depending on the -amount of data received/stored. To limit the amount of stored data users can -look into -[InfluxDB Retention Policies](https://docs.influxdata.com/influxdb/v0.9/query_language/database_management/#retention-policy-management). - -## Installation - -Installing InfluxDB is out of the scope of this document. Please refer to the -[InfluxDB documentation](https://docs.influxdata.com/influxdb/v0.9/). - -## InfluxDB Server Settings - -Since InfluxDB has many settings that users may wish to customize themselves -(e.g. what port to run InfluxDB on), we'll only cover the essentials. - -The configuration file in question is usually located at -`/etc/influxdb/influxdb.conf`. Whenever you make a change in this file, -InfluxDB needs to be restarted. - -### Storage Engine - -InfluxDB comes with different storage engines and as of InfluxDB 0.9.5 a new -storage engine is available, called [TSM Tree](https://www.influxdata.com/blog/new-storage-engine-time-structured-merge-tree/). -All users **must** use the new `tsm1` storage engine as this -[will be the default engine](https://github.com/influxdata/influxdb/commit/15d723dc77651bac83e09e2b1c94be480966cb0d) in -upcoming InfluxDB releases. - -Make sure you have the following in your configuration file: - -```toml -[data] - dir = "/var/lib/influxdb/data" - engine = "tsm1" -``` - -### Admin Panel - -Production environments should have the InfluxDB admin panel **disabled**. This -feature can be disabled by adding the following to your InfluxDB configuration -file: - -```toml -[admin] - enabled = false -``` - -### HTTP - -HTTP is required when using the [InfluxDB CLI](https://docs.influxdata.com/influxdb/v0.9/tools/shell/) -or other tools such as Grafana, thus it should be enabled. When enabling -make sure to _also_ enable authentication: - -```toml -[http] - enabled = true - auth-enabled = true -``` - -_**Note:** Before you enable authentication, you might want to [create an -admin user](#create-a-new-admin-user)._ - -### UDP - -GitLab writes data to InfluxDB via UDP and thus this must be enabled. Enabling -UDP can be done using the following settings: - -```toml -[[udp]] - enabled = true - bind-address = ":8089" - database = "gitlab" - batch-size = 1000 - batch-pending = 5 - batch-timeout = "1s" - read-buffer = 209715200 -``` - -This does the following: - -1. Enable UDP and bind it to port 8089 for all addresses. -1. Store any data received in the `gitlab` database. -1. Define a batch of points to be 1000 points in size and allow a maximum of - 5 batches _or_ flush them automatically after 1 second. -1. Define a UDP read buffer size of 200 MB. - -One of the most important settings here is the UDP read buffer size as if this -value is set too low, packets will be dropped. You must also make sure the OS -buffer size is set to the same value, the default value is almost never enough. - -To set the OS buffer size to 200 MB, on Linux you can run the following command: - -```shell -sysctl -w net.core.rmem_max=209715200 -``` - -To make this permanent, add the following to `/etc/sysctl.conf` and restart the -server: - -```shell -net.core.rmem_max=209715200 -``` - -It is **very important** to make sure the buffer sizes are large enough to -handle all data sent to InfluxDB as otherwise you _will_ lose data. The above -buffer sizes are based on the traffic for GitLab.com. Depending on the amount of -traffic, users may be able to use a smaller buffer size, but we highly recommend -using _at least_ 100 MB. - -When enabling UDP, users should take care to not expose the port to the public, -as doing so will allow anybody to write data into your InfluxDB database (as -[InfluxDB's UDP protocol](https://docs.influxdata.com/influxdb/v0.9/write_protocols/udp/) -doesn't support authentication). We recommend either -whitelisting the allowed IP addresses/ranges, or setting up a VLAN and only -allowing traffic from members of said VLAN. - -## Create a new admin user - -If you want to [enable authentication](#http), you might want to [create an -admin user](https://docs.influxdata.com/influxdb/v0.9/administration/authentication_and_authorization/#create-a-new-admin-user): - -```shell -influx -execute "CREATE USER jeff WITH PASSWORD '1234' WITH ALL PRIVILEGES" -``` - -## Create the `gitlab` database - -Once you get InfluxDB up and running, you need to create a database for GitLab. -Make sure you have changed the [storage engine](#storage-engine) to `tsm1` -before creating a database. - -_**Note:** If you [created an admin user](#create-a-new-admin-user) and enabled -[HTTP authentication](#http), remember to append the username (`-username <username>`) -and password (`-password <password>`) you set earlier to the commands below._ - -Run the following command to create a database named `gitlab`: - -```shell -influx -execute 'CREATE DATABASE gitlab' -``` - -The name **must** be `gitlab`, do not use any other name. - -Next, make sure that the database was successfully created: - -```shell -influx -execute 'SHOW DATABASES' -``` - -The output should be similar to: - -```plaintext -name: databases ---------------- -name -_internal -gitlab -``` - -That's it! Now your GitLab instance should send data to InfluxDB. - +--- +redirect_to: 'prometheus.md' --- -Read more on: - -- [Introduction to GitLab Performance Monitoring](index.md) -- [GitLab Configuration](gitlab_configuration.md) -- [InfluxDB Schema](influxdb_schema.md) -- [Grafana Install/Configuration](grafana_configuration.md) +Support for InfluxDB was removed in GitLab 13.0. Use [Prometheus](prometheus.md) for performance monitoring. diff --git a/doc/administration/monitoring/performance/influxdb_schema.md b/doc/administration/monitoring/performance/influxdb_schema.md index adbccdaaeb8..d793e014a18 100644 --- a/doc/administration/monitoring/performance/influxdb_schema.md +++ b/doc/administration/monitoring/performance/influxdb_schema.md @@ -1,101 +1,5 @@ -# InfluxDB Schema - -CAUTION: **InfluxDB is deprecated in favor of Prometheus:** -InfluxDB support is scheduled to be removed in GitLab 13.0. -You are advised to use [Prometheus](../prometheus/index.md) instead. - -The following measurements are currently stored in InfluxDB: - -- `PROCESS_file_descriptors` -- `PROCESS_gc_statistics` -- `PROCESS_memory_usage` -- `PROCESS_method_calls` -- `PROCESS_object_counts` -- `PROCESS_transactions` -- `PROCESS_views` -- `events` - -Here, `PROCESS` is replaced with either `rails` or `sidekiq` depending on the -process type. In all series, any form of duration is stored in milliseconds. - -## PROCESS_file_descriptors - -This measurement contains the number of open file descriptors over time. The -value field `value` contains the number of descriptors. - -## PROCESS_gc_statistics - -This measurement contains Ruby garbage collection statistics such as the amount -of minor/major GC runs (relative to the last sampling interval), the time spent -in garbage collection cycles, and all fields/values returned by `GC.stat`. - -## PROCESS_memory_usage - -This measurement contains the process' memory usage (in bytes) over time. The -value field `value` contains the number of bytes. - -## PROCESS_method_calls - -This measurement contains the methods called during a transaction along with -their duration, and a name of the transaction action that invoked the method (if -available). The method call duration is stored in the value field `duration`, -while the method name is stored in the tag `method`. The tag `action` contains -the full name of the transaction action. Both the `method` and `action` fields -are in the following format: - -```plaintext -ClassName#method_name -``` - -For example, a method called by the `show` method in the `UsersController` class -would have `action` set to `UsersController#show`. - -## PROCESS_object_counts - -This measurement is used to store retained Ruby objects (per class) and the -amount of retained objects. The number of objects is stored in the `count` value -field while the class name is stored in the `type` tag. - -## PROCESS_transactions - -This measurement is used to store basic transaction details such as the time it -took to complete a transaction, how much time was spent in SQL queries, etc. The -following value fields are available: - -| Value | Description | -| ----- | ----------- | -| `duration` | The total duration of the transaction | -| `allocated_memory` | The amount of bytes allocated while the transaction was running. This value is only reliable when using single-threaded application servers | -| `method_duration` | The total time spent in method calls | -| `sql_duration` | The total time spent in SQL queries | -| `view_duration` | The total time spent in views | - -## PROCESS_views - -This measurement is used to store view rendering timings for a transaction. The -following value fields are available: - -| Value | Description | -| ----- | ----------- | -| `duration` | The rendering time of the view | -| `view` | The path of the view, relative to the application's root directory | - -The `action` tag contains the action name of the transaction that rendered the -view. - -## events - -This measurement is used to store generic events such as the number of Git -pushes, Emails sent, etc. Each point in this measurement has a single value -field called `count`. The value of this field is simply set to `1`. Each point -also has at least one tag: `event`. This tag's value is set to the event name. -Depending on the event type additional tags may be available as well. - +--- +redirect_to: 'prometheus.md' --- -Read more on: - -- [Introduction to GitLab Performance Monitoring](index.md) -- [GitLab Configuration](gitlab_configuration.md) -- [InfluxDB Configuration](influxdb_configuration.md) -- [Grafana Install/Configuration](grafana_configuration.md) +Support for InfluxDB was removed in GitLab 13.0. Use [Prometheus](prometheus.md) for performance monitoring. diff --git a/doc/administration/monitoring/prometheus/gitlab_exporter.md b/doc/administration/monitoring/prometheus/gitlab_exporter.md index 9c5c67e7f67..3effca4a2bf 100644 --- a/doc/administration/monitoring/prometheus/gitlab_exporter.md +++ b/doc/administration/monitoring/prometheus/gitlab_exporter.md @@ -1,3 +1,9 @@ +--- +stage: Monitor +group: APM +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + # GitLab exporter >- Available since [Omnibus GitLab 8.17](https://gitlab.com/gitlab-org/omnibus-gitlab/-/merge_requests/1132). diff --git a/doc/administration/monitoring/prometheus/gitlab_metrics.md b/doc/administration/monitoring/prometheus/gitlab_metrics.md index 0833412ecb8..f725db9a039 100644 --- a/doc/administration/monitoring/prometheus/gitlab_metrics.md +++ b/doc/administration/monitoring/prometheus/gitlab_metrics.md @@ -1,3 +1,9 @@ +--- +stage: Monitor +group: APM +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + # GitLab Prometheus metrics >**Note:** @@ -31,16 +37,17 @@ The following metrics are available: | Metric | Type | Since | Description | Labels | |:---------------------------------------------------------------|:----------|-----------------------:|:----------------------------------------------------------------------------------------------------|:----------------------------------------------------| -| `gitlab_banzai_cached_render_real_duration_seconds` | Histogram | 9.4 | Duration of rendering Markdown into HTML when cached output exists | controller, action | -| `gitlab_banzai_cacheless_render_real_duration_seconds` | Histogram | 9.4 | Duration of rendering Markdown into HTML when cached output does not exist | controller, action | -| `gitlab_cache_misses_total` | Counter | 10.2 | Cache read miss | controller, action | +| `gitlab_banzai_cached_render_real_duration_seconds` | Histogram | 9.4 | Duration of rendering Markdown into HTML when cached output exists | `controller`, `action` | +| `gitlab_banzai_cacheless_render_real_duration_seconds` | Histogram | 9.4 | Duration of rendering Markdown into HTML when cached output does not exist | `controller`, `action` | +| `gitlab_cache_misses_total` | Counter | 10.2 | Cache read miss | `controller`, `action` | | `gitlab_cache_operation_duration_seconds` | Histogram | 10.2 | Cache access time | | -| `gitlab_cache_operations_total` | Counter | 12.2 | Cache operations by controller/action | controller, action, operation | -| `job_waiter_started_total` | Counter | 12.9 | Number of batches of jobs started where a web request is waiting for the jobs to complete | worker | -| `job_waiter_timeouts_total` | Counter | 12.9 | Number of batches of jobs that timed out where a web request is waiting for the jobs to complete | worker | +| `gitlab_cache_operations_total` | Counter | 12.2 | Cache operations by controller/action | `controller`, `action`, `operation` | +| `gitlab_ci_pipeline_creation_duration_seconds` | Histogram | 13.0 | Time in seconds it takes to create a CI/CD pipeline | | +| `job_waiter_started_total` | Counter | 12.9 | Number of batches of jobs started where a web request is waiting for the jobs to complete | `worker` | +| `job_waiter_timeouts_total` | Counter | 12.9 | Number of batches of jobs that timed out where a web request is waiting for the jobs to complete | `worker` | | `gitlab_database_transaction_seconds` | Histogram | 12.1 | Time spent in database transactions, in seconds | | -| `gitlab_method_call_duration_seconds` | Histogram | 10.2 | Method calls real duration | controller, action, module, method | -| `gitlab_page_out_of_bounds` | Counter | 12.8 | Counter for the PageLimiter pagination limit being hit | controller, action, bot | +| `gitlab_method_call_duration_seconds` | Histogram | 10.2 | Method calls real duration | `controller`, `action`, `module`, `method` | +| `gitlab_page_out_of_bounds` | Counter | 12.8 | Counter for the PageLimiter pagination limit being hit | `controller`, `action`, `bot` | | `gitlab_rails_queue_duration_seconds` | Histogram | 9.4 | Measures latency between GitLab Workhorse forwarding a request to Rails | | | `gitlab_sql_duration_seconds` | Histogram | 10.2 | SQL execution time, excluding SCHEMA operations and BEGIN / COMMIT | | | `gitlab_transaction_allocated_memory_bytes` | Histogram | 10.2 | Allocated memory for all transactions (gitlab_transaction_* metrics) | | @@ -48,27 +55,27 @@ The following metrics are available: | `gitlab_transaction_cache_<key>_duration_total` | Counter | 10.2 | Counter for total time (seconds) spent in Rails cache calls (per key) | | | `gitlab_transaction_cache_count_total` | Counter | 10.2 | Counter for total Rails cache calls (aggregate) | | | `gitlab_transaction_cache_duration_total` | Counter | 10.2 | Counter for total time (seconds) spent in Rails cache calls (aggregate) | | -| `gitlab_transaction_cache_read_hit_count_total` | Counter | 10.2 | Counter for cache hits for Rails cache calls | controller, action | -| `gitlab_transaction_cache_read_miss_count_total` | Counter | 10.2 | Counter for cache misses for Rails cache calls | controller, action | -| `gitlab_transaction_duration_seconds` | Histogram | 10.2 | Duration for all transactions (gitlab_transaction_* metrics) | controller, action | +| `gitlab_transaction_cache_read_hit_count_total` | Counter | 10.2 | Counter for cache hits for Rails cache calls | `controller`, `action` | +| `gitlab_transaction_cache_read_miss_count_total` | Counter | 10.2 | Counter for cache misses for Rails cache calls | `controller`, `action` | +| `gitlab_transaction_duration_seconds` | Histogram | 10.2 | Duration for all transactions (gitlab_transaction_* metrics) | `controller`, `action` | | `gitlab_transaction_event_build_found_total` | Counter | 9.4 | Counter for build found for API /jobs/request | | | `gitlab_transaction_event_build_invalid_total` | Counter | 9.4 | Counter for build invalid due to concurrency conflict for API /jobs/request | | | `gitlab_transaction_event_build_not_found_cached_total` | Counter | 9.4 | Counter for cached response of build not found for API /jobs/request | | | `gitlab_transaction_event_build_not_found_total` | Counter | 9.4 | Counter for build not found for API /jobs/request | | | `gitlab_transaction_event_change_default_branch_total` | Counter | 9.4 | Counter when default branch is changed for any repository | | | `gitlab_transaction_event_create_repository_total` | Counter | 9.4 | Counter when any repository is created | | -| `gitlab_transaction_event_etag_caching_cache_hit_total` | Counter | 9.4 | Counter for etag cache hit. | endpoint | -| `gitlab_transaction_event_etag_caching_header_missing_total` | Counter | 9.4 | Counter for etag cache miss - header missing | endpoint | -| `gitlab_transaction_event_etag_caching_key_not_found_total` | Counter | 9.4 | Counter for etag cache miss - key not found | endpoint | -| `gitlab_transaction_event_etag_caching_middleware_used_total` | Counter | 9.4 | Counter for etag middleware accessed | endpoint | -| `gitlab_transaction_event_etag_caching_resource_changed_total` | Counter | 9.4 | Counter for etag cache miss - resource changed | endpoint | +| `gitlab_transaction_event_etag_caching_cache_hit_total` | Counter | 9.4 | Counter for etag cache hit. | `endpoint` | +| `gitlab_transaction_event_etag_caching_header_missing_total` | Counter | 9.4 | Counter for etag cache miss - header missing | `endpoint` | +| `gitlab_transaction_event_etag_caching_key_not_found_total` | Counter | 9.4 | Counter for etag cache miss - key not found | `endpoint` | +| `gitlab_transaction_event_etag_caching_middleware_used_total` | Counter | 9.4 | Counter for etag middleware accessed | `endpoint` | +| `gitlab_transaction_event_etag_caching_resource_changed_total` | Counter | 9.4 | Counter for etag cache miss - resource changed | `endpoint` | | `gitlab_transaction_event_fork_repository_total` | Counter | 9.4 | Counter for repository forks (RepositoryForkWorker). Only incremented when source repository exists | | | `gitlab_transaction_event_import_repository_total` | Counter | 9.4 | Counter for repository imports (RepositoryImportWorker) | | | `gitlab_transaction_event_push_branch_total` | Counter | 9.4 | Counter for all branch pushes | | -| `gitlab_transaction_event_push_commit_total` | Counter | 9.4 | Counter for commits | branch | +| `gitlab_transaction_event_push_commit_total` | Counter | 9.4 | Counter for commits | `branch` | | `gitlab_transaction_event_push_tag_total` | Counter | 9.4 | Counter for tag pushes | | | `gitlab_transaction_event_rails_exception_total` | Counter | 9.4 | Counter for number of rails exceptions | | -| `gitlab_transaction_event_receive_email_total` | Counter | 9.4 | Counter for received emails | handler | +| `gitlab_transaction_event_receive_email_total` | Counter | 9.4 | Counter for received emails | `handler` | | `gitlab_transaction_event_remote_mirrors_failed_total` | Counter | 10.8 | Counter for failed remote mirrors | | | `gitlab_transaction_event_remote_mirrors_finished_total` | Counter | 10.8 | Counter for finished remote mirrors | | | `gitlab_transaction_event_remote_mirrors_running_total` | Counter | 10.8 | Counter for running remote mirrors | | @@ -76,15 +83,15 @@ The following metrics are available: | `gitlab_transaction_event_remove_repository_total` | Counter | 9.4 | Counter when a repository is removed | | | `gitlab_transaction_event_remove_tag_total` | Counter | 9.4 | Counter when a tag is remove for any repository | | | `gitlab_transaction_event_sidekiq_exception_total` | Counter | 9.4 | Counter of Sidekiq exceptions | | -| `gitlab_transaction_event_stuck_import_jobs_total` | Counter | 9.4 | Count of stuck import jobs | projects_without_jid_count, projects_with_jid_count | -| `gitlab_transaction_event_update_build_total` | Counter | 9.4 | Counter for update build for API /jobs/request/:id | | +| `gitlab_transaction_event_stuck_import_jobs_total` | Counter | 9.4 | Count of stuck import jobs | `projects_without_jid_count`, `projects_with_jid_count` | +| `gitlab_transaction_event_update_build_total` | Counter | 9.4 | Counter for update build for API `/jobs/request/:id` | | | `gitlab_transaction_new_redis_connections_total` | Counter | 9.4 | Counter for new Redis connections | | | `gitlab_transaction_queue_duration_total` | Counter | 9.4 | Duration jobs were enqueued before processing | | -| `gitlab_transaction_rails_queue_duration_total` | Counter | 9.4 | Measures latency between GitLab Workhorse forwarding a request to Rails | controller, action | -| `gitlab_transaction_view_duration_total` | Counter | 9.4 | Duration for views | controller, action, view | -| `gitlab_view_rendering_duration_seconds` | Histogram | 10.2 | Duration for views (histogram) | controller, action, view | -| `http_requests_total` | Counter | 9.4 | Rack request count | method | -| `http_request_duration_seconds` | Histogram | 9.4 | HTTP response time from rack middleware | method, status | +| `gitlab_transaction_rails_queue_duration_total` | Counter | 9.4 | Measures latency between GitLab Workhorse forwarding a request to Rails | `controller`, `action` | +| `gitlab_transaction_view_duration_total` | Counter | 9.4 | Duration for views | `controller`, `action`, `view` | +| `gitlab_view_rendering_duration_seconds` | Histogram | 10.2 | Duration for views (histogram) | `controller`, `action`, `view` | +| `http_requests_total` | Counter | 9.4 | Rack request count | `method` | +| `http_request_duration_seconds` | Histogram | 9.4 | HTTP response time from rack middleware | `method`, `status` | | `pipelines_created_total` | Counter | 9.4 | Counter of pipelines created | | | `rack_uncaught_errors_total` | Counter | 9.4 | Rack connections handling uncaught errors count | | | `user_session_logins_total` | Counter | 9.4 | Counter of how many users have logged in | | @@ -112,49 +119,52 @@ configuration option in `gitlab.yml`. These metrics are served from the | Metric | Type | Since | Description | Labels | |:---------------------------------------------- |:------- |:----- |:----------- |:------ | -| `sidekiq_jobs_cpu_seconds` | Histogram | 12.4 | Seconds of cpu time to run Sidekiq job | queue, boundary, external_dependencies, feature_category, job_status, urgency | -| `sidekiq_jobs_completion_seconds` | Histogram | 12.2 | Seconds to complete Sidekiq job | queue, boundary, external_dependencies, feature_category, job_status, urgency | -| `sidekiq_jobs_db_seconds` | Histogram | 12.9 | Seconds of DB time to run Sidekiq job | queue, boundary, external_dependencies, feature_category, job_status, urgency | -| `sidekiq_jobs_gitaly_seconds` | Histogram | 12.9 | Seconds of Gitaly time to run Sidekiq job | queue, boundary, external_dependencies, feature_category, job_status, urgency | -| `sidekiq_jobs_queue_duration_seconds` | Histogram | 12.5 | Duration in seconds that a Sidekiq job was queued before being executed | queue, boundary, external_dependencies, feature_category, urgency | -| `sidekiq_jobs_failed_total` | Counter | 12.2 | Sidekiq jobs failed | queue, boundary, external_dependencies, feature_category, urgency | -| `sidekiq_jobs_retried_total` | Counter | 12.2 | Sidekiq jobs retried | queue, boundary, external_dependencies, feature_category, urgency | -| `sidekiq_running_jobs` | Gauge | 12.2 | Number of Sidekiq jobs running | queue, boundary, external_dependencies, feature_category, urgency | +| `sidekiq_jobs_cpu_seconds` | Histogram | 12.4 | Seconds of cpu time to run Sidekiq job | `queue`, `boundary`, `external_dependencies`, `feature_category`, `job_status`, `urgency` | +| `sidekiq_jobs_completion_seconds` | Histogram | 12.2 | Seconds to complete Sidekiq job | `queue`, `boundary`, `external_dependencies`, `feature_category`, `job_status`, `urgency` | +| `sidekiq_jobs_db_seconds` | Histogram | 12.9 | Seconds of DB time to run Sidekiq job | `queue`, `boundary`, `external_dependencies`, `feature_category`, `job_status`, `urgency` | +| `sidekiq_jobs_gitaly_seconds` | Histogram | 12.9 | Seconds of Gitaly time to run Sidekiq job | `queue`, `boundary`, `external_dependencies`, `feature_category`, `job_status`, `urgency` | +| `sidekiq_jobs_queue_duration_seconds` | Histogram | 12.5 | Duration in seconds that a Sidekiq job was queued before being executed | `queue`, `boundary`, `external_dependencies`, `feature_category`, `urgency` | +| `sidekiq_jobs_failed_total` | Counter | 12.2 | Sidekiq jobs failed | `queue`, `boundary`, `external_dependencies`, `feature_category`, `urgency` | +| `sidekiq_jobs_retried_total` | Counter | 12.2 | Sidekiq jobs retried | `queue`, `boundary`, `external_dependencies`, `feature_category`, `urgency` | +| `sidekiq_running_jobs` | Gauge | 12.2 | Number of Sidekiq jobs running | `queue`, `boundary`, `external_dependencies`, `feature_category`, `urgency` | | `sidekiq_concurrency` | Gauge | 12.5 | Maximum number of Sidekiq jobs | | -| `geo_db_replication_lag_seconds` | Gauge | 10.2 | Database replication lag (seconds) | url | -| `geo_repositories` | Gauge | 10.2 | Total number of repositories available on primary | url | -| `geo_repositories_synced` | Gauge | 10.2 | Number of repositories synced on secondary | url | -| `geo_repositories_failed` | Gauge | 10.2 | Number of repositories failed to sync on secondary | url | -| `geo_lfs_objects` | Gauge | 10.2 | Total number of LFS objects available on primary | url | -| `geo_lfs_objects_synced` | Gauge | 10.2 | Number of LFS objects synced on secondary | url | -| `geo_lfs_objects_failed` | Gauge | 10.2 | Number of LFS objects failed to sync on secondary | url | -| `geo_attachments` | Gauge | 10.2 | Total number of file attachments available on primary | url | -| `geo_attachments_synced` | Gauge | 10.2 | Number of attachments synced on secondary | url | -| `geo_attachments_failed` | Gauge | 10.2 | Number of attachments failed to sync on secondary | url | -| `geo_last_event_id` | Gauge | 10.2 | Database ID of the latest event log entry on the primary | url | -| `geo_last_event_timestamp` | Gauge | 10.2 | UNIX timestamp of the latest event log entry on the primary | url | -| `geo_cursor_last_event_id` | Gauge | 10.2 | Last database ID of the event log processed by the secondary | url | -| `geo_cursor_last_event_timestamp` | Gauge | 10.2 | Last UNIX timestamp of the event log processed by the secondary | url | -| `geo_status_failed_total` | Counter | 10.2 | Number of times retrieving the status from the Geo Node failed | url | -| `geo_last_successful_status_check_timestamp` | Gauge | 10.2 | Last timestamp when the status was successfully updated | url | -| `geo_lfs_objects_synced_missing_on_primary` | Gauge | 10.7 | Number of LFS objects marked as synced due to the file missing on the primary | url | -| `geo_job_artifacts_synced_missing_on_primary` | Gauge | 10.7 | Number of job artifacts marked as synced due to the file missing on the primary | url | -| `geo_attachments_synced_missing_on_primary` | Gauge | 10.7 | Number of attachments marked as synced due to the file missing on the primary | url | -| `geo_repositories_checksummed_count` | Gauge | 10.7 | Number of repositories checksummed on primary | url | -| `geo_repositories_checksum_failed_count` | Gauge | 10.7 | Number of repositories failed to calculate the checksum on primary | url | -| `geo_wikis_checksummed_count` | Gauge | 10.7 | Number of wikis checksummed on primary | url | -| `geo_wikis_checksum_failed_count` | Gauge | 10.7 | Number of wikis failed to calculate the checksum on primary | url | -| `geo_repositories_verified_count` | Gauge | 10.7 | Number of repositories verified on secondary | url | -| `geo_repositories_verification_failed_count` | Gauge | 10.7 | Number of repositories failed to verify on secondary | url | -| `geo_repositories_checksum_mismatch_count` | Gauge | 10.7 | Number of repositories that checksum mismatch on secondary | url | -| `geo_wikis_verified_count` | Gauge | 10.7 | Number of wikis verified on secondary | url | -| `geo_wikis_verification_failed_count` | Gauge | 10.7 | Number of wikis failed to verify on secondary | url | -| `geo_wikis_checksum_mismatch_count` | Gauge | 10.7 | Number of wikis that checksum mismatch on secondary | url | -| `geo_repositories_checked_count` | Gauge | 11.1 | Number of repositories that have been checked via `git fsck` | url | -| `geo_repositories_checked_failed_count` | Gauge | 11.1 | Number of repositories that have a failure from `git fsck` | url | -| `geo_repositories_retrying_verification_count` | Gauge | 11.2 | Number of repositories verification failures that Geo is actively trying to correct on secondary | url | -| `geo_wikis_retrying_verification_count` | Gauge | 11.2 | Number of wikis verification failures that Geo is actively trying to correct on secondary | url | +| `geo_db_replication_lag_seconds` | Gauge | 10.2 | Database replication lag (seconds) | `url` | +| `geo_repositories` | Gauge | 10.2 | Total number of repositories available on primary | `url` | +| `geo_repositories_synced` | Gauge | 10.2 | Number of repositories synced on secondary | `url` | +| `geo_repositories_failed` | Gauge | 10.2 | Number of repositories failed to sync on secondary | `url` | +| `geo_lfs_objects` | Gauge | 10.2 | Total number of LFS objects available on primary | `url` | +| `geo_lfs_objects_synced` | Gauge | 10.2 | Number of LFS objects synced on secondary | `url` | +| `geo_lfs_objects_failed` | Gauge | 10.2 | Number of LFS objects failed to sync on secondary | `url` | +| `geo_attachments` | Gauge | 10.2 | Total number of file attachments available on primary | `url` | +| `geo_attachments_synced` | Gauge | 10.2 | Number of attachments synced on secondary | `url` | +| `geo_attachments_failed` | Gauge | 10.2 | Number of attachments failed to sync on secondary | `url` | +| `geo_last_event_id` | Gauge | 10.2 | Database ID of the latest event log entry on the primary | `url` | +| `geo_last_event_timestamp` | Gauge | 10.2 | UNIX timestamp of the latest event log entry on the primary | `url` | +| `geo_cursor_last_event_id` | Gauge | 10.2 | Last database ID of the event log processed by the secondary | `url` | +| `geo_cursor_last_event_timestamp` | Gauge | 10.2 | Last UNIX timestamp of the event log processed by the secondary | `url` | +| `geo_status_failed_total` | Counter | 10.2 | Number of times retrieving the status from the Geo Node failed | `url` | +| `geo_last_successful_status_check_timestamp` | Gauge | 10.2 | Last timestamp when the status was successfully updated | `url` | +| `geo_lfs_objects_synced_missing_on_primary` | Gauge | 10.7 | Number of LFS objects marked as synced due to the file missing on the primary | `url` | +| `geo_job_artifacts_synced_missing_on_primary` | Gauge | 10.7 | Number of job artifacts marked as synced due to the file missing on the primary | `url` | +| `geo_attachments_synced_missing_on_primary` | Gauge | 10.7 | Number of attachments marked as synced due to the file missing on the primary | `url` | +| `geo_repositories_checksummed_count` | Gauge | 10.7 | Number of repositories checksummed on primary | `url` | +| `geo_repositories_checksum_failed_count` | Gauge | 10.7 | Number of repositories failed to calculate the checksum on primary | `url` | +| `geo_wikis_checksummed_count` | Gauge | 10.7 | Number of wikis checksummed on primary | `url` | +| `geo_wikis_checksum_failed_count` | Gauge | 10.7 | Number of wikis failed to calculate the checksum on primary | `url` | +| `geo_repositories_verified_count` | Gauge | 10.7 | Number of repositories verified on secondary | `url` | +| `geo_repositories_verification_failed_count` | Gauge | 10.7 | Number of repositories failed to verify on secondary | `url` | +| `geo_repositories_checksum_mismatch_count` | Gauge | 10.7 | Number of repositories that checksum mismatch on secondary | `url` | +| `geo_wikis_verified_count` | Gauge | 10.7 | Number of wikis verified on secondary | `url` | +| `geo_wikis_verification_failed_count` | Gauge | 10.7 | Number of wikis failed to verify on secondary | `url` | +| `geo_wikis_checksum_mismatch_count` | Gauge | 10.7 | Number of wikis that checksum mismatch on secondary | `url` | +| `geo_repositories_checked_count` | Gauge | 11.1 | Number of repositories that have been checked via `git fsck` | `url` | +| `geo_repositories_checked_failed_count` | Gauge | 11.1 | Number of repositories that have a failure from `git fsck` | `url` | +| `geo_repositories_retrying_verification_count` | Gauge | 11.2 | Number of repositories verification failures that Geo is actively trying to correct on secondary | `url` | +| `geo_wikis_retrying_verification_count` | Gauge | 11.2 | Number of wikis verification failures that Geo is actively trying to correct on secondary | `url` | | `global_search_bulk_cron_queue_size` | Gauge | 12.10 | Number of database records waiting to be synchronized to Elasticsearch | | +| `package_files_count` | Gauge | 13.0 | Number of package files on primary | `url` | +| `package_files_checksummed_count` | Gauge | 13.0 | Number of package files checksummed on primary | `url` | +| `package_files_checksum_failed_count` | Gauge | 13.0 | Number of package files failed to calculate the checksum on primary ## Database load balancing metrics **(PREMIUM ONLY)** @@ -168,17 +178,18 @@ The following metrics are available: Some basic Ruby runtime metrics are available: -| Metric | Type | Since | Description | -|:------------------------------------ |:--------- |:----- |:----------- | -| `ruby_gc_duration_seconds` | Counter | 11.1 | Time spent by Ruby in GC | -| `ruby_gc_stat_...` | Gauge | 11.1 | Various metrics from [GC.stat](https://ruby-doc.org/core-2.6.5/GC.html#method-c-stat) | -| `ruby_file_descriptors` | Gauge | 11.1 | File descriptors per process | -| `ruby_memory_bytes` | Gauge | 11.1 | Memory usage by process | -| `ruby_sampler_duration_seconds` | Counter | 11.1 | Time spent collecting stats | -| `ruby_process_cpu_seconds_total` | Gauge | 12.0 | Total amount of CPU time per process | -| `ruby_process_max_fds` | Gauge | 12.0 | Maximum number of open file descriptors per process | -| `ruby_process_resident_memory_bytes` | Gauge | 12.0 | Memory usage by process | -| `ruby_process_start_time_seconds` | Gauge | 12.0 | UNIX timestamp of process start time | +| Metric | Type | Since | Description | +|:---------------------------------------- |:--------- |:----- |:----------- | +| `ruby_gc_duration_seconds` | Counter | 11.1 | Time spent by Ruby in GC | +| `ruby_gc_stat_...` | Gauge | 11.1 | Various metrics from [GC.stat](https://ruby-doc.org/core-2.6.5/GC.html#method-c-stat) | +| `ruby_file_descriptors` | Gauge | 11.1 | File descriptors per process | +| `ruby_sampler_duration_seconds` | Counter | 11.1 | Time spent collecting stats | +| `ruby_process_cpu_seconds_total` | Gauge | 12.0 | Total amount of CPU time per process | +| `ruby_process_max_fds` | Gauge | 12.0 | Maximum number of open file descriptors per process | +| `ruby_process_resident_memory_bytes` | Gauge | 12.0 | Memory usage by process (RSS/Resident Set Size) | +| `ruby_process_unique_memory_bytes` | Gauge | 13.0 | Memory usage by process (USS/Unique Set Size) | +| `ruby_process_proportional_memory_bytes` | Gauge | 13.0 | Memory usage by process (PSS/Proportional Set Size) | +| `ruby_process_start_time_seconds` | Gauge | 12.0 | UNIX timestamp of process start time | ## Unicorn Metrics diff --git a/doc/administration/monitoring/prometheus/index.md b/doc/administration/monitoring/prometheus/index.md index 03d86012cc7..cb93aca6e4e 100644 --- a/doc/administration/monitoring/prometheus/index.md +++ b/doc/administration/monitoring/prometheus/index.md @@ -1,3 +1,9 @@ +--- +stage: Monitor +group: APM +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + # Monitoring GitLab with Prometheus > **Notes:** @@ -49,7 +55,7 @@ To disable Prometheus and all of its exporters, as well as any added in the futu ### Changing the port and address Prometheus listens on NOTE: **Note:** -The following change was added in [GitLab Omnibus 8.17](https://gitlab.com/gitlab-org/omnibus-gitlab/-/merge_requests/1261). Although possible, +The following change was added in [Omnibus GitLab 8.17](https://gitlab.com/gitlab-org/omnibus-gitlab/-/merge_requests/1261). Although possible, it's not recommended to change the port Prometheus listens on, as this might affect or conflict with other services running on the GitLab server. Proceed at your own risk. @@ -80,7 +86,7 @@ To change the address/port that Prometheus listens on: ### Adding custom scrape configs -You can configure additional scrape targets for the GitLab Omnibus-bundled +You can configure additional scrape targets for the Omnibus GitLab-bundled Prometheus by editing `prometheus['scrape_configs']` in `/etc/gitlab/gitlab.rb` using the [Prometheus scrape target configuration](https://prometheus.io/docs/prometheus/latest/configuration/configuration/#%3Cscrape_config%3E) syntax. @@ -108,7 +114,7 @@ prometheus['scrape_configs'] = [ NOTE: **Note:** Prometheus and most exporters don't support authentication. We don't recommend exposing them outside the local network. -A few configuration changes are required to allow GitLab to be monitored by an external Prometheus server. External servers are recommended for highly available deployments of GitLab with multiple nodes. +A few configuration changes are required to allow GitLab to be monitored by an external Prometheus server. External servers are recommended for [GitLab deployments with multiple nodes](../../reference_architectures/index.md). To use an external Prometheus server: diff --git a/doc/administration/monitoring/prometheus/node_exporter.md b/doc/administration/monitoring/prometheus/node_exporter.md index d75b04f1ccd..357303ee4e1 100644 --- a/doc/administration/monitoring/prometheus/node_exporter.md +++ b/doc/administration/monitoring/prometheus/node_exporter.md @@ -1,3 +1,9 @@ +--- +stage: Monitor +group: APM +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + # Node exporter >**Note:** diff --git a/doc/administration/monitoring/prometheus/pgbouncer_exporter.md b/doc/administration/monitoring/prometheus/pgbouncer_exporter.md index ba8e464efcb..92ba2d9bb52 100644 --- a/doc/administration/monitoring/prometheus/pgbouncer_exporter.md +++ b/doc/administration/monitoring/prometheus/pgbouncer_exporter.md @@ -1,3 +1,9 @@ +--- +stage: Monitor +group: APM +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + # PgBouncer exporter >**Note:** diff --git a/doc/administration/monitoring/prometheus/postgres_exporter.md b/doc/administration/monitoring/prometheus/postgres_exporter.md index 853e3837184..77ca502b21d 100644 --- a/doc/administration/monitoring/prometheus/postgres_exporter.md +++ b/doc/administration/monitoring/prometheus/postgres_exporter.md @@ -1,3 +1,9 @@ +--- +stage: Monitor +group: APM +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + # PostgreSQL Server Exporter >**Note:** diff --git a/doc/administration/monitoring/prometheus/redis_exporter.md b/doc/administration/monitoring/prometheus/redis_exporter.md index 76f4add0c1b..bef87400f5a 100644 --- a/doc/administration/monitoring/prometheus/redis_exporter.md +++ b/doc/administration/monitoring/prometheus/redis_exporter.md @@ -1,3 +1,9 @@ +--- +stage: Monitor +group: APM +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + # Redis exporter >**Note:** diff --git a/doc/administration/monitoring/prometheus/registry_exporter.md b/doc/administration/monitoring/prometheus/registry_exporter.md index 2e2440389ed..3d28b26b685 100644 --- a/doc/administration/monitoring/prometheus/registry_exporter.md +++ b/doc/administration/monitoring/prometheus/registry_exporter.md @@ -1,3 +1,9 @@ +--- +stage: Monitor +group: APM +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + # Registry exporter > [Introduced](https://gitlab.com/gitlab-org/omnibus-gitlab/-/merge_requests/2884) in GitLab 11.9. diff --git a/doc/administration/object_storage.md b/doc/administration/object_storage.md index 1b0c7f8a907..c4f9b672923 100644 --- a/doc/administration/object_storage.md +++ b/doc/administration/object_storage.md @@ -5,7 +5,7 @@ type: reference # Object Storage GitLab supports using an object storage service for holding numerous types of data. -In a high availability setup, it's recommended over [NFS](high_availability/nfs.md) and +It's recommended over NFS and in general it's better in larger setups as object storage is typically much more performant, reliable, and scalable. @@ -37,8 +37,8 @@ For configuring GitLab to use Object Storage refer to the following guides: ### Other alternatives to filesystem storage -If you're working to [scale out](scaling/index.md) your GitLab implementation, -or add [fault tolerance and redundancy](high_availability/README.md) you may be +If you're working to [scale out](reference_architectures/index.md) your GitLab implementation, +or add fault tolerance and redundancy, you may be looking at removing dependencies on block or network filesystems. See the following guides and [note that Pages requires disk storage](#gitlab-pages-requires-nfs): @@ -77,7 +77,7 @@ with the Fog library that GitLab uses. Symptoms include: ### GitLab Pages requires NFS -If you're working to add more GitLab servers for [scaling or fault tolerance](scaling/index.md) +If you're working to add more GitLab servers for [scaling or fault tolerance](reference_architectures/index.md) and one of your requirements is [GitLab Pages](../user/project/pages/index.md) this currently requires NFS. There is [work in progress](https://gitlab.com/gitlab-org/gitlab-pages/issues/196) to remove this dependency. In the future, GitLab Pages may use @@ -98,9 +98,9 @@ object storage back end, like when Git clients request large files via LFS or wh downloading CI artifacts and logs. When the files are stored on local block storage or NFS, GitLab has to act as a proxy. -This is not the default behaviour with object storage. +This is not the default behavior with object storage. -The `proxy_download` setting controls this behaviour: the default is generally `false`. +The `proxy_download` setting controls this behavior: the default is generally `false`. Verify this in the documentation for each use case. Set it to `true` so that GitLab proxies the files. diff --git a/doc/administration/operations/extra_sidekiq_processes.md b/doc/administration/operations/extra_sidekiq_processes.md index 1c92a429982..8f54b82c325 100644 --- a/doc/administration/operations/extra_sidekiq_processes.md +++ b/doc/administration/operations/extra_sidekiq_processes.md @@ -1,13 +1,13 @@ -# Running multiple Sidekiq processes **(CORE ONLY)** - -NOTE: **Note:** -The information in this page applies only to Omnibus GitLab. +# Run multiple Sidekiq processes **(CORE ONLY)** GitLab allows you to start multiple Sidekiq processes. These processes can be used to consume a dedicated set of queues. This can be used to ensure certain queues always have dedicated workers, no matter the number of jobs that need to be processed. +NOTE: **Note:** +The information in this page applies only to Omnibus GitLab. + ## Available Sidekiq queues For a list of the existing Sidekiq queues, check the following files: @@ -18,28 +18,27 @@ For a list of the existing Sidekiq queues, check the following files: Each entry in the above files represents a queue on which Sidekiq processes can be started. -## Starting multiple processes +## Start multiple processes -To start multiple Sidekiq processes, you must enable `sidekiq-cluster`: +> - [Introduced](https://gitlab.com/gitlab-org/omnibus-gitlab/-/merge_requests/4006) in GitLab 12.10, starting multiple processes with Sidekiq cluster. +> - [Sidekiq cluster moved](https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/181) to GitLab [Core](https://about.gitlab.com/pricing/#self-managed) in GitLab 12.10. +> - [Sidekiq cluster became default](https://gitlab.com/gitlab-org/omnibus-gitlab/-/merge_requests/4140) in GitLab 13.0. -1. Edit `/etc/gitlab/gitlab.rb` and add: +To start multiple processes: - ```ruby - sidekiq_cluster['enable'] = true - ``` - -1. You will then need to specify how many additional processes to create via `sidekiq-cluster` - and which queue they should handle via the `sidekiq_cluster['queue_groups']` - array setting. Each item in the array equates to one additional Sidekiq +1. Using the `sidekiq['queue_groups']` array setting, specify how many processes to + create using `sidekiq-cluster` and which queue they should handle. + Each item in the array equates to one additional Sidekiq process, and values in each item determine the queues it works on. - For example, the following setting adds additional Sidekiq processes to two - queues, one to `elastic_indexer` and one to `mailers`: + For example, the following setting creates three Sidekiq processes, one to run on + `elastic_indexer`, one to run on `mailers`, and one process running all on queues: ```ruby - sidekiq_cluster['queue_groups'] = [ + sidekiq['queue_groups'] = [ "elastic_indexer", - "mailers" + "mailers", + "*" ] ``` @@ -47,9 +46,10 @@ To start multiple Sidekiq processes, you must enable `sidekiq-cluster`: queue names to its item delimited by commas. For example: ```ruby - sidekiq_cluster['queue_groups'] = [ + sidekiq['queue_groups'] = [ "elastic_indexer, elastic_commit_indexer", - "mailers" + "mailers", + "*" ] ``` @@ -58,7 +58,7 @@ To start multiple Sidekiq processes, you must enable `sidekiq-cluster`: processes, each handling all queues: ```ruby - sidekiq_cluster['queue_groups'] = [ + sidekiq['queue_groups'] = [ "*", "*" ] @@ -67,27 +67,35 @@ To start multiple Sidekiq processes, you must enable `sidekiq-cluster`: `*` cannot be combined with concrete queue names - `*, mailers` will just handle the `mailers` queue. + When `sidekiq-cluster` is only running on a single node, make sure that at least + one process is running on all queues using `*`. This means a process will + automatically pick up jobs in queues created in the future. + + If `sidekiq-cluster` is running on more than one node, you can also use + [`--negate`](#negate-settings) and list all the queues that are already being + processed. + 1. Save the file and reconfigure GitLab for the changes to take effect: ```shell sudo gitlab-ctl reconfigure ``` -Once the extra Sidekiq processes are added, you can visit the -**Admin Area > Monitoring > Background Jobs** (`/admin/background_jobs`) in GitLab. +After the extra Sidekiq processes are added, navigate to +**{admin}** **Admin Area > Monitoring > Background Jobs** (`/admin/background_jobs`) in GitLab.  -## Negating settings +## Negate settings To have the additional Sidekiq processes work on every queue **except** the ones you list: -1. After you follow the steps for [starting extra processes](#starting-multiple-processes), +1. After you follow the steps for [starting extra processes](#start-multiple-processes), edit `/etc/gitlab/gitlab.rb` and add: ```ruby - sidekiq_cluster['negate'] = true + sidekiq['negate'] = true ``` 1. Save the file and reconfigure GitLab for the changes to take effect: @@ -177,9 +185,9 @@ entire queue group selects all queues. In `/etc/gitlab/gitlab.rb`: ```ruby -sidekiq_cluster['enable'] = true -sidekiq_cluster['experimental_queue_selector'] = true -sidekiq_cluster['queue_groups'] = [ +sidekiq['enable'] = true +sidekiq['experimental_queue_selector'] = true +sidekiq['queue_groups'] = [ # Run all non-CPU-bound queues that are high urgency 'resource_boundary!=cpu&urgency=high', # Run all continuous integration and pages queues that are not high urgency @@ -189,35 +197,31 @@ sidekiq_cluster['queue_groups'] = [ ] ``` -### Using Sidekiq cluster by default (experimental) - -> [Introduced](https://gitlab.com/gitlab-org/omnibus-gitlab/-/merge_requests/4006) in GitLab 12.10. +### Disable Sidekiq cluster CAUTION: **Warning:** -This feature is experimental. +Sidekiq cluster is [scheduled](https://gitlab.com/gitlab-com/gl-infra/scalability/-/issues/240) +to be the only way to start Sidekiq in GitLab 14.0. -We're moving [Sidekiq cluster to -core](https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/181) and -plan to make it the default way of starting Sidekiq. - -Set the following to start Sidekiq (cluster) -process for handling for all queues (`/etc/gitlab/gitlab.rb`): +By default, the Sidekiq service will run `sidekiq-cluster`. To disable this behavior, +add the following to the Sidekiq configuration: ```ruby sidekiq['enable'] = true -sidekiq['cluster'] = true +sidekiq['cluster'] = false ``` -All of the aforementioned configuration options for `sidekiq_cluster` -are also available. By default, they will be configured as follows: +All of the aforementioned configuration options for `sidekiq` +are available. By default, they will be configured as follows: ```ruby sidekiq['experimental_queue_selector'] = false sidekiq['interval'] = nil -sidekiq['max_concurrency'] = nil +sidekiq['max_concurrency'] = 50 sidekiq['min_concurrency'] = nil sidekiq['negate'] = false sidekiq['queue_groups'] = ['*'] +sidekiq['shutdown_timeout'] = 25 ``` `sidekiq_cluster` must be disabled if you decide to configure the @@ -231,7 +235,7 @@ setting `sidekiq['cluster'] = true`. When using this feature, the service called `sidekiq` will now be running `sidekiq-cluster`. -The [concurrency](#managing-concurrency) and other options configured +The [concurrency](#manage-concurrency) and other options configured for Sidekiq will be respected. By default, logs for `sidekiq-cluster` go to `/var/log/gitlab/sidekiq` @@ -246,9 +250,9 @@ use all of its resources to perform those operations. To set up a separate 1. Edit `/etc/gitlab/gitlab.rb` and add: ```ruby - sidekiq_cluster['enable'] = true - sidekiq_cluster['negate'] = true - sidekiq_cluster['queue_groups'] = [ + sidekiq['enable'] = true + sidekiq['negate'] = true + sidekiq['queue_groups'] = [ "github_import_advance_stage", "github_importer:github_import_import_diff_note", "github_importer:github_import_import_issue", @@ -274,12 +278,12 @@ use all of its resources to perform those operations. To set up a separate ## Number of threads -Each process defined under `sidekiq_cluster` starts with a +Each process defined under `sidekiq` starts with a number of threads that equals the number of queues, plus one spare thread. For example, a process that handles the `process_commit` and `post_receive` queues will use three threads in total. -## Managing concurrency +## Manage concurrency When setting the maximum concurrency, keep in mind this normally should not exceed the number of CPU cores available. The values in the examples @@ -290,29 +294,15 @@ latency and potentially cause client timeouts. See the [Sidekiq documentation about Redis](https://github.com/mperham/sidekiq/wiki/Using-Redis) for more details. -### When running a single Sidekiq process (default) - -1. Edit `/etc/gitlab/gitlab.rb` and add: - - ```ruby - sidekiq['concurrency'] = 25 - ``` +### When running Sidekiq cluster (default) -1. Save the file and reconfigure GitLab for the changes to take effect: - - ```shell - sudo gitlab-ctl reconfigure - ``` - -This will set the concurrency (number of threads) for the Sidekiq process. - -### When running Sidekiq cluster +Running Sidekiq cluster is the default in GitLab 13.0 and later. 1. Edit `/etc/gitlab/gitlab.rb` and add: ```ruby - sidekiq_cluster['min_concurrency'] = 15 - sidekiq_cluster['max_concurrency'] = 25 + sidekiq['min_concurrency'] = 15 + sidekiq['max_concurrency'] = 25 ``` 1. Save the file and reconfigure GitLab for the changes to take effect: @@ -337,21 +327,44 @@ regardless of the number of queues. When `min_concurrency` is greater than `max_concurrency`, it is treated as being equal to `max_concurrency`. -## Modifying the check interval +### When running a single Sidekiq process + +Running a single Sidekiq process is the default in GitLab 12.10 and earlier. + +CAUTION: **Warning:** +Running Sidekiq directly is scheduled to be removed in GitLab +[14.0](https://gitlab.com/gitlab-com/gl-infra/scalability/-/issues/240). + +1. Edit `/etc/gitlab/gitlab.rb` and add: + + ```ruby + sidekiq['cluster'] = false + sidekiq['concurrency'] = 25 + ``` + +1. Save the file and reconfigure GitLab for the changes to take effect: + + ```shell + sudo gitlab-ctl reconfigure + ``` + +This will set the concurrency (number of threads) for the Sidekiq process. + +## Modify the check interval To modify the check interval for the additional Sidekiq processes: 1. Edit `/etc/gitlab/gitlab.rb` and add: ```ruby - sidekiq_cluster['interval'] = 5 + sidekiq['interval'] = 5 ``` 1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. This tells the additional processes how often to check for enqueued jobs. -## Troubleshooting using the CLI +## Troubleshoot using the CLI CAUTION: **Warning:** It's recommended to use `/etc/gitlab/gitlab.rb` to configure the Sidekiq processes. @@ -399,7 +412,7 @@ you'd use the following: /opt/gitlab/embedded/service/gitlab-rails/bin/sidekiq-cluster process_commit,post_receive gitlab_shell ``` -### Monitoring the `sidekiq-cluster` command +### Monitor the `sidekiq-cluster` command The `sidekiq-cluster` command will not terminate once it has started the desired amount of Sidekiq processes. Instead, the process will continue running and @@ -412,7 +425,7 @@ processes will terminate themselves after a few seconds. This ensures you don't end up with zombie Sidekiq processes. All of this makes monitoring the processes fairly easy. Simply hook up -`sidekiq-cluster` to your supervisor of choice (e.g. runit) and you're good to +`sidekiq-cluster` to your supervisor of choice (for example, runit) and you're good to go. If a child process died the `sidekiq-cluster` command will signal all remaining diff --git a/doc/administration/operations/fast_ssh_key_lookup.md b/doc/administration/operations/fast_ssh_key_lookup.md index 2d1e1c5bda8..6759c3f265d 100644 --- a/doc/administration/operations/fast_ssh_key_lookup.md +++ b/doc/administration/operations/fast_ssh_key_lookup.md @@ -68,11 +68,17 @@ sudo service sshd reload ``` Confirm that SSH is working by removing your user's SSH key in the UI, adding a -new one, and attempting to pull a repo. +new one, and attempting to pull a repository. NOTE: **Note:** For Omnibus Docker, `AuthorizedKeysCommand` is setup by default in GitLab 11.11 and later. +NOTE: **Note:** For Installations from source, the command would be located at +`/home/git/gitlab-shell/bin/gitlab-shell-authorized-keys-check` if [the install from source](../../install/installation.md#install-gitlab-shell) instructions were followed. +You might want to consider creating a wrapper script somewhere else since this command needs to be +owned by `root` and not be writable by group or others. You could also consider changing the ownership of this command +as required, but that might require temporary ownership changes during `gitlab-shell` upgrades. + CAUTION: **Caution:** Do not disable writes until SSH is confirmed to be working perfectly, because the file will quickly become out-of-date. @@ -87,7 +93,7 @@ installation.  Again, confirm that SSH is working by removing your user's SSH key in the UI, -adding a new one, and attempting to pull a repo. +adding a new one, and attempting to pull a repository. Then you can backup and delete your `authorized_keys` file for best performance. diff --git a/doc/administration/operations/index.md b/doc/administration/operations/index.md index c27832e67ef..45b8e5ad448 100644 --- a/doc/administration/operations/index.md +++ b/doc/administration/operations/index.md @@ -12,6 +12,7 @@ Keep your GitLab instance up and running smoothly. - [Sidekiq MemoryKiller](sidekiq_memory_killer.md): Configure Sidekiq MemoryKiller to restart Sidekiq. - [Multiple Sidekiq processes](extra_sidekiq_processes.md): Configure multiple Sidekiq processes to ensure certain queues always have dedicated workers, no matter the number of jobs that need to be processed. **(CORE ONLY)** +- [Puma](puma.md): Understand Puma and puma-worker-killer. - [Unicorn](unicorn.md): Understand Unicorn and unicorn-worker-killer. - Speed up SSH operations by [Authorizing SSH users via a fast, indexed lookup to the GitLab database](fast_ssh_key_lookup.md), and/or diff --git a/doc/administration/operations/puma.md b/doc/administration/operations/puma.md index 6f252a7d76e..af559cf00e9 100644 --- a/doc/administration/operations/puma.md +++ b/doc/administration/operations/puma.md @@ -3,7 +3,9 @@ ## Puma As of GitLab 12.9, [Puma](https://github.com/puma/puma) has replaced [Unicorn](https://yhbt.net/unicorn/). -as the default web server. +as the default web server. Starting with 13.0, both all-in-one package based +installations as well as Helm chart based installations will run Puma instead of +Unicorn unless explicitly specified not to. ## Why switch to Puma? @@ -14,6 +16,12 @@ Most Rails applications requests normally include a proportion of I/O wait time. During I/O wait time MRI Ruby will release the GVL (Global VM Lock) to other threads. Multi-threaded Puma can therefore still serve more requests than a single process. +## Configuring Puma to replace Unicorn + +If you are currently running Unicorn and would like to switch to Puma, server configuration +will _not_ carry over automatically. For details on matching Unicorn configuration settings with +the Puma equivalent, where applicable, see [Converting Unicorn settings to Puma](https://docs.gitlab.com/omnibus/settings/puma.html#converting-unicorn-settings-to-puma). + ## Performance caveat when using Puma with Rugged For deployments where NFS is used to store Git repository, we allow GitLab to use diff --git a/doc/administration/operations/sidekiq_memory_killer.md b/doc/administration/operations/sidekiq_memory_killer.md index 6438dbb9dab..fdccfacc8a9 100644 --- a/doc/administration/operations/sidekiq_memory_killer.md +++ b/doc/administration/operations/sidekiq_memory_killer.md @@ -2,13 +2,13 @@ The GitLab Rails application code suffers from memory leaks. For web requests this problem is made manageable using -[`unicorn-worker-killer`](https://github.com/kzk/unicorn-worker-killer) which -restarts Unicorn worker processes in between requests when needed. The Sidekiq +[`puma-worker-killer`](https://github.com/schneems/puma_worker_killer) which +restarts Puma worker processes if it exceeds a memory limit. The Sidekiq MemoryKiller applies the same approach to the Sidekiq processes used by GitLab to process background jobs. -Unlike unicorn-worker-killer, which is enabled by default for all GitLab -installations since GitLab 6.4, the Sidekiq MemoryKiller is enabled by default +Unlike puma-worker-killer, which is enabled by default for all GitLab +installations since GitLab 13.0, the Sidekiq MemoryKiller is enabled by default _only_ for Omnibus packages. The reason for this is that the MemoryKiller relies on runit to restart Sidekiq after a memory-induced shutdown and GitLab installations from source do not all use runit or an equivalent. diff --git a/doc/administration/operations/ssh_certificates.md b/doc/administration/operations/ssh_certificates.md index eaf0e4ab284..b652f282b7b 100644 --- a/doc/administration/operations/ssh_certificates.md +++ b/doc/administration/operations/ssh_certificates.md @@ -50,7 +50,7 @@ the GitLab server itself, but your setup may vary. If the CA is only used for GitLab consider putting this in the `Match User git` section (described below). -The SSH certificates being issued by that CA **MUST** have a "key id" +The SSH certificates being issued by that CA **MUST** have a "key ID" corresponding to that user's username on GitLab, e.g. (some output omitted for brevity): @@ -77,7 +77,7 @@ own `AuthorizedPrincipalsCommand` to do that mapping instead of using our provided default. The important part is that the `AuthorizedPrincipalsCommand` must be -able to map from the "key id" to a GitLab username in some way, the +able to map from the "key ID" to a GitLab username in some way, the default command we ship assumes there's a 1=1 mapping between the two, since the whole point of this is to allow us to extract a GitLab username from the key itself, instead of relying on something like the @@ -122,7 +122,7 @@ into multiple lines of `authorized_keys` output, as described in the Normally when using the `AuthorizedKeysCommand` with OpenSSH the principal is some "group" that's allowed to log into that server. However with GitLab it's only used to appease OpenSSH's -requirement for it, we effectively only care about the "key id" being +requirement for it, we effectively only care about the "key ID" being correct. Once that's extracted GitLab will enforce its own ACLs for that user (e.g. what projects the user can access). diff --git a/doc/administration/operations/unicorn.md b/doc/administration/operations/unicorn.md index bb817e71f5a..50481482f4c 100644 --- a/doc/administration/operations/unicorn.md +++ b/doc/administration/operations/unicorn.md @@ -1,5 +1,9 @@ # Understanding Unicorn and unicorn-worker-killer +NOTE: **Note:** +Starting with GitLab 13.0, Puma is the default web server used in GitLab +all-in-one package based installations as well as GitLab Helm chart deployments. + ## Unicorn GitLab uses [Unicorn](https://yhbt.net/unicorn/), a pre-forking Ruby web diff --git a/doc/administration/packages/container_registry.md b/doc/administration/packages/container_registry.md index aaf1ca29084..503e5fb4a2a 100644 --- a/doc/administration/packages/container_registry.md +++ b/doc/administration/packages/container_registry.md @@ -18,16 +18,22 @@ You can read more about the Docker Registry at **Omnibus GitLab installations** -If you are using the Omnibus GitLab built in [Let's Encrypt integration](https://docs.gitlab.com/omnibus/settings/ssl.html#lets-encrypt-integration), as of GitLab 12.5, the Container Registry will be automatically enabled on port 5050 of the default domain. +If you installed GitLab by using the Omnibus installation package, the Container Registry +may or may not be available by default. -If you would like to use a separate domain, all you have to do is configure the domain name under which the Container -Registry will listen to. Read -[#container-registry-domain-configuration](#container-registry-domain-configuration) -and pick one of the two options that fits your case. +The Container Registry is automatically enabled and available on your GitLab domain, port 5050 if: + +- You're using the built-in [Let's Encrypt integration](https://docs.gitlab.com/omnibus/settings/ssl.html#lets-encrypt-integration), and +- You're using GitLab 12.5 or later. + +Otherwise, the Container Registry is not enabled. To enable it: + +- You can configure it for your [GitLab domain](#configure-container-registry-under-an-existing-gitlab-domain), or +- You can configure it for [a different domain](#configure-container-registry-under-its-own-domain). NOTE: **Note:** -The container registry works under HTTPS by default. Using HTTP is possible -but not recommended and out of the scope of this document. +The Container Registry works under HTTPS by default. You can use HTTP +but it's not recommended and is out of the scope of this document. Read the [insecure Registry documentation](https://docs.docker.com/registry/insecure/) if you want to implement this. @@ -427,7 +433,7 @@ NOTE: **Note:** By default, users accessing a registry configured with a remote backend are redirected to the default backend for the storage driver. For example, registries can be configured using the `s3` storage driver, which redirects requests to a remote S3 bucket to alleviate load on the GitLab server. -However, this behaviour is undesirable for registries used by internal hosts that usually can't access public servers. To disable redirects, set the `disable` flag to true as follows. This makes all traffic to always go through the Registry service. This results in improved security (less surface attack as the storage backend is not publicly accessible), but worse performance (all traffic is redirected via the service). +However, this behavior is undesirable for registries used by internal hosts that usually can't access public servers. To disable redirects, set the `disable` flag to true as follows. This makes all traffic to always go through the Registry service. This results in improved security (less surface attack as the storage backend is not publicly accessible), but worse performance (all traffic is redirected via the service). **Omnibus GitLab installations** @@ -454,7 +460,7 @@ However, this behaviour is undesirable for registries used by internal hosts tha 1. Add the `redirect` flag to your registry configuration YML file: - ```yml + ```yaml storage: s3: accesskey: 'AKIAKIAKI' @@ -520,7 +526,7 @@ on how to achieve that. NOTE: **Note:** In using an external container registry, some features associated with the -container registry may be unavailable or have [inherant risks](./../../user/packages/container_registry/index.md#use-with-external-container-registries) +container registry may be unavailable or have [inherent risks](./../../user/packages/container_registry/index.md#use-with-external-container-registries) **Omnibus GitLab** @@ -650,13 +656,13 @@ notifications: NOTE: **Note:** The garbage collection tools are only available when you've installed GitLab -via an Omnibus package or the cloud native chart. +via an Omnibus package or the [cloud native chart](https://docs.gitlab.com/charts/charts/registry/#garbage-collection). DANGER: **Danger:** By running the built-in garbage collection command, it will cause downtime to -the Container Registry. Running this command on an instance in an HA environment -while one of your other instances is still writing to the Registry storage, -will remove referenced manifests. To avoid that, make sure Registry is set to +the Container Registry. If you run this command on an instance in an environment +where one of your other instances is still writing to the Registry storage, +referenced manifests will be removed. To avoid that, make sure Registry is set to [read-only mode](#performing-garbage-collection-without-downtime) before proceeding. Container Registry can use considerable amounts of disk space. To clear up @@ -710,7 +716,7 @@ built-in command: If you did not change the default location of the configuration file, run: -```sh +```shell sudo gitlab-ctl registry-garbage-collect ``` @@ -719,7 +725,7 @@ layers you have stored. If you changed the location of the Container Registry `config.yml`: -```sh +```shell sudo gitlab-ctl registry-garbage-collect /path/to/config.yml ``` @@ -743,7 +749,7 @@ referenced by the registry tag. The `registry-garbage-collect` command supports `-m` switch to allow you to remove all unreferenced manifests and layers that are not directly accessible via `tag`: -```sh +```shell sudo gitlab-ctl registry-garbage-collect -m ``` @@ -781,7 +787,7 @@ To enable the read-only mode: 1. Save and reconfigure GitLab: - ```sh + ```shell sudo gitlab-ctl reconfigure ``` @@ -789,7 +795,7 @@ To enable the read-only mode: 1. Next, trigger one of the garbage collect commands: - ```sh + ```shell # Recycling unused tags sudo /opt/gitlab/embedded/bin/registry garbage-collect /var/opt/gitlab/registry/config.yml @@ -816,7 +822,7 @@ To enable the read-only mode: 1. Save and reconfigure GitLab: - ```sh + ```shell sudo gitlab-ctl reconfigure ``` diff --git a/doc/administration/pages/index.md b/doc/administration/pages/index.md index 84133205bd3..21d13be47bd 100644 --- a/doc/administration/pages/index.md +++ b/doc/administration/pages/index.md @@ -80,7 +80,7 @@ added `gitlab.io` [in 2016](https://gitlab.com/gitlab-com/infrastructure/issues/ ### DNS configuration GitLab Pages expect to run on their own virtual host. In your DNS server/provider -you need to add a [wildcard DNS A record][wiki-wildcard-dns] pointing to the +you need to add a [wildcard DNS A record](https://en.wikipedia.org/wiki/Wildcard_DNS_record) pointing to the host that GitLab runs. For example, an entry would look like this: ```plaintext @@ -95,8 +95,6 @@ IPv6 address. If you don't have IPv6, you can omit the AAAA record. NOTE: **Note:** You should not use the GitLab domain to serve user pages. For more information see the [security section](#security). -[wiki-wildcard-dns]: https://en.wikipedia.org/wiki/Wildcard_DNS_record - ## Configuration Depending on your needs, you can set up GitLab Pages in 4 different ways. @@ -354,7 +352,7 @@ This usually results in this error: For installation from source this can be fixed by installing the custom Certificate Authority (CA) in the system certificate store. -For Omnibus, normally this would be fixed by [installing a custom CA in GitLab Omnibus](https://docs.gitlab.com/omnibus/settings/ssl.html#install-custom-public-certificates) +For Omnibus, normally this would be fixed by [installing a custom CA in Omnibus GitLab](https://docs.gitlab.com/omnibus/settings/ssl.html#install-custom-public-certificates) but a [bug](https://gitlab.com/gitlab-org/gitlab/issues/25411) is currently preventing that method from working. Use the following workaround: @@ -365,14 +363,14 @@ that method from working. Use the following workaround: echo -n | openssl s_client -connect gitlab-domain-example.com:443 | sed -ne '/-BEGIN CERTIFICATE-/,/-END CERTIFICATE-/p' | sudo tee --append /opt/gitlab/embedded/ssl/certs/cacert.pem ``` -1. [Restart](../restart_gitlab.md) the GitLab Pages Daemon. For GitLab Omnibus instances: +1. [Restart](../restart_gitlab.md) the GitLab Pages Daemon. For Omnibus GitLab instances: ```shell sudo gitlab-ctl restart gitlab-pages ``` CAUTION: **Caution:** -Some GitLab Omnibus upgrades will revert this workaround and you'll need to apply it again. +Some Omnibus GitLab upgrades will revert this workaround and you'll need to apply it again. ## Activate verbose logging for daemon @@ -457,9 +455,36 @@ You can run the GitLab Pages daemon on a separate server in order to decrease th To configure GitLab Pages on a separate server: +DANGER: **Danger:** +The following procedure includes steps to back up and edit the +`gitlab-secrets.json` file. This file contains secrets that control +database encryption. Proceed with caution. + +1. On the **GitLab server**, to enable Pages, add the following to `/etc/gitlab/gitlab.rb`: + + ```ruby + gitlab_pages['enable'] = true + ``` + +1. Optionally, to enable [access control](#access-control), add the following to `/etc/gitlab/gitlab.rb`: + + ```ruby + gitlab_pages['access_control'] = true + ``` + +1. [Reconfigure the **GitLab server**](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the + changes to take effect. The `gitlab-secrets.json` file is now updated with the + new configuration. + +1. Create a backup of the secrets file on the **GitLab server**: + + ```shell + cp /etc/gitlab/gitlab-secrets.json /etc/gitlab/gitlab-secrets.json.bak + ``` + 1. Set up a new server. This will become the **Pages server**. -1. Create an NFS share on the new server and configure this share to +1. Create an [NFS share](../high_availability/nfs_host_client_setup.md) on the new server and configure this share to allow access from your main **GitLab server**. For this example, we use the default GitLab Pages folder `/var/opt/gitlab/gitlab-rails/shared/pages` as the shared folder on the new server and we will mount it to `/mnt/pages` @@ -474,7 +499,7 @@ To configure GitLab Pages on a separate server: postgresql['enable'] = false redis['enable'] = false prometheus['enable'] = false - unicorn['enable'] = false + puma['enable'] = false sidekiq['enable'] = false gitlab_workhorse['enable'] = false gitaly['enable'] = false @@ -483,6 +508,15 @@ To configure GitLab Pages on a separate server: gitlab_rails['auto_migrate'] = false ``` +1. Create a backup of the secrets file on the **Pages server**: + + ```shell + cp /etc/gitlab/gitlab-secrets.json /etc/gitlab/gitlab-secrets.json.bak + ``` + +1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the **GitLab server** + to the **Pages server**. + 1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. 1. On the **GitLab server**, make the following changes to `/etc/gitlab/gitlab.rb`: @@ -502,61 +536,6 @@ configuring a load balancer to work at the IP level, and so on. If you wish to set up GitLab Pages on multiple servers, perform the above procedure for each Pages server. -### Access control when running GitLab Pages on a separate server - -If you are [running GitLab Pages on a separate server](#running-gitlab-pages-on-a-separate-server), -then you must use the following procedure to configure [access control](#access-control): - -1. On the **GitLab server**, add the following to `/etc/gitlab/gitlab.rb`: - - ```ruby - gitlab_pages['enable'] = true - gitlab_pages['access_control'] = true - ``` - -1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the - changes to take effect. The `gitlab-secrets.json` file is now updated with the - new configuration. - - DANGER: **Danger:** - The `gitlab-secrets.json` file contains secrets that control database encryption. - Do not edit or replace this file on the **GitLab server** or you might - experience permanent data loss. Make a backup copy of this file before proceeding, - as explained in the following steps. - -1. Create a backup of the secrets file on the **GitLab server**: - - ```shell - cp /etc/gitlab/gitlab-secrets.json /etc/gitlab/gitlab-secrets.json.bak - ``` - -1. Create a backup of the secrets file on the **Pages server**: - - ```shell - cp /etc/gitlab/gitlab-secrets.json /etc/gitlab/gitlab-secrets.json.bak - ``` - -1. Disable Pages on the **GitLab server** by setting the following in - `/etc/gitlab/gitlab.rb`: - - ```ruby - gitlab_pages['enable'] = false - ``` - -1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. - -1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the **GitLab server** - to the **Pages server**. - -1. On your **Pages server**, add the following to `/etc/gitlab/gitlab.rb`: - - ```ruby - gitlab_pages['gitlab_server'] = "https://<your-gitlab-server-URL>" - gitlab_pages['access_control'] = true - ``` - -1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. - ## Backup GitLab Pages are part of the [regular backup](../../raketasks/backup_restore.md), so there is no separate backup to configure. diff --git a/doc/administration/pages/source.md b/doc/administration/pages/source.md index 94d2c5420aa..1bb3b86b419 100644 --- a/doc/administration/pages/source.md +++ b/doc/administration/pages/source.md @@ -61,7 +61,7 @@ Before proceeding with the Pages configuration, make sure that: ### DNS configuration GitLab Pages expect to run on their own virtual host. In your DNS server/provider -you need to add a [wildcard DNS A record][wiki-wildcard-dns] pointing to the +you need to add a [wildcard DNS A record](https://en.wikipedia.org/wiki/Wildcard_DNS_record) pointing to the host that GitLab runs. For example, an entry would look like this: ```plaintext @@ -75,8 +75,6 @@ and `192.0.2.1` is the IP address of your GitLab instance. You should not use the GitLab domain to serve user pages. For more information see the [security section](#security). -[wiki-wildcard-dns]: https://en.wikipedia.org/wiki/Wildcard_DNS_record - ## Configuration Depending on your needs, you can set up GitLab Pages in 4 different ways. diff --git a/doc/administration/pseudonymizer.md b/doc/administration/pseudonymizer.md index 36bb446da78..22543a5c743 100644 --- a/doc/administration/pseudonymizer.md +++ b/doc/administration/pseudonymizer.md @@ -1,6 +1,6 @@ # Pseudonymizer **(ULTIMATE)** -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/5532) in [GitLab Ultimate][ee] 11.1. +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/5532) in [GitLab Ultimate](https://about.gitlab.com/pricing/) 11.1. As GitLab's database hosts sensitive information, using it unfiltered for analytics implies high security requirements. To help alleviate this constraint, the Pseudonymizer @@ -101,5 +101,3 @@ This will produce some CSV files that might be very large, so make sure the After the pseudonymizer has run, the output CSV files should be uploaded to the configured object storage and deleted from the local disk. - -[ee]: https://about.gitlab.com/pricing/ diff --git a/doc/administration/raketasks/check.md b/doc/administration/raketasks/check.md index c0edb43cc01..da5caf3966f 100644 --- a/doc/administration/raketasks/check.md +++ b/doc/administration/raketasks/check.md @@ -1,6 +1,8 @@ -# Integrity Check Rake Task +# Integrity check Rake task **(CORE ONLY)** -## Repository Integrity +GitLab provides Rake tasks to check the integrity of various components. + +## Repository integrity Even though Git is very resilient and tries to prevent data integrity issues, there are times when things go wrong. The following Rake tasks intend to @@ -43,7 +45,7 @@ sudo gitlab-rake gitlab:git:fsck sudo -u git -H bundle exec rake gitlab:git:fsck RAILS_ENV=production ``` -## Uploaded Files Integrity +## Uploaded files integrity Various types of files can be uploaded to a GitLab installation by users. These integrity checks can detect missing files. Additionally, for locally @@ -127,11 +129,9 @@ Checking integrity of Uploads Done! ``` -## LDAP Check +## LDAP check The LDAP check Rake task will test the bind DN and password credentials (if configured) and will list a sample of LDAP users. This task is also executed as part of the `gitlab:check` task, but can run independently. See [LDAP Rake Tasks - LDAP Check](ldap.md#check) for details. - -[git-fsck]: https://git-scm.com/docs/git-fsck diff --git a/doc/administration/raketasks/geo.md b/doc/administration/raketasks/geo.md index 707ed1dbee0..71e4f922348 100644 --- a/doc/administration/raketasks/geo.md +++ b/doc/administration/raketasks/geo.md @@ -1,9 +1,11 @@ # Geo Rake Tasks **(PREMIUM ONLY)** +The following Rake tasks are for [Geo installations](../geo/replication/index.md). + ## Git housekeeping There are few tasks you can run to schedule a Git housekeeping to start at the -next repository sync in a **Secondary node**: +next repository sync in a **secondary** node: ### Incremental Repack diff --git a/doc/administration/raketasks/github_import.md b/doc/administration/raketasks/github_import.md index 4c6521f38ec..83a3d2c8884 100644 --- a/doc/administration/raketasks/github_import.md +++ b/doc/administration/raketasks/github_import.md @@ -1,4 +1,4 @@ -# GitHub import +# GitHub import **(CORE ONLY)** > [Introduced]( https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/10308) in GitLab 9.1. @@ -9,7 +9,7 @@ which will become the owner of the project. You can resume an import with the same command. Bear in mind that the syntax is very specific. Remove any spaces within the argument block and -before/after the brackets. Also, Some shells (e.g., zsh) can interpret the open/close brackets +before/after the brackets. Also, some shells (for example, `zsh`) can interpret the open/close brackets (`[]`) separately. You may need to either escape the brackets or use double quotes. ## Importing multiple projects diff --git a/doc/administration/raketasks/ldap.md b/doc/administration/raketasks/ldap.md index 361ab28f6d4..2871a9235a3 100644 --- a/doc/administration/raketasks/ldap.md +++ b/doc/administration/raketasks/ldap.md @@ -1,4 +1,6 @@ -# LDAP Rake Tasks +# LDAP Rake tasks **(CORE ONLY)** + +The following are LDAP-related Rake tasks. ## Check @@ -26,7 +28,7 @@ limit by passing a number to the check task: rake gitlab:ldap:check[50] ``` -## Run a Group Sync +## Run a group sync **(STARTER ONLY)** > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/14735) in [GitLab Starter](https://about.gitlab.com/pricing/) 12.2. diff --git a/doc/administration/raketasks/maintenance.md b/doc/administration/raketasks/maintenance.md index c79a1aa6ba1..eee68c0da0a 100644 --- a/doc/administration/raketasks/maintenance.md +++ b/doc/administration/raketasks/maintenance.md @@ -1,8 +1,11 @@ -# Maintenance Rake Tasks +# Maintenance Rake tasks **(CORE ONLY)** -## Gather information about GitLab and the system it runs on +GitLab provides Rake tasks for general maintenance. -This command gathers information about your GitLab installation and the System it runs on. These may be useful when asking for help or reporting issues. +## Gather GitLab and system information + +This command gathers information about your GitLab installation and the system it runs on. +These may be useful when asking for help or reporting issues. **Omnibus Installation** @@ -50,20 +53,23 @@ Git: /usr/bin/git ## Check GitLab configuration -Runs the following Rake tasks: +The `gitlab:check` Rake task runs the following Rake tasks: - `gitlab:gitlab_shell:check` - `gitlab:gitaly:check` - `gitlab:sidekiq:check` - `gitlab:app:check` -It will check that each component was set up according to the installation guide and suggest fixes for issues found. -This command must be run from your app server and will not work correctly on component servers like [Gitaly](../gitaly/index.md#running-gitaly-on-its-own-server). +It will check that each component was set up according to the installation guide and suggest fixes +for issues found. This command must be run from your application server and will not work correctly on +component servers like [Gitaly](../gitaly/index.md#running-gitaly-on-its-own-server). + +You may also have a look at our troubleshooting guides for: -You may also have a look at our Troubleshooting Guides: +- [GitLab](../index.md#troubleshooting) +- [Omnibus GitLab](https://docs.gitlab.com/omnibus/README.html#troubleshooting) -- [Troubleshooting Guide (GitLab)](../index.md#troubleshooting) -- [Troubleshooting Guide (Omnibus GitLab)](https://docs.gitlab.com/omnibus/README.html#troubleshooting) +To run `gitlab:check`, run: **Omnibus Installation** @@ -77,7 +83,8 @@ sudo gitlab-rake gitlab:check bundle exec rake gitlab:check RAILS_ENV=production ``` -NOTE: Use `SANITIZE=true` for `gitlab:check` if you want to omit project names from the output. +NOTE: **Note:** +Use `SANITIZE=true` for `gitlab:check` if you want to omit project names from the output. Example output: @@ -126,7 +133,7 @@ Checking GitLab ... Finished ## Rebuild authorized_keys file -In some case it is necessary to rebuild the `authorized_keys` file. +In some case it is necessary to rebuild the `authorized_keys` file. To do this, run: **Omnibus Installation** @@ -141,6 +148,8 @@ cd /home/git/gitlab sudo -u git -H bundle exec rake gitlab:shell:setup RAILS_ENV=production ``` +Example output: + ```plaintext This will rebuild an authorized_keys file. You will lose any data stored in authorized_keys file. @@ -149,8 +158,8 @@ Do you want to continue (yes/no)? yes ## Clear Redis cache -If for some reason the dashboard shows wrong information you might want to -clear Redis' cache. +If for some reason the dashboard displays the wrong information, you might want to +clear Redis' cache. To do this, run: **Omnibus Installation** @@ -170,7 +179,7 @@ sudo -u git -H bundle exec rake cache:clear RAILS_ENV=production Sometimes during version upgrades you might end up with some wrong CSS or missing some icons. In that case, try to precompile the assets again. -Note that this only applies to source installations and does NOT apply to +This only applies to source installations and does NOT apply to Omnibus packages. **Source Installation** @@ -187,25 +196,6 @@ production machine after installing the package, there should be no reason to re `rake gitlab:assets:compile` on the production machine. If you suspect that assets have been corrupted, you should reinstall the omnibus package. -## Tracking Deployments - -GitLab provides a Rake task that lets you track deployments in GitLab -Performance Monitoring. This Rake task simply stores the current GitLab version -in the GitLab Performance Monitoring database. - -**Omnibus Installation** - -```shell -sudo gitlab-rake gitlab:track_deployment -``` - -**Source Installation** - -```shell -cd /home/git/gitlab -sudo -u git -H bundle exec rake gitlab:track_deployment RAILS_ENV=production -``` - ## Check TCP connectivity to a remote site Sometimes you need to know if your GitLab installation can connect to a TCP diff --git a/doc/administration/raketasks/praefect.md b/doc/administration/raketasks/praefect.md index 4f9cf0e10ba..c3dadb6bc30 100644 --- a/doc/administration/raketasks/praefect.md +++ b/doc/administration/raketasks/praefect.md @@ -1,4 +1,4 @@ -# Praefect Rake Tasks +# Praefect Rake Tasks **(CORE ONLY)** > [Introduced]( https://gitlab.com/gitlab-org/gitlab/-/merge_requests/28369) in GitLab 12.10. diff --git a/doc/administration/raketasks/project_import_export.md b/doc/administration/raketasks/project_import_export.md index 6d874d596e1..2ab8b13e29e 100644 --- a/doc/administration/raketasks/project_import_export.md +++ b/doc/administration/raketasks/project_import_export.md @@ -3,11 +3,13 @@ > - [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/issues/3050) in GitLab 8.9. > - From GitLab 11.3, import/export can use object storage automatically. -See also: +GitLab provides Rake tasks relating to project import and export. For more information, see: - [Project import/export documentation](../../user/project/settings/import_export.md). - [Project import/export API](../../api/project_import_export.md). +## Import/export tasks + The GitLab import/export version can be checked by using the following command: ```shell @@ -28,8 +30,6 @@ sudo gitlab-rake gitlab:import_export:data bundle exec rake gitlab:import_export:data RAILS_ENV=production ``` -## Important notes - Note the following: - Importing is only possible if the version of the import and export GitLab instances are diff --git a/doc/administration/raketasks/storage.md b/doc/administration/raketasks/storage.md index 1e487a317c9..30f50c24138 100644 --- a/doc/administration/raketasks/storage.md +++ b/doc/administration/raketasks/storage.md @@ -1,214 +1,162 @@ -# Repository Storage Rake Tasks +# Repository storage Rake tasks **(CORE ONLY)** -This is a collection of Rake tasks you can use to help you list and migrate -existing projects and attachments associated with it from Legacy storage to -the new Hashed storage type. +This is a collection of Rake tasks to help you list and migrate +existing projects and their attachments to the new +[hashed storage](../repository_storage_types.md) that GitLab +uses to organize the Git data. -You can read more about the storage types [here][storage-types]. +## List projects and attachments -## Migrate existing projects to Hashed storage +The following Rake tasks will list the projects and attachments that are +available on legacy and hashed storage. -Before migrating your existing projects, you should -[enable hashed storage][storage-migration] for the new projects as well. +### On legacy storage -This task will schedule all your existing projects and attachments associated with it to be migrated to the -**Hashed** storage type: +To have a summary and then a list of projects and their attachments using legacy storage: -**Omnibus Installation** +- **Omnibus installation** -```shell -sudo gitlab-rake gitlab:storage:migrate_to_hashed -``` - -**Source Installation** - -```shell -sudo -u git -H bundle exec rake gitlab:storage:migrate_to_hashed RAILS_ENV=production -``` - -They both also accept a range as environment variable: - -```shell -# to migrate any non migrated project from ID 20 to 50. -export ID_FROM=20 -export ID_TO=50 -``` - -You can monitor the progress in the **Admin Area > Monitoring > Background Jobs** page. -There is a specific Queue you can watch to see how long it will take to finish: -`hashed_storage:hashed_storage_project_migrate` - -After it reaches zero, you can confirm every project has been migrated by running the commands bellow. -If you find it necessary, you can run this migration script again to schedule missing projects. - -Any error or warning will be logged in Sidekiq's log file. - -NOTE: **Note:** -If Geo is enabled, each project that is successfully migrated generates an event to replicate the changes on any **secondary** nodes. - -You only need the `gitlab:storage:migrate_to_hashed` Rake task to migrate your repositories, but we have additional -commands below that helps you inspect projects and attachments in both legacy and hashed storage. - -## Rollback from Hashed storage to Legacy storage - -If you need to rollback the storage migration for any reason, you can follow the steps described here. - -NOTE: **Note:** Hashed Storage will be required in future version of GitLab. - -To prevent new projects from being created in the Hashed storage, -you need to undo the [enable hashed storage][storage-migration] changes. - -This task will schedule all your existing projects and associated attachments to be rolled back to the -Legacy storage type. - -For Omnibus installations, run the following: - -```shell -sudo gitlab-rake gitlab:storage:rollback_to_legacy -``` - -For source installations, run the following: - -```shell -sudo -u git -H bundle exec rake gitlab:storage:rollback_to_legacy RAILS_ENV=production -``` - -Both commands accept a range as environment variable: - -```shell -# to rollback any migrated project from ID 20 to 50. -export ID_FROM=20 -export ID_TO=50 -``` - -You can monitor the progress in the **Admin Area > Monitoring > Background Jobs** page. -On the **Queues** tab, you can watch the `hashed_storage:hashed_storage_project_rollback` queue to see how long the process will take to finish. - -After it reaches zero, you can confirm every project has been rolled back by running the commands bellow. -If some projects weren't rolled back, you can run this rollback script again to schedule further rollbacks. - -Any error or warning will be logged in Sidekiq's log file. - -## List projects on Legacy storage + ```shell + # Projects + sudo gitlab-rake gitlab:storage:legacy_projects + sudo gitlab-rake gitlab:storage:list_legacy_projects -To have a simple summary of projects using **Legacy** storage: + # Attachments + sudo gitlab-rake gitlab:storage:legacy_attachments + sudo gitlab-rake gitlab:storage:list_legacy_attachments + ``` -**Omnibus Installation** +- **Source installation** -```shell -sudo gitlab-rake gitlab:storage:legacy_projects -``` - -**Source Installation** - -```shell -sudo -u git -H bundle exec rake gitlab:storage:legacy_projects RAILS_ENV=production -``` + ```shell + # Projects + sudo -u git -H bundle exec rake gitlab:storage:legacy_projects RAILS_ENV=production + sudo -u git -H bundle exec rake gitlab:storage:list_legacy_projects RAILS_ENV=production -To list projects using **Legacy** storage: + # Attachments + sudo -u git -H bundle exec rake gitlab:storage:legacy_attachments RAILS_ENV=production + sudo -u git -H bundle exec rake gitlab:storage:list_legacy_attachments RAILS_ENV=production + ``` -**Omnibus Installation** - -```shell -sudo gitlab-rake gitlab:storage:list_legacy_projects -``` - -**Source Installation** - -```shell -sudo -u git -H bundle exec rake gitlab:storage:list_legacy_projects RAILS_ENV=production +### On hashed storage -``` +To have a summary and then a list of projects and their attachments using hashed storage: -## List projects on Hashed storage +- **Omnibus installation** -To have a simple summary of projects using **Hashed** storage: + ```shell + # Projects + sudo gitlab-rake gitlab:storage:hashed_projects + sudo gitlab-rake gitlab:storage:list_hashed_projects -**Omnibus Installation** + # Attachments + sudo gitlab-rake gitlab:storage:hashed_attachments + sudo gitlab-rake gitlab:storage:list_hashed_attachments + ``` -```shell -sudo gitlab-rake gitlab:storage:hashed_projects -``` +- **Source installation** -**Source Installation** + ```shell + # Projects + sudo -u git -H bundle exec rake gitlab:storage:hashed_projects RAILS_ENV=production + sudo -u git -H bundle exec rake gitlab:storage:list_hashed_projects RAILS_ENV=production -```shell -sudo -u git -H bundle exec rake gitlab:storage:hashed_projects RAILS_ENV=production -``` + # Attachments + sudo -u git -H bundle exec rake gitlab:storage:hashed_attachments RAILS_ENV=production + sudo -u git -H bundle exec rake gitlab:storage:list_hashed_attachments RAILS_ENV=production + ``` -To list projects using **Hashed** storage: +## Migrate to hashed storage -**Omnibus Installation** +NOTE: **Note:** +In GitLab 13.0, [hashed storage](../repository_storage_types.md#hashed-storage) +is enabled by default and the legacy storage is deprecated. +Support for legacy storage will be removed in GitLab 14.0. If you're on GitLab +13.0 and later, switching new projects to legacy storage is not possible. +The option to choose between hashed and legacy storage in the admin area has +been disabled. -```shell -sudo gitlab-rake gitlab:storage:list_hashed_projects -``` +This task will schedule all your existing projects and attachments associated +with it to be migrated to the **Hashed** storage type: -**Source Installation** +- **Omnibus installation** -```shell -sudo -u git -H bundle exec rake gitlab:storage:list_hashed_projects RAILS_ENV=production -``` + ```shell + sudo gitlab-rake gitlab:storage:migrate_to_hashed + ``` -## List attachments on Legacy storage +- **Source installation** -To have a simple summary of project attachments using **Legacy** storage: + ```shell + sudo -u git -H bundle exec rake gitlab:storage:migrate_to_hashed RAILS_ENV=production + ``` -**Omnibus Installation** +If you have any existing integration, you may want to do a small rollout first, +to validate. You can do so by specifying an ID range with the operation by using +the environment variables `ID_FROM` and `ID_TO`. For example, to limit the rollout +to project IDs 50 to 100 in an Omnibus GitLab installation: ```shell -sudo gitlab-rake gitlab:storage:legacy_attachments +sudo gitlab-rake gitlab:storage:migrate_to_hashed ID_FROM=50 ID_TO=100 ``` -**Source Installation** +You can monitor the progress in the **{admin}** **Admin Area > Monitoring > Background Jobs** page. +There is a specific queue you can watch to see how long it will take to finish: +`hashed_storage:hashed_storage_project_migrate`. -```shell -sudo -u git -H bundle exec rake gitlab:storage:legacy_attachments RAILS_ENV=production -``` - -To list project attachments using **Legacy** storage: - -**Omnibus Installation** +After it reaches zero, you can confirm every project has been migrated by running the commands bellow. +If you find it necessary, you can run this migration script again to schedule missing projects. -```shell -sudo gitlab-rake gitlab:storage:list_legacy_attachments -``` +Any error or warning will be logged in Sidekiq's log file. -**Source Installation** +NOTE: **Note:** +If [Geo](../geo/replication/index.md) is enabled, each project that is successfully migrated +generates an event to replicate the changes on any **secondary** nodes. -```shell -sudo -u git -H bundle exec rake gitlab:storage:list_legacy_attachments RAILS_ENV=production -``` +You only need the `gitlab:storage:migrate_to_hashed` Rake task to migrate your repositories, but we have additional +commands below that helps you inspect projects and attachments in both legacy and hashed storage. -## List attachments on Hashed storage +## Rollback from hashed storage to legacy storage -To have a simple summary of project attachments using **Hashed** storage: +NOTE: **Deprecated:** +In GitLab 13.0, [hashed storage](../repository_storage_types.md#hashed-storage) +is enabled by default and the legacy storage is deprecated. +Support for legacy storage will be removed in GitLab 14.0. If you're on GitLab +13.0 and later, switching new projects to legacy storage is not possible. +The option to choose between hashed and legacy storage in the admin area has +been disabled. -**Omnibus Installation** +This task will schedule all your existing projects and associated attachments to be rolled back to the +legacy storage type. -```shell -sudo gitlab-rake gitlab:storage:hashed_attachments -``` +- **Omnibus installation** -**Source Installation** + ```shell + sudo gitlab-rake gitlab:storage:rollback_to_legacy + ``` -```shell -sudo -u git -H bundle exec rake gitlab:storage:hashed_attachments RAILS_ENV=production -``` +- **Source installation** -To list project attachments using **Hashed** storage: + ```shell + sudo -u git -H bundle exec rake gitlab:storage:rollback_to_legacy RAILS_ENV=production + ``` -**Omnibus Installation** +If you have any existing integration, you may want to do a small rollback first, +to validate. You can do so by specifying an ID range with the operation by using +the environment variables `ID_FROM` and `ID_TO`. For example, to limit the rollout +to project IDs 50 to 100 in an Omnibus GitLab installation: ```shell -sudo gitlab-rake gitlab:storage:list_hashed_attachments +sudo gitlab-rake gitlab:storage:rollback_to_legacy ID_FROM=50 ID_TO=100 ``` -**Source Installation** +You can monitor the progress in the **{admin}** **Admin Area > Monitoring > Background Jobs** page. +On the **Queues** tab, you can watch the `hashed_storage:hashed_storage_project_rollback` queue to see how long the process will take to finish. -```shell -sudo -u git -H bundle exec rake gitlab:storage:list_hashed_attachments RAILS_ENV=production -``` +After it reaches zero, you can confirm every project has been rolled back by running the commands bellow. +If some projects weren't rolled back, you can run this rollback script again to schedule further rollbacks. +Any error or warning will be logged in Sidekiq's log file. -[storage-types]: ../repository_storage_types.md -[storage-migration]: ../repository_storage_types.md#how-to-migrate-to-hashed-storage +If you have a Geo setup, the rollback will not be reflected automatically +on the **secondary** node. You may need to wait for a backfill operation to kick-in and remove +the remaining repositories from the special `@hashed/` folder manually. diff --git a/doc/administration/raketasks/uploads/migrate.md b/doc/administration/raketasks/uploads/migrate.md index 851305d433f..d58b802b024 100644 --- a/doc/administration/raketasks/uploads/migrate.md +++ b/doc/administration/raketasks/uploads/migrate.md @@ -1,21 +1,26 @@ -# Uploads Migrate Rake Tasks +# Uploads migrate Rake tasks **(CORE ONLY)** -## Migrate to Object Storage +`gitlab:uploads:migrate` migrates uploads between different storage types. -After [configuring the object storage](../../uploads.md#using-object-storage-core-only) for GitLab's uploads, you may use this task to migrate existing uploads from the local storage to the remote storage. +## Migrate to object storage ->**Note:** -All of the processing will be done in a background worker and requires **no downtime**. +After [configuring the object storage](../../uploads.md#using-object-storage-core-only) for GitLab's +uploads, use this task to migrate existing uploads from the local storage to the remote storage. + +Read more about using [object storage with GitLab](../../object_storage.md). -[Read more about using object storage with GitLab](../../object_storage.md). +NOTE: **Note:** +All of the processing will be done in a background worker and requires **no downtime**. ### All-in-one Rake task -GitLab provides a wrapper Rake task that migrates all uploaded files - avatars, -logos, attachments, favicon, etc. - to object storage in one go. Under the hood, -it invokes individual Rake tasks to migrate files falling under each of this -category one by one. The specifications of these individual Rake tasks are -described in the next section. +GitLab provides a wrapper Rake task that migrates all uploaded files (for example avatars, logos, +attachments, and favicon) to object storage in one step. The wrapper task invokes individual Rake +tasks to migrate files falling under each of these categories one by one. + +These [individual Rake tasks](#individual-rake-tasks) are described in the next section. + +To migrate all uploads from local storage to object storage, run: **Omnibus Installation** @@ -31,26 +36,29 @@ sudo RAILS_ENV=production -u git -H bundle exec rake gitlab:uploads:migrate:all ### Individual Rake tasks ->**Note:** -If you already ran the Rake task mentioned above, no need to run these individual Rake tasks as that has been done automatically. +If you already ran the [all-in-one Rake task](#all-in-one-rake-task), there is no need to run these +individual tasks. + +The Rake task uses three parameters to find uploads to migrate: -The Rake task uses 3 parameters to find uploads to migrate. +| Parameter | Type | Description | +|:-----------------|:--------------|:-------------------------------------------------------| +| `uploader_class` | string | Type of the uploader to migrate from. | +| `model_class` | string | Type of the model to migrate from. | +| `mount_point` | string/symbol | Name of the model's column the uploader is mounted on. | -Parameter | Type | Description ---------- | ---- | ----------- -`uploader_class` | string | Type of the uploader to migrate from -`model_class` | string | Type of the model to migrate from -`mount_point` | string/symbol | Name of the model's column on which the uploader is mounted on. +NOTE: **Note:** +These parameters are mainly internal to GitLab's structure, you may want to refer to the task list +instead below. ->**Note:** -These parameters are mainly internal to GitLab's structure, you may want to refer to the task list instead below. +This task also accepts an environment variable which you can use to override +the default batch size: -This task also accepts some environment variables which you can use to override -certain values: +| Variable | Type | Description | +|:---------|:--------|:--------------------------------------------------| +| `BATCH` | integer | Specifies the size of the batch. Defaults to 200. | -Variable | Type | Description --------- | ---- | ----------- -`BATCH` | integer | Specifies the size of the batch. Defaults to 200. +The following shows how to run `gitlab:uploads:migrate` for individual types of uploads. **Omnibus Installation** @@ -76,13 +84,13 @@ gitlab-rake "gitlab:uploads:migrate[PersonalFileUploader, Snippet]" gitlab-rake "gitlab:uploads:migrate[NamespaceFileUploader, Snippet]" gitlab-rake "gitlab:uploads:migrate[FileUploader, MergeRequest]" -# Design Management design thumbnails (EE) +# Design Management design thumbnails gitlab-rake "gitlab:uploads:migrate[DesignManagement::DesignV432x230Uploader, DesignManagement::Action, :image_v432x230]" ``` **Source Installation** ->**Note:** +NOTE: **Note:** Use `RAILS_ENV=production` for every task. ```shell @@ -107,16 +115,14 @@ sudo -u git -H bundle exec rake "gitlab:uploads:migrate[PersonalFileUploader, Sn sudo -u git -H bundle exec rake "gitlab:uploads:migrate[NamespaceFileUploader, Snippet]" sudo -u git -H bundle exec rake "gitlab:uploads:migrate[FileUploader, MergeRequest]" -# Design Management design thumbnails (EE) +# Design Management design thumbnails sudo -u git -H bundle exec rake "gitlab:uploads:migrate[DesignManagement::DesignV432x230Uploader, DesignManagement::Action]" ``` -## Migrate from object storage to local storage +## Migrate to local storage -If you need to disable Object Storage for any reason, first you need to migrate -your data out of Object Storage and back into your local storage. - -**Before proceeding, it is important to disable both `direct_upload` and `background_upload` under `uploads` settings in `gitlab.rb`** +If you need to disable [object storage](../../object_storage.md) for any reason, you must first +migrate your data out of object storage and back into your local storage. CAUTION: **Warning:** **Extended downtime is required** so no new files are created in object storage during @@ -126,23 +132,29 @@ To follow progress, see the [relevant issue](https://gitlab.com/gitlab-org/gitla ### All-in-one Rake task -GitLab provides a wrapper Rake task that migrates all uploaded files - avatars, -logos, attachments, favicon, etc. - to local storage in one go. Under the hood, -it invokes individual Rake tasks to migrate files falling under each of this -category one by one. For details on these Rake tasks please [refer to the section above](#individual-rake-tasks), +GitLab provides a wrapper Rake task that migrates all uploaded files (for example, avatars, logos, +attachments, and favicon) to local storage in one step. The wrapper task invokes individual Rake +tasks to migrate files falling under each of these categories one by one. + +For details on these Rake tasks, refer to [Individual Rake tasks](#individual-rake-tasks), keeping in mind the task name in this case is `gitlab:uploads:migrate_to_local`. -**Omnibus Installation** +To migrate uploads from object storage to local storage: -```shell -gitlab-rake "gitlab:uploads:migrate_to_local:all" -``` +1. Disable both `direct_upload` and `background_upload` under `uploads` settings in `gitlab.rb`. +1. Run the Rake task: -**Source Installation** + **Omnibus Installation** -```shell -sudo RAILS_ENV=production -u git -H bundle exec rake gitlab:uploads:migrate_to_local:all -``` + ```shell + gitlab-rake "gitlab:uploads:migrate_to_local:all" + ``` + + **Source Installation** + + ```shell + sudo RAILS_ENV=production -u git -H bundle exec rake gitlab:uploads:migrate_to_local:all + ``` -After this is done, you may disable Object Storage by undoing the changes described -in the instructions to [configure object storage](../../uploads.md#using-object-storage-core-only) +After running the Rake task, you can disable object storage by undoing the changes described +in the instructions to [configure object storage](../../uploads.md#using-object-storage-core-only). diff --git a/doc/administration/raketasks/uploads/sanitize.md b/doc/administration/raketasks/uploads/sanitize.md index c00399a27cf..7c760b95c33 100644 --- a/doc/administration/raketasks/uploads/sanitize.md +++ b/doc/administration/raketasks/uploads/sanitize.md @@ -1,8 +1,13 @@ -# Uploads Sanitize tasks +# Uploads sanitize Rake tasks **(CORE ONLY)** + +Since GitLab 11.9, EXIF data is automatically stripped from JPG or TIFF image uploads. + +EXIF data may contain sensitive information (for example, GPS location), so you +can remove EXIF data from existing images that were uploaded to an earlier version of GitLab. ## Requirements -You need `exiftool` installed on your system. If you installed GitLab: +To run this Rake task, you need `exiftool` installed on your system. If you installed GitLab: - Using the Omnibus package, you're all set. - From source, make sure `exiftool` is installed: @@ -17,48 +22,48 @@ You need `exiftool` installed on your system. If you installed GitLab: ## Remove EXIF data from existing uploads -Since 11.9 EXIF data are automatically stripped from JPG or TIFF image uploads. -Because EXIF data may contain sensitive information (e.g. GPS location), you -can remove EXIF data also from existing images which were uploaded before -with the following command: +To remove EXIF data from existing uploads, run the following command: ```shell sudo RAILS_ENV=production -u git -H bundle exec rake gitlab:uploads:sanitize:remove_exif ``` -This command by default runs in dry mode and it doesn't remove EXIF data. It can be used for +By default, this command runs in "dry run" mode and doesn't remove EXIF data. It can be used for checking if (and how many) images should be sanitized. The Rake task accepts following parameters. -Parameter | Type | Description ---------- | ---- | ----------- -`start_id` | integer | Only uploads with equal or greater ID will be processed -`stop_id` | integer | Only uploads with equal or smaller ID will be processed -`dry_run` | boolean | Do not remove EXIF data, only check if EXIF data are present or not, default: true -`sleep_time` | float | Pause for number of seconds after processing each image, default: 0.3 seconds -`uploader` | string | Run sanitization only for uploads of the given uploader (`FileUploader`, `PersonalFileUploader`, `NamespaceFileUploader`) -`since` | date | Run sanitization only for uploads newer than given date (e.g. `2019-05-01`) +| Parameter | Type | Description | +|:-------------|:--------|:----------------------------------------------------------------------------------------------------------------------------| +| `start_id` | integer | Only uploads with equal or greater ID will be processed | +| `stop_id` | integer | Only uploads with equal or smaller ID will be processed | +| `dry_run` | boolean | Do not remove EXIF data, only check if EXIF data are present or not. Defaults to `true` | +| `sleep_time` | float | Pause for number of seconds after processing each image. Defaults to 0.3 seconds | +| `uploader` | string | Run sanitization only for uploads of the given uploader: `FileUploader`, `PersonalFileUploader`, or `NamespaceFileUploader` | +| `since` | date | Run sanitization only for uploads newer than given date. For example, `2019-05-01` | -If you have too many uploads, you can speed up sanitization by setting -`sleep_time` to a lower value or by running multiple Rake tasks in parallel, -each with a separate range of upload IDs (by setting `start_id` and `stop_id`). +If you have too many uploads, you can speed up sanitization by: -To run the command without dry mode and remove EXIF data from all uploads, you can use: +- Setting `sleep_time` to a lower value. +- Running multiple Rake tasks in parallel, each with a separate range of upload IDs (by setting + `start_id` and `stop_id`). + +To remove EXIF data from all uploads, use: ```shell sudo RAILS_ENV=production -u git -H bundle exec rake gitlab:uploads:sanitize:remove_exif[,,false,] 2>&1 | tee exif.log ``` -To run the command without dry mode on uploads with ID between 100 and 5000 and pause for 0.1 second, you can use: +To remove EXIF data on uploads with an ID between 100 and 5000 and pause for 0.1 second after each file, use: ```shell sudo RAILS_ENV=production -u git -H bundle exec rake gitlab:uploads:sanitize:remove_exif[100,5000,false,0.1] 2>&1 | tee exif.log ``` -Because the output of commands will be probably long, the output is written also into exif.log file. +The output is written into an `exif.log` file because it will probably be long. + +If sanitization fails for an upload, an error message should be in the output of the Rake task. +Typical reasons include that the file is missing in the storage or it's not a valid image. -If sanitization fails for an upload, an error message should be in the output of the Rake task (typical reasons may -be that the file is missing in the storage or it's not a valid image). Please -[report](https://gitlab.com/gitlab-org/gitlab-foss/issues/new) any issues at `gitlab.com` and use -prefix 'EXIF' in issue title with the error output and (if possible) the image. +[Report](https://gitlab.com/gitlab-org/gitlab/issues/new) any issues and use the prefix 'EXIF' in +the issue title with the error output and (if possible) the image. diff --git a/doc/administration/reference_architectures/10k_users.md b/doc/administration/reference_architectures/10k_users.md new file mode 100644 index 00000000000..7f31f336251 --- /dev/null +++ b/doc/administration/reference_architectures/10k_users.md @@ -0,0 +1,79 @@ +# Reference architecture: up to 10,000 users + +This page describes GitLab reference architecture for up to 10,000 users. +For a full list of reference architectures, see +[Available reference architectures](index.md#available-reference-architectures). + +> - **Supported users (approximate):** 10,000 +> - **High Availability:** True +> - **Test RPS rates:** API: 200 RPS, Web: 20 RPS, Git: 20 RPS + +| Service | Nodes | Configuration ([8](#footnotes)) | GCP | AWS ([9](#footnotes)) | Azure([9](#footnotes)) | +|--------------------------------------------------------------|-------|---------------------------------|----------------|-----------------------|------------------------| +| GitLab Rails ([1](#footnotes)) | 3 | 32 vCPU, 28.8GB Memory | n1-highcpu-32 | c5.9xlarge | F32s v2 | +| PostgreSQL | 3 | 4 vCPU, 15GB Memory | n1-standard-4 | m5.xlarge | D4s v3 | +| PgBouncer | 3 | 2 vCPU, 1.8GB Memory | n1-highcpu-2 | c5.large | F2s v2 | +| Gitaly ([2](#footnotes)) ([5](#footnotes)) ([7](#footnotes)) | X | 16 vCPU, 60GB Memory | n1-standard-16 | m5.4xlarge | D16s v3 | +| Redis ([3](#footnotes)) - Cache | 3 | 4 vCPU, 15GB Memory | n1-standard-4 | m5.xlarge | D4s v3 | +| Redis ([3](#footnotes)) - Queues / Shared State | 3 | 4 vCPU, 15GB Memory | n1-standard-4 | m5.xlarge | D4s v3 | +| Redis Sentinel ([3](#footnotes)) - Cache | 3 | 1 vCPU, 1.7GB Memory | g1-small | t2.small | B1MS | +| Redis Sentinel ([3](#footnotes)) - Queues / Shared State | 3 | 1 vCPU, 1.7GB Memory | g1-small | t2.small | B1MS | +| Consul | 3 | 2 vCPU, 1.8GB Memory | n1-highcpu-2 | c5.large | F2s v2 | +| Sidekiq | 4 | 4 vCPU, 15GB Memory | n1-standard-4 | m5.xlarge | D4s v3 | +| Object Storage ([4](#footnotes)) | - | - | - | - | - | +| NFS Server ([5](#footnotes)) ([7](#footnotes)) | 1 | 4 vCPU, 3.6GB Memory | n1-highcpu-4 | c5.xlarge | F4s v2 | +| Monitoring node | 1 | 4 vCPU, 3.6GB Memory | n1-highcpu-4 | c5.xlarge | F4s v2 | +| External load balancing node ([6](#footnotes)) | 1 | 2 vCPU, 1.8GB Memory | n1-highcpu-2 | c5.large | F2s v2 | +| Internal load balancing node ([6](#footnotes)) | 1 | 2 vCPU, 1.8GB Memory | n1-highcpu-2 | c5.large | F2s v2 | + +## Footnotes + +1. In our architectures we run each GitLab Rails node using the Puma webserver + and have its number of workers set to 90% of available CPUs along with four threads. For + nodes that are running Rails with other components the worker value should be reduced + accordingly where we've found 50% achieves a good balance but this is dependent + on workload. + +1. Gitaly node requirements are dependent on customer data, specifically the number of + projects and their sizes. We recommend two nodes as an absolute minimum for HA environments + and at least four nodes should be used when supporting 50,000 or more users. + We also recommend that each Gitaly node should store no more than 5TB of data + and have the number of [`gitaly-ruby` workers](../gitaly/index.md#gitaly-ruby) + set to 20% of available CPUs. Additional nodes should be considered in conjunction + with a review of expected data size and spread based on the recommendations above. + +1. Recommended Redis setup differs depending on the size of the architecture. + For smaller architectures (less than 3,000 users) a single instance should suffice. + For medium sized installs (3,000 - 5,000) we suggest one Redis cluster for all + classes and that Redis Sentinel is hosted alongside Consul. + For larger architectures (10,000 users or more) we suggest running a separate + [Redis Cluster](../high_availability/redis.md#running-multiple-redis-clusters) for the Cache class + and another for the Queues and Shared State classes respectively. We also recommend + that you run the Redis Sentinel clusters separately for each Redis Cluster. + +1. For data objects such as LFS, Uploads, Artifacts, etc. We recommend an [Object Storage service](../object_storage.md) + over NFS where possible, due to better performance and availability. + +1. NFS can be used as an alternative for both repository data (replacing Gitaly) and + object storage but this isn't typically recommended for performance reasons. Note however it is required for + [GitLab Pages](https://gitlab.com/gitlab-org/gitlab-pages/issues/196). + +1. Our architectures have been tested and validated with [HAProxy](https://www.haproxy.org/) + as the load balancer. Although other load balancers with similar feature sets + could also be used, those load balancers have not been validated. + +1. We strongly recommend that any Gitaly or NFS nodes be set up with SSD disks over + HDD with a throughput of at least 8,000 IOPS for read operations and 2,000 IOPS for write + as these components have heavy I/O. These IOPS values are recommended only as a starter + as with time they may be adjusted higher or lower depending on the scale of your + environment's workload. If you're running the environment on a Cloud provider + you may need to refer to their documentation on how configure IOPS correctly. + +1. The architectures were built and tested with the [Intel Xeon E5 v3 (Haswell)](https://cloud.google.com/compute/docs/cpu-platforms) + CPU platform on GCP. On different hardware you may find that adjustments, either lower + or higher, are required for your CPU or Node counts accordingly. For more information, a + [Sysbench](https://github.com/akopytov/sysbench) benchmark of the CPU can be found + [here](https://gitlab.com/gitlab-org/quality/performance/-/wikis/Reference-Architectures/GCP-CPU-Benchmarks). + +1. AWS-equivalent and Azure-equivalent configurations are rough suggestions + and may change in the future. They have not yet been tested and validated. diff --git a/doc/administration/reference_architectures/1k_users.md b/doc/administration/reference_architectures/1k_users.md new file mode 100644 index 00000000000..b6aaffa9488 --- /dev/null +++ b/doc/administration/reference_architectures/1k_users.md @@ -0,0 +1,82 @@ +# Reference architecture: up to 1,000 users + +This page describes GitLab reference architecture for up to 1,000 users. +For a full list of reference architectures, see +[Available reference architectures](index.md#available-reference-architectures). + +> - **Supported users (approximate):** 1,000 +> - **High Availability:** False + +| Users | Configuration([8](#footnotes)) | GCP | AWS([9](#footnotes)) | Azure([9](#footnotes)) | +|-------|--------------------------------|---------------|----------------------|------------------------| +| 100 | 2 vCPU, 7.2GB Memory | n1-standard-2 | m5.large | D2s v3 | +| 500 | 4 vCPU, 15GB Memory | n1-standard-4 | m5.xlarge | D4s v3 | +| 1000 | 8 vCPU, 30GB Memory | n1-standard-8 | m5.2xlarge | D8s v3 | + +For situations where you need to serve up to 1,000 users, a single-node +solution with [frequent backups](index.md#automated-backups-core-only) is appropriate +for many organizations. With automatic backup of the GitLab repositories, +configuration, and the database, if you don't have strict availability +requirements, this is the ideal solution. + +## Setup instructions + +- For this default reference architecture, use the standard [installation instructions](../../install/README.md) to install GitLab. + +NOTE: **Note:** +You can also optionally configure GitLab to use an +[external PostgreSQL service](../external_database.md) or an +[external object storage service](../high_availability/object_storage.md) for +added performance and reliability at a reduced complexity cost. + +## Footnotes + +1. In our architectures we run each GitLab Rails node using the Puma webserver + and have its number of workers set to 90% of available CPUs along with four threads. For + nodes that are running Rails with other components the worker value should be reduced + accordingly where we've found 50% achieves a good balance but this is dependent + on workload. + +1. Gitaly node requirements are dependent on customer data, specifically the number of + projects and their sizes. We recommend two nodes as an absolute minimum for HA environments + and at least four nodes should be used when supporting 50,000 or more users. + We also recommend that each Gitaly node should store no more than 5TB of data + and have the number of [`gitaly-ruby` workers](../gitaly/index.md#gitaly-ruby) + set to 20% of available CPUs. Additional nodes should be considered in conjunction + with a review of expected data size and spread based on the recommendations above. + +1. Recommended Redis setup differs depending on the size of the architecture. + For smaller architectures (less than 3,000 users) a single instance should suffice. + For medium sized installs (3,000 - 5,000) we suggest one Redis cluster for all + classes and that Redis Sentinel is hosted alongside Consul. + For larger architectures (10,000 users or more) we suggest running a separate + [Redis Cluster](../high_availability/redis.md#running-multiple-redis-clusters) for the Cache class + and another for the Queues and Shared State classes respectively. We also recommend + that you run the Redis Sentinel clusters separately for each Redis Cluster. + +1. For data objects such as LFS, Uploads, Artifacts, etc. We recommend an [Object Storage service](../object_storage.md) + over NFS where possible, due to better performance and availability. + +1. NFS can be used as an alternative for both repository data (replacing Gitaly) and + object storage but this isn't typically recommended for performance reasons. Note however it is required for + [GitLab Pages](https://gitlab.com/gitlab-org/gitlab-pages/issues/196). + +1. Our architectures have been tested and validated with [HAProxy](https://www.haproxy.org/) + as the load balancer. Although other load balancers with similar feature sets + could also be used, those load balancers have not been validated. + +1. We strongly recommend that any Gitaly or NFS nodes be set up with SSD disks over + HDD with a throughput of at least 8,000 IOPS for read operations and 2,000 IOPS for write + as these components have heavy I/O. These IOPS values are recommended only as a starter + as with time they may be adjusted higher or lower depending on the scale of your + environment's workload. If you're running the environment on a Cloud provider + you may need to refer to their documentation on how configure IOPS correctly. + +1. The architectures were built and tested with the [Intel Xeon E5 v3 (Haswell)](https://cloud.google.com/compute/docs/cpu-platforms) + CPU platform on GCP. On different hardware you may find that adjustments, either lower + or higher, are required for your CPU or Node counts accordingly. For more information, a + [Sysbench](https://github.com/akopytov/sysbench) benchmark of the CPU can be found + [here](https://gitlab.com/gitlab-org/quality/performance/-/wikis/Reference-Architectures/GCP-CPU-Benchmarks). + +1. AWS-equivalent and Azure-equivalent configurations are rough suggestions + and may change in the future. They have not yet been tested and validated. diff --git a/doc/administration/reference_architectures/25k_users.md b/doc/administration/reference_architectures/25k_users.md new file mode 100644 index 00000000000..2ee692d635c --- /dev/null +++ b/doc/administration/reference_architectures/25k_users.md @@ -0,0 +1,79 @@ +# Reference architecture: up to 25,000 users + +This page describes GitLab reference architecture for up to 25,000 users. +For a full list of reference architectures, see +[Available reference architectures](index.md#available-reference-architectures). + +> - **Supported users (approximate):** 25,000 +> - **High Availability:** True +> - **Test RPS rates:** API: 500 RPS, Web: 50 RPS, Git: 50 RPS + +| Service | Nodes | Configuration ([8](#footnotes)) | GCP | AWS ([9](#footnotes)) | Azure([9](#footnotes)) | +|--------------------------------------------------------------|-------|---------------------------------|----------------|-----------------------|------------------------| +| GitLab Rails ([1](#footnotes)) | 5 | 32 vCPU, 28.8GB Memory | n1-highcpu-32 | c5.9xlarge | F32s v2 | +| PostgreSQL | 3 | 8 vCPU, 30GB Memory | n1-standard-8 | m5.2xlarge | D8s v3 | +| PgBouncer | 3 | 2 vCPU, 1.8GB Memory | n1-highcpu-2 | c5.large | F2s v2 | +| Gitaly ([2](#footnotes)) ([5](#footnotes)) ([7](#footnotes)) | X | 32 vCPU, 120GB Memory | n1-standard-32 | m5.8xlarge | D32s v3 | +| Redis ([3](#footnotes)) - Cache | 3 | 4 vCPU, 15GB Memory | n1-standard-4 | m5.xlarge | D4s v3 | +| Redis ([3](#footnotes)) - Queues / Shared State | 3 | 4 vCPU, 15GB Memory | n1-standard-4 | m5.xlarge | D4s v3 | +| Redis Sentinel ([3](#footnotes)) - Cache | 3 | 1 vCPU, 1.7GB Memory | g1-small | t2.small | B1MS | +| Redis Sentinel ([3](#footnotes)) - Queues / Shared State | 3 | 1 vCPU, 1.7GB Memory | g1-small | t2.small | B1MS | +| Consul | 3 | 2 vCPU, 1.8GB Memory | n1-highcpu-2 | c5.large | F2s v2 | +| Sidekiq | 4 | 4 vCPU, 15GB Memory | n1-standard-4 | m5.xlarge | D4s v3 | +| Object Storage ([4](#footnotes)) | - | - | - | - | - | +| NFS Server ([5](#footnotes)) ([7](#footnotes)) | 1 | 4 vCPU, 3.6GB Memory | n1-highcpu-4 | c5.xlarge | F4s v2 | +| Monitoring node | 1 | 4 vCPU, 3.6GB Memory | n1-highcpu-4 | c5.xlarge | F4s v2 | +| External load balancing node ([6](#footnotes)) | 1 | 2 vCPU, 1.8GB Memory | n1-highcpu-2 | c5.large | F2s v2 | +| Internal load balancing node ([6](#footnotes)) | 1 | 4 vCPU, 3.6GB Memory | n1-highcpu-4 | c5.xlarge | F4s v2 | + +## Footnotes + +1. In our architectures we run each GitLab Rails node using the Puma webserver + and have its number of workers set to 90% of available CPUs along with four threads. For + nodes that are running Rails with other components the worker value should be reduced + accordingly where we've found 50% achieves a good balance but this is dependent + on workload. + +1. Gitaly node requirements are dependent on customer data, specifically the number of + projects and their sizes. We recommend two nodes as an absolute minimum for HA environments + and at least four nodes should be used when supporting 50,000 or more users. + We also recommend that each Gitaly node should store no more than 5TB of data + and have the number of [`gitaly-ruby` workers](../gitaly/index.md#gitaly-ruby) + set to 20% of available CPUs. Additional nodes should be considered in conjunction + with a review of expected data size and spread based on the recommendations above. + +1. Recommended Redis setup differs depending on the size of the architecture. + For smaller architectures (less than 3,000 users) a single instance should suffice. + For medium sized installs (3,000 - 5,000) we suggest one Redis cluster for all + classes and that Redis Sentinel is hosted alongside Consul. + For larger architectures (10,000 users or more) we suggest running a separate + [Redis Cluster](../high_availability/redis.md#running-multiple-redis-clusters) for the Cache class + and another for the Queues and Shared State classes respectively. We also recommend + that you run the Redis Sentinel clusters separately for each Redis Cluster. + +1. For data objects such as LFS, Uploads, Artifacts, etc. We recommend an [Object Storage service](../object_storage.md) + over NFS where possible, due to better performance and availability. + +1. NFS can be used as an alternative for both repository data (replacing Gitaly) and + object storage but this isn't typically recommended for performance reasons. Note however it is required for + [GitLab Pages](https://gitlab.com/gitlab-org/gitlab-pages/issues/196). + +1. Our architectures have been tested and validated with [HAProxy](https://www.haproxy.org/) + as the load balancer. Although other load balancers with similar feature sets + could also be used, those load balancers have not been validated. + +1. We strongly recommend that any Gitaly or NFS nodes be set up with SSD disks over + HDD with a throughput of at least 8,000 IOPS for read operations and 2,000 IOPS for write + as these components have heavy I/O. These IOPS values are recommended only as a starter + as with time they may be adjusted higher or lower depending on the scale of your + environment's workload. If you're running the environment on a Cloud provider + you may need to refer to their documentation on how configure IOPS correctly. + +1. The architectures were built and tested with the [Intel Xeon E5 v3 (Haswell)](https://cloud.google.com/compute/docs/cpu-platforms) + CPU platform on GCP. On different hardware you may find that adjustments, either lower + or higher, are required for your CPU or Node counts accordingly. For more information, a + [Sysbench](https://github.com/akopytov/sysbench) benchmark of the CPU can be found + [here](https://gitlab.com/gitlab-org/quality/performance/-/wikis/Reference-Architectures/GCP-CPU-Benchmarks). + +1. AWS-equivalent and Azure-equivalent configurations are rough suggestions + and may change in the future. They have not yet been tested and validated. diff --git a/doc/administration/reference_architectures/2k_users.md b/doc/administration/reference_architectures/2k_users.md new file mode 100644 index 00000000000..874e00e6722 --- /dev/null +++ b/doc/administration/reference_architectures/2k_users.md @@ -0,0 +1,90 @@ +# Reference architecture: up to 2,000 users + +This page describes GitLab reference architecture for up to 2,000 users. +For a full list of reference architectures, see +[Available reference architectures](index.md#available-reference-architectures). + +> - **Supported users (approximate):** 2,000 +> - **High Availability:** False +> - **Test RPS rates:** API: 40 RPS, Web: 4 RPS, Git: 4 RPS + +| Service | Nodes | Configuration ([8](#footnotes)) | GCP | AWS ([9](#footnotes)) | Azure([9](#footnotes)) | +|--------------------------------------------------------------|-------|---------------------------------|---------------|-----------------------|----------------| +| External load balancing node ([6](#footnotes)) | 1 | 2 vCPU, 1.8GB Memory | n1-highcpu-2 | c5.large | F2s v2 | +| Object Storage ([4](#footnotes)) | - | - | - | - | - | +| NFS Server ([5](#footnotes)) ([7](#footnotes)) | 1 | 4 vCPU, 3.6GB Memory | n1-highcpu-4 | c5.xlarge | F4s v2 | +| PostgreSQL | 1 | 2 vCPU, 7.5GB Memory | n1-standard-2 | m5.large | D2s v3 | +| Redis ([3](#footnotes)) | 1 | 1 vCPU, 3.75GB Memory | n1-standard-1 | m5.large | D2s v3 | +| Gitaly ([5](#footnotes)) ([7](#footnotes)) | X ([2](#footnotes)) | 4 vCPU, 15GB Memory | n1-standard-4 | m5.xlarge | D4s v3 | +| GitLab Rails ([1](#footnotes)) | 2 | 8 vCPU, 7.2GB Memory | n1-highcpu-8 | c5.2xlarge | F8s v2 | +| Monitoring node | 1 | 2 vCPU, 1.8GB Memory | n1-highcpu-2 | c5.large | F2s v2 | + +## Setup instructions + +1. [Configure the external load balancing node](../high_availability/load_balancer.md) + that will handle the load balancing of the two GitLab application services nodes. +1. [Configure the Object Storage](../object_storage.md) ([4](#footnotes)) used for shared data objects. +1. (Optional) [Configure NFS](../high_availability/nfs.md) to have + shared disk storage service as an alternative to Gitaly and/or + [Object Storage](../object_storage.md) (although not recommended). + NFS is required for GitLab Pages, you can skip this step if you're not using that feature. +1. [Configure PostgreSQL](../high_availability/load_balancer.md), the database for GitLab. +1. [Configure Redis](../high_availability/redis.md). +1. [Configure Gitaly](../gitaly/index.md#running-gitaly-on-its-own-server), + which is used to provide access to the Git repositories. +1. [Configure the main GitLab Rails application](../high_availability/gitlab.md) + to run Puma/Unicorn, Workhorse, GitLab Shell, and to serve all + frontend requests (UI, API, Git over HTTP/SSH). +1. [Configure Prometheus](../high_availability/monitoring_node.md) to monitor your GitLab environment. + +## Footnotes + +1. In our architectures we run each GitLab Rails node using the Puma webserver + and have its number of workers set to 90% of available CPUs along with four threads. For + nodes that are running Rails with other components the worker value should be reduced + accordingly where we've found 50% achieves a good balance but this is dependent + on workload. + +1. Gitaly node requirements are dependent on customer data, specifically the number of + projects and their sizes. We recommend two nodes as an absolute minimum for HA environments + and at least four nodes should be used when supporting 50,000 or more users. + We also recommend that each Gitaly node should store no more than 5TB of data + and have the number of [`gitaly-ruby` workers](../gitaly/index.md#gitaly-ruby) + set to 20% of available CPUs. Additional nodes should be considered in conjunction + with a review of expected data size and spread based on the recommendations above. + +1. Recommended Redis setup differs depending on the size of the architecture. + For smaller architectures (less than 3,000 users) a single instance should suffice. + For medium sized installs (3,000 - 5,000) we suggest one Redis cluster for all + classes and that Redis Sentinel is hosted alongside Consul. + For larger architectures (10,000 users or more) we suggest running a separate + [Redis Cluster](../high_availability/redis.md#running-multiple-redis-clusters) for the Cache class + and another for the Queues and Shared State classes respectively. We also recommend + that you run the Redis Sentinel clusters separately for each Redis Cluster. + +1. For data objects such as LFS, Uploads, Artifacts, etc. We recommend an [Object Storage service](../object_storage.md) + over NFS where possible, due to better performance and availability. + +1. NFS can be used as an alternative for both repository data (replacing Gitaly) and + object storage but this isn't typically recommended for performance reasons. Note however it is required for + [GitLab Pages](https://gitlab.com/gitlab-org/gitlab-pages/issues/196). + +1. Our architectures have been tested and validated with [HAProxy](https://www.haproxy.org/) + as the load balancer. Although other load balancers with similar feature sets + could also be used, those load balancers have not been validated. + +1. We strongly recommend that any Gitaly or NFS nodes be set up with SSD disks over + HDD with a throughput of at least 8,000 IOPS for read operations and 2,000 IOPS for write + as these components have heavy I/O. These IOPS values are recommended only as a starter + as with time they may be adjusted higher or lower depending on the scale of your + environment's workload. If you're running the environment on a Cloud provider + you may need to refer to their documentation on how configure IOPS correctly. + +1. The architectures were built and tested with the [Intel Xeon E5 v3 (Haswell)](https://cloud.google.com/compute/docs/cpu-platforms) + CPU platform on GCP. On different hardware you may find that adjustments, either lower + or higher, are required for your CPU or Node counts accordingly. For more information, a + [Sysbench](https://github.com/akopytov/sysbench) benchmark of the CPU can be found + [here](https://gitlab.com/gitlab-org/quality/performance/-/wikis/Reference-Architectures/GCP-CPU-Benchmarks). + +1. AWS-equivalent and Azure-equivalent configurations are rough suggestions + and may change in the future. They have not yet been tested and validated. diff --git a/doc/administration/reference_architectures/3k_users.md b/doc/administration/reference_architectures/3k_users.md new file mode 100644 index 00000000000..bd429fbc4b4 --- /dev/null +++ b/doc/administration/reference_architectures/3k_users.md @@ -0,0 +1,82 @@ +# Reference architecture: up to 3,000 users + +This page describes GitLab reference architecture for up to 3,000 users. +For a full list of reference architectures, see +[Available reference architectures](index.md#available-reference-architectures). + +NOTE: **Note:** The 3,000-user reference architecture documented below is +designed to help your organization achieve a highly-available GitLab deployment. +If you do not have the expertise or need to maintain a highly-available +environment, you can have a simpler and less costly-to-operate environment by +following the [2,000-user reference architecture](2k_users.md). + +> - **Supported users (approximate):** 3,000 +> - **High Availability:** True +> - **Test RPS rates:** API: 60 RPS, Web: 6 RPS, Git: 6 RPS + +| Service | Nodes | Configuration ([8](#footnotes)) | GCP | AWS ([9](#footnotes)) | Azure([9](#footnotes)) | +|--------------------------------------------------------------|-------|---------------------------------|---------------|-----------------------|------------------------| +| GitLab Rails ([1](#footnotes)) | 3 | 8 vCPU, 7.2GB Memory | n1-highcpu-8 | c5.2xlarge | F8s v2 | +| PostgreSQL | 3 | 2 vCPU, 7.5GB Memory | n1-standard-2 | m5.large | D2s v3 | +| PgBouncer | 3 | 2 vCPU, 1.8GB Memory | n1-highcpu-2 | c5.large | F2s v2 | +| Gitaly ([2](#footnotes)) ([5](#footnotes)) ([7](#footnotes)) | X | 4 vCPU, 15GB Memory | n1-standard-4 | m5.xlarge | D4s v3 | +| Redis ([3](#footnotes)) | 3 | 2 vCPU, 7.5GB Memory | n1-standard-2 | m5.large | D2s v3 | +| Consul + Sentinel ([3](#footnotes)) | 3 | 2 vCPU, 1.8GB Memory | n1-highcpu-2 | c5.large | F2s v2 | +| Sidekiq | 4 | 2 vCPU, 7.5GB Memory | n1-standard-2 | m5.large | D2s v3 | +| Object Storage ([4](#footnotes)) | - | - | - | - | - | +| NFS Server ([5](#footnotes)) ([7](#footnotes)) | 1 | 4 vCPU, 3.6GB Memory | n1-highcpu-4 | c5.xlarge | F4s v2 | +| Monitoring node | 1 | 2 vCPU, 1.8GB Memory | n1-highcpu-2 | c5.large | F2s v2 | +| External load balancing node ([6](#footnotes)) | 1 | 2 vCPU, 1.8GB Memory | n1-highcpu-2 | c5.large | F2s v2 | +| Internal load balancing node ([6](#footnotes)) | 1 | 2 vCPU, 1.8GB Memory | n1-highcpu-2 | c5.large | F2s v2 | + +## Footnotes + +1. In our architectures we run each GitLab Rails node using the Puma webserver + and have its number of workers set to 90% of available CPUs along with four threads. For + nodes that are running Rails with other components the worker value should be reduced + accordingly where we've found 50% achieves a good balance but this is dependent + on workload. + +1. Gitaly node requirements are dependent on customer data, specifically the number of + projects and their sizes. We recommend two nodes as an absolute minimum for HA environments + and at least four nodes should be used when supporting 50,000 or more users. + We also recommend that each Gitaly node should store no more than 5TB of data + and have the number of [`gitaly-ruby` workers](../gitaly/index.md#gitaly-ruby) + set to 20% of available CPUs. Additional nodes should be considered in conjunction + with a review of expected data size and spread based on the recommendations above. + +1. Recommended Redis setup differs depending on the size of the architecture. + For smaller architectures (less than 3,000 users) a single instance should suffice. + For medium sized installs (3,000 - 5,000) we suggest one Redis cluster for all + classes and that Redis Sentinel is hosted alongside Consul. + For larger architectures (10,000 users or more) we suggest running a separate + [Redis Cluster](../high_availability/redis.md#running-multiple-redis-clusters) for the Cache class + and another for the Queues and Shared State classes respectively. We also recommend + that you run the Redis Sentinel clusters separately for each Redis Cluster. + +1. For data objects such as LFS, Uploads, Artifacts, etc. We recommend an [Object Storage service](../object_storage.md) + over NFS where possible, due to better performance and availability. + +1. NFS can be used as an alternative for both repository data (replacing Gitaly) and + object storage but this isn't typically recommended for performance reasons. Note however it is required for + [GitLab Pages](https://gitlab.com/gitlab-org/gitlab-pages/issues/196). + +1. Our architectures have been tested and validated with [HAProxy](https://www.haproxy.org/) + as the load balancer. Although other load balancers with similar feature sets + could also be used, those load balancers have not been validated. + +1. We strongly recommend that any Gitaly or NFS nodes be set up with SSD disks over + HDD with a throughput of at least 8,000 IOPS for read operations and 2,000 IOPS for write + as these components have heavy I/O. These IOPS values are recommended only as a starter + as with time they may be adjusted higher or lower depending on the scale of your + environment's workload. If you're running the environment on a Cloud provider + you may need to refer to their documentation on how configure IOPS correctly. + +1. The architectures were built and tested with the [Intel Xeon E5 v3 (Haswell)](https://cloud.google.com/compute/docs/cpu-platforms) + CPU platform on GCP. On different hardware you may find that adjustments, either lower + or higher, are required for your CPU or Node counts accordingly. For more information, a + [Sysbench](https://github.com/akopytov/sysbench) benchmark of the CPU can be found + [here](https://gitlab.com/gitlab-org/quality/performance/-/wikis/Reference-Architectures/GCP-CPU-Benchmarks). + +1. AWS-equivalent and Azure-equivalent configurations are rough suggestions + and may change in the future. They have not yet been tested and validated. diff --git a/doc/administration/reference_architectures/50k_users.md b/doc/administration/reference_architectures/50k_users.md new file mode 100644 index 00000000000..67f773a021f --- /dev/null +++ b/doc/administration/reference_architectures/50k_users.md @@ -0,0 +1,79 @@ +# Reference architecture: up to 50,000 users + +This page describes GitLab reference architecture for up to 50,000 users. +For a full list of reference architectures, see +[Available reference architectures](index.md#available-reference-architectures). + +> - **Supported users (approximate):** 50,000 +> - **High Availability:** True +> - **Test RPS rates:** API: 1000 RPS, Web: 100 RPS, Git: 100 RPS + +| Service | Nodes | Configuration ([8](#footnotes)) | GCP | AWS ([9](#footnotes)) | Azure([9](#footnotes)) | +|--------------------------------------------------------------|-------|---------------------------------|----------------|-----------------------|------------------------| +| GitLab Rails ([1](#footnotes)) | 12 | 32 vCPU, 28.8GB Memory | n1-highcpu-32 | c5.9xlarge | F32s v2 | +| PostgreSQL | 3 | 16 vCPU, 60GB Memory | n1-standard-16 | m5.4xlarge | D16s v3 | +| PgBouncer | 3 | 2 vCPU, 1.8GB Memory | n1-highcpu-2 | c5.large | F2s v2 | +| Gitaly ([2](#footnotes)) ([5](#footnotes)) ([7](#footnotes)) | X | 64 vCPU, 240GB Memory | n1-standard-64 | m5.16xlarge | D64s v3 | +| Redis ([3](#footnotes)) - Cache | 3 | 4 vCPU, 15GB Memory | n1-standard-4 | m5.xlarge | D4s v3 | +| Redis ([3](#footnotes)) - Queues / Shared State | 3 | 4 vCPU, 15GB Memory | n1-standard-4 | m5.xlarge | D4s v3 | +| Redis Sentinel ([3](#footnotes)) - Cache | 3 | 1 vCPU, 1.7GB Memory | g1-small | t2.small | B1MS | +| Redis Sentinel ([3](#footnotes)) - Queues / Shared State | 3 | 1 vCPU, 1.7GB Memory | g1-small | t2.small | B1MS | +| Consul | 3 | 2 vCPU, 1.8GB Memory | n1-highcpu-2 | c5.large | F2s v2 | +| Sidekiq | 4 | 4 vCPU, 15GB Memory | n1-standard-4 | m5.xlarge | D4s v3 | +| NFS Server ([5](#footnotes)) ([7](#footnotes)) | 1 | 4 vCPU, 3.6GB Memory | n1-highcpu-4 | c5.xlarge | F4s v2 | +| Object Storage ([4](#footnotes)) | - | - | - | - | - | +| Monitoring node | 1 | 4 vCPU, 3.6GB Memory | n1-highcpu-4 | c5.xlarge | F4s v2 | +| External load balancing node ([6](#footnotes)) | 1 | 2 vCPU, 1.8GB Memory | n1-highcpu-2 | c5.large | F2s v2 | +| Internal load balancing node ([6](#footnotes)) | 1 | 8 vCPU, 7.2GB Memory | n1-highcpu-8 | c5.2xlarge | F8s v2 | + +## Footnotes + +1. In our architectures we run each GitLab Rails node using the Puma webserver + and have its number of workers set to 90% of available CPUs along with four threads. For + nodes that are running Rails with other components the worker value should be reduced + accordingly where we've found 50% achieves a good balance but this is dependent + on workload. + +1. Gitaly node requirements are dependent on customer data, specifically the number of + projects and their sizes. We recommend two nodes as an absolute minimum for HA environments + and at least four nodes should be used when supporting 50,000 or more users. + We also recommend that each Gitaly node should store no more than 5TB of data + and have the number of [`gitaly-ruby` workers](../gitaly/index.md#gitaly-ruby) + set to 20% of available CPUs. Additional nodes should be considered in conjunction + with a review of expected data size and spread based on the recommendations above. + +1. Recommended Redis setup differs depending on the size of the architecture. + For smaller architectures (less than 3,000 users) a single instance should suffice. + For medium sized installs (3,000 - 5,000) we suggest one Redis cluster for all + classes and that Redis Sentinel is hosted alongside Consul. + For larger architectures (10,000 users or more) we suggest running a separate + [Redis Cluster](../high_availability/redis.md#running-multiple-redis-clusters) for the Cache class + and another for the Queues and Shared State classes respectively. We also recommend + that you run the Redis Sentinel clusters separately for each Redis Cluster. + +1. For data objects such as LFS, Uploads, Artifacts, etc. We recommend an [Object Storage service](../object_storage.md) + over NFS where possible, due to better performance and availability. + +1. NFS can be used as an alternative for both repository data (replacing Gitaly) and + object storage but this isn't typically recommended for performance reasons. Note however it is required for + [GitLab Pages](https://gitlab.com/gitlab-org/gitlab-pages/issues/196). + +1. Our architectures have been tested and validated with [HAProxy](https://www.haproxy.org/) + as the load balancer. Although other load balancers with similar feature sets + could also be used, those load balancers have not been validated. + +1. We strongly recommend that any Gitaly or NFS nodes be set up with SSD disks over + HDD with a throughput of at least 8,000 IOPS for read operations and 2,000 IOPS for write + as these components have heavy I/O. These IOPS values are recommended only as a starter + as with time they may be adjusted higher or lower depending on the scale of your + environment's workload. If you're running the environment on a Cloud provider + you may need to refer to their documentation on how configure IOPS correctly. + +1. The architectures were built and tested with the [Intel Xeon E5 v3 (Haswell)](https://cloud.google.com/compute/docs/cpu-platforms) + CPU platform on GCP. On different hardware you may find that adjustments, either lower + or higher, are required for your CPU or Node counts accordingly. For more information, a + [Sysbench](https://github.com/akopytov/sysbench) benchmark of the CPU can be found + [here](https://gitlab.com/gitlab-org/quality/performance/-/wikis/Reference-Architectures/GCP-CPU-Benchmarks). + +1. AWS-equivalent and Azure-equivalent configurations are rough suggestions + and may change in the future. They have not yet been tested and validated. diff --git a/doc/administration/reference_architectures/5k_users.md b/doc/administration/reference_architectures/5k_users.md new file mode 100644 index 00000000000..41ef6f369c2 --- /dev/null +++ b/doc/administration/reference_architectures/5k_users.md @@ -0,0 +1,76 @@ +# Reference architecture: up to 5,000 users + +This page describes GitLab reference architecture for up to 5,000 users. +For a full list of reference architectures, see +[Available reference architectures](index.md#available-reference-architectures). + +> - **Supported users (approximate):** 5,000 +> - **High Availability:** True +> - **Test RPS rates:** API: 100 RPS, Web: 10 RPS, Git: 10 RPS + +| Service | Nodes | Configuration ([8](#footnotes)) | GCP | AWS ([9](#footnotes)) | Azure([9](#footnotes)) | +|--------------------------------------------------------------|-------|---------------------------------|---------------|-----------------------|------------------------| +| GitLab Rails ([1](#footnotes)) | 3 | 16 vCPU, 14.4GB Memory | n1-highcpu-16 | c5.4xlarge | F16s v2 | +| PostgreSQL | 3 | 2 vCPU, 7.5GB Memory | n1-standard-2 | m5.large | D2s v3 | +| PgBouncer | 3 | 2 vCPU, 1.8GB Memory | n1-highcpu-2 | c5.large | F2s v2 | +| Gitaly ([2](#footnotes)) ([5](#footnotes)) ([7](#footnotes)) | X | 8 vCPU, 30GB Memory | n1-standard-8 | m5.2xlarge | D8s v3 | +| Redis ([3](#footnotes)) | 3 | 2 vCPU, 7.5GB Memory | n1-standard-2 | m5.large | D2s v3 | +| Consul + Sentinel ([3](#footnotes)) | 3 | 2 vCPU, 1.8GB Memory | n1-highcpu-2 | c5.large | F2s v2 | +| Sidekiq | 4 | 2 vCPU, 7.5GB Memory | n1-standard-2 | m5.large | D2s v3 | +| Object Storage ([4](#footnotes)) | - | - | - | - | - | +| NFS Server ([5](#footnotes)) ([7](#footnotes)) | 1 | 4 vCPU, 3.6GB Memory | n1-highcpu-4 | c5.xlarge | F4s v2 | +| Monitoring node | 1 | 2 vCPU, 1.8GB Memory | n1-highcpu-2 | c5.large | F2s v2 | +| External load balancing node ([6](#footnotes)) | 1 | 2 vCPU, 1.8GB Memory | n1-highcpu-2 | c5.large | F2s v2 | +| Internal load balancing node ([6](#footnotes)) | 1 | 2 vCPU, 1.8GB Memory | n1-highcpu-2 | c5.large | F2s v2 | + +## Footnotes + +1. In our architectures we run each GitLab Rails node using the Puma webserver + and have its number of workers set to 90% of available CPUs along with four threads. For + nodes that are running Rails with other components the worker value should be reduced + accordingly where we've found 50% achieves a good balance but this is dependent + on workload. + +1. Gitaly node requirements are dependent on customer data, specifically the number of + projects and their sizes. We recommend two nodes as an absolute minimum for HA environments + and at least four nodes should be used when supporting 50,000 or more users. + We also recommend that each Gitaly node should store no more than 5TB of data + and have the number of [`gitaly-ruby` workers](../gitaly/index.md#gitaly-ruby) + set to 20% of available CPUs. Additional nodes should be considered in conjunction + with a review of expected data size and spread based on the recommendations above. + +1. Recommended Redis setup differs depending on the size of the architecture. + For smaller architectures (less than 3,000 users) a single instance should suffice. + For medium sized installs (3,000 - 5,000) we suggest one Redis cluster for all + classes and that Redis Sentinel is hosted alongside Consul. + For larger architectures (10,000 users or more) we suggest running a separate + [Redis Cluster](../high_availability/redis.md#running-multiple-redis-clusters) for the Cache class + and another for the Queues and Shared State classes respectively. We also recommend + that you run the Redis Sentinel clusters separately for each Redis Cluster. + +1. For data objects such as LFS, Uploads, Artifacts, etc. We recommend an [Object Storage service](../object_storage.md) + over NFS where possible, due to better performance and availability. + +1. NFS can be used as an alternative for both repository data (replacing Gitaly) and + object storage but this isn't typically recommended for performance reasons. Note however it is required for + [GitLab Pages](https://gitlab.com/gitlab-org/gitlab-pages/issues/196). + +1. Our architectures have been tested and validated with [HAProxy](https://www.haproxy.org/) + as the load balancer. Although other load balancers with similar feature sets + could also be used, those load balancers have not been validated. + +1. We strongly recommend that any Gitaly or NFS nodes be set up with SSD disks over + HDD with a throughput of at least 8,000 IOPS for read operations and 2,000 IOPS for write + as these components have heavy I/O. These IOPS values are recommended only as a starter + as with time they may be adjusted higher or lower depending on the scale of your + environment's workload. If you're running the environment on a Cloud provider + you may need to refer to their documentation on how configure IOPS correctly. + +1. The architectures were built and tested with the [Intel Xeon E5 v3 (Haswell)](https://cloud.google.com/compute/docs/cpu-platforms) + CPU platform on GCP. On different hardware you may find that adjustments, either lower + or higher, are required for your CPU or Node counts accordingly. For more information, a + [Sysbench](https://github.com/akopytov/sysbench) benchmark of the CPU can be found + [here](https://gitlab.com/gitlab-org/quality/performance/-/wikis/Reference-Architectures/GCP-CPU-Benchmarks). + +1. AWS-equivalent and Azure-equivalent configurations are rough suggestions + and may change in the future. They have not yet been tested and validated. diff --git a/doc/administration/reference_architectures/img/reference-architectures.png b/doc/administration/reference_architectures/img/reference-architectures.png Binary files differnew file mode 100644 index 00000000000..e15609e78e1 --- /dev/null +++ b/doc/administration/reference_architectures/img/reference-architectures.png diff --git a/doc/administration/reference_architectures/index.md b/doc/administration/reference_architectures/index.md new file mode 100644 index 00000000000..26244368234 --- /dev/null +++ b/doc/administration/reference_architectures/index.md @@ -0,0 +1,225 @@ +--- +type: reference, concepts +--- +# Reference architectures + +<!-- TBD to be reviewed by Eric --> + +You can set up GitLab on a single server or scale it up to serve many users. +This page details the recommended Reference Architectures that were built and verified by GitLab's Quality and Support teams. + +Below is a chart representing each architecture tier and the number of users they can handle. As your number of users grow with time, it’s recommended that you scale GitLab accordingly. + + +<!-- Internal link: https://docs.google.com/spreadsheets/d/1obYP4fLKkVVDOljaI3-ozhmCiPtEeMblbBKkf2OADKs/edit#gid=1403207183 --> + +Testing on these reference architectures were performed with [GitLab's Performance Tool](https://gitlab.com/gitlab-org/quality/performance) +at specific coded workloads, and the throughputs used for testing were calculated based on sample customer data. +After selecting the reference architecture that matches your scale, refer to +[Configure GitLab to Scale](#configure-gitlab-to-scale) to see the components +involved, and how to configure them. + +Each endpoint type is tested with the following number of requests per second (RPS) per 1000 users: + +- API: 20 RPS +- Web: 2 RPS +- Git: 2 RPS + +For GitLab instances with less than 2,000 users, it's recommended that you use the [default setup](#automated-backups-core-only) +by [installing GitLab](../../install/README.md) on a single machine to minimize maintenance and resource costs. + +If your organization has more than 2,000 users, the recommendation is to scale GitLab's components to multiple +machine nodes. The machine nodes are grouped by component(s). The addition of these +nodes increases the performance and scalability of to your GitLab instance. + +When scaling GitLab, there are several factors to consider: + +- Multiple application nodes to handle frontend traffic. +- A load balancer is added in front to distribute traffic across the application nodes. +- The application nodes connects to a shared file server and PostgreSQL and Redis services on the backend. + +NOTE: **Note:** Depending on your workflow, the following recommended +reference architectures may need to be adapted accordingly. Your workload +is influenced by factors including how active your users are, +how much automation you use, mirroring, and repository/change size. Additionally the +displayed memory values are provided by [GCP machine types](https://cloud.google.com/compute/docs/machine-types). +For different cloud vendors, attempt to select options that best match the provided architecture. + +## Available reference architectures + +The following reference architectures are available: + +- [Up to 1,000 users](1k_users.md) +- [Up to 2,000 users](2k_users.md) +- [Up to 3,000 users](3k_users.md) +- [Up to 5,000 users](5k_users.md) +- [Up to 10,000 users](10k_users.md) +- [Up to 25,000 users](25k_users.md) +- [Up to 50,000 users](50k_users.md) + +## Availability Components + +GitLab comes with the following components for your use, listed from +least to most complex: + +1. [Automated backups](#automated-backups-core-only) +1. [Traffic load balancer](#traffic-load-balancer-starter-only) +1. [Zero downtime updates](#zero-downtime-updates-starter-only) +1. [Automated database failover](#automated-database-failover-premium-only) +1. [Instance level replication with GitLab Geo](#instance-level-replication-with-gitlab-geo-premium-only) + +As you implement these components, begin with a single server and then do +backups. Only after completing the first server should you proceed to the next. + +Also, not implementing extra servers for GitLab doesn't necessarily mean that you'll have +more downtime. Depending on your needs and experience level, single servers can +have more actual perceived uptime for your users. + +### Automated backups **(CORE ONLY)** + +> - Level of complexity: **Low** +> - Required domain knowledge: PostgreSQL, GitLab configurations, Git +> - Supported tiers: [GitLab Core, Starter, Premium, and Ultimate](https://about.gitlab.com/pricing/) + +This solution is appropriate for many teams that have the default GitLab installation. +With automatic backups of the GitLab repositories, configuration, and the database, +this can be an optimal solution if you don't have strict requirements. +[Automated backups](../../raketasks/backup_restore.md#configuring-cron-to-make-daily-backups) +is the least complex to setup. This provides a point-in-time recovery of a predetermined schedule. + +### Traffic load balancer **(STARTER ONLY)** + +> - Level of complexity: **Medium** +> - Required domain knowledge: HAProxy, shared storage, distributed systems +> - Supported tiers: [GitLab Starter, Premium, and Ultimate](https://about.gitlab.com/pricing/) + +This requires separating out GitLab into multiple application nodes with an added +[load balancer](../high_availability/load_balancer.md). The load balancer will distribute traffic +across GitLab application nodes. Meanwhile, each application node connects to a +shared file server and database systems on the back end. This way, if one of the +application servers fails, the workflow is not interrupted. +[HAProxy](https://www.haproxy.org/) is recommended as the load balancer. + +With this added component you have a number of advantages compared +to the default installation: + +- Increase the number of users. +- Enable zero-downtime upgrades. +- Increase availability. + +### Zero downtime updates **(STARTER ONLY)** + +> - Level of complexity: **Medium** +> - Required domain knowledge: PostgreSQL, HAProxy, shared storage, distributed systems +> - Supported tiers: [GitLab Starter, Premium, and Ultimate](https://about.gitlab.com/pricing/) + +GitLab supports [zero-downtime updates](https://docs.gitlab.com/omnibus/update/#zero-downtime-updates). +Although you can perform zero-downtime updates with a single GitLab node, the recommendation is to separate GitLab into several application nodes. +As long as at least one of each component is online and capable of handling the instance's usage load, your team's productivity will not be interrupted during the update. + +### Automated database failover **(PREMIUM ONLY)** + +> - Level of complexity: **High** +> - Required domain knowledge: PgBouncer, Repmgr, shared storage, distributed systems +> - Supported tiers: [GitLab Premium and Ultimate](https://about.gitlab.com/pricing/) + +By adding automatic failover for database systems, you can enable higher uptime +with additional database nodes. This extends the default database with +cluster management and failover policies. +[PgBouncer](../../development/architecture.md#pgbouncer) in conjunction with +[Repmgr](../high_availability/database.md) is recommended. + +### Instance level replication with GitLab Geo **(PREMIUM ONLY)** + +> - Level of complexity: **Very High** +> - Required domain knowledge: Storage replication +> - Supported tiers: [GitLab Premium and Ultimate](https://about.gitlab.com/pricing/) + +[GitLab Geo](../geo/replication/index.md) allows you to replicate your GitLab +instance to other geographical locations as a read-only fully operational instance +that can also be promoted in case of disaster. + +## Configure GitLab to scale + +The following components are the ones you need to configure in order to scale +GitLab. They are listed in the order you'll typically configure them if they are +required by your [reference architecture](#reference-architectures) of choice. + +Most of them are bundled in the GitLab deb/rpm package (called Omnibus GitLab), +but depending on your system architecture, you may require some components which are +not included in it. If required, those should be configured before +setting up components provided by GitLab. Advice on how to select the right +solution for your organization is provided in the configuration instructions +column. + +| Component | Description | Configuration instructions | Bundled with Omnibus GitLab | +|-----------|-------------|----------------------------| +| Load balancer(s) ([6](#footnotes)) | Handles load balancing, typically when you have multiple GitLab application services nodes | [Load balancer configuration](../high_availability/load_balancer.md) ([6](#footnotes)) | No | +| Object storage service ([4](#footnotes)) | Recommended store for shared data objects | [Object Storage configuration](../object_storage.md) | No | +| NFS ([5](#footnotes)) ([7](#footnotes)) | Shared disk storage service. Can be used as an alternative Object Storage. Required for GitLab Pages | [NFS configuration](../high_availability/nfs.md) | No | +| [Consul](../../development/architecture.md#consul) ([3](#footnotes)) | Service discovery and health checks/failover | [Consul configuration](../high_availability/consul.md) **(PREMIUM ONLY)** | Yes | +| [PostgreSQL](../../development/architecture.md#postgresql) | Database | [PostgreSQL configuration](https://docs.gitlab.com/omnibus/settings/database.html) | Yes | +| [PgBouncer](../../development/architecture.md#pgbouncer) | Database connection pooler | [PgBouncer configuration](../high_availability/pgbouncer.md#running-pgbouncer-as-part-of-a-non-ha-gitlab-installation) **(PREMIUM ONLY)** | Yes | +| Repmgr | PostgreSQL cluster management and failover | [PostgreSQL and Repmgr configuration](../high_availability/database.md) | Yes | +| [Redis](../../development/architecture.md#redis) ([3](#footnotes)) | Key/value store for fast data lookup and caching | [Redis configuration](../high_availability/redis.md) | Yes | +| Redis Sentinel | Redis | [Redis Sentinel configuration](../high_availability/redis.md) | Yes | +| [Gitaly](../../development/architecture.md#gitaly) ([2](#footnotes)) ([7](#footnotes)) ([10](#footnotes)) | Provides access to Git repositories | [Gitaly configuration](../gitaly/index.md#running-gitaly-on-its-own-server) | Yes | +| [Sidekiq](../../development/architecture.md#sidekiq) | Asynchronous/background jobs | [Sidekiq configuration](../high_availability/sidekiq.md) | Yes | +| [GitLab application services](../../development/architecture.md#unicorn)([1](#footnotes)) | Puma/Unicorn, Workhorse, GitLab Shell - serves front-end requests (UI, API, Git over HTTP/SSH) | [GitLab app scaling configuration](../high_availability/gitlab.md) | Yes | +| [Prometheus](../../development/architecture.md#prometheus) and [Grafana](../../development/architecture.md#grafana) | GitLab environment monitoring | [Monitoring node for scaling](../high_availability/monitoring_node.md) | Yes | + +## Footnotes + +1. In our architectures we run each GitLab Rails node using the Puma webserver + and have its number of workers set to 90% of available CPUs along with four threads. For + nodes that are running Rails with other components the worker value should be reduced + accordingly where we've found 50% achieves a good balance but this is dependent + on workload. + +1. Gitaly node requirements are dependent on customer data, specifically the number of + projects and their sizes. We recommend two nodes as an absolute minimum, + and at least four nodes should be used when supporting 50,000 or more users. + We also recommend that each Gitaly node should store no more than 5TB of data + and have the number of [`gitaly-ruby` workers](../gitaly/index.md#gitaly-ruby) + set to 20% of available CPUs. Additional nodes should be considered in conjunction + with a review of expected data size and spread based on the recommendations above. + +1. Recommended Redis setup differs depending on the size of the architecture. + For smaller architectures (less than 3,000 users) a single instance should suffice. + For medium sized installs (3,000 - 5,000) we suggest one Redis cluster for all + classes and that Redis Sentinel is hosted alongside Consul. + For larger architectures (10,000 users or more) we suggest running a separate + [Redis Cluster](../high_availability/redis.md#running-multiple-redis-clusters) for the Cache class + and another for the Queues and Shared State classes respectively. We also recommend + that you run the Redis Sentinel clusters separately for each Redis Cluster. + +1. For data objects such as LFS, Uploads, Artifacts, etc. We recommend an [Object Storage service](../object_storage.md) + over NFS where possible, due to better performance. + +1. NFS can be used as an alternative for object storage but this isn't typically + recommended for performance reasons. Note however it is required for [GitLab + Pages](https://gitlab.com/gitlab-org/gitlab-pages/issues/196). + +1. Our architectures have been tested and validated with [HAProxy](https://www.haproxy.org/) + as the load balancer. Although other load balancers with similar feature sets + could also be used, those load balancers have not been validated. + +1. We strongly recommend that any Gitaly or NFS nodes be set up with SSD disks over + HDD with a throughput of at least 8,000 IOPS for read operations and 2,000 IOPS for write + as these components have heavy I/O. These IOPS values are recommended only as a starter + as with time they may be adjusted higher or lower depending on the scale of your + environment's workload. If you're running the environment on a Cloud provider + you may need to refer to their documentation on how configure IOPS correctly. + +1. The architectures were built and tested with the [Intel Xeon E5 v3 (Haswell)](https://cloud.google.com/compute/docs/cpu-platforms) + CPU platform on GCP. On different hardware you may find that adjustments, either lower + or higher, are required for your CPU or Node counts accordingly. For more information, a + [Sysbench](https://github.com/akopytov/sysbench) benchmark of the CPU can be found + [here](https://gitlab.com/gitlab-org/quality/performance/-/wikis/Reference-Architectures/GCP-CPU-Benchmarks). + +1. AWS-equivalent and Azure-equivalent configurations are rough suggestions + and may change in the future. They have not yet been tested and validated. + +1. From GitLab 13.0, using NFS for Git repositories is deprecated. In GitLab + 14.0, support for NFS for Git repositories is scheduled to be removed. + Upgrade to [Gitaly Cluster](../gitaly/praefect.md) as soon as possible. diff --git a/doc/administration/reply_by_email_postfix_setup.md b/doc/administration/reply_by_email_postfix_setup.md index 8a24514aec2..86854a2a7b6 100644 --- a/doc/administration/reply_by_email_postfix_setup.md +++ b/doc/administration/reply_by_email_postfix_setup.md @@ -1,7 +1,7 @@ # Set up Postfix for incoming email This document will take you through the steps of setting up a basic Postfix mail -server with IMAP authentication on Ubuntu, to be used with [incoming email]. +server with IMAP authentication on Ubuntu, to be used with [incoming email](incoming_email.md). The instructions make the assumption that you will be using the email address `incoming@gitlab.example.com`, that is, username `incoming` on host `gitlab.example.com`. Don't forget to change it to your actual host when executing the example code snippets. @@ -333,10 +333,8 @@ Courier, which we will install later to add IMAP authentication, requires mailbo ## Done -If all the tests were successful, Postfix is all set up and ready to receive email! Continue with the [incoming email] guide to configure GitLab. +If all the tests were successful, Postfix is all set up and ready to receive email! Continue with the [incoming email](incoming_email.md) guide to configure GitLab. --- _This document was adapted from <https://help.ubuntu.com/community/PostfixBasicSetupHowto>, by contributors to the Ubuntu documentation wiki._ - -[incoming email]: incoming_email.md diff --git a/doc/administration/repository_checks.md b/doc/administration/repository_checks.md index a647bc82660..6d9ab723d2f 100644 --- a/doc/administration/repository_checks.md +++ b/doc/administration/repository_checks.md @@ -1,14 +1,14 @@ # Repository checks -> [Introduced][ce-3232] in GitLab 8.7. +> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/3232) in GitLab 8.7. -Git has a built-in mechanism, [`git fsck`][git-fsck], to verify the +Git has a built-in mechanism, [`git fsck`](https://git-scm.com/docs/git-fsck), to verify the integrity of all data committed to a repository. GitLab administrators can trigger such a check for a project via the project page under the admin panel. The checks run asynchronously so it may take a few minutes before the check result is visible on the project admin page. If the -checks failed you can see their output on the admin log page under -'repocheck.log'. +checks failed you can see their output on in the [`repocheck.log` +file.](logs.md#repochecklog) NOTE: **Note:** It is OFF by default because it still causes too many false alarms. @@ -31,17 +31,11 @@ panel. ## What to do if a check failed If the repository check fails for some repository you should look up the error -in `repocheck.log`: +in the [`repocheck.log` file](logs.md#repochecklog) on disk: -- in the [admin panel](logs.md#repochecklog) -- or on disk, see: - - `/var/log/gitlab/gitlab-rails` for Omnibus installations - - `/home/git/gitlab/log` for installations from source +- `/var/log/gitlab/gitlab-rails` for Omnibus installations +- `/home/git/gitlab/log` for installations from source If the periodic repository check causes false alarms, you can clear all repository check states by navigating to **Admin Area > Settings > Repository** (`/admin/application_settings/repository`) and clicking **Clear all repository checks**. - ---- -[ce-3232]: https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/3232 "Auto git fsck" -[git-fsck]: https://git-scm.com/docs/git-fsck "git fsck documentation" diff --git a/doc/administration/repository_storage_paths.md b/doc/administration/repository_storage_paths.md index 857fb0b6a90..283401dafff 100644 --- a/doc/administration/repository_storage_paths.md +++ b/doc/administration/repository_storage_paths.md @@ -1,6 +1,6 @@ # Repository storage paths -> [Introduced][ce-4578] in GitLab 8.10. +> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/4578) in GitLab 8.10. GitLab allows you to define multiple repository storage paths (sometimes called storage shards) to distribute the storage load between several mount points. @@ -34,7 +34,7 @@ storage2: ## Configure GitLab > **Warning:** -> In order for [backups] to work correctly, the storage path must **not** be a +> In order for [backups](../raketasks/backup_restore.md) to work correctly, the storage path must **not** be a > mount point and the GitLab user should have correct permissions for the parent > directory of the path. In Omnibus GitLab this is taken care of automatically, > but for source installations you should be extra careful. @@ -47,7 +47,7 @@ storage2: > `gitlab.yml`. > > This little detail matters because while restoring a backup, the current -> contents of `/home/git/repositories` [are moved to][raketask] `/home/git/repositories.old`, +> contents of `/home/git/repositories` [are moved to](https://gitlab.com/gitlab-org/gitlab/blob/033e5423a2594e08a7ebcd2379bd2331f4c39032/lib/backup/repository.rb#L54-56) `/home/git/repositories.old`, > so if `/home/git/repositories` is the mount point, then `mv` would be moving > things between mount points, and bad things could happen. Ideally, > `/home/git` would be the mount point, so then things would be moving within the @@ -79,10 +79,10 @@ NOTE: **Note:** This example uses NFS. We do not recommend using EFS for storage path: /mnt/nfs2/repositories ``` -1. [Restart GitLab][restart-gitlab] for the changes to take effect. +1. [Restart GitLab](restart_gitlab.md#installations-from-source) for the changes to take effect. >**Note:** -The [`gitlab_shell: repos_path` entry][repospath] in `gitlab.yml` will be +The [`gitlab_shell: repos_path` entry](https://gitlab.com/gitlab-org/gitlab-foss/-/blob/8-9-stable/config/gitlab.yml.example#L457) in `gitlab.yml` will be deprecated and replaced by `repositories: storages` in the future, so if you are upgrading from a version prior to 8.10, make sure to add the configuration as described in the step above. After you make the changes and confirm they are @@ -114,9 +114,3 @@ Repository storage > Storage nodes for new repositories**. Beginning with GitLab 8.13.4, multiple paths can be chosen. New repositories will be randomly placed on one of the selected paths. - -[ce-4578]: https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/4578 -[restart-gitlab]: restart_gitlab.md#installations-from-source -[backups]: ../raketasks/backup_restore.md -[raketask]: https://gitlab.com/gitlab-org/gitlab/blob/033e5423a2594e08a7ebcd2379bd2331f4c39032/lib/backup/repository.rb#L54-56 -[repospath]: https://gitlab.com/gitlab-org/gitlab-foss/-/blob/8-9-stable/config/gitlab.yml.example#L457 diff --git a/doc/administration/repository_storage_types.md b/doc/administration/repository_storage_types.md index 2e2ed431c8b..a95178c01e2 100644 --- a/doc/administration/repository_storage_types.md +++ b/doc/administration/repository_storage_types.md @@ -1,16 +1,15 @@ -# Repository Storage Types +# Repository storage types **(CORE ONLY)** -> [Introduced][ce-28283] in GitLab 10.0. - -Two different storage layouts can be used -to store the repositories on disk and their characteristics. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/issues/28283) in GitLab 10.0. +> - Hashed storage became the default for new installations in GitLab 12.0 +> - Hashed storage is enabled by default for new and renamed projects in GitLab 13.0. GitLab can be configured to use one or multiple repository storage paths/shard locations that can be: - Mounted to the local disk - Exposed as an NFS shared volume -- Accessed via [Gitaly] on its own machine. +- Accessed via [Gitaly](gitaly/index.md) on its own machine. In GitLab, this is configured in `/etc/gitlab/gitlab.rb` by the `git_data_dirs({})` configuration hash. The storage layouts discussed here will apply to any shard @@ -20,40 +19,17 @@ The `default` repository shard that is available in any installations that haven't customized it, points to the local folder: `/var/opt/gitlab/git-data`. Anything discussed below is expected to be part of that folder. -## Legacy Storage - -Legacy Storage is the storage behavior prior to version 10.0. For historical -reasons, GitLab replicated the same mapping structure from the projects URLs: +## Hashed storage -- Project's repository: `#{namespace}/#{project_name}.git` -- Project's wiki: `#{namespace}/#{project_name}.wiki.git` +NOTE: **Note:** +In GitLab 13.0, hashed storage is enabled by default and the legacy storage is +deprecated. Support for legacy storage will be removed in GitLab 14.0. +If you haven't migrated yet, check the +[migration instructions](raketasks/storage.md#migrate-to-hashed-storage). +The option to choose between hashed and legacy storage in the admin area has +been disabled. -This structure made it simple to migrate from existing solutions to GitLab and -easy for Administrators to find where the repository is stored. - -On the other hand this has some drawbacks: - -Storage location will concentrate huge amount of top-level namespaces. The -impact can be reduced by the introduction of -[multiple storage paths](repository_storage_paths.md). - -Because backups are a snapshot of the same URL mapping, if you try to recover a -very old backup, you need to verify whether any project has taken the place of -an old removed or renamed project sharing the same URL. This means that -`mygroup/myproject` from your backup may not be the same original project that -is at that same URL today. - -Any change in the URL will need to be reflected on disk (when groups / users or -projects are renamed). This can add a lot of load in big installations, -especially if using any type of network based filesystem. - -## Hashed Storage - -CAUTION: **Important:** -Geo requires Hashed Storage since 12.0. If you haven't migrated yet, -check the [migration instructions](#how-to-migrate-to-hashed-storage) ASAP. - -Hashed Storage is the new storage behavior we rolled out with 10.0. Instead +Hashed storage is the storage behavior we rolled out with 10.0. Instead of coupling project URL and the folder structure where the repository will be stored on disk, we are coupling a hash, based on the project's ID. This makes the folder structure immutable, and therefore eliminates any requirement to @@ -124,7 +100,7 @@ GitLab server. For example, on a default Omnibus installation this would be `/var/opt/gitlab/git-data/repositories/@hashed/b1/7e/b17ef6d19c7a5b1ee83b907c595526dcb1eb06db8227d650d5dda0a9f4ce8cd9.git` with `.git` from the end of the directory name removed. -The output includes the project id and the project name: +The output includes the project ID and the project name: ```plaintext => #<Project id:16 it/supportteam/ticketsystem> @@ -134,6 +110,11 @@ The output includes the project id and the project name: > [Introduced](https://gitlab.com/gitlab-org/gitaly/issues/1606) in GitLab 12.1. +DANGER: **Danger:** +Do not run `git prune` or `git gc` in pool repositories! This can +cause data loss in "real" repositories that depend on the pool in +question. + Forks of public projects are deduplicated by creating a third repository, the object pool, containing the objects from the source project. Using `objects/info/alternates`, the source project and forks use the object pool for @@ -145,71 +126,15 @@ when housekeeping is run on the source project. "@pools/#{hash[0..1]}/#{hash[2..3]}/#{hash}.git" ``` -DANGER: **Danger:** -Do not run `git prune` or `git gc` in pool repositories! This can -cause data loss in "real" repositories that depend on the pool in -question. - -### How to migrate to Hashed Storage - -To start a migration, enable Hashed Storage for new projects: - -1. Go to **Admin > Settings > Repository** and expand the **Repository Storage** section. -1. Select the **Use hashed storage paths for newly created and renamed projects** checkbox. - -Check if the change breaks any existing integration you may have that -either runs on the same machine as your repositories are located, or may login to that machine -to access data (for example, a remote backup solution). - -To schedule a complete rollout, see the -[Rake task documentation for storage migration][rake/migrate-to-hashed] for instructions. - -If you do have any existing integration, you may want to do a small rollout first, -to validate. You can do so by specifying a range with the operation. - -This is an example of how to limit the rollout to Project IDs 50 to 100, running in -an Omnibus GitLab installation: - -```shell -sudo gitlab-rake gitlab:storage:migrate_to_hashed ID_FROM=50 ID_TO=100 -``` - -Check the [documentation][rake/migrate-to-hashed] for additional information and instructions for -source-based installation. - -#### Rollback - -Similar to the migration, to disable Hashed Storage for new -projects: - -1. Go to **Admin > Settings > Repository** and expand the **Repository Storage** section. -1. Uncheck the **Use hashed storage paths for newly created and renamed projects** checkbox. - -To schedule a complete rollback, see the -[Rake task documentation for storage rollback](raketasks/storage.md#rollback-from-hashed-storage-to-legacy-storage) for instructions. - -The rollback task also supports specifying a range of Project IDs. Here is an example -of limiting the rollout to Project IDs 50 to 100, in an Omnibus GitLab installation: - -```shell -sudo gitlab-rake gitlab:storage:rollback_to_legacy ID_FROM=50 ID_TO=100 -``` +### Hashed storage coverage migration -If you have a Geo setup, please note that the rollback will not be reflected automatically -on the **secondary** node. You may need to wait for a backfill operation to kick-in and remove -the remaining repositories from the special `@hashed/` folder manually. - -### Hashed Storage coverage - -We are incrementally moving every storable object in GitLab to the Hashed -Storage pattern. You can check the current coverage status below (and also see -the [issue][ce-2821]). - -Note that things stored in an S3 compatible endpoint will not have the downsides +Files stored in an S3 compatible endpoint will not have the downsides mentioned earlier, if they are not prefixed with `#{namespace}/#{project_name}`, which is true for CI Cache and LFS Objects. -| Storable Object | Legacy Storage | Hashed Storage | S3 Compatible | GitLab Version | +In the table below, you can find the coverage of the migration to the hashed storage. + +| Storable Object | Legacy storage | Hashed storage | S3 Compatible | GitLab Version | | --------------- | -------------- | -------------- | ------------- | -------------- | | Repository | Yes | Yes | - | 10.0 | | Attachments | Yes | Yes | - | 10.2 | @@ -222,18 +147,16 @@ which is true for CI Cache and LFS Objects. | LFS Objects | Yes | Similar | Yes | 10.0 / 10.7 | | Repository pools| No | Yes | - | 11.6 | -#### Implementation Details - -##### Avatars +#### Avatars Each file is stored in a folder with its `id` from the database. The filename is always `avatar.png` for user avatars. When avatar is replaced, `Upload` model is destroyed and a new one takes place with different `id`. -##### CI Artifacts +#### CI artifacts CI Artifacts are S3 compatible since **9.4** (GitLab Premium), and available in GitLab Core since **10.6**. -##### LFS Objects +#### LFS objects [LFS Objects in GitLab](../topics/git/lfs/index.md) implement a similar storage pattern using 2 chars, 2 level folders, following Git's own implementation: @@ -247,7 +170,38 @@ storage pattern using 2 chars, 2 level folders, following Git's own implementati LFS objects are also [S3 compatible](lfs/index.md#storing-lfs-objects-in-remote-object-storage). -[ce-2821]: https://gitlab.com/gitlab-com/infrastructure/issues/2821 -[ce-28283]: https://gitlab.com/gitlab-org/gitlab-foss/issues/28283 -[rake/migrate-to-hashed]: raketasks/storage.md#migrate-existing-projects-to-hashed-storage -[gitaly]: gitaly/index.md +## Legacy storage + +NOTE: **Deprecated:** +In GitLab 13.0, hashed storage is enabled by default and the legacy storage is +deprecated. If you haven't migrated yet, check the +[migration instructions](raketasks/storage.md#migrate-to-hashed-storage). +Support for legacy storage will be removed in GitLab 14.0. If you're on GitLab +13.0 and later, switching new projects to legacy storage is not possible. +The option to choose between hashed and legacy storage in the admin area has +been disabled. + +Legacy storage is the storage behavior prior to version 10.0. For historical +reasons, GitLab replicated the same mapping structure from the projects URLs: + +- Project's repository: `#{namespace}/#{project_name}.git` +- Project's wiki: `#{namespace}/#{project_name}.wiki.git` + +This structure made it simple to migrate from existing solutions to GitLab and +easy for Administrators to find where the repository is stored. + +On the other hand this has some drawbacks: + +Storage location will concentrate huge amount of top-level namespaces. The +impact can be reduced by the introduction of +[multiple storage paths](repository_storage_paths.md). + +Because backups are a snapshot of the same URL mapping, if you try to recover a +very old backup, you need to verify whether any project has taken the place of +an old removed or renamed project sharing the same URL. This means that +`mygroup/myproject` from your backup may not be the same original project that +is at that same URL today. + +Any change in the URL will need to be reflected on disk (when groups / users or +projects are renamed). This can add a lot of load in big installations, +especially if using any type of network based filesystem. diff --git a/doc/administration/restart_gitlab.md b/doc/administration/restart_gitlab.md index 907d7bb307a..57f53fd6cc4 100644 --- a/doc/administration/restart_gitlab.md +++ b/doc/administration/restart_gitlab.md @@ -12,18 +12,18 @@ If you want the TL;DR versions, jump to: ## Omnibus installations -If you have used the [Omnibus packages][omnibus-dl] to install GitLab, then +If you have used the [Omnibus packages](https://about.gitlab.com/install/) to install GitLab, then you should already have `gitlab-ctl` in your `PATH`. `gitlab-ctl` interacts with the Omnibus packages and can be used to restart the -GitLab Rails application (Unicorn) as well as the other components, like: +GitLab Rails application (Puma) as well as the other components, like: - GitLab Workhorse - Sidekiq - PostgreSQL (if you are using the bundled one) - NGINX (if you are using the bundled one) - Redis (if you are using the bundled one) -- [Mailroom][] +- [Mailroom](reply_by_email.md) - Logrotate ### Omnibus GitLab restart @@ -45,7 +45,7 @@ ok: run: nginx: (pid 11309) 0s ok: run: postgresql: (pid 11316) 1s ok: run: redis: (pid 11325) 0s ok: run: sidekiq: (pid 11331) 1s -ok: run: unicorn: (pid 11338) 0s +ok: run: puma: (pid 11338) 0s ``` To restart a component separately, you can append its service name to the @@ -86,7 +86,7 @@ sudo gitlab-ctl reconfigure Reconfiguring GitLab should occur in the event that something in its configuration (`/etc/gitlab/gitlab.rb`) has changed. -When you run this command, [Chef], the underlying configuration management +When you run this command, [Chef](https://www.chef.io/products/chef-infra/), the underlying configuration management application that powers Omnibus GitLab, will make sure that all things like directories, permissions, and services are in place and in the same shape that they were initially shipped. @@ -101,7 +101,7 @@ depend on those files. ## Installations from source If you have followed the official installation guide to [install GitLab from -source][install], run the following command to restart GitLab: +source](../install/installation.md), run the following command to restart GitLab: ```shell sudo service gitlab restart @@ -110,46 +110,39 @@ sudo service gitlab restart The output should be similar to this: ```plaintext -Shutting down GitLab Unicorn +Shutting down GitLab Puma Shutting down GitLab Sidekiq Shutting down GitLab Workhorse Shutting down GitLab MailRoom ... GitLab is not running. -Starting GitLab Unicorn +Starting GitLab Puma Starting GitLab Sidekiq Starting GitLab Workhorse Starting GitLab MailRoom ... -The GitLab Unicorn web server with pid 28059 is running. +The GitLab Puma web server with pid 28059 is running. The GitLab Sidekiq job dispatcher with pid 28176 is running. The GitLab Workhorse with pid 28122 is running. The GitLab MailRoom email processor with pid 28114 is running. GitLab and all its components are up and running. ``` -This should restart Unicorn, Sidekiq, GitLab Workhorse, and [Mailroom][] +This should restart Puma, Sidekiq, GitLab Workhorse, and [Mailroom](reply_by_email.md) (if enabled). The init service file that does all the magic can be found on your server in `/etc/init.d/gitlab`. --- If you are using other init systems, like systemd, you can check the -[GitLab Recipes][gl-recipes] repository for some unofficial services. These are +[GitLab Recipes](https://gitlab.com/gitlab-org/gitlab-recipes/tree/master/init) repository for some unofficial services. These are **not** officially supported so use them at your own risk. -[omnibus-dl]: https://about.gitlab.com/install/ "Download the Omnibus packages" -[install]: ../install/installation.md "Documentation to install GitLab from source" -[mailroom]: reply_by_email.md "Used for replying by email in GitLab issues and merge requests" -[chef]: https://www.chef.io/products/chef-infra/ "Chef official website" -[src-service]: https://gitlab.com/gitlab-org/gitlab/blob/master/lib/support/init.d/gitlab "GitLab init service file" -[gl-recipes]: https://gitlab.com/gitlab-org/gitlab-recipes/tree/master/init "GitLab Recipes repository" - ## Helm chart installations There is no single command to restart the entire GitLab application installed via the [cloud native Helm Chart](https://docs.gitlab.com/charts/). Usually, it should be -enough to restart a specific component separately (for example, `gitaly`, `unicorn`, +enough to restart a specific component separately (for example, `gitaly`, `puma`, `workhorse`, or `gitlab-shell`) by deleting all the pods related to it: ```shell diff --git a/doc/administration/scaling/index.md b/doc/administration/scaling/index.md index ec7492883cc..748373c8941 100644 --- a/doc/administration/scaling/index.md +++ b/doc/administration/scaling/index.md @@ -1,256 +1,5 @@ --- -type: reference, concepts +redirect_to: ../reference_architectures/index.md --- -# Scaling - -GitLab supports a number of scaling options to ensure that your self-managed -instance is able to scale out to meet your organization's needs when scaling up -a single-box GitLab installation is no longer practical or feasible. - -Please consult our [high availability documentation](../availability/index.md) -if your organization requires fault tolerance and redundancy features, such as -automatic database system failover. - -## GitLab components and scaling instructions - -Here's a list of components directly provided by Omnibus GitLab or installed as -part of a source installation and their configuration instructions for scaling. - -| Component | Description | Configuration instructions | -|-----------|-------------|----------------------------| -| [PostgreSQL](../../development/architecture.md#postgresql) | Database | [PostgreSQL configuration](https://docs.gitlab.com/omnibus/settings/database.html) | -| [Redis](../../development/architecture.md#redis) | Key/value store for fast data lookup and caching | [Redis configuration](../high_availability/redis.md) | -| [GitLab application services](../../development/architecture.md#unicorn) | Unicorn/Puma, Workhorse, GitLab Shell - serves front-end requests (UI, API, Git over HTTP/SSH) | [GitLab app scaling configuration](../high_availability/gitlab.md) | -| [PgBouncer](../../development/architecture.md#pgbouncer) | Database connection pooler | [PgBouncer configuration](../high_availability/pgbouncer.md#running-pgbouncer-as-part-of-a-non-ha-gitlab-installation) **(PREMIUM ONLY)** | -| [Sidekiq](../../development/architecture.md#sidekiq) | Asynchronous/background jobs | [Sidekiq configuration](../high_availability/sidekiq.md) | -| [Gitaly](../../development/architecture.md#gitaly) | Provides access to Git repositories | [Gitaly configuration](../gitaly/index.md#running-gitaly-on-its-own-server) | -| [Prometheus](../../development/architecture.md#prometheus) and [Grafana](../../development/architecture.md#grafana) | GitLab environment monitoring | [Monitoring node for scaling](../high_availability/monitoring_node.md) | - -## Third-party services used for scaling - -Here's a list of third-party services you may require as part of scaling GitLab. -The services can be provided by numerous applications or vendors and further -advice is given on how best to select the right choice for your organization's -needs. - -| Component | Description | Configuration instructions | -|-----------|-------------|----------------------------| -| Load balancer(s) | Handles load balancing, typically when you have multiple GitLab application services nodes | [Load balancer configuration](../high_availability/load_balancer.md) | -| Object storage service | Recommended store for shared data objects | [Cloud Object Storage configuration](../high_availability/object_storage.md) | -| NFS | Shared disk storage service. Can be used as an alternative for Gitaly or Object Storage. Required for GitLab Pages | [NFS configuration](../high_availability/nfs.md) | - -## Reference architectures - -- 1 - 1000 Users: A single-node [Omnibus](https://docs.gitlab.com/omnibus/) setup with frequent backups. Refer to the [Single-node Omnibus installation](#single-node-installation) section below. -- 1000 to 50000+ Users: A [Scaled-out Omnibus installation with multiple servers](#multi-node-installation-scaled-out-for-availability), it can be with or without high-availability components applied. - - To decide the level of Availability please refer to our [Availability](../availability/index.md) page. - -### Single-node installation - -This solution is appropriate for many teams that have a single server at their disposal. With automatic backup of the GitLab repositories, configuration, and the database, this can be an optimal solution if you don't have strict availability requirements. - -You can also optionally configure GitLab to use an [external PostgreSQL service](../external_database.md) -or an [external object storage service](../high_availability/object_storage.md) for added -performance and reliability at a relatively low complexity cost. - -References: - -- [Installation Docs](../../install/README.md) -- [Backup/Restore Docs](https://docs.gitlab.com/omnibus/settings/backups.html#backup-and-restore-omnibus-gitlab-configuration) - -### Multi-node installation (scaled out for availability) - -This solution is appropriate for teams that are starting to scale out when -scaling up is no longer meeting their needs. In this configuration, additional application nodes will handle frontend traffic, with a load balancer in front to distribute traffic across those nodes. Meanwhile, each application node connects to a shared file server and PostgreSQL and Redis services on the back end. - -The additional application servers adds limited fault tolerance to your GitLab -instance. As long as one application node is online and capable of handling the -instance's usage load, your team's productivity will not be interrupted. Having -multiple application nodes also enables [zero-downtime updates](https://docs.gitlab.com/omnibus/update/#zero-downtime-updates). - -References: - -- [Configure your load balancer for GitLab](../high_availability/load_balancer.md) -- [Configure your NFS server to work with GitLab](../high_availability/nfs.md) -- [Configure packaged PostgreSQL server to listen on TCP/IP](https://docs.gitlab.com/omnibus/settings/database.html#configure-packaged-postgresql-server-to-listen-on-tcpip) -- [Setting up a Redis-only server](https://docs.gitlab.com/omnibus/settings/redis.html#setting-up-a-redis-only-server) - -In this section we'll detail the Reference Architectures that can support large numbers -of users. These were built, tested and verified by our Quality and Support teams. - -Testing was done with our GitLab Performance Tool at specific coded workloads, and the -throughputs used for testing were calculated based on sample customer data. We -test each endpoint type with the following number of requests per second (RPS) -per 1000 users: - -- API: 20 RPS -- Web: 2 RPS -- Git: 2 RPS - -NOTE: **Note:** Note that depending on your workflow the below recommended -reference architectures may need to be adapted accordingly. Your workload -is influenced by factors such as - but not limited to - how active your users are, -how much automation you use, mirroring, and repo/change size. Additionally the -shown memory values are given directly by [GCP machine types](https://cloud.google.com/compute/docs/machine-types). -On different cloud vendors a best effort like for like can be used. - -#### 2,000 user configuration - -- **Supported users (approximate):** 2,000 -- **Test RPS rates:** API: 40 RPS, Web: 4 RPS, Git: 4 RPS -- **Known issues:** [List of known performance issues](https://gitlab.com/gitlab-org/gitlab/issues?label_name%5B%5D=Quality%3Aperformance-issues) - -| Service | Nodes | Configuration[^8] | GCP type | AWS type[^9] | -| ----------------------------|-------|-----------------------|---------------|--------------| -| GitLab Rails[^1] | 3 | 8 vCPU, 7.2GB Memory | n1-highcpu-8 | c5.2xlarge | -| PostgreSQL | 3 | 2 vCPU, 7.5GB Memory | n1-standard-2 | m5.large | -| PgBouncer | 3 | 2 vCPU, 1.8GB Memory | n1-highcpu-2 | c5.large | -| Gitaly[^2] [^5] [^7] | X | 4 vCPU, 15GB Memory | n1-standard-4 | m5.xlarge | -| Redis[^3] | 3 | 2 vCPU, 7.5GB Memory | n1-standard-2 | m5.large | -| Consul + Sentinel[^3] | 3 | 2 vCPU, 1.8GB Memory | n1-highcpu-2 | c5.large | -| Sidekiq | 4 | 2 vCPU, 7.5GB Memory | n1-standard-2 | m5.large | -| Cloud Object Storage[^4] | - | - | - | - | -| NFS Server[^5] [^7] | 1 | 4 vCPU, 3.6GB Memory | n1-highcpu-4 | c5.xlarge | -| Monitoring node | 1 | 2 vCPU, 1.8GB Memory | n1-highcpu-2 | c5.large | -| External load balancing node[^6] | 1 | 2 vCPU, 1.8GB Memory | n1-highcpu-2 | c5.large | -| Internal load balancing node[^6] | 1 | 2 vCPU, 1.8GB Memory | n1-highcpu-2 | c5.large | - -#### 5,000 user configuration - -- **Supported users (approximate):** 5,000 -- **Test RPS rates:** API: 100 RPS, Web: 10 RPS, Git: 10 RPS -- **Known issues:** [List of known performance issues](https://gitlab.com/gitlab-org/gitlab/issues?label_name%5B%5D=Quality%3Aperformance-issues) - -| Service | Nodes | Configuration[^8] | GCP type | AWS type[^9] | -| ----------------------------|-------|------------------------|---------------|--------------| -| GitLab Rails[^1] | 3 | 16 vCPU, 14.4GB Memory | n1-highcpu-16 | c5.4xlarge | -| PostgreSQL | 3 | 2 vCPU, 7.5GB Memory | n1-standard-2 | m5.large | -| PgBouncer | 3 | 2 vCPU, 1.8GB Memory | n1-highcpu-2 | c5.large | -| Gitaly[^2] [^5] [^7] | X | 8 vCPU, 30GB Memory | n1-standard-8 | m5.2xlarge | -| Redis[^3] | 3 | 2 vCPU, 7.5GB Memory | n1-standard-2 | m5.large | -| Consul + Sentinel[^3] | 3 | 2 vCPU, 1.8GB Memory | n1-highcpu-2 | c5.large | -| Sidekiq | 4 | 2 vCPU, 7.5GB Memory | n1-standard-2 | m5.large | -| Cloud Object Storage[^4] | - | - | - | - | -| NFS Server[^5] [^7] | 1 | 4 vCPU, 3.6GB Memory | n1-highcpu-4 | c5.xlarge | -| Monitoring node | 1 | 2 vCPU, 1.8GB Memory | n1-highcpu-2 | c5.large | -| External load balancing node[^6] | 1 | 2 vCPU, 1.8GB Memory | n1-highcpu-2 | c5.large | -| Internal load balancing node[^6] | 1 | 2 vCPU, 1.8GB Memory | n1-highcpu-2 | c5.large | - -#### 10,000 user configuration - -- **Supported users (approximate):** 10,000 -- **Test RPS rates:** API: 200 RPS, Web: 20 RPS, Git: 20 RPS -- **Known issues:** [List of known performance issues](https://gitlab.com/gitlab-org/gitlab/issues?label_name%5B%5D=Quality%3Aperformance-issues) - -| Service | Nodes | GCP Configuration[^8] | GCP type | AWS type[^9] | -| ----------------------------|-------|------------------------|----------------|--------------| -| GitLab Rails[^1] | 3 | 32 vCPU, 28.8GB Memory | n1-highcpu-32 | c5.9xlarge | -| PostgreSQL | 3 | 4 vCPU, 15GB Memory | n1-standard-4 | m5.xlarge | -| PgBouncer | 3 | 2 vCPU, 1.8GB Memory | n1-highcpu-2 | c5.large | -| Gitaly[^2] [^5] [^7] | X | 16 vCPU, 60GB Memory | n1-standard-16 | m5.4xlarge | -| Redis[^3] - Cache | 3 | 4 vCPU, 15GB Memory | n1-standard-4 | m5.xlarge | -| Redis[^3] - Queues / Shared State | 3 | 4 vCPU, 15GB Memory | n1-standard-4 | m5.xlarge | -| Redis Sentinel[^3] - Cache | 3 | 1 vCPU, 1.7GB Memory | g1-small | t2.small | -| Redis Sentinel[^3] - Queues / Shared State | 3 | 1 vCPU, 1.7GB Memory | g1-small | t2.small | -| Consul | 3 | 2 vCPU, 1.8GB Memory | n1-highcpu-2 | c5.large | -| Sidekiq | 4 | 4 vCPU, 15GB Memory | n1-standard-4 | m5.xlarge | -| Cloud Object Storage[^4] | - | - | - | - | -| NFS Server[^5] [^7] | 1 | 4 vCPU, 3.6GB Memory | n1-highcpu-4 | c5.xlarge | -| Monitoring node | 1 | 4 vCPU, 3.6GB Memory | n1-highcpu-4 | c5.xlarge | -| External load balancing node[^6] | 1 | 2 vCPU, 1.8GB Memory | n1-highcpu-2 | c5.large | -| Internal load balancing node[^6] | 1 | 2 vCPU, 1.8GB Memory | n1-highcpu-2 | c5.large | - -#### 25,000 user configuration - -- **Supported users (approximate):** 25,000 -- **Test RPS rates:** API: 500 RPS, Web: 50 RPS, Git: 50 RPS -- **Known issues:** [List of known performance issues](https://gitlab.com/gitlab-org/gitlab/issues?label_name%5B%5D=Quality%3Aperformance-issues) - -| Service | Nodes | Configuration[^8] | GCP type | AWS type[^9] | -| ----------------------------|-------|------------------------|----------------|--------------| -| GitLab Rails[^1] | 5 | 32 vCPU, 28.8GB Memory | n1-highcpu-32 | c5.9xlarge | -| PostgreSQL | 3 | 8 vCPU, 30GB Memory | n1-standard-8 | m5.2xlarge | -| PgBouncer | 3 | 2 vCPU, 1.8GB Memory | n1-highcpu-2 | c5.large | -| Gitaly[^2] [^5] [^7] | X | 32 vCPU, 120GB Memory | n1-standard-32 | m5.8xlarge | -| Redis[^3] - Cache | 3 | 4 vCPU, 15GB Memory | n1-standard-4 | m5.xlarge | -| Redis[^3] - Queues / Shared State | 3 | 4 vCPU, 15GB Memory | n1-standard-4 | m5.xlarge | -| Redis Sentinel[^3] - Cache | 3 | 1 vCPU, 1.7GB Memory | g1-small | t2.small | -| Redis Sentinel[^3] - Queues / Shared State | 3 | 1 vCPU, 1.7GB Memory | g1-small | t2.small | -| Consul | 3 | 2 vCPU, 1.8GB Memory | n1-highcpu-2 | c5.large | -| Sidekiq | 4 | 4 vCPU, 15GB Memory | n1-standard-4 | m5.xlarge | -| Cloud Object Storage[^4] | - | - | - | - | -| NFS Server[^5] [^7] | 1 | 4 vCPU, 3.6GB Memory | n1-highcpu-4 | c5.xlarge | -| Monitoring node | 1 | 4 vCPU, 3.6GB Memory | n1-highcpu-4 | c5.xlarge | -| External load balancing node[^6] | 1 | 2 vCPU, 1.8GB Memory | n1-highcpu-2 | c5.large | -| Internal load balancing node[^6] | 1 | 4 vCPU, 3.6GB Memory | n1-highcpu-4 | c5.xlarge | - -#### 50,000 user configuration - -- **Supported users (approximate):** 50,000 -- **Test RPS rates:** API: 1000 RPS, Web: 100 RPS, Git: 100 RPS -- **Known issues:** [List of known performance issues](https://gitlab.com/gitlab-org/gitlab/issues?label_name%5B%5D=Quality%3Aperformance-issues) - -| Service | Nodes | Configuration[^8] | GCP type | AWS type[^9] | -| ----------------------------|-------|------------------------|----------------|--------------| -| GitLab Rails[^1] | 12 | 32 vCPU, 28.8GB Memory | n1-highcpu-32 | c5.9xlarge | -| PostgreSQL | 3 | 16 vCPU, 60GB Memory | n1-standard-16 | m5.4xlarge | -| PgBouncer | 3 | 2 vCPU, 1.8GB Memory | n1-highcpu-2 | c5.large | -| Gitaly[^2] [^5] [^7] | X | 64 vCPU, 240GB Memory | n1-standard-64 | m5.16xlarge | -| Redis[^3] - Cache | 3 | 4 vCPU, 15GB Memory | n1-standard-4 | m5.xlarge | -| Redis[^3] - Queues / Shared State | 3 | 4 vCPU, 15GB Memory | n1-standard-4 | m5.xlarge | -| Redis Sentinel[^3] - Cache | 3 | 1 vCPU, 1.7GB Memory | g1-small | t2.small | -| Redis Sentinel[^3] - Queues / Shared State | 3 | 1 vCPU, 1.7GB Memory | g1-small | t2.small | -| Consul | 3 | 2 vCPU, 1.8GB Memory | n1-highcpu-2 | c5.large | -| Sidekiq | 4 | 4 vCPU, 15GB Memory | n1-standard-4 | m5.xlarge | -| NFS Server[^5] [^7] | 1 | 4 vCPU, 3.6GB Memory | n1-highcpu-4 | c5.xlarge | -| Cloud Object Storage[^4] | - | - | - | - | -| Monitoring node | 1 | 4 vCPU, 3.6GB Memory | n1-highcpu-4 | c5.xlarge | -| External load balancing node[^6] | 1 | 2 vCPU, 1.8GB Memory | n1-highcpu-2 | c5.large | -| Internal load balancing node[^6] | 1 | 8 vCPU, 7.2GB Memory | n1-highcpu-8 | c5.2xlarge | - -[^1]: In our architectures we run each GitLab Rails node using the Puma webserver - and have its number of workers set to 90% of available CPUs along with 4 threads. - -[^2]: Gitaly node requirements are dependent on customer data, specifically the number of - projects and their sizes. We recommend 2 nodes as an absolute minimum for HA environments - and at least 4 nodes should be used when supporting 50,000 or more users. - We also recommend that each Gitaly node should store no more than 5TB of data - and have the number of [`gitaly-ruby` workers](../gitaly/index.md#gitaly-ruby) - set to 20% of available CPUs. Additional nodes should be considered in conjunction - with a review of expected data size and spread based on the recommendations above. - -[^3]: Recommended Redis setup differs depending on the size of the architecture. - For smaller architectures (up to 5,000 users) we suggest one Redis cluster for all - classes and that Redis Sentinel is hosted alongside Consul. - For larger architectures (10,000 users or more) we suggest running a separate - [Redis Cluster](../high_availability/redis.md#running-multiple-redis-clusters) for the Cache class - and another for the Queues and Shared State classes respectively. We also recommend - that you run the Redis Sentinel clusters separately as well for each Redis Cluster. - -[^4]: For data objects such as LFS, Uploads, Artifacts, etc. We recommend a [Cloud Object Storage service](../object_storage.md) - over NFS where possible, due to better performance and availability. - -[^5]: NFS can be used as an alternative for both repository data (replacing Gitaly) and - object storage but this isn't typically recommended for performance reasons. Note however it is required for - [GitLab Pages](https://gitlab.com/gitlab-org/gitlab-pages/issues/196). - -[^6]: Our architectures have been tested and validated with [HAProxy](https://www.haproxy.org/) - as the load balancer. However other reputable load balancers with similar feature sets - should also work instead but be aware these aren't validated. - -[^7]: We strongly recommend that any Gitaly and / or NFS nodes are set up with SSD disks over - HDD with a throughput of at least 8,000 IOPS for read operations and 2,000 IOPS for write - as these components have heavy I/O. These IOPS values are recommended only as a starter - as with time they may be adjusted higher or lower depending on the scale of your - environment's workload. If you're running the environment on a Cloud provider - you may need to refer to their documentation on how configure IOPS correctly. - -[^8]: The architectures were built and tested with the [Intel Xeon E5 v3 (Haswell)](https://cloud.google.com/compute/docs/cpu-platforms) - CPU platform on GCP. On different hardware you may find that adjustments, either lower - or higher, are required for your CPU or Node counts accordingly. For more information, a - [Sysbench](https://github.com/akopytov/sysbench) benchmark of the CPU can be found - [here](https://gitlab.com/gitlab-org/quality/performance/-/wikis/Reference-Architectures/GCP-CPU-Benchmarks). - -[^9]: AWS-equivalent configurations are rough suggestions and may change in the - future. They have not yet been tested and validated. +This document was moved to [another location](../reference_architectures/index.md). diff --git a/doc/administration/server_hooks.md b/doc/administration/server_hooks.md index 06c560a01ca..0b8c66805ae 100644 --- a/doc/administration/server_hooks.md +++ b/doc/administration/server_hooks.md @@ -11,12 +11,13 @@ disqus_identifier: 'https://docs.gitlab.com/ee/administration/custom_hooks.html' > - Server hooks must be configured on the filesystem of the GitLab server. Only GitLab server administrators will be able to complete these tasks. Please explore [webhooks](../user/project/integrations/webhooks.md) and [GitLab CI/CD](../ci/README.md) as an option if you do not have filesystem access. For a user-configurable Git hook interface, see [Push Rules](../push_rules/push_rules.md), available in GitLab Starter **(STARTER)**. > - Server hooks won't be replicated to secondary nodes if you use [GitLab Geo](geo/replication/index.md). -Git natively supports hooks that are executed on different actions. -Examples of server-side Git hooks include pre-receive, post-receive, and update. -See [Git SCM Server-Side Hooks](https://git-scm.com/book/en/v2/Customizing-Git-Git-Hooks#Server-Side-Hooks) for more information about each hook type. +Git natively supports hooks that are executed on different actions. These hooks run +on the server and can be used to enforce specific commit policies or perform other +tasks based on the state of the repository. -As of GitLab Shell version 2.2.0 (which requires GitLab 7.5+), GitLab -administrators can add custom Git hooks to any GitLab project. +Examples of server-side Git hooks include `pre-receive`, `post-receive`, and `update`. +See [Git SCM Server-Side Hooks](https://git-scm.com/book/en/v2/Customizing-Git-Git-Hooks#Server-Side-Hooks) +for more information about each hook type. ## Create a server hook for a repository @@ -24,14 +25,21 @@ Server-side Git hooks are typically placed in the repository's `hooks` subdirectory. In GitLab, hook directories are symlinked to the GitLab Shell `hooks` directory for ease of maintenance between GitLab Shell upgrades. Server hooks are implemented differently, but the behavior is exactly the same -once the hook is created. Follow the steps below to set up a server hook for a +once the hook is created. + +NOTE: **Note:** +If you are not using [hashed storage](repository_storage_types.md#hashed-storage), the project's +repository directory might not exactly match the instructions below. In that case, +for an installation from source the path is usually `/home/git/repositories/<group>/<project>.git`. +For Omnibus installs the path is usually `/var/opt/gitlab/git-data/repositories/<group>/<project>.git`. + +Follow the steps below to set up a server hook for a repository: -1. Pick a project that needs a server hook. -1. On the GitLab server, navigate to the project's repository directory. - For an installation from source the path is usually - `/home/git/repositories/<group>/<project>.git`. For Omnibus installs the path is - usually `/var/opt/gitlab/git-data/repositories/<group>/<project>.git`. +1. Find that project's path on the GitLab server, by navigating to the + **Admin area > Projects**. From there, select the project for which you + would like to add a hook. You can find the path to the project's repository + under **Gitaly relative path** on that page. 1. Create a new directory in this location called `custom_hooks`. 1. Inside the new `custom_hooks` directory, create a file with a name matching the hook type. For a pre-receive hook the file name should be `pre-receive` @@ -42,8 +50,7 @@ repository: type. For example, if the script is in Ruby the shebang will probably be `#!/usr/bin/env ruby`. -That's it! Assuming the hook code is properly implemented the hook will fire -as appropriate. +Assuming the hook code is properly implemented the hook will run as appropriate. ## Set a global server hook for all repositories diff --git a/doc/administration/smime_signing_email.md b/doc/administration/smime_signing_email.md index ed7447c0da9..bab7c5c260d 100644 --- a/doc/administration/smime_signing_email.md +++ b/doc/administration/smime_signing_email.md @@ -18,6 +18,9 @@ files must be provided: intervention. - Only RSA keys are supported. +Optionally, you can also provide a bundle of CA certs (PEM-encoded) to be +included on each signature. This will typically be an intermediate CA. + NOTE: **Note:** Be mindful of the access levels for your private keys and visibility to third parties. @@ -29,6 +32,8 @@ third parties. gitlab_rails['gitlab_email_smime_enabled'] = true gitlab_rails['gitlab_email_smime_key_file'] = '/etc/gitlab/ssl/gitlab_smime.key' gitlab_rails['gitlab_email_smime_cert_file'] = '/etc/gitlab/ssl/gitlab_smime.crt' + # Optional + gitlab_rails['gitlab_email_smime_ca_certs_file'] = '/etc/gitlab/ssl/gitlab_smime_cas.crt' ``` 1. Save the file and [reconfigure GitLab](restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. @@ -49,6 +54,9 @@ NOTE: **Note:** The key needs to be readable by the GitLab system user (`git` by # S/MIME public certificate key in PEM format, will be attached to signed messages # Default is '.gitlab_smime_cert' relative to Rails.root (i.e. root of the GitLab app). cert_file: /etc/pki/smime/certs/gitlab.crt + # S/MIME extra CA public certificates in PEM format, will be attached to signed messages + # Optional + ca_certs_file: /etc/pki/smime/certs/gitlab_cas.crt ``` 1. Save the file and [restart GitLab](restart_gitlab.md#installations-from-source) for the changes to take effect. diff --git a/doc/administration/static_objects_external_storage.md b/doc/administration/static_objects_external_storage.md index f649a1ebcd2..973f4304115 100644 --- a/doc/administration/static_objects_external_storage.md +++ b/doc/administration/static_objects_external_storage.md @@ -63,7 +63,7 @@ other CDNs or Function as a Service (FaaS) systems should work using the same pr `pwgen -cn1 64` on a UNIX machine). Save this token for the admin panel, as described in the [configuring](#configuring) section. - ```js + ```javascript const ORIGIN_HOSTNAME = 'gitlab.installation.com' // FIXME: SET CORRECT VALUE const STORAGE_TOKEN = 'very-secure-token' // FIXME: SET CORRECT VALUE const CACHE_PRIVATE_OBJECTS = false diff --git a/doc/administration/terraform_state.md b/doc/administration/terraform_state.md index 0956edaf252..77d5495418e 100644 --- a/doc/administration/terraform_state.md +++ b/doc/administration/terraform_state.md @@ -30,7 +30,7 @@ below. gitlab_rails['terraform_state_storage_path'] = "/mnt/storage/terraform_state" ``` -1. Save the file and [reconfigure GitLab][] for the changes to take effect. +1. Save the file and [reconfigure GitLab](restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. **In installations from source:** @@ -43,7 +43,7 @@ below. storage_path: /mnt/storage/terraform_state ``` -1. Save the file and [restart GitLab][] for the changes to take effect. +1. Save the file and [restart GitLab](restart_gitlab.md#installations-from-source) for the changes to take effect. ## Using object storage **(CORE ONLY)** @@ -62,7 +62,7 @@ The following settings are: | Setting | Description | Default | |---------|-------------|---------| -| `enabled` | Enable/disable object storage | `true` | +| `enabled` | Enable/disable object storage | `false` | | `remote_directory` | The bucket name where Terraform state files will be stored | | | `connection` | Various connection options described below | | @@ -111,7 +111,7 @@ The connection settings match those provided by [Fog](https://github.com/fog), a } ``` -1. Save the file and [reconfigure GitLab][] for the changes to take effect. +1. Save the file and [reconfigure GitLab](restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. **In installations from source:** @@ -131,7 +131,4 @@ The connection settings match those provided by [Fog](https://github.com/fog), a region: eu-central-1 ``` -1. Save the file and [restart GitLab][] for the changes to take effect. - -[reconfigure gitlab]: restart_gitlab.md#omnibus-gitlab-reconfigure "How to reconfigure Omnibus GitLab" -[restart gitlab]: restart_gitlab.md#installations-from-source "How to restart GitLab" +1. Save the file and [restart GitLab](restart_gitlab.md#installations-from-source) for the changes to take effect. diff --git a/doc/administration/timezone.md b/doc/administration/timezone.md index fa0fbb43433..d6112c45141 100644 --- a/doc/administration/timezone.md +++ b/doc/administration/timezone.md @@ -2,7 +2,7 @@ The global time zone configuration parameter can be changed in `config/gitlab.yml`: -```text +```plaintext # time_zone: 'UTC' ``` @@ -15,7 +15,7 @@ To see all available time zones, run `bundle exec rake time:zones:all`. For Omnibus installations, run `gitlab-rake time:zones:all`. NOTE: **Note:** -Currently, this Rake task does not list timezones in TZInfo format required by GitLab Omnibus during a reconfigure: [#58672](https://gitlab.com/gitlab-org/gitlab-foss/issues/58672). +Currently, this Rake task does not list timezones in TZInfo format required by Omnibus GitLab during a reconfigure: [#58672](https://gitlab.com/gitlab-org/gitlab-foss/issues/58672). ## Changing time zone in Omnibus installations diff --git a/doc/administration/troubleshooting/elasticsearch.md b/doc/administration/troubleshooting/elasticsearch.md index 77a63112ea3..a39fe4ba8c3 100644 --- a/doc/administration/troubleshooting/elasticsearch.md +++ b/doc/administration/troubleshooting/elasticsearch.md @@ -1,5 +1,8 @@ # Troubleshooting Elasticsearch +To install and configure Elasticsearch, and for common and known issues, +visit the [administrator documentation](../../integration/elasticsearch.md). + Troubleshooting Elasticsearch requires: - Knowledge of common terms. @@ -203,7 +206,7 @@ The best place to start is to determine if the issue is with creating an empty i If it is, check on the Elasticsearch side to determine if the `gitlab-production` (the name for the GitLab index) exists. If it exists, manually delete it on the Elasticsearch side and attempt to recreate it from the -[`create_empty_index`](../../integration/elasticsearch.md#gitlab-elasticsearch-rake-tasks) +[`recreate_index`](../../integration/elasticsearch.md#gitlab-elasticsearch-rake-tasks) Rake task. If you still encounter issues, try creating an index manually on the Elasticsearch diff --git a/doc/administration/troubleshooting/gitlab_rails_cheat_sheet.md b/doc/administration/troubleshooting/gitlab_rails_cheat_sheet.md index ba2224e3fc7..2cbc994fb4c 100644 --- a/doc/administration/troubleshooting/gitlab_rails_cheat_sheet.md +++ b/doc/administration/troubleshooting/gitlab_rails_cheat_sheet.md @@ -150,9 +150,9 @@ Project.update_all(visibility_level: 0) # projects = Project.where(pending_delete: true) projects.each do |p| - puts "Project name: #{p.id}" + puts "Project ID: #{p.id}" puts "Project name: #{p.name}" - puts "Repository path: #{p.repository.storage_path}" + puts "Repository path: #{p.repository.full_path}" end # @@ -214,6 +214,20 @@ p.each do |project| end ``` +### Bulk update to disable the Slack Notification service + +To disable notifications for all projects that have Slack service enabled, do: + +```ruby +# Grab all projects that have the Slack notifications enabled +p = Project.find_by_sql("SELECT p.id FROM projects p LEFT JOIN services s ON p.id = s.project_id WHERE s.type = 'SlackService' AND s.active = true") + +# Disable the service on each of the projects that were found. +p.each do |project| + project.slack_service.update_attribute(:active, false) +end +``` + ## Wikis ### Recreate @@ -558,29 +572,7 @@ Ci::Pipeline.where(project_id: p.id).where(status: 'pending').count ### Remove artifacts more than a week old -The Latest version of these steps can be found in the [job artifacts documentation](../job_artifacts.md) - -```ruby -### SELECTING THE BUILDS TO CLEAR -# For a single project: -project = Project.find_by_full_path('') -builds_with_artifacts = project.builds.with_artifacts_archive - -# Instance-wide: -builds_with_artifacts = Ci::Build.with_artifacts_archive - -# Prior to 10.6 the above lines would be: -# builds_with_artifacts = project.builds.with_artifacts -# builds_with_artifacts = Ci::Build.with_artifacts - -### CLEAR THEM OUT -# Note that this will also erase artifacts that developers marked to "Keep" -builds_to_clear = builds_with_artifacts.where("finished_at < ?", 1.week.ago) -builds_to_clear.each do |build| - build.artifacts_expire_at = Time.now - build.erase_erasable_artifacts! -end -``` +This section has been moved to the [job artifacts troubleshooting documentation](../job_artifacts.md#delete-job-artifacts-from-jobs-completed-before-a-specific-date). ### Find reason failure (for when build trace is empty) (Introduced in 10.3.0) @@ -603,6 +595,14 @@ m = project.merge_requests.find_by(iid: ) m.project.try(:ci_service) ``` +### Validate the `.gitlab-ci.yml` + +```ruby +project = Project.find_by_full_path 'group/project' +content = project.repository.gitlab_ci_yml_for(project.repository.root_ref_sha) +Gitlab::Ci::YamlProcessor.validation_message(content, user: User.first) +``` + ### Disable AutoDevOps on Existing Projects ```ruby diff --git a/doc/administration/troubleshooting/kubernetes_cheat_sheet.md b/doc/administration/troubleshooting/kubernetes_cheat_sheet.md index 30ef3da3a99..cab073b9924 100644 --- a/doc/administration/troubleshooting/kubernetes_cheat_sheet.md +++ b/doc/administration/troubleshooting/kubernetes_cheat_sheet.md @@ -52,7 +52,7 @@ and they will assist you with any issues you are having. - Check logs via Kubectl: ```shell - kubectl logs <unicorn pod> -c dependencies + kubectl logs <webservice pod> -c dependencies ``` - How to tail all Kubernetes cluster events in real time: @@ -72,7 +72,7 @@ and they will assist you with any issues you are having. This is the principle of Kubernetes, read [Twelve-factor app](https://12factor.net/) for details. -- How to get cronjobs configured on a cluster +- How to get cron jobs configured on a cluster ```shell kubectl get cronjobs @@ -87,20 +87,20 @@ and they will assist you with any issues you are having. - Minimal config that can be used to test a Kubernetes Helm chart can be found [here](https://gitlab.com/gitlab-org/charts/gitlab/issues/620). -- Tailing logs of a separate pod. An example for a Unicorn pod: +- Tailing logs of a separate pod. An example for a Webservice pod: ```shell - kubectl logs gitlab-unicorn-7656fdd6bf-jqzfs -c unicorn + kubectl logs gitlab-webservice-54fbf6698b-hpckq -c webservice ``` -- Tail and follow all pods that share a label (in this case, `unicorn`): +- Tail and follow all pods that share a label (in this case, `webservice`): ```shell - # all containers in the unicorn pods - kubectl logs -f -l app=unicorn --all-containers=true --max-log-requests=50 + # all containers in the webservice pods + kubectl logs -f -l app=webservice --all-containers=true --max-log-requests=50 - # only the unicorn containers in all unicorn pods - kubectl logs -f -l app=unicorn -c unicorn --max-log-requests=50 + # only the webservice containers in all webservice pods + kubectl logs -f -l app=webservice -c webservice --max-log-requests=50 ``` - One can stream logs from all containers at once, similar to the Omnibus @@ -132,7 +132,7 @@ and they will assist you with any issues you are having. /srv/gitlab/bin/rails console # source-style commands should also work - /srv/gitlab && bundle exec rake gitlab:check RAILS_ENV=production + cd /srv/gitlab && bundle exec rake gitlab:check RAILS_ENV=production # run GitLab check. Note that the output can be confusing and invalid because of the specific structure of GitLab installed via helm chart /usr/local/bin/gitlab-rake gitlab:check @@ -206,7 +206,7 @@ all Kubernetes resources and dependent charts: helm get manifest <release name> ``` -## Installation of minimal GitLab config via Minukube on macOS +## Installation of minimal GitLab config via Minikube on macOS This section is based on [Developing for Kubernetes with Minikube](https://docs.gitlab.com/charts/development/minikube/index.html) and [Helm](https://docs.gitlab.com/charts/installation/tools.html#helm). Refer @@ -230,31 +230,33 @@ to those documents for details. ```shell minikube start --cpus 3 --memory 8192 # minimum amount for GitLab to work minikube addons enable ingress - minikube addons enable kube-dns ``` - Install Helm via Homebrew and initialize it: ```shell - brew install kubernetes-helm - helm init --service-account tiller + brew install helm ``` -- Copy the file <https://gitlab.com/gitlab-org/charts/gitlab/raw/master/examples/values-minikube-minimum.yaml> - to your workstation. +- Copy the [Minikube minimum values YAML file](https://gitlab.com/gitlab-org/charts/gitlab/raw/master/examples/values-minikube-minimum.yaml) + to your workstation: -- Find the IP address in the output of `minikube ip` and update the yaml file with + ```shell + curl --output values.yaml "https://gitlab.com/gitlab-org/charts/gitlab/raw/master/examples/values-minikube-minimum.yaml" + ``` + +- Find the IP address in the output of `minikube ip` and update the YAML file with this IP address. - Install the GitLab Helm Chart: ```shell helm repo add gitlab https://charts.gitlab.io - helm install --name gitlab -f <path-to-yaml-file> gitlab/gitlab + helm install gitlab -f <path-to-yaml-file> gitlab/gitlab ``` If you want to modify some GitLab settings, you can use the above-mentioned config - as a base and create your own yaml file. + as a base and create your own YAML file. - Monitor the installation progress via `helm status gitlab` and `minikube dashboard`. The installation could take up to 20-30 minutes depending on the amount of resources @@ -263,7 +265,7 @@ to those documents for details. - When all the pods show either a `Running` or `Completed` status, get the GitLab password as described in [Initial login](https://docs.gitlab.com/charts/installation/deployment.html#initial-login), and log in to GitLab via the UI. It will be accessible via `https://gitlab.domain` - where `domain` is the value provided in the yaml file. + where `domain` is the value provided in the YAML file. <!-- ## Troubleshooting diff --git a/doc/administration/troubleshooting/log_parsing.md b/doc/administration/troubleshooting/log_parsing.md index 0413e5ce953..dcd1df2f423 100644 --- a/doc/administration/troubleshooting/log_parsing.md +++ b/doc/administration/troubleshooting/log_parsing.md @@ -16,19 +16,19 @@ include use cases targeted for parsing GitLab log files. #### Pipe colorized `jq` output into `less` -```sh +```shell jq . <FILE> -C | less -R ``` #### Search for a term and pretty-print all matching lines -```sh +```shell grep <TERM> <FILE> | jq . ``` #### Skip invalid lines of JSON -```sh +```shell jq -cR 'fromjson?' file.json | jq <COMMAND> ``` @@ -39,49 +39,49 @@ This skips over all invalid lines and parses the rest. #### Find all requests with a 5XX status code -```sh +```shell jq 'select(status >= 500)' <FILE> ``` #### Top 10 slowest requests -```sh +```shell jq -s 'sort_by(-.duration) | limit(10; .[])' <FILE> ``` #### Find and pretty print all requests related to a project -```sh +```shell grep <PROJECT_NAME> <FILE> | jq . ``` #### Find all requests with a total duration > 5 seconds -```sh +```shell jq 'select(.duration > 5000)' <FILE> ``` #### Find all project requests with more than 5 rugged calls -```sh +```shell grep <PROJECT_NAME> <FILE> | jq 'select(.rugged_calls > 5)' ``` #### Find all requests with a Gitaly duration > 10 seconds -```sh +```shell jq 'select(.gitaly_duration > 10000)' <FILE> ``` #### Find all requests with a queue duration > 10 seconds -```sh +```shell jq 'select(.queue_duration > 10000)' <FILE> ``` #### Top 10 requests by # of Gitaly calls -```sh +```shell jq -s 'map(select(.gitaly_calls != null)) | sort_by(-.gitaly_calls) | limit(10; .[])' <FILE> ``` @@ -89,7 +89,7 @@ jq -s 'map(select(.gitaly_calls != null)) | sort_by(-.gitaly_calls) | limit(10; #### Print the top three controller methods by request volume and their three longest durations -```sh +```shell jq -s -r 'group_by(.controller+.action) | sort_by(-length) | limit(3; .[]) | sort_by(-.duration) | "CT: \(length)\tMETHOD: \(.[0].controller)#\(.[0].action)\tDURS: \(.[0].duration), \(.[1].duration), \(.[2].duration)"' production_json.log ``` @@ -105,7 +105,7 @@ CT: 1328 METHOD: Projects::NotesController#index DURS: 403.99, 386.29, 384.3 #### Print top three routes with request count and their three longest durations -```sh +```shell jq -s -r 'group_by(.route) | sort_by(-length) | limit(3; .[]) | sort_by(-.duration) | "CT: \(length)\tROUTE: \(.[0].route)\tDURS: \(.[0].duration), \(.[1].duration), \(.[2].duration)"' api_json.log ``` @@ -121,25 +121,25 @@ CT: 190 ROUTE: /api/:version/projects/:id/repository/commits DURS: 1079.02, #### Find all Gitaly requests sent from web UI -```sh +```shell jq 'select(."grpc.meta.client_name" == "gitlab-web")' current ``` #### Find all failed Gitaly requests -```sh +```shell jq 'select(."grpc.code" != null and ."grpc.code" != "OK")' current ``` #### Find all requests that took longer than 30 seconds -```sh +```shell jq 'select(."grpc.time_ms" > 30000)' current ``` #### Print top three projects by request volume and their three longest durations -```sh +```shell jq -s -r 'map(select(."grpc.request.glProjectPath" != null and ."grpc.request.glProjectPath" != "" and ."grpc.time_ms" != null)) | group_by(."grpc.request.glProjectPath") | sort_by(-length) | limit(3; .[]) | sort_by(-."grpc.time_ms") | "CT: \(length)\tPROJECT: \(.[0]."grpc.request.glProjectPath")\tDURS: \(.[0]."grpc.time_ms"), \(.[1]."grpc.time_ms"), \(.[2]."grpc.time_ms")"' current ``` diff --git a/doc/administration/troubleshooting/postgresql.md b/doc/administration/troubleshooting/postgresql.md index 8e7727ee214..65a6bffca44 100644 --- a/doc/administration/troubleshooting/postgresql.md +++ b/doc/administration/troubleshooting/postgresql.md @@ -33,7 +33,7 @@ This section is for links to information elsewhere in the GitLab documentation. - [More about external PostgreSQL](../external_database.md) -- [Running GEO with external PostgreSQL](../geo/replication/external_database.md) +- [Running Geo with external PostgreSQL](../geo/replication/external_database.md) - [Upgrades when running PostgreSQL configured for HA.](https://docs.gitlab.com/omnibus/settings/database.html#upgrading-a-gitlab-ha-cluster) @@ -45,7 +45,7 @@ This section is for links to information elsewhere in the GitLab documentation. - Managing Omnibus PostgreSQL versions [from the development docs](https://docs.gitlab.com/omnibus/development/managing-postgresql-versions.html) -- [PostgreSQL scaling and HA](../high_availability/database.md) +- [PostgreSQL scaling](../high_availability/database.md) - including [troubleshooting](../high_availability/database.md#troubleshooting) `gitlab-ctl repmgr-check-master` and PgBouncer errors - [Developer database documentation](../../development/README.md#database-guides) - some of which is absolutely not for production use. Including: @@ -71,7 +71,7 @@ This section is for links to information elsewhere in the GitLab documentation. HINT: Free one or increase max_replication_slots. ``` -- GEO [replication errors](../geo/replication/troubleshooting.md#fixing-replication-errors) including: +- Geo [replication errors](../geo/replication/troubleshooting.md#fixing-replication-errors) including: ```plaintext ERROR: replication slots can only be used if max_replication_slots > 0 @@ -83,11 +83,11 @@ This section is for links to information elsewhere in the GitLab documentation. PANIC: could not write to file ‘pg_xlog/xlogtemp.123’: No space left on device ``` -- [Checking GEO configuration](../geo/replication/troubleshooting.md#checking-configuration) including +- [Checking Geo configuration](../geo/replication/troubleshooting.md#checking-configuration) including - reconfiguring hosts/ports - checking and fixing user/password mappings -- [Common GEO errors](../geo/replication/troubleshooting.md#fixing-common-errors) +- [Common Geo errors](../geo/replication/troubleshooting.md#fixing-common-errors) ## Support topics diff --git a/doc/administration/troubleshooting/sidekiq.md b/doc/administration/troubleshooting/sidekiq.md index 31e41725834..ca21c038267 100644 --- a/doc/administration/troubleshooting/sidekiq.md +++ b/doc/administration/troubleshooting/sidekiq.md @@ -282,7 +282,8 @@ Commonly, `<condition>` references the job arguments, which depend on the type o For example, `repository_import` has `project_id` as the job argument, while `update_merge_requests` has `project_id, user_id, oldrev, newrev, ref`. -NOTE: **Note:** Arguments need to be referenced by their sequence id using `job.args[<id>]` because `job.args` is a list of all arguments provided to the Sidekiq job. +NOTE: **Note:** +Arguments need to be referenced by their sequence ID using `job.args[<id>]` because `job.args` is a list of all arguments provided to the Sidekiq job. Here are some examples: diff --git a/doc/administration/troubleshooting/ssl.md b/doc/administration/troubleshooting/ssl.md index f230f047ded..e6c081e1eea 100644 --- a/doc/administration/troubleshooting/ssl.md +++ b/doc/administration/troubleshooting/ssl.md @@ -46,6 +46,44 @@ After configuring a GitLab instance with an internal CA certificate, you might n If you have the problems listed above, add your certificate to `/etc/gitlab/trusted-certs` and run `sudo gitlab-ctl reconfigure`. +## X.509 key values mismatch error + +After configuring your instance with a certificate bundle, NGINX may throw the +following error: + +`SSL: error:0B080074:x509 certificate routines:X509_check_private_key:key values mismatch` + +This error means that the server certificate and key you have provided do not +match. You can confirm this by running the following command and comparing the +output: + +```shell +openssl rsa -noout -modulus -in path/to/your/.key | openssl md5 +openssl x509 -noout -modulus -in path/to/your/.crt | openssl md5 +``` + +The following is an example of an md5 output between a matching key and certificate. Note the +matching md5 hashes: + +```shell +$ openssl rsa -noout -modulus -in private.key | openssl md5 +4f49b61b25225abeb7542b29ae20e98c +$ openssl x509 -noout -modulus -in public.crt | openssl md5 +4f49b61b25225abeb7542b29ae20e98c +``` + +This is an opposing output with a non-matching key and certificate which shows different md5 hashes: + +```shell +$ openssl rsa -noout -modulus -in private.key | openssl md5 +d418865077299af27707b1d1fa83cd99 +$ openssl x509 -noout -modulus -in public.crt | openssl md5 +4f49b61b25225abeb7542b29ae20e98c +``` + +If the two outputs differ like the above example, there is a mismatch between the certificate +and key. You should contact the provider of the SSL certificate for further support. + ## Using GitLab Runner with a GitLab instance configured with internal CA certificate or self-signed certificate Besides getting the errors mentioned in diff --git a/doc/administration/uploads.md b/doc/administration/uploads.md index f29deba3d40..0a300084342 100644 --- a/doc/administration/uploads.md +++ b/doc/administration/uploads.md @@ -25,7 +25,7 @@ _The uploads are stored by default in `/var/opt/gitlab/gitlab-rails/uploads`._ gitlab_rails['uploads_base_dir'] = "uploads" ``` -1. Save the file and [reconfigure GitLab][] for the changes to take effect. +1. Save the file and [reconfigure GitLab](restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. **In installations from source:** @@ -41,13 +41,13 @@ _The uploads are stored by default in base_dir: uploads ``` -1. Save the file and [restart GitLab][] for the changes to take effect. +1. Save the file and [restart GitLab](restart_gitlab.md#installations-from-source) for the changes to take effect. ## Using object storage **(CORE ONLY)** > **Notes:** > -> - [Introduced][ee-3867] in [GitLab Premium][eep] 10.5. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/3867) in [GitLab Premium](https://about.gitlab.com/pricing/) 10.5. > - [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/17358) in [GitLab Core](https://about.gitlab.com/pricing/) 10.7. > - Since version 11.1, we support direct_upload to S3. @@ -65,7 +65,7 @@ For source installations the following settings are nested under `uploads:` and |---------|-------------|---------| | `enabled` | Enable/disable object storage | `false` | | `remote_directory` | The bucket name where Uploads will be stored| | -| `direct_upload` | Set to true to remove Unicorn from the Upload path. Workhorse handles the actual Artifact Upload to Object Storage while Unicorn does minimal processing to keep track of the upload. There is no need for local shared storage. The option may be removed if support for a single storage type for all files is introduced. Read more on [direct upload](../development/uploads.md#direct-upload). | `false` | +| `direct_upload` | Set to true to remove Puma from the Upload path. Workhorse handles the actual Artifact Upload to Object Storage while Puma does minimal processing to keep track of the upload. There is no need for local shared storage. The option may be removed if support for a single storage type for all files is introduced. Read more on [direct upload](../development/uploads.md#direct-upload). | `false` | | `background_upload` | Set to false to disable automatic upload. Option may be removed once upload is direct to S3 (if `direct_upload` is set to `true` it will override `background_upload`) | `true` | | `proxy_download` | Set to true to enable proxying all files served. Option allows to reduce egress traffic as this allows clients to download directly from remote storage instead of proxying all data | `false` | | `connection` | Various connection options described below | | @@ -76,16 +76,16 @@ The connection settings match those provided by [Fog](https://github.com/fog), a | Setting | Description | Default | |---------|-------------|---------| -| `provider` | Always `AWS` for compatible hosts | AWS | +| `provider` | Always `AWS` for compatible hosts | `AWS` | | `aws_access_key_id` | AWS credentials, or compatible | | | `aws_secret_access_key` | AWS credentials, or compatible | | -| `aws_signature_version` | AWS signature version to use. 2 or 4 are valid options. Digital Ocean Spaces and other providers may need 2. | 4 | -| `enable_signature_v4_streaming` | Set to true to enable HTTP chunked transfers with [AWS v4 signatures](https://docs.aws.amazon.com/AmazonS3/latest/API/sigv4-streaming.html). Oracle Cloud S3 needs this to be false | true | +| `aws_signature_version` | AWS signature version to use. `2` or `4` are valid options. Digital Ocean Spaces and other providers may need `2`. | `4` | +| `enable_signature_v4_streaming` | Set to true to enable HTTP chunked transfers with [AWS v4 signatures](https://docs.aws.amazon.com/AmazonS3/latest/API/sigv4-streaming.html). Oracle Cloud S3 needs this to be `false`. | `true` | | `region` | AWS region | us-east-1 | -| `host` | S3 compatible host for when not using AWS, e.g. `localhost` or `storage.example.com` | s3.amazonaws.com | +| `host` | S3 compatible host for when not using AWS, e.g. `localhost` or `storage.example.com` | `s3.amazonaws.com` | | `endpoint` | Can be used when configuring an S3 compatible service such as [MinIO](https://min.io), by entering a URL such as `http://127.0.0.1:9000` | (optional) | -| `path_style` | Set to true to use `host/bucket_name/object` style paths instead of `bucket_name.host/object`. Leave as false for AWS S3 | false | -| `use_iam_profile` | Set to true to use IAM profile instead of access keys | false +| `path_style` | Set to `true` to use `host/bucket_name/object` style paths instead of `bucket_name.host/object`. Leave as `false` for AWS S3. | `false` | +| `use_iam_profile` | Set to `true` to use IAM profile instead of access keys | false **In Omnibus installations:** @@ -117,7 +117,7 @@ _The uploads are stored by default in } ``` -1. Save the file and [reconfigure GitLab][] for the changes to take effect. +1. Save the file and [reconfigure GitLab](restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. 1. Migrate any existing local uploads to the object storage using [`gitlab:uploads:migrate` Rake task](raketasks/uploads/migrate.md). **In installations from source:** @@ -140,7 +140,7 @@ _The uploads are stored by default in region: eu-central-1 ``` -1. Save the file and [restart GitLab][] for the changes to take effect. +1. Save the file and [restart GitLab](restart_gitlab.md#installations-from-source) for the changes to take effect. 1. Migrate any existing local uploads to the object storage using [`gitlab:uploads:migrate` Rake task](raketasks/uploads/migrate.md). ### Oracle Cloud S3 connection settings @@ -149,8 +149,8 @@ Note that Oracle Cloud S3 must be sure to use the following settings: | Setting | Value | |---------|-------| -| `enable_signature_v4_streaming` | false | -| `path_style` | true | +| `enable_signature_v4_streaming` | `false` | +| `path_style` | `true` | If `enable_signature_v4_streaming` is set to `true`, you may see the following error: @@ -165,7 +165,7 @@ The connection settings match those provided by [Fog](https://github.com/fog), a | Setting | Description | Default | |---------|-------------|---------| -| `provider` | Always `OpenStack` for compatible hosts | OpenStack | +| `provider` | Always `OpenStack` for compatible hosts | `OpenStack` | | `openstack_username` | OpenStack username | | | `openstack_api_key` | OpenStack API key | | | `openstack_temp_url_key` | OpenStack key for generating temporary urls | | @@ -194,7 +194,7 @@ _The uploads are stored by default in } ``` -1. Save the file and [reconfigure GitLab][] for the changes to take effect. +1. Save the file and [reconfigure GitLab](restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. 1. Migrate any existing local uploads to the object storage using [`gitlab:uploads:migrate` Rake task](raketasks/uploads/migrate.md). --- @@ -225,10 +225,5 @@ _The uploads are stored by default in openstack_tenant: 'TENANT_ID' ``` -1. Save the file and [reconfigure GitLab][] for the changes to take effect. +1. Save the file and [reconfigure GitLab](restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. 1. Migrate any existing local uploads to the object storage using [`gitlab:uploads:migrate` Rake task](raketasks/uploads/migrate.md). - -[reconfigure gitlab]: restart_gitlab.md#omnibus-gitlab-reconfigure "How to reconfigure Omnibus GitLab" -[restart gitlab]: restart_gitlab.md#installations-from-source "How to restart GitLab" -[eep]: https://about.gitlab.com/pricing/ "GitLab Premium" -[ee-3867]: https://gitlab.com/gitlab-org/gitlab/-/merge_requests/3867 |
