diff options
| author | GitLab Bot <gitlab-bot@gitlab.com> | 2020-04-21 15:21:10 +0000 |
|---|---|---|
| committer | GitLab Bot <gitlab-bot@gitlab.com> | 2020-04-21 15:21:10 +0000 |
| commit | e33f87ac0fabaab468ce4b457996cc0f1b1bb648 (patch) | |
| tree | 8bf0de72a9acac014cfdaddab7d463b208294af2 /doc/administration | |
| parent | 5baf990db20a75078684702782c24399ef9eb0fa (diff) | |
| download | gitlab-ce-e33f87ac0fabaab468ce4b457996cc0f1b1bb648.tar.gz | |
Add latest changes from gitlab-org/gitlab@master
Diffstat (limited to 'doc/administration')
34 files changed, 502 insertions, 394 deletions
diff --git a/doc/administration/audit_events.md b/doc/administration/audit_events.md index 26b4434de77..9019cf0a630 100644 --- a/doc/administration/audit_events.md +++ b/doc/administration/audit_events.md @@ -1,6 +1,6 @@ # Audit Events **(STARTER)** -GitLab offers a way to view the changes made within the GitLab server for owners and administrators on a [paid plan][ee]. +GitLab offers a way to view the changes made within the GitLab server for owners and administrators on a [paid plan](https://about.gitlab.com/pricing/). GitLab system administrators can also take advantage of the logs located on the filesystem. See [the logs system documentation](logs.md) for more details. @@ -31,7 +31,7 @@ There are two kinds of events logged: ### Group events **(STARTER)** NOTE: **Note:** -You need Owner [permissions] to view the group Audit Events page. +You need Owner [permissions](../user/permissions.md) to view the group Audit Events page. To view a group's audit events, navigate to **Group > Settings > Audit Events**. From there, you can see the following actions: @@ -40,14 +40,14 @@ From there, you can see the following actions: - Group repository size limit changed - Group created or deleted - Group changed visibility -- User was added to group and with which [permissions] +- User was added to group and with which [permissions](../user/permissions.md) - User sign-in via [Group SAML](../user/group/saml_sso/index.md) - Permissions changes of a user assigned to a group - Removed user from group - Project added to group and with which visibility level - Project removed from group - [Project shared with group](../user/project/members/share_project_with_groups.md) - and with which [permissions] + and with which [permissions](../user/permissions.md) - Removal of a previously shared group with a project - LFS enabled or disabled - Shared runners minutes limit changed @@ -61,7 +61,7 @@ Group events can also be accessed via the [Group Audit Events API](../api/audit_ ### Project events **(STARTER)** NOTE: **Note:** -You need Maintainer [permissions] or higher to view the project Audit Events page. +You need Maintainer [permissions](../user/permissions.md) or higher to view the project Audit Events page. To view a project's audit events, navigate to **Project > Settings > Audit Events**. From there, you can see the following actions: @@ -69,7 +69,7 @@ From there, you can see the following actions: - Added or removed deploy keys - Project created, deleted, renamed, moved(transferred), changed path - Project changed visibility level -- User was added to project and with which [permissions] +- User was added to project and with which [permissions](../user/permissions.md) - Permission changes of a user assigned to a project - User was removed from project - Project export was downloaded @@ -86,7 +86,7 @@ From there, you can see the following actions: ### Instance events **(PREMIUM ONLY)** -> [Introduced][ee-2336] in [GitLab Premium][ee] 9.3. +> [Introduced](https://gitlab.com/gitlab-org/gitlab/issues/2336) in [GitLab Premium](https://about.gitlab.com/pricing/) 9.3. Server-wide audit logging introduces the ability to observe user actions across the entire instance of your GitLab server, making it easy to understand who @@ -157,7 +157,3 @@ the steps bellow. ```ruby Feature.enable(:repository_push_audit_event) ``` - -[ee-2336]: https://gitlab.com/gitlab-org/gitlab/issues/2336 -[ee]: https://about.gitlab.com/pricing/ -[permissions]: ../user/permissions.md diff --git a/doc/administration/availability/index.md b/doc/administration/availability/index.md index a0d4ea7919f..d567a21aa16 100644 --- a/doc/administration/availability/index.md +++ b/doc/administration/availability/index.md @@ -4,136 +4,10 @@ type: reference, concepts # Availability -GitLab offers high availability options for organizations that require -the fault tolerance and redundancy necessary to maintain high-uptime operations. - -Please consult our [scaling documentation](../scaling) if you want to resolve -performance bottlenecks you encounter in individual GitLab components without -incurring the additional complexity costs associated with maintaining a -highly-available architecture. - -On this page, we present examples of self-managed instances which demonstrate -how GitLab can be scaled out and made highly available. These examples progress -from simple to complex as scaling or highly-available components are added. - -For larger setups serving 2,000 or more users, we provide -[reference architectures](../scaling/index.md#reference-architectures) based on GitLab's -experience with GitLab.com and internal scale testing that aim to achieve the -right balance of scalability and availability. - -For detailed insight into how GitLab scales and configures GitLab.com, you can -watch [this 1 hour Q&A](https://www.youtube.com/watch?v=uCU8jdYzpac) -with [John Northrup](https://gitlab.com/northrup), and live questions coming -in from some of our customers. - GitLab offers a number of options to manage availability and resiliency. Below are the options to consider with trade-offs. | Event | GitLab Feature | Recovery Point Objective (RPO) | Recovery Time Objective (RTO) | Cost | | ----- | -------------- | --- | --- | ---- | | Availability Zone failure | "GitLab HA" | No loss | No loss | 2x Git storage, multiple nodes balanced across AZ's | -| Region failure | "GitLab Disaster Recovery" | 5-10 minutes | 30 minutes | 2x primary cost | +| Region failure | [GitLab Geo Disaster Recovery](../geo/disaster_recovery/index.md) | 5-10 minutes | 30 minutes | 2x primary cost | | All failures | Backup/Restore | Last backup | Hours to Days | Cost of storing the backups | - -## High availability - -### Omnibus installation with automatic database failover - -By adding automatic failover for database systems, we can enable higher uptime with an additional layer of complexity. - -- For PostgreSQL, we provide repmgr for server cluster management and failover - and a combination of [PgBouncer](../high_availability/pgbouncer.md) and [Consul](../high_availability/consul.md) for - database client cutover. -- For Redis, we use [Redis Sentinel](../high_availability/redis.md) for server failover and client cutover. - -You can also optionally run [additional Sidekiq processes on dedicated hardware](../high_availability/sidekiq.md) -and configure individual Sidekiq processes to -[process specific background job queues](../operations/extra_sidekiq_processes.md) -if you need to scale out background job processing. - -### GitLab Geo - -GitLab Geo allows you to replicate your GitLab instance to other geographical locations as a read-only fully operational instance that can also be promoted in case of disaster. - -This configuration is supported in [GitLab Premium and Ultimate](https://about.gitlab.com/pricing/). - -References: - -- [Geo Documentation](../geo/replication/index.md) -- [GitLab Geo with a highly available configuration](../geo/replication/high_availability.md) - -## GitLab components and configuration instructions - -The GitLab application depends on the following [components](../../development/architecture.md#component-diagram). -It can also depend on several third party services depending on -your environment setup. Here we'll detail both in the order in which -you would typically configure them along with our recommendations for -their use and configuration. - -### Third party services - -Here's some details of several third party services a typical environment -will depend on. The services can be provided by numerous applications -or providers and further advice can be given on how best to select. -These should be configured first, before the [GitLab components](#gitlab-components). - -| Component | Description | Configuration instructions | -|--------------------------------------------------------|---------------------------------------------------------------------------------------------------------------------|---------------------------------------------------------| -| [Load Balancer(s)](../high_availability/load_balancer.md)[^6] | Handles load balancing for the GitLab nodes where required | [Load balancer HA configuration](../high_availability/load_balancer.md) | -| [Cloud Object Storage service](../high_availability/object_storage.md)[^4] | Recommended store for shared data objects | [Cloud Object Storage configuration](../high_availability/object_storage.md) | -| [NFS](../high_availability/nfs.md)[^5] [^7] | Shared disk storage service. Can be used as an alternative for Gitaly or Object Storage. Required for GitLab Pages | [NFS configuration](../high_availability/nfs.md) | - -### GitLab components - -Next are all of the components provided directly by GitLab. As mentioned -earlier, they are presented in the typical order you would configure -them. - -| Component | Description | Configuration instructions | -|---------------------------------------------------------------------------------------------------------------------|---------------------------------------------------------------------|---------------------------------------------------------------| -| [Consul](../../development/architecture.md#consul)[^3] | Service discovery and health checks/failover | [Consul HA configuration](../high_availability/consul.md) **(PREMIUM ONLY)** | -| [PostgreSQL](../../development/architecture.md#postgresql) | Database | [Database HA configuration](../high_availability/database.md) | -| [PgBouncer](../../development/architecture.md#pgbouncer) | Database Pool Manager | [PgBouncer HA configuration](../high_availability/pgbouncer.md) **(PREMIUM ONLY)** | -| [Redis](../../development/architecture.md#redis)[^3] with Redis Sentinel | Key/Value store for shared data with HA watcher service | [Redis HA configuration](../high_availability/redis.md) | -| [Gitaly](../../development/architecture.md#gitaly)[^2] [^5] [^7] | Recommended high-level storage for Git repository data | [Gitaly HA configuration](../high_availability/gitaly.md) | -| [Sidekiq](../../development/architecture.md#sidekiq) | Asynchronous/Background jobs | [Sidekiq configuration](../high_availability/sidekiq.md) | -| [GitLab application nodes](../../development/architecture.md#unicorn)[^1] | (Unicorn / Puma, Workhorse) - Web-requests (UI, API, Git over HTTP) | [GitLab app HA/scaling configuration](../high_availability/gitlab.md) | -| [Prometheus](../../development/architecture.md#prometheus) and [Grafana](../../development/architecture.md#grafana) | GitLab environment monitoring | [Monitoring node for scaling/HA](../high_availability/monitoring_node.md) | - -In some cases, components can be combined on the same nodes to reduce complexity as well. - -[^1]: In our architectures we run each GitLab Rails node using the Puma webserver - and have its number of workers set to 90% of available CPUs along with 4 threads. - -[^2]: Gitaly node requirements are dependent on customer data, specifically the number of - projects and their sizes. We recommend 2 nodes as an absolute minimum for HA environments - and at least 4 nodes should be used when supporting 50,000 or more users. - We also recommend that each Gitaly node should store no more than 5TB of data - and have the number of [`gitaly-ruby` workers](../gitaly/index.md#gitaly-ruby) - set to 20% of available CPUs. Additional nodes should be considered in conjunction - with a review of expected data size and spread based on the recommendations above. - -[^3]: Recommended Redis setup differs depending on the size of the architecture. - For smaller architectures (up to 5,000 users) we suggest one Redis cluster for all - classes and that Redis Sentinel is hosted alongside Consul. - For larger architectures (10,000 users or more) we suggest running a separate - [Redis Cluster](../high_availability/redis.md#running-multiple-redis-clusters) for the Cache class - and another for the Queues and Shared State classes respectively. We also recommend - that you run the Redis Sentinel clusters separately as well for each Redis Cluster. - -[^4]: For data objects such as LFS, Uploads, Artifacts, etc. We recommend a [Cloud Object Storage service](../object_storage.md) - over NFS where possible, due to better performance and availability. - -[^5]: NFS can be used as an alternative for both repository data (replacing Gitaly) and - object storage but this isn't typically recommended for performance reasons. Note however it is required for - [GitLab Pages](https://gitlab.com/gitlab-org/gitlab-pages/issues/196). - -[^6]: Our architectures have been tested and validated with [HAProxy](https://www.haproxy.org/) - as the load balancer. However other reputable load balancers with similar feature sets - should also work instead but be aware these aren't validated. - -[^7]: We strongly recommend that any Gitaly and / or NFS nodes are set up with SSD disks over - HDD with a throughput of at least 8,000 IOPS for read operations and 2,000 IOPS for write - as these components have heavy I/O. These IOPS values are recommended only as a starter - as with time they may be adjusted higher or lower depending on the scale of your - environment's workload. If you're running the environment on a Cloud provider - you may need to refer to their documentation on how configure IOPS correctly. diff --git a/doc/administration/database_load_balancing.md b/doc/administration/database_load_balancing.md index b6ce6883dd6..cecc372f387 100644 --- a/doc/administration/database_load_balancing.md +++ b/doc/administration/database_load_balancing.md @@ -1,6 +1,6 @@ # Database Load Balancing **(PREMIUM ONLY)** -> [Introduced][ee-1283] in [GitLab Premium][eep] 9.0. +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/1283) in [GitLab Premium](https://about.gitlab.com/pricing/) 9.0. Distribute read-only queries among multiple database servers. @@ -11,7 +11,7 @@ multiple computing resources. Load balancing aims to optimize resource use, maximize throughput, minimize response time, and avoid overload of any single resource. Using multiple components with load balancing instead of a single component may increase reliability and availability through redundancy. -[_Wikipedia article_][wikipedia] +[_Wikipedia article_](https://en.wikipedia.org/wiki/Load_balancing_(computing)) When database load balancing is enabled in GitLab, the load is balanced using a simple round-robin algorithm, without any external dependencies such as Redis. @@ -27,8 +27,8 @@ sent to the primary (unless necessary), the primary (`db3`) hardly has any load. ## Requirements For load balancing to work you will need at least PostgreSQL 9.2 or newer, -[**MySQL is not supported**][db-req]. You also need to make sure that you have -at least 1 secondary in [hot standby][hot-standby] mode. +[**MySQL is not supported**](../install/requirements.md#database). You also need to make sure that you have +at least 1 secondary in [hot standby](https://www.postgresql.org/docs/9.6/hot-standby.html) mode. Load balancing also requires that the configured hosts **always** point to the primary, even after a database failover. Furthermore, the additional hosts to @@ -78,7 +78,7 @@ the following. This will balance the load between `host1.example.com` and gitlab_rails['db_load_balancing'] = { 'hosts' => ['host1.example.com', 'host2.example.com'] } ``` -1. Save the file and [reconfigure GitLab][] for the changes to take effect. +1. Save the file and [reconfigure GitLab](restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. --- @@ -97,11 +97,11 @@ the following. This will balance the load between `host1.example.com` and - host2.example.com ``` -1. Save the file and [restart GitLab][] for the changes to take effect. +1. Save the file and [restart GitLab](restart_gitlab.md#installations-from-source) for the changes to take effect. ## Service Discovery -> [Introduced][ee-5883] in [GitLab Premium][eep] 11.0. +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/5883) in [GitLab Premium](https://about.gitlab.com/pricing/) 11.0. Service discovery allows GitLab to automatically retrieve a list of secondary databases to use, instead of having to manually specify these in the @@ -161,7 +161,7 @@ When the list of hosts is updated, it might take a while for the old connections to be terminated. The `disconnect_timeout` setting can be used to enforce an upper limit on the time it will take to terminate all old database connections. -Some nameservers (like [Consul][consul-udp]) can return a truncated list of hosts when +Some nameservers (like [Consul](https://www.consul.io/docs/agent/dns.html#udp-based-dns-queries)) can return a truncated list of hosts when queried over UDP. To overcome this issue, you can use TCP for querying by setting `use_tcp` to `true`. @@ -239,7 +239,7 @@ For example: ## Handling Stale Reads -> [Introduced][ee-3526] in [GitLab Premium][eep] 10.3. +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/3526) in [GitLab Premium](https://about.gitlab.com/pricing/) 10.3. To prevent reading from an outdated secondary the load balancer will check if it is in sync with the primary. If the data is determined to be recent enough the @@ -270,14 +270,3 @@ production: max_replication_lag_time: 30 replica_check_interval: 30 ``` - -[hot-standby]: https://www.postgresql.org/docs/9.6/hot-standby.html -[ee-1283]: https://gitlab.com/gitlab-org/gitlab/-/merge_requests/1283 -[eep]: https://about.gitlab.com/pricing/ -[reconfigure gitlab]: restart_gitlab.md#omnibus-gitlab-reconfigure "How to reconfigure Omnibus GitLab" -[restart gitlab]: restart_gitlab.md#installations-from-source "How to restart GitLab" -[wikipedia]: https://en.wikipedia.org/wiki/Load_balancing_(computing) -[db-req]: ../install/requirements.md#database -[ee-3526]: https://gitlab.com/gitlab-org/gitlab/-/merge_requests/3526 -[ee-5883]: https://gitlab.com/gitlab-org/gitlab/-/merge_requests/5883 -[consul-udp]: https://www.consul.io/docs/agent/dns.html#udp-based-dns-queries diff --git a/doc/administration/file_hooks.md b/doc/administration/file_hooks.md index 7d935af8e37..21ade36a2a5 100644 --- a/doc/administration/file_hooks.md +++ b/doc/administration/file_hooks.md @@ -23,22 +23,22 @@ see the [system hooks] documentation. ## Setup -The file hooks must be placed directly into the `plugins` directory, subdirectories +The file hooks must be placed directly into the `file_hooks` directory, subdirectories will be ignored. There is an -[`example` directory inside `plugins`](https://gitlab.com/gitlab-org/gitlab/tree/master/plugins/examples) +[`example` directory inside `file_hooks`](https://gitlab.com/gitlab-org/gitlab/tree/master/file_hooks/examples) where you can find some basic examples. Follow the steps below to set up a custom hook: 1. On the GitLab server, navigate to the plugin directory. For an installation from source the path is usually - `/home/git/gitlab/plugins/`. For Omnibus installs the path is - usually `/opt/gitlab/embedded/service/gitlab-rails/plugins`. + `/home/git/gitlab/file_hooks/`. For Omnibus installs the path is + usually `/opt/gitlab/embedded/service/gitlab-rails/file_hooks`. For [highly available] configurations, your hook file should exist on each application server. -1. Inside the `plugins` directory, create a file with a name of your choice, +1. Inside the `file_hooks` directory, create a file with a name of your choice, without spaces or special characters. 1. Make the hook file executable and make sure it's owned by the Git user. 1. Write the code to make the file hook function as expected. That can be @@ -106,9 +106,9 @@ bundle exec rake file_hooks:validate RAILS_ENV=production Example of output: ```plaintext -Validating file hooks from /plugins directory -* /home/git/gitlab/plugins/save_to_file.clj succeed (zero exit code) -* /home/git/gitlab/plugins/save_to_file.rb failure (non-zero exit code) +Validating file hooks from /file_hooks directory +* /home/git/gitlab/file_hooks/save_to_file.clj succeed (zero exit code) +* /home/git/gitlab/file_hooks/save_to_file.rb failure (non-zero exit code) ``` [system hooks]: ../system_hooks/system_hooks.md diff --git a/doc/administration/geo/disaster_recovery/index.md b/doc/administration/geo/disaster_recovery/index.md index 0a5c39665f4..6f417f955ac 100644 --- a/doc/administration/geo/disaster_recovery/index.md +++ b/doc/administration/geo/disaster_recovery/index.md @@ -248,6 +248,10 @@ The data can be removed with the following command: sudo rm -rf /var/opt/gitlab/geo-postgresql ``` +If you have any `geo_secondary[]` configuration options enabled in your `gitlab.rb` +file, these can be safely commented out or removed, and then [reconfigure GitLab](../../restart_gitlab.md#omnibus-gitlab-reconfigure) +for the changes to take effect. + ## Promoting secondary Geo replica in multi-secondary configurations If you have more than one **secondary** node and you need to promote one of them, we suggest you follow diff --git a/doc/administration/geo/replication/external_database.md b/doc/administration/geo/replication/external_database.md index fd14c100ffb..e305718580e 100644 --- a/doc/administration/geo/replication/external_database.md +++ b/doc/administration/geo/replication/external_database.md @@ -45,13 +45,17 @@ Skip to the [Configure secondary application node](#configure-secondary-applicat The [geo_primary_role](https://docs.gitlab.com/omnibus/roles/#gitlab-geo-roles) configures the **primary** node's database to be replicated by making changes to `pg_hba.conf` and `postgresql.conf`. Make the following configuration changes -manually to your external database configuration: +manually to your external database configuration and ensure that you restart PostgreSQL +afterwards for the changes to take effect: ```plaintext ## ## Geo Primary Role ## - pg_hba.conf ## +host all all <trusted primary IP>/32 md5 +host replication gitlab_replicator <trusted primary IP>/32 md5 +host all all <trusted secondary IP>/32 md5 host replication gitlab_replicator <trusted secondary IP>/32 md5 ``` @@ -60,7 +64,6 @@ host replication gitlab_replicator <trusted secondary IP>/32 md5 ## Geo Primary Role ## - postgresql.conf ## -sql_replication_user = gitlab_replicator wal_level = hot_standby max_wal_senders = 10 wal_keep_segments = 50 @@ -72,8 +75,19 @@ hot_standby = on ### Manually configure the replica database -Make the following configuration changes manually to your `postgresql.conf` -of external replica database: +Make the following configuration changes manually to your `pg_hba.conf` and `postgresql.conf` +of your external replica database and ensure that you restart PostgreSQL afterwards +for the changes to take effect: + +```plaintext +## +## Geo Secondary Role +## - pg_hba.conf +## +host all all <trusted secondary IP>/32 md5 +host replication gitlab_replicator <trusted secondary IP>/32 md5 +host all all <trusted primary IP>/24 md5 +``` ```plaintext ## @@ -145,6 +159,19 @@ outbound rules do not apply to RDS PostgreSQL databases. Therefore, you need to rule to the read-replica's security group allowing any TCP traffic from the tracking database on port 5432. +1. Ensure that your secondary node can communicate with your tracking database by + manually changing the `pg_hba.conf` that is associated with your tracking database. + Remember to restart PostgreSQL afterwards for the changes to take effect: + + ```plaintext + ## + ## Geo Tracking Database Role + ## - pg_hba.conf + ## + host all all <trusted tracking IP>/32 md5 + host all all <trusted secondary IP>/32 md5 + ``` + 1. SSH into a GitLab **secondary** server and login as root: ```shell diff --git a/doc/administration/geo/replication/img/geo_node_dashboard.png b/doc/administration/geo/replication/img/geo_node_dashboard.png Binary files differindex 0d3dc5895af..53ca4767c5b 100644 --- a/doc/administration/geo/replication/img/geo_node_dashboard.png +++ b/doc/administration/geo/replication/img/geo_node_dashboard.png diff --git a/doc/administration/geo/replication/updating_the_geo_nodes.md b/doc/administration/geo/replication/updating_the_geo_nodes.md index df66b1b36ec..3ce35798406 100644 --- a/doc/administration/geo/replication/updating_the_geo_nodes.md +++ b/doc/administration/geo/replication/updating_the_geo_nodes.md @@ -44,7 +44,7 @@ and all **secondary** nodes: Now that the update process is complete, you may want to check whether everything is working correctly: -1. Run the Geo raketask on all nodes, everything should be green: +1. Run the Geo Rake task on all nodes, everything should be green: ```shell sudo gitlab-rake gitlab:geo:check diff --git a/doc/administration/gitaly/img/praefect_architecture_v12_10.png b/doc/administration/gitaly/img/praefect_architecture_v12_10.png Binary files differindex 7b8f1138b23..024a12b0a5d 100644 --- a/doc/administration/gitaly/img/praefect_architecture_v12_10.png +++ b/doc/administration/gitaly/img/praefect_architecture_v12_10.png diff --git a/doc/administration/gitaly/index.md b/doc/administration/gitaly/index.md index 20b9f2104de..cd3c61c1604 100644 --- a/doc/administration/gitaly/index.md +++ b/doc/administration/gitaly/index.md @@ -101,23 +101,43 @@ Omnibus GitLab or install it from source: **_do not_** provide the `EXTERNAL_URL=` value. - From source: [Install Gitaly](../../install/installation.md#install-gitaly). -### 2. Client side token configuration +### 2. Authentication -Configure a token on the instance that runs the GitLab Rails application. +Gitaly and GitLab use two shared secrets for authentication, one to authenticate gRPC requests +to Gitaly, and a second for authentication callbacks from Gitaly to the GitLab internal API. **For Omnibus GitLab** -1. On the client node(s), edit `/etc/gitlab/gitlab.rb`: +There are two ways to configure the required tokens: + +1. Copy `/etc/gitlab/gitlab-secrets.json` from the client server to same path on the Gitaly server. +1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). + +**OR** + +1. On the client server, edit `/etc/gitlab/gitlab.rb`: ```ruby gitlab_rails['gitaly_token'] = 'abc123secret' + gitlab_shell['secret_token'] = 'shellsecret' ``` 1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). +1. On the Gitaly server, edit `/etc/gitlab/gitlab.rb`: + + ```ruby + gitaly['auth_token'] = 'abc123secret' + gitlab_shell['secret_token'] = 'shellsecret' + ``` + +1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). + **For installations from source** -1. On the client node(s), edit `/home/git/gitlab/config/gitlab.yml`: +1. Copy `/home/git/gitlab/.gitlab_shell_secret` from the client server to the same path on the Gitaly +server. +1. On the client server, edit `/home/git/gitlab/config/gitlab.yml`: ```yaml gitlab: @@ -138,12 +158,6 @@ documentation on configuring Gitaly authentication](https://gitlab.com/gitlab-org/gitaly/blob/master/doc/configuration/README.md#authentication) . -Gitaly must trigger some callbacks to GitLab via GitLab Shell. As a result, -the GitLab Shell secret must be the same between the other GitLab servers and -the Gitaly server. The easiest way to accomplish this is to copy `/etc/gitlab/gitlab-secrets.json` -from an existing GitLab server to the Gitaly server. Without this shared secret, -Git operations in GitLab will result in an API error. - **For Omnibus GitLab** 1. Edit `/etc/gitlab/gitlab.rb`: @@ -189,10 +203,6 @@ Git operations in GitLab will result in an API error. # Don't forget to copy `/etc/gitlab/gitlab-secrets.json` from web server to Gitaly server. gitlab_rails['internal_api_url'] = 'https://gitlab.example.com' - # Authentication token to ensure only authorized servers can communicate with - # Gitaly server - gitaly['auth_token'] = 'abc123secret' - # Make Gitaly accept connections on all network interfaces. You must use # firewalls to restrict access to this address/port. # Comment out following line if you only want to support TLS connections @@ -230,6 +240,8 @@ Git operations in GitLab will result in an API error. ``` 1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). +1. Run `sudo /opt/gitlab/embedded/service/gitlab-shell/bin/check -config /opt/gitlab/embedded/service/gitlab-shell/config.yml` +to confirm that Gitaly can perform callbacks to the internal API. **For installations from source** @@ -271,7 +283,15 @@ Git operations in GitLab will result in an API error. path = '/srv/gitlab/git-data/repositories' ``` +1. On each Gitaly server, edit `/home/git/gitlab-shell/config.yml`: + + ```yaml + gitlab_url: https://gitlab.example.com + ``` + 1. Save the file and [restart GitLab](../restart_gitlab.md#installations-from-source). +1. Run `sudo -u git /home/git/gitlab-shell/bin/check -config /home/git/gitlab-shell/config.yml` +to confirm that Gitaly can perform callbacks to the internal API. ### 4. Converting clients to use the Gitaly server @@ -302,11 +322,10 @@ can read and write to `/mnt/gitlab/storage2`. 'storage1' => { 'gitaly_address' => 'tcp://gitaly1.internal:8075' }, 'storage2' => { 'gitaly_address' => 'tcp://gitaly2.internal:8075' }, }) - - gitlab_rails['gitaly_token'] = 'abc123secret' ``` 1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). +1. Run `sudo gitlab-rake gitlab:gitaly:check` to confirm the client can connect to Gitaly. 1. Tail the logs to see the requests: ```shell @@ -330,9 +349,6 @@ can read and write to `/mnt/gitlab/storage2`. storage2: gitaly_address: tcp://gitaly2.internal:8075 path: /some/dummy/path - - gitaly: - token: 'abc123secret' ``` NOTE: **Note:** @@ -341,6 +357,8 @@ can read and write to `/mnt/gitlab/storage2`. [this issue](https://gitlab.com/gitlab-org/gitaly/issues/1282) is resolved. 1. Save the file and [restart GitLab](../restart_gitlab.md#installations-from-source). +1. Run `sudo -u git -H bundle exec rake gitlab:gitaly:check RAILS_ENV=production` to +confirm the client can connect to Gitaly. 1. Tail the logs to see the requests: ```shell @@ -430,17 +448,32 @@ To configure Gitaly with TLS: }) gitlab_rails['gitaly_token'] = 'abc123secret' + gitlab_shell['secret_token'] = 'shellsecret' ``` 1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) on client node(s). +1. On the client node(s), copy the cert into the `/etc/gitlab/trusted-certs`: + + ```shell + sudo cp cert.pem /etc/gitlab/trusted-certs/ + ``` + 1. On the Gitaly server, create the `/etc/gitlab/ssl` directory and copy your key and certificate there: ```shell sudo mkdir -p /etc/gitlab/ssl sudo chmod 755 /etc/gitlab/ssl sudo cp key.pem cert.pem /etc/gitlab/ssl/ + sudo chmod 644 key.pem cert.pem ``` +1. Copy the cert to `/etc/gitlab/trusted-certs` so Gitaly will trust the cert when +calling into itself: + + ```shell + sudo cp /etc/gitlab/ssl/cert.pem /etc/gitlab/trusted-certs/ + ``` + 1. On the Gitaly server node(s), edit `/etc/gitlab/gitlab.rb` and add: <!-- @@ -463,6 +496,13 @@ To configure Gitaly with TLS: **For installations from source** +1. On the client node(s), add the cert to the system trusted certs: + + ```shell + sudo cp cert.pem /usr/local/share/ca-certificates/gitaly.crt + sudo update-ca-certificates + ``` + 1. On the client node(s), edit `/home/git/gitlab/config/gitlab.yml` as follows: ```yaml @@ -488,13 +528,32 @@ To configure Gitaly with TLS: data will be stored in this folder. This will no longer be necessary after [this issue](https://gitlab.com/gitlab-org/gitaly/issues/1282) is resolved. -1. Save the file and [restart GitLab](../restart_gitlab.md#installations-from-source) on client node(s). +1. Save the file and[restart GitLab](../restart_gitlab.md#installations-from-source) +on client node(s). +1. Copy `/home/git/gitlab/.gitlab_shell_secret` from the client server to the same +path on the Gitaly server. +1. On the Gitaly server, create or edit `/etc/default/gitlab` and add: + + ```shell + export SSL_CERT_DIR=/etc/gitlab/ssl + ``` + +1. Save the file. 1. Create the `/etc/gitlab/ssl` directory and copy your key and certificate there: ```shell sudo mkdir -p /etc/gitlab/ssl - sudo chmod 700 /etc/gitlab/ssl + sudo chmod 755 /etc/gitlab/ssl sudo cp key.pem cert.pem /etc/gitlab/ssl/ + sudo chmod 644 key.pem cert.pem + ``` + +1. On the Gitaly server, add the cert to the system trusted certs so Gitaly will trust it +when calling into itself: + + ```shell + sudo cp cert.pem /usr/local/share/ca-certificates/gitaly.crt + sudo update-ca-certificates ``` 1. On the Gitaly server node(s), edit `/home/git/gitaly/config.toml` and add: diff --git a/doc/administration/gitaly/praefect.md b/doc/administration/gitaly/praefect.md index c16d705db1d..bdac66ffeff 100644 --- a/doc/administration/gitaly/praefect.md +++ b/doc/administration/gitaly/praefect.md @@ -1,34 +1,36 @@ # Praefect: High Availability -NOTE: **Note:** Praefect is an experimental service, and data loss is likely. +NOTE: **Note:** Praefect is a +[beta](https://about.gitlab.com/handbook/product/#alpha-beta-ga) component that +allows Gitaly to be run in a highly available configuration. While unexpected +data loss is not likely, Praefect is not yet ready for production environments. Praefect is an optional reverse-proxy for [Gitaly](../index.md) to manage a -cluster of Gitaly nodes for high availability. Initially, high availability -be implemented through asynchronous replication. If a Gitaly node becomes -unavailable, it will be possible to fail over to a warm Gitaly replica. +cluster of Gitaly nodes for high availability. High availability is currently +implemented through asynchronous replication. If a Gitaly node becomes +unavailable, Praefect will automatically route traffic to a warm Gitaly replica. -The first minimal version will support: +The current version supports: - Eventual consistency of the secondary replicas. - Automatic fail over from the primary to the secondary. - Reporting of possible data loss if replication queue is non empty. Follow the [HA Gitaly epic](https://gitlab.com/groups/gitlab-org/-/epics/1489) -for updates and roadmap. +for improvements including +[strong consistency](https://gitlab.com/groups/gitlab-org/-/epics/1189) and +[horizontally distributing reads](https://gitlab.com/groups/gitlab-org/-/epics/2013). ## Requirements for configuring Gitaly for High Availability -NOTE: **Note:** this reference architecture is not highly available because -Praefect is a single point of failure. +A minimum highly available configuration requires: -The minimal [alpha](https://about.gitlab.com/handbook/product/#alpha-beta-ga) -reference architecture additionally requires: - -- 1 Praefect node -- 1 PostgreSQL server (PostgreSQL 9.6 or newer) +- 1 highly available load balancer +- 1 highly available PostgreSQL server (PostgreSQL 9.6 or newer) +- 2 Praefect nodes - 3 Gitaly nodes (1 primary, 2 secondary) - + See the [design document](https://gitlab.com/gitlab-org/gitaly/-/blob/master/doc/design_ha.md) @@ -64,6 +66,7 @@ GitLab](https://about.gitlab.com/install/). You will need the IP/host address for each node. +1. `LOADBALANCER_SERVER_ADDRESS`: the IP/hots address of the load balancer 1. `POSTGRESQL_SERVER_ADDRESS`: the IP/host address of the PostgreSQL server 1. `PRAEFECT_HOST`: the IP/host address of the Praefect server 1. `GITALY_HOST`: the IP/host address of each Gitaly server @@ -98,8 +101,10 @@ We will note in the instructions below where these secrets are required. ### PostgreSQL -NOTE: **Note:** don't reuse the GitLab application database for the Praefect -database. +NOTE: **Note:** do not store the GitLab application database and the Praefect +database on the same PostgreSQL server if using +[Geo](../geo/replication/index.md). The replication state is internal to each instance +of GitLab and should not be replicated. To complete this section you will need: @@ -245,9 +250,9 @@ application server, or a Gitaly node. 1. Configure the **Praefect** cluster to connect to each Gitaly node in the cluster by editing `/etc/gitlab/gitlab.rb`. - In the example below we have configured one cluster named `praefect`. This - cluster has three Gitaly nodes `gitaly-1`, `gitaly-2`, and `gitaly-3`, which - will be replicas of each other. + In the example below we have configured one virtual storage (or shard) named + `storage-1`. This cluster has three Gitaly nodes `gitaly-1`, `gitaly-2`, and + `gitaly-3`, which will be replicas of each other. Replace `PRAEFECT_INTERNAL_TOKEN` with a strong secret, which will be used by Praefect when communicating with Gitaly nodes in the cluster. This token is @@ -266,7 +271,7 @@ application server, or a Gitaly node. # Name of storage hash must match storage name in git_data_dirs on GitLab # server ('praefect') and in git_data_dirs on Gitaly nodes ('gitaly-1') praefect['virtual_storages'] = { - 'praefect' => { + 'storage-1' => { 'gitaly-1' => { 'address' => 'tcp://GITALY_HOST:8075', 'token' => 'PRAEFECT_INTERNAL_TOKEN', @@ -284,6 +289,12 @@ application server, or a Gitaly node. } ``` +1. Enable the replication queue: + + ```ruby + praefect['postgres_queue_enabled'] = true + ``` + 1. Save the changes to `/etc/gitlab/gitlab.rb` and [reconfigure Praefect](../restart_gitlab.md#omnibus-gitlab-reconfigure): ```shell @@ -300,6 +311,60 @@ application server, or a Gitaly node. edit `/etc/gitlab/gitlab.rb`, remember to run `sudo gitlab-ctl reconfigure` again before trying the `sql-ping` command. +#### Automatic failover + +When automatic failover is enabled, Praefect will do automatic detection of the health of internal Gitaly nodes. If the +primary has a certain amount of health checks fail, it will decide to promote one of the secondaries to be primary, and +demote the primary to be a secondary. + +1. To enable automatic failover, edit `/etc/gitlab/gitlab.rb`: + + ```ruby + # failover_enabled turns on automatic failover + praefect['failover_enabled'] = true + praefect['virtual_storages'] = { + 'storage-1' => { + 'gitaly-1' => { + 'address' => 'tcp://GITALY_HOST:8075', + 'token' => 'PRAEFECT_INTERNAL_TOKEN', + 'primary' => true + }, + 'gitaly-2' => { + 'address' => 'tcp://GITALY_HOST:8075', + 'token' => 'PRAEFECT_INTERNAL_TOKEN' + }, + 'gitaly-3' => { + 'address' => 'tcp://GITALY_HOST:8075', + 'token' => 'PRAEFECT_INTERNAL_TOKEN' + } + } + } + ``` + +1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). + +Below is the picture when Praefect starts up with the config.toml above: + +```mermaid +graph TD + A[Praefect] -->|Mutator RPC| B(internal_storage_0) + B --> |Replication|C[internal_storage_1] +``` + +Let's say suddenly `internal_storage_0` goes down. Praefect will detect this and +automatically switch over to `internal_storage_1`, and `internal_storage_0` will serve as a secondary: + +```mermaid +graph TD + A[Praefect] -->|Mutator RPC| B(internal_storage_1) + B --> |Replication|C[internal_storage_0] +``` + +NOTE: **Note:**: Currently this feature is supported for setups that only have 1 Praefect instance. Praefect instances running, +for example behind a load balancer, `failover_enabled` should be disabled. The reason is The reason is because there +is no coordination that currently happens across different Praefect instances, so there could be a situation where +two Praefect instances think two different Gitaly nodes are the primary. + ### Gitaly NOTE: **Note:** Complete these steps for **each** Gitaly node. @@ -501,7 +566,7 @@ Particular attention should be shown to: - the storage name added to `git_data_dirs` in this section must match the storage name under `praefect['virtual_storages']` on the Praefect node. This was set in the [Praefect](#praefect) section of this guide. This document uses - `praefect` as the Praefect storage name. + `storage-1` as the Praefect storage name. 1. SSH into the **GitLab** node and login as root: @@ -523,7 +588,7 @@ Particular attention should be shown to: "default" => { "gitaly_address" => "tcp://GITLAB_HOST:8075" }, - "praefect" => { + "storage-1" => { "gitaly_address" => "tcp://PRAEFECT_HOST:2305", "gitaly_token" => 'PRAEFECT_EXTERNAL_TOKEN' } diff --git a/doc/administration/high_availability/README.md b/doc/administration/high_availability/README.md index 55ec3b8d6c4..c84456e17a8 100644 --- a/doc/administration/high_availability/README.md +++ b/doc/administration/high_availability/README.md @@ -2,6 +2,9 @@ type: reference, concepts --- -# High Availability +The page have been deprecated, please see: -This content has been moved to the [availability page](../availability/index.md). +1. [Availability page](../availability/index.md) +1. [Scaling page](../scaling/index.md) +1. [Docs page for high availability](./gitlab.md) +1. [High availability solutions page](https://about.gitlab.com/solutions/high-availability/) diff --git a/doc/administration/high_availability/gitaly.md b/doc/administration/high_availability/gitaly.md index 5d66d3c5c94..0b5bf8d0c4c 100644 --- a/doc/administration/high_availability/gitaly.md +++ b/doc/administration/high_availability/gitaly.md @@ -19,7 +19,7 @@ See [Running Gitaly on its own server](../gitaly/index.md#running-gitaly-on-its- in Gitaly documentation. Continue configuration of other components by going back to the -[High Availability](../availability/index.md#gitlab-components-and-configuration-instructions) page. +[Scaling](../scaling/index.md#components-provided-by-omnibus-gitlab) page. ## Enable Monitoring diff --git a/doc/administration/high_availability/redis.md b/doc/administration/high_availability/redis.md index a1e5482e5dc..11e930cb53b 100644 --- a/doc/administration/high_availability/redis.md +++ b/doc/administration/high_availability/redis.md @@ -8,10 +8,15 @@ type: reference The following are the requirements for providing your own Redis instance: -- GitLab 12.0 and later requires Redis version 3.2 or higher. Version 3.2 or higher is recommend as this is - what ships with the GitLab Omnibus package. Older Redis versions do not - support an optional count argument to SPOP which is now required for - [Merge Trains](../../ci/merge_request_pipelines/pipelines_for_merged_results/merge_trains/index.md). +- Redis version 5.0 or higher is recommended, as this is what ships with + Omnibus GitLab packages starting with GitLab 12.7. +- Support for Redis 3.2 is deprecated with GitLab 12.10 and will be completely + removed in GitLab 13.0. +- GitLab 12.0 and later requires Redis version 3.2 or higher. Older Redis + versions do not support an optional count argument to SPOP which is now + required for [Merge Trains](../../ci/merge_request_pipelines/pipelines_for_merged_results/merge_trains/index.md). +- In addition, if Redis 4 or later is available, GitLab makes use of certain + commands like `UNLINK` and `USAGE` which were introduced only in Redis 4. - Standalone Redis or Redis high availability with Sentinel are supported. Redis Cluster is not supported. - Managed Redis from cloud providers such as AWS ElastiCache will work. If these @@ -84,7 +89,7 @@ Advanced configuration options are supported and can be added if needed. Continue configuration of other components by going back to the -[High Availability](../availability/index.md#gitlab-components-and-configuration-instructions) page. +[Scaling](../scaling/index.md#components-provided-by-omnibus-gitlab) page. ### High Availability with GitLab Omnibus **(PREMIUM ONLY)** diff --git a/doc/administration/index.md b/doc/administration/index.md index 1f4e23fce8a..9b360d3b253 100644 --- a/doc/administration/index.md +++ b/doc/administration/index.md @@ -76,7 +76,7 @@ Learn how to install, configure, update, and maintain your GitLab instance. ### Maintaining GitLab -- [Raketasks](../raketasks/README.md): Perform various tasks for maintenance, backups, automatic webhooks setup, and more. +- [Rake tasks](../raketasks/README.md): Perform various tasks for maintenance, backups, automatic webhooks setup, and more. - [Backup and restore](../raketasks/backup_restore.md): Backup and restore your GitLab instance. - [Operations](operations/index.md): Keeping GitLab up and running (clean up Redis sessions, moving repositories, Sidekiq MemoryKiller, Unicorn). - [Restart GitLab](restart_gitlab.md): Learn how to restart GitLab and its components. @@ -98,7 +98,7 @@ Learn how to install, configure, update, and maintain your GitLab instance. - [Mattermost](https://docs.gitlab.com/omnibus/gitlab-mattermost/): Integrate with [Mattermost](https://mattermost.com), an open source, private cloud workplace for web messaging. - [PlantUML](integration/plantuml.md): Create simple diagrams in AsciiDoc and Markdown documents - created in snippets, wikis, and repos. + created in snippets, wikis, and repositories. - [Web terminals](integration/terminal.md): Provide terminal access to your applications deployed to Kubernetes from within GitLab's CI/CD [environments](../ci/environments.md#web-terminals). ## User settings and permissions diff --git a/doc/administration/instance_review.md b/doc/administration/instance_review.md index bb64c7967b7..8ea4bff252e 100644 --- a/doc/administration/instance_review.md +++ b/doc/administration/instance_review.md @@ -1,6 +1,6 @@ # Instance Review **(CORE ONLY)** -> [Introduced][6995] in [GitLab Core][ee] 11.3. +> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/6995) in [GitLab Core](https://about.gitlab.com/pricing/) 11.3. If you are running a medium size instance of GitLab Core edition you are qualified for a free Instance Review. You can find the button in the User menu. @@ -11,6 +11,3 @@ When you click the button you will be redirected to a form with prefilled data o Once you submit the data to GitLab Inc. you can see the initial report. Additionally you will be contacted by our team for further review which should help you to improve your usage of GitLab. - -[6995]: https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/6995 -[ee]: https://about.gitlab.com/pricing/ diff --git a/doc/administration/issue_closing_pattern.md b/doc/administration/issue_closing_pattern.md index 7b815143597..579b957eb47 100644 --- a/doc/administration/issue_closing_pattern.md +++ b/doc/administration/issue_closing_pattern.md @@ -28,7 +28,7 @@ Because Rubular doesn't understand `%{issue_ref}`, you can replace this by expression of your liking: ```ruby - gitlab_rails['gitlab_issue_closing_pattern'] = "\b((?:[Cc]los(?:e[sd]|ing)|\b[Ff]ix(?:e[sd]|ing)?) +(?:(?:issues? +)?%{issue_ref}(?:(?:, *| +and +)?))+)" + gitlab_rails['gitlab_issue_closing_pattern'] = "\b((?:[Cc]los(?:e[sd]?|ing)|\b[Ff]ix(?:e[sd]|ing)?|\b[Rr]esolv(?:e[sd]?|ing)|\b[Ii]mplement(?:s|ed|ing)?)(:?) +(?:(?:issues? +)?%{issue_ref}(?:(?: *,? +and +| *,? *)?)|([A-Z][A-Z0-9_]+-\d+))+)" ``` 1. [Reconfigure](restart_gitlab.md#omnibus-gitlab-reconfigure) GitLab for the changes to take effect. @@ -39,7 +39,7 @@ Because Rubular doesn't understand `%{issue_ref}`, you can replace this by 1. Change the value of `issue_closing_pattern`: ```yaml - issue_closing_pattern: "\b((?:[Cc]los(?:e[sd]|ing)|\b[Ff]ix(?:e[sd]|ing)?) +(?:(?:issues? +)?%{issue_ref}(?:(?:, *| +and +)?))+)" + issue_closing_pattern: "\b((?:[Cc]los(?:e[sd]?|ing)|\b[Ff]ix(?:e[sd]|ing)?|\b[Rr]esolv(?:e[sd]?|ing)|\b[Ii]mplement(?:s|ed|ing)?)(:?) +(?:(?:issues? +)?%{issue_ref}(?:(?: *,? +and +| *,? *)?)|([A-Z][A-Z0-9_]+-\d+))+)" ``` 1. [Restart](restart_gitlab.md#installations-from-source) GitLab for the changes to take effect. diff --git a/doc/administration/job_artifacts.md b/doc/administration/job_artifacts.md index 3777f551996..9bbbc7d228b 100644 --- a/doc/administration/job_artifacts.md +++ b/doc/administration/job_artifacts.md @@ -378,7 +378,7 @@ and [projects APIs](../api/projects.md). When GitLab receives an artifacts archive, an archive metadata file is also generated by [GitLab Workhorse](https://gitlab.com/gitlab-org/gitlab-workhorse). This metadata file describes all the entries that are located in the artifacts archive itself. -The metadata file is in a binary format, with additional GZIP compression. +The metadata file is in a binary format, with additional Gzip compression. GitLab does not extract the artifacts archive in order to save space, memory and disk I/O. It instead inspects the metadata file which contains all the diff --git a/doc/administration/logs.md b/doc/administration/logs.md index c43406fb647..b43a6546082 100644 --- a/doc/administration/logs.md +++ b/doc/administration/logs.md @@ -30,17 +30,21 @@ Line breaks have been added to this example for legibility: "controller":"Projects::IssuesController", "action":"show", "status":200, - "duration":229.03, - "view":174.07, - "db":13.24, "time":"2017-08-08T20:15:54.821Z", "params":[{"key":"param_key","value":"param_value"}], "remote_ip":"18.245.0.1", "user_id":1, "username":"admin", - "gitaly_calls":76, - "gitaly_duration":7.41, - "queue_duration": 112.47 + "queue_duration_s":0.0, + "gitaly_calls":16, + "gitaly_duration_s":0.16, + "redis_calls":115, + "redis_duration_s":0.13, + "correlation_id":"O1SdybnnIq7", + "cpu_s":17.50, + "db_duration_s":0.08, + "view_duration_s":2.39, + "duration_s":20.54 } ``` @@ -48,12 +52,15 @@ This example was a GET request for a specific issue. Each line also contains performance data, with times in milliseconds: -1. `duration`: total time taken to retrieve the request -1. `queue_duration`: total time that the request was queued inside GitLab Workhorse -1. `view`: total time taken inside the Rails views -1. `db`: total time to retrieve data from the database +1. `duration_s`: total time taken to retrieve the request +1. `queue_duration_s`: total time that the request was queued inside GitLab Workhorse +1. `view_duration_s`: total time taken inside the Rails views +1. `db_duration_s`: total time to retrieve data from PostgreSQL +1. `redis_duration_s`: total time to retrieve data from Redis +1. `cpu_s`: total time spent on CPU +1. `gitaly_duration_s`: total time taken by Gitaly calls 1. `gitaly_calls`: total number of calls made to Gitaly -1. `gitaly_duration`: total time taken by Gitaly calls +1. `redis_calls`: total number of calls made to Redis User clone and fetch activity using HTTP transport appears in this log as `action: git_upload_pack`. @@ -73,9 +80,6 @@ NOTE: **Note:** Starting with GitLab 12.5, if an error occurs, an "controller": "Admin::DashboardController", "action": "index", "status": 500, - "duration": 2584.11, - "view": 0, - "db": 9.21, "time": "2019-11-14T13:12:46.156Z", "params": [], "remote_ip": "127.0.0.1", @@ -84,7 +88,16 @@ NOTE: **Note:** Starting with GitLab 12.5, if an error occurs, an "ua": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.14; rv:70.0) Gecko/20100101 Firefox/70.0", "queue_duration": 274.35, "correlation_id": "KjDVUhNvvV3", - "cpu_s": 2.837645135999999, + "queue_duration_s":0.0, + "gitaly_calls":16, + "gitaly_duration_s":0.16, + "redis_calls":115, + "redis_duration_s":0.13, + "correlation_id":"O1SdybnnIq7", + "cpu_s":17.50, + "db_duration_s":0.08, + "view_duration_s":2.39, + "duration_s":20.54 "exception.class": "NameError", "exception.message": "undefined local variable or method `adsf' for #<Admin::DashboardController:0x00007ff3c9648588>", "exception.backtrace": [ @@ -417,15 +430,85 @@ you've configured this for Sidekiq as mentioned above. ## `gitlab-shell.log` -This file lives in `/var/log/gitlab/gitaly/gitlab-shell.log` for -Omnibus GitLab packages or in `/home/git/gitaly/gitlab-shell.log` for -installations from source. +GitLab Shell is used by GitLab for executing Git commands and provide SSH access to Git repositories. + +### For GitLab versions 12.10 and up + +For GitLab version 12.10 and later, there are 2 `gitlab-shell.log` files. Information containing `git-{upload-pack,receive-pack}` requests lives in `/var/log/gitlab/gitlab-shell/gitlab-shell.log`. Information about hooks to GitLab Shell from Gitaly lives in `/var/log/gitlab/gitaly/gitlab-shell.log`. + +Example log entries for `/var/log/gitlab/gitlab-shell/gitlab-shell.log`: + +```json +{ + "duration_ms": 74.104, + "level": "info", + "method": "POST", + "msg": "Finished HTTP request", + "time": "2020-04-17T20:28:46Z", + "url": "http://127.0.0.1:8080/api/v4/internal/allowed" +} +{ + "command": "git-upload-pack", + "git_protocol": "", + "gl_project_path": "root/example", + "gl_repository": "project-1", + "level": "info", + "msg": "executing git command", + "time": "2020-04-17T20:28:46Z", + "user_id": "user-1", + "username": "root" +} +``` + +Example log entries for `/var/log/gitlab/gitaly/gitlab-shell.log`: + +```json +{ + "method": "POST", + "url": "http://127.0.0.1:8080/api/v4/internal/allowed", + "duration": 0.058012959, + "gitaly_embedded": true, + "pid": 16636, + "level": "info", + "msg": "finished HTTP request", + "time": "2020-04-17T20:29:08+00:00" +} +{ + "method": "POST", + "url": "http://127.0.0.1:8080/api/v4/internal/pre_receive", + "duration": 0.031022552, + "gitaly_embedded": true, + "pid": 16636, + "level": "info", + "msg": "finished HTTP request", + "time": "2020-04-17T20:29:08+00:00" +} +``` + +### For GitLab versions 12.5 through 12.9 + +For GitLab 12.5 to 12.9, this file lives in `/var/log/gitlab/gitaly/gitlab-shell.log` for Omnibus GitLab packages or in `/home/git/gitaly/gitlab-shell.log` for installations from source. + +Example log entries: + +```json +{ + "method": "POST", + "url": "http://127.0.0.1:8080/api/v4/internal/post_receive", + "duration": 0.031809164, + "gitaly_embedded": true, + "pid": 27056, + "level": "info", + "msg": "finished HTTP request", + "time": "2020-04-17T16:24:38+00:00" +} +``` + +### For GitLab 12.5 and earlier -NOTE: **Note:** For GitLab 12.5 and earlier, the file lives in `/var/log/gitlab/gitlab-shell/gitlab-shell.log`. -GitLab Shell is used by GitLab for executing Git commands and provide -SSH access to Git repositories. For example: +Example log entries: ```plaintext I, [2015-02-13T06:17:00.671315 #9291] INFO -- : Adding project root/example.git at </var/opt/gitlab/git-data/repositories/root/dcdcdcdcd.git>. diff --git a/doc/administration/pseudonymizer.md b/doc/administration/pseudonymizer.md index 36bb446da78..22543a5c743 100644 --- a/doc/administration/pseudonymizer.md +++ b/doc/administration/pseudonymizer.md @@ -1,6 +1,6 @@ # Pseudonymizer **(ULTIMATE)** -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/5532) in [GitLab Ultimate][ee] 11.1. +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/5532) in [GitLab Ultimate](https://about.gitlab.com/pricing/) 11.1. As GitLab's database hosts sensitive information, using it unfiltered for analytics implies high security requirements. To help alleviate this constraint, the Pseudonymizer @@ -101,5 +101,3 @@ This will produce some CSV files that might be very large, so make sure the After the pseudonymizer has run, the output CSV files should be uploaded to the configured object storage and deleted from the local disk. - -[ee]: https://about.gitlab.com/pricing/ diff --git a/doc/administration/raketasks/check.md b/doc/administration/raketasks/check.md index c0edb43cc01..f55c6e8e86b 100644 --- a/doc/administration/raketasks/check.md +++ b/doc/administration/raketasks/check.md @@ -133,5 +133,3 @@ The LDAP check Rake task will test the bind DN and password credentials (if configured) and will list a sample of LDAP users. This task is also executed as part of the `gitlab:check` task, but can run independently. See [LDAP Rake Tasks - LDAP Check](ldap.md#check) for details. - -[git-fsck]: https://git-scm.com/docs/git-fsck diff --git a/doc/administration/raketasks/praefect.md b/doc/administration/raketasks/praefect.md new file mode 100644 index 00000000000..4f9cf0e10ba --- /dev/null +++ b/doc/administration/raketasks/praefect.md @@ -0,0 +1,22 @@ +# Praefect Rake Tasks + +> [Introduced]( https://gitlab.com/gitlab-org/gitlab/-/merge_requests/28369) in GitLab 12.10. + +## Replica checksums + +Prints out checksums of the repository of a given project_id on the primary as well as secondary internal Gitaly nodes. + +NOTE: **Note:** +This only is relevant and works for projects that have been created on a praefect storage. See the [Praefect Documentation](../gitaly/praefect.md) for configuring Praefect. + +**Omnibus Installation** + +```shell +sudo gitlab-rake "gitlab:praefect:replicas[project_id]" +``` + +**Source Installation** + +```shell +sudo -u git -H bundle exec rake "gitlab:praefect:replicas[project_id]" RAILS_ENV=production +``` diff --git a/doc/administration/raketasks/storage.md b/doc/administration/raketasks/storage.md index 1e487a317c9..e5ed9cd0dc1 100644 --- a/doc/administration/raketasks/storage.md +++ b/doc/administration/raketasks/storage.md @@ -4,12 +4,12 @@ This is a collection of Rake tasks you can use to help you list and migrate existing projects and attachments associated with it from Legacy storage to the new Hashed storage type. -You can read more about the storage types [here][storage-types]. +You can read more about the storage types [here](../repository_storage_types.md). ## Migrate existing projects to Hashed storage Before migrating your existing projects, you should -[enable hashed storage][storage-migration] for the new projects as well. +[enable hashed storage](../repository_storage_types.md#how-to-migrate-to-hashed-storage) for the new projects as well. This task will schedule all your existing projects and attachments associated with it to be migrated to the **Hashed** storage type: @@ -56,7 +56,7 @@ If you need to rollback the storage migration for any reason, you can follow the NOTE: **Note:** Hashed Storage will be required in future version of GitLab. To prevent new projects from being created in the Hashed storage, -you need to undo the [enable hashed storage][storage-migration] changes. +you need to undo the [enable hashed storage](../repository_storage_types.md#how-to-migrate-to-hashed-storage) changes. This task will schedule all your existing projects and associated attachments to be rolled back to the Legacy storage type. @@ -209,6 +209,3 @@ sudo gitlab-rake gitlab:storage:list_hashed_attachments ```shell sudo -u git -H bundle exec rake gitlab:storage:list_hashed_attachments RAILS_ENV=production ``` - -[storage-types]: ../repository_storage_types.md -[storage-migration]: ../repository_storage_types.md#how-to-migrate-to-hashed-storage diff --git a/doc/administration/reply_by_email_postfix_setup.md b/doc/administration/reply_by_email_postfix_setup.md index 8a24514aec2..86854a2a7b6 100644 --- a/doc/administration/reply_by_email_postfix_setup.md +++ b/doc/administration/reply_by_email_postfix_setup.md @@ -1,7 +1,7 @@ # Set up Postfix for incoming email This document will take you through the steps of setting up a basic Postfix mail -server with IMAP authentication on Ubuntu, to be used with [incoming email]. +server with IMAP authentication on Ubuntu, to be used with [incoming email](incoming_email.md). The instructions make the assumption that you will be using the email address `incoming@gitlab.example.com`, that is, username `incoming` on host `gitlab.example.com`. Don't forget to change it to your actual host when executing the example code snippets. @@ -333,10 +333,8 @@ Courier, which we will install later to add IMAP authentication, requires mailbo ## Done -If all the tests were successful, Postfix is all set up and ready to receive email! Continue with the [incoming email] guide to configure GitLab. +If all the tests were successful, Postfix is all set up and ready to receive email! Continue with the [incoming email](incoming_email.md) guide to configure GitLab. --- _This document was adapted from <https://help.ubuntu.com/community/PostfixBasicSetupHowto>, by contributors to the Ubuntu documentation wiki._ - -[incoming email]: incoming_email.md diff --git a/doc/administration/repository_checks.md b/doc/administration/repository_checks.md index a647bc82660..d14b829bdcc 100644 --- a/doc/administration/repository_checks.md +++ b/doc/administration/repository_checks.md @@ -1,8 +1,8 @@ # Repository checks -> [Introduced][ce-3232] in GitLab 8.7. +> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/3232) in GitLab 8.7. -Git has a built-in mechanism, [`git fsck`][git-fsck], to verify the +Git has a built-in mechanism, [`git fsck`](https://git-scm.com/docs/git-fsck), to verify the integrity of all data committed to a repository. GitLab administrators can trigger such a check for a project via the project page under the admin panel. The checks run asynchronously so it may take a few minutes @@ -41,7 +41,3 @@ in `repocheck.log`: If the periodic repository check causes false alarms, you can clear all repository check states by navigating to **Admin Area > Settings > Repository** (`/admin/application_settings/repository`) and clicking **Clear all repository checks**. - ---- -[ce-3232]: https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/3232 "Auto git fsck" -[git-fsck]: https://git-scm.com/docs/git-fsck "git fsck documentation" diff --git a/doc/administration/repository_storage_paths.md b/doc/administration/repository_storage_paths.md index 857fb0b6a90..283401dafff 100644 --- a/doc/administration/repository_storage_paths.md +++ b/doc/administration/repository_storage_paths.md @@ -1,6 +1,6 @@ # Repository storage paths -> [Introduced][ce-4578] in GitLab 8.10. +> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/4578) in GitLab 8.10. GitLab allows you to define multiple repository storage paths (sometimes called storage shards) to distribute the storage load between several mount points. @@ -34,7 +34,7 @@ storage2: ## Configure GitLab > **Warning:** -> In order for [backups] to work correctly, the storage path must **not** be a +> In order for [backups](../raketasks/backup_restore.md) to work correctly, the storage path must **not** be a > mount point and the GitLab user should have correct permissions for the parent > directory of the path. In Omnibus GitLab this is taken care of automatically, > but for source installations you should be extra careful. @@ -47,7 +47,7 @@ storage2: > `gitlab.yml`. > > This little detail matters because while restoring a backup, the current -> contents of `/home/git/repositories` [are moved to][raketask] `/home/git/repositories.old`, +> contents of `/home/git/repositories` [are moved to](https://gitlab.com/gitlab-org/gitlab/blob/033e5423a2594e08a7ebcd2379bd2331f4c39032/lib/backup/repository.rb#L54-56) `/home/git/repositories.old`, > so if `/home/git/repositories` is the mount point, then `mv` would be moving > things between mount points, and bad things could happen. Ideally, > `/home/git` would be the mount point, so then things would be moving within the @@ -79,10 +79,10 @@ NOTE: **Note:** This example uses NFS. We do not recommend using EFS for storage path: /mnt/nfs2/repositories ``` -1. [Restart GitLab][restart-gitlab] for the changes to take effect. +1. [Restart GitLab](restart_gitlab.md#installations-from-source) for the changes to take effect. >**Note:** -The [`gitlab_shell: repos_path` entry][repospath] in `gitlab.yml` will be +The [`gitlab_shell: repos_path` entry](https://gitlab.com/gitlab-org/gitlab-foss/-/blob/8-9-stable/config/gitlab.yml.example#L457) in `gitlab.yml` will be deprecated and replaced by `repositories: storages` in the future, so if you are upgrading from a version prior to 8.10, make sure to add the configuration as described in the step above. After you make the changes and confirm they are @@ -114,9 +114,3 @@ Repository storage > Storage nodes for new repositories**. Beginning with GitLab 8.13.4, multiple paths can be chosen. New repositories will be randomly placed on one of the selected paths. - -[ce-4578]: https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/4578 -[restart-gitlab]: restart_gitlab.md#installations-from-source -[backups]: ../raketasks/backup_restore.md -[raketask]: https://gitlab.com/gitlab-org/gitlab/blob/033e5423a2594e08a7ebcd2379bd2331f4c39032/lib/backup/repository.rb#L54-56 -[repospath]: https://gitlab.com/gitlab-org/gitlab-foss/-/blob/8-9-stable/config/gitlab.yml.example#L457 diff --git a/doc/administration/repository_storage_types.md b/doc/administration/repository_storage_types.md index 2e2ed431c8b..1281131fab1 100644 --- a/doc/administration/repository_storage_types.md +++ b/doc/administration/repository_storage_types.md @@ -1,6 +1,6 @@ # Repository Storage Types -> [Introduced][ce-28283] in GitLab 10.0. +> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/issues/28283) in GitLab 10.0. Two different storage layouts can be used to store the repositories on disk and their characteristics. @@ -10,7 +10,7 @@ locations that can be: - Mounted to the local disk - Exposed as an NFS shared volume -- Accessed via [Gitaly] on its own machine. +- Accessed via [Gitaly](gitaly/index.md) on its own machine. In GitLab, this is configured in `/etc/gitlab/gitlab.rb` by the `git_data_dirs({})` configuration hash. The storage layouts discussed here will apply to any shard @@ -162,7 +162,7 @@ either runs on the same machine as your repositories are located, or may login t to access data (for example, a remote backup solution). To schedule a complete rollout, see the -[Rake task documentation for storage migration][rake/migrate-to-hashed] for instructions. +[Rake task documentation for storage migration](raketasks/storage.md#migrate-existing-projects-to-hashed-storage) for instructions. If you do have any existing integration, you may want to do a small rollout first, to validate. You can do so by specifying a range with the operation. @@ -174,7 +174,7 @@ an Omnibus GitLab installation: sudo gitlab-rake gitlab:storage:migrate_to_hashed ID_FROM=50 ID_TO=100 ``` -Check the [documentation][rake/migrate-to-hashed] for additional information and instructions for +Check the [documentation](raketasks/storage.md#migrate-existing-projects-to-hashed-storage) for additional information and instructions for source-based installation. #### Rollback @@ -203,7 +203,7 @@ the remaining repositories from the special `@hashed/` folder manually. We are incrementally moving every storable object in GitLab to the Hashed Storage pattern. You can check the current coverage status below (and also see -the [issue][ce-2821]). +the [issue](https://gitlab.com/gitlab-com/infrastructure/issues/2821)). Note that things stored in an S3 compatible endpoint will not have the downsides mentioned earlier, if they are not prefixed with `#{namespace}/#{project_name}`, @@ -246,8 +246,3 @@ storage pattern using 2 chars, 2 level folders, following Git's own implementati ``` LFS objects are also [S3 compatible](lfs/index.md#storing-lfs-objects-in-remote-object-storage). - -[ce-2821]: https://gitlab.com/gitlab-com/infrastructure/issues/2821 -[ce-28283]: https://gitlab.com/gitlab-org/gitlab-foss/issues/28283 -[rake/migrate-to-hashed]: raketasks/storage.md#migrate-existing-projects-to-hashed-storage -[gitaly]: gitaly/index.md diff --git a/doc/administration/restart_gitlab.md b/doc/administration/restart_gitlab.md index 907d7bb307a..c2e750ed34e 100644 --- a/doc/administration/restart_gitlab.md +++ b/doc/administration/restart_gitlab.md @@ -12,7 +12,7 @@ If you want the TL;DR versions, jump to: ## Omnibus installations -If you have used the [Omnibus packages][omnibus-dl] to install GitLab, then +If you have used the [Omnibus packages](https://about.gitlab.com/install/) to install GitLab, then you should already have `gitlab-ctl` in your `PATH`. `gitlab-ctl` interacts with the Omnibus packages and can be used to restart the @@ -23,7 +23,7 @@ GitLab Rails application (Unicorn) as well as the other components, like: - PostgreSQL (if you are using the bundled one) - NGINX (if you are using the bundled one) - Redis (if you are using the bundled one) -- [Mailroom][] +- [Mailroom](reply_by_email.md) - Logrotate ### Omnibus GitLab restart @@ -86,7 +86,7 @@ sudo gitlab-ctl reconfigure Reconfiguring GitLab should occur in the event that something in its configuration (`/etc/gitlab/gitlab.rb`) has changed. -When you run this command, [Chef], the underlying configuration management +When you run this command, [Chef](https://www.chef.io/products/chef-infra/), the underlying configuration management application that powers Omnibus GitLab, will make sure that all things like directories, permissions, and services are in place and in the same shape that they were initially shipped. @@ -101,7 +101,7 @@ depend on those files. ## Installations from source If you have followed the official installation guide to [install GitLab from -source][install], run the following command to restart GitLab: +source](../install/installation.md), run the following command to restart GitLab: ```shell sudo service gitlab restart @@ -128,23 +128,16 @@ The GitLab MailRoom email processor with pid 28114 is running. GitLab and all its components are up and running. ``` -This should restart Unicorn, Sidekiq, GitLab Workhorse, and [Mailroom][] +This should restart Unicorn, Sidekiq, GitLab Workhorse, and [Mailroom](reply_by_email.md) (if enabled). The init service file that does all the magic can be found on your server in `/etc/init.d/gitlab`. --- If you are using other init systems, like systemd, you can check the -[GitLab Recipes][gl-recipes] repository for some unofficial services. These are +[GitLab Recipes](https://gitlab.com/gitlab-org/gitlab-recipes/tree/master/init) repository for some unofficial services. These are **not** officially supported so use them at your own risk. -[omnibus-dl]: https://about.gitlab.com/install/ "Download the Omnibus packages" -[install]: ../install/installation.md "Documentation to install GitLab from source" -[mailroom]: reply_by_email.md "Used for replying by email in GitLab issues and merge requests" -[chef]: https://www.chef.io/products/chef-infra/ "Chef official website" -[src-service]: https://gitlab.com/gitlab-org/gitlab/blob/master/lib/support/init.d/gitlab "GitLab init service file" -[gl-recipes]: https://gitlab.com/gitlab-org/gitlab-recipes/tree/master/init "GitLab Recipes repository" - ## Helm chart installations There is no single command to restart the entire GitLab application installed via diff --git a/doc/administration/scaling/index.md b/doc/administration/scaling/index.md index ec7492883cc..f04517bc379 100644 --- a/doc/administration/scaling/index.md +++ b/doc/administration/scaling/index.md @@ -5,69 +5,39 @@ type: reference, concepts # Scaling GitLab supports a number of scaling options to ensure that your self-managed -instance is able to scale out to meet your organization's needs when scaling up -a single-box GitLab installation is no longer practical or feasible. +instance is able to scale to meet your organization's needs. -Please consult our [high availability documentation](../availability/index.md) -if your organization requires fault tolerance and redundancy features, such as -automatic database system failover. +On this page, we present examples of self-managed instances which demonstrate +how GitLab can be scaled up, scaled out or made highly available. These +examples progress from simple to complex as scaling or highly-available +components are added. -## GitLab components and scaling instructions - -Here's a list of components directly provided by Omnibus GitLab or installed as -part of a source installation and their configuration instructions for scaling. - -| Component | Description | Configuration instructions | -|-----------|-------------|----------------------------| -| [PostgreSQL](../../development/architecture.md#postgresql) | Database | [PostgreSQL configuration](https://docs.gitlab.com/omnibus/settings/database.html) | -| [Redis](../../development/architecture.md#redis) | Key/value store for fast data lookup and caching | [Redis configuration](../high_availability/redis.md) | -| [GitLab application services](../../development/architecture.md#unicorn) | Unicorn/Puma, Workhorse, GitLab Shell - serves front-end requests (UI, API, Git over HTTP/SSH) | [GitLab app scaling configuration](../high_availability/gitlab.md) | -| [PgBouncer](../../development/architecture.md#pgbouncer) | Database connection pooler | [PgBouncer configuration](../high_availability/pgbouncer.md#running-pgbouncer-as-part-of-a-non-ha-gitlab-installation) **(PREMIUM ONLY)** | -| [Sidekiq](../../development/architecture.md#sidekiq) | Asynchronous/background jobs | [Sidekiq configuration](../high_availability/sidekiq.md) | -| [Gitaly](../../development/architecture.md#gitaly) | Provides access to Git repositories | [Gitaly configuration](../gitaly/index.md#running-gitaly-on-its-own-server) | -| [Prometheus](../../development/architecture.md#prometheus) and [Grafana](../../development/architecture.md#grafana) | GitLab environment monitoring | [Monitoring node for scaling](../high_availability/monitoring_node.md) | - -## Third-party services used for scaling - -Here's a list of third-party services you may require as part of scaling GitLab. -The services can be provided by numerous applications or vendors and further -advice is given on how best to select the right choice for your organization's -needs. - -| Component | Description | Configuration instructions | -|-----------|-------------|----------------------------| -| Load balancer(s) | Handles load balancing, typically when you have multiple GitLab application services nodes | [Load balancer configuration](../high_availability/load_balancer.md) | -| Object storage service | Recommended store for shared data objects | [Cloud Object Storage configuration](../high_availability/object_storage.md) | -| NFS | Shared disk storage service. Can be used as an alternative for Gitaly or Object Storage. Required for GitLab Pages | [NFS configuration](../high_availability/nfs.md) | +For detailed insight into how GitLab scales and configures GitLab.com, you can +watch [this 1 hour Q&A](https://www.youtube.com/watch?v=uCU8jdYzpac) +with [John Northrup](https://gitlab.com/northrup), and live questions coming +in from some of our customers. ## Reference architectures -- 1 - 1000 Users: A single-node [Omnibus](https://docs.gitlab.com/omnibus/) setup with frequent backups. Refer to the [Single-node Omnibus installation](#single-node-installation) section below. -- 1000 to 50000+ Users: A [Scaled-out Omnibus installation with multiple servers](#multi-node-installation-scaled-out-for-availability), it can be with or without high-availability components applied. - - To decide the level of Availability please refer to our [Availability](../availability/index.md) page. +GitLab can be set up on a single machine or scaled out to handle large number of users. In this section we'll detail the Reference Architectures that were built and verified by our Quality and Support teams. +Testing was done with our [GitLab Performance Tool](https://gitlab.com/gitlab-org/quality/performance) at specific coded workloads, and the throughputs used for testing were calculated based on sample customer data. -### Single-node installation +We test each endpoint type with the following number of requests per second (RPS) per 1000 users: -This solution is appropriate for many teams that have a single server at their disposal. With automatic backup of the GitLab repositories, configuration, and the database, this can be an optimal solution if you don't have strict availability requirements. - -You can also optionally configure GitLab to use an [external PostgreSQL service](../external_database.md) -or an [external object storage service](../high_availability/object_storage.md) for added -performance and reliability at a relatively low complexity cost. - -References: - -- [Installation Docs](../../install/README.md) -- [Backup/Restore Docs](https://docs.gitlab.com/omnibus/settings/backups.html#backup-and-restore-omnibus-gitlab-configuration) +- API: 20 RPS +- Web: 2 RPS +- Git: 2 RPS -### Multi-node installation (scaled out for availability) +For up to 2,000 users we recommend going with a simple setup. Going above 2,000 users, we recommend scaling GitLab components to multiple machine nodes. +The machine nodes are grouped by component(s). The addition of these nodes adds limited fault tolerance to your GitLab instance. +As long as there is at least one of each component online and capable of handling the instance's usage load, your team's productivity will not be interrupted. +The same is true if you are looking to perform [zero-downtime updates](https://docs.gitlab.com/omnibus/update/#zero-downtime-updates). -This solution is appropriate for teams that are starting to scale out when -scaling up is no longer meeting their needs. In this configuration, additional application nodes will handle frontend traffic, with a load balancer in front to distribute traffic across those nodes. Meanwhile, each application node connects to a shared file server and PostgreSQL and Redis services on the back end. +When scaling GitLab there's a few factors to consider: -The additional application servers adds limited fault tolerance to your GitLab -instance. As long as one application node is online and capable of handling the -instance's usage load, your team's productivity will not be interrupted. Having -multiple application nodes also enables [zero-downtime updates](https://docs.gitlab.com/omnibus/update/#zero-downtime-updates). +- Multiple application nodes to handle frontend traffic. +- A load balancer is added in front to distribute traffic across the application nodes. +- The application nodes connects to a shared file server and PostgreSQL and Redis services on the backend. References: @@ -76,18 +46,6 @@ References: - [Configure packaged PostgreSQL server to listen on TCP/IP](https://docs.gitlab.com/omnibus/settings/database.html#configure-packaged-postgresql-server-to-listen-on-tcpip) - [Setting up a Redis-only server](https://docs.gitlab.com/omnibus/settings/redis.html#setting-up-a-redis-only-server) -In this section we'll detail the Reference Architectures that can support large numbers -of users. These were built, tested and verified by our Quality and Support teams. - -Testing was done with our GitLab Performance Tool at specific coded workloads, and the -throughputs used for testing were calculated based on sample customer data. We -test each endpoint type with the following number of requests per second (RPS) -per 1000 users: - -- API: 20 RPS -- Web: 2 RPS -- Git: 2 RPS - NOTE: **Note:** Note that depending on your workflow the below recommended reference architectures may need to be adapted accordingly. Your workload is influenced by factors such as - but not limited to - how active your users are, @@ -95,7 +53,24 @@ how much automation you use, mirroring, and repo/change size. Additionally the shown memory values are given directly by [GCP machine types](https://cloud.google.com/compute/docs/machine-types). On different cloud vendors a best effort like for like can be used. -#### 2,000 user configuration +### Up to 1,000 users + +From 1 to 1,000 users, a single-node [Omnibus](https://docs.gitlab.com/omnibus/) setup with frequent backups is adequate. +Please refer to the [installation documentation](../../install/README.md) and [backup/restore documentation](https://docs.gitlab.com/omnibus/settings/backups.html#backup-and-restore-omnibus-gitlab-configuration). + +This solution is appropriate for many teams that have a single server at their disposal. With automatic backup of the GitLab repositories, configuration, and the database, this can be an optimal solution if you don't have strict availability requirements. + +You can also optionally configure GitLab to use an [external PostgreSQL service](../external_database.md) or an [external object storage service](../high_availability/object_storage.md) for added performance and reliability at a relatively low complexity cost. + +### Up to 2,000 users + +NOTE: **Note:** The 2,000-user reference architecture documented below is +designed to help your organization achieve a highly-available GitLab deployment. +If you do not have the expertise or need to maintain a highly-available +environment, you can have a simpler and less costly-to-operate environment by +deploying two or more GitLab Rails servers, external load balancing, an NFS +server, a PostgreSQL server and a Redis server. A reference architecture with +this alternative in mind is [being worked on](https://gitlab.com/gitlab-org/quality/performance/-/issues/223). - **Supported users (approximate):** 2,000 - **Test RPS rates:** API: 40 RPS, Web: 4 RPS, Git: 4 RPS @@ -116,7 +91,7 @@ On different cloud vendors a best effort like for like can be used. | External load balancing node[^6] | 1 | 2 vCPU, 1.8GB Memory | n1-highcpu-2 | c5.large | | Internal load balancing node[^6] | 1 | 2 vCPU, 1.8GB Memory | n1-highcpu-2 | c5.large | -#### 5,000 user configuration +### Up to 5,000 users - **Supported users (approximate):** 5,000 - **Test RPS rates:** API: 100 RPS, Web: 10 RPS, Git: 10 RPS @@ -137,7 +112,7 @@ On different cloud vendors a best effort like for like can be used. | External load balancing node[^6] | 1 | 2 vCPU, 1.8GB Memory | n1-highcpu-2 | c5.large | | Internal load balancing node[^6] | 1 | 2 vCPU, 1.8GB Memory | n1-highcpu-2 | c5.large | -#### 10,000 user configuration +### Up to 10,000 users - **Supported users (approximate):** 10,000 - **Test RPS rates:** API: 200 RPS, Web: 20 RPS, Git: 20 RPS @@ -161,7 +136,7 @@ On different cloud vendors a best effort like for like can be used. | External load balancing node[^6] | 1 | 2 vCPU, 1.8GB Memory | n1-highcpu-2 | c5.large | | Internal load balancing node[^6] | 1 | 2 vCPU, 1.8GB Memory | n1-highcpu-2 | c5.large | -#### 25,000 user configuration +### Up to 25,000 users - **Supported users (approximate):** 25,000 - **Test RPS rates:** API: 500 RPS, Web: 50 RPS, Git: 50 RPS @@ -185,7 +160,7 @@ On different cloud vendors a best effort like for like can be used. | External load balancing node[^6] | 1 | 2 vCPU, 1.8GB Memory | n1-highcpu-2 | c5.large | | Internal load balancing node[^6] | 1 | 4 vCPU, 3.6GB Memory | n1-highcpu-4 | c5.xlarge | -#### 50,000 user configuration +### Up to 50,000 users - **Supported users (approximate):** 50,000 - **Test RPS rates:** API: 1000 RPS, Web: 100 RPS, Git: 100 RPS @@ -209,6 +184,43 @@ On different cloud vendors a best effort like for like can be used. | External load balancing node[^6] | 1 | 2 vCPU, 1.8GB Memory | n1-highcpu-2 | c5.large | | Internal load balancing node[^6] | 1 | 8 vCPU, 7.2GB Memory | n1-highcpu-8 | c5.2xlarge | +## Configuring GitLab to scale + +### Components not provided by Omnibus GitLab + +Depending on your system architecture, you may require some components which are +not provided in Omnibus GitLab. If required, these should be configured before +setting up components provided by GitLab. Advice on how to select the right +solution for your organization is provided in the configuration instructions +listed below. + +| Component | Description | Configuration instructions | +|-----------|-------------|----------------------------| +| Load balancer(s)[^6] | Handles load balancing, typically when you have multiple GitLab application services nodes | [Load balancer configuration](../high_availability/load_balancer.md)[^6] | +| Object storage service[^4] | Recommended store for shared data objects | [Cloud Object Storage configuration](../object_storage.md) | +| NFS[^5] [^7] | Shared disk storage service. Can be used as an alternative for Gitaly or Object Storage. Required for GitLab Pages | [NFS configuration](../high_availability/nfs.md) | + +### Components provided by Omnibus GitLab + +The following components are provided by Omnibus GitLab. They are listed in the +order you'll typically configure them if they are required by your +[reference architecture](#reference-architectures) of choice. + +| Component | Description | Configuration instructions | +|-----------|-------------|----------------------------| +| [Consul](../../development/architecture.md#consul)[^3] | Service discovery and health checks/failover | [Consul HA configuration](../high_availability/consul.md) **(PREMIUM ONLY)** | +| [PostgreSQL](../../development/architecture.md#postgresql) | Database | [PostgreSQL configuration](https://docs.gitlab.com/omnibus/settings/database.html) | +| [PgBouncer](../../development/architecture.md#pgbouncer) | Database connection pooler | [PgBouncer configuration](../high_availability/pgbouncer.md#running-pgbouncer-as-part-of-a-non-ha-gitlab-installation) **(PREMIUM ONLY)** | +| Repmgr | PostgreSQL cluster management and failover | [PostgreSQL and Repmgr configuration](../high_availability/database.md) | +| [Redis](../../development/architecture.md#redis)[^3] | Key/value store for fast data lookup and caching | [Redis configuration](../high_availability/redis.md) | +| Redis Sentinel | High availability for Redis | [Redis Sentinel configuration](../high_availability/redis.md) | +| [Gitaly](../../development/architecture.md#gitaly)[^2] [^5] [^7] | Provides access to Git repositories | [Gitaly configuration](../gitaly/index.md#running-gitaly-on-its-own-server) | +| [Sidekiq](../../development/architecture.md#sidekiq) | Asynchronous/background jobs | [Sidekiq configuration](../high_availability/sidekiq.md) | +| [GitLab application services](../../development/architecture.md#unicorn)[^1] | Unicorn/Puma, Workhorse, GitLab Shell - serves front-end requests (UI, API, Git over HTTP/SSH) | [GitLab app scaling configuration](../high_availability/gitlab.md) | +| [Prometheus](../../development/architecture.md#prometheus) and [Grafana](../../development/architecture.md#grafana) | GitLab environment monitoring | [Monitoring node for scaling](../high_availability/monitoring_node.md) | + +## Footnotes + [^1]: In our architectures we run each GitLab Rails node using the Puma webserver and have its number of workers set to 90% of available CPUs along with 4 threads. diff --git a/doc/administration/smime_signing_email.md b/doc/administration/smime_signing_email.md index ed7447c0da9..bab7c5c260d 100644 --- a/doc/administration/smime_signing_email.md +++ b/doc/administration/smime_signing_email.md @@ -18,6 +18,9 @@ files must be provided: intervention. - Only RSA keys are supported. +Optionally, you can also provide a bundle of CA certs (PEM-encoded) to be +included on each signature. This will typically be an intermediate CA. + NOTE: **Note:** Be mindful of the access levels for your private keys and visibility to third parties. @@ -29,6 +32,8 @@ third parties. gitlab_rails['gitlab_email_smime_enabled'] = true gitlab_rails['gitlab_email_smime_key_file'] = '/etc/gitlab/ssl/gitlab_smime.key' gitlab_rails['gitlab_email_smime_cert_file'] = '/etc/gitlab/ssl/gitlab_smime.crt' + # Optional + gitlab_rails['gitlab_email_smime_ca_certs_file'] = '/etc/gitlab/ssl/gitlab_smime_cas.crt' ``` 1. Save the file and [reconfigure GitLab](restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. @@ -49,6 +54,9 @@ NOTE: **Note:** The key needs to be readable by the GitLab system user (`git` by # S/MIME public certificate key in PEM format, will be attached to signed messages # Default is '.gitlab_smime_cert' relative to Rails.root (i.e. root of the GitLab app). cert_file: /etc/pki/smime/certs/gitlab.crt + # S/MIME extra CA public certificates in PEM format, will be attached to signed messages + # Optional + ca_certs_file: /etc/pki/smime/certs/gitlab_cas.crt ``` 1. Save the file and [restart GitLab](restart_gitlab.md#installations-from-source) for the changes to take effect. diff --git a/doc/administration/troubleshooting/elasticsearch.md b/doc/administration/troubleshooting/elasticsearch.md index 77a63112ea3..c049fdf617e 100644 --- a/doc/administration/troubleshooting/elasticsearch.md +++ b/doc/administration/troubleshooting/elasticsearch.md @@ -203,7 +203,7 @@ The best place to start is to determine if the issue is with creating an empty i If it is, check on the Elasticsearch side to determine if the `gitlab-production` (the name for the GitLab index) exists. If it exists, manually delete it on the Elasticsearch side and attempt to recreate it from the -[`create_empty_index`](../../integration/elasticsearch.md#gitlab-elasticsearch-rake-tasks) +[`recreate_index`](../../integration/elasticsearch.md#gitlab-elasticsearch-rake-tasks) Rake task. If you still encounter issues, try creating an index manually on the Elasticsearch diff --git a/doc/administration/troubleshooting/img/AzureAD-scim_attribute_mapping.png b/doc/administration/troubleshooting/img/AzureAD-scim_attribute_mapping.png Binary files differindex 807936423e5..5ad82003eed 100644 --- a/doc/administration/troubleshooting/img/AzureAD-scim_attribute_mapping.png +++ b/doc/administration/troubleshooting/img/AzureAD-scim_attribute_mapping.png diff --git a/doc/administration/troubleshooting/kubernetes_cheat_sheet.md b/doc/administration/troubleshooting/kubernetes_cheat_sheet.md index 30ef3da3a99..2a6a336bbfb 100644 --- a/doc/administration/troubleshooting/kubernetes_cheat_sheet.md +++ b/doc/administration/troubleshooting/kubernetes_cheat_sheet.md @@ -240,10 +240,10 @@ to those documents for details. helm init --service-account tiller ``` -- Copy the file <https://gitlab.com/gitlab-org/charts/gitlab/raw/master/examples/values-minikube-minimum.yaml> +- Copy the [Minikube minimum values YAML file](https://gitlab.com/gitlab-org/charts/gitlab/raw/master/examples/values-minikube-minimum.yaml) to your workstation. -- Find the IP address in the output of `minikube ip` and update the yaml file with +- Find the IP address in the output of `minikube ip` and update the YAML file with this IP address. - Install the GitLab Helm Chart: @@ -254,7 +254,7 @@ to those documents for details. ``` If you want to modify some GitLab settings, you can use the above-mentioned config - as a base and create your own yaml file. + as a base and create your own YAML file. - Monitor the installation progress via `helm status gitlab` and `minikube dashboard`. The installation could take up to 20-30 minutes depending on the amount of resources @@ -263,7 +263,7 @@ to those documents for details. - When all the pods show either a `Running` or `Completed` status, get the GitLab password as described in [Initial login](https://docs.gitlab.com/charts/installation/deployment.html#initial-login), and log in to GitLab via the UI. It will be accessible via `https://gitlab.domain` - where `domain` is the value provided in the yaml file. + where `domain` is the value provided in the YAML file. <!-- ## Troubleshooting diff --git a/doc/administration/uploads.md b/doc/administration/uploads.md index f29deba3d40..294c776980d 100644 --- a/doc/administration/uploads.md +++ b/doc/administration/uploads.md @@ -25,7 +25,7 @@ _The uploads are stored by default in `/var/opt/gitlab/gitlab-rails/uploads`._ gitlab_rails['uploads_base_dir'] = "uploads" ``` -1. Save the file and [reconfigure GitLab][] for the changes to take effect. +1. Save the file and [reconfigure GitLab](restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. **In installations from source:** @@ -41,13 +41,13 @@ _The uploads are stored by default in base_dir: uploads ``` -1. Save the file and [restart GitLab][] for the changes to take effect. +1. Save the file and [restart GitLab](restart_gitlab.md#installations-from-source) for the changes to take effect. ## Using object storage **(CORE ONLY)** > **Notes:** > -> - [Introduced][ee-3867] in [GitLab Premium][eep] 10.5. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/3867) in [GitLab Premium](https://about.gitlab.com/pricing/) 10.5. > - [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/17358) in [GitLab Core](https://about.gitlab.com/pricing/) 10.7. > - Since version 11.1, we support direct_upload to S3. @@ -117,7 +117,7 @@ _The uploads are stored by default in } ``` -1. Save the file and [reconfigure GitLab][] for the changes to take effect. +1. Save the file and [reconfigure GitLab](restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. 1. Migrate any existing local uploads to the object storage using [`gitlab:uploads:migrate` Rake task](raketasks/uploads/migrate.md). **In installations from source:** @@ -140,7 +140,7 @@ _The uploads are stored by default in region: eu-central-1 ``` -1. Save the file and [restart GitLab][] for the changes to take effect. +1. Save the file and [restart GitLab](restart_gitlab.md#installations-from-source) for the changes to take effect. 1. Migrate any existing local uploads to the object storage using [`gitlab:uploads:migrate` Rake task](raketasks/uploads/migrate.md). ### Oracle Cloud S3 connection settings @@ -194,7 +194,7 @@ _The uploads are stored by default in } ``` -1. Save the file and [reconfigure GitLab][] for the changes to take effect. +1. Save the file and [reconfigure GitLab](restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. 1. Migrate any existing local uploads to the object storage using [`gitlab:uploads:migrate` Rake task](raketasks/uploads/migrate.md). --- @@ -225,10 +225,5 @@ _The uploads are stored by default in openstack_tenant: 'TENANT_ID' ``` -1. Save the file and [reconfigure GitLab][] for the changes to take effect. +1. Save the file and [reconfigure GitLab](restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. 1. Migrate any existing local uploads to the object storage using [`gitlab:uploads:migrate` Rake task](raketasks/uploads/migrate.md). - -[reconfigure gitlab]: restart_gitlab.md#omnibus-gitlab-reconfigure "How to reconfigure Omnibus GitLab" -[restart gitlab]: restart_gitlab.md#installations-from-source "How to restart GitLab" -[eep]: https://about.gitlab.com/pricing/ "GitLab Premium" -[ee-3867]: https://gitlab.com/gitlab-org/gitlab/-/merge_requests/3867 |
