diff options
Diffstat (limited to 'doc/administration/operations')
| -rw-r--r-- | doc/administration/operations/extra_sidekiq_processes.md | 39 | ||||
| -rw-r--r-- | doc/administration/operations/extra_sidekiq_routing.md | 4 | ||||
| -rw-r--r-- | doc/administration/operations/puma.md | 150 | ||||
| -rw-r--r-- | doc/administration/operations/sidekiq_memory_killer.md | 4 | ||||
| -rw-r--r-- | doc/administration/operations/ssh_certificates.md | 16 |
5 files changed, 158 insertions, 55 deletions
diff --git a/doc/administration/operations/extra_sidekiq_processes.md b/doc/administration/operations/extra_sidekiq_processes.md index b910a789d29..1f195bcc378 100644 --- a/doc/administration/operations/extra_sidekiq_processes.md +++ b/doc/administration/operations/extra_sidekiq_processes.md @@ -74,8 +74,9 @@ To start multiple processes: just handles the `mailers` queue. When `sidekiq-cluster` is only running on a single node, make sure that at least - one process is running on all queues using `*`. This means a process is - This includes queues that have dedicated processes. + one process is running on all queues using `*`. This ensures a process + automatically picks up jobs in queues created in the future, + including queues that have dedicated processes. If `sidekiq-cluster` is running on more than one node, you can also use [`--negate`](#negate-settings) and list all the queues that are already being @@ -95,13 +96,16 @@ To view the Sidekiq processes in GitLab: ## Negate settings To have the additional Sidekiq processes work on every queue **except** the ones -you list: +you list. In this example, we exclude all import-related jobs from a Sidekiq node: 1. After you follow the steps for [starting extra processes](#start-multiple-processes), edit `/etc/gitlab/gitlab.rb` and add: ```ruby sidekiq['negate'] = true + sidekiq['queue_groups'] = [ + "feature_category=importers" + ] ``` 1. Save the file and reconfigure GitLab for the changes to take effect: @@ -171,7 +175,7 @@ When disabling `sidekiq_cluster`, you must copy your configuration for `sidekiq_cluster` is overridden by the options for `sidekiq` when setting `sidekiq['cluster'] = true`. -When using this feature, the service called `sidekiq` is now +When using this feature, the service called `sidekiq` is now running `sidekiq-cluster`. The [concurrency](#manage-concurrency) and other options configured @@ -180,32 +184,21 @@ for Sidekiq are respected. By default, logs for `sidekiq-cluster` go to `/var/log/gitlab/sidekiq` like regular Sidekiq logs. -## Ignore all GitHub import queues +## Ignore all import queues -When [importing from GitHub](../../user/project/import/github.md), Sidekiq might -use all of its resources to perform those operations. To set up a separate -`sidekiq-cluster` process to ignore all GitHub import-related queues: +When [importing from GitHub](../../user/project/import/github.md) or +other sources, Sidekiq might use all of its resources to perform those +operations. To set up two separate `sidekiq-cluster` processes, where +one only processes imports and the other processes all other queues: 1. Edit `/etc/gitlab/gitlab.rb` and add: ```ruby sidekiq['enable'] = true - sidekiq['negate'] = true + sidekiq['queue_selector'] = true sidekiq['queue_groups'] = [ - "github_import_advance_stage", - "github_importer:github_import_import_diff_note", - "github_importer:github_import_import_issue", - "github_importer:github_import_import_note", - "github_importer:github_import_import_lfs_object", - "github_importer:github_import_import_pull_request", - "github_importer:github_import_refresh_import_jid", - "github_importer:github_import_stage_finish_import", - "github_importer:github_import_stage_import_base_data", - "github_importer:github_import_stage_import_issues_and_diff_notes", - "github_importer:github_import_stage_import_notes", - "github_importer:github_import_stage_import_lfs_objects", - "github_importer:github_import_stage_import_pull_requests", - "github_importer:github_import_stage_import_repository" + "feature_category=importers", + "feature_category!=importers" ] ``` diff --git a/doc/administration/operations/extra_sidekiq_routing.md b/doc/administration/operations/extra_sidekiq_routing.md index 93cf8bd4f43..80540b7ba46 100644 --- a/doc/administration/operations/extra_sidekiq_routing.md +++ b/doc/administration/operations/extra_sidekiq_routing.md @@ -41,7 +41,7 @@ In `/etc/gitlab/gitlab.rb`: ```ruby sidekiq['routing_rules'] = [ # Route all non-CPU-bound workers that are high urgency to `high-urgency` queue - ['resource_boundary!=cpu&urgency=high', 'high-urgency'], + ['resource_boundary!=cpu&urgency=high', 'high-urgency'], # Route all database, gitaly and global search workers that are throttled to `throttled` queue ['feature_category=database,gitaly,global_search&urgency=throttled', 'throttled'], # Route all workers having contact with outside work to a `network-intenstive` queue @@ -99,7 +99,7 @@ based on a subset of worker attributes: - `urgency` - how important it is that this queue's jobs run quickly. Can be `high`, `low`, or `throttled`. For example, the `authorized_projects` queue is used to refresh user permissions, and - is high urgency. + is `high` urgency. - `worker_name` - the worker name. The other attributes are typically more useful as they are more general, but this is available in case a particular worker needs to be selected. diff --git a/doc/administration/operations/puma.md b/doc/administration/operations/puma.md index fffff78b9d6..e8477eaf686 100644 --- a/doc/administration/operations/puma.md +++ b/doc/administration/operations/puma.md @@ -4,35 +4,102 @@ group: Distribution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# Switching to Puma **(FREE SELF)** +# Puma **(FREE SELF)** -As of GitLab 12.9, [Puma](https://github.com/puma/puma) has replaced [Unicorn](https://yhbt.net/unicorn/) -as the default web server. From GitLab 14.0, the following run Puma: +Puma is a simple, fast, multi-threaded, and highly concurrent HTTP 1.1 server for +Ruby applications. It's the default GitLab web server since GitLab 13.0 +and has replaced Unicorn. From GitLab 14.0, Unicorn is no longer supported. -- All-in-one package-based installations. -- Helm chart-based installations. +NOTE: +Starting with GitLab 13.0, Puma is the default web server and Unicorn has been disabled. +In GitLab 14.0, Unicorn was removed from the Linux package and only Puma is available. -## Why switch to Puma? +## Configure Puma -Puma has a multi-thread architecture which uses less memory than a multi-process -application server like Unicorn. On GitLab.com, we saw a 40% reduction in memory -consumption. +To configure Puma: -Most Rails applications requests normally include a proportion of I/O wait time. -During I/O wait time MRI Ruby will release the GVL (Global VM Lock) to other threads. -Multi-threaded Puma can therefore still serve more requests than a single process. +1. Determine suitable Puma worker and thread [settings](../../install/requirements.md#puma-settings). +1. If you're switching from Unicorn, [convert any custom settings to Puma](#convert-unicorn-settings-to-puma). +1. For multi-node deployments, configure the load balancer to use the + [readiness check](../load_balancer.md#readiness-check). +1. Reconfigure GitLab so the above changes take effect: + + ```shell + sudo gitlab-ctl reconfigure + ``` + +For Helm-based deployments, see the +[`webservice` chart documentation](https://docs.gitlab.com/charts/charts/gitlab/webservice/index.html). + +For more details about the Puma configuration, see the +[Puma documentation](https://github.com/puma/puma#configuration). + +## Puma Worker Killer + +By default: + +- The [Puma Worker Killer](https://github.com/schneems/puma_worker_killer) restarts a worker if it + exceeds a [memory limit](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/cluster/puma_worker_killer_initializer.rb). +- Rolling restarts of Puma workers are performed every 12 hours. + +To change the memory limit setting: + +1. Edit `/etc/gitlab/gitlab.rb`: + + ```ruby + puma['per_worker_max_memory_mb'] = 1024 + ``` + +1. Reconfigure GitLab for the changes to take effect: + + ```shell + sudo gitlab-ctl reconfigure + ``` + +## Worker timeout + +A [timeout of 60 seconds](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/initializers/rack_timeout.rb) +is used when Puma is enabled. + +NOTE: +Unlike Unicorn, the `puma['worker_timeout']` setting does not set the maximum request duration. -## Configuring Puma to replace Unicorn +To change the worker timeout: -Beginning with GitLab 13.0, Puma is the default application server. We removed support for -Unicorn in GitLab 14.0. +1. Edit `/etc/gitlab/gitlab.rb`: -When switching to Puma, Unicorn server configuration -will _not_ carry over automatically, due to differences between the two application servers. For Omnibus-based -deployments, see [Configuring Puma Settings](https://docs.gitlab.com/omnibus/settings/puma.html#configuring-puma-settings). -For Helm based deployments, see the [`webservice` chart documentation](https://docs.gitlab.com/charts/charts/gitlab/webservice/index.html). + ```ruby + gitlab_rails['env'] = { + 'GITLAB_RAILS_RACK_TIMEOUT' => 600 + } + ``` -Additionally we strongly recommend that multi-node deployments [configure their load balancers to use the readiness check](../load_balancer.md#readiness-check) due to a difference between Unicorn and Puma in how they handle connections during a restart of the service. +1. Reconfigure GitLab for the changes to take effect: + + ```shell + sudo gitlab-ctl reconfigure + ``` + +## Memory-constrained environments + +In a memory-constrained environment with less than 4GB of RAM available, consider disabling Puma +[Clustered mode](https://github.com/puma/puma#clustered-mode). + +Configuring Puma by setting the amount of `workers` to `0` could reduce memory usage by hundreds of MB. +For details on Puma worker and thread settings, see the [Puma requirements](../../install/requirements.md#puma-settings). + +Unlike in a Clustered mode, which is set up by default, only a single Puma process would serve the application. + +The downside of running Puma with such configuration is the reduced throughput, which could be +considered as a fair tradeoff in a memory-constraint environment. + +When running Puma in Single mode, some features are not supported: + +- Phased restart do not work: [issue](https://gitlab.com/gitlab-org/gitlab/-/issues/300665) +- [Phased restart](https://gitlab.com/gitlab-org/gitlab/-/issues/300665) +- [Puma Worker Killer](https://gitlab.com/gitlab-org/gitlab/-/issues/300664) + +To learn more, visit [epic 5303](https://gitlab.com/groups/gitlab-org/-/epics/5303). ## Performance caveat when using Puma with Rugged @@ -66,3 +133,46 @@ optimal configuration: Rugged, single-threaded Puma works the same as Unicorn. - To force Rugged to be used with multi-threaded Puma, you can use [feature flags](../../development/gitaly.md#legacy-rugged-code). + +## Convert Unicorn settings to Puma + +NOTE: +Starting with GitLab 13.0, Puma is the default web server and Unicorn has been +disabled by default. In GitLab 14.0, Unicorn was removed from the Linux package +and only Puma is available. + +Puma has a multi-thread architecture which uses less memory than a multi-process +application server like Unicorn. On GitLab.com, we saw a 40% reduction in memory +consumption. Most Rails applications requests normally include a proportion of I/O wait time. + +During I/O wait time MRI Ruby releases the GVL (Global VM Lock) to other threads. +Multi-threaded Puma can therefore still serve more requests than a single process. + +When switching to Puma, any Unicorn server configuration will _not_ carry over +automatically, due to differences between the two application servers. + +The table below summarizes which Unicorn configuration keys correspond to those +in Puma when using the Linux package, and which ones have no corresponding counterpart. + +| Unicorn | Puma | +| ------------------------------------ | ---------------------------------- | +| `unicorn['enable']` | `puma['enable']` | +| `unicorn['worker_timeout']` | `puma['worker_timeout']` | +| `unicorn['worker_processes']` | `puma['worker_processes']` | +| n/a | `puma['ha']` | +| n/a | `puma['min_threads']` | +| n/a | `puma['max_threads']` | +| `unicorn['listen']` | `puma['listen']` | +| `unicorn['port']` | `puma['port']` | +| `unicorn['socket']` | `puma['socket']` | +| `unicorn['pidfile']` | `puma['pidfile']` | +| `unicorn['tcp_nopush']` | n/a | +| `unicorn['backlog_socket']` | n/a | +| `unicorn['somaxconn']` | `puma['somaxconn']` | +| n/a | `puma['state_path']` | +| `unicorn['log_directory']` | `puma['log_directory']` | +| `unicorn['worker_memory_limit_min']` | n/a | +| `unicorn['worker_memory_limit_max']` | `puma['per_worker_max_memory_mb']` | +| `unicorn['exporter_enabled']` | `puma['exporter_enabled']` | +| `unicorn['exporter_address']` | `puma['exporter_address']` | +| `unicorn['exporter_port']` | `puma['exporter_port']` | diff --git a/doc/administration/operations/sidekiq_memory_killer.md b/doc/administration/operations/sidekiq_memory_killer.md index d3019e2c580..598baa4fcc7 100644 --- a/doc/administration/operations/sidekiq_memory_killer.md +++ b/doc/administration/operations/sidekiq_memory_killer.md @@ -25,7 +25,7 @@ minute of delay for incoming background jobs. Some background jobs rely on long-running external processes. To ensure these are cleanly terminated when Sidekiq is restarted, each Sidekiq process should be -run as a process group leader (e.g., using `chpst -P`). If using Omnibus or the +run as a process group leader (for example, using `chpst -P`). If using Omnibus or the `bin/background_jobs` script with `runit` installed, this is handled for you. ## Configuring the MemoryKiller @@ -80,4 +80,4 @@ The MemoryKiller is controlled using environment variables. If the process hard shutdown/restart is not performed by Sidekiq, the Sidekiq process is forcefully terminated after `Sidekiq.options[:timeout] + 2` seconds. An external supervision mechanism - (e.g. runit) must restart Sidekiq afterwards. + (for example, runit) must restart Sidekiq afterwards. diff --git a/doc/administration/operations/ssh_certificates.md b/doc/administration/operations/ssh_certificates.md index 508d284b0bd..374eebeb773 100644 --- a/doc/administration/operations/ssh_certificates.md +++ b/doc/administration/operations/ssh_certificates.md @@ -41,11 +41,11 @@ uploading user SSH keys to GitLab entirely. How to fully set up SSH certificates is outside the scope of this document. See [OpenSSH's `PROTOCOL.certkeys`](https://cvsweb.openbsd.org/cgi-bin/cvsweb/src/usr.bin/ssh/PROTOCOL.certkeys?annotate=HEAD) -for how it works, and e.g. [RedHat's documentation about +for how it works, for example [RedHat's documentation about it](https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/6/html/deployment_guide/sec-using_openssh_certificate_authentication). We assume that you already have SSH certificates set up, and have -added the `TrustedUserCAKeys` of your CA to your `sshd_config`, e.g.: +added the `TrustedUserCAKeys` of your CA to your `sshd_config`, for example: ```plaintext TrustedUserCAKeys /etc/security/mycompany_user_ca.pub @@ -58,7 +58,7 @@ used for GitLab consider putting this in the `Match User git` section (described below). The SSH certificates being issued by that CA **MUST** have a "key ID" -corresponding to that user's username on GitLab, e.g. (some output +corresponding to that user's username on GitLab, for example (some output omitted for brevity): ```shell @@ -77,7 +77,7 @@ $ ssh-add -L | grep cert | ssh-keygen -L -f - [...] ``` -Technically that's not strictly true, e.g. it could be +Technically that's not strictly true, for example, it could be `prod-aearnfjord` if it's a SSH certificate you'd normally log in to servers as the `prod-aearnfjord` user, but then you must specify your own `AuthorizedPrincipalsCommand` to do that mapping instead of using @@ -107,13 +107,13 @@ command="/opt/gitlab/embedded/service/gitlab-shell/bin/gitlab-shell username-{KE ``` Where `{KEY_ID}` is the `%i` argument passed to the script -(e.g. `aeanfjord`), and `{PRINCIPAL}` is the principal passed to it -(e.g. `sshUsers`). +(for example, `aeanfjord`), and `{PRINCIPAL}` is the principal passed to it +(for example, `sshUsers`). You need to customize the `sshUsers` part of that. It should be some principal that's guaranteed to be part of the key for all users who can log in to GitLab, or you must provide a list of principals, -one of which is present for the user, e.g.: +one of which is present for the user, for example: ```plaintext [...] @@ -131,7 +131,7 @@ principal is some "group" that's allowed to log into that server. However with GitLab it's only used to appease OpenSSH's requirement for it, we effectively only care about the "key ID" being correct. Once that's extracted GitLab enforces its own ACLs for -that user (e.g. what projects the user can access). +that user (for example, what projects the user can access). So it's OK to e.g. be overly generous in what you accept, since if the user e.g. has no access to GitLab at all it just errors out with a |