From 311b0269b4eb9839fa63f80c8d7a58f32b8138a0 Mon Sep 17 00:00:00 2001 From: GitLab Bot Date: Thu, 18 Nov 2021 13:16:36 +0000 Subject: Add latest changes from gitlab-org/gitlab@14-5-stable-ee --- .../operations/extra_sidekiq_processes.md | 39 +++++-- .../operations/fast_ssh_key_lookup.md | 130 ++++++--------------- .../operations/moving_repositories.md | 4 +- doc/administration/operations/puma.md | 5 + doc/administration/operations/ssh_certificates.md | 6 +- 5 files changed, 75 insertions(+), 109 deletions(-) (limited to 'doc/administration/operations') diff --git a/doc/administration/operations/extra_sidekiq_processes.md b/doc/administration/operations/extra_sidekiq_processes.md index 31cbd6c457b..02cb7ad0bca 100644 --- a/doc/administration/operations/extra_sidekiq_processes.md +++ b/doc/administration/operations/extra_sidekiq_processes.md @@ -30,6 +30,11 @@ can be started. > - [Sidekiq cluster moved](https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/181) to GitLab Free in 12.10. > - [Sidekiq cluster became default](https://gitlab.com/gitlab-org/omnibus-gitlab/-/merge_requests/4140) in GitLab 13.0. +When starting multiple processes, the number of processes should +equal (and **not** exceed) the number of CPU cores you want to +dedicate to Sidekiq. Each Sidekiq process can use only 1 CPU +core, subject to the available workload and concurrency settings. + To start multiple processes: 1. Using the `sidekiq['queue_groups']` array setting, specify how many processes to @@ -166,23 +171,41 @@ one only processes imports and the other processes all other queues: ## Number of threads -Each process defined under `sidekiq` starts with a +By default each process defined under `sidekiq` starts with a number of threads that equals the number of queues, plus one spare thread. For example, a process that handles the `process_commit` and `post_receive` queues uses three threads in total. -## Manage concurrency +These thread run inside a single Ruby process, and each process +can only use a single CPU core. The usefulness of threading depends +on the work having some external dependencies to wait on, like database queries or +HTTP requests. Most Sidekiq deployments benefit from this threading, and when +running fewer queues in a process, increasing the thread count might be +even more desirable to make the most effective use of CPU resources. + +### Manage thread counts explicitly + +The correct maximum thread count (also called concurrency) depends on the workload. +Typical values range from `1` for highly CPU-bound tasks to `15` or higher for mixed +low-priority work. A reasonable starting range is `15` to `25` for a non-specialized +deployment. + +You can find example values used by GitLab.com by searching for `concurrency:` in +[the Helm charts](https://gitlab.com/gitlab-com/gl-infra/k8s-workloads/gitlab-com/-/blob/master/releases/gitlab/values/gprd.yaml.gotmpl). +The values vary according to the work each specific deployment of Sidekiq does. +Any other specialized deployments with processes dedicated to specific queues should +have the concurrency tuned according to: +have the concurrency tuned according to: -When setting the maximum concurrency, keep in mind this normally should -not exceed the number of CPU cores available. The values in the examples -below are arbitrary and not particular recommendations. +- The CPU usage of each type of process. +- The throughput achieved. Each thread requires a Redis connection, so adding threads may increase Redis latency and potentially cause client timeouts. See the [Sidekiq documentation about Redis](https://github.com/mperham/sidekiq/wiki/Using-Redis) for more details. -### When running Sidekiq cluster (default) +#### When running Sidekiq cluster (default) Running Sidekiq cluster is the default in GitLab 13.0 and later. @@ -203,7 +226,7 @@ Running Sidekiq cluster is the default in GitLab 13.0 and later. the other. Setting `min_concurrency` to `0` disables the limit. For each queue group, let `N` be one more than the number of queues. The -concurrency factor are set to: +concurrency is set to: 1. `N`, if it's between `min_concurrency` and `max_concurrency`. 1. `max_concurrency`, if `N` exceeds this value. @@ -215,7 +238,7 @@ regardless of the number of queues. When `min_concurrency` is greater than `max_concurrency`, it is treated as being equal to `max_concurrency`. -### When running a single Sidekiq process +#### When running a single Sidekiq process Running a single Sidekiq process is the default in GitLab 12.10 and earlier. diff --git a/doc/administration/operations/fast_ssh_key_lookup.md b/doc/administration/operations/fast_ssh_key_lookup.md index 8aa5af4c2bf..c99f94589d7 100644 --- a/doc/administration/operations/fast_ssh_key_lookup.md +++ b/doc/administration/operations/fast_ssh_key_lookup.md @@ -26,11 +26,8 @@ indexed lookup in the GitLab database. This page describes how to enable the fas lookup of authorized SSH keys. WARNING: -OpenSSH version 6.9+ is required because -`AuthorizedKeysCommand` must be able to accept a fingerprint. These -instructions break installations that use older versions of OpenSSH, such as -those included with CentOS 6 as of September 2017. If you want to use this -feature for CentOS 6, follow [the instructions on how to build and install a custom OpenSSH package](#compiling-a-custom-version-of-openssh-for-centos-6) before continuing. +OpenSSH version 6.9+ is required because `AuthorizedKeysCommand` must be +able to accept a fingerprint. Check the version of OpenSSH on your server. ## Fast lookup is required for Geo **(PREMIUM)** @@ -132,100 +129,43 @@ This is a brief overview. Please refer to the above instructions for more contex 1. Remove the `AuthorizedKeysCommand` lines from `/etc/ssh/sshd_config` or from `/assets/sshd_config` if you are using Omnibus Docker. 1. Reload `sshd`: `sudo service sshd reload`. -## Compiling a custom version of OpenSSH for CentOS 6 +## Use `gitlab-sshd` instead of OpenSSH -Building a custom version of OpenSSH is not necessary for Ubuntu 16.04 users, -since Ubuntu 16.04 ships with OpenSSH 7.2. +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/299109) in GitLab 14.5. -It is also unnecessary for CentOS 7.4 users, as that version ships with -OpenSSH 7.4. If you are using CentOS 7.0 - 7.3, we strongly recommend that you -upgrade to CentOS 7.4 instead of following this procedure. This should be as -simple as running `yum update`. - -CentOS 6 users must build their own OpenSSH package to enable SSH lookups via -the database. The following instructions can be used to build OpenSSH 7.5: - -1. First, download the package and install the required packages: - - ```shell - sudo su - - cd /tmp - curl --remote-name "https://cdn.openbsd.org/pub/OpenBSD/OpenSSH/portable/openssh-7.5p1.tar.gz" - tar xzvf openssh-7.5p1.tar.gz - yum install rpm-build gcc make wget openssl-devel krb5-devel pam-devel libX11-devel xmkmf libXt-devel - ``` - -1. Prepare the build by copying files to the right place: - - ```shell - mkdir -p /root/rpmbuild/{SOURCES,SPECS} - cp ./openssh-7.5p1/contrib/redhat/openssh.spec /root/rpmbuild/SPECS/ - cp openssh-7.5p1.tar.gz /root/rpmbuild/SOURCES/ - cd /root/rpmbuild/SPECS - ``` - -1. Next, set the spec settings properly: - - ```shell - sed -i -e "s/%define no_gnome_askpass 0/%define no_gnome_askpass 1/g" openssh.spec - sed -i -e "s/%define no_x11_askpass 0/%define no_x11_askpass 1/g" openssh.spec - sed -i -e "s/BuildPreReq/BuildRequires/g" openssh.spec - ``` - -1. Build the RPMs: - - ```shell - rpmbuild -bb openssh.spec - ``` - -1. Ensure the RPMs were built: - - ```shell - ls -al /root/rpmbuild/RPMS/x86_64/ - ``` - - You should see something as the following: - - ```plaintext - total 1324 - drwxr-xr-x. 2 root root 4096 Jun 20 19:37 . - drwxr-xr-x. 3 root root 19 Jun 20 19:37 .. - -rw-r--r--. 1 root root 470828 Jun 20 19:37 openssh-7.5p1-1.x86_64.rpm - -rw-r--r--. 1 root root 490716 Jun 20 19:37 openssh-clients-7.5p1-1.x86_64.rpm - -rw-r--r--. 1 root root 17020 Jun 20 19:37 openssh-debuginfo-7.5p1-1.x86_64.rpm - -rw-r--r--. 1 root root 367516 Jun 20 19:37 openssh-server-7.5p1-1.x86_64.rpm - ``` - -1. Install the packages. OpenSSH packages replace `/etc/pam.d/sshd` - with their own versions, which may prevent users from logging in, so be sure - that the file is backed up and restored after installation: - - ```shell - timestamp=$(date +%s) - cp /etc/pam.d/sshd pam-ssh-conf-$timestamp - rpm -Uvh /root/rpmbuild/RPMS/x86_64/*.rpm - yes | cp pam-ssh-conf-$timestamp /etc/pam.d/sshd - ``` - -1. Verify the installed version. In another window, attempt to sign in to the - server: - - ```shell - ssh -v +WARNING: +`gitlab-sshd` is in [**Alpha**](https://about.gitlab.com/handbook/product/gitlab-the-product/#alpha-beta-ga). +It is not ready for production use. + +`gitlab-sshd` is [a standalone SSH server](https://gitlab.com/gitlab-org/gitlab-shell/-/tree/main/internal/sshd) + written in Go. It is provided as a part of `gitlab-shell` package. It has a lower memory + use as a OpenSSH alternative and supports + [group access restriction by IP address](../../user/group/index.md) for applications + running behind the proxy. + +If you are considering switching from OpenSSH to `gitlab-sshd`, consider these concerns: + +- The `gitlab-sshd` component is only available for + [Cloud Native Helm Charts](https://docs.gitlab.com/charts/) deployments. +- `gitlab-sshd` supports the PROXY protocol. It can run behind proxy servers that rely + on it, such as HAProxy. +- `gitlab-sshd` does not share a SSH port with the system administrator's OpenSSH, + and requires a bind to port 22. +- `gitlab-sshd` **does not** support SSH certificates. + +To switch from OpenSSH to `gitlab-sshd`: + +1. Set the `gitlab-shell` charts `sshDaemon` option to + [`gitlab-sshd`](https://docs.gitlab.com/charts/charts/gitlab/gitlab-shell/index.html#installation-command-line-options). + For example: + + ```yaml + gitlab: + gitlab-shell: + sshDaemon: gitlab-sshd ``` - You should see a line that reads: "debug1: Remote protocol version 2.0, remote software version OpenSSH_7.5" - - If not, you may need to restart `sshd` (for example, `systemctl restart sshd.service`). - -1. *IMPORTANT!* Open a new SSH session to your server before exiting to make - sure everything is working! If you need to downgrade, simple install the - older package: - - ```shell - # Only run this if you run into a problem logging in - yum downgrade openssh-server openssh openssh-clients - ``` +1. Perform a Helm upgrade. ## SELinux support and limitations diff --git a/doc/administration/operations/moving_repositories.md b/doc/administration/operations/moving_repositories.md index 8aeaadc17e9..9cf7ac18c81 100644 --- a/doc/administration/operations/moving_repositories.md +++ b/doc/administration/operations/moving_repositories.md @@ -49,8 +49,8 @@ until the move has completed. To move repositories: -1. Ensure all storages are accessible to the GitLab instance. In this example, these are - `` and ``. +1. Ensure all [local and cluster storages](../gitaly/configure_gitaly.md#mixed-configuration) are accessible to the GitLab instance. In + this example, these are `` and ``. 1. [Configure repository storage weights](../repository_storage_paths.md#configure-where-new-repositories-are-stored) so that the new storages receives all new projects. This stops new projects from being created on existing storages while the migration is in progress. diff --git a/doc/administration/operations/puma.md b/doc/administration/operations/puma.md index 775761d655d..f1f02b606f5 100644 --- a/doc/administration/operations/puma.md +++ b/doc/administration/operations/puma.md @@ -224,3 +224,8 @@ in Puma when using the Linux package, and which ones have no corresponding count | `unicorn['exporter_enabled']` | `puma['exporter_enabled']` | | `unicorn['exporter_address']` | `puma['exporter_address']` | | `unicorn['exporter_port']` | `puma['exporter_port']` | + +## Puma exporter + +You can use the Puma exporter to measure various Puma metrics. For more information, see +[Puma exporter](../monitoring/prometheus/puma_exporter.md). diff --git a/doc/administration/operations/ssh_certificates.md b/doc/administration/operations/ssh_certificates.md index 77dc4eb180b..4fcc5d7c916 100644 --- a/doc/administration/operations/ssh_certificates.md +++ b/doc/administration/operations/ssh_certificates.md @@ -19,10 +19,8 @@ In such setups some external automated process is needed to constantly upload the new keys to GitLab. WARNING: -OpenSSH version 6.9+ is required because that version -introduced the `AuthorizedPrincipalsCommand` configuration option. If -using CentOS 6, you can [follow these instructions](fast_ssh_key_lookup.md#compiling-a-custom-version-of-openssh-for-centos-6) -to compile an up-to-date version. +OpenSSH version 6.9+ is required because `AuthorizedKeysCommand` must be +able to accept a fingerprint. Check the version of OpenSSH on your server. ## Why use OpenSSH certificates? -- cgit v1.2.1