diff options
Diffstat (limited to 'doc/administration/operations')
8 files changed, 126 insertions, 60 deletions
diff --git a/doc/administration/operations/cleaning_up_redis_sessions.md b/doc/administration/operations/cleaning_up_redis_sessions.md index a94f88439f2..8f90231a713 100644 --- a/doc/administration/operations/cleaning_up_redis_sessions.md +++ b/doc/administration/operations/cleaning_up_redis_sessions.md @@ -34,7 +34,7 @@ rcli() { # This example works for Omnibus installations of GitLab 7.3 or newer. For an # installation from source you will have to change the socket path and the # path to redis-cli. - sudo /opt/gitlab/embedded/bin/redis-cli -s /var/opt/gitlab/redis/redis.shared_state.socket "$@" + sudo /opt/gitlab/embedded/bin/redis-cli -s /var/opt/gitlab/redis/redis.socket "$@" } # test the new shell function; the response should be PONG diff --git a/doc/administration/operations/extra_sidekiq_processes.md b/doc/administration/operations/extra_sidekiq_processes.md index 76dc9bf5510..5de79882083 100644 --- a/doc/administration/operations/extra_sidekiq_processes.md +++ b/doc/administration/operations/extra_sidekiq_processes.md @@ -1,6 +1,6 @@ --- -stage: none -group: unassigned +stage: Enablement +group: Distribution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers --- @@ -38,7 +38,7 @@ To start multiple processes: process, and values in each item determine the queues it works on. For example, the following setting creates three Sidekiq processes, one to run on - `elastic_indexer`, one to run on `mailers`, and one process running all on queues: + `elastic_indexer`, one to run on `mailers`, and one process running on all queues: ```ruby sidekiq['queue_groups'] = [ @@ -110,34 +110,29 @@ you list: sudo gitlab-ctl reconfigure ``` -## Queue selector (experimental) +## Queue selector -> [Introduced](https://gitlab.com/gitlab-com/gl-infra/scalability/-/issues/45) in [GitLab Starter](https://about.gitlab.com/pricing/) 12.8. +> - [Introduced](https://gitlab.com/gitlab-com/gl-infra/scalability/-/issues/45) in [GitLab Starter](https://about.gitlab.com/pricing/) 12.8. +> - [Sidekiq cluster including queue selector moved](https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/181) to GitLab [Core](https://about.gitlab.com/pricing/#self-managed) in GitLab 12.10. +> - [Marked as supported](https://gitlab.com/gitlab-com/gl-infra/scalability/-/issues/147) in GitLab [Core](https://about.gitlab.com/pricing/#self-managed) in GitLab 13.6. Renamed from `experimental_queue_selector` to `queue_selector`. -CAUTION: **Caution:** -As this is marked as **experimental**, it is subject to change at any -time, including **breaking backwards compatibility**. This is so that we -can react to changes we need for our GitLab.com deployment. We have a -tracking issue open to [remove the experimental -designation](https://gitlab.com/gitlab-com/gl-infra/scalability/-/issues/147) -from this feature; please comment there if you are interested in using -this in your own deployment. - -In addition to selecting queues by name, as above, the -`experimental_queue_selector` option allows queue groups to be selected -in a more general way using the following components: +In addition to selecting queues by name, as above, the `queue_selector` +option allows queue groups to be selected in a more general way using +the following components: - Attributes that can be selected. - Operators used to construct a query. +When `queue_selector` is set, all `queue_groups` must be in the queue +selector syntax. + ### Available attributes - [Introduced](https://gitlab.com/gitlab-com/gl-infra/scalability/-/issues/261) in GitLab 13.1, `tags`. From the [list of all available attributes](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/workers/all_queues.yml), -`experimental_queue_selector` allows selecting of queues by the -following attributes: +`queue_selector` allows selecting of queues by the following attributes: - `feature_category` - the [GitLab feature category](https://about.gitlab.com/direction/maturity/#category-maturity) the @@ -169,8 +164,8 @@ neither of those tags. ### Available operators -`experimental_queue_selector` supports the following operators, listed -from highest to lowest precedence: +`queue_selector` supports the following operators, listed from highest +to lowest precedence: - `|` - the logical OR operator. For example, `query_a|query_b` (where `query_a` and `query_b` are queries made up of the other operators here) will include @@ -201,7 +196,7 @@ In `/etc/gitlab/gitlab.rb`: ```ruby sidekiq['enable'] = true -sidekiq['experimental_queue_selector'] = true +sidekiq['queue_selector'] = true sidekiq['queue_groups'] = [ # Run all non-CPU-bound queues that are high urgency 'resource_boundary!=cpu&urgency=high', @@ -230,7 +225,7 @@ All of the aforementioned configuration options for `sidekiq` are available. By default, they will be configured as follows: ```ruby -sidekiq['experimental_queue_selector'] = false +sidekiq['queue_selector'] = false sidekiq['interval'] = nil sidekiq['max_concurrency'] = 50 sidekiq['min_concurrency'] = nil @@ -327,9 +322,9 @@ Running Sidekiq cluster is the default in GitLab 13.0 and later. ``` `min_concurrency` and `max_concurrency` are independent; one can be set without -the other. Setting `min_concurrency` to 0 will disable the limit. +the other. Setting `min_concurrency` to `0` will disable the limit. -For each queue group, let N be one more than the number of queues. The +For each queue group, let `N` be one more than the number of queues. The concurrency factor will be set to: 1. `N`, if it's between `min_concurrency` and `max_concurrency`. diff --git a/doc/administration/operations/fast_ssh_key_lookup.md b/doc/administration/operations/fast_ssh_key_lookup.md index 88ef69b29f2..c8830a45fb2 100644 --- a/doc/administration/operations/fast_ssh_key_lookup.md +++ b/doc/administration/operations/fast_ssh_key_lookup.md @@ -196,7 +196,8 @@ the database. The following instructions can be used to build OpenSSH 7.5: yes | cp pam-ssh-conf-$timestamp /etc/pam.d/sshd ``` -1. Verify the installed version. In another window, attempt to login to the server: +1. Verify the installed version. In another window, attempt to sign in to the + server: ```shell ssh -v <your-centos-machine> @@ -204,7 +205,7 @@ the database. The following instructions can be used to build OpenSSH 7.5: You should see a line that reads: "debug1: Remote protocol version 2.0, remote software version OpenSSH_7.5" - If not, you may need to restart `sshd` (e.g. `systemctl restart sshd.service`). + If not, you may need to restart `sshd` (for example, `systemctl restart sshd.service`). 1. *IMPORTANT!* Open a new SSH session to your server before exiting to make sure everything is working! If you need to downgrade, simple install the diff --git a/doc/administration/operations/index.md b/doc/administration/operations/index.md index aaeb8c723d0..864bbb3233e 100644 --- a/doc/administration/operations/index.md +++ b/doc/administration/operations/index.md @@ -13,6 +13,9 @@ Keep your GitLab instance up and running smoothly. you have been running a large GitLab server (thousands of users) since before GitLab 7.3 we recommend cleaning up stale sessions to compact the Redis database after you upgrade to GitLab 7.3. +- [Rake tasks](../../raketasks/README.md): Tasks for common administration and operational processes such as + [cleaning up unneeded items from GitLab instance](../../raketasks/cleanup.md), integrity checks, + and more. - [Moving repositories](moving_repositories.md): Moving all repositories managed by GitLab to another file system or another server. - [Sidekiq MemoryKiller](sidekiq_memory_killer.md): Configure Sidekiq MemoryKiller @@ -28,6 +31,8 @@ Keep your GitLab instance up and running smoothly. performance can have a big impact on GitLab performance, especially for actions that read or write Git repositories. This information will help benchmark filesystem performance against known good and bad real-world systems. +- [The Rails Console](rails_console.md): Provides a way to interact with your GitLab instance from the command line. + Used for troubleshooting a problem or retrieving some data that can only be done through direct access to GitLab. - [ChatOps Scripts](https://gitlab.com/gitlab-com/chatops): The GitLab.com Infrastructure team uses this repository to house common ChatOps scripts they use to troubleshoot and maintain the production instance of GitLab.com. These scripts are likely useful to administrators of GitLab instances of all sizes. diff --git a/doc/administration/operations/moving_repositories.md b/doc/administration/operations/moving_repositories.md index 94eea1e25b8..b311bee1a5b 100644 --- a/doc/administration/operations/moving_repositories.md +++ b/doc/administration/operations/moving_repositories.md @@ -1,26 +1,69 @@ --- -stage: none -group: unassigned +stage: Create +group: Gitaly info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +type: reference --- -# Moving repositories managed by GitLab +# Moving repositories managed by GitLab **(CORE ONLY)** Sometimes you need to move all repositories managed by GitLab to -another filesystem or another server. In this document we will look +another file system or another server. + +## Moving data within a GitLab instance + +The GitLab API is the recommended way to move Git repositories: + +- Between servers. +- Between different storage. +- From single-node Gitaly to Gitaly Cluster. + +For more information, see: + +- [Configuring additional storage for Gitaly](../gitaly/index.md#network-architecture). Within this + example, additional storage called `storage1` and `storage2` is configured. +- [The API documentation](../../api/project_repository_storage_moves.md) details the endpoints for + querying and scheduling repository moves. +- [Migrate existing repositories to Gitaly Cluster](../gitaly/praefect.md#migrate-existing-repositories-to-gitaly-cluster). + +### Limitations + +Read more in the [API documentation](../../api/project_repository_storage_moves.md#limitations). + +## Migrating to another GitLab instance + +[Using the API](#moving-data-within-a-gitlab-instance) isn't an option if you are migrating to a new +GitLab environment, for example: + +- From a single-node GitLab to a scaled-out architecture. +- From a GitLab instance in your private datacenter to a cloud provider. + +The rest of the document looks at some of the ways you can copy all your repositories from `/var/opt/gitlab/git-data/repositories` to `/mnt/gitlab/repositories`. -We will look at three scenarios: the target directory is empty, the -target directory contains an outdated copy of the repositories, and -how to deal with thousands of repositories. +We look at three scenarios: + +- The target directory is empty. +- The target directory contains an outdated copy of the repositories. +- How to deal with thousands of repositories. + +DANGER: **Warning:** +Each of the approaches we list can or does overwrite data in the target directory +`/mnt/gitlab/repositories`. Do not mix up the source and the target. + +### Recommended approach in all cases + +GitLab's [backup and restore capability](../../raketasks/backup_restore.md) should be used. Git +repositories are accessed, managed, and stored on GitLab servers by Gitaly as a database. Data loss +can result from directly accessing and copying Gitaly's files using tools like `rsync`. -DANGER: **Danger:** -Each of the approaches we list can/will overwrite data in the -target directory `/mnt/gitlab/repositories`. Do not mix up the -source and the target. +- From GitLab 13.3, backup performance can be improved by + [processing multiple repositories concurrently](../../raketasks/backup_restore.md#back-up-git-repositories-concurrently). +- Backups can be created of just the repositories using the + [skip feature](../../raketasks/backup_restore.md#excluding-specific-directories-from-the-backup). -## Target directory is empty: use a `tar` pipe +### Target directory is empty: use a `tar` pipe If the target directory `/mnt/gitlab/repositories` is empty the simplest thing to do is to use a `tar` pipe. This method has low @@ -35,7 +78,7 @@ sudo -u git sh -c 'tar -C /var/opt/gitlab/git-data/repositories -cf - -- . |\ If you want to see progress, replace `-xf` with `-xvf`. -### `tar` pipe to another server +#### `tar` pipe to another server You can also use a `tar` pipe to copy data to another server. If your `git` user has SSH access to the new server as `git@newserver`, you @@ -47,15 +90,19 @@ sudo -u git sh -c 'tar -C /var/opt/gitlab/git-data/repositories -cf - -- . |\ ``` If you want to compress the data before it goes over the network -(which will cost you CPU cycles) you can replace `ssh` with `ssh -C`. +(which costs you CPU cycles) you can replace `ssh` with `ssh -C`. -## The target directory contains an outdated copy of the repositories: use `rsync` +### The target directory contains an outdated copy of the repositories: use `rsync` + +DANGER: **Warning:** +Using `rsync` to migrate Git data can cause data loss and repository corruption. +[These instructions are being reviewed](https://gitlab.com/gitlab-org/gitlab/-/issues/270422). If the target directory already contains a partial / outdated copy of the repositories it may be wasteful to copy all the data again with `tar`. In this scenario it is better to use `rsync`. This utility is either already installed on your system or easily installable -via `apt`, `yum`, etc. +via `apt`, `yum`, and so on. ```shell sudo -u git sh -c 'rsync -a --delete /var/opt/gitlab/git-data/repositories/. \ @@ -66,7 +113,11 @@ The `/.` in the command above is very important, without it you can easily get the wrong directory structure in the target directory. If you want to see progress, replace `-a` with `-av`. -### Single `rsync` to another server +#### Single `rsync` to another server + +DANGER: **Warning:** +Using `rsync` to migrate Git data can cause data loss and repository corruption. +[These instructions are being reviewed](https://gitlab.com/gitlab-org/gitlab/-/issues/270422). If the `git` user on your source system has SSH access to the target server you can send the repositories over the network with `rsync`. @@ -76,7 +127,11 @@ sudo -u git sh -c 'rsync -a --delete /var/opt/gitlab/git-data/repositories/. \ git@newserver:/mnt/gitlab/repositories' ``` -## Thousands of Git repositories: use one `rsync` per repository +### Thousands of Git repositories: use one `rsync` per repository + +DANGER: **Warning:** +Using `rsync` to migrate Git data can cause data loss and repository corruption. +[These instructions are being reviewed](https://gitlab.com/gitlab-org/gitlab/-/issues/270422). Every time you start an `rsync` job it has to inspect all files in the source directory, all files in the target directory, and then @@ -86,20 +141,20 @@ for your GitLab server. In cases like this you can make `rsync`'s life easier by dividing its work in smaller pieces, and sync one repository at a time. -In addition to `rsync` we will use [GNU -Parallel](http://www.gnu.org/software/parallel/). This utility is -not included in GitLab so you need to install it yourself with `apt` -or `yum`. Also note that the GitLab scripts we used below were added -in GitLab 8.1. +In addition to `rsync` we use [GNU Parallel](http://www.gnu.org/software/parallel/). +This utility is not included in GitLab so you need to install it yourself with `apt` +or `yum`. Also note that the GitLab scripts we used below were added in GitLab 8.1. **This process does not clean up repositories at the target location that no -longer exist at the source.** If you start using your GitLab instance with -`/mnt/gitlab/repositories`, you need to run `gitlab-rake gitlab:cleanup:repos` -after switching to the new repository storage directory. +longer exist at the source.** -### Parallel `rsync` for all repositories known to GitLab +#### Parallel `rsync` for all repositories known to GitLab -This will sync repositories with 10 `rsync` processes at a time. We keep +DANGER: **Warning:** +Using `rsync` to migrate Git data can cause data loss and repository corruption. +[These instructions are being reviewed](https://gitlab.com/gitlab-org/gitlab/-/issues/270422). + +This syncs repositories with 10 `rsync` processes at a time. We keep track of progress so that the transfer can be restarted if necessary. First we create a new directory, owned by `git`, to hold transfer @@ -154,7 +209,11 @@ cat /home/git/transfer-logs/* | sort | uniq -u |\ ` ``` -### Parallel `rsync` only for repositories with recent activity +#### Parallel `rsync` only for repositories with recent activity + +DANGER: **Warning:** +Using `rsync` to migrate Git data can cause data loss and repository corruption. +[These instructions are being reviewed](https://gitlab.com/gitlab-org/gitlab/-/issues/270422). Suppose you have already done one sync that started after 2015-10-1 12:00 UTC. Then you might only want to sync repositories that were changed via GitLab diff --git a/doc/administration/operations/puma.md b/doc/administration/operations/puma.md index 2d53a790428..5104b65c86d 100644 --- a/doc/administration/operations/puma.md +++ b/doc/administration/operations/puma.md @@ -33,7 +33,7 @@ will _not_ carry over automatically, due to differences between the two applicat deployments, see [Configuring Puma Settings](https://docs.gitlab.com/omnibus/settings/puma.html#configuring-puma-settings). For Helm based deployments, see the [Webservice Chart documentation](https://docs.gitlab.com/charts/charts/gitlab/webservice/index.html). -Additionally we strongly recommend that multi-node deployments [configure their load balancers to utilize the readiness check](../load_balancer.md#readiness-check) due to a difference between Unicorn and Puma in how they handle connections during a restart of the service. +Additionally we strongly recommend that multi-node deployments [configure their load balancers to use the readiness check](../load_balancer.md#readiness-check) due to a difference between Unicorn and Puma in how they handle connections during a restart of the service. ## Performance caveat when using Puma with Rugged diff --git a/doc/administration/operations/rails_console.md b/doc/administration/operations/rails_console.md index 9db9a885baf..dac36135a8e 100644 --- a/doc/administration/operations/rails_console.md +++ b/doc/administration/operations/rails_console.md @@ -1,3 +1,9 @@ +--- +stage: none +group: unassigned +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +--- + # The Rails Console The [Rails console](https://guides.rubyonrails.org/command_line.html#rails-console). diff --git a/doc/administration/operations/sidekiq_memory_killer.md b/doc/administration/operations/sidekiq_memory_killer.md index d99468411e3..0de8b681dd8 100644 --- a/doc/administration/operations/sidekiq_memory_killer.md +++ b/doc/administration/operations/sidekiq_memory_killer.md @@ -1,6 +1,6 @@ --- -stage: none -group: unassigned +stage: Enablement +group: Memory info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers --- |