diff options
Diffstat (limited to 'doc/administration')
74 files changed, 1074 insertions, 386 deletions
diff --git a/doc/administration/audit_events.md b/doc/administration/audit_events.md index ccb4ccbd525..7d3be9e1bd3 100644 --- a/doc/administration/audit_events.md +++ b/doc/administration/audit_events.md @@ -18,7 +18,7 @@ permission level, who added a new user, or who removed a user. ## Use-cases -- Check who was the person who changed the permission level of a particular +- Check who the person was that changed the permission level of a particular user for a project in GitLab. - Use it to track which users have access to a certain group of projects in GitLab, and who gave them that permission level. @@ -80,6 +80,9 @@ From there, you can see the following actions: - Project was archived - Project was unarchived - Added/removed/updated protected branches +- Release was added to a project +- Release was updated +- Release milestone associations changed ### Instance events **(PREMIUM ONLY)** @@ -104,7 +107,7 @@ recorded: - Started/stopped user impersonation It is possible to filter particular actions by choosing an audit data type from -the filter drop-down. You can further filter by specific group, project or user +the filter dropdown box. You can further filter by specific group, project or user (for authentication events).  diff --git a/doc/administration/auditor_users.md b/doc/administration/auditor_users.md index 9b4d0f443cf..46f8ae25916 100644 --- a/doc/administration/auditor_users.md +++ b/doc/administration/auditor_users.md @@ -9,7 +9,7 @@ resources on the GitLab instance. Auditor users can have full access to their own resources (projects, groups, snippets, etc.), and read-only access to **all** other resources, except the -Admin area. To put another way, they are just regular users (who can be added +Admin Area. To put another way, they are just regular users (who can be added to projects, create personal snippets, create milestones on their groups, etc.) who also happen to have read-only access to all projects on the system that they haven't been explicitly [given access][permissions] to. @@ -28,7 +28,7 @@ To sum up, assuming you have logged-in as an Auditor user: have the same access as the [permissions] they were given to. For example, if they were added as a Developer, they could then push commits or comment on issues. -- The Auditor cannot view the Admin area, or perform any admin actions. +- The Auditor cannot view the Admin Area, or perform any admin actions. For more information about what an Auditor can or can't do, see the [Permissions and restrictions of an Auditor user](#permissions-and-restrictions-of-an-auditor-user) @@ -73,7 +73,7 @@ instance, with the following permissions/restrictions: - Can read issues / MRs - Can read project snippets - Cannot be Admin and Auditor at the same time -- Cannot access the Admin area +- Cannot access the Admin Area - In a group / project they're not a member of: - Cannot access project settings - Cannot access group settings diff --git a/doc/administration/auth/how_to_configure_ldap_gitlab_ce/index.md b/doc/administration/auth/how_to_configure_ldap_gitlab_ce/index.md index 743893d984a..800bb28c664 100644 --- a/doc/administration/auth/how_to_configure_ldap_gitlab_ce/index.md +++ b/doc/administration/auth/how_to_configure_ldap_gitlab_ce/index.md @@ -23,7 +23,7 @@ For example, [Active Directory](https://docs.microsoft.com/en-us/previous-versio - [Oracle Internet Directory](https://www.oracle.com/middleware/technologies/internet-directory.html) - [OpenLDAP](http://www.openldap.org/) - [389 Directory](http://directory.fedoraproject.org/) -- [OpenDJ (Renamed to Foregerock Directory Services)](https://www.forgerock.com/platform/directory-services) +- [OpenDJ (Renamed to Forgerock Directory Services)](https://www.forgerock.com/platform/directory-services) - [ApacheDS](https://directory.apache.org/) > GitLab uses the [Net::LDAP](https://rubygems.org/gems/net-ldap) library under the hood. This means it supports all [IETF](https://tools.ietf.org/html/rfc2251) compliant LDAPv3 servers. diff --git a/doc/administration/auth/ldap-ee.md b/doc/administration/auth/ldap-ee.md index 34fd97a24ee..a15e34c33a5 100644 --- a/doc/administration/auth/ldap-ee.md +++ b/doc/administration/auth/ldap-ee.md @@ -452,7 +452,7 @@ things to check to debug the situation. links by visiting the GitLab group, then **Settings dropdown > LDAP groups**. - Check that the user has an LDAP identity: 1. Sign in to GitLab as an administrator user. - 1. Navigate to **Admin area > Users**. + 1. Navigate to **Admin Area > Users**. 1. Search for the user 1. Open the user, by clicking on their name. Do not click 'Edit'. 1. Navigate to the **Identities** tab. There should be an LDAP identity with diff --git a/doc/administration/auth/ldap.md b/doc/administration/auth/ldap.md index d449a5a72af..857f554f2fe 100644 --- a/doc/administration/auth/ldap.md +++ b/doc/administration/auth/ldap.md @@ -52,7 +52,7 @@ If a user is deleted from the LDAP server, they will be blocked in GitLab as well. Users will be immediately blocked from logging in. However, there is an LDAP check cache time of one hour (see note) which means users that are already logged in or are using Git over SSH will still be able to access -GitLab for up to one hour. Manually block the user in the GitLab Admin area to +GitLab for up to one hour. Manually block the user in the GitLab Admin Area to immediately block all access. NOTE: **Note**: diff --git a/doc/administration/file_hooks.md b/doc/administration/file_hooks.md new file mode 100644 index 00000000000..5ef739a4289 --- /dev/null +++ b/doc/administration/file_hooks.md @@ -0,0 +1,116 @@ +# File hooks + +> Introduced in GitLab 10.6. +> Until 12.8 the feature name was Plugins. + +With custom file hooks, GitLab administrators can introduce custom integrations +without modifying GitLab's source code. + +NOTE: **Note:** +Instead of writing and supporting your own file hook you can make changes +directly to the GitLab source code and contribute back upstream. This way we can +ensure functionality is preserved across versions and covered by tests. + +NOTE: **Note:** +File hooks must be configured on the filesystem of the GitLab server. Only GitLab +server administrators will be able to complete these tasks. Explore +[system hooks] or [webhooks] as an option if you do not have filesystem access. + +A file hook will run on each event so it's up to you to filter events or projects +within a file hook code. You can have as many file hooks as you want. Each file hook will +be triggered by GitLab asynchronously in case of an event. For a list of events +see the [system hooks] documentation. + +## Setup + +The file hooks must be placed directly into the `plugins` directory, subdirectories +will be ignored. There is an +[`example` directory inside `plugins`](https://gitlab.com/gitlab-org/gitlab/tree/master/plugins/examples) +where you can find some basic examples. + +Follow the steps below to set up a custom hook: + +1. On the GitLab server, navigate to the plugin directory. + For an installation from source the path is usually + `/home/git/gitlab/plugins/`. For Omnibus installs the path is + usually `/opt/gitlab/embedded/service/gitlab-rails/plugins`. + + For [highly available] configurations, your hook file should exist on each + application server. + +1. Inside the `plugins` directory, create a file with a name of your choice, + without spaces or special characters. +1. Make the hook file executable and make sure it's owned by the Git user. +1. Write the code to make the file hook function as expected. That can be + in any language, and ensure the 'shebang' at the top properly reflects the + language type. For example, if the script is in Ruby the shebang will + probably be `#!/usr/bin/env ruby`. +1. The data to the file hook will be provided as JSON on STDIN. It will be exactly + same as for [system hooks] + +That's it! Assuming the file hook code is properly implemented, the hook will fire +as appropriate. The file hooks file list is updated for each event, there is no +need to restart GitLab to apply a new file hook. + +If a file hook executes with non-zero exit code or GitLab fails to execute it, a +message will be logged to: + +- `gitlab-rails/plugin.log` in an Omnibus installation. +- `log/plugin.log` in a source installation. + +## Creating file hooks + +Below is an example that will only response on the event `project_create` and +will inform the admins from the GitLab instance that a new project has been created. + +```ruby +# By using the embedded ruby version we eliminate the possibility that our chosen language +# would be unavailable from +#!/opt/gitlab/embedded/bin/ruby +require 'json' +require 'mail' + +# The incoming variables are in JSON format so we need to parse it first. +ARGS = JSON.parse(STDIN.read) + +# We only want to trigger this file hook on the event project_create +return unless ARGS['event_name'] == 'project_create' + +# We will inform our admins of our gitlab instance that a new project is created +Mail.deliver do + from 'info@gitlab_instance.com' + to 'admin@gitlab_instance.com' + subject "new project " + ARGS['name'] + body ARGS['owner_name'] + 'created project ' + ARGS['name'] +end +``` + +## Validation + +Writing your own file hook can be tricky and it's easier if you can check it +without altering the system. A rake task is provided so that you can use it +in a staging environment to test your file hook before using it in production. +The rake task will use a sample data and execute each of file hook. The output +should be enough to determine if the system sees your file hook and if it was +executed without errors. + +```bash +# Omnibus installations +sudo gitlab-rake file_hooks:validate + +# Installations from source +cd /home/git/gitlab +bundle exec rake file_hooks:validate RAILS_ENV=production +``` + +Example of output: + +``` +Validating file hooks from /plugins directory +* /home/git/gitlab/plugins/save_to_file.clj succeed (zero exit code) +* /home/git/gitlab/plugins/save_to_file.rb failure (non-zero exit code) +``` + +[system hooks]: ../system_hooks/system_hooks.md +[webhooks]: ../user/project/integrations/webhooks.md +[highly available]: ./high_availability/README.md diff --git a/doc/administration/geo/replication/database.md b/doc/administration/geo/replication/database.md index 72c3692716b..bddd30dbb2a 100644 --- a/doc/administration/geo/replication/database.md +++ b/doc/administration/geo/replication/database.md @@ -266,13 +266,13 @@ There is an [issue where support is being discussed](https://gitlab.com/gitlab-o 1. SSH into your GitLab **secondary** server and login as root: - ``` + ```sh sudo -i ``` 1. Stop application server and Sidekiq - ``` + ```sh gitlab-ctl stop unicorn gitlab-ctl stop sidekiq ``` @@ -295,7 +295,7 @@ There is an [issue where support is being discussed](https://gitlab.com/gitlab-o 1. Create a file `server.crt` in the **secondary** server, with the content you got on the last step of the **primary** node's setup: - ``` + ```sh editor server.crt ``` diff --git a/doc/administration/geo/replication/datatypes.md b/doc/administration/geo/replication/datatypes.md new file mode 100644 index 00000000000..75ce7503c34 --- /dev/null +++ b/doc/administration/geo/replication/datatypes.md @@ -0,0 +1,168 @@ +# Geo data types support + +A Geo data type is a specific class of data that is required by one or more GitLab features to +store relevant information. + +To replicate data produced by these features with Geo, we use several strategies to access, transfer, and verify them. + +## Data types + +We currently distinguish between three different data types: + +- [Git repositories](#git-repositories) +- [Blobs](#blobs) +- [Database](#database) + +See the list below of each feature or component we replicate, its corresponding data type, replication and +verification methods: + +| Type | Feature / component | Replication method | Verification method | +|----------|-----------------------------------------------|---------------------------------------------|----------------------| +| Database | Application data in PostgreSQL | Native | Native | +| Database | Redis | _N/A_ (*1*) | _N/A_ | +| Database | Elasticsearch | Native | Native | +| Database | Personal snippets | Postgres Replication | Postgres Replication | +| Database | Project snippets | Postgres Replication | Postgres Replication | +| Database | SSH public keys | Postgres Replication | Postgres Replication | +| Git | Project repository | Geo with Gitaly | Gitaly Checksum | +| Git | Project wiki repository | Geo with Gitaly | Gitaly Checksum | +| Git | Project designs repository | Geo with Gitaly | Gitaly Checksum | +| Git | Object pools for forked project deduplication | Geo with Gitaly | _Not implemented_ | +| Blobs | User uploads _(filesystem)_ | Geo with API | _Not implemented_ | +| Blobs | User uploads _(object storage)_ | Geo with API/Managed (*2*) | _Not implemented_ | +| Blobs | LFS objects _(filesystem)_ | Geo with API | _Not implemented_ | +| Blobs | LFS objects _(object storage)_ | Geo with API/Managed (*2*) | _Not implemented_ | +| Blobs | CI job artifacts _(filesystem)_ | Geo with API | _Not implemented_ | +| Blobs | CI job artifacts _(object storage)_ | Geo with API/Managed (*2*) | _Not implemented_ | +| Blobs | Archived CI build traces _(filesystem)_ | Geo with API | _Not implemented_ | +| Blobs | Archived CI build traces _(object storage)_ | Geo with API/Managed (*2*) | _Not implemented_ | +| Blobs | Container registry _(filesystem)_ | Geo with API/Docker API | _Not implemented_ | +| Blobs | Container registry _(object storage)_ | Geo with API/Managed/Docker API (*2*) | _Not implemented_ | + +- (*1*): Redis replication can be used as part of HA with Redis sentinel. It's not used between Geo nodes. +- (*2*): Object storage replication can be performed by Geo or by your object storage provider/appliance + native replication feature. + +### Git repositories + +A GitLab instance can have one or more repository shards. Each shard has a Gitaly instance that +is responsible for allowing access and operations on the locally stored Git repositories. It can run +on a machine with a single disk, multiple disks mounted as a single mount-point (like with a RAID array), +or using LVM. + +It requires no special filesystem and can work with NFS or a mounted Storage Appliance (there may be +performance limitations when using a remote filesystem). + +Communication is done via Gitaly's own gRPC API. There are three possible ways of synchronization: + +- Using regular Git clone/fetch from one Geo node to another (with special authentication). +- Using repository snapshots (for when the first method fails or repository is corrupt). +- Manual trigger from the Admin UI (a combination of both of the above). + +Each project can have at most 3 different repositories: + +- A project repository, where the source code is stored. +- A wiki repository, where the wiki content is stored. +- A design repository, where design artifacts are indexed (assets are actually in LFS). + +They all live in the same shard and share the same base name with a `-wiki` and `-design` suffix +for Wiki and Design Repository cases. + +### Blobs + +GitLab stores files and blobs such as Issue attachments or LFS objects into either: + +- The filesystem in a specific location. +- An Object Storage solution. Object Storage solutions can be: + - Cloud based like Amazon S3 Google Cloud Storage. + - Self hosted (like MinIO). + - A Storage Appliance that exposes an Object Storage-compatible API. + +When using the filesystem store instead of Object Storage, you need to use network mounted filesystems +to run GitLab when using more than one server (for example with a High Availability setup). + +With respect to replication and verification: + +- We transfer files and blobs using an internal API request. +- With Object Storage, you can either: + - Use a cloud provider replication functionality. + - Have GitLab replicate it for you. + +### Database + +GitLab relies on data stored in multiple databases, for different use-cases. +PostgreSQL is the single point of truth for user-generated content in the Web interface, like issues content, comments +as well as permissions and credentials. + +PostgreSQL can also hold some level of cached data like HTML rendered Markdown, cached merge-requests diff (this can +also be configured to be offloaded to object storage). + +We use PostgreSQL's own replication functionality to replicate data from the primary to secondary nodes. + +We use Redis both as a cache store and to hold persistent data for our background jobs system. Because both +use-cases has data that are exclusive to the same Geo node, we don't replicate it between nodes. + +Elasticsearch is an optional database, that can enable advanced searching capabilities, like improved Global Search +in both source-code level and user generated content in Issues / Merge-Requests and discussions. Currently it's not +supported in Geo. + +## Limitations on replication/verification + +The following table lists the GitLab features along with their replication +and verification status on a **secondary** node. + +You can keep track of the progress to implement the missing items in +these epics/issues: + +- [Unreplicated Data Types](https://gitlab.com/groups/gitlab-org/-/epics/893) +- [Verify all replicated data](https://gitlab.com/groups/gitlab-org/-/epics/1430) + +DANGER: **DANGER** +Features not on this list, or with **No** in the **Replicated** column, +are not replicated on the **secondary** node. Failing over without manually +replicating data from those features will cause the data to be **lost**. +If you wish to use those features on a **secondary** node, or to execute a failover +successfully, you must replicate their data using some other means. + +| Feature | Replicated | Verified | Notes | +|-----------------------------------------------------|--------------------------|-----------------------------|---------------------------------------------| +| Application data in PostgreSQL | **Yes** | **Yes** | | +| Project repository | **Yes** | **Yes** | | +| Project wiki repository | **Yes** | **Yes** | | +| Project designs repository | **Yes** | [No][design-verification] | | +| Uploads | **Yes** | [No][upload-verification] | Verified only on transfer, or manually (*1*)| +| LFS objects | **Yes** | [No][lfs-verification] | Verified only on transfer, or manually (*1*)| +| CI job artifacts (other than traces) | **Yes** | [No][artifact-verification] | Verified only manually (*1*) | +| Archived traces | **Yes** | [No][artifact-verification] | Verified only on transfer, or manually (*1*)| +| Personal snippets | **Yes** | **Yes** | | +| Project snippets | **Yes** | **Yes** | | +| Object pools for forked project deduplication | **Yes** | No | | +| [Server-side Git Hooks][custom-hooks] | No | No | | +| [Elasticsearch integration][elasticsearch] | No | No | | +| [GitLab Pages][gitlab-pages] | [No][pages-replication] | No | | +| [Container Registry][container-registry] | **Yes** | No | | +| [NPM Registry][npm-registry] | No | No | | +| [Maven Repository][maven-repository] | No | No | | +| [Conan Repository][conan-repository] | No | No | | +| [External merge request diffs][merge-request-diffs] | [No][diffs-replication] | No | | +| Content in object storage | **Yes** | No | | + +- (*1*): The integrity can be verified manually using + [Integrity Check Rake Task](../../raketasks/check.md) on both nodes and comparing the output between them. + +[design-replication]: https://gitlab.com/groups/gitlab-org/-/epics/1633 +[design-verification]: https://gitlab.com/gitlab-org/gitlab/issues/32467 +[upload-verification]: https://gitlab.com/groups/gitlab-org/-/epics/1817 +[lfs-verification]: https://gitlab.com/gitlab-org/gitlab/issues/8922 +[artifact-verification]: https://gitlab.com/gitlab-org/gitlab/issues/8923 +[diffs-replication]: https://gitlab.com/gitlab-org/gitlab/issues/33817 +[pages-replication]: https://gitlab.com/groups/gitlab-org/-/epics/589 + +[custom-hooks]: ../../custom_hooks.md +[elasticsearch]: ../../../integration/elasticsearch.md +[gitlab-pages]: ../../pages/index.md +[container-registry]: ../../packages/container_registry.md +[npm-registry]: ../../../user/packages/npm_registry/index.md +[maven-repository]: ../../../user/packages/maven_repository/index.md +[conan-repository]: ../../../user/packages/conan_repository/index.md +[merge-request-diffs]: ../../merge_request_diffs.md diff --git a/doc/administration/geo/replication/external_database.md b/doc/administration/geo/replication/external_database.md index 4451d3c6c08..6948dcc0c68 100644 --- a/doc/administration/geo/replication/external_database.md +++ b/doc/administration/geo/replication/external_database.md @@ -13,13 +13,13 @@ developed and tested. We aim to be compatible with most external 1. SSH into a GitLab **primary** application server and login as root: - ```sh + ```bash sudo -i ``` 1. Execute the command below to define the node as **primary** node: - ```sh + ```bash gitlab-ctl set-geo-primary-node ``` @@ -47,7 +47,7 @@ configures the **primary** node's database to be replicated by making changes to `pg_hba.conf` and `postgresql.conf`. Make the following configuration changes manually to your external database configuration: -``` +```plaintext ## ## Geo Primary Role ## - pg_hba.conf @@ -55,7 +55,7 @@ manually to your external database configuration: host replication gitlab_replicator <trusted secondary IP>/32 md5 ``` -``` +```plaintext ## ## Geo Primary Role ## - postgresql.conf @@ -75,7 +75,7 @@ hot_standby = on Make the following configuration changes manually to your `postgresql.conf` of external replica database: -``` +```plaintext ## ## Geo Secondary Role ## - postgresql.conf diff --git a/doc/administration/geo/replication/high_availability.md b/doc/administration/geo/replication/high_availability.md index faa9d051107..19266a6b358 100644 --- a/doc/administration/geo/replication/high_availability.md +++ b/doc/administration/geo/replication/high_availability.md @@ -123,6 +123,20 @@ a single node only, rather than as a PostgreSQL cluster. Configure the [**secondary** database](database.md) as a read-only replica of the **primary** database. Use the following as a guide. +1. Generate an MD5 hash of the desired password for the database user that the + GitLab application will use to access the read-replica database: + + Note that the username (`gitlab` by default) is incorporated into the hash. + + ```sh + gitlab-ctl pg-password-md5 gitlab + # Enter password: <your_password_here> + # Confirm password: <your_password_here> + # fca0b89a972d69f00eb3ec98a5838484 + ``` + + Use this hash to fill in `<md5_hash_of_your_password>` in the next step. + 1. Edit `/etc/gitlab/gitlab.rb` in the replica database machine, and add the following: @@ -167,6 +181,22 @@ only a single machine, rather than as a PostgreSQL cluster. Configure the tracking database. +1. Generate an MD5 hash of the desired password for the database user that the + GitLab application will use to access the tracking database: + + Note that the username (`gitlab_geo` by default) is incorporated into the + hash. + + ```sh + gitlab-ctl pg-password-md5 gitlab_geo + # Enter password: <your_password_here> + # Confirm password: <your_password_here> + # fca0b89a972d69f00eb3ec98a5838484 + ``` + + Use this hash to fill in `<tracking_database_password_md5_hash>` in the next + step. + 1. Edit `/etc/gitlab/gitlab.rb` in the tracking database machine, and add the following: diff --git a/doc/administration/geo/replication/img/adding_a_secondary_node.png b/doc/administration/geo/replication/img/adding_a_secondary_node.png Binary files differindex 5421b578672..e33b690da18 100644 --- a/doc/administration/geo/replication/img/adding_a_secondary_node.png +++ b/doc/administration/geo/replication/img/adding_a_secondary_node.png diff --git a/doc/administration/geo/replication/img/single_git_add_geolocation_rule.png b/doc/administration/geo/replication/img/single_git_add_geolocation_rule.png Binary files differindex 4b04ba8d1f1..0d1b12e925f 100644 --- a/doc/administration/geo/replication/img/single_git_add_geolocation_rule.png +++ b/doc/administration/geo/replication/img/single_git_add_geolocation_rule.png diff --git a/doc/administration/geo/replication/img/single_git_add_traffic_policy_endpoints.png b/doc/administration/geo/replication/img/single_git_add_traffic_policy_endpoints.png Binary files differindex c19ad57c953..4dfb78986da 100644 --- a/doc/administration/geo/replication/img/single_git_add_traffic_policy_endpoints.png +++ b/doc/administration/geo/replication/img/single_git_add_traffic_policy_endpoints.png diff --git a/doc/administration/geo/replication/img/single_git_clone_panel.png b/doc/administration/geo/replication/img/single_git_clone_panel.png Binary files differindex 8aa0bd2f7d8..427224f5b78 100644 --- a/doc/administration/geo/replication/img/single_git_clone_panel.png +++ b/doc/administration/geo/replication/img/single_git_clone_panel.png diff --git a/doc/administration/geo/replication/img/single_git_create_policy_records_with_traffic_policy.png b/doc/administration/geo/replication/img/single_git_create_policy_records_with_traffic_policy.png Binary files differindex a554532f3b8..ecc4859ca17 100644 --- a/doc/administration/geo/replication/img/single_git_create_policy_records_with_traffic_policy.png +++ b/doc/administration/geo/replication/img/single_git_create_policy_records_with_traffic_policy.png diff --git a/doc/administration/geo/replication/img/single_git_created_policy_record.png b/doc/administration/geo/replication/img/single_git_created_policy_record.png Binary files differindex 74c42395e15..f541c0dd236 100644 --- a/doc/administration/geo/replication/img/single_git_created_policy_record.png +++ b/doc/administration/geo/replication/img/single_git_created_policy_record.png diff --git a/doc/administration/geo/replication/img/single_git_name_policy.png b/doc/administration/geo/replication/img/single_git_name_policy.png Binary files differindex 1a976539e94..5571a41cb3c 100644 --- a/doc/administration/geo/replication/img/single_git_name_policy.png +++ b/doc/administration/geo/replication/img/single_git_name_policy.png diff --git a/doc/administration/geo/replication/img/single_git_policy_diagram.png b/doc/administration/geo/replication/img/single_git_policy_diagram.png Binary files differindex d62952dbbb3..eacd4de0e29 100644 --- a/doc/administration/geo/replication/img/single_git_policy_diagram.png +++ b/doc/administration/geo/replication/img/single_git_policy_diagram.png diff --git a/doc/administration/geo/replication/img/single_git_traffic_policies.png b/doc/administration/geo/replication/img/single_git_traffic_policies.png Binary files differindex b3193c23d99..197b0ac182f 100644 --- a/doc/administration/geo/replication/img/single_git_traffic_policies.png +++ b/doc/administration/geo/replication/img/single_git_traffic_policies.png diff --git a/doc/administration/geo/replication/index.md b/doc/administration/geo/replication/index.md index 0d2ca9ea9d9..04f61775b29 100644 --- a/doc/administration/geo/replication/index.md +++ b/doc/administration/geo/replication/index.md @@ -252,74 +252,13 @@ This list of limitations only reflects the latest version of GitLab. If you are ### Limitations on replication/verification -The following table lists the GitLab features along with their replication -and verification status on a **secondary** node. - You can keep track of the progress to implement the missing items in these epics/issues: - [Unreplicated Data Types](https://gitlab.com/groups/gitlab-org/-/epics/893) - [Verify all replicated data](https://gitlab.com/groups/gitlab-org/-/epics/1430) -| Feature | Replicated | Verified | Notes | -|-----------------------------------------------------|--------------------------|-----------------------------|--------------------------------------------| -| All database content | **Yes** | **Yes** | | -| Project repository | **Yes** | **Yes** | | -| Project wiki repository | **Yes** | **Yes** | | -| Project designs repository | **Yes** | [No][design-verification] | Behind feature flag (2) | -| Uploads | **Yes** | [No][upload-verification] | Verified only on transfer, or manually (1) | -| LFS Objects | **Yes** | [No][lfs-verification] | Verified only on transfer, or manually (1) | -| CI job artifacts (other than traces) | **Yes** | [No][artifact-verification] | Verified only manually (1) | -| Archived traces | **Yes** | [No][artifact-verification] | Verified only on transfer, or manually (1) | -| Personal snippets | **Yes** | **Yes** | | -| Version-controlled personal snippets | No | No | [Not yet supported][unsupported-snippets] | -| Project snippets | **Yes** | **Yes** | | -| Version-controlled project snippets | No | No | [Not yet supported][unsupported-snippets] | -| Object pools for forked project deduplication | **Yes** | No | | -| [Server-side Git Hooks][custom-hooks] | No | No | | -| [Elasticsearch integration][elasticsearch] | No | No | | -| [GitLab Pages][gitlab-pages] | [No][pages-replication] | No | | -| [Container Registry][container-registry] | **Yes** | No | | -| [NPM Registry][npm-registry] | No | No | | -| [Maven Repository][maven-repository] | No | No | | -| [Conan Repository][conan-repository] | No | No | | -| [External merge request diffs][merge-request-diffs] | [No][diffs-replication] | No | | -| Content in object storage | **Yes** | No | | - -[design-replication]: https://gitlab.com/groups/gitlab-org/-/epics/1633 -[design-verification]: https://gitlab.com/gitlab-org/gitlab/issues/32467 -[upload-verification]: https://gitlab.com/groups/gitlab-org/-/epics/1817 -[lfs-verification]: https://gitlab.com/gitlab-org/gitlab/issues/8922 -[artifact-verification]: https://gitlab.com/gitlab-org/gitlab/issues/8923 -[diffs-replication]: https://gitlab.com/gitlab-org/gitlab/issues/33817 -[pages-replication]: https://gitlab.com/groups/gitlab-org/-/epics/589 - -[unsupported-snippets]: https://gitlab.com/gitlab-org/gitlab/issues/14228 -[custom-hooks]: ../../custom_hooks.md -[elasticsearch]: ../../../integration/elasticsearch.md -[gitlab-pages]: ../../pages/index.md -[container-registry]: ../../packages/container_registry.md -[npm-registry]: ../../../user/packages/npm_registry/index.md -[maven-repository]: ../../../user/packages/maven_repository/index.md -[conan-repository]: ../../../user/packages/conan_repository/index.md -[merge-request-diffs]: ../../merge_request_diffs.md - -1. The integrity can be verified manually using -[Integrity Check Rake Task](../../raketasks/check.md) -on both nodes and comparing the output between them. -1. Enable the `enable_geo_design_sync` feature flag by running the -following in a Rails console: - - ```ruby - Feature.disable(:enable_geo_design_sync) - ``` - -DANGER: **DANGER** -Features not on this list, or with **No** in the **Replicated** column, -are not replicated on the **secondary** node. Failing over without manually -replicating data from those features will cause the data to be **lost**. -If you wish to use those features on a **secondary** node, or to execute a failover -successfully, you must replicate their data using some other means. +There is a complete list of all GitLab [data types](datatypes.md) and [existing support for replication and verification](datatypes.md#limitations-on-replicationverification). ## Frequently Asked Questions diff --git a/doc/administration/geo/replication/troubleshooting.md b/doc/administration/geo/replication/troubleshooting.md index 0a2602261d1..7c36d55027a 100644 --- a/doc/administration/geo/replication/troubleshooting.md +++ b/doc/administration/geo/replication/troubleshooting.md @@ -51,6 +51,7 @@ Checking Geo ... GitLab Geo is available ... yes GitLab Geo is enabled ... yes +This machine's Geo node name matches a database record ... yes, found a secondary node named "Shanghai" GitLab Geo secondary database is correctly configured ... yes Database replication enabled? ... yes Database replication working? ... yes @@ -115,34 +116,36 @@ Any **secondary** nodes should point only to read-only instances. #### Can Geo detect the current node correctly? -Geo finds the current machine's name in `/etc/gitlab/gitlab.rb` by: +Geo finds the current machine's Geo node name in `/etc/gitlab/gitlab.rb` by: - Using the `gitlab_rails['geo_node_name']` setting. - If that is not defined, using the `external_url` setting. -To get a machine's name, run: - -```sh -sudo gitlab-rails runner "puts GeoNode.current_node_name" -``` - This name is used to look up the node with the same **Name** in **Admin Area > Geo**. -To check if current machine is correctly finding its node: +To check if the current machine has a node name that matches a node in the +database, run the check task: ```sh -sudo gitlab-rails runner "puts Gitlab::Geo.current_node.inspect" +sudo gitlab-rake gitlab:geo:check ``` -and expect something like: +It displays the current machine's node name and whether the matching database +record is a **primary** or **secondary** node. -```ruby -#<GeoNode id: 2, schema: "https", host: "gitlab.example.com", port: 443, relative_url_root: "", primary: false, ...> +``` +This machine's Geo node name matches a database record ... yes, found a secondary node named "Shanghai" ``` -By running the command above, `primary` should be `true` when executed in -the **primary** node, and `false` on any **secondary** node. +``` +This machine's Geo node name matches a database record ... no + Try fixing it: + You could add or update a Geo node database record, setting the name to "https://example.com/". + Or you could set this machine's Geo node name to match the name of an existing database record: "London", "Shanghai" + For more information see: + doc/administration/geo/replication/troubleshooting.md#can-geo-detect-the-current-node-correctly +``` ## Fixing errors found when running the Geo check rake task @@ -208,9 +211,9 @@ sudo gitlab-rake gitlab:geo:check Checking Geo ... Finished ``` - - Ensure that you have added the secondary node in the admin area of the primary node. + - Ensure that you have added the secondary node in the Admin Area of the primary node. - Ensure that you entered the `external_url` or `gitlab_rails['geo_node_name']` when adding the secondary node in the admin are of the primary node. - - Prior to GitLab 12.4, edit the secondary node in the admin area of the primary node and ensure that there is a trailing `/` in the `Name` field. + - Prior to GitLab 12.4, edit the secondary node in the Admin Area of the primary node and ensure that there is a trailing `/` in the `Name` field. 1. Check returns Exception: PG::UndefinedTable: ERROR: relation "geo_nodes" does not exist @@ -295,7 +298,7 @@ log data to build up in `pg_xlog`. Removing the unused slots can reduce the amou 1. Start a PostgreSQL console session: ```sh - sudo gitlab-psql gitlabhq_production + sudo gitlab-psql ``` Note: **Note:** Using `gitlab-rails dbconsole` will not work, because managing replication slots requires superuser permissions. @@ -318,6 +321,40 @@ Slots where `active` is `f` are not active. SELECT pg_drop_replication_slot('<name_of_extra_slot>'); ``` +### Message: "ERROR: canceling statement due to conflict with recovery" + +This error may rarely occur under normal usage, and the system is resilient +enough to recover. + +However, under certain conditions, some database queries on secondaries may run +excessively long, which increases the frequency of this error. At some point, +some of these queries will never be able to complete due to being canceled +every time. + +These long-running queries are +[planned to be removed in the future](https://gitlab.com/gitlab-org/gitlab/issues/34269), +but as a workaround, we recommend enabling +[hot_standby_feedback](https://www.postgresql.org/docs/10/hot-standby.html#HOT-STANDBY-CONFLICT). +This increases the likelihood of bloat on the **primary** node as it prevents +`VACUUM` from removing recently-dead rows. However, it has been used +successfully in production on GitLab.com. + +To enable `hot_standby_feedback`, add the following to `/etc/gitlab/gitlab.rb` +on the **secondary** node: + +```ruby +postgresql['hot_standby_feedback'] = 'on' +``` + +Then reconfigure GitLab: + +```sh +sudo gitlab-ctl reconfigure +``` + +To help us resolve this problem, consider commenting on +[the issue](https://gitlab.com/gitlab-org/gitlab/issues/4489). + ### Very large repositories never successfully synchronize on the **secondary** node GitLab places a timeout on all repository clones, including project imports @@ -457,16 +494,55 @@ The following steps are for Omnibus installs only. Using Geo with source-based i To check the configuration: +1. SSH into an app node in the **secondary**: + + ```sh + sudo -i + ``` + + Note: An app node is any machine running at least one of the following services: + + - `puma` + - `unicorn` + - `sidekiq` + - `geo-logcursor` + 1. Enter the database console: + If the tracking database is running on the same node: + ```sh gitlab-geo-psql ``` -1. Check whether any tables are present. If everything is working, you - should see something like this: + Or, if the tracking database is running on a different node, you must specify + the user and host when entering the database console: + + ```sh + gitlab-geo-psql -U gitlab_geo -h <IP of tracking database> + ``` + + You will be prompted for the password of the `gitlab_geo` user. You can find + it in plaintext in `/etc/gitlab/gitlab.rb` at: + + ```ruby + geo_secondary['db_password'] = '<geo_tracking_db_password>' + ``` + + This password is normally set on the tracking database during + [Step 3: Configure the tracking database on the secondary node](high_availability.md#step-3-configure-the-tracking-database-on-the-secondary-node), + and it is set on the app nodes during + [Step 4: Configure the frontend application servers on the secondary node](high_availability.md#step-4-configure-the-frontend-application-servers-on-the-secondary-node). + +1. Check whether any tables are present with the following statement: ```sql + SELECT * from information_schema.foreign_tables; + ``` + + If everything is working, you should see something like this: + + ``` gitlabhq_geo_production=# SELECT * from information_schema.foreign_tables; foreign_table_catalog | foreign_table_schema | foreign_table_name | foreign_server_catalog | foreign_server_name -------------------------+----------------------+-------------------------------------------------+-------------------------+--------------------- @@ -482,7 +558,7 @@ To check the configuration: 1. Check that the foreign server mapping is correct via `\des+`. The results should look something like this: - ```sql + ``` gitlabhq_geo_production=# \des+ List of foreign servers -[ RECORD 1 ]--------+------------------------------------------------------------ @@ -518,7 +594,7 @@ To check the configuration: 1. Check that the user mapping is configured properly via `\deu+`: - ```sql + ``` gitlabhq_geo_production=# \deu+ List of user mappings Server | User name | FDW Options diff --git a/doc/administration/geo/replication/version_specific_updates.md b/doc/administration/geo/replication/version_specific_updates.md index 5288cc6e186..7ce38d80c88 100644 --- a/doc/administration/geo/replication/version_specific_updates.md +++ b/doc/administration/geo/replication/version_specific_updates.md @@ -357,7 +357,7 @@ is prepended with the relevant node for better clarity: 1. **(secondary)** Save the snippet below in a file, let's say `/tmp/replica.sh`. Modify the embedded paths if necessary: - ``` + ```bash #!/bin/bash PORT="5432" diff --git a/doc/administration/git_protocol.md b/doc/administration/git_protocol.md index c1742ff87a7..436f1a55369 100644 --- a/doc/administration/git_protocol.md +++ b/doc/administration/git_protocol.md @@ -37,10 +37,9 @@ service is already configured to accept the `GIT_PROTOCOL` environment and users need not do anything more. For Omnibus GitLab and installations from source, you have to manually update -the SSH configuration of your server: +the SSH configuration of your server by adding the line below to the `/etc/ssh/sshd_config` file: -``` -# /etc/ssh/sshd_config +```plaintext AcceptEnv GIT_PROTOCOL ``` @@ -69,7 +68,7 @@ GIT_TRACE_CURL=1 git -c protocol.version=2 ls-remote https://your-gitlab-instanc You should see that the `Git-Protocol` header is sent: -``` +```plaintext 16:29:44.577888 http.c:657 => Send header: Git-Protocol: version=2 ``` @@ -105,7 +104,7 @@ GIT_SSH_COMMAND="ssh -v" git -c protocol.version=2 ls-remote ssh://your-gitlab-i You should see that the `GIT_PROTOCOL` environment variable is sent: -``` +```plaintext debug1: Sending env GIT_PROTOCOL = version=2 ``` diff --git a/doc/administration/gitaly/img/architecture_v12_4.png b/doc/administration/gitaly/img/architecture_v12_4.png Binary files differindex 1054083bb28..6a3955a483b 100644 --- a/doc/administration/gitaly/img/architecture_v12_4.png +++ b/doc/administration/gitaly/img/architecture_v12_4.png diff --git a/doc/administration/gitaly/index.md b/doc/administration/gitaly/index.md index 9218ffa4006..1aad0d80db4 100644 --- a/doc/administration/gitaly/index.md +++ b/doc/administration/gitaly/index.md @@ -38,19 +38,21 @@ This is an optional way to deploy Gitaly which can benefit GitLab installations that are larger than a single machine. Most installations will be better served with the default configuration used by Omnibus and the GitLab source installation guide. +Follow transition to Gitaly on its own server, [Gitaly servers will need to be upgraded before other servers in your cluster](https://docs.gitlab.com/omnibus/update/#upgrading-gitaly-servers). Starting with GitLab 11.4, Gitaly is able to serve all Git requests without requiring a shared NFS mount for Git repository data. Between 11.4 and 11.8 the exception was the [Elasticsearch indexer](https://gitlab.com/gitlab-org/gitlab-elasticsearch-indexer). But since 11.8 the indexer uses Gitaly for data access as well. NFS can still -be leveraged for redudancy on block level of the Git data. But only has to +be leveraged for redundancy on block level of the Git data. But only has to be mounted on the Gitaly server. -Starting with GitLab 11.8, it is possible to use Elasticsearch in conjunction with +From GitLab v11.8 to v12.2, it is possible to use Elasticsearch in conjunction with a Gitaly setup that isn't utilising NFS. In order to use Elasticsearch in this -scenario, the [new repository indexer](../../integration/elasticsearch.md#elasticsearch-repository-indexer-beta) -needs to be enabled in your GitLab configuration. +scenario, the [new repository indexer](../../integration/elasticsearch.md#elasticsearch-repository-indexer) +needs to be enabled in your GitLab configuration. [Since GitLab v12.3](https://gitlab.com/gitlab-org/gitlab/issues/6481), +the new indexer becomes the default and no configuration is required. NOTE: **Note:** While Gitaly can be used as a replacement for NFS, it's not recommended to use EFS as it may impact GitLab's performance. Review the [relevant documentation](../high_availability/nfs.md#avoid-using-awss-elastic-file-system-efs) @@ -162,11 +164,21 @@ Git operations in GitLab will result in an API error. postgresql['enable'] = false redis['enable'] = false nginx['enable'] = false - prometheus['enable'] = false unicorn['enable'] = false sidekiq['enable'] = false gitlab_workhorse['enable'] = false + # If you don't want to run monitoring services uncomment the following (not recommended) + # alertmanager['enable'] = false + # gitlab_exporter['enable'] = false + # grafana['enable'] = false + # node_exporter['enable'] = false + # prometheus['enable'] = false + + # Enable prometheus monitoring - comment out if you disable monitoring services above. + # This makes Prometheus listen on all interfaces. You must use firewalls to restrict access to this address/port. + prometheus['listen_address'] = '0.0.0.0:9090' + # Prevent database connections during 'gitlab-ctl reconfigure' gitlab_rails['rake_cache_clear'] = false gitlab_rails['auto_migrate'] = false @@ -189,9 +201,14 @@ Git operations in GitLab will result in an API error. 1. Append the following to `/etc/gitlab/gitlab.rb` for each respective server: + <!-- + updates to following example must also be made at + https://gitlab.com/gitlab-org/charts/gitlab/blob/master/doc/advanced/external-gitaly/external-omnibus-gitaly.md#configure-omnibus-gitlab + --> + On `gitaly1.internal`: - ``` + ```ruby git_data_dirs({ 'default' => { 'path' => '/var/opt/gitlab/git-data' @@ -204,7 +221,7 @@ Git operations in GitLab will result in an API error. On `gitaly2.internal`: - ``` + ```ruby git_data_dirs({ 'storage2' => { 'path' => '/srv/gitlab/git-data' @@ -502,7 +519,7 @@ To configure Gitaly with TLS: To observe what type of connections are actually being used in a production environment you can use the following Prometheus query: -``` +```prometheus sum(rate(gitaly_connections_total[5m])) by (type) ``` @@ -559,14 +576,14 @@ a few things that you need to do: 1. Make sure the [`git` user home directory](https://docs.gitlab.com/omnibus/settings/configuration.html#moving-the-home-directory-for-a-user) is on local disk. 1. Configure [database lookup of SSH keys](../operations/fast_ssh_key_lookup.md) - to eliminate the need for a shared authorized_keys file. + to eliminate the need for a shared `authorized_keys` file. 1. Configure [object storage for job artifacts](../job_artifacts.md#using-object-storage) including [incremental logging](../job_logs.md#new-incremental-logging-architecture). 1. Configure [object storage for LFS objects](../lfs/lfs_administration.md#storing-lfs-objects-in-remote-object-storage). 1. Configure [object storage for uploads](../uploads.md#using-object-storage-core-only). -1. Configure [object storage for Merge Request Diffs](../merge_request_diffs.md#using-object-storage). -1. Configure [object storage for Packages](../packages/index.md#using-object-storage) (Optional Feature). -1. Configure [object storage for Dependency Proxy](../packages/dependency_proxy.md#using-object-storage) (Optional Feature). +1. Configure [object storage for merge request diffs](../merge_request_diffs.md#using-object-storage). +1. Configure [object storage for packages](../packages/index.md#using-object-storage) (optional feature). +1. Configure [object storage for dependency proxy](../packages/dependency_proxy.md#using-object-storage) (optional feature). NOTE: **Note:** One current feature of GitLab that still requires a shared directory (NFS) is @@ -601,7 +618,7 @@ This will limit the number of in-flight RPC calls for the given RPC's. The limit is applied per repository. In the example above, each on the Gitaly server can have at most 20 simultaneous PostUploadPack calls in flight, and the same for SSHUploadPack. If another request comes in for -a repository that hase used up its 20 slots, that request will get +a repository that has used up its 20 slots, that request will get queued. You can observe the behavior of this queue via the Gitaly logs and via @@ -631,14 +648,14 @@ machine. Use Prometheus to see what the current authentication behavior of your GitLab installation is. -``` +```prometheus sum(rate(gitaly_authentications_total[5m])) by (enforced, status) ``` In a system where authentication is configured correctly, and where you have live traffic, you will see something like this: -``` +```prometheus {enforced="true",status="ok"} 4424.985419441742 ``` @@ -667,7 +684,7 @@ gitaly['auth_transitioning'] = true After you have applied this, your Prometheus query should return something like this: -``` +```prometheus {enforced="false",status="would be ok"} 4424.985419441742 ``` @@ -713,7 +730,7 @@ gitaly['auth_transitioning'] = false Refresh your Prometheus query. You should now see the same kind of result as you did in the beginning: -``` +```prometheus {enforced="true",status="ok"} 4424.985419441742 ``` @@ -745,7 +762,7 @@ Git implementation itself. Because Rugged+Unicorn was so efficient, GitLab's application code ended up with lots of duplicate Git object lookups (like looking up the -`master` commmit a dozen times in one request). We could write +`master` commit a dozen times in one request). We could write inefficient code without being punished for it. When we migrated these Git lookups to Gitaly calls, we were suddenly @@ -853,14 +870,14 @@ gitaly-debug -h ### Commits, pushes, and clones return a 401 -``` +```plaintext remote: GitLab: 401 Unauthorized ``` You will need to sync your `gitlab-secrets.json` file with your GitLab app nodes. -### Client side GRPC logs +### Client side gRPC logs Gitaly uses the [gRPC](https://grpc.io/) RPC framework. The Ruby gRPC client has its own log file which may contain useful information when @@ -885,7 +902,7 @@ Assuming your `grpc_client_handled_total` counter only observes Gitaly, the following query shows you RPCs are (most likely) internally implemented as calls to `gitaly-ruby`: -``` +```prometheus sum(rate(grpc_client_handled_total[5m])) by (grpc_method) > 0 ``` @@ -910,7 +927,7 @@ Confirm the following are all true: ``` - When any user adds or modifies a file from the repository using the GitLab - UI, it immediatley fails with a red `401 Unauthorized` banner. + UI, it immediately fails with a red `401 Unauthorized` banner. - Creating a new project and [initializing it with a README](../../gitlab-basics/create-project.md#blank-projects) successfully creates the project but doesn't create the README. - When [tailing the logs](https://docs.gitlab.com/omnibus/settings/logs.html#tail-logs-in-a-console-on-the-server) on an app node and reproducing the error, you get `401` errors diff --git a/doc/administration/gitaly/praefect.md b/doc/administration/gitaly/praefect.md index 6193a40ac4f..72c3f996841 100644 --- a/doc/administration/gitaly/praefect.md +++ b/doc/administration/gitaly/praefect.md @@ -25,15 +25,22 @@ The most common architecture for Praefect is simplified in the diagram below: ```mermaid graph TB GitLab --> Praefect; - Praefect --> Gitaly-1; - Praefect --> Gitaly-2; - Praefect --> Gitaly-3; + Praefect --- PostgreSQL; + Praefect --> Gitaly1; + Praefect --> Gitaly2; + Praefect --> Gitaly3; ``` Where `GitLab` is the collection of clients that can request Git operations. -The Praefect node has threestorage nodes attached. Praefect itself doesn't +The Praefect node has three storage nodes attached. Praefect itself doesn't store data, but connects to three Gitaly nodes, `Gitaly-1`, `Gitaly-2`, and `Gitaly-3`. +In order to keep track of replication state, Praefect relies on a +PostgreSQL database. This database is a single point of failure so you +should use a highly available PostgreSQL server for this. GitLab +itself needs a HA PostgreSQL server too, so you could optionally co-locate the Praefect +SQL database on the PostgreSQL server you use for the rest of GitLab. + Praefect may be enabled on its own node or can be run on the GitLab server. In the example below we will use a separate server, but the optimal configuration for Praefect is still being determined. @@ -62,6 +69,53 @@ We need to manage the following secrets and make them match across hosts: `PRAEFECT_EXTERNAL_TOKEN` because Gitaly clients must not be able to access internal nodes of the Praefect cluster directly; that could lead to data loss. +1. `PRAEFECT_SQL_PASSWORD`: this password is used by Praefect to connect to + PostgreSQL. + +#### Network addresses + +1. `POSTGRESQL_SERVER`: the host name or IP address of your PostgreSQL server + +#### PostgreSQL + +To set up a Praefect cluster you need a highly available PostgreSQL +server. You need PostgreSQL 9.6 or newer. Praefect needs to have a SQL +user with the right to create databases. + +In the instructions below we assume you have administrative access to +your PostgreSQL server via `psql`. Depending on your environment, you +may also be able to do this via the web interface of your cloud +platform, or via your configuration management system, etc. + +Below we assume that you have administrative access as the `postgres` +user. First open a `psql` session as the `postgres` user: + +```shell +psql -h POSTGRESQL_SERVER -U postgres -d template1 +``` + +Once you are connected, run the following command. Replace +`PRAEFECT_SQL_PASSWORD` with the actual (random) password you +generated for the `praefect` SQL user: + +```sql +CREATE ROLE praefect WITH LOGIN CREATEDB PASSWORD 'PRAEFECT_SQL_PASSWORD'; +\q # exit psql +``` + +Now connect as the `praefect` user to create the database. This has +the side effect of verifying that you have access: + +```shell +psql -h POSTGRESQL_SERVER -U praefect -d template1 +``` + +Once you have connected as the `praefect` user, run: + +```sql +CREATE DATABASE praefect_production WITH ENCODING=UTF8; +\q # quit psql +``` #### Praefect @@ -118,10 +172,39 @@ praefect['virtual_storages'] = { } } } + +praefect['database_host'] = 'POSTGRESQL_SERVER' +praefect['database_port'] = 5432 +praefect['database_user'] = 'praefect' +praefect['database_password'] = 'PRAEFECT_SQL_PASSWORD' +praefect['database_dbname'] = 'praefect_production' + +# Uncomment the line below if you do not want to use an encrypted +# connection to PostgreSQL +# praefect['database_sslmode'] = 'disable' + +# Uncomment and modify these lines if you are using a TLS client +# certificate to connect to PostgreSQL +# praefect['database_sslcert'] = '/path/to/client-cert' +# praefect['database_sslkey'] = '/path/to/client-key' + +# Uncomment and modify this line if your PostgreSQL server uses a custom +# CA +# praefect['database_sslrootcert'] = '/path/to/rootcert' ``` Save the file and [reconfigure Praefect](../restart_gitlab.md#omnibus-gitlab-reconfigure). +After you reconfigure, verify that Praefect can reach PostgreSQL: + +```shell +sudo -u git /opt/gitlab/embedded/bin/praefect -config /var/opt/gitlab/praefect/config.toml sql-ping +``` + +If the check fails, make sure you have followed the steps correctly. If you edit `/etc/gitlab/gitlab.rb`, +remember to run `sudo gitlab-ctl reconfigure` again before trying the +`sql-ping` command. + #### Gitaly Next we will configure each Gitaly server assigned to Praefect. Configuration for these diff --git a/doc/administration/high_availability/README.md b/doc/administration/high_availability/README.md index d411fb7f20f..13b6bd88453 100644 --- a/doc/administration/high_availability/README.md +++ b/doc/administration/high_availability/README.md @@ -224,14 +224,9 @@ users are, how much automation you use, mirroring, and repo/change size. - **Supported Users (approximate):** 2,000 - **Test RPS Rates:** API: 40 RPS, Web: 4 RPS, Git: 4 RPS -- **Status:** Work-in-progress - **Known Issues:** For the latest list of known performance issues head [here](https://gitlab.com/gitlab-org/gitlab/issues?label_name%5B%5D=Quality%3Aperformance-issues). -NOTE: **Note:** This architecture is a work-in-progress of the work so far. The -Quality team will be certifying this environment in late 2019 or early 2020. The specifications -may be adjusted prior to certification based on performance testing. - | Service | Nodes | Configuration | GCP type | | ----------------------------|-------|-----------------------|---------------| | GitLab Rails <br> - Puma workers on each node set to 90% of available CPUs with 8 threads | 3 | 8 vCPU, 7.2GB Memory | n1-highcpu-8 | @@ -255,14 +250,9 @@ vendors a best effort like for like can be used. - **Supported Users (approximate):** 5,000 - **Test RPS Rates:** API: 100 RPS, Web: 10 RPS, Git: 10 RPS -- **Status:** Work-in-progress - **Known Issues:** For the latest list of known performance issues head [here](https://gitlab.com/gitlab-org/gitlab/issues?label_name%5B%5D=Quality%3Aperformance-issues). -NOTE: **Note:** This architecture is a work-in-progress of the work so far. The -Quality team will be certifying this environment in late 2019 or early 2020. The specifications -may be adjusted prior to certification based on performance testing. - | Service | Nodes | Configuration | GCP type | | ----------------------------|-------|-----------------------|---------------| | GitLab Rails <br> - Puma workers on each node set to 90% of available CPUs with 16 threads | 3 | 16 vCPU, 14.4GB Memory | n1-highcpu-16 | diff --git a/doc/administration/high_availability/consul.md b/doc/administration/high_availability/consul.md index 392b9b76c31..71d380dbec7 100644 --- a/doc/administration/high_availability/consul.md +++ b/doc/administration/high_availability/consul.md @@ -64,7 +64,7 @@ command to verify all server nodes are communicating: The output should be similar to: -``` +```plaintext Node Address Status Type Build Protocol DC CONSUL_NODE_ONE XXX.XXX.XXX.YYY:8301 alive server 0.9.2 2 gitlab_consul CONSUL_NODE_TWO XXX.XXX.XXX.YYY:8301 alive server 0.9.2 2 gitlab_consul @@ -80,8 +80,8 @@ check the [Troubleshooting section](#troubleshooting) before proceeding. To see which nodes are part of the cluster, run the following on any member in the cluster -``` -# /opt/gitlab/embedded/bin/consul members +```shell +$ /opt/gitlab/embedded/bin/consul members Node Address Status Type Build Protocol DC consul-b XX.XX.X.Y:8301 alive server 0.9.0 2 gitlab_consul consul-c XX.XX.X.Y:8301 alive server 0.9.0 2 gitlab_consul @@ -100,7 +100,7 @@ If it is necessary to restart the server cluster, it is important to do this in To be safe, we recommend you only restart one server agent at a time to ensure the cluster remains intact. -For larger clusters, it is possible to restart multiple agents at a time. See the [Consul consensus document](https://www.consul.io/docs/internals/consensus.html#deployment-table) for how many failures it can tolerate. This will be the number of simulateneous restarts it can sustain. +For larger clusters, it is possible to restart multiple agents at a time. See the [Consul consensus document](https://www.consul.io/docs/internals/consensus.html#deployment-table) for how many failures it can tolerate. This will be the number of simultaneous restarts it can sustain. ## Upgrades for bundled Consul @@ -127,7 +127,7 @@ By default, the server agents will attempt to [bind](https://www.consul.io/docs/ You will see messages like the following in `gitlab-ctl tail consul` output if you are running into this issue: -``` +```plaintext 2017-09-25_19:53:39.90821 2017/09/25 19:53:39 [WARN] raft: no known peers, aborting election 2017-09-25_19:53:41.74356 2017/09/25 19:53:41 [ERR] agent: failed to sync remote state: No cluster leader ``` @@ -154,7 +154,7 @@ In the case that a node has multiple private IPs the agent be confused as to whi You will see messages like the following in `gitlab-ctl tail consul` output if you are running into this issue: -``` +```plaintext 2017-11-09_17:41:45.52876 ==> Starting Consul agent... 2017-11-09_17:41:45.53057 ==> Error creating agent: Failed to get advertise address: Multiple private IPs found. Please configure one. ``` @@ -181,10 +181,10 @@ If you lost enough server agents in the cluster to break quorum, then the cluste By default, GitLab does not store anything in the Consul cluster that cannot be recreated. To erase the Consul database and reinitialize -``` -# gitlab-ctl stop consul -# rm -rf /var/opt/gitlab/consul/data -# gitlab-ctl start consul +```shell +gitlab-ctl stop consul +rm -rf /var/opt/gitlab/consul/data +gitlab-ctl start consul ``` After this, the cluster should start back up, and the server agents rejoin. Shortly after that, the client agents should rejoin as well. diff --git a/doc/administration/high_availability/database.md b/doc/administration/high_availability/database.md index 02684f575d4..8e57b049730 100644 --- a/doc/administration/high_availability/database.md +++ b/doc/administration/high_availability/database.md @@ -229,7 +229,7 @@ available database connections. In this document we are assuming 3 database nodes, which makes this configuration: -``` +```ruby postgresql['max_wal_senders'] = 4 ``` @@ -352,7 +352,7 @@ When installing the GitLab package, do not supply `EXTERNAL_URL` value. to inform `gitlab-ctl` that they are standby nodes initially and it need not attempt to register them as primary node - ``` + ```ruby # HA setting to specify if a node should attempt to be master on initialization repmgr['master_on_initialization'] = false ``` @@ -396,7 +396,7 @@ Select one node as a primary node. The output should be similar to the following: - ``` + ```plaintext Role | Name | Upstream | Connection String ----------+----------|----------|---------------------------------------- * master | HOSTNAME | | host=HOSTNAME user=gitlab_repmgr dbname=gitlab_repmgr @@ -442,7 +442,7 @@ Select one node as a primary node. The output should be similar to the following: - ``` + ```plaintext Role | Name | Upstream | Connection String ----------+---------|-----------|------------------------------------------------ * master | MASTER | | host=MASTER_NODE_NAME user=gitlab_repmgr dbname=gitlab_repmgr @@ -463,7 +463,7 @@ gitlab-ctl repmgr cluster show The output should be similar to: -``` +```plaintext Role | Name | Upstream | Connection String ----------+--------------|--------------|-------------------------------------------------------------------- * master | MASTER | | host=MASTER port=5432 user=gitlab_repmgr dbname=gitlab_repmgr @@ -652,7 +652,7 @@ On secondary nodes, edit `/etc/gitlab/gitlab.rb` and add all the configuration added to primary node, noted above. In addition, append the following configuration: -``` +```ruby # HA setting to specify if a node should attempt to be master on initialization repmgr['master_on_initialization'] = false ``` @@ -706,7 +706,7 @@ After deploying the configuration follow these steps: gitlab-psql -d gitlabhq_production ``` - ``` + ```shell CREATE EXTENSION pg_trgm; ``` @@ -804,7 +804,7 @@ consul['configuration'] = { On secondary nodes, edit `/etc/gitlab/gitlab.rb` and add all the information added to primary node, noted above. In addition, append the following configuration -``` +```ruby # HA setting to specify if a node should attempt to be master on initialization repmgr['master_on_initialization'] = false ``` @@ -908,7 +908,7 @@ after it has been restored to service. It will output something like: - ``` + ```plaintext 959789412 ``` @@ -1052,7 +1052,7 @@ Now there should not be errors. If errors still occur then there is another prob You may get this error when running `gitlab-rake gitlab:db:configure` or you may see the error in the PgBouncer log file. -``` +```plaintext PG::ConnectionBad: ERROR: pgbouncer cannot connect to server ``` @@ -1063,13 +1063,13 @@ You can confirm that this is the issue by checking the PostgreSQL log on the mas database node. If you see the following error then `trust_auth_cidr_addresses` is the problem. -``` +```plaintext 2018-03-29_13:59:12.11776 FATAL: no pg_hba.conf entry for host "123.123.123.123", user "pgbouncer", database "gitlabhq_production", SSL off ``` To fix the problem, add the IP address to `/etc/gitlab/gitlab.rb`. -``` +```ruby postgresql['trust_auth_cidr_addresses'] = %w(123.123.123.123/32 <other_cidrs>) ``` diff --git a/doc/administration/high_availability/gitlab.md b/doc/administration/high_availability/gitlab.md index 71ab169a801..b4269cd4e38 100644 --- a/doc/administration/high_availability/gitlab.md +++ b/doc/administration/high_availability/gitlab.md @@ -11,7 +11,7 @@ these additional steps before proceeding with GitLab installation. 1. If necessary, install the NFS client utility packages using the following commands: - ``` + ```shell # Ubuntu/Debian apt-get install nfs-common @@ -24,7 +24,7 @@ these additional steps before proceeding with GitLab installation. to configure your NFS server. See [NFS documentation](nfs.md) for the various options. Here is an example snippet to add to `/etc/fstab`: - ``` + ```plaintext 10.1.0.1:/var/opt/gitlab/.ssh /var/opt/gitlab/.ssh nfs4 defaults,soft,rsize=1048576,wsize=1048576,noatime,nofail,lookupcache=positive 0 2 10.1.0.1:/var/opt/gitlab/gitlab-rails/uploads /var/opt/gitlab/gitlab-rails/uploads nfs4 defaults,soft,rsize=1048576,wsize=1048576,noatime,nofail,lookupcache=positive 0 2 10.1.0.1:/var/opt/gitlab/gitlab-rails/shared /var/opt/gitlab/gitlab-rails/shared nfs4 defaults,soft,rsize=1048576,wsize=1048576,noatime,nofail,lookupcache=positive 0 2 @@ -35,7 +35,7 @@ these additional steps before proceeding with GitLab installation. 1. Create the shared directories. These may be different depending on your NFS mount locations. - ``` + ```shell mkdir -p /var/opt/gitlab/.ssh /var/opt/gitlab/gitlab-rails/uploads /var/opt/gitlab/gitlab-rails/shared /var/opt/gitlab/gitlab-ci/builds /var/opt/gitlab/git-data ``` diff --git a/doc/administration/high_availability/nfs.md b/doc/administration/high_availability/nfs.md index f7c5593e211..1d0dc420987 100644 --- a/doc/administration/high_availability/nfs.md +++ b/doc/administration/high_availability/nfs.md @@ -132,7 +132,7 @@ For supported database architecture, please see our documentation on Below is an example of an NFS mount point defined in `/etc/fstab` we use on GitLab.com: -``` +```plaintext 10.1.1.1:/var/opt/gitlab/git-data /var/opt/gitlab/git-data nfs4 defaults,soft,rsize=1048576,wsize=1048576,noatime,nofail,lookupcache=positive 0 2 ``` @@ -149,7 +149,7 @@ Note there are several options that you should consider using: It's recommended to nest all GitLab data dirs within a mount, that allows automatic restore of backups without manually moving existing data. -``` +```plaintext mountpoint └── gitlab-data ├── builds diff --git a/doc/administration/high_availability/pgbouncer.md b/doc/administration/high_availability/pgbouncer.md index 09b33c3554a..7b93159628d 100644 --- a/doc/administration/high_availability/pgbouncer.md +++ b/doc/administration/high_availability/pgbouncer.md @@ -83,7 +83,7 @@ In a HA setup, it's recommended to run a PgBouncer node separately for each data The output should be similar to the following: - ``` + ```plaintext name | host | port | database | force_user | pool_size | reserve_pool | pool_mode | max_connections | current_connections ---------------------+-------------+------+---------------------+------------+-----------+--------------+-----------+-----------------+--------------------- gitlabhq_production | MASTER_HOST | 5432 | gitlabhq_production | | 20 | 0 | | 0 | 0 @@ -102,7 +102,7 @@ If you're running more than one PgBouncer node as recommended, then at this time As an example here's how you could do it with [HAProxy](https://www.haproxy.org/): -``` +```plaintext global log /dev/log local0 log localhost local1 notice diff --git a/doc/administration/high_availability/redis.md b/doc/administration/high_availability/redis.md index 9b733034f5b..8e94b56a940 100644 --- a/doc/administration/high_availability/redis.md +++ b/doc/administration/high_availability/redis.md @@ -391,7 +391,7 @@ The prerequisites for a HA Redis setup are the following: prevent database migrations from running on upgrade, add the following configuration to your `/etc/gitlab/gitlab.rb` file: - ``` + ```ruby gitlab_rails['auto_migrate'] = false ``` @@ -439,7 +439,7 @@ The prerequisites for a HA Redis setup are the following: 1. To prevent reconfigure from running automatically on upgrade, run: - ``` + ```shell sudo touch /etc/gitlab/skip-auto-reconfigure ``` @@ -569,7 +569,7 @@ multiple machines with the Sentinel daemon. 1. To prevent database migrations from running on upgrade, run: - ``` + ```shell sudo touch /etc/gitlab/skip-auto-reconfigure ``` @@ -898,14 +898,14 @@ Before proceeding with the troubleshooting below, check your firewall rules: You can check if everything is correct by connecting to each server using `redis-cli` application, and sending the `info replication` command as below. -``` +```shell /opt/gitlab/embedded/bin/redis-cli -h <redis-host-or-ip> -a '<redis-password>' info replication ``` When connected to a `master` Redis, you will see the number of connected `slaves`, and a list of each with connection details: -``` +```plaintext # Replication role:master connected_slaves:1 @@ -920,7 +920,7 @@ repl_backlog_histlen:1048576 When it's a `slave`, you will see details of the master connection and if its `up` or `down`: -``` +```plaintext # Replication role:slave master_host:10.133.1.58 @@ -959,7 +959,7 @@ To make sure your configuration is correct: 1. SSH into your GitLab application server 1. Enter the Rails console: - ``` + ```shell # For Omnibus installations sudo gitlab-rails console @@ -985,7 +985,7 @@ To make sure your configuration is correct: 1. Then back in the Rails console from the first step, run: - ``` + ```ruby redis.info ``` diff --git a/doc/administration/housekeeping.md b/doc/administration/housekeeping.md index 9083619841e..ca3480f1146 100644 --- a/doc/administration/housekeeping.md +++ b/doc/administration/housekeeping.md @@ -5,14 +5,13 @@ ## Automatic housekeeping GitLab automatically runs `git gc` and `git repack` on repositories -after Git pushes. If needed you can change how often this happens, or -to turn it off, go to **Admin area > Settings** -(`/admin/application_settings`). +after Git pushes. You can change how often this happens or turn it off in +**Admin Area > Settings > Repository** (`/admin/application_settings/repository`). ## Manual housekeeping -The housekeeping function will run a `repack` or `gc` depending on the -"Automatic Git repository housekeeping" settings configured in **Admin area > Settings** +The housekeeping function runs `repack` or `gc` depending on the +**Housekeeping** settings configured in **Admin Area > Settings > Repository**. For example in the following scenario a `git repack -d` will be executed: diff --git a/doc/administration/img/repository_storages_admin_ui.png b/doc/administration/img/repository_storages_admin_ui.png Binary files differindex 315b4b0144c..51b2f5f8c15 100644 --- a/doc/administration/img/repository_storages_admin_ui.png +++ b/doc/administration/img/repository_storages_admin_ui.png diff --git a/doc/administration/incoming_email.md b/doc/administration/incoming_email.md index a0360f1d252..1550787d532 100644 --- a/doc/administration/incoming_email.md +++ b/doc/administration/incoming_email.md @@ -7,7 +7,7 @@ GitLab has several features based on receiving incoming emails: - [New issue by email](../user/project/issues/managing_issues.md#new-issue-via-email): allow GitLab users to create a new issue by sending an email to a user-specific email address. -- [New merge request by email](../user/project/merge_requests/creating_merge_requests.md#create-new-merge-requests-by-email): +- [New merge request by email](../user/project/merge_requests/creating_merge_requests.md#new-merge-request-by-email-core-only): allow GitLab users to create a new merge request by sending an email to a user-specific email address. - [Service Desk](../user/project/service_desk.md): provide e-mail support to @@ -79,7 +79,7 @@ email address in order to sign up. If you also host a public-facing GitLab instance at `hooli.com` and set your incoming email domain to `hooli.com`, an attacker could abuse the "Create new issue by email" or -"[Create new merge request by email](../user/project/merge_requests/creating_merge_requests.md#create-new-merge-requests-by-email)" +"[Create new merge request by email](../user/project/merge_requests/creating_merge_requests.md#new-merge-request-by-email-core-only)" features by using a project's unique address as the email when signing up for Slack, which would send a confirmation email, which would create a new issue or merge request on the project owned by the attacker, allowing them to click the diff --git a/doc/administration/index.md b/doc/administration/index.md index 2a9980cddb3..8172acd09b4 100644 --- a/doc/administration/index.md +++ b/doc/administration/index.md @@ -119,7 +119,7 @@ Learn how to install, configure, update, and maintain your GitLab instance. - [Auditor users](auditor_users.md): Users with read-only access to all projects, groups, and other resources on the GitLab instance. **(PREMIUM ONLY)** - [Incoming email](incoming_email.md): Configure incoming emails to allow users to [reply by email](reply_by_email.md), create [issues by email](../user/project/issues/managing_issues.md#new-issue-via-email) and - [merge requests by email](../user/project/merge_requests/creating_merge_requests.md#create-new-merge-requests-by-email), and to enable [Service Desk](../user/project/service_desk.md). + [merge requests by email](../user/project/merge_requests/creating_merge_requests.md#new-merge-request-by-email-core-only), and to enable [Service Desk](../user/project/service_desk.md). - [Postfix for incoming email](reply_by_email_postfix_setup.md): Set up a basic Postfix mail server with IMAP authentication on Ubuntu for incoming emails. @@ -200,6 +200,7 @@ Learn how to install, configure, update, and maintain your GitLab instance. - [Log system](logs.md): Where to look for logs. - [Sidekiq Troubleshooting](troubleshooting/sidekiq.md): Debug when Sidekiq appears hung and is not processing jobs. - [Troubleshooting Elasticsearch](troubleshooting/elasticsearch.md) +- [GitLab application limits](instance_limits.md) ### Support Team Docs diff --git a/doc/administration/instance_limits.md b/doc/administration/instance_limits.md new file mode 100644 index 00000000000..d68b825ed88 --- /dev/null +++ b/doc/administration/instance_limits.md @@ -0,0 +1,44 @@ +--- +type: reference +--- + +# GitLab application limits + +GitLab, like most large applications, enforces limits within certain features to maintain a +minimum quality of performance. Allowing some features to be limitless could affect security, +performance, data, or could even exhaust the allocated resources for the application. + +## Number of comments per issue, merge request or commit + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/issues/22388) in GitLab 12.4. + +There's a limit to the number of comments that can be submitted on an issue, +merge request, or commit. When the limit is reached, system notes can still be +added so that the history of events is not lost, but user-submitted comments +will fail. + +- **Max limit:** 5.000 comments + +## Number of pipelines per Git push + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/issues/51401) in GitLab 11.10. + +The number of pipelines that can be created in a single push is 4. +This is to prevent the accidental creation of pipelines when `git push --all` +or `git push --mirror` is used. + +Read more in the [CI documentation](../ci/yaml/README.md#processing-git-pushes). + +## Retention of activity history + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/issues/21164) in GitLab 8.12. + +Activity history for projects and individuals' profiles was limited to one year until [GitLab 11.4](https://gitlab.com/gitlab-org/gitlab-foss/issues/52246) when it was extended to two years, and in [GitLab 12.4](https://gitlab.com/gitlab-org/gitlab/issues/33840) to three years. + +## Number of project webhooks + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/merge_requests/20730) in GitLab 12.6. + +A maximum number of project webhooks applies to each GitLab.com tier. Check the +[Maximum number of webhooks (per tier)](../user/project/integrations/webhooks.md#maximum-number-of-webhooks-per-tier) +section in the Webhooks page. diff --git a/doc/administration/integration/terminal.md b/doc/administration/integration/terminal.md index 1af15648b97..42acc4cb80e 100644 --- a/doc/administration/integration/terminal.md +++ b/doc/administration/integration/terminal.md @@ -96,6 +96,6 @@ they will receive a `Connection failed` message. in GitLab 8.17. Terminal sessions use long-lived connections; by default, these may last -forever. You can configure a maximum session time in the Admin area of your +forever. You can configure a maximum session time in the Admin Area of your GitLab instance if you find this undesirable from a scalability or security point of view. diff --git a/doc/administration/job_artifacts.md b/doc/administration/job_artifacts.md index ec2f40700f5..7a3d116ea58 100644 --- a/doc/administration/job_artifacts.md +++ b/doc/administration/job_artifacts.md @@ -267,7 +267,7 @@ you can flip the feature flag from a Rails console. ## Set the maximum file size of the artifacts Provided the artifacts are enabled, you can change the maximum file size of the -artifacts through the [Admin area settings](../user/admin_area/settings/continuous_integration.md#maximum-artifacts-size-core-only). +artifacts through the [Admin Area settings](../user/admin_area/settings/continuous_integration.md#maximum-artifacts-size-core-only). ## Storage statistics @@ -294,3 +294,149 @@ memory and disk I/O. [reconfigure gitlab]: restart_gitlab.md#omnibus-gitlab-reconfigure "How to reconfigure Omnibus GitLab" [restart gitlab]: restart_gitlab.md#installations-from-source "How to restart GitLab" [gitlab workhorse]: https://gitlab.com/gitlab-org/gitlab-workhorse "GitLab Workhorse repository" + +## Troubleshooting + +### Job artifacts using too much disk space + +Job artifacts can fill up your disk space quicker than expected. Some possible +reasons are: + +- Users have configured job artifacts expiration to be longer than necessary. +- The number of jobs run, and hence artifacts generated, is higher than expected. +- Job logs are larger than expected, and have accumulated over time. + +In these and other cases, you'll need to identify the projects most responsible +for disk space usage, figure out what types of artifacts are using the most +space, and in some cases, manually delete job artifacts to reclaim disk space. + +#### List projects by total size of job artifacts stored + +List the top 20 projects, sorted by the total size of job artifacts stored, by +running the following code in the Rails console (`sudo gitlab-rails console`): + +```ruby +include ActionView::Helpers::NumberHelper +ProjectStatistics.order(build_artifacts_size: :desc).limit(20).each do |s| + puts "#{number_to_human_size(s.build_artifacts_size)} \t #{s.project.full_path}" +end +``` + +You can change the number of projects listed by modifying `.limit(20)` to the +number you want. + +#### List largest artifacts in a single project + +List the 50 largest job artifacts in a single project by running the following +code in the Rails console (`sudo gitlab-rails console`): + +```ruby +include ActionView::Helpers::NumberHelper +project = Project.find_by_full_path('path/to/project') +Ci::JobArtifact.where(project: project).order(size: :desc).limit(50).map { |a| puts "ID: #{a.id} - #{a.file_type}: #{number_to_human_size(a.size)}" } +``` + +You can change the number of job artifacts listed by modifying `.limit(50)` to +the number you want. + +#### Delete job artifacts from jobs completed before a specific date + +CAUTION: **CAUTION:** +These commands remove data permanently from the database and from disk. We +highly recommend running them only under the guidance of a Support Engineer, or +running them in a test environment with a backup of the instance ready to be +restored, just in case. + +If you need to manually remove job artifacts associated with multiple jobs while +**retaining their job logs**, this can be done from the Rails console (`sudo gitlab-rails console`): + +1. Select jobs to be deleted: + + To select all jobs with artifacts for a single project: + + ```ruby + project = Project.find_by_full_path('path/to/project') + builds_with_artifacts = project.builds.with_artifacts_archive + ``` + + To select all jobs with artifacts across the entire GitLab instance: + + ```ruby + builds_with_artifacts = Ci::Build.with_artifacts_archive + ``` + +1. Delete job artifacts older than a specific date: + + NOTE: **NOTE:** + This step will also erase artifacts that users have chosen to + ["keep"](../user/project/pipelines/job_artifacts.html#browsing-artifacts). + + ```ruby + builds_to_clear = builds_with_artifacts.where("finished_at < ?", 1.week.ago) + builds_to_clear.find_each do |build| + build.artifacts_expire_at = Time.now + build.erase_erasable_artifacts! + end + ``` + + `1.week.ago` is a Rails `ActiveSupport::Duration` method which calculates a new + date or time in the past. Other valid examples are: + + - `7.days.ago` + - `3.months.ago` + - `1.year.ago` + +#### Delete job artifacts and logs from jobs completed before a specific date + +CAUTION: **CAUTION:** +These commands remove data permanently from the database and from disk. We +highly recommend running them only under the guidance of a Support Engineer, or +running them in a test environment with a backup of the instance ready to be +restored, just in case. + +If you need to manually remove ALL job artifacts associated with multiple jobs, +**including job logs**, this can be done from the Rails console (`sudo gitlab-rails console`): + +1. Select jobs to be deleted: + + To select jobs with artifacts for a single project: + + ```ruby + project = Project.find_by_full_path('path/to/project') + builds_with_artifacts = project.builds.with_existing_job_artifacts + ``` + + To select jobs with artifacts across the entire GitLab instance: + + ```ruby + builds_with_artifacts = Ci::Build.with_existing_job_artifacts + ``` + +1. Select the user which will be mentioned in the web UI as erasing the job: + + ```ruby + admin_user = User.find_by(username: 'username') + ``` + +1. Erase job artifacts and logs older than a specific date: + + ```ruby + builds_to_clear = builds_with_artifacts.where("finished_at < ?", 1.week.ago) + builds_to_clear.find_each do |build| + print "Ci::Build ID #{build.id}... " + + if build.erasable? + build.erase(erased_by: admin_user) + puts "Erased" + else + puts "Skipped (Nothing to erase or not erasable)" + end + end + ``` + + `1.week.ago` is a Rails `ActiveSupport::Duration` method which calculates a new + date or time in the past. Other valid examples are: + + - `7.days.ago` + - `3.months.ago` + - `1.year.ago` diff --git a/doc/administration/job_logs.md b/doc/administration/job_logs.md index 6042786d101..fc37fbb170d 100644 --- a/doc/administration/job_logs.md +++ b/doc/administration/job_logs.md @@ -81,7 +81,7 @@ with one change: _the stored path of the first two phases is different_. This in log architecture stores chunks of logs in Redis and a persistent store (object storage or database) instead of file storage. Redis is used as first-class storage, and it stores up-to 128KB of data. Once the full chunk is sent, it is flushed to a persistent store, either object storage (temporary directory) or database. -After a while, the data in Redis and a persitent store will be archived to [object storage](#uploading-logs-to-object-storage). +After a while, the data in Redis and a persistent store will be archived to [object storage](#uploading-logs-to-object-storage). The data are stored in the following Redis namespace: `Gitlab::Redis::SharedState`. diff --git a/doc/administration/lfs/lfs_administration.md b/doc/administration/lfs/lfs_administration.md index f3b8029f487..fbf48619854 100644 --- a/doc/administration/lfs/lfs_administration.md +++ b/doc/administration/lfs/lfs_administration.md @@ -238,8 +238,8 @@ and [projects APIs](../../api/projects.md). ## Troubleshooting: `Google::Apis::TransmissionError: execution expired` -If LFS integration is configred with Google Cloud Storage and background uploads (`background_upload: true` and `direct_upload: false`), -Sidekiq workers may encouter this error. This is because the uploading timed out with very large files. +If LFS integration is configured with Google Cloud Storage and background uploads (`background_upload: true` and `direct_upload: false`), +Sidekiq workers may encounter this error. This is because the uploading timed out with very large files. LFS files up to 6Gb can be uploaded without any extra steps, otherwise you need to use the following workaround. ```shell diff --git a/doc/administration/logs.md b/doc/administration/logs.md index f4a1c754252..c69f787a5d8 100644 --- a/doc/administration/logs.md +++ b/doc/administration/logs.md @@ -96,7 +96,7 @@ request that have been performed and how much time it took. This task is more useful for GitLab contributors and developers. Use part of this log file when you are going to report bug. For example: -``` +```plaintext Started GET "/gitlabhq/yaml_db/tree/master" for 168.111.56.1 at 2015-02-12 19:34:53 +0200 Processing by Projects::TreeController#show as HTML Parameters: {"project_id"=>"gitlabhq/yaml_db", "id"=>"master"} @@ -151,7 +151,7 @@ installations from source. It helps you discover events happening in your instance such as user creation, project removing and so on. For example: -``` +```plaintext October 06, 2014 11:56: User "Administrator" (admin@example.com) was created October 06, 2014 11:56: Documentcloud created a new project "Documentcloud / Underscore" October 06, 2014 11:56: Gitlab Org created a new project "Gitlab Org / Gitlab Ce" @@ -167,7 +167,7 @@ installations from source. It contains information about [integrations](../user/project/integrations/project_services.md) activities such as Jira, Asana and Irker services. It uses JSON format like the example below: -``` json +```json {"severity":"ERROR","time":"2018-09-06T14:56:20.439Z","service_class":"JiraService","project_id":8,"project_path":"h5bp/html5-boilerplate","message":"Error sending message","client_url":"http://jira.gitlap.com:8080","error":"execution expired"} {"severity":"INFO","time":"2018-09-06T17:15:16.365Z","service_class":"JiraService","project_id":3,"project_path":"namespace2/project2","message":"Successfully posted","client_url":"http://jira.example.com"} ``` @@ -249,7 +249,7 @@ Instead of the format above, you can opt to generate JSON logs for Sidekiq. For example: ```json -{"severity":"INFO","time":"2018-04-03T22:57:22.071Z","queue":"cronjob:update_all_mirrors","args":[],"class":"UpdateAllMirrorsWorker","retry":false,"queue_namespace":"cronjob","jid":"06aeaa3b0aadacf9981f368e","created_at":"2018-04-03T22:57:21.930Z","enqueued_at":"2018-04-03T22:57:21.931Z","pid":10077,"message":"UpdateAllMirrorsWorker JID-06aeaa3b0aadacf9981f368e: done: 0.139 sec","job_status":"done","duration":0.139,"completed_at":"2018-04-03T22:57:22.071Z"} +{"severity":"INFO","time":"2018-04-03T22:57:22.071Z","queue":"cronjob:update_all_mirrors","args":[],"class":"UpdateAllMirrorsWorker","retry":false,"queue_namespace":"cronjob","jid":"06aeaa3b0aadacf9981f368e","created_at":"2018-04-03T22:57:21.930Z","enqueued_at":"2018-04-03T22:57:21.931Z","pid":10077,"message":"UpdateAllMirrorsWorker JID-06aeaa3b0aadacf9981f368e: done: 0.139 sec","job_status":"done","duration":0.139,"completed_at":"2018-04-03T22:57:22.071Z","db_duration":0.05,"db_duration_s":0.0005,"gitaly_duration":0,"gitaly_calls":0} ``` For Omnibus GitLab installations, add the configuration option: @@ -276,7 +276,7 @@ installations from source. GitLab Shell is used by GitLab for executing Git commands and provide SSH access to Git repositories. For example: -``` +```plaintext I, [2015-02-13T06:17:00.671315 #9291] INFO -- : Adding project root/example.git at </var/opt/gitlab/git-data/repositories/root/dcdcdcdcd.git>. I, [2015-02-13T06:17:00.679433 #9291] INFO -- : Moving existing hooks directory and symlinking global hooks directory for /var/opt/gitlab/git-data/repositories/root/example.git. ``` @@ -294,7 +294,7 @@ serving the GitLab application. You can look at this log if, for example, your application does not respond. This log contains all information about the state of Unicorn processes at any given time. -``` +```plaintext I, [2015-02-13T06:14:46.680381 #9047] INFO -- : Refreshing Gem list I, [2015-02-13T06:14:56.931002 #9047] INFO -- : listening on addr=127.0.0.1:8080 fd=12 I, [2015-02-13T06:14:56.931381 #9047] INFO -- : listening on addr=/var/opt/gitlab/gitlab-rails/sockets/gitlab.socket fd=13 @@ -421,6 +421,47 @@ etc. For example: {"severity":"DEBUG","time":"2019-10-17T06:23:13.227Z","correlation_id":null,"message":"redacted_search_result","class_name":"Milestone","id":2,"ability":"read_milestone","current_user_id":2,"query":"project"} ``` +## `exceptions_json.log` + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/merge_requests/17819) in GitLab 12.6. + +This file lives in +`/var/log/gitlab/gitlab-rails/exceptions_json.log` for Omnibus GitLab +packages or in `/home/git/gitlab/log/exceptions_json.log` for installations +from source. + +It logs the information about exceptions being tracked by `Gitlab::ErrorTracking` which provides standard and consistent way of [processing rescued exceptions](https://gitlab.com/gitlab-org/gitlab/blob/master/doc/development/logging.md#exception-handling). + +Each line contains a JSON line that can be ingested by Elasticsearch. For example: + +```json +{ + "severity": "ERROR", + "time": "2019-12-17T11:49:29.485Z", + "correlation_id": "AbDVUrrTvM1", + "extra.server": { + "os": { + "name": "Darwin", + "version": "Darwin Kernel Version 19.2.0", + "build": "19.2.0", + }, + "runtime": { + "name": "ruby", + "version": "ruby 2.6.5p114 (2019-10-01 revision 67812) [x86_64-darwin18]" + } + }, + "extra.project_id": 55, + "extra.relation_key": "milestones", + "extra.relation_index": 1, + "exception.class": "NoMethodError", + "exception.message": "undefined method `strong_memoize' for #<Gitlab::ImportExport::RelationFactory:0x00007fb5d917c4b0>", + "exception.backtrace": [ + "lib/gitlab/import_export/relation_factory.rb:329:in `unique_relation?'", + "lib/gitlab/import_export/relation_factory.rb:345:in `find_or_create_object!'" + ] +} +``` + [repocheck]: repository_checks.md [Rack Attack]: ../security/rack_attack.md [Rate Limit]: ../user/admin_area/settings/rate_limits_on_raw_endpoints.md diff --git a/doc/administration/monitoring/gitlab_instance_administration_project/index.md b/doc/administration/monitoring/gitlab_instance_administration_project/index.md index b07bbafaf7d..8675521ddb1 100644 --- a/doc/administration/monitoring/gitlab_instance_administration_project/index.md +++ b/doc/administration/monitoring/gitlab_instance_administration_project/index.md @@ -1,7 +1,9 @@ # GitLab instance administration project NOTE: **Note:** -This feature is not yet available and is [planned for 12.6](https://gitlab.com/gitlab-org/gitlab/issues/32351). +This feature is available behind a feature flag called `self_monitoring_project` +since [12.7](https://gitlab.com/gitlab-org/gitlab/issues/32351). The feature flag +will be removed once we [add dashboards to display metrics](https://gitlab.com/groups/gitlab-org/-/epics/2367). GitLab has been adding the ability for administrators to see insights into the health of their GitLab instance. In order to surface this experience in a native way, similar to how diff --git a/doc/administration/monitoring/performance/gitlab_configuration.md b/doc/administration/monitoring/performance/gitlab_configuration.md index 9a25cc04ee7..1bff170768a 100644 --- a/doc/administration/monitoring/performance/gitlab_configuration.md +++ b/doc/administration/monitoring/performance/gitlab_configuration.md @@ -1,8 +1,12 @@ # GitLab Configuration +CAUTION: **InfluxDB is deprecated in favor of Prometheus:** +InfluxDB support is scheduled to be removed in GitLab 13.0. +You are advised to use [Prometheus](../prometheus/index.md) instead. + GitLab Performance Monitoring is disabled by default. To enable it and change any of its -settings, navigate to the Admin area in **Settings > Metrics** -(`/admin/application_settings`). +settings, navigate to **Admin Area > Settings > Metrics and profiling** +(`/admin/application_settings/metrics_and_profiling`). The minimum required settings you need to set are the InfluxDB host and port. Make sure _Enable InfluxDB Metrics_ is checked and hit **Save** to save the @@ -28,7 +32,7 @@ have been performed. Read more on: -- [Introduction to GitLab Performance Monitoring](introduction.md) +- [Introduction to GitLab Performance Monitoring](index.md) - [InfluxDB Configuration](influxdb_configuration.md) - [InfluxDB Schema](influxdb_schema.md) - [Grafana Install/Configuration](grafana_configuration.md) diff --git a/doc/administration/monitoring/performance/grafana_configuration.md b/doc/administration/monitoring/performance/grafana_configuration.md index ccba0a55479..2fbbeb0b774 100644 --- a/doc/administration/monitoring/performance/grafana_configuration.md +++ b/doc/administration/monitoring/performance/grafana_configuration.md @@ -1,5 +1,9 @@ # Grafana Configuration +CAUTION: **InfluxDB is deprecated in favor of Prometheus:** +InfluxDB support is scheduled to be removed in GitLab 13.0. +You are advised to use [Prometheus](../prometheus/index.md) instead. + [Grafana](https://grafana.com/) is a tool that allows you to visualize time series metrics through graphs and dashboards. It supports several backend data stores, including InfluxDB. GitLab writes performance data to InfluxDB @@ -53,14 +57,14 @@ repository. To use this repository you must first clone it: -``` +```shell git clone https://gitlab.com/gitlab-org/influxdb-management.git cd influxdb-management ``` Next you must install the required dependencies: -``` +```shell gem install bundler bundle install ``` @@ -109,14 +113,14 @@ repository for more information on this process. If you have set up Grafana, you can enable a link to access it easily from the sidebar: -1. Go to the admin area under **Settings > Metrics and profiling** - and expand "Metrics - Grafana". +1. Go to the **Admin Area > Settings > Metrics and profiling**. +1. Expand **Metrics - Grafana**. 1. Check the "Enable access to Grafana" checkbox. 1. If Grafana is enabled through Omnibus GitLab and on the same server, leave "Grafana URL" unchanged. In any other case, enter the full URL path of the Grafana instance. 1. Click **Save changes**. -1. The new link will be available in the admin area under **Monitoring > Metrics Dashboard**. +1. The new link will be available in the **Admin Area > Monitoring > Metrics Dashboard**. ## Security Update @@ -135,7 +139,7 @@ echo "0" > /var/opt/gitlab/grafana/CVE_reset_status To reinstate your old data, move it back into its original location: -``` +```shell sudo mv /var/opt/gitlab/grafana/data.bak.xxxx/ /var/opt/gitlab/grafana/data/ ``` @@ -152,7 +156,7 @@ For more information and further mitigation details, please refer to our [blog p Read more on: -- [Introduction to GitLab Performance Monitoring](introduction.md) +- [Introduction to GitLab Performance Monitoring](index.md) - [GitLab Configuration](gitlab_configuration.md) - [InfluxDB Installation/Configuration](influxdb_configuration.md) - [InfluxDB Schema](influxdb_schema.md) diff --git a/doc/administration/monitoring/performance/img/performance_bar.png b/doc/administration/monitoring/performance/img/performance_bar.png Binary files differindex d206d5a4268..be06e0b2f94 100644 --- a/doc/administration/monitoring/performance/img/performance_bar.png +++ b/doc/administration/monitoring/performance/img/performance_bar.png diff --git a/doc/administration/monitoring/performance/img/performance_bar_frontend.png b/doc/administration/monitoring/performance/img/performance_bar_frontend.png Binary files differindex 489f855fe33..32a241e032b 100644 --- a/doc/administration/monitoring/performance/img/performance_bar_frontend.png +++ b/doc/administration/monitoring/performance/img/performance_bar_frontend.png diff --git a/doc/administration/monitoring/performance/img/performance_bar_gitaly_threshold.png b/doc/administration/monitoring/performance/img/performance_bar_gitaly_threshold.png Binary files differindex d4bf5c69ffa..4e42d904cdf 100644 --- a/doc/administration/monitoring/performance/img/performance_bar_gitaly_threshold.png +++ b/doc/administration/monitoring/performance/img/performance_bar_gitaly_threshold.png diff --git a/doc/administration/monitoring/performance/img/performance_bar_request_selector_warning.png b/doc/administration/monitoring/performance/img/performance_bar_request_selector_warning.png Binary files differindex 966549554a4..74711387ffe 100644 --- a/doc/administration/monitoring/performance/img/performance_bar_request_selector_warning.png +++ b/doc/administration/monitoring/performance/img/performance_bar_request_selector_warning.png diff --git a/doc/administration/monitoring/performance/img/performance_bar_request_selector_warning_expanded.png b/doc/administration/monitoring/performance/img/performance_bar_request_selector_warning_expanded.png Binary files differindex 3362186bb48..36553f513e1 100644 --- a/doc/administration/monitoring/performance/img/performance_bar_request_selector_warning_expanded.png +++ b/doc/administration/monitoring/performance/img/performance_bar_request_selector_warning_expanded.png diff --git a/doc/administration/monitoring/performance/index.md b/doc/administration/monitoring/performance/index.md index 5204ab40dc9..6569f6a8c6d 100644 --- a/doc/administration/monitoring/performance/index.md +++ b/doc/administration/monitoring/performance/index.md @@ -1,5 +1,9 @@ # GitLab Performance Monitoring +CAUTION: **InfluxDB is deprecated in favor of Prometheus:** +InfluxDB support is scheduled to be removed in GitLab 13.0. +You are advised to use [Prometheus](../prometheus/index.md) instead. + GitLab comes with its own application performance measuring system as of GitLab 8.4, simply called "GitLab Performance Monitoring". GitLab Performance Monitoring is available in both the Community and Enterprise editions. diff --git a/doc/administration/monitoring/performance/influxdb_configuration.md b/doc/administration/monitoring/performance/influxdb_configuration.md index f1f588a924d..b18be09ef4b 100644 --- a/doc/administration/monitoring/performance/influxdb_configuration.md +++ b/doc/administration/monitoring/performance/influxdb_configuration.md @@ -1,5 +1,9 @@ # InfluxDB Configuration +CAUTION: **InfluxDB is deprecated in favor of Prometheus:** +InfluxDB support is scheduled to be removed in GitLab 13.0. +You are advised to use [Prometheus](../prometheus/index.md) instead. + The default settings provided by [InfluxDB] are not sufficient for a high traffic GitLab environment. The settings discussed in this document are based on the settings GitLab uses for GitLab.com, depending on your own needs you may need to @@ -44,7 +48,7 @@ upcoming InfluxDB releases. Make sure you have the following in your configuration file: -``` +```toml [data] dir = "/var/lib/influxdb/data" engine = "tsm1" @@ -56,7 +60,7 @@ Production environments should have the InfluxDB admin panel **disabled**. This feature can be disabled by adding the following to your InfluxDB configuration file: -``` +```toml [admin] enabled = false ``` @@ -67,7 +71,7 @@ HTTP is required when using the [InfluxDB CLI] or other tools such as Grafana, thus it should be enabled. When enabling make sure to _also_ enable authentication: -``` +```toml [http] enabled = true auth-enabled = true @@ -81,7 +85,7 @@ admin user](#create-a-new-admin-user)._ GitLab writes data to InfluxDB via UDP and thus this must be enabled. Enabling UDP can be done using the following settings: -``` +```toml [[udp]] enabled = true bind-address = ":8089" @@ -134,7 +138,7 @@ allowing traffic from members of said VLAN. If you want to [enable authentication](#http), you might want to [create an admin user][influx-admin]: -``` +```shell influx -execute "CREATE USER jeff WITH PASSWORD '1234' WITH ALL PRIVILEGES" ``` @@ -164,7 +168,7 @@ influx -execute 'SHOW DATABASES' The output should be similar to: -``` +```plaintext name: databases --------------- name @@ -178,7 +182,7 @@ That's it! Now your GitLab instance should send data to InfluxDB. Read more on: -- [Introduction to GitLab Performance Monitoring](introduction.md) +- [Introduction to GitLab Performance Monitoring](index.md) - [GitLab Configuration](gitlab_configuration.md) - [InfluxDB Schema](influxdb_schema.md) - [Grafana Install/Configuration](grafana_configuration.md) diff --git a/doc/administration/monitoring/performance/influxdb_schema.md b/doc/administration/monitoring/performance/influxdb_schema.md index eff0e29f58d..adbccdaaeb8 100644 --- a/doc/administration/monitoring/performance/influxdb_schema.md +++ b/doc/administration/monitoring/performance/influxdb_schema.md @@ -1,5 +1,9 @@ # InfluxDB Schema +CAUTION: **InfluxDB is deprecated in favor of Prometheus:** +InfluxDB support is scheduled to be removed in GitLab 13.0. +You are advised to use [Prometheus](../prometheus/index.md) instead. + The following measurements are currently stored in InfluxDB: - `PROCESS_file_descriptors` @@ -39,7 +43,7 @@ while the method name is stored in the tag `method`. The tag `action` contains the full name of the transaction action. Both the `method` and `action` fields are in the following format: -``` +```plaintext ClassName#method_name ``` @@ -91,7 +95,7 @@ Depending on the event type additional tags may be available as well. Read more on: -- [Introduction to GitLab Performance Monitoring](introduction.md) +- [Introduction to GitLab Performance Monitoring](index.md) - [GitLab Configuration](gitlab_configuration.md) - [InfluxDB Configuration](influxdb_configuration.md) - [Grafana Install/Configuration](grafana_configuration.md) diff --git a/doc/administration/monitoring/performance/performance_bar.md b/doc/administration/monitoring/performance/performance_bar.md index 98c611ea140..fe4c29fbb01 100644 --- a/doc/administration/monitoring/performance/performance_bar.md +++ b/doc/administration/monitoring/performance/performance_bar.md @@ -52,8 +52,9 @@ And requests with warnings are indicated in the request selector with a ## Enable the Performance Bar via the Admin panel GitLab Performance Bar is disabled by default. To enable it for a given group, -navigate to the Admin area in **Settings > Metrics and Profiling > Profiling - Performance bar** -(`admin/application_settings/metrics_and_profiling`). +navigate to **Admin Area > Settings > Metrics and profiling** +(`admin/application_settings/metrics_and_profiling`), and expand the section +**Profiling - Performance bar**. The only required setting you need to set is the full path of the group that will be allowed to display the Performance Bar. diff --git a/doc/administration/monitoring/prometheus/gitlab_metrics.md b/doc/administration/monitoring/prometheus/gitlab_metrics.md index 57048059476..f3da5a6dd2f 100644 --- a/doc/administration/monitoring/prometheus/gitlab_metrics.md +++ b/doc/administration/monitoring/prometheus/gitlab_metrics.md @@ -6,8 +6,8 @@ installations from source you'll have to configure it yourself. To enable the GitLab Prometheus metrics: -1. Log into GitLab as an administrator, and go to the Admin area. -1. Navigate to GitLab's **Settings > Metrics and profiling**. +1. Log into GitLab as an administrator. +1. Navigate to **Admin Area > Settings > Metrics and profiling**. 1. Find the **Metrics - Prometheus** section, and click **Enable Prometheus Metrics**. 1. [Restart GitLab](../../restart_gitlab.md#omnibus-gitlab-restart) for the changes to take effect. @@ -26,7 +26,7 @@ The following metrics are available: | Metric | Type | Since | Description | Labels | |:---------------------------------------------------------------|:----------|-----------------------:|:----------------------------------------------------------------------------------------------------|:----------------------------------------------------| | `gitlab_banzai_cached_render_real_duration_seconds` | Histogram | 9.4 | Duration of rendering Markdown into HTML when cached output exists | controller, action | -| `gitlab_banzai_cacheless_render_real_duration_seconds` | Histogram | 9.4 | Duration of rendering Markdown into HTML when cached outupt does not exist | controller, action | +| `gitlab_banzai_cacheless_render_real_duration_seconds` | Histogram | 9.4 | Duration of rendering Markdown into HTML when cached output does not exist | controller, action | | `gitlab_cache_misses_total` | Counter | 10.2 | Cache read miss | controller, action | | `gitlab_cache_operation_duration_seconds` | Histogram | 10.2 | Cache access time | | | `gitlab_cache_operations_total` | Counter | 12.2 | Cache operations by controller/action | controller, action, operation | @@ -59,7 +59,7 @@ The following metrics are available: | `gitlab_transaction_event_push_commit_total` | Counter | 9.4 | Counter for commits | branch | | `gitlab_transaction_event_push_tag_total` | Counter | 9.4 | Counter for tag pushes | | | `gitlab_transaction_event_rails_exception_total` | Counter | 9.4 | Counter for number of rails exceptions | | -| `gitlab_transaction_event_receive_email_total` | Counter | 9.4 | Counter for recieved emails | handler | +| `gitlab_transaction_event_receive_email_total` | Counter | 9.4 | Counter for received emails | handler | | `gitlab_transaction_event_remote_mirrors_failed_total` | Counter | 10.8 | Counter for failed remote mirrors | | | `gitlab_transaction_event_remote_mirrors_finished_total` | Counter | 10.8 | Counter for finished remote mirrors | | | `gitlab_transaction_event_remote_mirrors_running_total` | Counter | 10.8 | Counter for running remote mirrors | | @@ -154,10 +154,10 @@ Some basic Ruby runtime metrics are available: | `ruby_sampler_duration_seconds` | Counter | 11.1 | Time spent collecting stats | | `ruby_process_cpu_seconds_total` | Gauge | 12.0 | Total amount of CPU time per process | | `ruby_process_max_fds` | Gauge | 12.0 | Maximum number of open file descriptors per process | -| `ruby_process_resident_memory_bytes` | Gauge | 12.0 | Memory usage by process, measured in bytes | +| `ruby_process_resident_memory_bytes` | Gauge | 12.0 | Memory usage by process | | `ruby_process_start_time_seconds` | Gauge | 12.0 | UNIX timestamp of process start time | -[GC.stat]: https://ruby-doc.org/core-2.6.3/GC.html#method-c-stat +[GC.stat]: https://ruby-doc.org/core-2.6.5/GC.html#method-c-stat ## Unicorn Metrics diff --git a/doc/administration/monitoring/prometheus/index.md b/doc/administration/monitoring/prometheus/index.md index eb7a2d791c1..62bacf9791e 100644 --- a/doc/administration/monitoring/prometheus/index.md +++ b/doc/administration/monitoring/prometheus/index.md @@ -245,7 +245,7 @@ To add a Prometheus dashboard for a single server GitLab setup: 1. Create a new data source in Grafana. 1. Name your data source i.e GitLab. -1. Select `Prometheus` in the type drop down. +1. Select `Prometheus` in the type dropdown box. 1. Add your Prometheus listen address as the URL and set access to `Browser`. 1. Set the HTTP method to `GET`. 1. Save & Test your configuration to verify that it works. diff --git a/doc/administration/operations/cleaning_up_redis_sessions.md b/doc/administration/operations/cleaning_up_redis_sessions.md index fd469ae23e3..38fac8a0eca 100644 --- a/doc/administration/operations/cleaning_up_redis_sessions.md +++ b/doc/administration/operations/cleaning_up_redis_sessions.md @@ -22,7 +22,7 @@ settings outlined in First we define a shell function with the proper Redis connection details. -``` +```shell rcli() { # This example works for Omnibus installations of GitLab 7.3 or newer. For an # installation from source you will have to change the socket path and the @@ -37,7 +37,7 @@ rcli ping Now we do a search to see if there are any session keys in the old format for us to clean up. -``` +```shell # returns the number of old-format session keys in Redis rcli keys '*' | grep '^[a-f0-9]\{32\}$' | wc -l ``` @@ -45,7 +45,7 @@ rcli keys '*' | grep '^[a-f0-9]\{32\}$' | wc -l If the number is larger than zero, you can proceed to expire the keys from Redis. If the number is zero there is nothing to clean up. -``` +```shell # Tell Redis to expire each matched key after 600 seconds. rcli keys '*' | grep '^[a-f0-9]\{32\}$' | awk '{ print "expire", $0, 600 }' | rcli # This will print '(integer) 1' for each key that gets expired. diff --git a/doc/administration/operations/extra_sidekiq_processes.md b/doc/administration/operations/extra_sidekiq_processes.md index e15f91ebab2..1be89f759da 100644 --- a/doc/administration/operations/extra_sidekiq_processes.md +++ b/doc/administration/operations/extra_sidekiq_processes.md @@ -59,8 +59,8 @@ To start extra Sidekiq processes, you must enable `sidekiq-cluster`: sudo gitlab-ctl reconfigure ``` -Once the extra Sidekiq processes are added, you can visit the "Background Jobs" -section under the admin area in GitLab (`/admin/background_jobs`). +Once the extra Sidekiq processes are added, you can visit the +**Admin Area > Monitoring > Background Jobs** (`/admin/background_jobs`) in GitLab.  diff --git a/doc/administration/operations/fast_ssh_key_lookup.md b/doc/administration/operations/fast_ssh_key_lookup.md index 9a38e8ddd23..96571b0a5d9 100644 --- a/doc/administration/operations/fast_ssh_key_lookup.md +++ b/doc/administration/operations/fast_ssh_key_lookup.md @@ -53,7 +53,7 @@ Add the following to your `sshd_config` file. This is usually located at `/etc/ssh/sshd_config`, but it will be `/assets/sshd_config` if you're using Omnibus Docker: -``` +```plaintext AuthorizedKeysCommand /opt/gitlab/embedded/service/gitlab-shell/bin/gitlab-shell-authorized-keys-check git %u %k AuthorizedKeysCommandUser git ``` @@ -117,7 +117,7 @@ the database. The following instructions can be used to build OpenSSH 7.5: 1. First, download the package and install the required packages: - ``` + ```shell sudo su - cd /tmp curl --remote-name https://mirrors.evowise.com/pub/OpenBSD/OpenSSH/portable/openssh-7.5p1.tar.gz @@ -127,7 +127,7 @@ the database. The following instructions can be used to build OpenSSH 7.5: 1. Prepare the build by copying files to the right place: - ``` + ```shell mkdir -p /root/rpmbuild/{SOURCES,SPECS} cp ./openssh-7.5p1/contrib/redhat/openssh.spec /root/rpmbuild/SPECS/ cp openssh-7.5p1.tar.gz /root/rpmbuild/SOURCES/ @@ -136,7 +136,7 @@ the database. The following instructions can be used to build OpenSSH 7.5: 1. Next, set the spec settings properly: - ``` + ```shell sed -i -e "s/%define no_gnome_askpass 0/%define no_gnome_askpass 1/g" openssh.spec sed -i -e "s/%define no_x11_askpass 0/%define no_x11_askpass 1/g" openssh.spec sed -i -e "s/BuildPreReq/BuildRequires/g" openssh.spec @@ -144,19 +144,19 @@ the database. The following instructions can be used to build OpenSSH 7.5: 1. Build the RPMs: - ``` + ```shell rpmbuild -bb openssh.spec ``` 1. Ensure the RPMs were built: - ``` + ```shell ls -al /root/rpmbuild/RPMS/x86_64/ ``` You should see something as the following: - ``` + ```plaintext total 1324 drwxr-xr-x. 2 root root 4096 Jun 20 19:37 . drwxr-xr-x. 3 root root 19 Jun 20 19:37 .. @@ -170,7 +170,7 @@ the database. The following instructions can be used to build OpenSSH 7.5: with its own version, which may prevent users from logging in, so be sure that the file is backed up and restored after installation: - ``` + ```shell timestamp=$(date +%s) cp /etc/pam.d/sshd pam-ssh-conf-$timestamp rpm -Uvh /root/rpmbuild/RPMS/x86_64/*.rpm @@ -179,7 +179,7 @@ the database. The following instructions can be used to build OpenSSH 7.5: 1. Verify the installed version. In another window, attempt to login to the server: - ``` + ```shell ssh -v <your-centos-machine> ``` @@ -191,7 +191,7 @@ the database. The following instructions can be used to build OpenSSH 7.5: sure everything is working! If you need to downgrade, simple install the older package: - ``` + ```shell # Only run this if you run into a problem logging in yum downgrade openssh-server openssh openssh-clients ``` diff --git a/doc/administration/operations/puma.md b/doc/administration/operations/puma.md new file mode 100644 index 00000000000..2490cf1f0ae --- /dev/null +++ b/doc/administration/operations/puma.md @@ -0,0 +1,46 @@ +# Switching to Puma + +## Puma + +GitLab plans to use [Puma](https://github.com/puma/puma) to replace +[Unicorn](https://bogomips.org/unicorn/). + +## Why switch to Puma? + +Puma has a multi-thread architecture which uses less memory than a multi-process +application server like Unicorn. + +Most Rails applications requests normally include a proportion of I/O wait time. +During I/O wait time MRI Ruby will release the GVL (Global VM Lock) to other threads. +Multi-threaded Puma can therefore still serve more requests than a single process. + +## Performance caveat when using Puma with Rugged + +For deployments where NFS is used to store Git repository, we allow GitLab to use +[Direct Git Access](../gitaly/#direct-git-access-in-gitlab-rails) to improve performance via usage of [Rugged](https://github.com/libgit2/rugged). + +Rugged usage is automatically enabled if Direct Git Access is present, unless it +is disabled by [feature flags](../../development/gitaly.md#legacy-rugged-code). + +MRI Ruby uses a GVL. This allows MRI Ruby to be multi-threaded, but running at +most on a single core. Since Rugged can use a thread for long periods of +time (due to intensive I/O operations of Git access), this can starve other threads +that might be processing requests. This is not a case for Unicorn or Puma running +in a single thread mode, as concurrently at most one request is being processed. + +We are actively working on removing Rugged usage. Even though performance without Rugged +is acceptable today, in some cases it might be still beneficial to run with it. + +Given the caveat of running Rugged with multi-threaded Puma, and acceptable +performance of Gitaly, we are disabling Rugged usage if Puma multi-threaded is +used (when Puma is configured to run with more than one thread). + +This default behavior may not be the optimal configuration in some situations. If Rugged +plays an important role in your deployment, we suggest you benchmark to find the +optimal configuration: + +- The safest option is to start with single-threaded Puma. When working with +Rugged, single-threaded Puma does work the same as Unicorn. + +- To force Rugged auto detect with multi-threaded Puma, you can use [feature +flags](../../development/gitaly.md#legacy-rugged-code). diff --git a/doc/administration/packages/container_registry.md b/doc/administration/packages/container_registry.md index e735d8dd97e..6ef1a3ec607 100644 --- a/doc/administration/packages/container_registry.md +++ b/doc/administration/packages/container_registry.md @@ -98,7 +98,7 @@ There are two ways you can configure the Registry's external domain. Either: for that domain. Since the container Registry requires a TLS certificate, in the end it all boils -down to how easy or pricey is to get a new one. +down to how easy or pricey it is to get a new one. Please take this into consideration before configuring the Container Registry for the first time. @@ -398,6 +398,9 @@ To configure the `s3` storage driver in Omnibus: 1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. +NOTE: **Note:** +`your-s3-bucket` should only be the name of a bucket that exists, and can't include subdirectories. + **Installations from source** Configuring the storage driver is done in your registry config YML file created @@ -408,9 +411,9 @@ when you [deployed your docker registry](https://docs.docker.com/registry/deploy ```yml storage: s3: - accesskey: 'AKIAKIAKI' - secretkey: 'secret123' - bucket: 'gitlab-registry-bucket-AKIAKIAKI' + accesskey: 's3-access-key' + secretkey: 's3-secret-key-for-access-key' + bucket: 'your-s3-bucket' region: 'your-s3-region' regionendpoint: 'your-s3-regionendpoint' cache: @@ -419,6 +422,9 @@ storage: enabled: true ``` +NOTE: **Note:** +`your-s3-bucket` should only be the name of a bucket that exists, and can't include subdirectories. + ## Change the registry's internal port NOTE: **Note:** @@ -625,13 +631,36 @@ mounting the docker-daemon and setting `privileged = false` in the Runner's ```toml [runners.docker] - image = "ruby:2.1" + image = "ruby:2.6" privileged = false volumes = ["/var/run/docker.sock:/var/run/docker.sock", "/cache"] ``` Additional information about this: [issue 18239](https://gitlab.com/gitlab-org/gitlab-foss/issues/18239). +### `unauthorized: authentication required` when pushing large images + +Example error: + +```shell +docker push gitlab.example.com/myproject/docs:latest +The push refers to a repository [gitlab.example.com/myproject/docs] +630816f32edb: Preparing +530d5553aec8: Preparing +... +4b0bab9ff599: Waiting +d1c800db26c7: Waiting +42755cf4ee95: Waiting +unauthorized: authentication required +``` + +GitLab has a default token expiration of 5 minutes for the registry. When pushing +larger images, or images that take longer than 5 minutes to push, users may +encounter this error. + +Administrators can increase the token duration in **Admin area > Settings > +Container Registry > Authorization token duration (minutes)**. + ### AWS S3 with the GitLab registry error when pushing large images When using AWS S3 with the GitLab registry, an error may occur when pushing diff --git a/doc/administration/packages/index.md b/doc/administration/packages/index.md index 58dd8201c15..432e72e03e3 100644 --- a/doc/administration/packages/index.md +++ b/doc/administration/packages/index.md @@ -8,6 +8,7 @@ The Packages feature allows GitLab to act as a repository for the following: | Software repository | Description | Available in GitLab version | | ------------------- | ----------- | --------------------------- | +| [NuGet Repository](../../user/packages/nuget_repository/index.md) | The GitLab NuGet Repository enables every project in GitLab to have its own space to store [NuGet](https://www.nuget.org/) packages. | 12.8+ | | [Conan Repository](../../user/packages/conan_repository/index.md) | The GitLab Conan Repository enables every project in GitLab to have its own space to store [Conan](https://conan.io/) packages. | 12.4+ | | [Maven Repository](../../user/packages/maven_repository/index.md) | The GitLab Maven Repository enables every project in GitLab to have its own space to store [Maven](https://maven.apache.org/) packages. | 11.3+ | | [NPM Registry](../../user/packages/npm_registry/index.md) | The GitLab NPM Registry enables every project in GitLab to have its own space to store [NPM](https://www.npmjs.com/) packages. | 11.7+ | diff --git a/doc/administration/pages/index.md b/doc/administration/pages/index.md index d1b58f2ee18..434cb2447c8 100644 --- a/doc/administration/pages/index.md +++ b/doc/administration/pages/index.md @@ -32,11 +32,11 @@ In the case of [custom domains](#custom-domains) (but not ports `80` and/or `443`. For that reason, there is some flexibility in the way which you can set it up: -1. Run the Pages daemon in the same server as GitLab, listening on a secondary IP. -1. Run the Pages daemon in a separate server. In that case, the +- Run the Pages daemon in the same server as GitLab, listening on a **secondary IP**. +- Run the Pages daemon in a [separate server](#running-gitlab-pages-on-a-separate-server). In that case, the [Pages path](#change-storage-path) must also be present in the server that the Pages daemon is installed, so you will have to share it via network. -1. Run the Pages daemon in the same server as GitLab, listening on the same IP +- Run the Pages daemon in the same server as GitLab, listening on the same IP but on different ports. In that case, you will have to proxy the traffic with a loadbalancer. If you choose that route note that you should use TCP load balancing for HTTPS. If you use TLS-termination (HTTPS-load balancing) the @@ -182,7 +182,7 @@ The [GitLab Pages README](https://gitlab.com/gitlab-org/gitlab-pages#caveats) ha In addition to the wildcard domains, you can also have the option to configure GitLab Pages to work with custom domains. Again, there are two options here: support custom domains with and without TLS certificates. The easiest setup is -that without TLS certificates. In either case, you'll need a secondary IP. If +that without TLS certificates. In either case, you'll need a **secondary IP**. If you have IPv6 as well as IPv4 addresses, you can use them both. ### Custom domains @@ -257,8 +257,8 @@ When adding a custom domain, users will be required to prove they own it by adding a GitLab-controlled verification code to the DNS records for that domain. If your userbase is private or otherwise trusted, you can disable the -verification requirement. Navigate to `Admin area ➔ Settings` and uncheck -**Require users to prove ownership of custom domains** in the Pages section. +verification requirement. Navigate to **Admin Area > Settings > Preferences** and +uncheck **Require users to prove ownership of custom domains** in the **Pages** section. This setting is enabled by default. ### Let's Encrypt integration @@ -307,6 +307,27 @@ Pages access control is disabled by default. To enable it: 1. [Reconfigure GitLab][reconfigure]. 1. Users can now configure it in their [projects' settings](../../user/project/pages/pages_access_control.md). +#### Disabling public access to all Pages websites + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/issues/32095) in GitLab 12.7. + +You can enforce [Access Control](#access-control) for all GitLab Pages websites hosted +on your GitLab instance. By doing so, only logged-in users will have access to them. +This setting overrides Access Control set by users in individual projects. + +This can be useful to preserve information published with Pages websites to the users +of your instance only. +To do that: + +1. Navigate to your instance's **Admin Area > Settings > Preferences** and expand **Pages** settings. +1. Check the **Disable public access to Pages sites** checkbox. +1. Click **Save changes**. + +CAUTION: **Warning:** +This action will not make all currently public web-sites private until they redeployed. +This issue among others will be resolved by +[changing GitLab Pages configuration mechanism](https://gitlab.com/gitlab-org/gitlab-pages/issues/282). + ### Running behind a proxy Like the rest of GitLab, Pages can be used in those environments where external @@ -395,10 +416,26 @@ Omnibus GitLab 11.1. ## Set maximum pages size -The maximum size of the unpacked archive per project can be configured in the -Admin area under the Application settings in the **Maximum size of pages (MB)**. +You can configure the maximum size of the unpacked archive per project in +**Admin Area > Settings > Preferences > Pages**, in **Maximum size of pages (MB)**. The default is 100MB. +### Override maximum pages size per project or group **(PREMIUM ONLY)** + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/issues/16610) in GitLab 12.7. + +To override the global maximum pages size for a specific project: + +1. Navigate to your project's **Settings > Pages** page. +1. Edit the **Maximum size of pages**. +1. Click **Save changes**. + +To override the global maximum pages size for a specific group: + +1. Navigate to your group's **Settings > General** page and expand **Pages**. +1. Edit the **Maximum size of pages**. +1. Click **Save changes**. + ## Running GitLab Pages on a separate server You can run the GitLab Pages daemon on a separate server in order to decrease the load on your main application server. diff --git a/doc/administration/pages/source.md b/doc/administration/pages/source.md index be8bba3c95b..738eb87d53d 100644 --- a/doc/administration/pages/source.md +++ b/doc/administration/pages/source.md @@ -433,8 +433,8 @@ are stored. ## Set maximum Pages size -The maximum size of the unpacked archive per project can be configured in the -Admin area under the Application settings in the **Maximum size of pages (MB)**. +The maximum size of the unpacked archive per project can be configured in +**Admin Area > Settings > Preferences > Pages**, in **Maximum size of pages (MB)**. The default is 100MB. ## Backup diff --git a/doc/administration/plugins.md b/doc/administration/plugins.md index df75d3a24bc..dbb733b9b19 100644 --- a/doc/administration/plugins.md +++ b/doc/administration/plugins.md @@ -1,115 +1,5 @@ -# GitLab Plugin system +--- +redirect_to: 'file_hooks.md' +--- -> Introduced in GitLab 10.6. - -With custom plugins, GitLab administrators can introduce custom integrations -without modifying GitLab's source code. - -NOTE: **Note:** -Instead of writing and supporting your own plugin you can make changes -directly to the GitLab source code and contribute back upstream. This way we can -ensure functionality is preserved across versions and covered by tests. - -NOTE: **Note:** -Plugins must be configured on the filesystem of the GitLab server. Only GitLab -server administrators will be able to complete these tasks. Explore -[system hooks] or [webhooks] as an option if you do not have filesystem access. - -A plugin will run on each event so it's up to you to filter events or projects -within a plugin code. You can have as many plugins as you want. Each plugin will -be triggered by GitLab asynchronously in case of an event. For a list of events -see the [system hooks] documentation. - -## Setup - -The plugins must be placed directly into the `plugins` directory, subdirectories -will be ignored. There is an -[`example` directory inside `plugins`](https://gitlab.com/gitlab-org/gitlab-foss/tree/master/plugins/examples) -where you can find some basic examples. - -Follow the steps below to set up a custom hook: - -1. On the GitLab server, navigate to the plugin directory. - For an installation from source the path is usually - `/home/git/gitlab/plugins/`. For Omnibus installs the path is - usually `/opt/gitlab/embedded/service/gitlab-rails/plugins`. - - For [highly available] configurations, your hook file should exist on each - application server. - -1. Inside the `plugins` directory, create a file with a name of your choice, - without spaces or special characters. -1. Make the hook file executable and make sure it's owned by the Git user. -1. Write the code to make the plugin function as expected. That can be - in any language, and ensure the 'shebang' at the top properly reflects the - language type. For example, if the script is in Ruby the shebang will - probably be `#!/usr/bin/env ruby`. -1. The data to the plugin will be provided as JSON on STDIN. It will be exactly - same as for [system hooks] - -That's it! Assuming the plugin code is properly implemented, the hook will fire -as appropriate. The plugins file list is updated for each event, there is no -need to restart GitLab to apply a new plugin. - -If a plugin executes with non-zero exit code or GitLab fails to execute it, a -message will be logged to: - -- `gitlab-rails/plugin.log` in an Omnibus installation. -- `log/plugin.log` in a source installation. - -## Creating plugins - -Below is an example that will only response on the event `project_create` and -will inform the admins from the GitLab instance that a new project has been created. - -```ruby -# By using the embedded ruby version we eliminate the possibility that our chosen language -# would be unavailable from -#!/opt/gitlab/embedded/bin/ruby -require 'json' -require 'mail' - -# The incoming variables are in JSON format so we need to parse it first. -ARGS = JSON.parse(STDIN.read) - -# We only want to trigger this plugin on the event project_create -return unless ARGS['event_name'] == 'project_create' - -# We will inform our admins of our gitlab instance that a new project is created -Mail.deliver do - from 'info@gitlab_instance.com' - to 'admin@gitlab_instance.com' - subject "new project " + ARGS['name'] - body ARGS['owner_name'] + 'created project ' + ARGS['name'] -end -``` - -## Validation - -Writing your own plugin can be tricky and it's easier if you can check it -without altering the system. A rake task is provided so that you can use it -in a staging environment to test your plugin before using it in production. -The rake task will use a sample data and execute each of plugin. The output -should be enough to determine if the system sees your plugin and if it was -executed without errors. - -```bash -# Omnibus installations -sudo gitlab-rake plugins:validate - -# Installations from source -cd /home/git/gitlab -bundle exec rake plugins:validate RAILS_ENV=production -``` - -Example of output: - -``` -Validating plugins from /plugins directory -* /home/git/gitlab/plugins/save_to_file.clj succeed (zero exit code) -* /home/git/gitlab/plugins/save_to_file.rb failure (non-zero exit code) -``` - -[system hooks]: ../system_hooks/system_hooks.md -[webhooks]: ../user/project/integrations/webhooks.md -[highly available]: ./high_availability/README.md +This document was moved to [File Hooks](file_hooks.md), after the Plugins feature was renamed to File Hooks. diff --git a/doc/administration/raketasks/uploads/migrate.md b/doc/administration/raketasks/uploads/migrate.md index 26c811ca54d..aef15e3f388 100644 --- a/doc/administration/raketasks/uploads/migrate.md +++ b/doc/administration/raketasks/uploads/migrate.md @@ -1,4 +1,4 @@ -# Uploads Migrate Rake Task +# Uploads Migrate Rake Tasks ## Migrate to Object Storage @@ -110,7 +110,15 @@ sudo -u git -H bundle exec rake "gitlab:uploads:migrate[FileUploader, MergeReque To migrate all uploads created by legacy uploaders, run: -```shell +**Omnibus Installation** + +```bash +gitlab-rake gitlab:uploads:legacy:migrate +``` + +**Source Installation** + +```bash bundle exec rake gitlab:uploads:legacy:migrate ``` diff --git a/doc/administration/repository_checks.md b/doc/administration/repository_checks.md index 6bf10441369..decd708a85d 100644 --- a/doc/administration/repository_checks.md +++ b/doc/administration/repository_checks.md @@ -36,10 +36,9 @@ in `repocheck.log`: - `/var/log/gitlab/gitlab-rails` for Omnibus installations - `/home/git/gitlab/log` for installations from source -If for some reason the periodic repository check caused a lot of false -alarms you can choose to clear *all* repository check states by -clicking "Clear all repository checks" on the **Settings** page of the -admin panel (`/admin/application_settings`). +If the periodic repository check causes false alarms, you can clear all repository check states by +navigating to **Admin Area > Settings > Repository** +(`/admin/application_settings/repository`) and clicking **Clear all repository checks**. --- [ce-3232]: https://gitlab.com/gitlab-org/gitlab-foss/merge_requests/3232 "Auto git fsck" diff --git a/doc/administration/repository_storage_paths.md b/doc/administration/repository_storage_paths.md index d31b1d7fcd6..dfd0a618a73 100644 --- a/doc/administration/repository_storage_paths.md +++ b/doc/administration/repository_storage_paths.md @@ -110,7 +110,7 @@ Once you set the multiple storage paths, you can choose where new projects will be stored under **Admin Area > Settings > Repository > Repository storage > Storage nodes for new projects**. - + Beginning with GitLab 8.13.4, multiple paths can be chosen. New projects will be randomly placed on one of the selected paths. diff --git a/doc/administration/troubleshooting/debug.md b/doc/administration/troubleshooting/debug.md index 3007b711405..b754b954391 100644 --- a/doc/administration/troubleshooting/debug.md +++ b/doc/administration/troubleshooting/debug.md @@ -196,7 +196,7 @@ is a Unicorn worker that is spinning via `top`. Try to use the `gdb` techniques above. In addition, using `strace` may help isolate issues: ```shell -strace -tt -T -f -s 1024 -p <PID of unicorn worker> -o /tmp/unicorn.txt +strace -ttTfyyy -s 1024 -p <PID of unicorn worker> -o /tmp/unicorn.txt ``` If you cannot isolate which Unicorn worker is the issue, try to run `strace` @@ -204,7 +204,7 @@ on all the Unicorn workers to see where the `/internal/allowed` endpoint gets stuck: ```shell -ps auwx | grep unicorn | awk '{ print " -p " $2}' | xargs strace -tt -T -f -s 1024 -o /tmp/unicorn.txt +ps auwx | grep unicorn | awk '{ print " -p " $2}' | xargs strace -ttTfyyy -s 1024 -o /tmp/unicorn.txt ``` The output in `/tmp/unicorn.txt` may help diagnose the root cause. diff --git a/doc/administration/troubleshooting/elasticsearch.md b/doc/administration/troubleshooting/elasticsearch.md index 5846514c574..a582e07b141 100644 --- a/doc/administration/troubleshooting/elasticsearch.md +++ b/doc/administration/troubleshooting/elasticsearch.md @@ -106,7 +106,7 @@ graph TD; D2 --> |Yes| D4 D4 --> |No| D5 D4 --> |Yes| D6 - D{Is the error concerning<br>the beta indexer?} + D{Is the error concerning<br>the Go indexer?} D1[It would be best<br>to speak with an<br>Elasticsearch admin.] D2{Is the ICU development<br>package installed?} D3>This package is required.<br>Install the package<br>and retry.] @@ -245,12 +245,13 @@ much to "integrate" here. If the issue is: -- Not concerning the beta indexer, it is almost always an +- With the Go indexer, check if the ICU development package is installed. + This is a required package so make sure you install it. + Go indexer was a beta indexer which can be optionally turned on/off, but in 12.3 it reached stable status and is now the default. +- Not concerning the Go indexer, it is almost always an Elasticsearch-side issue. This means you should reach out to your Elasticsearch admin regarding the error(s) you are seeing. If you are unsure here, it never hurts to reach out to GitLab support. -- With the beta indexer, check if the ICU development package is installed. - This is a required package so make sure you install it. Beyond that, you will want to review the error. If it is: @@ -324,7 +325,7 @@ feel free to update that page with issues you encounter and solutions. ## Replication -Setting up Elasticsearch isn't too bad, but it can be a bit finnicky and time consuming. +Setting up Elasticsearch isn't too bad, but it can be a bit finicky and time consuming. The easiest method is to spin up a docker container with the required version and bind ports 9200/9300 so it can be used. diff --git a/doc/administration/troubleshooting/gitlab_rails_cheat_sheet.md b/doc/administration/troubleshooting/gitlab_rails_cheat_sheet.md index cb0b24ae026..0ab8d629b61 100644 --- a/doc/administration/troubleshooting/gitlab_rails_cheat_sheet.md +++ b/doc/administration/troubleshooting/gitlab_rails_cheat_sheet.md @@ -743,6 +743,8 @@ Namespace.find_by_full_path("user/proj").namespace_statistics.update(shared_runn ### Remove artifacts more than a week old +The Latest version of these steps can be found in the [job artifacts documentation](../job_artifacts.md) + ```ruby ### SELECTING THE BUILDS TO CLEAR # For a single project: diff --git a/doc/administration/troubleshooting/postgresql.md b/doc/administration/troubleshooting/postgresql.md index 65c6952bf1c..ab302c919b2 100644 --- a/doc/administration/troubleshooting/postgresql.md +++ b/doc/administration/troubleshooting/postgresql.md @@ -41,7 +41,7 @@ This section is for links to information elsewhere in the GitLab documentation. - [Using Slony to update PostgreSQL](../../update/upgrading_postgresql_using_slony.md) - Uses replication to handle PostgreSQL upgrades - providing the schemas are the same. - - Reduces downtime to a short window for swinging over to the newer vewrsion. + - Reduces downtime to a short window for swinging over to the newer version. - Managing Omnibus PostgreSQL versions [from the development docs](https://docs.gitlab.com/omnibus/development/managing-postgresql-versions.html) diff --git a/doc/administration/troubleshooting/sidekiq.md b/doc/administration/troubleshooting/sidekiq.md index 41657368ea4..91361dddf02 100644 --- a/doc/administration/troubleshooting/sidekiq.md +++ b/doc/administration/troubleshooting/sidekiq.md @@ -174,7 +174,7 @@ the query details. ## Managing Sidekiq queues It is possible to use [Sidekiq API](https://github.com/mperham/sidekiq/wiki/API) -to perform a number of troubleshoting on Sidekiq. +to perform a number of troubleshooting on Sidekiq. These are the administrative commands and it should only be used if currently admin interface is not suitable due to scale of installation. |
