summaryrefslogtreecommitdiff
path: root/doc/integration
diff options
context:
space:
mode:
authorGitLab Bot <gitlab-bot@gitlab.com>2020-09-19 01:45:44 +0000
committerGitLab Bot <gitlab-bot@gitlab.com>2020-09-19 01:45:44 +0000
commit85dc423f7090da0a52c73eb66faf22ddb20efff9 (patch)
tree9160f299afd8c80c038f08e1545be119f5e3f1e1 /doc/integration
parent15c2c8c66dbe422588e5411eee7e68f1fa440bb8 (diff)
downloadgitlab-ce-85dc423f7090da0a52c73eb66faf22ddb20efff9.tar.gz
Add latest changes from gitlab-org/gitlab@13-4-stable-ee
Diffstat (limited to 'doc/integration')
-rw-r--r--doc/integration/README.md3
-rw-r--r--doc/integration/elasticsearch.md639
-rw-r--r--doc/integration/external-issue-tracker.md3
-rw-r--r--doc/integration/gitpod.md74
-rw-r--r--doc/integration/img/gitpod_button_project_page_v13_4.pngbin0 -> 25773 bytes
-rw-r--r--doc/integration/img/gitpod_web_interface_v13_4.pngbin0 -> 99925 bytes
-rw-r--r--doc/integration/img/jira_dev_panel_jira_setup_1-1.pngbin13286 -> 0 bytes
-rw-r--r--doc/integration/jira_development_panel.md21
-rw-r--r--doc/integration/kerberos.md2
-rw-r--r--doc/integration/omniauth.md11
-rw-r--r--doc/integration/salesforce.md2
11 files changed, 433 insertions, 322 deletions
diff --git a/doc/integration/README.md b/doc/integration/README.md
index fdaff74ca58..c5c21644d1c 100644
--- a/doc/integration/README.md
+++ b/doc/integration/README.md
@@ -15,6 +15,7 @@ GitLab can be integrated with the following external issue trackers:
- Jira
- Redmine
- Bugzilla
+- EWM
- YouTrack
## Authentication sources
@@ -56,7 +57,7 @@ GitLab can be integrated with the following enhancements:
- Configure [PlantUML](../administration/integration/plantuml.md) to use diagrams in AsciiDoc documents.
- Attach merge requests to [Trello](trello_power_up.md) cards.
- Enable integrated code intelligence powered by [Sourcegraph](sourcegraph.md).
-- Add [Elasticsearch](elasticsearch.md) for [Advanced Global Search](../user/search/advanced_global_search.md),
+- Add [Elasticsearch](elasticsearch.md) for [Advanced Search](../user/search/advanced_global_search.md),
[Advanced System Search](../user/search/advanced_search_syntax.md), and faster searching.
## Integrations
diff --git a/doc/integration/elasticsearch.md b/doc/integration/elasticsearch.md
index 67b256cc944..f40955ad8ff 100644
--- a/doc/integration/elasticsearch.md
+++ b/doc/integration/elasticsearch.md
@@ -1,43 +1,85 @@
+---
+type: reference
+stage: Enablement
+group: Global Search
+info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers
+---
+
# Elasticsearch integration **(STARTER ONLY)**
> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/109 "Elasticsearch Merge Request") in GitLab [Starter](https://about.gitlab.com/pricing/) 8.4.
> - Support for [Amazon Elasticsearch](https://docs.aws.amazon.com/elasticsearch-service/latest/developerguide/es-gsg.html) was [introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/1305) in GitLab [Starter](https://about.gitlab.com/pricing/) 9.0.
-This document describes how to set up Elasticsearch with GitLab. Once enabled,
-you'll have the benefit of fast search response times and the advantage of two
-special searches:
+This document describes how to set up Elasticsearch with GitLab. After
+Elasticsearch is enabled, you'll have the benefit of fast search response times
+and the advantage of the following special searches:
+
+- [Advanced Search](../user/search/advanced_global_search.md)
+- [Advanced Search Syntax](../user/search/advanced_search_syntax.md)
+
+## Version requirements
+
+<!-- Remember to update ee/lib/system_check/app/elasticsearch_check.rb if this changes -->
-- [Advanced Global Search](../user/search/advanced_global_search.md)
-- [Advanced Syntax Search](../user/search/advanced_search_syntax.md)
+| GitLab version | Elasticsearch version |
+|---------------------------------------------|-------------------------------|
+| GitLab Enterprise Edition 12.7 or greater | Elasticsearch 6.x through 7.x |
+| GitLab Enterprise Edition 11.5 through 12.6 | Elasticsearch 5.6 through 6.x |
+| GitLab Enterprise Edition 9.0 through 11.4 | Elasticsearch 5.1 through 5.5 |
+| GitLab Enterprise Edition 8.4 through 8.17 | Elasticsearch 2.4 with [Delete By Query Plugin](https://www.elastic.co/guide/en/elasticsearch/plugins/2.4/plugins-delete-by-query.html) installed |
-## Version Requirements
+## System requirements
+
+Elasticsearch requires additional resources in excess of those documented in the
+[GitLab system requirements](../install/requirements.md).
-<!-- Please remember to update ee/lib/system_check/app/elasticsearch_check.rb if this changes -->
+The amount of resources (memory, CPU, storage) will vary greatly, based on the
+amount of data being indexed into the Elasticsearch cluster. According to
+[Elasticsearch official guidelines](https://www.elastic.co/guide/en/elasticsearch/guide/current/hardware.html#_memory),
+each node should have:
-| GitLab version | Elasticsearch version |
-| -------------- | --------------------- |
-| GitLab Enterprise Edition 8.4 - 8.17 | Elasticsearch 2.4 with [Delete By Query Plugin](https://www.elastic.co/guide/en/elasticsearch/plugins/2.4/plugins-delete-by-query.html) installed |
-| GitLab Enterprise Edition 9.0 - 11.4 | Elasticsearch 5.1 - 5.5 |
-| GitLab Enterprise Edition 11.5 - 12.6 | Elasticsearch 5.6 - 6.x |
-| GitLab Enterprise Edition 12.7+ | Elasticsearch 6.x - 7.x |
+- [Memory](https://www.elastic.co/guide/en/elasticsearch/guide/current/hardware.html#_memory): 8 GiB (minimum).
+- [CPU](https://www.elastic.co/guide/en/elasticsearch/guide/current/hardware.html#_cpus): Modern processor with multiple cores.
+- [Storage](https://www.elastic.co/guide/en/elasticsearch/guide/current/hardware.html#_disks): Use SSD storage. You will need enough storage for 50% of the total size of your Git repositories.
+
+A few notes on CPU and storage:
+
+- CPU requirements for Elasticsearch tend to be minimal. There are specific
+ scenarios where this isn't true, but GitLab.com isn't using Elasticsearch in
+ an exceptionally CPU-heavy way. More cores will be more performant than faster
+ CPUs. Extra concurrency from multiple cores will far outweigh a slightly
+ faster clock speed in Elasticsearch.
+
+- Storage requirements for Elasticsearch are important, especially for
+ indexing-heavy clusters. When possible use SSDs, whose speed is far superior
+ to any spinning media for Elasticsearch. In testing, nodes that use SSD storage
+ see boosts in both query and indexing performance.
+
+Keep in mind, these are **minimum requirements** for Elasticsearch.
+Heavily-utilized Elasticsearch clusters will likely require considerably more
+resources.
## Installing Elasticsearch
-Elasticsearch is _not_ included in the Omnibus packages or when you install from source. You must
-[install it separately](https://www.elastic.co/guide/en/elasticsearch/reference/7.x/install-elasticsearch.html "Elasticsearch 7.x installation documentation"). Be sure to select your version.
-Providing detailed information on installing Elasticsearch is out of the scope
-of this document.
+Elasticsearch is *not* included in the Omnibus packages or when you install from
+source. You must [install it separately](https://www.elastic.co/guide/en/elasticsearch/reference/7.x/install-elasticsearch.html "Elasticsearch 7.x installation documentation").
+Be sure to select your version. Providing detailed information on installing
+Elasticsearch is out of the scope of this document.
NOTE: **Note:**
Elasticsearch should be installed on a separate server, whether you install
-it yourself or use a cloud hosted offering like Elastic's [Elasticsearch Service](https://www.elastic.co/elasticsearch/service) (available on AWS, GCP, or Azure) or the
-[Amazon Elasticsearch](https://docs.aws.amazon.com/elasticsearch-service/latest/developerguide/es-gsg.html) service. Running Elasticsearch on the same server as GitLab is not recommended
-and will likely cause a degradation in GitLab instance performance.
+it yourself or use a cloud hosted offering like Elastic's [Elasticsearch Service](https://www.elastic.co/elasticsearch/service)
+(available on AWS, GCP, or Azure) or the [Amazon Elasticsearch](https://docs.aws.amazon.com/elasticsearch-service/latest/developerguide/es-gsg.html)
+service. Running Elasticsearch on the same server as GitLab is not recommended
+and can cause a degradation in GitLab instance performance.
NOTE: **Note:**
-**For a single node Elasticsearch cluster the functional cluster health status will be yellow** (will never be green) because the primary shard is allocated but replicas can not be as there is no other node to which Elasticsearch can assign a replica.
+**For a single node Elasticsearch cluster the functional cluster health status
+will be yellow** (will never be green) because the primary shard is allocated but
+replicas can not be as there is no other node to which Elasticsearch can assign a
+replica.
-Once the data is added to the database or repository and [Elasticsearch is
+After the data is added to the database or repository and [Elasticsearch is
enabled in the Admin Area](#enabling-elasticsearch) the search index will be
updated automatically.
@@ -47,12 +89,13 @@ For indexing Git repository data, GitLab uses an [indexer written in Go](https:/
The way you install the Go indexer depends on your version of GitLab:
-- For Omnibus GitLab 11.8 and above, see [Omnibus GitLab](#omnibus-gitlab).
-- For installations from source or older versions of Omnibus GitLab, install the indexer [From Source](#from-source).
+- For Omnibus GitLab 11.8 or greater, see [Omnibus GitLab](#omnibus-gitlab).
+- For installations from source or older versions of Omnibus GitLab,
+ [install the indexer from source](#from-source).
### Omnibus GitLab
-Since GitLab 11.8 the Go indexer is included in Omnibus GitLab.
+Starting with GitLab 11.8, the Go indexer is included in Omnibus GitLab.
The former Ruby-based indexer was removed in [GitLab 12.3](https://gitlab.com/gitlab-org/gitlab/-/issues/6481).
### From source
@@ -80,7 +123,7 @@ To install on CentOS or RHEL, run:
sudo yum install libicu-devel
```
-##### Mac OSX
+#### Mac OSX
To install on macOS, run:
@@ -112,60 +155,87 @@ Example:
PREFIX=/usr sudo -E make install
```
-Once installed, enable it under your instance's Elasticsearch settings explained [below](#enabling-elasticsearch).
+After installation, be sure to [enable Elasticsearch](#enabling-elasticsearch).
-## System Requirements
+## Enabling Elasticsearch
-Elasticsearch requires additional resources in excess of those documented in the
-[GitLab system requirements](../install/requirements.md).
+NOTE: **Note:**
+For GitLab instances with more than 50GB repository data you can follow the instructions for [Indexing large
+instances](#indexing-large-instances) below.
-The amount of resources (memory, CPU, storage) will vary greatly, based on the amount of data being indexed into the Elasticsearch cluster. According to [Elasticsearch official guidelines](https://www.elastic.co/guide/en/elasticsearch/guide/current/hardware.html#_memory), each node should have:
+To enable Elasticsearch, you need to have admin access to GitLab:
-- [RAM](https://www.elastic.co/guide/en/elasticsearch/guide/current/hardware.html#_disks): **8 GiB as the bare minimum**
-- [CPU](https://www.elastic.co/guide/en/elasticsearch/guide/current/hardware.html#_cpus): Modern processor with multiple cores
-- [Storage](https://www.elastic.co/guide/en/elasticsearch/guide/current/hardware.html#_disks): Use SSD storage. As a guide you will need enough storage for 50% of the total size of your Git repositories.
+1. Navigate to **Admin Area** (wrench icon), then **Settings > General**
+ and expand the **Advanced Search** section.
-A few notes on CPU and storage:
+ NOTE: **Note:**
+ To see the Advanced Search section, you need an active Starter
+ [license](../user/admin_area/license.md).
-- CPU requirements for Elasticsearch tend to be light. There are specific scenarios where this isn't true, but GitLab.com isn't using Elasticsearch in an exceptionally CPU-heavy way. More cores will be more performant than faster CPUs. Extra concurrency from multiple cores will far outweigh a slightly faster clock speed in Elasticsearch.
+1. Configure the [Elasticsearch settings](#elasticsearch-configuration) for
+ your Elasticsearch cluster. Do not enable **Elasticsearch indexing** or
+ **Search with Elasticsearch enabled** yet.
+1. Click **Save changes** for the changes to take effect.
+1. Before enabling **Elasticsearch indexing** you need to create an index by
+ running the Rake task:
-- Storage requirements for Elasticsearch are important, especially for indexing-heavy clusters. When possible, use SSDs, Their speed is far superior to any spinning media for Elasticsearch. In testing, nodes that use SSD storage see boosts in both query and indexing performance.
+ ```shell
+ # Omnibus installations
+ sudo gitlab-rake gitlab:elastic:create_empty_index
-Keep in mind, these are **minimum requirements** for Elasticsearch. Heavily-utilized Elasticsearch clusters will likely require considerably more resources.
+ # Installations from source
+ bundle exec rake gitlab:elastic:create_empty_index RAILS_ENV=production
+ ```
-## Enabling Elasticsearch
+1. Now enable `Elasticsearch indexing` in **Admin Area > Settings >
+ General > Advanced Search** and click **Save changes**.
+1. Click **Index all projects**.
+1. Click **Check progress** in the confirmation message to see the status of
+ the background jobs.
+1. Personal snippets need to be indexed using another Rake task:
+
+ ```shell
+ # Omnibus installations
+ sudo gitlab-rake gitlab:elastic:index_snippets
+
+ # Installations from source
+ bundle exec rake gitlab:elastic:index_snippets RAILS_ENV=production
+ ```
-In order to enable Elasticsearch, you need to have admin access. Navigate to
-**Admin Area** (wrench icon), then **Settings > Integrations** and expand the **Elasticsearch** section.
+1. After the indexing has completed, enable **Search with Elasticsearch enabled** in
+ **Admin Area > Settings > General > Advanced Search** and click **Save
+ changes**.
-Click **Save changes** for the changes to take effect.
+### Elasticsearch configuration
The following Elasticsearch settings are available:
| Parameter | Description |
-| ----------------------------------------------------- | ----------- |
-| `Elasticsearch indexing` | Enables/disables Elasticsearch indexing. You may want to enable indexing but disable search in order to give the index time to be fully completed, for example. Also, keep in mind that this option doesn't have any impact on existing data, this only enables/disables background indexer which tracks data changes. So by enabling this you will not get your existing data indexed, use special Rake task for that as explained in [Adding GitLab's data to the Elasticsearch index](#adding-gitlabs-data-to-the-elasticsearch-index). |
-| `Elasticsearch pause indexing` | Enables/disables temporary indexing pause. This is useful for cluster migration/reindexing. All changes are still tracked, but they are not committed to the Elasticsearch index until unpaused. |
-| `Search with Elasticsearch enabled` | Enables/disables using Elasticsearch in search. |
+|-------------------------------------------------------|-------------|
+| `Elasticsearch indexing` | Enables or disables Elasticsearch indexing. You may want to enable indexing but disable search in order to give the index time to be fully completed, for example. Also, keep in mind that this option doesn't have any impact on existing data, this only enables/disables the background indexer which tracks data changes and ensures new data is indexed. |
+| `Pause Elasticsearch indexing` | Enables or disables temporary indexing pause. This is useful for cluster migration/reindexing. All changes are still tracked, but they are not committed to the Elasticsearch index until unpaused. |
+| `Search with Elasticsearch enabled` | Enables or disables using Elasticsearch in search. |
| `URL` | The URL to use for connecting to Elasticsearch. Use a comma-separated list to support clustering (e.g., `http://host1, https://host2:9200`). If your Elasticsearch instance is password protected, pass the `username:password` in the URL (e.g., `http://<username>:<password>@<elastic_host>:9200/`). |
| `Number of Elasticsearch shards` | Elasticsearch indexes are split into multiple shards for performance reasons. In general, larger indexes need to have more shards. Changes to this value do not take effect until the index is recreated. You can read more about tradeoffs in the [Elasticsearch documentation](https://www.elastic.co/guide/en/elasticsearch/reference/current/scalability.html). |
| `Number of Elasticsearch replicas` | Each Elasticsearch shard can have a number of replicas. These are a complete copy of the shard, and can provide increased query performance or resilience against hardware failure. Increasing this value will greatly increase total disk space required by the index. |
| `Limit namespaces and projects that can be indexed` | Enabling this will allow you to select namespaces and projects to index. All other namespaces and projects will use database search instead. Please note that if you enable this option but do not select any namespaces or projects, none will be indexed. [Read more below](#limiting-namespaces-and-projects).
-| `Using AWS hosted Elasticsearch with IAM credentials` | Sign your Elasticsearch requests using [AWS IAM authorization](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_access-keys.html), [AWS EC2 Instance Profile Credentials](https://docs.aws.amazon.com/codedeploy/latest/userguide/getting-started-create-iam-instance-profile.html#getting-started-create-iam-instance-profile-cli), or [AWS ECS Tasks Credentials](https://docs.aws.amazon.com/AmazonECS/latest/userguide/task-iam-roles.html). The policies must be configured to allow `es:*` actions. |
-| `AWS Region` | The AWS region your Elasticsearch service is located in. |
+| `Using AWS hosted Elasticsearch with IAM credentials` | Sign your Elasticsearch requests using [AWS IAM authorization](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_access-keys.html), [AWS EC2 Instance Profile Credentials](https://docs.aws.amazon.com/codedeploy/latest/userguide/getting-started-create-iam-instance-profile.html#getting-started-create-iam-instance-profile-cli), or [AWS ECS Tasks Credentials](https://docs.aws.amazon.com/AmazonECS/latest/userguide/task-iam-roles.html). Please refer to [Identity and Access Management in Amazon Elasticsearch Service](https://docs.aws.amazon.com/elasticsearch-service/latest/developerguide/es-ac.html) for details of AWS hosted Elasticsearch domain access policy configuration. |
+| `AWS Region` | The AWS region in which your Elasticsearch service is located. |
| `AWS Access Key` | The AWS access key. |
| `AWS Secret Access Key` | The AWS secret access key. |
| `Maximum file size indexed` | See [the explanation in instance limits.](../administration/instance_limits.md#maximum-file-size-indexed). |
| `Maximum field length` | See [the explanation in instance limits.](../administration/instance_limits.md#maximum-field-length). |
| `Maximum bulk request size (MiB)` | The Maximum Bulk Request size is used by GitLab's Golang-based indexer processes and indicates how much data it ought to collect (and store in memory) in a given indexing process before submitting the payload to Elasticsearch’s Bulk API. This setting should be used with the Bulk request concurrency setting (see below) and needs to accommodate the resource constraints of both the Elasticsearch host(s) and the host(s) running GitLab's Golang-based indexer either from the `gitlab-rake` command or the Sidekiq tasks. |
-| `Bulk request concurrency` | The Bulk request concurrency indicates how many of GitLab's Golang-based indexer processes (or threads) can run in parallel to collect data to subsequently submit to Elasticsearch’s Bulk API. This increases indexing performance, but fills the Elasticsearch bulk requests queue faster. This setting should be used together with the Maximum bulk request size setting (see above) and needs to accommodate the resource constraints of both the Elasticsearch host(s) and the host(s) running GitLab's Golang-based indexer either from the `gitlab-rake` command or the Sidekiq tasks. |
+| `Bulk request concurrency` | The Bulk request concurrency indicates how many of GitLab's Golang-based indexer processes (or threads) can run in parallel to collect data to subsequently submit to Elasticsearch’s Bulk API. This increases indexing performance, but fills the Elasticsearch bulk requests queue faster. This setting should be used together with the Maximum bulk request size setting (see above) and needs to accommodate the resource constraints of both the Elasticsearch host(s) and the host(s) running GitLab's Golang-based indexer either from the `gitlab-rake` command or the Sidekiq tasks. |
+| `Client request timeout` | Elasticsearch HTTP client request timeout value in seconds. `0` means using the system default timeout value, which depends on the libraries that GitLab application is built upon. |
### Limiting namespaces and projects
-If you select `Limit namespaces and projects that can be indexed`, more options will become available
+If you select `Limit namespaces and projects that can be indexed`, more options will become available.
+
![limit namespaces and projects options](img/limit_namespaces_projects_options.png)
-You can select namespaces and projects to index exclusively. Please note that if the namespace is a group it will include
+You can select namespaces and projects to index exclusively. Note that if the namespace is a group it will include
any sub-groups and projects belonging to those sub-groups to be indexed as well.
Elasticsearch only provides cross-group code/commit search (global) if all name-spaces are indexed. In this particular scenario where only a subset of namespaces are indexed, a global search will not provide a code or commit scope. This will be possible only in the scope of an indexed namespace. Currently there is no way to code/commit search in multiple indexed namespaces (when only a subset of namespaces has been indexed). For example if two groups are indexed, there is no way to run a single code search on both. You can only run a code search on the first group and then on the second.
@@ -187,8 +257,8 @@ from the Elasticsearch index as expected.
To disable the Elasticsearch integration:
-1. Navigate to the **Admin Area** (wrench icon), then **Settings > Integrations**.
-1. Expand the **Elasticsearch** section and uncheck **Elasticsearch indexing**
+1. Navigate to the **Admin Area** (wrench icon), then **Settings > General**.
+1. Expand the **Advanced Search** section and uncheck **Elasticsearch indexing**
and **Search with Elasticsearch enabled**.
1. Click **Save changes** for the changes to take effect.
1. (Optional) Delete the existing index:
@@ -201,253 +271,39 @@ To disable the Elasticsearch integration:
bundle exec rake gitlab:elastic:delete_index RAILS_ENV=production
```
-## Adding GitLab's data to the Elasticsearch index
-
-While Elasticsearch indexing is enabled, new changes in your GitLab instance will be automatically indexed as they happen.
-To backfill existing data, you can use one of the methods below to index it in background jobs.
-
-### Indexing through the administration UI
-
-> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/15390) in [GitLab Starter](https://about.gitlab.com/pricing/) 12.3.
-
-To index via the Admin Area:
-
-1. [Configure your Elasticsearch host and port](#enabling-elasticsearch).
-1. Create empty indexes:
-
- ```shell
- # Omnibus installations
- sudo gitlab-rake gitlab:elastic:create_empty_index
-
- # Installations from source
- bundle exec rake gitlab:elastic:create_empty_index RAILS_ENV=production
- ```
-
-1. [Enable **Elasticsearch indexing**](#enabling-elasticsearch).
-1. Click **Index all projects** in **Admin Area > Settings > Integrations > Elasticsearch**.
-1. Click **Check progress** in the confirmation message to see the status of the background jobs.
-1. Personal snippets need to be indexed manually:
-
- ```shell
- # Omnibus installations
- sudo gitlab-rake gitlab:elastic:index_snippets
-
- # Installations from source
- bundle exec rake gitlab:elastic:index_snippets RAILS_ENV=production
- ```
-
-1. After the indexing has completed, enable [**Search with Elasticsearch**](#enabling-elasticsearch).
-
-### Indexing through Rake tasks
-
-Indexing can be performed using Rake tasks.
-
-#### Indexing small instances
-
-CAUTION: **Warning:**
-This will delete your existing indexes.
-
-If the database size is less than 500 MiB, and the size of all hosted repos is less than 5 GiB:
-
-1. [Configure your Elasticsearch host and port](#enabling-elasticsearch).
-1. Index your data:
-
- ```shell
- # Omnibus installations
- sudo gitlab-rake gitlab:elastic:index
-
- # Installations from source
- bundle exec rake gitlab:elastic:index RAILS_ENV=production
- ```
-
-1. After the indexing has completed, enable [**Search with Elasticsearch**](#enabling-elasticsearch).
-
-#### Indexing large instances
-
-CAUTION: **Warning:**
-Performing asynchronous indexing will generate a lot of Sidekiq jobs.
-Make sure to prepare for this task by having a [Scalable and Highly Available Setup](README.md)
-or creating [extra Sidekiq processes](../administration/operations/extra_sidekiq_processes.md)
-
-1. [Configure your Elasticsearch host and port](#enabling-elasticsearch).
-1. Create empty indexes:
-
- ```shell
- # Omnibus installations
- sudo gitlab-rake gitlab:elastic:create_empty_index
-
- # Installations from source
- bundle exec rake gitlab:elastic:create_empty_index RAILS_ENV=production
- ```
-
-1. If this is a re-index of your GitLab instance, clear the index status:
-
- ```shell
- # Omnibus installations
- sudo gitlab-rake gitlab:elastic:clear_index_status
-
- # Installations from source
- bundle exec rake gitlab:elastic:clear_index_status RAILS_ENV=production
- ```
-
-1. [Enable **Elasticsearch indexing**](#enabling-elasticsearch).
-1. Indexing large Git repositories can take a while. To speed up the process, you can [tune for indexing speed](https://www.elastic.co/guide/en/elasticsearch/reference/current/tune-for-indexing-speed.html#tune-for-indexing-speed):
-
- - You can temporarily disable [`refresh`](https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-refresh.html), the operation responsible for making changes to an index available to search.
-
- - You can set the number of replicas to 0. This setting controls the number of copies each primary shard of an index will have. Thus, having 0 replicas effectively disables the replication of shards across nodes, which should increase the indexing performance. This is an important trade-off in terms of reliability and query performance. It is important to remember to set the replicas to a considered value after the initial indexing is complete.
-
- In our experience, you can expect a 20% decrease in indexing time. After completing indexing in a later step, you can return `refresh` and `number_of_replicas` to their desired settings.
-
- NOTE: **Note:**
- This step is optional but may help significantly speed up large indexing operations.
-
- ```shell
- curl --request PUT localhost:9200/gitlab-production/_settings --header 'Content-Type: application/json' --data '{
- "index" : {
- "refresh_interval" : "-1",
- "number_of_replicas" : 0
- } }'
- ```
-
-1. Index projects and their associated data:
-
- ```shell
- # Omnibus installations
- sudo gitlab-rake gitlab:elastic:index_projects
-
- # Installations from source
- bundle exec rake gitlab:elastic:index_projects RAILS_ENV=production
- ```
-
- This enqueues a Sidekiq job for each project that needs to be indexed.
- You can view the jobs in **Admin Area > Monitoring > Background Jobs > Queues Tab**
- and click `elastic_indexer`, or you can query indexing status using a Rake task:
-
- ```shell
- # Omnibus installations
- sudo gitlab-rake gitlab:elastic:index_projects_status
-
- # Installations from source
- bundle exec rake gitlab:elastic:index_projects_status RAILS_ENV=production
-
- Indexing is 65.55% complete (6555/10000 projects)
- ```
-
- If you want to limit the index to a range of projects you can provide the
- `ID_FROM` and `ID_TO` parameters:
-
- ```shell
- # Omnibus installations
- sudo gitlab-rake gitlab:elastic:index_projects ID_FROM=1001 ID_TO=2000
-
- # Installations from source
- bundle exec rake gitlab:elastic:index_projects ID_FROM=1001 ID_TO=2000 RAILS_ENV=production
- ```
-
- Where `ID_FROM` and `ID_TO` are project IDs. Both parameters are optional.
- The above example will index all projects from ID `1001` up to (and including) ID `2000`.
-
- TIP: **Troubleshooting:**
- Sometimes the project indexing jobs queued by `gitlab:elastic:index_projects`
- can get interrupted. This may happen for many reasons, but it's always safe
- to run the indexing task again. It will skip repositories that have
- already been indexed.
-
- As the indexer stores the last commit SHA of every indexed repository in the
- database, you can run the indexer with the special parameter `UPDATE_INDEX` and
- it will check every project repository again to make sure that every commit in
- a repository is indexed, which can be useful in case if your index is outdated:
-
- ```shell
- # Omnibus installations
- sudo gitlab-rake gitlab:elastic:index_projects UPDATE_INDEX=true ID_TO=1000
-
- # Installations from source
- bundle exec rake gitlab:elastic:index_projects UPDATE_INDEX=true ID_TO=1000 RAILS_ENV=production
- ```
-
- You can also use the `gitlab:elastic:clear_index_status` Rake task to force the
- indexer to "forget" all progress, so it will retry the indexing process from the
- start.
-
-1. Personal snippets are not associated with a project and need to be indexed separately:
-
- ```shell
- # Omnibus installations
- sudo gitlab-rake gitlab:elastic:index_snippets
-
- # Installations from source
- bundle exec rake gitlab:elastic:index_snippets RAILS_ENV=production
- ```
-
-1. Enable replication and refreshing again after indexing (only if you previously disabled it):
-
- ```shell
- curl --request PUT localhost:9200/gitlab-production/_settings --header 'Content-Type: application/json' --data '{
- "index" : {
- "number_of_replicas" : 1,
- "refresh_interval" : "1s"
- } }'
- ```
-
- A force merge should be called after enabling the refreshing above.
-
- For Elasticsearch 6.x, the index should be in read-only mode before proceeding with the force merge:
-
- ```shell
- curl --request PUT localhost:9200/gitlab-production/_settings --header 'Content-Type: application/json' --data '{
- "settings": {
- "index.blocks.write": true
- } }'
- ```
-
- Then, initiate the force merge:
-
- ```shell
- curl --request POST 'localhost:9200/gitlab-production/_forcemerge?max_num_segments=5'
- ```
-
- After this, if your index is in read-only mode, switch back to read-write:
-
- ```shell
- curl --request PUT localhost:9200/gitlab-production/_settings --header 'Content-Type: application/json' --data '{
- "settings": {
- "index.blocks.write": false
- } }'
- ```
-
-1. After the indexing has completed, enable [**Search with Elasticsearch**](#enabling-elasticsearch).
-
-### Indexing limitations
-
-For repository and snippet files, GitLab will only index up to 1 MiB of content, in order to avoid indexing timeouts.
-
## Zero downtime reindexing
-The idea behind this reindexing method is to leverage Elasticsearch index alias feature to atomically swap between two indices.
-We will refer to each index as `primary` (online and used by GitLab for read/writes) and `secondary` (offline, for reindexing purpose).
+The idea behind this reindexing method is to leverage Elasticsearch index alias
+feature to atomically swap between two indices. We'll refer to each index as
+`primary` (online and used by GitLab for read/writes) and `secondary`
+(offline, for reindexing purpose).
-Instead of connecting directly to the `primary` index, we'll setup an index alias such as we can change the underlying index at will.
+Instead of connecting directly to the `primary` index, we'll setup an index
+alias such as we can change the underlying index at will.
NOTE: **Note:**
-Any index attached to the production alias is deemed a `primary` and will end up being used by the GitLab Elasticsearch integration.
+Any index attached to the production alias is deemed a `primary` and will be
+used by the GitLab Elasticsearch integration.
### Pause the indexing
-Under **Admin Area > Integration > Elasticsearch**, check the **Pause Elasticsearch Indexing** setting and save.
+In the **Admin Area > Settings > General > Advanced Search** section, select the
+**Pause Elasticsearch Indexing** setting, and then save your change.
-With this, all updates that should happen on your Elasticsearch index will be buffered and caught up once unpaused.
+With this, all updates that should happen on your Elasticsearch index will be
+buffered and caught up once unpaused.
### Setup
TIP: **Tip:**
-If your index has been created with GitLab v13.0+ you can skip directly to [trigger the reindex](#trigger-the-reindex-via-the-elasticsearch-administration).
+If your index was created with GitLab 13.0 or greater, you can directly
+[trigger the reindex](#trigger-the-reindex-via-the-advanced-search-administration).
-This process involves multiple shell commands and curl invocations, so a good initial setup will help down the road:
+This process involves several shell commands and curl invocations, so a good
+initial setup will help for later:
```shell
-# You can find this value under Admin Area > Integration > Elasticsearch > URL
+# You can find this value under Admin Area > Settings > General > Advanced Search > URL
export CLUSTER_URL="http://localhost:9200"
export PRIMARY_INDEX="gitlab-production"
export SECONDARY_INDEX="gitlab-production-$(date +%s)"
@@ -456,10 +312,12 @@ export SECONDARY_INDEX="gitlab-production-$(date +%s)"
### Reclaiming the `gitlab-production` index name
CAUTION: **Caution:**
-It is highly recommended that you take a snapshot of your cluster to make sure there is a recovery path if anything goes wrong.
+It is highly recommended that you take a snapshot of your cluster to ensure
+there is a recovery path if anything goes wrong.
NOTE: **Note:**
-Due to a technical limitation, there will be a slight downtime because of the fact that we need to reclaim the current `primary` index to be used as the alias.
+Due to a technical limitation, there will be a slight downtime because of the
+fact that we need to reclaim the current `primary` index to be used as the alias.
To reclaim the `gitlab-production` index name, you need to first create a `secondary` index and then trigger the re-index from `primary`.
@@ -477,7 +335,8 @@ sudo SKIP_ALIAS=1 gitlab-rake "gitlab:elastic:create_empty_index[$SECONDARY_INDE
SKIP_ALIAS=1 bundle exec rake "gitlab:elastic:create_empty_index[$SECONDARY_INDEX]"
```
-The index should be created successfully, with the latest index options and mappings.
+The index should be created successfully, with the latest index options and
+mappings.
#### Trigger the re-index from `primary`
@@ -498,9 +357,9 @@ To trigger the re-index from `primary` index:
{"task":"3qw_Tr0YQLq7PF16Xek8YA:1012"}
```
- Note the `task` value here as it will be useful to follow the reindex progress.
+ Note the `task` value, as it will be useful to follow the reindex progress.
-1. Wait for the reindex process to complete, by checking the `completed` value.
+1. Wait for the reindex process to complete by checking the `completed` value.
Using the `task` value form the previous step:
```shell
@@ -514,10 +373,10 @@ To trigger the re-index from `primary` index:
{"completed":false, …}
```
- Once the returned value is `true`, you may continue to the next step.
+ After the returned value is `true`, continue to the next step.
-1. Make sure that the secondary index has data in it. You can use the Elasticsearch
- API to look for the index size and compare our two indices:
+1. Ensure that the secondary index has data in it. You can use the
+ Elasticsearch API to look for the index size and compare our two indices:
```shell
curl $CLUSTER_URL/$PRIMARY_INDEX/_count => 123123
@@ -527,7 +386,8 @@ To trigger the re-index from `primary` index:
TIP: **Tip:**
Comparing the document count is more accurate than using the index size, as improvements to the storage might cause the new index to be smaller than the original one.
-1. Once you are confident your `secondary` index is valid, you can process to the creation of the alias.
+1. After you are confident your `secondary` index is valid, you can process to
+ the creation of the alias.
```shell
# Delete the original index
@@ -540,18 +400,18 @@ To trigger the re-index from `primary` index:
$CLUSTER_URL/_aliases
```
- The reindexing is now completed. Your GitLab instance is now ready to use the [automated in-cluster reindexing](#trigger-the-reindex-via-the-elasticsearch-administration) feature for future reindexing.
+ The reindexing is now completed. Your GitLab instance is now ready to use the [automated in-cluster reindexing](#trigger-the-reindex-via-the-advanced-search-administration) feature for future reindexing.
1. Unpause the indexing
- Under **Admin Area > Integration > Elasticsearch**, uncheck the **Pause Elasticsearch Indexing** setting and save.
+ Under **Admin Area > Settings > General > Advanced Search**, uncheck the **Pause Elasticsearch Indexing** setting and save.
-### Trigger the reindex via the Elasticsearch administration
+### Trigger the reindex via the Advanced Search administration
> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/34069) in [GitLab Starter](https://about.gitlab.com/pricing/) 13.2.
> - A scheduled index deletion and the ability to cancel it was [introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/38914) in GitLab Starter 13.3.
-Under **Admin Area > Integration > Elasticsearch zero-downtime reindexing**, click on **Trigger cluster reindexing**.
+Under **Admin Area > Settings > General > Advanced Search > Elasticsearch zero-downtime reindexing**, click on **Trigger cluster reindexing**.
NOTE: **Note:**
Reindexing can be a lengthy process depending on the size of your Elasticsearch cluster.
@@ -567,13 +427,13 @@ Rake tasks are available to:
- [Build and install](#building-and-installing) the indexer.
- Delete indexes when [disabling Elasticsearch](#disabling-elasticsearch).
-- [Add GitLab data](#adding-gitlabs-data-to-the-elasticsearch-index) to an index.
+- Add GitLab data to an index.
The following are some available Rake tasks:
| Task | Description |
|:--------------------------------------------------------------------------------------------------------------------------------------------------------|:------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
-| [`sudo gitlab-rake gitlab:elastic:index`](https://gitlab.com/gitlab-org/gitlab/blob/master/ee/lib/tasks/gitlab/elastic.rake) | Enables Elasticsearch Indexing and run `gitlab:elastic:create_empty_index`, `gitlab:elastic:clear_index_status`, `gitlab:elastic:index_projects`, and `gitlab:elastic:index_snippets`. |
+| [`sudo gitlab-rake gitlab:elastic:index`](https://gitlab.com/gitlab-org/gitlab/blob/master/ee/lib/tasks/gitlab/elastic.rake) | Enables Elasticsearch indexing and run `gitlab:elastic:create_empty_index`, `gitlab:elastic:clear_index_status`, `gitlab:elastic:index_projects`, and `gitlab:elastic:index_snippets`. |
| [`sudo gitlab-rake gitlab:elastic:index_projects`](https://gitlab.com/gitlab-org/gitlab/blob/master/ee/lib/tasks/gitlab/elastic.rake) | Iterates over all projects and queues Sidekiq jobs to index them in the background. |
| [`sudo gitlab-rake gitlab:elastic:index_projects_status`](https://gitlab.com/gitlab-org/gitlab/blob/master/ee/lib/tasks/gitlab/elastic.rake) | Determines the overall status of the indexing. It is done by counting the total number of indexed projects, dividing by a count of the total number of projects, then multiplying by 100. |
| [`sudo gitlab-rake gitlab:elastic:clear_index_status`](https://gitlab.com/gitlab-org/gitlab/blob/master/ee/lib/tasks/gitlab/elastic.rake) | Deletes all instances of IndexStatus for all projects. |
@@ -644,6 +504,168 @@ For basic guidance on choosing a cluster configuration you may refer to [Elastic
- The `Number of Elasticsearch shards` setting usually corresponds with the number of CPUs available in your cluster. For example, if you have a 3-node cluster with 4 cores each, this means you will benefit from having at least 3*4=12 shards in the cluster. Please note, it's only possible to change the shards number by using [Split index API](https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-split-index.html) or by reindexing to a different index with a changed number of shards.
- The `Number of Elasticsearch replicas` setting should most of the time be equal to `1` (each shard will have 1 replica). Using `0` is not recommended, because losing one node will corrupt the index.
+### Indexing large instances
+
+This section may be helpful in the event that the other
+[basic instructions](#enabling-elasticsearch) cause problems
+due to large volumes of data being indexed.
+
+CAUTION: **Warning:**
+Indexing a large instance will generate a lot of Sidekiq jobs.
+Make sure to prepare for this task by having a [Scalable and Highly Available
+Setup](../administration/reference_architectures/index.md) or creating [extra
+Sidekiq processes](../administration/operations/extra_sidekiq_processes.md).
+
+1. [Configure your Elasticsearch host and port](#enabling-elasticsearch).
+1. Create empty indexes:
+
+ ```shell
+ # Omnibus installations
+ sudo gitlab-rake gitlab:elastic:create_empty_index
+
+ # Installations from source
+ bundle exec rake gitlab:elastic:create_empty_index RAILS_ENV=production
+ ```
+
+1. If this is a re-index of your GitLab instance, clear the index status:
+
+ ```shell
+ # Omnibus installations
+ sudo gitlab-rake gitlab:elastic:clear_index_status
+
+ # Installations from source
+ bundle exec rake gitlab:elastic:clear_index_status RAILS_ENV=production
+ ```
+
+1. [Enable **Elasticsearch indexing**](#enabling-elasticsearch).
+1. Indexing large Git repositories can take a while. To speed up the process, you can [tune for indexing speed](https://www.elastic.co/guide/en/elasticsearch/reference/current/tune-for-indexing-speed.html#tune-for-indexing-speed):
+
+ - You can temporarily disable [`refresh`](https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-refresh.html), the operation responsible for making changes to an index available to search.
+
+ - You can set the number of replicas to 0. This setting controls the number of copies each primary shard of an index will have. Thus, having 0 replicas effectively disables the replication of shards across nodes, which should increase the indexing performance. This is an important trade-off in terms of reliability and query performance. It is important to remember to set the replicas to a considered value after the initial indexing is complete.
+
+ In our experience, you can expect a 20% decrease in indexing time. After completing indexing in a later step, you can return `refresh` and `number_of_replicas` to their desired settings.
+
+ NOTE: **Note:**
+ This step is optional but may help significantly speed up large indexing operations.
+
+ ```shell
+ curl --request PUT localhost:9200/gitlab-production/_settings --header 'Content-Type: application/json' --data '{
+ "index" : {
+ "refresh_interval" : "-1",
+ "number_of_replicas" : 0
+ } }'
+ ```
+
+1. Index projects and their associated data:
+
+ ```shell
+ # Omnibus installations
+ sudo gitlab-rake gitlab:elastic:index_projects
+
+ # Installations from source
+ bundle exec rake gitlab:elastic:index_projects RAILS_ENV=production
+ ```
+
+ This enqueues a Sidekiq job for each project that needs to be indexed.
+ You can view the jobs in **Admin Area > Monitoring > Background Jobs > Queues Tab**
+ and click `elastic_indexer`, or you can query indexing status using a Rake task:
+
+ ```shell
+ # Omnibus installations
+ sudo gitlab-rake gitlab:elastic:index_projects_status
+
+ # Installations from source
+ bundle exec rake gitlab:elastic:index_projects_status RAILS_ENV=production
+
+ Indexing is 65.55% complete (6555/10000 projects)
+ ```
+
+ If you want to limit the index to a range of projects you can provide the
+ `ID_FROM` and `ID_TO` parameters:
+
+ ```shell
+ # Omnibus installations
+ sudo gitlab-rake gitlab:elastic:index_projects ID_FROM=1001 ID_TO=2000
+
+ # Installations from source
+ bundle exec rake gitlab:elastic:index_projects ID_FROM=1001 ID_TO=2000 RAILS_ENV=production
+ ```
+
+ Where `ID_FROM` and `ID_TO` are project IDs. Both parameters are optional.
+ The above example will index all projects from ID `1001` up to (and including) ID `2000`.
+
+ TIP: **Troubleshooting:**
+ Sometimes the project indexing jobs queued by `gitlab:elastic:index_projects`
+ can get interrupted. This may happen for many reasons, but it's always safe
+ to run the indexing task again. It will skip repositories that have
+ already been indexed.
+
+ As the indexer stores the last commit SHA of every indexed repository in the
+ database, you can run the indexer with the special parameter `UPDATE_INDEX` and
+ it will check every project repository again to make sure that every commit in
+ a repository is indexed, which can be useful in case if your index is outdated:
+
+ ```shell
+ # Omnibus installations
+ sudo gitlab-rake gitlab:elastic:index_projects UPDATE_INDEX=true ID_TO=1000
+
+ # Installations from source
+ bundle exec rake gitlab:elastic:index_projects UPDATE_INDEX=true ID_TO=1000 RAILS_ENV=production
+ ```
+
+ You can also use the `gitlab:elastic:clear_index_status` Rake task to force the
+ indexer to "forget" all progress, so it will retry the indexing process from the
+ start.
+
+1. Personal snippets are not associated with a project and need to be indexed separately:
+
+ ```shell
+ # Omnibus installations
+ sudo gitlab-rake gitlab:elastic:index_snippets
+
+ # Installations from source
+ bundle exec rake gitlab:elastic:index_snippets RAILS_ENV=production
+ ```
+
+1. Enable replication and refreshing again after indexing (only if you previously disabled it):
+
+ ```shell
+ curl --request PUT localhost:9200/gitlab-production/_settings --header 'Content-Type: application/json' --data '{
+ "index" : {
+ "number_of_replicas" : 1,
+ "refresh_interval" : "1s"
+ } }'
+ ```
+
+ A force merge should be called after enabling the refreshing above.
+
+ For Elasticsearch 6.x, the index should be in read-only mode before proceeding with the force merge:
+
+ ```shell
+ curl --request PUT localhost:9200/gitlab-production/_settings --header 'Content-Type: application/json' --data '{
+ "settings": {
+ "index.blocks.write": true
+ } }'
+ ```
+
+ Then, initiate the force merge:
+
+ ```shell
+ curl --request POST 'localhost:9200/gitlab-production/_forcemerge?max_num_segments=5'
+ ```
+
+ After this, if your index is in read-only mode, switch back to read-write:
+
+ ```shell
+ curl --request PUT localhost:9200/gitlab-production/_settings --header 'Content-Type: application/json' --data '{
+ "settings": {
+ "index.blocks.write": false
+ } }'
+ ```
+
+1. After the indexing has completed, enable [**Search with Elasticsearch enabled**](#enabling-elasticsearch).
+
### Deleted documents
Whenever a change or deletion is made to an indexed GitLab object (a merge request description is changed, a file is deleted from the master branch in a repository, a project is deleted, etc), a document in the index is deleted. However, since these are "soft" deletes, the overall number of "deleted documents", and therefore wasted space, increases. Elasticsearch does intelligent merging of segments in order to remove these deleted documents. However, depending on the amount and type of activity in your GitLab installation, it's possible to see as much as 50% wasted space in the index.
@@ -701,11 +723,11 @@ Here are some common pitfalls and how to overcome them:
We continuously make updates to our indexing strategies and aim to support
newer versions of Elasticsearch. When indexing changes are made, it may
- be necessary for you to [reindex](#adding-gitlabs-data-to-the-elasticsearch-index) after updating GitLab.
+ be necessary for you to [reindex](#zero-downtime-reindexing) after updating GitLab.
- **I indexed all the repositories but I can't find anything**
- Make sure you indexed all the database data [as stated above](#adding-gitlabs-data-to-the-elasticsearch-index).
+ Make sure you indexed all the database data [as stated above](#enabling-elasticsearch).
Beyond that, check via the [Elasticsearch Search API](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-search.html) to see if the data shows up on the Elasticsearch side.
@@ -734,6 +756,17 @@ Here are some common pitfalls and how to overcome them:
You can run `sudo gitlab-rake gitlab:elastic:projects_not_indexed` to display projects that aren't indexed.
+- **No new data is added to the Elasticsearch index when I push code**
+
+ NOTE: **Note:**
+ This was [fixed in GitLab 13.2](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/35936) and the Rake task is not available for versions greater than that.
+
+ When performing the initial indexing of blobs, we lock all projects until the project finishes indexing. It could happen that an error during the process causes one or multiple projects to remain locked. In order to unlock them, run:
+
+ ```shell
+ sudo gitlab-rake gitlab:elastic:clear_locked_projects
+ ```
+
- **"Can't specify parent if no parent field has been configured"**
If you enabled Elasticsearch before GitLab 8.12 and have not rebuilt indexes you will get
@@ -801,7 +834,7 @@ Here are some common pitfalls and how to overcome them:
```
You probably have not used either `http://` or `https://` as part of your value in the **"URL"** field of the Elasticsearch Integration Menu. Please make sure you are using either `http://` or `https://` in this field as the [Elasticsearch client for Go](https://github.com/olivere/elastic) that we are using [needs the prefix for the URL to be accepted as valid](https://github.com/olivere/elastic/commit/a80af35aa41856dc2c986204e2b64eab81ccac3a).
- Once you have corrected the formatting of the URL, delete the index (via the [dedicated Rake task](#gitlab-elasticsearch-rake-tasks)) and [reindex the content of your instance](#adding-gitlabs-data-to-the-elasticsearch-index).
+ Once you have corrected the formatting of the URL, delete the index (via the [dedicated Rake task](#gitlab-elasticsearch-rake-tasks)) and [reindex the content of your instance](#enabling-elasticsearch).
### Low-level troubleshooting
@@ -815,7 +848,7 @@ There is a [more structured, lower-level troubleshooting document](../administra
Improvements to the `code_analyzer` pattern and filters is being discussed in [epic 3621](https://gitlab.com/groups/gitlab-org/-/epics/3621).
-### Reverting to basic search
+### Reverting to Basic Search
Sometimes there may be issues with your Elasticsearch index data and as such
GitLab will allow you to revert to "basic search" when there are no search
diff --git a/doc/integration/external-issue-tracker.md b/doc/integration/external-issue-tracker.md
index 8b4ebc337de..cde0093f53e 100644
--- a/doc/integration/external-issue-tracker.md
+++ b/doc/integration/external-issue-tracker.md
@@ -1,7 +1,7 @@
# External issue tracker
GitLab has a great [issue tracker](../user/project/issues/index.md) but you can also use an external one
-such as Jira, Redmine, YouTrack, or Bugzilla. External issue trackers are configurable per GitLab project.
+such as Jira, Redmine, YouTrack, Bugzilla, or EWM. External issue trackers are configurable per GitLab project.
Once configured, you can reference external issues using the format `CODE-123`, where:
@@ -26,6 +26,7 @@ Visit the links below for details:
- [YouTrack](../user/project/integrations/youtrack.md)
- [Jira](../user/project/integrations/jira.md)
- [Bugzilla](../user/project/integrations/bugzilla.md)
+- [EWM](../user/project/integrations/ewm.md)
- [Custom Issue Tracker](../user/project/integrations/custom_issue_tracker.md)
### Service Template
diff --git a/doc/integration/gitpod.md b/doc/integration/gitpod.md
new file mode 100644
index 00000000000..f26483e3b5e
--- /dev/null
+++ b/doc/integration/gitpod.md
@@ -0,0 +1,74 @@
+---
+type: reference, how-to
+stage: Create
+group: Editor
+info: "To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers"
+---
+
+# Gitpod Integration
+
+> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/228893) in GitLab 13.4.
+> - It's [deployed behind a feature flag](#enable-or-disable-the-gitpod-integration), disabled by default.
+> - It's enabled on GitLab.com.
+> - It's recommended for production use.
+> - To use it in GitLab self-managed instances, ask a GitLab administrator to [enable it](#configure-your-gitlab-instance-with-gitpod). **(CORE ONLY)**
+
+CAUTION: **Warning:**
+This feature might not be available to you. Check the **version history** note above for details.
+
+With [Gitpod](https://gitpod.io/) you can describe your dev environment as code to get fully set
+up, compiled, and tested dev environments for any GitLab project. The dev environments are not only
+automated but also prebuilt which means that Gitpod continuously builds your Git branches like a CI
+server. By that you don’t have to wait for dependencies to be downloaded and builds to finish, but
+you can start coding immediately.
+
+In short: With Gitpod you can start coding instantly on any project, branch, and merge request from
+any device, at any time.
+
+![Gitpod interface](img/gitpod_web_interface_v13_4.png)
+
+You can launch Gitpod directly from GitLab by clicking the **Gitpod** button from the **Web IDE**
+dropdown on the project page:
+
+![Gitpod Button on Project Page](img/gitpod_button_project_page_v13_4.png)
+
+To learn more about Gitpod, see their [features](https://www.gitpod.io/features/) and
+[documentation](https://www.gitpod.io/docs/).
+
+To use the GitLab-Gitpod integration, you need to enable it from your user preferences:
+
+1. From the GitLab UI, click your avatar in the top-right corner, then click **Settings**.
+1. On the left-hand nav, click **Preferences**.
+1. Under **Integrations**, find the **Gitpod** section.
+1. Check **Enable Gitpod**.
+
+Users of GitLab.com can enable it and start using straightaway. Users of GitLab self-managed instances
+can follow the same steps once the integration has been enabled and configured by a GitLab administrator.
+
+## Configure your GitLab instance with Gitpod **(CORE ONLY)**
+
+If you are new to Gitpod, head over to the [Gitpod documentation](https://www.gitpod.io/docs/self-hosted/latest/self-hosted/)
+and get your instance up and running.
+
+1. In GitLab, go to **Admin Area > Settings > Integrations**.
+1. Expand the **Gitpod** configuration section.
+1. Check **Enable Gitpod**.
+1. Add your Gitpod instance URL (for example, `https://gitpod.example.com`).
+
+## Enable or disable the Gitpod integration **(CORE ONLY)**
+
+The Gitpod integration is under development and not ready for production use. It is deployed behind a
+feature flag that is **disabled by default**.
+[GitLab administrators with access to the GitLab Rails console](../administration/feature_flags.md)
+can enable it.
+
+To enable it:
+
+```ruby
+Feature.enable(:gitpod)
+```
+
+To disable it:
+
+```ruby
+Feature.disable(:gitpod)
diff --git a/doc/integration/img/gitpod_button_project_page_v13_4.png b/doc/integration/img/gitpod_button_project_page_v13_4.png
new file mode 100644
index 00000000000..55a70d89169
--- /dev/null
+++ b/doc/integration/img/gitpod_button_project_page_v13_4.png
Binary files differ
diff --git a/doc/integration/img/gitpod_web_interface_v13_4.png b/doc/integration/img/gitpod_web_interface_v13_4.png
new file mode 100644
index 00000000000..5cd9a6aad0f
--- /dev/null
+++ b/doc/integration/img/gitpod_web_interface_v13_4.png
Binary files differ
diff --git a/doc/integration/img/jira_dev_panel_jira_setup_1-1.png b/doc/integration/img/jira_dev_panel_jira_setup_1-1.png
deleted file mode 100644
index cef903ac9b4..00000000000
--- a/doc/integration/img/jira_dev_panel_jira_setup_1-1.png
+++ /dev/null
Binary files differ
diff --git a/doc/integration/jira_development_panel.md b/doc/integration/jira_development_panel.md
index 7c646b95ae7..9b7aa5829c1 100644
--- a/doc/integration/jira_development_panel.md
+++ b/doc/integration/jira_development_panel.md
@@ -4,9 +4,10 @@ group: Ecosystem
info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers
---
-# GitLab Jira Development Panel integration **(PREMIUM)**
+# GitLab Jira Development Panel integration **(CORE)**
-> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/2381) in [GitLab Premium](https://about.gitlab.com/pricing/) 10.0.
+> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/2381) in [GitLab Premium](https://about.gitlab.com/pricing/) 10.0.
+> - [Moved](https://gitlab.com/gitlab-org/gitlab/-/issues/233149) to [GitLab Core](https://about.gitlab.com/pricing/) in 13.4.
The Jira Development Panel integration allows you to reference Jira issues within GitLab, displaying activity in the [Development panel](https://support.atlassian.com/jira-software-cloud/docs/view-development-information-for-an-issue/) in the issue. It complements the [GitLab Jira integration](../user/project/integrations/jira.md). You may choose to configure both integrations to take advantage of both sets of features. (See a [feature comparison](../user/project/integrations/jira_integrations.md#feature-comparison)).
@@ -92,9 +93,6 @@ If you're using GitLab.com and Jira Cloud, we recommend you use the [GitLab for
If you're using Jira Cloud, go to **Settings (gear) > Products > DVCS accounts**.
1. Click **Link GitHub Enterprise account** to start creating a new integration.
(We're pretending to be GitHub in this integration, until there's additional platform support in Jira.)
-
- ![Jira Settings](img/jira_dev_panel_jira_setup_1-1.png)
-
1. Complete the form:
Select **GitHub Enterprise** for the **Host** field.
@@ -202,13 +200,12 @@ Potential resolutions:
- If you're using GitLab versions 11.10-12.7, upgrade to GitLab 12.8.10 or later
to resolve an identified [issue](https://gitlab.com/gitlab-org/gitlab/-/issues/37012).
-- The Jira Development Panel integration requires GitLab Premium, GitLab.com Silver,
- or a higher tier. If you're using a lower tier of GitLab, you'll need to upgrade
- to use this feature.
+- If you're using GitLab Core or GitLab Starter, be sure you're using
+ GitLab 13.4 or later.
[Contact GitLab Support](https://about.gitlab.com/support) if none of these reasons apply.
-#### Fixing synchonization issues
+#### Fixing synchronization issues
If Jira displays incorrect information (such as deleted branches), you may need to
resynchronize the information. To do so:
@@ -237,12 +234,14 @@ For a walkthrough of the integration with GitLab for Jira, watch [Configure GitL
1. After installing, click **Get started** to go to the configurations page. This page is always available under **Jira Settings > Apps > Manage apps**.
![Start GitLab App configuration on Jira](img/jira_dev_panel_setup_com_2.png)
-1. Enter the group or personal namespace in the **Namespace** field and click **Link namespace to Jira**. Make sure you are logged in on GitLab.com and the namespace has a Silver or above license. The user setting up _GitLab for Jira_ must have **Maintainer** access to the GitLab namespace.
+1. In **Namespace**, enter the group or personal namespace, and then click
+ **Link namespace to Jira**. The user setting up *GitLab for Jira* must have
+ *Maintainer* access to the GitLab namespace.
NOTE: **Note:**
The GitLab user only needs access when adding a new namespace. For syncing with Jira, we do not depend on the user's token.
- ![Confure namespace on GitLab Jira App](img/jira_dev_panel_setup_com_3.png)
+ ![Configure namespace on GitLab Jira App](img/jira_dev_panel_setup_com_3.png)
After a namespace is added, all future commits, branches and merge requests of all projects under that namespace will be synced to Jira. Past data cannot be synced at the moment.
diff --git a/doc/integration/kerberos.md b/doc/integration/kerberos.md
index 09bc795f7ef..1b14b5a986f 100644
--- a/doc/integration/kerberos.md
+++ b/doc/integration/kerberos.md
@@ -103,7 +103,7 @@ enabled, then your users will be automatically linked to their LDAP accounts on
first login. For this to work, some prerequisites must be met:
The Kerberos username must match the LDAP user's UID. You can choose which LDAP
-attribute is used as the UID in GitLab's [LDAP configuration](../administration/auth/ldap/index.md#configuration-core-only)
+attribute is used as the UID in GitLab's [LDAP configuration](../administration/auth/ldap/index.md#configuration)
but for Active Directory, this should be `sAMAccountName`.
The Kerberos realm must match the domain part of the LDAP user's Distinguished
diff --git a/doc/integration/omniauth.md b/doc/integration/omniauth.md
index 9dd7f2cd9e1..dd183ad9eb0 100644
--- a/doc/integration/omniauth.md
+++ b/doc/integration/omniauth.md
@@ -142,19 +142,22 @@ The chosen OmniAuth provider is now active and can be used to sign in to GitLab
## Automatically Link Existing Users to OmniAuth Users
-You can automatically link OmniAuth users with existing GitLab users if their email addresses match by adding the following setting:
+> [Introduced in GitLab 13.4.](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/36664)
+
+You can automatically link OmniAuth users with existing GitLab users if their email addresses match.
+For example, the following setting is used to enable the auto link feature for both a SAML provider and the Twitter OAuth provider:
**For Omnibus installations**
```ruby
-gitlab_rails['omniauth_auto_link_user'] = true
+gitlab_rails['omniauth_auto_link_user'] = ["saml", "twitter"]
```
**For installations from source**
```yaml
omniauth:
- auto_link_user: true
+ auto_link_user: ["saml", "twitter"]
```
## Configure OmniAuth Providers as External
@@ -299,7 +302,7 @@ providers without two factor authentication.
Define the allowed providers using an array, e.g. `["twitter", 'google_oauth2']`, or as
`true`/`false` to allow all providers or none. This option should only be configured
for providers which already have two factor authentication (default: false).
-This configration dose not apply to SAML.
+This configuration dose not apply to SAML.
```ruby
gitlab_rails['omniauth_allow_bypass_two_factor'] = ['twitter', 'google_oauth2']
diff --git a/doc/integration/salesforce.md b/doc/integration/salesforce.md
index 64eac2f9e66..7e0b2518e76 100644
--- a/doc/integration/salesforce.md
+++ b/doc/integration/salesforce.md
@@ -21,7 +21,7 @@ To get the credentials (a pair of Client ID and Client Secret), you must [create
1. Select **API (Enable OAuth Settings)** and click on **Enable OAuth Settings**.
1. Fill in the application details into the following fields:
- **Callback URL**: The callback URL of your GitLab installation. For example, `https://gitlab.example.com/users/auth/salesforce/callback`.
- - **Selected OAuth Scopes**: Move **Access your basic information (id, profile, email, address, phone)** and **Allow access to your unique identifier (OpenID)** to the right column.
+ - **Selected OAuth Scopes**: Move `Access your basic information (id, profile, email, address, phone)` and `Allow access to your unique identifier (openid)` to the right column.
![Salesforce OAuth App Details](img/salesforce_oauth_app_details.png)