diff options
| author | Achilleas Pipinellis <axil@gitlab.com> | 2019-03-12 16:22:37 +0100 |
|---|---|---|
| committer | Achilleas Pipinellis <axil@gitlab.com> | 2019-03-12 16:22:37 +0100 |
| commit | c19ed72155378c7e684d771964ad027f57c4ca34 (patch) | |
| tree | aee24cf4cd24ca3962a2adf68c7b89cfb37ef242 | |
| parent | a817f7905c084f725b6fa01955be4fd8ad28c747 (diff) | |
| download | gitlab-ce-docs/ee-to-ce.tar.gz | |
Merge EE docs into CEdocs/ee-to-ce
634 files changed, 20964 insertions, 665 deletions
diff --git a/doc/README.md b/doc/README.md index ecc214d97c8..a06554d89aa 100644 --- a/doc/README.md +++ b/doc/README.md @@ -4,7 +4,7 @@ description: 'Learn how to use and administer GitLab, the most scalable Git-base --- <div class="display-none"> - <em>Visit <a href="https://docs.gitlab.com/ce/">docs.gitlab.com</a> for optimized + <em>Visit <a href="https://docs.gitlab.com/ee/">docs.gitlab.com</a> for optimized navigation, discoverability, and readability.</em> </div> <!-- the div above will not display on the docs site but will display on /help --> @@ -107,13 +107,18 @@ The following documentation relates to the DevOps **Plan** stage: | Plan Topics | Description | |:-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:-------------------------------------------------------------------------------------------------------------------------------------------------| +| [Burndown Charts](user/project/milestones/burndown_charts.md) **[STARTER]** | Watch your project's progress throughout a specific milestone. | | [Discussions](user/discussions/index.md) | Threads, comments, and resolvable discussions in issues, commits, and merge requests. | | [Due Dates](user/project/issues/due_dates.md) | Keep track of issue deadlines. | -| [Quick Actions](user/project/quick_actions.md) | Shortcuts for common actions on issues or merge requests, replacing the need to click buttons or use dropdowns in GitLab's UI. | +| [Epics](user/group/epics/index.md) **[ULTIMATE]** | Tracking groups of issues that share a theme. | | [Issues](user/project/issues/index.md), including [confidential issues](user/project/issues/confidential_issues.md),<br/>[issue and merge request templates](user/project/description_templates.md),<br/>and [moving issues](user/project/issues/moving_issues.md) | Project issues, restricting access to issues, create templates for submitting new issues and merge requests, and moving issues between projects. | | [Labels](user/project/labels.md) | Categorize issues or merge requests with descriptive labels. | | [Milestones](user/project/milestones/index.md) | Set milestones for delivery of issues and merge requests, with optional due date. | | [Project Issue Board](user/project/issue_board.md) | Display issues on a Scrum or Kanban board. | +| [Quick Actions](user/project/quick_actions.md) | Shortcuts for common actions on issues or merge requests, replacing the need to click buttons or use dropdowns in GitLab's UI. | +| [Related Issues](user/project/issues/related_issues.md) **[STARTER]** | Create a relationship between issues. | +| [Roadmap](user/group/roadmap/index.md) **[ULTIMATE]** | Visualize epic timelines. | +| [Service Desk](user/project/service_desk.md) **[PREMIUM]** | A simple way to allow people to create issues in your GitLab instance without needing their own user account. | | [Time Tracking](workflow/time_tracking.md) | Track time spent on issues and merge requests. | | [Todos](workflow/todos.md) | Keep track of work requiring attention with a chronological list displayed on a simple dashboard. | @@ -136,16 +141,21 @@ The following documentation relates to the DevOps **Create** stage: #### Projects and Groups -| Create Topics - Projects and Groups | Description | -|:-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:-------------------------------------------------------------------------------| -| [Create](gitlab-basics/create-project.md) and [fork](gitlab-basics/fork-project.md) projects, and<br/>[import and export<br/>projects between instances](user/project/settings/import_export.md) | Create, duplicate, and move projects. | -| [GitLab Pages](user/project/pages/index.md) | Build, test, and deploy your static website with GitLab Pages. | -| [Groups](user/group/index.md) and [Subgroups](user/group/subgroups/index.md) | Organize your projects in groups. | -| [Projects](user/project/index.md), including [project access](public_access/public_access.md)<br/>and [settings](user/project/settings/index.md) | Host source code, and control your project's visibility and set configuration. | -| [Search through GitLab](user/search/index.md) | Search for issues, merge requests, projects, groups, and todos. | -| [Snippets](user/snippets.md) | Snippets allow you to create little bits of code. | -| [Web IDE](user/project/web_ide/index.md) | Edit files within GitLab's user interface. | -| [Wikis](user/project/wiki/index.md) | Enhance your repository documentation with built-in wikis. | +| Create Topics - Projects and Groups | Description | +|:-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:-------------------------------------------------------------------------------------------------| +| [Advanced global search](user/search/advanced_global_search.md) **[STARTER]** | Leverage Elasticsearch for faster, more advanced code search across your entire GitLab instance. | +| [Advanced syntax search](user/search/advanced_search_syntax.md) **[STARTER]** | Use advanced queries for more targeted search results. | +| [Contribution analytics](user/group/contribution_analytics/index.md) **[STARTER]** | See detailed statistics of group contributors. | +| [Create](gitlab-basics/create-project.md) and [fork](gitlab-basics/fork-project.md) projects, and<br/>[import and export projects<br/>between instances](user/project/settings/import_export.md) | Create, duplicate, and move projects. | +| [File locking](user/project/file_lock.md) **[PREMIUM]** | Lock files to avoid merge conflicts. | +| [GitLab Pages](user/project/pages/index.md) | Build, test, and deploy your static website with GitLab Pages. | +| [Groups](user/group/index.md) and [Subgroups](user/group/subgroups/index.md) | Organize your projects in groups. | +| [Issues Analytics](user/group/issues_analytics/index.md) **[PREMIUM]** | Check how many issues were created per month. | +| [Projects](user/project/index.md), including [project access](public_access/public_access.md)<br/>and [settings](user/project/settings/index.md) | Host source code, and control your project's visibility and set configuration. | +| [Search through GitLab](user/search/index.md) | Search for issues, merge requests, projects, groups, and todos. | +| [Snippets](user/snippets.md) | Snippets allow you to create little bits of code. | +| [Web IDE](user/project/web_ide/index.md) | Edit files within GitLab's user interface. | +| [Wikis](user/project/wiki/index.md) | Enhance your repository documentation with built-in wikis. | <div align="right"> <a type="button" class="btn btn-default" href="#overview"> @@ -165,7 +175,9 @@ The following documentation relates to the DevOps **Create** stage: | [Files](user/project/repository/index.md#files) | Files management. | | [Jupyter Notebook files](user/project/repository/index.md#jupyter-notebook-files) | GitLab's support for `.ipynb` files. | | [Protected branches](user/project/protected_branches.md) | Use protected branches. | +| [Push rules](push_rules/push_rules.md) **[STARTER]** | Additional control over pushes to your projects. | | [Repositories](user/project/repository/index.md) | Manage source code repositories in GitLab's user interface. | +| [Repository mirroring](workflow/repository_mirroring.md) **[STARTER]** | Push to or pull from repositories outside of GitLab | | [Start a merge request](user/project/repository/web_editor.md#tips) | Start merge request when committing via GitLab's user interface. | <div align="right"> @@ -192,13 +204,14 @@ The following documentation relates to the DevOps **Create** stage: #### Integration and Automation -| Create Topics - Integration and Automation | Description | -|:------------------------------------------------------------------|:-----------------------------------------------------------------------------------------------------------------------| -| [GitLab API](api/README.md) | Integrate GitLab via a simple and powerful API. | -| [GitLab Integration](integration/README.md) | Integrate with multiple third-party services with GitLab to allow external issue trackers and external authentication. | -| [GitLab Webhooks](user/project/integrations/webhooks.md) | Let GitLab notify you when new code has been pushed to your project. | -| [Project Services](user/project/integrations/project_services.md) | Integrate a project with external services, such as CI and chat. | -| [Trello Power-Up](integration/trello_power_up.md) | Integrate with GitLab's Trello Power-Up. | +| Create Topics - Integration and Automation | Description | +|:------------------------------------------------------------------------------|:-----------------------------------------------------------------------------------------------------------------------| +| [GitLab API](api/README.md) | Integrate GitLab via a simple and powerful API. | +| [GitLab Integration](integration/README.md) | Integrate with multiple third-party services with GitLab to allow external issue trackers and external authentication. | +| [GitLab Webhooks](user/project/integrations/webhooks.md) | Let GitLab notify you when new code has been pushed to your project. | +| [JIRA Development Panel](integration/jira_development_panel.md) **[PREMIUM]** | See GitLab information in the JIRA Development Panel. | +| [Project Services](user/project/integrations/project_services.md) | Integrate a project with external services, such as CI and chat. | +| [Trello Power-Up](integration/trello_power_up.md) | Integrate with GitLab's Trello Power-Up. | <div align="right"> <a type="button" class="btn btn-default" href="#overview"> @@ -218,12 +231,14 @@ scales to run your tests faster. The following documentation relates to the DevOps **Verify** stage: -| Verify Topics | Description | -|:---------------------------------------------------|:-----------------------------------------------------------------------------| -| [GitLab CI/CD](ci/README.md) | Explore the features and capabilities of Continuous Integration with GitLab. | -| [JUnit test reports](ci/junit_test_reports.md) | Display JUnit test reports on merge requests. | -| [Pipeline Graphs](ci/pipelines.md#pipeline-graphs) | Visualize builds. | -| [Review Apps](ci/review_apps/index.md) | Preview changes to your application right from a merge request. | +| Verify Topics | Description | +|:----------------------------------------------------------------------------------|:--------------------------------------------------------------------------------------------------------| +| [Code Quality reports](user/project/merge_requests/code_quality.md) **[STARTER]** | Analyze source code quality. | +| [GitLab CI/CD](ci/README.md) | Explore the features and capabilities of Continuous Integration with GitLab. | +| [JUnit test reports](ci/junit_test_reports.md) | Display JUnit test reports on merge requests. | +| [Multi-project pipelines](ci/multi_project_pipelines.md) **[PREMIUM]** | Visualize entire pipelines that span multiple projects, including all cross-project inter-dependencies. | +| [Pipeline Graphs](ci/pipelines.md#pipeline-graphs) | Visualize builds. | +| [Review Apps](ci/review_apps/index.md) | Preview changes to your application right from a merge request. | <div align="right"> <a type="button" class="btn btn-default" href="#overview"> @@ -242,6 +257,7 @@ The following documentation relates to the DevOps **Package** stage: | Package Topics | Description | |:----------------------------------------------------------------|:-------------------------------------------------------| | [GitLab Container Registry](user/project/container_registry.md) | Learn how to use GitLab's built-in Container Registry. | +| [GitLab Packages](administration/packages.md) **[PREMIUM]** | Use GitLab as an NPM registry or Maven repository. | <div align="right"> <a type="button" class="btn btn-default" href="#overview"> @@ -257,14 +273,17 @@ confidently and securely with GitLab’s built-in Continuous Delivery and Deploy The following documentation relates to the DevOps **Release** stage: -| Release Topics | Description | -|:------------------------------------------------------------|:---------------------------------------------------------------------------------------------| -| [Auto Deploy](topics/autodevops/index.md#auto-deploy) | Configure GitLab for the deployment of your application. | -| [Environments and deployments](ci/environments.md) | With environments, you can control the continuous deployment of your software within GitLab. | -| [GitLab CI/CD](ci/README.md) | Explore the features and capabilities of Continuous Deployment and Delivery with GitLab. | -| [GitLab Pages](user/project/pages/index.md) | Build, test, and deploy a static site directly from GitLab. | -| [Protected Runners](ci/runners/README.md#protected-runners) | Select Runners to only pick jobs for protected branches and tags. | -| [Scheduled Pipelines](user/project/pipelines/schedules.md) | Execute pipelines on a schedule. | +| Release Topics | Description | +|:----------------------------------------------------------------------------------------------------------------|:--------------------------------------------------------------------------------------------------------------------------------------| +| [Auto Deploy](topics/autodevops/index.md#auto-deploy) | Configure GitLab for the deployment of your application. | +| [Canary Deployments](user/project/canary_deployments.md) **[PREMIUM]** | Employ a popular CI strategy where a small portion of the fleet is updated to the new version first. | +| [Deploy Boards](user/project/deploy_boards.md) **[PREMIUM]** | View the current health and status of each CI environment running on Kubernetes, displaying the status of the pods in the deployment. | +| [Environments and deployments](ci/environments.md) | With environments, you can control the continuous deployment of your software within GitLab. | +| [Environment-specific variables](ci/variables/README.md#limiting-environment-scopes-of-variables-premium) **[PREMIUM]** | Limit scope of variables to specific environments. | +| [GitLab CI/CD](ci/README.md) | Explore the features and capabilities of Continuous Deployment and Delivery with GitLab. | +| [GitLab Pages](user/project/pages/index.md) | Build, test, and deploy a static site directly from GitLab. | +| [Protected Runners](ci/runners/README.md#protected-runners) | Select Runners to only pick jobs for protected branches and tags. | +| [Scheduled Pipelines](user/project/pipelines/schedules.md) | Execute pipelines on a schedule. | <div align="right"> <a type="button" class="btn btn-default" href="#overview"> @@ -288,6 +307,7 @@ The following documentation relates to the DevOps **Configure** stage: | [GitLab ChatOps](ci/chatops/README.md) | Interact with CI/CD jobs through chat services. | | [Installing Applications](user/project/clusters/index.md#installing-applications) | Deploy Helm, Ingress, and Prometheus on Kubernetes. | | [Mattermost slash commands](user/project/integrations/mattermost_slash_commands.md) | Enable and use slash commands from within Mattermost. | +| [Multiple Kubernetes Clusters](user/project/clusters/index.md#multiple-kubernetes-clusters) **[PREMIUM]** | Associate more than one Kubernetes clusters to your project. | | [Protected variables](ci/variables/README.md#protected-variables) | Restrict variables to protected branches and tags. | | [Serverless](user/project/clusters/serverless/index.md) | Run serverless workloads on Kubernetes. | | [Slack slash commands](user/project/integrations/slack_slash_commands.md) | Enable and use slash commands from within Slack. | @@ -323,16 +343,23 @@ The following documentation relates to the DevOps **Monitor** stage: ### Secure -GitLab can help you secure your applications from within your development lifecycle. +Check your application for security vulnerabilities that may lead to unauthorized access, +data leaks, and denial of services. GitLab will perform static and dynamic tests on the +code of your application, looking for known flaws and report them in the merge request +so you can fix them before merging. Security teams can use dashboards to get a +high-level view on projects and groups, and start remediation processes when needed. The following documentation relates to the DevOps **Secure** stage: -| Monitor Topics | Description | -|:----------------------------------------------------------------|:-----------------------------------------------------------------------------------------------------------| -| [Container Scanning example](ci/examples/container_scanning.md) | `.gitlab-ci.yml` example of using Clair and clair-scanner to scan docker images for known vulnerabilities. | - -NOTE: **Note:** -Viewing [Container Scanning reports](https://docs.gitlab.com/ee/user/project/merge_requests/container_scanning.html) within merge requests requires [GitLab Ultimate](https://about.gitlab.com/pricing/). +| Monitor Topics | Description | +|:--------------------------------------------------------------------------------------------------|:-----------------------------------------------------------------------| +| [Container Scanning](user/project/merge_requests/container_scanning.md) **[ULTIMATE]** | Use Clair to scan docker images for known vulnerabilities. | +| [Dependency Scanning](user/project/merge_requests/dependency_scanning.md) **[ULTIMATE]** | Analyze your dependencies for known vulnerabilities. | +| [Dynamic Application Security Testing (DAST)](user/project/merge_requests/dast.md) **[ULTIMATE]** | Analyze running web applications for known vulnerabilities. | +| [Group Security Dashboard](user/group/security_dashboard/index.md) **[ULTIMATE]** | View vulnerabilities in all the projects in a group and its subgroups. | +| [License Management](user/project/merge_requests/license_management.md) **[ULTIMATE]** | Search your project's dependencies for their licenses. | +| [Project Security Dashboard](user/project/security_dashboard.md) **[ULTIMATE]** | View the latest security reports for your project. | +| [Static Application Security Testing (SAST)](user/project/merge_requests/sast.md) **[ULTIMATE]** | Analyze source code for known vulnerabilities. | ## Subscribe to GitLab diff --git a/doc/administration/audit_events.md b/doc/administration/audit_events.md new file mode 100644 index 00000000000..7a2fedd141b --- /dev/null +++ b/doc/administration/audit_events.md @@ -0,0 +1,116 @@ +--- +last_updated: 2019-02-04 +--- + +# Audit Events + +GitLab offers a way to view the changes made within the GitLab server for owners and administrators on a [paid plan][ee]. + +GitLab system administrators can also take advantage of the logs located on the +filesystem, see [the logs system documentation](logs.md) for more details. + +## Overview + +**Audit Events** is a tool for GitLab owners and administrators to be +able to track important events such as who performed certain actions and the +time they happened. These actions could be, for example, change a user +permission level, who added a new user, or who removed a user. + +## Use-cases + +- Check who was the person who changed the permission level of a particular + user for a project in GitLab. +- Use it to track which users have access to a certain group of projects + in GitLab, and who gave them that permission level. + +## List of events + +There are two kinds of events logged: + +- Events scoped to the group or project, used by group / project managers + to look up who made what change. +- Instance events scoped to the whole GitLab instance, used by your Compliance team to + perform formal audits. + +### Group events **[STARTER]** + +NOTE: **Note:** +You need Owner [permissions] to view the group Audit Events page. + +To view a group's audit events, navigate to **Group > Settings > Audit Events**. +From there, you can see the following actions: + +- Group name/path changed +- Group repository size limit changed +- Group created/deleted +- Group changed visibility +- User was added to group and with which [permissions] +- Permissions changes of a user assigned to a group +- Removed user from group +- Project added to group and with which visibility level +- Project removed from group +- [Project shared with group](../user/project/members/share_project_with_groups.md) + and with which [permissions] +- Removal of a previously shared group with a project +- LFS enabled/disabled +- Shared runners minutes limit changed +- Membership lock enabled/disabled +- Request access enabled/disabled +- 2FA enforcement/grace period changed +- Roles allowed to create project changed + +### Project events **[STARTER]** + +NOTE: **Note:** +You need Maintainer [permissions] or higher to view the project Audit Events page. + +To view a project's audit events, navigate to **Project > Settings > Audit Events**. +From there, you can see the following actions: + +- Added/removed deploy keys +- Project created/deleted/renamed/moved(transferred)/changed path +- Project changed visibility level +- User was added to project and with which [permissions] +- Permission changes of a user assigned to a project +- User was removed from project + +### Instance events **[PREMIUM ONLY]** + +> [Introduced][ee-2336] in [GitLab Premium][ee] 9.3. + +Server-wide audit logging introduces the ability to observe user actions across +the entire instance of your GitLab server, making it easy to understand who +changed what and when for audit purposes. + +To view the server-wide admin log, visit **Admin Area > Monitoring > Audit Log**. + +In addition to the group and project events, the following user actions are also +recorded: + +- Failed Logins +- Sign-in events and the authentication type (standard, LDAP, OmniAuth, etc.) +- Added SSH key +- Added/removed email +- Changed password +- Ask for password reset +- Grant OAuth access + +It is possible to filter particular actions by choosing an audit data type from +the filter drop-down. You can further filter by specific group, project or user +(for authentication events). + + + +### Missing events + +Some events are not being tracked in Audit Events. Please see the following +epics for more detail on which events are not being tracked and our progress +on adding these events into GitLab: + +- [Project settings and activity](https://gitlab.com/groups/gitlab-org/-/epics/474) +- [Group settings and activity](https://gitlab.com/groups/gitlab-org/-/epics/475) +- [Instance-level settings and activity](https://gitlab.com/groups/gitlab-org/-/epics/476) + +[ee-2336]: https://gitlab.com/gitlab-org/gitlab-ee/issues/2336 +[ee]: https://about.gitlab.com/pricing/ +[permissions]: ../user/permissions.md diff --git a/doc/administration/audit_log.png b/doc/administration/audit_log.png Binary files differnew file mode 100644 index 00000000000..d4f4c2abf38 --- /dev/null +++ b/doc/administration/audit_log.png diff --git a/doc/administration/auditor_access_form.png b/doc/administration/auditor_access_form.png Binary files differnew file mode 100644 index 00000000000..c179a7d3b0a --- /dev/null +++ b/doc/administration/auditor_access_form.png diff --git a/doc/administration/auditor_users.md b/doc/administration/auditor_users.md new file mode 100644 index 00000000000..ef8c8197d6d --- /dev/null +++ b/doc/administration/auditor_users.md @@ -0,0 +1,88 @@ +# Auditor users **[PREMIUM ONLY]** + +>[Introduced][ee-998] in [GitLab Premium][eep] 8.17. + +Auditor users are given read-only access to all projects, groups, and other +resources on the GitLab instance. + +## Overview + +Auditor users can have full access to their own resources (projects, groups, +snippets, etc.), and read-only access to **all** other resources, except the +Admin area. To put another way, they are just regular users (who can be added +to projects, create personal snippets, create milestones on their groups, etc.) +who also happen to have read-only access to all projects on the system that +they haven't been explicitly [given access][permissions] to. + +The Auditor role is _not_ a read-only version of the Admin role. Auditor users +will not be able to access the project/group settings pages, or the Admin Area. + +To sum up, assuming you have logged-in as an Auditor user: + +- For a project the Auditor is not member of, the Auditor should have + read-only access. If the project is public or internal, they would have the + same access as the users that are not members of that project/group. +- For a project the Auditor owns, the Auditor should have full access to + everything. +- For a project the Auditor has been added to as a member, the Auditor should + have the same access as the [permissions] they were given to. For example, if + they were added as a Developer, they could then push commits or comment on + issues. +- The Auditor cannot view the Admin area, or perform any admin actions. + +For more information about what an Auditor can or can't do, see the +[Permissions and restrictions of an Auditor user](#permissions-and-restrictions-of-an-auditor-user) +section. + +## Use cases + +1. Your compliance department wants to run tests against the entire GitLab base + to ensure users are complying with password, credit card, and other sensitive + data policies. With Auditor users, this can be achieved very easily without + resulting to tactics like giving a user admin rights or having to use the API + to add them to all projects. +1. If particular users need visibility or access to most of all projects in + your GitLab instance, instead of manually adding the user to all projects, + you can simply create an Auditor user and share the credentials with those + that you want to grant access to. + +## Adding an Auditor user + +1. Create a new user or edit an existing one by navigating to + **Admin Area > Users**. You will find the option of the access level under + the 'Access' section. + +  + +1. Click **Save changes** or **Create user** for the changes to take effect. + +To revoke the Auditor permissions from a user, simply make them a Regular user +following the same steps as above. + +## Permissions and restrictions of an Auditor user + +An Auditor user should be able to access all projects and groups of a GitLab +instance, with the following permissions/restrictions: + +- Has read-only access to the API +- Can access projects that are: + - Private + - Public + - Internal +- Can read all files in a repository +- Can read issues / MRs +- Can read project snippets +- Cannot be Admin and Auditor at the same time +- Cannot access the Admin area +- In a group / project they're not a member of: + - Cannot access project settings + - Cannot access group settings + - Cannot commit to repository + - Cannot create / comment on issues / MRs + - Cannot create/modify files from the Web UI + - Cannot merge a merge request + - Cannot create project snippets + +[ee-998]: https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/998 +[eep]: https://about.gitlab.com/pricing/ +[permissions]: ../user/permissions.md diff --git a/doc/administration/auth/README.md b/doc/administration/auth/README.md index 54be7b616cc..c5c1806376f 100644 --- a/doc/administration/auth/README.md +++ b/doc/administration/auth/README.md @@ -9,9 +9,11 @@ providers. - [LDAP](ldap.md) Includes Active Directory, Apple Open Directory, Open LDAP, and 389 Server + - [LDAP for GitLab EE](ldap-ee.md): LDAP additions to GitLab Enterprise Editions **[STARTER ONLY]** - [OmniAuth](../../integration/omniauth.md) Sign in via Twitter, GitHub, GitLab.com, Google, Bitbucket, Facebook, Shibboleth, Crowd, Azure, Authentiq ID, and JWT - [CAS](../../integration/cas.md) Configure GitLab to sign in using CAS - [SAML](../../integration/saml.md) Configure GitLab as a SAML 2.0 Service Provider - [Okta](okta.md) Configure GitLab to sign in using Okta - [Authentiq](authentiq.md): Enable the Authentiq OmniAuth provider for passwordless authentication +- [Smartcard](smartcard.md) Smartcard authentication diff --git a/doc/administration/auth/how_to_configure_ldap_gitlab_ce/index.md b/doc/administration/auth/how_to_configure_ldap_gitlab_ce/index.md index 15276d364a0..5f4ae278acf 100644 --- a/doc/administration/auth/how_to_configure_ldap_gitlab_ce/index.md +++ b/doc/administration/auth/how_to_configure_ldap_gitlab_ce/index.md @@ -14,7 +14,7 @@ Managing a large number of users in GitLab can become a burden for system admini In this guide we will focus on configuring GitLab with Active Directory. [Active Directory](https://en.wikipedia.org/wiki/Active_Directory) is a popular LDAP compatible directory service provided by Microsoft, included in all modern Windows Server operating systems. -GitLab has supported LDAP integration since [version 2.2](https://about.gitlab.com/2012/02/22/gitlab-version-2-2/). With GitLab LDAP [group syncing](https://docs.gitlab.com/ee/administration/auth/how_to_configure_ldap_gitlab_ee/index.html#group-sync) being added to GitLab Enterprise Edition in [version 6.0](https://about.gitlab.com/2013/08/20/gitlab-6-dot-0-released/). LDAP integration has become one of the most popular features in GitLab. +GitLab has supported LDAP integration since [version 2.2](https://about.gitlab.com/2012/02/22/gitlab-version-2-2/). With GitLab LDAP [group syncing](../how_to_configure_ldap_gitlab_ee/index.html#group-sync) being added to GitLab Enterprise Edition in [version 6.0](https://about.gitlab.com/2013/08/20/gitlab-6-dot-0-released/). LDAP integration has become one of the most popular features in GitLab. ## Getting started @@ -111,7 +111,7 @@ The initial configuration of LDAP in GitLab requires changes to the `gitlab.rb` The two Active Directory specific values are `active_directory: true` and `uid: 'sAMAccountName'`. `sAMAccountName` is an attribute returned by Active Directory used for GitLab usernames. See the example output from `ldapsearch` for a full list of attributes a "person" object (user) has in **AD** - [`ldapsearch` example](#using-ldapsearch-unix) -> Both group_base and admin_group configuration options are only available in GitLab Enterprise Edition. See [GitLab EE - LDAP Features](https://docs.gitlab.com/ee/administration/auth/how_to_configure_ldap_gitlab_ee/index.html#gitlab-enterprise-edition---ldap-features) +> Both group_base and admin_group configuration options are only available in GitLab Enterprise Edition. See [GitLab EE - LDAP Features](../how_to_configure_ldap_gitlab_ee/index.html#gitlab-enterprise-edition---ldap-features) ### Example `gitlab.rb` LDAP @@ -267,4 +267,4 @@ have extended functionalities with LDAP, such as: - Updating user permissions - Multiple LDAP servers -Read through the article on [LDAP for GitLab EE](https://docs.gitlab.com/ee/administration/auth/how_to_configure_ldap_gitlab_ee/) for an overview. +Read through the article on [LDAP for GitLab EE](../how_to_configure_ldap_gitlab_ee/index.md) for an overview. diff --git a/doc/administration/auth/how_to_configure_ldap_gitlab_ee/img/admin_group.png b/doc/administration/auth/how_to_configure_ldap_gitlab_ee/img/admin_group.png Binary files differnew file mode 100644 index 00000000000..9896379d669 --- /dev/null +++ b/doc/administration/auth/how_to_configure_ldap_gitlab_ee/img/admin_group.png diff --git a/doc/administration/auth/how_to_configure_ldap_gitlab_ee/img/group_link_final.png b/doc/administration/auth/how_to_configure_ldap_gitlab_ee/img/group_link_final.png Binary files differnew file mode 100644 index 00000000000..21fb5a7d0ce --- /dev/null +++ b/doc/administration/auth/how_to_configure_ldap_gitlab_ee/img/group_link_final.png diff --git a/doc/administration/auth/how_to_configure_ldap_gitlab_ee/img/group_linking.gif b/doc/administration/auth/how_to_configure_ldap_gitlab_ee/img/group_linking.gif Binary files differnew file mode 100644 index 00000000000..d35cf55804f --- /dev/null +++ b/doc/administration/auth/how_to_configure_ldap_gitlab_ee/img/group_linking.gif diff --git a/doc/administration/auth/how_to_configure_ldap_gitlab_ee/img/manual_permissions.gif b/doc/administration/auth/how_to_configure_ldap_gitlab_ee/img/manual_permissions.gif Binary files differnew file mode 100644 index 00000000000..29b28df1cbd --- /dev/null +++ b/doc/administration/auth/how_to_configure_ldap_gitlab_ee/img/manual_permissions.gif diff --git a/doc/administration/auth/how_to_configure_ldap_gitlab_ee/img/multi_login.gif b/doc/administration/auth/how_to_configure_ldap_gitlab_ee/img/multi_login.gif Binary files differnew file mode 100644 index 00000000000..d317add9837 --- /dev/null +++ b/doc/administration/auth/how_to_configure_ldap_gitlab_ee/img/multi_login.gif diff --git a/doc/administration/auth/how_to_configure_ldap_gitlab_ee/index.md b/doc/administration/auth/how_to_configure_ldap_gitlab_ee/index.md new file mode 100644 index 00000000000..4e0a0035790 --- /dev/null +++ b/doc/administration/auth/how_to_configure_ldap_gitlab_ee/index.md @@ -0,0 +1,119 @@ +--- +author: Chris Wilson +author_gitlab: MrChrisW +level: intermediary +article_type: admin guide +date: 2017-05-03 +--- + +# How to configure LDAP with GitLab EE + +## Introduction + +The present article follows [How to Configure LDAP with GitLab CE](../how_to_configure_ldap_gitlab_ce/index.md). Make sure to read through it before moving forward. + +## GitLab Enterprise Edition - LDAP features + +[GitLab Enterprise Edition (EE)](https://about.gitlab.com/pricing/) has a number of advantages when it comes to integrating with Active Directory (LDAP): + +- [Administrator Sync](../ldap-ee.md#administrator-sync): As an extension of group sync, you can automatically manage your global GitLab administrators. Specify a group CN for `admin_group` and all members of the LDAP group will be given administrator privileges. +- [Group Sync](#group-sync): This allows GitLab group membership to be automatically updated based on LDAP group members. +- [Multiple LDAP servers](#multiple-ldap-servers): The ability to configure multiple LDAP servers. This is useful if an organization has different LDAP servers within departments. This is not designed for failover. We're working on [supporting LDAP failover](https://gitlab.com/gitlab-org/gitlab-ee/issues/139) in GitLab. + +- Daily user synchronization: Once a day, GitLab will run a synchronization to check and update GitLab users against LDAP. This process updates all user details automatically. + +On the following section, you'll find a description for each of these features. Read through [LDAP GitLab EE docs](../ldap-ee.md) for complementary information. + + + +All members of the group `Global Admins` will be given **administrator** access to GitLab, allowing them to view the `/admin` dashboard. + +### Group Sync + +Group syncing allows AD (LDAP) groups to be mapped to GitLab groups. This provides more control over per-group user management. To configure group syncing edit the `group_base` **DN** (`'OU=Global Groups,OU=GitLab INT,DC=GitLab,DC=org'`). This **OU** contains all groups that will be associated with [GitLab groups](../../../user/group/index.md). + +#### Creating group links - example + +As an example, let's suppose we have a "UKGov" GitLab group, which deals with confidential government information. Therefore, it is important that users of this group are given the correct permissions to projects contained within the group. Granular group permissions can be applied based on the AD group. + +**UK Developers** of our "UKGov" group are given **"developer"** permissions. + +_The developer permission allows the development staff to effectively manage all project code, issues, and merge requests._ + +**UK Support** staff of our "UKGov" group are given **"reporter"** permissions. + +_The reporter permission allows support staff to manage issues, labels, and review project code._ + +**US People Ops** of our "UKGov" group are given **"guest"** permissions. + + + +> Guest permissions allows people ops staff to review and lodge new issues while allowing no read or write access to project code or [confidential issues](../../../user/project/issues/confidential_issues.md#permissions-and-access-to-confidential-issues) created by other users. + +See the [permission list](../../../user/permissions.md) for complementary info. + +#### Group permissions - example + +Considering the previous example, our staff will have +access to our GitLab instance with the following structure: + + + +Using this permission structure in our example allows only UK staff access to sensitive information stored in the projects code, while still allowing other teams to work effectively. As all permissions are controlled via AD groups new users can be quickly added to existing groups. New group members will then automatically inherit the required permissions. + +> [More information](../ldap-ee.md#group-sync) on group syncing. + +### Updating user permissions - new feature + +Since GitLab [v8.15](https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/822) LDAP user permissions can now be manually overridden by an admin user. To override a user's permissions visit the groups **Members** page and select **Edit permissions**. + + + +### Multiple LDAP servers + +GitLab EE can support multiple LDAP servers. Simply configure another server in the `gitlab.rb` file within the `ldap_servers` block. In the example below we configure a new secondary server with the label **GitLab Secondary AD**. This is shown on the GitLab login screen. Large enterprises often utilize multiple LDAP servers for segregating organizational departments. + + + +Considering the example illustrated on the image above, +our `gitlab.rb` configuration would look like: + +```ruby +gitlab_rails['ldap_enabled'] = true +gitlab_rails['ldap_servers'] = { +'main' => { + 'label' => 'GitLab AD', + 'host' => 'ad.example.org', + 'port' => 636, + 'uid' => 'sAMAccountName', + 'method' => 'ssl', + 'bind_dn' => 'CN=GitLabSRV,CN=Users,DC=GitLab,DC=org', + 'password' => 'Password1', + 'active_directory' => true, + 'base' => 'OU=GitLab INT,DC=GitLab,DC=org', + 'group_base' => 'OU=Global Groups,OU=GitLab INT,DC=GitLab,DC=org', + 'admin_group' => 'Global Admins' + }, + +'secondary' => { + 'label' => 'GitLab Secondary AD', + 'host' => 'ad-secondary.example.net', + 'port' => 636, + 'uid' => 'sAMAccountName', + 'method' => 'ssl', + 'bind_dn' => 'CN=GitLabSRV,CN=Users,DC=GitLab,DC=com', + 'password' => 'Password1', + 'active_directory' => true, + 'base' => 'OU=GitLab Secondary,DC=GitLab,DC=com', + 'group_base' => 'OU=Global Groups,OU=GitLab INT,DC=GitLab,DC=com', + 'admin_group' => 'Global Admins' + } +} +``` + +## Conclusion + +Integration of GitLab with Active Directory (LDAP) reduces the complexity of user management. +It has the advantage of improving user permission controls, whilst easing the deployment of GitLab into an existing [IT environment](https://www.techopedia.com/definition/29199/it-infrastructure). GitLab EE offers advanced group management and multiple LDAP servers. + +With the assistance of the [GitLab Support](https://about.gitlab.com/support) team, setting up GitLab with an existing AD/LDAP solution will be a smooth and painless process. diff --git a/doc/administration/auth/ldap-ee.md b/doc/administration/auth/ldap-ee.md new file mode 100644 index 00000000000..aab3367086d --- /dev/null +++ b/doc/administration/auth/ldap-ee.md @@ -0,0 +1,557 @@ +# LDAP Additions in GitLab EE **[STARTER ONLY]** + +This is a continuation of the main [LDAP documentation](ldap.md), detailing LDAP +features specific to GitLab Enterprise Edition Starter, Premium and Ultimate. + +## Overview + +[LDAP](https://en.wikipedia.org/wiki/Lightweight_Directory_Access_Protocol) +stands for **Lightweight Directory Access Protocol**, which +is a standard application protocol for +accessing and maintaining distributed directory information services +over an Internet Protocol (IP) network. + +GitLab integrates with LDAP to support **user authentication**. This integration +works with most LDAP-compliant directory servers, including Microsoft Active +Directory, Apple Open Directory, Open LDAP, and 389 Server. +**GitLab Enterprise Edition** includes enhanced integration, including group +membership syncing. + +## Use cases + +- User sync: Once a day, GitLab will update users against LDAP +- Group sync: Once an hour, GitLab will update group membership + based on LDAP group members + +## Multiple LDAP servers + +With GitLab Enterprise Edition Starter, you can configure multiple LDAP servers +that your GitLab instance will connect to. + +To add another LDAP server, you can start by duplicating the settings under +[the main configuration](ldap.md#configuration) and edit them to match the +additional LDAP server. + +Be sure to choose a different provider ID made of letters a-z and numbers 0-9. +This ID will be stored in the database so that GitLab can remember which LDAP +server a user belongs to. + +## User sync + +Once per day, GitLab will run a worker to check and update GitLab +users against LDAP. + +The process will execute the following access checks: + +1. Ensure the user is still present in LDAP +1. If the LDAP server is Active Directory, ensure the user is active (not + blocked/disabled state). This will only be checked if + `active_directory: true` is set in the LDAP configuration [^1] + +The user will be set to `ldap_blocked` state in GitLab if the above conditions +fail. This means the user will not be able to login or push/pull code. + +The process will also update the following user information: + +1. Email address +1. If `sync_ssh_keys` is set, SSH public keys +1. If Kerberos is enabled, Kerberos identity + +NOTE: **Note:** +The LDAP sync process updates existing users while new users will +be created on first sign in. + +## Group Sync + +If your LDAP supports the `memberof` property, GitLab will add the user to any +new groups they might be added to when the user logs in. That way they don't need +to wait for the hourly sync to be granted access to the groups that they are in +in LDAP. + +In GitLab Premium, we can also add a GitLab group to sync with one or multiple LDAP groups or we can +also add a filter. The filter must comply with the syntax defined in [RFC 2254](https://tools.ietf.org/search/rfc2254). + +A group sync process will run every hour on the hour, and `group_base` must be set +in LDAP configuration for LDAP synchronizations based on group CN to work. This allows +GitLab group membership to be automatically updated based on LDAP group members. + +The `group_base` configuration should be a base LDAP 'container', such as an +'organization' or 'organizational unit', that contains LDAP groups that should +be available to GitLab. For example, `group_base` could be +`ou=groups,dc=example,dc=com`. In the config file it will look like the +following. + +**Omnibus configuration** + +1. Edit `/etc/gitlab/gitlab.rb`: + + ```ruby + gitlab_rails['ldap_servers'] = YAML.load <<-EOS + main: + ## snip... + ## + ## Base where we can search for groups + ## + ## Ex. ou=groups,dc=gitlab,dc=example + ## + ## + group_base: ou=groups,dc=example,dc=com + EOS + ``` + +1. [Reconfigure GitLab][reconfigure] for the changes to take effect. + +**Source configuration** + +1. Edit `/home/git/gitlab/config/gitlab.yml`: + + ```yaml + production: + ldap: + servers: + main: + # snip... + group_base: ou=groups,dc=example,dc=com + ``` + +1. [Restart GitLab][restart] for the changes to take effect. + +--- + +To take advantage of group sync, group owners or maintainers will need to create an +LDAP group link in their group **Settings > LDAP Groups** page. Multiple LDAP +groups and/or filters can be linked with a single GitLab group. When the link is +created, an access level/role is specified (Guest, Reporter, Developer, Maintainer, +or Owner). + +## Administrator sync + +As an extension of group sync, you can automatically manage your global GitLab +administrators. Specify a group CN for `admin_group` and all members of the +LDAP group will be given administrator privileges. The configuration will look +like the following. + +NOTE: **Note:** +Administrators will not be synced unless `group_base` is also +specified alongside `admin_group`. Also, only specify the CN of the admin +group, as opposed to the full DN. + +**Omnibus configuration** + +1. Edit `/etc/gitlab/gitlab.rb`: + + ```ruby + gitlab_rails['ldap_servers'] = YAML.load <<-EOS + main: + ## snip... + ## + ## Base where we can search for groups + ## + ## Ex. ou=groups,dc=gitlab,dc=example + ## + ## + group_base: ou=groups,dc=example,dc=com + + ## + ## The CN of a group containing GitLab administrators + ## + ## Ex. administrators + ## + ## Note: Not `cn=administrators` or the full DN + ## + ## + admin_group: my_admin_group + + EOS + ``` + +1. [Reconfigure GitLab][reconfigure] for the changes to take effect. + +**Source configuration** + +1. Edit `/home/git/gitlab/config/gitlab.yml`: + + ```yaml + production: + ldap: + servers: + main: + # snip... + group_base: ou=groups,dc=example,dc=com + admin_group: my_admin_group + ``` + +1. [Restart GitLab][restart] for the changes to take effect. + +## Adjusting LDAP user sync schedule + +> Introduced in GitLab Enterprise Edition Starter. + +NOTE: **Note:** +These are cron formatted values. You can use a crontab generator to create +these values, for example http://www.crontabgenerator.com/. + +By default, GitLab will run a worker once per day at 01:30 a.m. server time to +check and update GitLab users against LDAP. + +You can manually configure LDAP user sync times by setting the +following configuration values. The example below shows how to set LDAP user +sync to run once every 12 hours at the top of the hour. + +**Omnibus installations** + +1. Edit `/etc/gitlab/gitlab.rb`: + + ```ruby + gitlab_rails['ldap_sync_worker_cron'] = "0 */12 * * *" + ``` + +1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. + +**Source installations** + +1. Edit `config/gitlab.yaml`: + + ```yaml + cron_jobs: + ldap_sync_worker_cron: + "0 */12 * * *" + ``` + +1. [Restart GitLab](../restart_gitlab.md#installations-from-source) for the changes to take effect. + +## Adjusting LDAP group sync schedule + +NOTE: **Note:** +These are cron formatted values. You can use a crontab generator to create +these values, for example http://www.crontabgenerator.com/. + +By default, GitLab will run a group sync process every hour, on the hour. + +CAUTION: **Important:** +It's recommended that you do not run too short intervals as this +could lead to multiple syncs running concurrently. This is primarily a concern +for installations with a large number of LDAP users. Please review the +[LDAP group sync benchmark metrics](#benchmarks) to see how +your installation compares before proceeding. + +You can manually configure LDAP group sync times by setting the +following configuration values. The example below shows how to set group +sync to run once every 2 hours at the top of the hour. + +**Omnibus installations** + +1. Edit `/etc/gitlab/gitlab.rb`: + + ```ruby + gitlab_rails['ldap_group_sync_worker_cron'] = "0 */2 * * * *" + ``` + +1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. + +**Source installations** + +1. Edit `config/gitlab.yaml`: + + ```yaml + cron_jobs: + ldap_group_sync_worker_cron: + "*/30 * * * *" + ``` + +1. [Restart GitLab](../restart_gitlab.md#installations-from-source) for the changes to take effect. + +## External groups + +> Introduced in GitLab Enterprise Edition Starter 8.9. + +Using the `external_groups` setting will allow you to mark all users belonging +to these groups as [external users](../../user/permissions.md#external-users-permissions). +Group membership is checked periodically through the `LdapGroupSync` background +task. + +**Omnibus configuration** + +1. Edit `/etc/gitlab/gitlab.rb`: + + ```ruby + gitlab_rails['ldap_servers'] = YAML.load <<-EOS + main: + ## snip... + ## + ## An array of CNs of groups containing users that should be considered external + ## + ## Ex. ['interns', 'contractors'] + ## + ## Note: Not `cn=interns` or the full DN + ## + external_groups: ['interns', 'contractors'] + EOS + ``` + +1. [Reconfigure GitLab][reconfigure] for the changes to take effect. + +**Source configuration** + +1. Edit `config/gitlab.yaml`: + + ```yaml + production: + ldap: + servers: + main: + # snip... + external_groups: ['interns', 'contractors'] + ``` + +1. [Restart GitLab][restart] for the changes to take effect. + +## Group sync technical details + +There is a lot going on with group sync 'under the hood'. This section +outlines what LDAP queries are executed and what behavior you can expect +from group sync. + +Group member access will be downgraded from a higher level if their LDAP group +membership changes. For example, if a user has 'Owner' rights in a group and the +next group sync reveals they should only have 'Developer' privileges, their +access will be adjusted accordingly. The only exception is if the user is the +*last* owner in a group. Groups need at least one owner to fulfill +administrative duties. + +### Supported LDAP group types/attributes + +GitLab supports LDAP groups that use member attributes `member`, `submember`, +`uniquemember`, `memberof` and `memberuid`. This means group sync supports, at +least, LDAP groups with object class `groupOfNames`, `posixGroup`, and +`groupOfUniqueName`. Other object classes should work fine as long as members +are defined as one of the mentioned attributes. This also means GitLab supports +Microsoft Active Directory, Apple Open Directory, Open LDAP, and 389 Server. +Other LDAP servers should work, too. + +Active Directory also supports nested groups. Group sync will recursively +resolve membership if `active_directory: true` is set in the configuration file. + +> **Note:** Nested group membership will only be resolved if the nested group + also falls within the configured `group_base`. For example, if GitLab sees a + nested group with DN `cn=nested_group,ou=special_groups,dc=example,dc=com` but + the configured `group_base` is `ou=groups,dc=example,dc=com`, `cn=nested_group` + will be ignored. + +### Queries + +- Each LDAP group is queried a maximum of one time with base `group_base` and + filter `(cn=<cn_from_group_link>)`. +- If the LDAP group has the `memberuid` attribute, GitLab will execute another + LDAP query per member to obtain each user's full DN. These queries are + executed with base `base`, scope 'base object', and a filter depending on + whether `user_filter` is set. Filter may be `(uid=<uid_from_group>)` or a + joining of `user_filter`. + +### Benchmarks + +Group sync was written to be as performant as possible. Data is cached, database +queries are optimized, and LDAP queries are minimized. The last benchmark run +revealed the following metrics: + +For 20,000 LDAP users, 11,000 LDAP groups and 1,000 GitLab groups with 10 +LDAP group links each: + +- Initial sync (no existing members assigned in GitLab) took 1.8 hours +- Subsequent syncs (checking membership, no writes) took 15 minutes + +These metrics are meant to provide a baseline and performance may vary based on +any number of factors. This was a pretty extreme benchmark and most instances will +not have near this many users or groups. Disk speed, database performance, +network and LDAP server response time will affect these metrics. + +## Troubleshooting + +### Referral error + +If you see `LDAP search error: Referral` in the logs, or when troubleshooting +LDAP Group Sync, this error may indicate a configuration problem. The LDAP +configuration `/etc/gitlab/gitlab.rb` (Omnibus) or `config/gitlab.yml` (source) +is in YAML format and is sensitive to indentation. Check that `group_base` and +`admin_group` configuration keys are indented 2 spaces past the server +identifier. The default identifier is `main` and an example snippet looks like +the following: + +```yaml +main: # 'main' is the GitLab 'provider ID' of this LDAP server + label: 'LDAP' + host: 'ldap.example.com' + ... + group_base: 'cn=my_group,ou=groups,dc=example,dc=com' + admin_group: 'my_admin_group' +``` + +[reconfigure]: ../restart_gitlab.md#omnibus-gitlab-reconfigure +[restart]: ../restart_gitlab.md#installations-from-source + +[^1]: In Active Directory, a user is marked as disabled/blocked if the user + account control attribute (`userAccountControl:1.2.840.113556.1.4.803`) + has bit 2 set. See https://ctogonewild.com/2009/09/03/bitmask-searches-in-ldap/ + for more information. + +### User DN has changed + +When an LDAP user is created in GitLab, their LDAP DN is stored for later reference. + +If GitLab cannot find a user by their DN, it will attempt to fallback +to finding the user by their email. If the lookup is successful, GitLab will +update the stored DN to the new value. + +### User is not being added to a group + +Sometimes you may think a particular user should be added to a GitLab group via +LDAP group sync, but for some reason it's not happening. There are several +things to check to debug the situation. + +- Ensure LDAP configuration has a `group_base` specified. This configuration is + required for group sync to work properly. +- Ensure the correct LDAP group link is added to the GitLab group. Check group + links by visiting the GitLab group, then **Settings dropdown -> LDAP groups**. +- Check that the user has an LDAP identity + 1. Sign in to GitLab as an administrator user. + 1. Navigate to **Admin area -> Users**. + 1. Search for the user + 1. Open the user, by clicking on their name. Do not click 'Edit'. + 1. Navigate to the **Identities** tab. There should be an LDAP identity with + an LDAP DN as the 'Identifier'. + +If all of the above looks good, jump in to a little more advanced debugging. +Often, the best way to learn more about why group sync is behaving a certain +way is to enable debug logging. There is verbose output that details every +step of the sync. + +1. Start a Rails console + + ```bash + # For Omnibus installations + sudo gitlab-rails console + + # For installations from source + sudo -u git -H bundle exec rails console production + ``` +1. Set the log level to debug (only for this session): + + ```ruby + Rails.logger.level = Logger::DEBUG + ``` +1. Choose a GitLab group to test with. This group should have an LDAP group link + already configured. If the output is `nil`, the group could not be found. + If a bunch of group attributes are output, your group was found successfully. + + ```ruby + group = Group.find_by(name: 'my_group') + + # Output + => #<Group:0x007fe825196558 id: 1234, name: "my_group"...> + ``` +1. Run a group sync for this particular group. + + ```ruby + EE::Gitlab::Auth::LDAP::Sync::Group.execute_all_providers(group) + ``` +1. Look through the output of the sync. See [example log output](#example-log-output) + below for more information about the output. +1. If you still aren't able to see why the user isn't being added, query the + LDAP group directly to see what members are listed. Still in the Rails console, + run the following query: + + ```ruby + adapter = Gitlab::Auth::LDAP::Adapter.new('ldapmain') # If `main` is the LDAP provider + ldap_group = EE::Gitlab::Auth::LDAP::Group.find_by_cn('group_cn_here', adapter) + + # Output + => #<EE::Gitlab::Auth::LDAP::Group:0x007fcbdd0bb6d8 + ``` +1. Query the LDAP group's member DNs and see if the user's DN is in the list. + One of the DNs here should match the 'Identifier' from the LDAP identity + checked earlier. If it doesn't, the user does not appear to be in the LDAP + group. + + ```ruby + ldap_group.member_dns + + # Output + => ["uid=john,ou=people,dc=example,dc=com", "uid=mary,ou=people,dc=example,dc=com"] + ``` +1. Some LDAP servers don't store members by DN. Rather, they use UIDs instead. + If you didn't see results from the last query, try querying by UIDs instead. + + ```ruby + ldap_group.member_uids + + # Output + => ['john','mary'] + ``` + +#### Example log output + +The output of the last command will be very verbose, but contains lots of +helpful information. For the most part you can ignore log entries that are SQL +statements. + +Indicates the point where syncing actually begins: + +```bash +Started syncing all providers for 'my_group' group +``` + +The follow entry shows an array of all user DNs GitLab sees in the LDAP server. +Note that these are the users for a single LDAP group, not a GitLab group. If +you have multiple LDAP groups linked to this GitLab group, you will see multiple +log entries like this - one for each LDAP group. If you don't see an LDAP user +DN in this log entry, LDAP is not returning the user when we do the lookup. +Verify the user is actually in the LDAP group. + +```bash +Members in 'ldap_group_1' LDAP group: ["uid=john0,ou=people,dc=example,dc=com", +"uid=mary0,ou=people,dc=example,dc=com", "uid=john1,ou=people,dc=example,dc=com", +"uid=mary1,ou=people,dc=example,dc=com", "uid=john2,ou=people,dc=example,dc=com", +"uid=mary2,ou=people,dc=example,dc=com", "uid=john3,ou=people,dc=example,dc=com", +"uid=mary3,ou=people,dc=example,dc=com", "uid=john4,ou=people,dc=example,dc=com", +"uid=mary4,ou=people,dc=example,dc=com"] +``` + +Shortly after each of the above entries, you will see a hash of resolved member +access levels. This hash represents all user DNs GitLab thinks should have +access to this group, and at which access level (role). This hash is additive, +and more DNs may be added, or existing entries modified, based on additional +LDAP group lookups. The very last occurrence of this entry should indicate +exactly which users GitLab believes should be added to the group. + +> **Note:** 10 is 'Guest', 20 is 'Reporter', 30 is 'Developer', 40 is 'Maintainer' + and 50 is 'Owner' + +```bash +Resolved 'my_group' group member access: {"uid=john0,ou=people,dc=example,dc=com"=>30, +"uid=mary0,ou=people,dc=example,dc=com"=>30, "uid=john1,ou=people,dc=example,dc=com"=>30, +"uid=mary1,ou=people,dc=example,dc=com"=>30, "uid=john2,ou=people,dc=example,dc=com"=>30, +"uid=mary2,ou=people,dc=example,dc=com"=>30, "uid=john3,ou=people,dc=example,dc=com"=>30, +"uid=mary3,ou=people,dc=example,dc=com"=>30, "uid=john4,ou=people,dc=example,dc=com"=>30, +"uid=mary4,ou=people,dc=example,dc=com"=>30} +``` + +It's not uncommon to see warnings like the following. These indicate that GitLab +would have added the user to a group, but the user could not be found in GitLab. +Usually this is not a cause for concern. + +If you think a particular user should already exist in GitLab, but you're seeing +this entry, it could be due to a mismatched DN stored in GitLab. See +[User DN has changed](#User-DN-has-changed) to update the user's LDAP identity. + +```bash +User with DN `uid=john0,ou=people,dc=example,dc=com` should have access +to 'my_group' group but there is no user in GitLab with that +identity. Membership will be updated once the user signs in for +the first time. +``` + +Finally, the following entry says syncing has finished for this group: + +```bash +Finished syncing all providers for 'my_group' group +``` diff --git a/doc/administration/auth/ldap.md b/doc/administration/auth/ldap.md index 440c2b1285a..90406bf0f5b 100644 --- a/doc/administration/auth/ldap.md +++ b/doc/administration/auth/ldap.md @@ -14,7 +14,7 @@ including group membership syncing as well as multiple LDAP servers support. The information on this page is relevant for both GitLab CE and EE. For more details about EE-specific LDAP features, see the -[LDAP Enterprise Edition documentation](https://docs.gitlab.com/ee/administration/auth/ldap-ee.html). +[LDAP Enterprise Edition documentation](ldap-ee.md). ## Security @@ -39,7 +39,7 @@ immediately block all access. NOTE: **Note**: GitLab Enterprise Edition Starter supports a -[configurable sync time](https://docs.gitlab.com/ee/administration/auth/ldap-ee.html#adjusting-ldap-user-and-group-sync-schedules), +[configurable sync time](ldap-ee.md#adjusting-ldap-user-sync-schedule), with a default of one hour. ## Git password authentication @@ -56,6 +56,7 @@ to connect to one GitLab server. For a complete guide on configuring LDAP with GitLab Community Edition, please check the admin guide [How to configure LDAP with GitLab CE](how_to_configure_ldap_gitlab_ce/index.md). +For GitLab Enterprise Editions, see also [How to configure LDAP with GitLab EE](how_to_configure_ldap_gitlab_ee/index.md). To enable LDAP integration you need to add your LDAP server settings in `/etc/gitlab/gitlab.rb` or `/home/git/gitlab/config/gitlab.yml` for Omnibus @@ -380,7 +381,7 @@ group, you can use the following syntax: Find more information about this "LDAP_MATCHING_RULE_IN_CHAIN" filter at <https://docs.microsoft.com/en-us/windows/desktop/ADSI/search-filter-syntax>. Support for nested members in the user filter should not be confused with -[group sync nested groups support (EE only)](https://docs.gitlab.com/ee/administration/auth/ldap-ee.html#supported-ldap-group-types-attributes). +[group sync nested groups support (EE only)](ldap-ee.md#supported-ldap-group-typesattributes). Please note that GitLab does not support the custom filter syntax used by omniauth-ldap. diff --git a/doc/administration/auth/smartcard.md b/doc/administration/auth/smartcard.md new file mode 100644 index 00000000000..1107b955c4a --- /dev/null +++ b/doc/administration/auth/smartcard.md @@ -0,0 +1,186 @@ +# Smartcard authentication + +GitLab supports authentication using smartcards. + +## Authentication methods + +GitLab supports two authentication methods: + +- X.509 certificates with local databases. +- LDAP servers. + +### Authentication against a local database with X.509 certificates + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/issues/726) in +[GitLab Premium](https://about.gitlab.com/pricing/) 11.6 as an experimental +feature. Smartcard authentication against local databases may change or be +removed completely in future releases. + +Smartcards with X.509 certificates can be used to authenticate with GitLab. + +To use a smartcard with an X.509 certificate to authenticate against a local +database with GitLab, `CN` and `emailAddress` must be defined in the +certificate. For example: + +``` +Certificate: + Data: + Version: 1 (0x0) + Serial Number: 12856475246677808609 (0xb26b601ecdd555e1) + Signature Algorithm: sha256WithRSAEncryption + Issuer: O=Random Corp Ltd, CN=Random Corp + Validity + Not Before: Oct 30 12:00:00 2018 GMT + Not After : Oct 30 12:00:00 2019 GMT + Subject: CN=Gitlab User, emailAddress=gitlab-user@example.com +``` + +### Authentication against an LDAP server + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/issues/7693) in +[GitLab Premium](https://about.gitlab.com/pricing/) 11.8 as an experimental +feature. Smartcard authentication against an LDAP server may change or be +removed completely in future releases. + +GitLab implements a standard way of certificate matching following +[RFC4523](https://tools.ietf.org/html/rfc4523). It uses the +`certificateExactMatch` certificate matching rule against the `userCertificate` +attribute. As a prerequisite, you must use an LDAP server that: + +- Supports the `certificateExactMatch` matching rule. +- Has the certificate stored in the `userCertificate` attribute. + +## Configure GitLab for smartcard authentication + +**For Omnibus installations** + +1. Edit `/etc/gitlab/gitlab.rb`: + + ```ruby + gitlab_rails['smartcard_enabled'] = true + gitlab_rails['smartcard_ca_file'] = "/etc/ssl/certs/CA.pem" + gitlab_rails['smartcard_client_certificate_required_port'] = 3444 + ``` + +1. Save the file and [reconfigure](../restart_gitlab.md#omnibus-gitlab-reconfigure) + GitLab for the changes to take effect. + +--- + +**For installations from source** + +1. Configure NGINX to request a client side certificate + + In NGINX configuration, an **additional** server context must be defined with + the same configuration except: + + - The additional NGINX server context must be configured to run on a different + port: + + ``` + listen *:3444 ssl; + ``` + + - The additional NGINX server context must be configured to require the client + side certificate: + + ``` + ssl_verify_depth 2; + ssl_client_certificate /etc/ssl/certs/CA.pem; + ssl_verify_client on; + ``` + + - The additional NGINX server context must be configured to forward the client + side certificate: + + ``` + proxy_set_header X-SSL-Client-Certificate $ssl_client_escaped_cert; + ``` + + For example, the following is an example server context in an NGINX + configuration file (eg. in `/etc/nginx/sites-available/gitlab-ssl`): + + ``` + server { + listen *:3444 ssl; + + # certificate for configuring SSL + ssl_certificate /path/to/example.com.crt; + ssl_certificate_key /path/to/example.com.key; + + ssl_verify_depth 2; + # CA certificate for client side certificate verification + ssl_client_certificate /etc/ssl/certs/CA.pem; + ssl_verify_client on; + + location / { + proxy_set_header Host $http_host; + proxy_set_header X-Real-IP $remote_addr; + proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; + proxy_set_header X-Forwarded-Proto $scheme; + proxy_set_header Upgrade $http_upgrade; + proxy_set_header Connection $connection_upgrade; + + proxy_set_header X-SSL-Client-Certificate $ssl_client_escaped_cert; + + proxy_read_timeout 300; + + proxy_pass http://gitlab-workhorse; + } + } + ``` + +1. Edit `config/gitlab.yml`: + + ```yaml + ## Smartcard authentication settings + smartcard: + # Allow smartcard authentication + enabled: true + + # Path to a file containing a CA certificate + ca_file: '/etc/ssl/certs/CA.pem' + + # Port where the client side certificate is requested by NGINX + client_certificate_required_port: 3444 + ``` + +1. Save the file and [restart](../restart_gitlab.md#installations-from-source) + GitLab for the changes to take effect. + +### Additional steps when authenticating against an LDAP server + +**For Omnibus installations** + +1. Edit `/etc/gitlab/gitlab.rb`: + + ```ruby + gitlab_rails['ldap_servers'] = YAML.load <<-EOS + main: + # snip... + # Enable smartcard authentication against the LDAP server. Valid values + # are "false", "optional", and "required". + smartcard_auth: optional + EOS + ``` + +1. Save the file and [reconfigure](../restart_gitlab.md#omnibus-gitlab-reconfigure) + GitLab for the changes to take effect. + +**For installations from source** + +1. Edit `config/gitlab.yml`: + + ```yaml + production: + ldap: + servers: + main: + # snip... + # Enable smartcard authentication against the LDAP server. Valid values + # are "false", "optional", and "required". + smartcard_auth: optional + ``` + +1. Save the file and [restart](../restart_gitlab.md#installations-from-source) + GitLab for the changes to take effect. diff --git a/doc/administration/compliance.md b/doc/administration/compliance.md index 72cb57fb36c..246addb6dc9 100644 --- a/doc/administration/compliance.md +++ b/doc/administration/compliance.md @@ -11,8 +11,8 @@ GitLab’s [security features](../security/README.md) may also help you meet rel |**[Enforce TOS acceptance](../user/admin_area/settings/terms.md)**<br>Enforce your users accepting new terms of service by blocking GitLab traffic.|Core+|| |**[Email all users of a project, group, or entire server](../user/admin_area/settings/terms.md)**<br>An admin can email groups of users based on project or group membership, or email everyone using the GitLab instance. This is great for scheduled maintenance or upgrades.|Starter+|| |**[Omnibus package supports log forwarding](https://docs.gitlab.com/omnibus/settings/logs.html#udp-log-forwarding)**<br>Forward your logs to a central system.|Starter+|| -|**[Lock project membership to group](https://docs.gitlab.com/ee/user/group/index.html#member-lock-starter)**<br>Group owners can prevent new members from being added to projects within a group.|Starter+|✓| -|**[LDAP group sync](https://docs.gitlab.com/ee/administration/auth/ldap-ee.html#group-sync)**<br>GitLab Enterprise Edition gives admins the ability to automatically sync groups and manage SSH keys, permissions, and authentication, so you can focus on building your product, not configuring your tools.|Starter+|| -|**[LDAP group sync filters](https://docs.gitlab.com/ee/administration/auth/ldap-ee.html#group-sync)**<br>GitLab Enterprise Edition Premium gives more flexibility to synchronize with LDAP based on filters, meaning you can leverage LDAP attributes to map GitLab permissions.|Premium+|| -|**[Audit logs](https://docs.gitlab.com/ee/administration/audit_events.html)**<br>To maintain the integrity of your code, GitLab Enterprise Edition Premium gives admins the ability to view any modifications made within the GitLab server in an advanced audit log system, so you can control, analyze and track every change.|Premium+|| -|**[Auditor users](https://docs.gitlab.com/ee/administration/auditor_users.html)**<br>Auditor users are users who are given read-only access to all projects, groups, and other resources on the GitLab instance.|Premium+||
\ No newline at end of file +|**[Lock project membership to group](../user/group/index.md#member-lock-starter)**<br>Group owners can prevent new members from being added to projects within a group.|Starter+|✓| +|**[LDAP group sync](auth/ldap-ee.md#group-sync)**<br>GitLab Enterprise Edition gives admins the ability to automatically sync groups and manage SSH keys, permissions, and authentication, so you can focus on building your product, not configuring your tools.|Starter+|| +|**[LDAP group sync filters](auth/ldap-ee.md#group-sync)**<br>GitLab Enterprise Edition Premium gives more flexibility to synchronize with LDAP based on filters, meaning you can leverage LDAP attributes to map GitLab permissions.|Premium+|| +|**[Audit logs](audit_events.md)**<br>To maintain the integrity of your code, GitLab Enterprise Edition Premium gives admins the ability to view any modifications made within the GitLab server in an advanced audit log system, so you can control, analyze and track every change.|Premium+|| +|**[Auditor users](auditor_users.md)**<br>Auditor users are users who are given read-only access to all projects, groups, and other resources on the GitLab instance.|Premium+|| diff --git a/doc/administration/custom_hooks.md b/doc/administration/custom_hooks.md index 60ad4bf4e8f..da661b7f121 100644 --- a/doc/administration/custom_hooks.md +++ b/doc/administration/custom_hooks.md @@ -1,11 +1,13 @@ # Custom Git Hooks > **Note:** Custom Git hooks must be configured on the filesystem of the GitLab -server. Only GitLab server administrators will be able to complete these tasks. -Please explore [webhooks] and [CI] as an option if you do not -have filesystem access. For a user configurable Git hook interface, see -[Push Rules](https://docs.gitlab.com/ee/push_rules/push_rules.html), -available in GitLab Enterprise Edition. +> server. Only GitLab server administrators will be able to complete these tasks. +> Please explore [webhooks] and [CI] as an option if you do not +> have filesystem access. For a user configurable Git hook interface, see +> [Push Rules](https://docs.gitlab.com/ee/push_rules/push_rules.html), +> available in GitLab Enterprise Edition. +> +> **Note:** Custom Git hooks won't be replicated to secondary nodes if you use [GitLab Geo][gitlab-geo] Git natively supports hooks that are executed on different actions. Examples of server-side git hooks include pre-receive, post-receive, and update. @@ -85,5 +87,6 @@ STDERR takes precedence over STDOUT. [CI]: ../ci/README.md [hooks]: https://git-scm.com/book/en/v2/Customizing-Git-Git-Hooks#Server-Side-Hooks [webhooks]: ../user/project/integrations/webhooks.md +[gitlab-geo]: ../administration/geo/replication/index.md [5073]: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/5073 [93]: https://gitlab.com/gitlab-org/gitlab-shell/merge_requests/93 diff --git a/doc/administration/database_load_balancing.md b/doc/administration/database_load_balancing.md new file mode 100644 index 00000000000..7f3be402b84 --- /dev/null +++ b/doc/administration/database_load_balancing.md @@ -0,0 +1,277 @@ +# Database Load Balancing **[PREMIUM ONLY]** + +> [Introduced][ee-1283] in [GitLab Premium][eep] 9.0. + +Distribute read-only queries among multiple database servers. + +## Overview + +Database load balancing improves the distribution of database workloads across +multiple computing resources. Load balancing aims to optimize resource use, +maximize throughput, minimize response time, and avoid overload of any single +resource. Using multiple components with load balancing instead of a single +component may increase reliability and availability through redundancy. +[_Wikipedia article_][wikipedia] + +When database load balancing is enabled in GitLab, the load is balanced using +a simple round-robin algorithm, without any external dependencies such as Redis. +Load balancing is not enabled for Sidekiq as this would lead to consistency +problems, and Sidekiq mostly performs writes anyway. + +In the following image, you can see the load is balanced rather evenly among +all the secondaries (`db4`, `db5`, `db6`). Because `SELECT` queries are not +sent to the primary (unless necessary), the primary (`db3`) hardly has any load. + + + +## Requirements + +For load balancing to work you will need at least PostgreSQL 9.2 or newer, +[**MySQL is not supported**][db-req]. You also need to make sure that you have +at least 1 secondary in [hot standby][hot-standby] mode. + +Load balancing also requires that the configured hosts **always** point to the +primary, even after a database failover. Furthermore, the additional hosts to +balance load among must **always** point to secondary databases. This means that +you should put a load balance in front of every database, and have GitLab connect +to those load balancers. + +For example, say you have a primary (`db1.gitlab.com`) and two secondaries, +`db2.gitlab.com` and `db3.gitlab.com`. For this setup you will need to have 3 +load balancers, one for every host. For example: + +* `primary.gitlab.com` forwards to `db1.gitlab.com` +* `secondary1.gitlab.com` forwards to `db2.gitlab.com` +* `secondary2.gitlab.com` forwards to `db3.gitlab.com` + +Now let's say that a failover happens and db2 becomes the new primary. This +means forwarding should now happen as follows: + +* `primary.gitlab.com` forwards to `db2.gitlab.com` +* `secondary1.gitlab.com` forwards to `db1.gitlab.com` +* `secondary2.gitlab.com` forwards to `db3.gitlab.com` + +GitLab does not take care of this for you, so you will need to do so yourself. + +Finally, load balancing requires that GitLab can connect to all hosts using the +same credentials and port as configured in the +[Enabling load balancing](#enabling-load-balancing) section. Using +different ports or credentials for different hosts is not supported. + +## Use cases + +- For GitLab instances with thousands of users and high traffic, you can use + database load balancing to reduce the load on the primary database and + increase responsiveness, thus resulting in faster page load inside GitLab. + +## Enabling load balancing + +For the environment in which you want to use load balancing, you'll need to add +the following. This will balance the load between `host1.example.com` and +`host2.example.com`. + +**In Omnibus installations:** + +1. Edit `/etc/gitlab/gitlab.rb` and add the following line: + + ```ruby + gitlab_rails['db_load_balancing'] = { 'hosts' => ['host1.example.com', 'host2.example.com'] } + ``` + +1. Save the file and [reconfigure GitLab][] for the changes to take effect. + +--- + +**In installations from source:** + +1. Edit `/home/git/gitlab/config/database.yml` and add or amend the following lines: + + ```yaml + production: + username: gitlab + database: gitlab + encoding: unicode + load_balancing: + hosts: + - host1.example.com + - host2.example.com + ``` + +1. Save the file and [restart GitLab][] for the changes to take effect. + +## Service Discovery + +> [Introduced][ee-5883] in [GitLab Premium][eep] 11.0. + +Service discovery allows GitLab to automatically retrieve a list of secondary +databases to use, instead of having to manually specify these in the +`database.yml` configuration file. Service discovery works by periodically +checking a DNS A record, using the IPs returned by this record as the addresses +for the secondaries. For service discovery to work, all you need is a DNS server +and an A record containing the IP addresses of your secondaries. + +To use service discovery you need to change your `database.yml` configuration +file so it looks like the following: + +```yaml +production: + username: gitlab + database: gitlab + encoding: unicode + load_balancing: + discover: + nameserver: localhost + record: secondary.postgresql.service.consul + port: 8600 + interval: 60 + disconnect_timeout: 120 +``` + +Here the `discover:` section specifies the configuration details to use for +service discovery. + +### Configuration + +The following options can be set: + +| Option | Description | Default | +|----------------------|---------------------------------------------------------------------------------------------------|-----------| +| `nameserver` | The nameserver to use for looking up the DNS record. | localhost | +| `record` | The A record to look up. This option is required for service discovery to work. | | +| `port` | The port of the nameserver. | 8600 | +| `interval` | The minimum time in seconds between checking the DNS record. | 60 | +| `disconnect_timeout` | The time in seconds after which an old connection is closed, after the list of hosts was updated. | 120 | +| `use_tcp` | Lookup DNS resources using TCP instead of UDP | false | + +The `interval` value specifies the _minimum_ time between checks. If the A +record has a TTL greater than this value, then service discovery will honor said +TTL. For example, if the TTL of the A record is 90 seconds, then service +discovery will wait at least 90 seconds before checking the A record again. + +When the list of hosts is updated, it might take a while for the old connections +to be terminated. The `disconnect_timeout` setting can be used to enforce an +upper limit on the time it will take to terminate all old database connections. + +Some nameservers (like [Consul][consul-udp]) can return a truncated list of hosts when +queried over UDP. To overcome this issue, you can use TCP for querying by setting +`use_tcp` to `true`. + +### Forking + +If you use an application server that forks, such as Unicorn, you _have to_ +update your Unicorn configuration to start service discovery _after_ a fork. +Failure to do so will lead to service discovery only running in the parent +process. If you are using Unicorn, then you can add the following to your +Unicorn configuration file: + +```ruby +after_fork do |server, worker| + defined?(Gitlab::Database::LoadBalancing) && + Gitlab::Database::LoadBalancing.start_service_discovery +end +``` + +This will ensure that service discovery is started in both the parent and all +child processes. + +## Balancing queries + +Read-only `SELECT` queries will be balanced among all the secondary hosts. +Everything else (including transactions) will be executed on the primary. +Queries such as `SELECT ... FOR UPDATE` are also executed on the primary. + +## Prepared statements + +Prepared statements don't work well with load balancing and are disabled +automatically when load balancing is enabled. This should have no impact on +response timings. + +## Primary sticking + +After a write has been performed, GitLab will stick to using the primary for a +certain period of time, scoped to the user that performed the write. GitLab will +revert back to using secondaries when they have either caught up, or after 30 +seconds. + +## Failover handling + +In the event of a failover or an unresponsive database, the load balancer will +try to use the next available host. If no secondaries are available the +operation is performed on the primary instead. + +In the event of a connection error being produced when writing data, the +operation will be retried up to 3 times using an exponential back-off. + +When using load balancing, you should be able to safely restart a database server +without it immediately leading to errors being presented to the users. + +## Logging + +The load balancer logs various messages, such as: + +* When a host is marked as offline +* When a host comes back online +* When all secondaries are offline + +Each log message contains the tag `[DB-LB]` to make searching/filtering of such +log entries easier. For example: + +``` +[DB-LB] Host 10.123.2.5 came back online +[DB-LB] Marking host 10.123.2.7 as offline +[DB-LB] Marking host 10.123.2.7 as offline +[DB-LB] Marking host 10.123.2.7 as offline +[DB-LB] Marking host 10.123.2.7 as offline +[DB-LB] Marking host 10.123.2.7 as offline +[DB-LB] Host 10.123.2.6 came back online +[DB-LB] Marking host 10.123.2.7 as offline +[DB-LB] Marking host 10.123.2.7 as offline +[DB-LB] Marking host 10.123.2.7 as offline +[DB-LB] Host 10.123.2.7 came back online +[DB-LB] Host 10.123.2.7 came back online +``` + +## Handling Stale Reads + +> [Introduced][ee-3526] in [GitLab Premium][eep] 10.3. + +To prevent reading from an outdated secondary the load balancer will check if it +is in sync with the primary. If the data is determined to be recent enough the +secondary can be used, otherwise it will be ignored. To reduce the overhead of +these checks we only perform these checks at certain intervals. + +There are three configuration options that influence this behaviour: + +| Option | Description | Default | +|------------------------------|----------------------------------------------------------------------------------------------------------------|------------| +| `max_replication_difference` | The amount of data (in bytes) a secondary is allowed to lag behind when it hasn't replicated data for a while. | 8 MB | +| `max_replication_lag_time` | The maximum number of seconds a secondary is allowed to lag behind before we stop using it. | 60 seconds | +| `replica_check_interval` | The minimum number of seconds we have to wait before checking the status of a secondary. | 60 seconds | + +The defaults should be sufficient for most users. Should you want to change them +you can specify them in `config/database.yml` like so: + +```yaml +production: + username: gitlab + database: gitlab + encoding: unicode + load_balancing: + hosts: + - host1.example.com + - host2.example.com + max_replication_difference: 16777216 # 16 MB + max_replication_lag_time: 30 + replica_check_interval: 30 +``` + +[hot-standby]: https://www.postgresql.org/docs/9.6/static/hot-standby.html +[ee-1283]: https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/1283 +[eep]: https://about.gitlab.com/pricing/ +[reconfigure gitlab]: restart_gitlab.md#omnibus-gitlab-reconfigure "How to reconfigure Omnibus GitLab" +[restart gitlab]: restart_gitlab.md#installations-from-source "How to restart GitLab" +[wikipedia]: https://en.wikipedia.org/wiki/Load_balancing_(computing) +[db-req]: ../install/requirements.md#database +[ee-3526]: https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/3526 +[ee-5883]: https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/5883 +[consul-udp]: https://www.consul.io/docs/agent/dns.html#udp-based-dns-queries diff --git a/doc/administration/geo/disaster_recovery/background_verification.md b/doc/administration/geo/disaster_recovery/background_verification.md new file mode 100644 index 00000000000..7d2fd51f834 --- /dev/null +++ b/doc/administration/geo/disaster_recovery/background_verification.md @@ -0,0 +1,172 @@ +# Automatic background verification **[PREMIUM ONLY]** + +NOTE: **Note:** +Automatic background verification of repositories and wikis was added in +GitLab EE 10.6 but is enabled by default only on GitLab EE 11.1. You can +disable or enable this feature manually by following +[these instructions](#disabling-or-enabling-the-automatic-background-verification). + +Automatic background verification ensures that the transferred data matches a +calculated checksum. If the checksum of the data on the **primary** node matches checksum of the +data on the **secondary** node, the data transferred successfully. Following a planned failover, +any corrupted data may be **lost**, depending on the extent of the corruption. + +If verification fails on the **primary** node, this indicates that Geo is +successfully replicating a corrupted object; restore it from backup or remove it +it from the **primary** node to resolve the issue. + +If verification succeeds on the **primary** node but fails on the **secondary** node, +this indicates that the object was corrupted during the replication process. +Geo actively try to correct verification failures marking the repository to +be resynced with a backoff period. If you want to reset the verification for +these failures, so you should follow [these instructions][reset-verification]. + +If verification is lagging significantly behind replication, consider giving +the node more time before scheduling a planned failover. + +## Disabling or enabling the automatic background verification + +Run the following commands in a Rails console on the **primary** node: + +```sh +# Omnibus GitLab +gitlab-rails console + +# Installation from source +cd /home/git/gitlab +sudo -u git -H bin/rails console RAILS_ENV=production +``` + +To check if automatic background verification is enabled: + +```ruby +Gitlab::Geo.repository_verification_enabled? +``` + +To disable automatic background verification: + +```ruby +Feature.disable('geo_repository_verification') +``` + +To enable automatic background verification: + +```ruby +Feature.enable('geo_repository_verification') +``` + +## Repository verification + +Navigate to the **Admin Area > Geo** dashboard on the **primary** node and expand +the **Verification information** tab for that node to view automatic checksumming +status for repositories and wikis. Successes are shown in green, pending work +in grey, and failures in red. + + + +Navigate to the **Admin Area > Geo** dashboard on the **secondary** node and expand +the **Verification information** tab for that node to view automatic verification +status for repositories and wikis. As with checksumming, successes are shown in +green, pending work in grey, and failures in red. + + + +## Using checksums to compare Geo nodes + +To check the health of Geo **secondary** nodes, we use a checksum over the list of +Git references and their values. The checksum includes `HEAD`, `heads`, `tags`, +`notes`, and GitLab-specific references to ensure true consistency. If two nodes +have the same checksum, then they definitely hold the same references. We compute +the checksum for every node after every update to make sure that they are all +in sync. + +## Repository re-verification + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/8550) in GitLab Enterprise Edition 11.6. Available in [GitLab Premium](https://about.gitlab.com/pricing/). + +Due to bugs or transient infrastructure failures, it is possible for Git +repositories to change unexpectedly without being marked for verification. +Geo constantly reverifies the repositories to ensure the integrity of the +data. The default and recommended re-verification interval is 7 days, though +an interval as short as 1 day can be set. Shorter intervals reduce risk but +increase load and vice versa. + +Navigate to the **Admin Area > Geo** dashboard on the **primary** node, and +click the **Edit** button for the **primary** node to customize the minimum +re-verification interval: + + + +The automatic background re-verification is enabled by default, but you can +disable if you need. Run the following commands in a Rails console on the +**primary** node: + +```sh +# Omnibus GitLab +gitlab-rails console + +# Installation from source +cd /home/git/gitlab +sudo -u git -H bin/rails console RAILS_ENV=production +``` + +To disable automatic background re-verification: + +```ruby +Feature.disable('geo_repository_reverification') +``` + +To enable automatic background re-verification: + +```ruby +Feature.enable('geo_repository_reverification') +``` + +## Reset verification for projects where verification has failed + +Geo actively try to correct verification failures marking the repository to +be resynced with a backoff period. If you want to reset them manually, this +rake task marks projects where verification has failed or the checksum mismatch +to be resynced without the backoff period: + +For repositories: + +- Omnibus Installation + + ```sh + sudo gitlab-rake geo:verification:repository:reset + ``` + +- Source Installation + + ```sh + sudo -u git -H bundle exec rake geo:verification:repository:reset RAILS_ENV=production + ``` + +For wikis: + +- Omnibus Installation + + ```sh + sudo gitlab-rake geo:verification:wiki:reset + ``` + +- Source Installation + + ```sh + sudo -u git -H bundle exec rake geo:verification:wiki:reset RAILS_ENV=production + ``` + +## Current limitations + +Until [issue #5064][ee-5064] is completed, background verification doesn't cover +CI job artifacts and traces, LFS objects, or user uploads in file storage. +Verify their integrity manually by following [these instructions][foreground-verification] +on both nodes, and comparing the output between them. + +Data in object storage is **not verified**, as the object store is responsible +for ensuring the integrity of the data. + +[reset-verification]: background_verification.md#reset-verification-for-projects-where-verification-has-failed +[foreground-verification]: ../../raketasks/check.md +[ee-5064]: https://gitlab.com/gitlab-org/gitlab-ee/issues/5064 diff --git a/doc/administration/geo/disaster_recovery/bring_primary_back.md b/doc/administration/geo/disaster_recovery/bring_primary_back.md new file mode 100644 index 00000000000..209ca2f50d0 --- /dev/null +++ b/doc/administration/geo/disaster_recovery/bring_primary_back.md @@ -0,0 +1,61 @@ +# Bring a demoted primary node back online + +After a failover, it is possible to fail back to the demoted **primary** node to +restore your original configuration. This process consists of two steps: + +1. Making the old **primary** node a **secondary** node. +1. Promoting a **secondary** node to a **primary** node. + +CAUTION: **Caution:** +If you have any doubts about the consistency of the data on this node, we recommend setting it up from scratch. + +## Configure the former **primary** node to be a **secondary** node + +Since the former **primary** node will be out of sync with the current **primary** node, the first step is to bring the former **primary** node up to date. Note, deletion of data stored on disk like +repositories and uploads will not be replayed when bringing the former **primary** node back +into sync, which may result in increased disk usage. +Alternatively, you can [set up a new **secondary** GitLab instance][setup-geo] to avoid this. + +To bring the former **primary** node up to date: + +1. SSH into the former **primary** node that has fallen behind. +1. Make sure all the services are up: + + ```sh + sudo gitlab-ctl start + ``` + + > **Note 1:** If you [disabled the **primary** node permanently][disaster-recovery-disable-primary], + > you need to undo those steps now. For Debian/Ubuntu you just need to run + > `sudo systemctl enable gitlab-runsvdir`. For CentOS 6, you need to install + > the GitLab instance from scratch and set it up as a **secondary** node by + > following [Setup instructions][setup-geo]. In this case, you don't need to follow the next step. + > + > **Note 2:** If you [changed the DNS records](index.md#step-4-optional-updating-the-primary-domain-dns-record) + > for this node during disaster recovery procedure you may need to [block + > all the writes to this node](https://gitlab.com/gitlab-org/gitlab-ee/blob/master/doc/gitlab-geo/planned-failover.md#block-primary-traffic) + > during this procedure. + +1. [Setup database replication][database-replication]. Note that in this + case, **primary** node refers to the current **primary** node, and **secondary** node refers to the + former **primary** node. + +If you have lost your original **primary** node, follow the +[setup instructions][setup-geo] to set up a new **secondary** node. + +## Promote the **secondary** node to **primary** node + +When the initial replication is complete and the **primary** node and **secondary** node are +closely in sync, you can do a [planned failover]. + +## Restore the **secondary** node + +If your objective is to have two nodes again, you need to bring your **secondary** +node back online as well by repeating the first step +([configure the former **primary** node to be a **secondary** node](#configure-the-former-primary-node-to-be-a-secondary-node)) +for the **secondary** node. + +[setup-geo]: ../replication/index.md#setup-instructions +[database-replication]: ../replication/database.md +[disaster-recovery-disable-primary]: index.md#step-2-permanently-disable-the-primary-node +[planned failover]: planned_failover.md diff --git a/doc/administration/geo/disaster_recovery/img/replication-status.png b/doc/administration/geo/disaster_recovery/img/replication-status.png Binary files differnew file mode 100644 index 00000000000..d7085927c75 --- /dev/null +++ b/doc/administration/geo/disaster_recovery/img/replication-status.png diff --git a/doc/administration/geo/disaster_recovery/img/reverification-interval.png b/doc/administration/geo/disaster_recovery/img/reverification-interval.png Binary files differnew file mode 100644 index 00000000000..ad4597a4f49 --- /dev/null +++ b/doc/administration/geo/disaster_recovery/img/reverification-interval.png diff --git a/doc/administration/geo/disaster_recovery/img/verification-status-primary.png b/doc/administration/geo/disaster_recovery/img/verification-status-primary.png Binary files differnew file mode 100644 index 00000000000..2503408ec5d --- /dev/null +++ b/doc/administration/geo/disaster_recovery/img/verification-status-primary.png diff --git a/doc/administration/geo/disaster_recovery/img/verification-status-secondary.png b/doc/administration/geo/disaster_recovery/img/verification-status-secondary.png Binary files differnew file mode 100644 index 00000000000..462274d8b14 --- /dev/null +++ b/doc/administration/geo/disaster_recovery/img/verification-status-secondary.png diff --git a/doc/administration/geo/disaster_recovery/index.md b/doc/administration/geo/disaster_recovery/index.md new file mode 100644 index 00000000000..1de25671090 --- /dev/null +++ b/doc/administration/geo/disaster_recovery/index.md @@ -0,0 +1,323 @@ +# Disaster Recovery **[PREMIUM ONLY]** + +Geo replicates your database, your Git repositories, and few other assets. +We will support and replicate more data in the future, that will enable you to +failover with minimal effort, in a disaster situation. + +See [Geo current limitations][geo-limitations] for more information. + +CAUTION: **Warning:** +Disaster recovery for multi-secondary configurations is in **Alpha**. +For the latest updates, check the multi-secondary [Disaster Recovery epic][gitlab-org&65]. + +## Promoting a **secondary** Geo node in single-secondary configurations + +We don't currently provide an automated way to promote a Geo replica and do a +failover, but you can do it manually if you have `root` access to the machine. + +This process promotes a **secondary** Geo node to a **primary** node. To regain +geographic redundancy as quickly as possible, you should add a new **secondary** node +immediately after following these instructions. + +### Step 1. Allow replication to finish if possible + +If the **secondary** node is still replicating data from the **primary** node, follow +[the planned failover docs][planned-failover] as closely as possible in +order to avoid unnecessary data loss. + +### Step 2. Permanently disable the **primary** node + +CAUTION: **Warning:** +If the **primary** node goes offline, there may be data saved on the **primary** node +that has not been replicated to the **secondary** node. This data should be treated +as lost if you proceed. + +If an outage on the **primary** node happens, you should do everything possible to +avoid a split-brain situation where writes can occur in two different GitLab +instances, complicating recovery efforts. So to prepare for the failover, we +must disable the **primary** node. + +1. SSH into the **primary** node to stop and disable GitLab, if possible: + + ```sh + sudo gitlab-ctl stop + ``` + + Prevent GitLab from starting up again if the server unexpectedly reboots: + + ```sh + sudo systemctl disable gitlab-runsvdir + ``` + + > **CentOS only**: In CentOS 6 or older, there is no easy way to prevent GitLab from being + > started if the machine reboots isn't available (see [gitlab-org/omnibus-gitlab#3058]). + > It may be safest to uninstall the GitLab package completely: + + ```sh + yum remove gitlab-ee + ``` + + > **Ubuntu 14.04 LTS**: If you are using an older version of Ubuntu + > or any other distro based on the Upstart init system, you can prevent GitLab + > from starting if the machine reboots by doing the following: + + ```sh + initctl stop gitlab-runsvvdir + echo 'manual' > /etc/init/gitlab-runsvdir.override + initctl reload-configuration + ``` + +1. If you do not have SSH access to the **primary** node, take the machine offline and + prevent it from rebooting by any means at your disposal. + Since there are many ways you may prefer to accomplish this, we will avoid a + single recommendation. You may need to: + - Reconfigure the load balancers. + - Change DNS records (e.g., point the primary DNS record to the **secondary** + node in order to stop usage of the **primary** node). + - Stop the virtual servers. + - Block traffic through a firewall. + - Revoke object storage permissions from the **primary** node. + - Physically disconnect a machine. + +1. If you plan to + [update the primary domain DNS record](#step-4-optional-updating-the-primary-domain-dns-record), + you may wish to lower the TTL now to speed up propagation. + +### Step 3. Promoting a **secondary** node + +NOTE: **Note:** +A new **secondary** should not be added at this time. If you want to add a new +**secondary**, do this after you have completed the entire process of promoting +the **secondary** to the **primary**. + +#### Promoting a **secondary** node running on a single machine + +1. SSH in to your **secondary** node and login as root: + + ```sh + sudo -i + ``` + +1. Edit `/etc/gitlab/gitlab.rb` to reflect its new status as **primary** by + removing any lines that enabled the `geo_secondary_role`: + + ```ruby + ## In pre-11.5 documentation, the role was enabled as follows. Remove this line. + geo_secondary_role['enable'] = true + + ## In 11.5+ documentation, the role was enabled as follows. Remove this line. + roles ['geo_secondary_role'] + ``` + +1. Promote the **secondary** node to the **primary** node. Execute: + + ```sh + gitlab-ctl promote-to-primary-node + ``` + +1. Verify you can connect to the newly promoted **primary** node using the URL used + previously for the **secondary** node. +1. If successful, the **secondary** node has now been promoted to the **primary** node. + +#### Promoting a **secondary** node with HA + +The `gitlab-ctl promote-to-primary-node` command cannot be used yet in +conjunction with High Availability or with multiple machines, as it can only +perform changes on a **secondary** with only a single machine. Instead, you must +do this manually. + +1. SSH in to the database node in the **secondary** and trigger PostgreSQL to + promote to read-write: + + ```bash + sudo gitlab-pg-ctl promote + ``` + +1. Edit `/etc/gitlab/gitlab.rb` on every machine in the **secondary** to + reflect its new status as **primary** by removing any lines that enabled the + `geo_secondary_role`: + + ```ruby + ## In pre-11.5 documentation, the role was enabled as follows. Remove this line. + geo_secondary_role['enable'] = true + + ## In 11.5+ documentation, the role was enabled as follows. Remove this line. + roles ['geo_secondary_role'] + ``` + + After making these changes [Reconfigure GitLab](../../restart_gitlab.md#omnibus-gitlab-reconfigure) each + machine so the changes take effect. + +1. Promote the **secondary** to **primary**. SSH into a single application + server and execute: + + ```bash + sudo gitlab-rake geo:set_secondary_as_primary + ``` + +1. Verify you can connect to the newly promoted **primary** using the URL used + previously for the **secondary**. +1. Success! The **secondary** has now been promoted to **primary**. + +### Step 4. (Optional) Updating the primary domain DNS record + +Updating the DNS records for the primary domain to point to the **secondary** node +will prevent the need to update all references to the primary domain to the +secondary domain, like changing Git remotes and API URLs. + +1. SSH into the **secondary** node and login as root: + + ```sh + sudo -i + ``` + +1. Update the primary domain's DNS record. After updating the primary domain's + DNS records to point to the **secondary** node, edit `/etc/gitlab/gitlab.rb` on the + **secondary** node to reflect the new URL: + + ```ruby + # Change the existing external_url configuration + external_url 'https://gitlab.example.com' + ``` + + NOTE: **Note** + Changing `external_url` won't prevent access via the old secondary URL, as + long as the secondary DNS records are still intact. + +1. Reconfigure the **secondary** node for the change to take effect: + + ```sh + gitlab-ctl reconfigure + ``` + +1. Execute the command below to update the newly promoted **primary** node URL: + + ```sh + gitlab-rake geo:update_primary_node_url + ``` + + This command will use the changed `external_url` configuration defined + in `/etc/gitlab/gitlab.rb`. + +1. Verify you can connect to the newly promoted **primary** using its URL. + If you updated the DNS records for the primary domain, these changes may + not have yet propagated depending on the previous DNS records TTL. + +### Step 5. (Optional) Add **secondary** Geo node to a promoted **primary** node + +Promoting a **secondary** node to **primary** node using the process above does not enable +Geo on the new **primary** node. + +To bring a new **secondary** node online, follow the [Geo setup instructions][setup-geo]. + +### Step 6. (Optional) Removing the secondary's tracking database + +Every **secondary** has a special tracking database that is used to save the status of the synchronization of all the items from the **primary**. +Because the **secondary** is already promoted, that data in the tracking database is no longer required. + +The data can be removed with the following command: + +```sh +sudo rm -rf /var/opt/gitlab/geo-postgresql +``` + +## Promoting secondary Geo replica in multi-secondary configurations + +If you have more than one **secondary** node and you need to promote one of them, we suggest you follow +[Promoting a **secondary** Geo node in single-secondary configurations](#promoting-a-secondary-geo-node-in-single-secondary-configurations) +and after that you also need two extra steps. + +### Step 1. Prepare the new **primary** node to serve one or more **secondary** nodes + +1. SSH into the new **primary** node and login as root: + + ```sh + sudo -i + ``` + +1. Edit `/etc/gitlab/gitlab.rb` + + ```ruby + ## Enable a Geo Primary role (if you haven't yet) + roles ['geo_primary_role'] + + ## + # Primary and Secondary addresses + # - replace '198.51.100.1' with the public or VPC address of your Geo primary node + # - replace '198.51.100.2' with the public or VPC address of your Geo secondary node + ## + postgresql['md5_auth_cidr_addresses'] = ['198.51.100.1/32', '198.51.100.2/32'] + + # Every secondary server needs to have its own slot so specify the number of secondary nodes you're going to have + postgresql['max_replication_slots'] = 1 + + ## + ## Disable automatic database migrations temporarily + ## (until PostgreSQL is restarted and listening on the private address). + ## + gitlab_rails['auto_migrate'] = false + + ``` + + (For more details about these settings you can read [Configure the primary server][configure-the-primary-server]) + +1. Save the file and reconfigure GitLab for the database listen changes and + the replication slot changes to be applied. + + ```sh + gitlab-ctl reconfigure + ``` + + Restart PostgreSQL for its changes to take effect: + + ```sh + gitlab-ctl restart postgresql + ``` + +1. Re-enable migrations now that PostgreSQL is restarted and listening on the + private address. + + Edit `/etc/gitlab/gitlab.rb` and **change** the configuration to `true`: + + ```ruby + gitlab_rails['auto_migrate'] = true + ``` + + Save the file and reconfigure GitLab: + + ```sh + gitlab-ctl reconfigure + ``` + +### Step 2. Initiate the replication process + +Now we need to make each **secondary** node listen to changes on the new **primary** node. To do that you need +to [initiate the replication process][initiate-the-replication-process] again but this time +for another **primary** node. All the old replication settings will be overwritten. + +## Troubleshooting + +### I followed the disaster recovery instructions and now two-factor auth is broken! + +The setup instructions for Geo prior to 10.5 failed to replicate the +`otp_key_base` secret, which is used to encrypt the two-factor authentication +secrets stored in the database. If it differs between **primary** and **secondary** +nodes, users with two-factor authentication enabled won't be able to log in +after a failover. + +If you still have access to the old **primary** node, you can follow the +instructions in the +[Upgrading to GitLab 10.5][updating-geo] +section to resolve the error. Otherwise, the secret is lost and you'll need to +[reset two-factor authentication for all users][sec-tfa]. + +[gitlab-org&65]: https://gitlab.com/groups/gitlab-org/-/epics/65 +[geo-limitations]: ../replication/index.md#current-limitations +[planned-failover]: planned_failover.md +[setup-geo]: ../replication/index.md#setup-instructions +[updating-geo]: ../replication/updating_the_geo_nodes.md#upgrading-to-gitlab-105 +[sec-tfa]: ../../../security/two_factor_authentication.md#disabling-2fa-for-everyone +[gitlab-org/omnibus-gitlab#3058]: https://gitlab.com/gitlab-org/omnibus-gitlab/issues/3058 +[gitlab-org/gitlab-ee#4284]: https://gitlab.com/gitlab-org/gitlab-ee/issues/4284 +[initiate-the-replication-process]: ../replication/database.html#step-3-initiate-the-replication-process +[configure-the-primary-server]: ../replication/database.html#step-1-configure-the-primary-server diff --git a/doc/administration/geo/disaster_recovery/planned_failover.md b/doc/administration/geo/disaster_recovery/planned_failover.md new file mode 100644 index 00000000000..9875a29d4c0 --- /dev/null +++ b/doc/administration/geo/disaster_recovery/planned_failover.md @@ -0,0 +1,229 @@ +# Disaster recovery for planned failover + +The primary use-case of Disaster Recovery is to ensure business continuity in +the event of unplanned outage, but it can be used in conjunction with a planned +failover to migrate your GitLab instance between regions without extended +downtime. + +As replication between Geo nodes is asynchronous, a planned failover requires +a maintenance window in which updates to the **primary** node are blocked. The +length of this window is determined by your replication capacity - once the +**secondary** node is completely synchronized with the **primary** node, the failover can occur without +data loss. + +This document assumes you already have a fully configured, working Geo setup. +Please read it and the [Disaster Recovery][disaster-recovery] failover +documentation in full before proceeding. Planned failover is a major operation, +and if performed incorrectly, there is a high risk of data loss. Consider +rehearsing the procedure until you are comfortable with the necessary steps and +have a high degree of confidence in being able to perform them accurately. + +## Not all data is automatically replicated + +If you are using any GitLab features that Geo [doesn't support][limitations], +you must make separate provisions to ensure that the **secondary** node has an +up-to-date copy of any data associated with that feature. This may extend the +required scheduled maintenance period significantly. + +A common strategy for keeping this period as short as possible for data stored +in files is to use `rsync` to transfer the data. An initial `rsync` can be +performed ahead of the maintenance window; subsequent `rsync`s (including a +final transfer inside the maintenance window) will then transfer only the +*changes* between the **primary** node and the **secondary** nodes. + +Repository-centric strategies for using `rsync` effectively can be found in the +[moving repositories][moving-repositories] documentation; these strategies can +be adapted for use with any other file-based data, such as GitLab Pages (to +be found in `/var/opt/gitlab/gitlab-rails/shared/pages` if using Omnibus). + +## Pre-flight checks + +Follow these steps before scheduling a planned failover to ensure the process +will go smoothly. + +### Object storage + +Some classes of non-repository data can use object storage in preference to +file storage. Geo [does not replicate data in object storage](../replication/object_storage.md), +leaving that task up to the object store itself. For a planned failover, this +means you can decouple the replication of this data from the failover of the +GitLab service. + +If you're already using object storage, simply verify that your **secondary** +node has access to the same data as the **primary** node - they must either they share the +same object storage configuration, or the **secondary** node should be configured to +access a [geographically-replicated][os-repl] copy provided by the object store +itself. + +If you have a large GitLab installation or cannot tolerate downtime, consider +[migrating to Object Storage][os-conf] **before** scheduling a planned failover. +Doing so reduces both the length of the maintenance window, and the risk of data +loss as a result of a poorly executed planned failover. + +### Review the configuration of each **secondary** node + +Database settings are automatically replicated to the **secondary** node, but the +`/etc/gitlab/gitlab.rb` file must be set up manually, and differs between +nodes. If features such as Mattermost, OAuth or LDAP integration are enabled +on the **primary** node but not the **secondary** node, they will be lost during failover. + +Review the `/etc/gitlab/gitlab.rb` file for both nodes and ensure the **secondary** node +supports everything the **primary** node does **before** scheduling a planned failover. + +### Run system checks + +Run the following on both **primary** and **secondary** nodes: + +```sh +gitlab-rake gitlab:check +gitlab-rake gitlab:geo:check +``` + +If any failures are reported on either node, they should be resolved **before** +scheduling a planned failover. + +### Check that secrets match between nodes + +The SSH host keys and `/etc/gitlab/gitlab-secrets.json` files should be +identical on all nodes. Check this by running the following on all nodes and +comparing the output: + +```sh +sudo sha256sum /etc/ssh/ssh_host* /etc/gitlab/gitlab-secrets.json +``` + +If any files differ, replace the content on the **secondary** node with the +content from the **primary** node. + +### Ensure Geo replication is up-to-date + +The maintenance window won't end until Geo replication and verification is +completely finished. To keep the window as short as possible, you should +ensure these processes are close to 100% as possible during active use. + +Navigate to the **Admin Area > Geo** dashboard on the **secondary** node to +review status. Replicated objects (shown in green) should be close to 100%, +and there should be no failures (shown in red). If a large proportion of +objects aren't yet replicated (shown in grey), consider giving the node more +time to complete + + + +If any objects are failing to replicate, this should be investigated before +scheduling the maintenance window. Following a planned failover, anything that +failed to replicate will be **lost**. + +You can use the [Geo status API](../../../api/geo_nodes.md#retrieve-project-sync-or-verification-failures-that-occurred-on-the-current-node) to review failed objects and +the reasons for failure. + +A common cause of replication failures is the data being missing on the +**primary** node - you can resolve these failures by restoring the data from backup, +or removing references to the missing data. + +### Verify the integrity of replicated data + +This [content was moved to another location][background-verification]. + +### Notify users of scheduled maintenance + +On the **primary** node, navigate to **Admin Area > Messages**, add a broadcast +message. You can check under **Admin Area > Geo** to estimate how long it +will take to finish syncing. An example message would be: + +> A scheduled maintenance will take place at XX:XX UTC. We expect it to take +> less than 1 hour. + +## Prevent updates to the **primary** node + +Until a [read-only mode][ce-19739] is implemented, updates must be prevented +from happening manually. Note that your **secondary** node still needs read-only +access to the **primary** node during the maintenance window. + +1. At the scheduled time, using your cloud provider or your node's firewall, block + all HTTP, HTTPS and SSH traffic to/from the **primary** node, **except** for your IP and + the **secondary** node's IP. + + For instance, if your **secondary** node originates all its traffic from `5.6.7.8` and + your IP is `100.0.0.1`, you might run the following commands on the server(s) + making up your **primary** node: + + ```sh + sudo iptables -A INPUT -p tcp -s 5.6.7.8 --destination-port 22 -j ACCEPT + sudo iptables -A INPUT -p tcp -s 100.0.0.1 --destination-port 22 -j ACCEPT + sudo iptables -A INPUT --destination-port 22 -j REJECT + + sudo iptables -A INPUT -p tcp -s 5.6.7.8 --destination-port 80 -j ACCEPT + sudo iptables -A INPUT -p tcp -s 100.0.0.1 --destination-port 80 -j ACCEPT + sudo iptables -A INPUT --tcp-dport 80 -j REJECT + + sudo iptables -A INPUT -p tcp -s 5.6.7.8 --destination-port 443 -j ACCEPT + sudo iptables -A INPUT -p tcp -s 100.0.0.1 --destination-port 443 -j ACCEPT + sudo iptables -A INPUT --tcp-dport 443 -j REJECT + ``` + + From this point, users will be unable to view their data or make changes on the + **primary** node. They will also be unable to log in to the **secondary** node. + However, existing sessions will work for the remainder of the maintenance period, and + public data will be accessible throughout. + +1. Verify the **primary** node is blocked to HTTP traffic by visiting it in browser via + another IP. The server should refuse connection. + +1. Verify the **primary** node is blocked to Git over SSH traffic by attempting to pull an + existing Git repository with an SSH remote URL. The server should refuse + connection. + +1. Disable non-Geo periodic background jobs on the primary node by navigating + to **Admin Area > Monitoring > Background Jobs > Cron** , pressing `Disable All`, + and then pressing `Enable` for the `geo_sidekiq_cron_config_worker` cron job. + This job will re-enable several other cron jobs that are essential for planned + failover to complete successfully. + +## Finish replicating and verifying all data + +1. If you are manually replicating any data not managed by Geo, trigger the + final replication process now. +1. On the **primary** node, navigate to **Admin Area > Monitoring > Background Jobs > Queues** + and wait for all queues except those with `geo` in the name to drop to 0. + These queues contain work that has been submitted by your users; failing over + before it is completed will cause the work to be lost. +1. On the **primary** node, navigate to **Admin Area > Geo** and wait for the + following conditions to be true of the **secondary** node you are failing over to: + - All replication meters to each 100% replicated, 0% failures. + - All verification meters reach 100% verified, 0% failures. + - Database replication lag is 0ms. + - The Geo log cursor is up to date (0 events behind). + +1. On the **secondary** node, navigate to **Admin Area > Monitoring > Background Jobs > Queues** + and wait for all the `geo` queues to drop to 0 queued and 0 running jobs. +1. On the **secondary** node, use [these instructions][foreground-verification] + to verify the integrity of CI artifacts, LFS objects and uploads in file + storage. + +At this point, your **secondary** node will contain an up-to-date copy of everything the +**primary** node has, meaning nothing will be lost when you fail over. + +## Promote the **secondary** node + +Finally, follow the [Disaster Recovery docs][disaster-recovery] to promote the +**secondary** node to a **primary** node. This process will cause a brief outage on the **secondary** node, and users may need to log in again. + +Once it is completed, the maintenance window is over! Your new **primary** node will now +begin to diverge from the old one. If problems do arise at this point, failing +back to the old **primary** node [is possible][bring-primary-back], but likely to result +in the loss of any data uploaded to the new primary in the meantime. + +Don't forget to remove the broadcast message after failover is complete. + +[bring-primary-back]: bring_primary_back.md +[ce-19739]: https://gitlab.com/gitlab-org/gitlab-ce/issues/19739 +[container-registry]: ../replication/container_registry.md +[disaster-recovery]: index.md +[ee-4930]: https://gitlab.com/gitlab-org/gitlab-ee/issues/4930 +[ee-5064]: https://gitlab.com/gitlab-org/gitlab-ee/issues/5064 +[foreground-verification]: ../../raketasks/check.md +[background-verification]: background_verification.md +[limitations]: ../replication/index.md#current-limitations +[moving-repositories]: ../../operations/moving_repositories.md +[os-conf]: ../replication/object_storage.md#configuration +[os-repl]: ../replication/object_storage.md#replication diff --git a/doc/administration/geo/replication/configuration.md b/doc/administration/geo/replication/configuration.md new file mode 100644 index 00000000000..8c2e91dd0a0 --- /dev/null +++ b/doc/administration/geo/replication/configuration.md @@ -0,0 +1,315 @@ +# Geo configuration (GitLab Omnibus) + +NOTE: **Note:** +This is the documentation for the Omnibus GitLab packages. For installations +from source, follow the [**Geo nodes configuration for installations +from source**][configuration-source] guide. + +## Configuring a new **secondary** node + +NOTE: **Note:** +This is the final step in setting up a **secondary** Geo node. Stages of the +setup process must be completed in the documented order. +Before attempting the steps in this stage, [complete all prior stages][setup-geo-omnibus]. + +The basic steps of configuring a **secondary** node are to: + +- Replicate required configurations between the **primary** node and the **secondary** nodes. +- Configure a tracking database on each **secondary** node. +- Start GitLab on each **secondary** node. + +You are encouraged to first read through all the steps before executing them +in your testing/production environment. + +> **Notes:** +> - **Do not** setup any custom authentication for the **secondary** nodes. This will be + handled by the **primary** node. +> - Any change that requires access to the **Admin Area** needs to be done in the + **primary** node because the **secondary** node is a read-only replica. + +### Step 1. Manually replicate secret GitLab values + +GitLab stores a number of secret values in the `/etc/gitlab/gitlab-secrets.json` +file which *must* be the same on all nodes. Until there is +a means of automatically replicating these between nodes (see issue [gitlab-org/gitlab-ee#3789]), +they must be manually replicated to the **secondary** node. + +1. SSH into the **primary** node, and execute the command below: + + ```sh + sudo cat /etc/gitlab/gitlab-secrets.json + ``` + + This will display the secrets that need to be replicated, in JSON format. + +1. SSH into the **secondary** node and login as the `root` user: + + ```sh + sudo -i + ``` + +1. Make a backup of any existing secrets: + + ```sh + mv /etc/gitlab/gitlab-secrets.json /etc/gitlab/gitlab-secrets.json.`date +%F` + ``` + +1. Copy `/etc/gitlab/gitlab-secrets.json` from the **primary** node to the **secondary** node, or + copy-and-paste the file contents between nodes: + + ```sh + sudo editor /etc/gitlab/gitlab-secrets.json + + # paste the output of the `cat` command you ran on the primary + # save and exit + ``` + +1. Ensure the file permissions are correct: + + ```sh + chown root:root /etc/gitlab/gitlab-secrets.json + chmod 0600 /etc/gitlab/gitlab-secrets.json + ``` + +1. Reconfigure the **secondary** node for the change to take effect: + + ```sh + gitlab-ctl reconfigure + gitlab-ctl restart + ``` + +### Step 2. Manually replicate the **primary** node's SSH host keys + +GitLab integrates with the system-installed SSH daemon, designating a user +(typically named git) through which all access requests are handled. + +In a [Disaster Recovery] situation, GitLab system +administrators will promote a **secondary** node to the **primary** node. DNS records for the +**primary** domain should also be updated to point to the new **primary** node +(previously a **secondary** node). Doing so will avoid the need to update Git remotes and API URLs. + +This will cause all SSH requests to the newly promoted **primary** node to +fail due to SSH host key mismatch. To prevent this, the primary SSH host +keys must be manually replicated to the **secondary** node. + +1. SSH into the **secondary** node and login as the `root` user: + + ```sh + sudo -i + ``` + +1. Make a backup of any existing SSH host keys: + + ```sh + find /etc/ssh -iname ssh_host_* -exec cp {} {}.backup.`date +%F` \; + ``` + +1. Copy OpenSSH host keys from the **primary** node: + + If you can access your **primary** node using the **root** user: + + ```sh + # Run this from the secondary node, change `primary-node-fqdn` for the IP or FQDN of the server + scp root@primary-node-fqdn:/etc/ssh/ssh_host_*_key* /etc/ssh + ``` + + If you only have access through a user with **sudo** privileges: + + ```sh + # Run this from your primary node: + sudo tar --transform 's/.*\///g' -zcvf ~/geo-host-key.tar.gz /etc/ssh/ssh_host_*_key* + + # Run this from your secondary node: + scp user-with-sudo@primary-node-fqdn:geo-host-key.tar.gz . + tar zxvf ~/geo-host-key.tar.gz -C /etc/ssh + ``` + +1. On your **secondary** node, ensure the file permissions are correct: + + ```sh + chown root:root /etc/ssh/ssh_host_*_key* + chmod 0600 /etc/ssh/ssh_host_*_key* + ``` + +1. To verify key fingerprint matches, execute the following command on both nodes: + + ```sh + for file in /etc/ssh/ssh_host_*_key; do ssh-keygen -lf $file; done + ``` + + You should get an output similar to this one and they should be identical on both nodes: + + ```sh + 1024 SHA256:FEZX2jQa2bcsd/fn/uxBzxhKdx4Imc4raXrHwsbtP0M root@serverhostname (DSA) + 256 SHA256:uw98R35Uf+fYEQ/UnJD9Br4NXUFPv7JAUln5uHlgSeY root@serverhostname (ECDSA) + 256 SHA256:sqOUWcraZQKd89y/QQv/iynPTOGQxcOTIXU/LsoPmnM root@serverhostname (ED25519) + 2048 SHA256:qwa+rgir2Oy86QI+PZi/QVR+MSmrdrpsuH7YyKknC+s root@serverhostname (RSA) + ``` + +1. Verify that you have the correct public keys for the existing private keys: + + ```sh + # This will print the fingerprint for private keys: + for file in /etc/ssh/ssh_host_*_key; do ssh-keygen -lf $file; done + + # This will print the fingerprint for public keys: + for file in /etc/ssh/ssh_host_*_key.pub; do ssh-keygen -lf $file; done + ``` + + NOTE: **Note**: + The output for private keys and public keys command should generate the same fingerprint. + +1. Restart sshd on your **secondary** node: + + ```sh + # Debian or Ubuntu installations + sudo service ssh reload + + # CentOS installations + sudo service sshd reload + ``` + +### Step 3. Add the **secondary** node + +1. Visit the **primary** node's **Admin Area > Geo** + (`/admin/geo/nodes`) in your browser. +1. Add the **secondary** node by providing its full URL. **Do NOT** check the + **This is a primary node** checkbox. +1. Optionally, choose which namespaces should be replicated by the + **secondary** node. Leave blank to replicate all. Read more in + [selective synchronization](#selective-synchronization). +1. Click the **Add node** button. +1. SSH into your GitLab **secondary** server and restart the services: + + ```sh + gitlab-ctl restart + ``` + + Check if there are any common issue with your Geo setup by running: + + ```sh + gitlab-rake gitlab:geo:check + ``` + +1. SSH into your **primary** server and login as root to verify the + **secondary** node is reachable or there are any common issue with your Geo setup: + + ```sh + gitlab-rake gitlab:geo:check + ``` + +Once added to the admin panel and restarted, the **secondary** node will automatically start +replicating missing data from the **primary** node in a process known as **backfill**. +Meanwhile, the **primary** node will start to notify each **secondary** node of any changes, so +that the **secondary** node can act on those notifications immediately. + +Make sure the **secondary** node is running and accessible. +You can login to the **secondary** node with the same credentials as used for the **primary** node. + +### Step 4. Enabling Hashed Storage + +Using Hashed Storage significantly improves Geo replication. Project and group +renames no longer require synchronization between nodes. + +1. Visit the **primary** node's **Admin Area > Settings > Repository** + (`/admin/application_settings/repository`) in your browser. +1. In the **Repository storage** section, check **Use hashed storage paths for newly created and renamed projects**. + +### Step 5. (Optional) Configuring the **secondary** node to trust the **primary** node + +You can safely skip this step if your **primary** node uses a CA-issued HTTPS certificate. + +If your **primary** node is using a self-signed certificate for *HTTPS* support, you will +need to add that certificate to the **secondary** node's trust store. Retrieve the +certificate from the **primary** node and follow +[these instructions][omnibus-ssl] +on the **secondary** node. + +### Step 6. Enable Git access over HTTP/HTTPS + +Geo synchronizes repositories over HTTP/HTTPS, and therefore requires this clone +method to be enabled. Navigate to **Admin Area > Settings** +(`/admin/application_settings`) on the **primary** node, and set +`Enabled Git access protocols` to `Both SSH and HTTP(S)` or `Only HTTP(S)`. + +### Step 7. Verify proper functioning of the **secondary** node + +Your **secondary** node is now configured! + +You can login to the **secondary** node with the same credentials you used for the +**primary** node. Visit the **secondary** node's **Admin Area > Geo** +(`/admin/geo/nodes`) in your browser to check if it's correctly identified as a +**secondary** Geo node and if Geo is enabled. + +The initial replication, or 'backfill', will probably still be in progress. You +can monitor the synchronization process on each geo node from the **primary** +node's Geo Nodes dashboard in your browser. + + + +If your installation isn't working properly, check the +[troubleshooting document]. + +The two most obvious issues that can become apparent in the dashboard are: + +1. Database replication not working well. +1. Instance to instance notification not working. In that case, it can be + something of the following: + - You are using a custom certificate or custom CA (see the + [troubleshooting document]). + - The instance is firewalled (check your firewall rules). + +Please note that disabling a **secondary** node will stop the synchronization process. + +Please note that if `git_data_dirs` is customized on the **primary** node for multiple +repository shards you must duplicate the same configuration on each **secondary** node. + +Point your users to the ["Using a Geo Server" guide][using-geo]. + +Currently, this is what is synced: + +- Git repositories. +- Wikis. +- LFS objects. +- Issues, merge requests, snippets, and comment attachments. +- Users, groups, and project avatars. + +## Selective synchronization + +Geo supports selective synchronization, which allows admins to choose +which projects should be synchronized by **secondary** nodes. + +It is important to note that selective synchronization does not: + +1. Restrict permissions from **secondary** nodes. +1. Hide project metadata from **secondary** nodes. + - Since Geo currently relies on PostgreSQL replication, all project metadata + gets replicated to **secondary** nodes, but repositories that have not been + selected will be empty. +1. Reduce the number of events generated for the Geo event log. + - The **primary** node generates events as long as any **secondary** nodes are present. + Selective synchronization restrictions are implemented on the **secondary** nodes, + not the **primary** node. + +A subset of projects can be chosen, either by group or by storage shard. The +former is ideal for replicating data belonging to a subset of users, while the +latter is more suited to progressively rolling out Geo to a large GitLab +instance. + +## Upgrading Geo + +See the [updating the Geo nodes document](updating_the_geo_nodes.md). + +## Troubleshooting + +See the [troubleshooting document](troubleshooting.md). + +[configuration-source]: configuration_source.md +[setup-geo-omnibus]: index.md#using-omnibus-gitlab +[Hashed Storage]: ../../repository_storage_types.md +[Disaster Recovery]: ../disaster_recovery/index.md +[gitlab-org/gitlab-ee#3789]: https://gitlab.com/gitlab-org/gitlab-ee/issues/3789 +[gitlab-com/infrastructure#2821]: https://gitlab.com/gitlab-com/infrastructure/issues/2821 +[omnibus-ssl]: https://docs.gitlab.com/omnibus/settings/ssl.html +[troubleshooting document]: troubleshooting.md +[using-geo]: using_a_geo_server.md diff --git a/doc/administration/geo/replication/configuration_source.md b/doc/administration/geo/replication/configuration_source.md new file mode 100644 index 00000000000..72d5831b1cb --- /dev/null +++ b/doc/administration/geo/replication/configuration_source.md @@ -0,0 +1,173 @@ +# Geo configuration (source) + +NOTE: **Note:** +This documentation applies to GitLab source installations. In GitLab 11.5, this documentation was deprecated and will be removed in a future release. +Please consider [migrating to GitLab Omnibus install](https://docs.gitlab.com/omnibus/update/convert_to_omnibus.html). For installations +using the Omnibus GitLab packages, follow the +[**Omnibus Geo nodes configuration**][configuration] guide. + +## Configuring a new **secondary** node + +NOTE: **Note:** +This is the final step in setting up a **secondary** node. Stages of the setup +process must be completed in the documented order. Before attempting the steps +in this stage, [complete all prior stages](index.md#using-gitlab-installed-from-source-deprecated). + +The basic steps of configuring a **secondary** node are to: + +- Replicate required configurations between the **primary** and **secondary** nodes. +- Configure a tracking database on each **secondary** node. +- Start GitLab on the **secondary** node. + +You are encouraged to first read through all the steps before executing them +in your testing/production environment. + +NOTE: **Note:** +**Do not** set up any custom authentication on **secondary** nodes, this will be handled by the **primary** node. + +NOTE: **Note:** +**Do not** add anything in the **secondary** node's admin area (**Admin Area > Geo**). This is handled solely by the **primary** node. + +### Step 1. Manually replicate secret GitLab values + +GitLab stores a number of secret values in the `/home/git/gitlab/config/secrets.yml` +file which *must* match between the **primary** and **secondary** nodes. Until there is +a means of automatically replicating these between nodes (see [gitlab-org/gitlab-ee#3789]), they must +be manually replicated to **secondary** nodes. + +1. SSH into the **primary** node, and execute the command below: + + ```sh + sudo cat /home/git/gitlab/config/secrets.yml + ``` + + This will display the secrets that need to be replicated, in YAML format. + +1. SSH into the **secondary** node and login as the `git` user: + + ```sh + sudo -i -u git + ``` + +1. Make a backup of any existing secrets: + + ```sh + mv /home/git/gitlab/config/secrets.yml /home/git/gitlab/config/secrets.yml.`date +%F` + ``` + +1. Copy `/home/git/gitlab/config/secrets.yml` from the **primary** node to the **secondary** node, or + copy-and-paste the file contents between nodes: + + ```sh + sudo editor /home/git/gitlab/config/secrets.yml + + # paste the output of the `cat` command you ran on the primary + # save and exit + ``` + +1. Ensure the file permissions are correct: + + ```sh + chown git:git /home/git/gitlab/config/secrets.yml + chmod 0600 /home/git/gitlab/config/secrets.yml + ``` + +1. Restart GitLab + + ```sh + service gitlab restart + ``` + +Once restarted, the **secondary** node will automatically start replicating missing data +from the **primary** node in a process known as backfill. Meanwhile, the **primary** node +will start to notify the **secondary** node of any changes, so that the **secondary** node can +act on those notifications immediately. + +Make sure the **secondary** node is running and accessible. You can login to +the **secondary** node with the same credentials as used for the **primary** node. + +### Step 2. Manually replicate the **primary** node's SSH host keys + +Read [Manually replicate the **primary** node's SSH host keys](configuration.md#step-2-manually-replicate-the-primary-nodes-ssh-host-keys) + +### Step 3. Add the **secondary** GitLab node + +1. Navigate to the **primary** node's **Admin Area > Geo** + (`/admin/geo/nodes`) in your browser. +1. Add the **secondary** node by providing its full URL. **Do NOT** check the + **This is a primary node** checkbox. +1. Optionally, choose which namespaces should be replicated by the + **secondary** node. Leave blank to replicate all. Read more in + [selective synchronization](#selective-synchronization). +1. Click the **Add node** button. +1. SSH into your GitLab **secondary** server and restart the services: + + ```sh + service gitlab restart + ``` + + Check if there are any common issue with your Geo setup by running: + + ```sh + bundle exec rake gitlab:geo:check + ``` + +1. SSH into your GitLab **primary** server and login as root to verify the + **secondary** node is reachable or there are any common issue with your Geo setup: + + ```sh + bundle exec rake gitlab:geo:check + ``` + +Once reconfigured, the **secondary** node will automatically start +replicating missing data from the **primary** node in a process known as backfill. +Meanwhile, the **primary** node will start to notify the **secondary** node of any changes, so +that the **secondary** node can act on those notifications immediately. + +Make sure the **secondary** node is running and accessible. +You can log in to the **secondary** node with the same credentials as used for the **primary** node. + +### Step 4. Enabling Hashed Storage + +Read [Enabling Hashed Storage](configuration.md#step-4-enabling-hashed-storage). + +### Step 5. (Optional) Configuring the secondary to trust the primary + +You can safely skip this step if your **primary** node uses a CA-issued HTTPS certificate. + +If your **primary** node is using a self-signed certificate for *HTTPS* support, you will +need to add that certificate to the **secondary** node's trust store. Retrieve the +certificate from the **primary** node and follow your distribution's instructions for +adding it to the **secondary** node's trust store. In Debian/Ubuntu, for example, with a +certificate file of `primary.geo.example.com.crt`, you would follow these steps: + +```sh +sudo -i +cp primary.geo.example.com.crt /usr/local/share/ca-certificates +update-ca-certificates +``` + +### Step 6. Enable Git access over HTTP/HTTPS + +Geo synchronizes repositories over HTTP/HTTPS, and therefore requires this clone +method to be enabled. Navigate to **Admin Area > Settings** +(`/admin/application_settings`) on the **primary** node, and set +`Enabled Git access protocols` to `Both SSH and HTTP(S)` or `Only HTTP(S)`. + +### Step 7. Verify proper functioning of the secondary node + +Read [Verify proper functioning of the secondary node][configuration-verify-node]. + +## Selective synchronization + +Read [Selective synchronization][configuration-selective-replication]. + +## Troubleshooting + +Read the [troubleshooting document][troubleshooting]. + +[gitlab-org/gitlab-ee#3789]: https://gitlab.com/gitlab-org/gitlab-ee/issues/3789 +[configuration]: configuration.md +[configuration-selective-replication]: configuration.md#selective-synchronization +[configuration-verify-node]: configuration.md#step-7-verify-proper-functioning-of-the-secondary-node +[troubleshooting]: troubleshooting.md diff --git a/doc/administration/geo/replication/database.md b/doc/administration/geo/replication/database.md new file mode 100644 index 00000000000..10e5409124c --- /dev/null +++ b/doc/administration/geo/replication/database.md @@ -0,0 +1,496 @@ +# Geo database replication (GitLab Omnibus) + +NOTE: **Note:** +This is the documentation for the Omnibus GitLab packages. For installations +from source, follow the +[Geo database replication (source)](database_source.md) guide. + +NOTE: **Note:** +If your GitLab installation uses external (not managed by Omnibus) PostgreSQL +instances, the Omnibus roles will not be able to perform all necessary +configuration steps. In this case, refer to +[additional instructions](external_database.md). + +NOTE: **Note:** +The stages of the setup process must be completed in the documented order. +Before attempting the steps in this stage, [complete all prior stages][toc]. + +This document describes the minimal steps you have to take in order to +replicate your **primary** GitLab database to a **secondary** node's database. You may +have to change some values according to your database setup, how big it is, etc. + +You are encouraged to first read through all the steps before executing them +in your testing/production environment. + +## PostgreSQL replication + +The GitLab **primary** node where the write operations happen will connect to +the **primary** database server, and **secondary** nodes will +connect to their own database servers (which are also read-only). + +NOTE: **Note:** +In database documentation, you may see "**primary**" being referenced as "master" +and "**secondary**" as either "slave" or "standby" server (read-only). + +We recommend using [PostgreSQL replication slots][replication-slots-article] +to ensure that the **primary** node retains all the data necessary for the **secondary** nodes to +recover. See below for more details. + +The following guide assumes that: + +- You are using Omnibus and therefore you are using PostgreSQL 9.6 or later + which includes the [`pg_basebackup` tool][pgback] and improved + [Foreign Data Wrapper][FDW] support. +- You have a **primary** node already set up (the GitLab server you are + replicating from), running Omnibus' PostgreSQL (or equivalent version), and + you have a new **secondary** server set up with the same versions of the OS, + PostgreSQL, and GitLab on all nodes. +- The IP of the **primary** server for our examples is `198.51.100.1`, whereas the + **secondary** node's IP is `198.51.100.2`. Note that the **primary** and **secondary** servers + **must** be able to communicate over these addresses. More on this in the + guide below. + +CAUTION: **Warning:** +Geo works with streaming replication. Logical replication is not supported at this time. +There is an [issue where support is being discussed](https://gitlab.com/gitlab-org/gitlab-ee/issues/7420). + +### Step 1. Configure the **primary** server + +1. SSH into your GitLab **primary** server and login as root: + + ```sh + sudo -i + ``` + +1. Execute the command below to define the node as **primary** node: + + ```sh + gitlab-ctl set-geo-primary-node + ``` + + This command will use your defined `external_url` in `/etc/gitlab/gitlab.rb`. + +1. GitLab 10.4 and up only: Do the following to make sure the `gitlab` database user has a password defined: + + Generate a MD5 hash of the desired password: + + ```sh + gitlab-ctl pg-password-md5 gitlab + # Enter password: mypassword + # Confirm password: mypassword + # fca0b89a972d69f00eb3ec98a5838484 + ``` + + Edit `/etc/gitlab/gitlab.rb`: + + ```ruby + # Fill with the hash generated by `gitlab-ctl pg-password-md5 gitlab` + postgresql['sql_user_password'] = 'fca0b89a972d69f00eb3ec98a5838484' + + # Every node that runs Unicorn or Sidekiq needs to have the database + # password specified as below. If you have a high-availability setup, this + # must be present in all application nodes. + gitlab_rails['db_password'] = 'mypassword' + ``` + +1. Omnibus GitLab already has a [replication user] + called `gitlab_replicator`. You must set the password for this user manually. + You will be prompted to enter a password: + + ```sh + gitlab-ctl set-replication-password + ``` + + This command will also read the `postgresql['sql_replication_user']` Omnibus + setting in case you have changed `gitlab_replicator` username to something + else. + + If you are using an external database not managed by Omnibus GitLab, you need + to create the replicator user and define a password to it manually. + For information on how to create a replication user, refer to the + [appropriate step](database_source.md#step-1-configure-the-primary-server) + in [Geo database replication (source)](database_source.md). + +1. Configure PostgreSQL to listen on network interfaces: + + For security reasons, PostgreSQL does not listen on any network interfaces + by default. However, Geo requires the **secondary** node to be able to + connect to the **primary** node's database. For this reason, we need the address of + each node. Note: For external PostgreSQL instances, see [additional instructions](external_database.md). + + If you are using a cloud provider, you can lookup the addresses for each + Geo node through your cloud provider's management console. + + To lookup the address of a Geo node, SSH in to the Geo node and execute: + + ```sh + ## + ## Private address + ## + ip route get 255.255.255.255 | awk '{print "Private address:", $NF; exit}' + + ## + ## Public address + ## + echo "External address: $(curl --silent ipinfo.io/ip)" + ``` + + In most cases, the following addresses will be used to configure GitLab + Geo: + + | Configuration | Address | + |:----------------------------------------|:------------------------------------------------------| + | `postgresql['listen_address']` | **Primary** node's public or VPC private address. | + | `postgresql['md5_auth_cidr_addresses']` | **Secondary** node's public or VPC private addresses. | + + If you are using Google Cloud Platform, SoftLayer, or any other vendor that + provides a virtual private cloud (VPC) you can use the **secondary** node's private + address (corresponds to "internal address" for Google Cloud Platform) for + `postgresql['md5_auth_cidr_addresses']` and `postgresql['listen_address']`. + + The `listen_address` option opens PostgreSQL up to network connections + with the interface corresponding to the given address. See [the PostgreSQL + documentation][pg-docs-runtime-conn] for more details. + + Depending on your network configuration, the suggested addresses may not + be correct. If your **primary** node and **secondary** nodes connect over a local + area network, or a virtual network connecting availability zones like + [Amazon's VPC](https://aws.amazon.com/vpc/) or [Google's VPC](https://cloud.google.com/vpc/) + you should use the **secondary** node's private address for `postgresql['md5_auth_cidr_addresses']`. + + Edit `/etc/gitlab/gitlab.rb` and add the following, replacing the IP + addresses with addresses appropriate to your network configuration: + + ```ruby + ## + ## Geo Primary role + ## - configure dependent flags automatically to enable Geo + ## + roles ['geo_primary_role'] + + ## + ## Primary address + ## - replace '198.51.100.1' with the public or VPC address of your Geo primary node + ## + postgresql['listen_address'] = '198.51.100.1' + + ## + # Primary and Secondary addresses + # - replace '198.51.100.1' with the public or VPC address of your Geo primary node + # - replace '198.51.100.2' with the public or VPC address of your Geo secondary node + ## + postgresql['md5_auth_cidr_addresses'] = ['198.51.100.1/32','198.51.100.2/32'] + + ## + ## Replication settings + ## - set this to be the number of Geo secondary nodes you have + ## + postgresql['max_replication_slots'] = 1 + # postgresql['max_wal_senders'] = 10 + # postgresql['wal_keep_segments'] = 10 + + ## + ## Disable automatic database migrations temporarily + ## (until PostgreSQL is restarted and listening on the private address). + ## + gitlab_rails['auto_migrate'] = false + ``` + +1. Optional: If you want to add another **secondary** node, the relevant setting would look like: + + ```ruby + postgresql['md5_auth_cidr_addresses'] = ['198.51.100.1/32', '198.51.100.2/32','198.51.100.3/32'] + ``` + + You may also want to edit the `wal_keep_segments` and `max_wal_senders` to + match your database replication requirements. Consult the [PostgreSQL - + Replication documentation][pg-docs-runtime-replication] + for more information. + +1. Save the file and reconfigure GitLab for the database listen changes and + the replication slot changes to be applied: + + ```sh + gitlab-ctl reconfigure + ``` + + Restart PostgreSQL for its changes to take effect: + + ```sh + gitlab-ctl restart postgresql + ``` + +1. Re-enable migrations now that PostgreSQL is restarted and listening on the + private address. + + Edit `/etc/gitlab/gitlab.rb` and **change** the configuration to `true`: + + ```ruby + gitlab_rails['auto_migrate'] = true + ``` + + Save the file and reconfigure GitLab: + + ```sh + gitlab-ctl reconfigure + ``` + +1. Now that the PostgreSQL server is set up to accept remote connections, run + `netstat -plnt | grep 5432` to make sure that PostgreSQL is listening on port + `5432` to the **primary** server's private address. + +1. A certificate was automatically generated when GitLab was reconfigured. This + will be used automatically to protect your PostgreSQL traffic from + eavesdroppers, but to protect against active ("man-in-the-middle") attackers, + the **secondary** node needs a copy of the certificate. Make a copy of the PostgreSQL + `server.crt` file on the **primary** node by running this command: + + ```sh + cat ~gitlab-psql/data/server.crt + ``` + + Copy the output into a clipboard or into a local file. You + will need it when setting up the **secondary** node! The certificate is not sensitive + data. + +### Step 2. Configure the **secondary** server + +1. SSH into your GitLab **secondary** server and login as root: + + ``` + sudo -i + ``` + +1. Stop application server and Sidekiq + + ``` + gitlab-ctl stop unicorn + gitlab-ctl stop sidekiq + ``` + + NOTE: **Note**: + This step is important so we don't try to execute anything before the node is fully configured. + +1. [Check TCP connectivity][rake-maintenance] to the **primary** node's PostgreSQL server: + + ```sh + gitlab-rake gitlab:tcp_check[198.51.100.1,5432] + ``` + + NOTE: **Note**: + If this step fails, you may be using the wrong IP address, or a firewall may + be preventing access to the server. Check the IP address, paying close + attention to the difference between public and private addresses and ensure + that, if a firewall is present, the **secondary** node is permitted to connect to the + **primary** node on port 5432. + +1. Create a file `server.crt` in the **secondary** server, with the content you got on the last step of the **primary** node's setup: + + ``` + editor server.crt + ``` + +1. Set up PostgreSQL TLS verification on the **secondary** node: + + Install the `server.crt` file: + + ```sh + install -D -o gitlab-psql -g gitlab-psql -m 0400 -T server.crt ~gitlab-psql/.postgresql/root.crt + ``` + + PostgreSQL will now only recognize that exact certificate when verifying TLS + connections. The certificate can only be replicated by someone with access + to the private key, which is **only** present on the **primary** node. + +1. Test that the `gitlab-psql` user can connect to the **primary** node's database: + + ```sh + sudo -u gitlab-psql /opt/gitlab/embedded/bin/psql --list -U gitlab_replicator -d "dbname=gitlabhq_production sslmode=verify-ca" -W -h 198.51.100.1 + ``` + + When prompted enter the password you set in the first step for the + `gitlab_replicator` user. If all worked correctly, you should see + the list of **primary** node's databases. + + A failure to connect here indicates that the TLS configuration is incorrect. + Ensure that the contents of `~gitlab-psql/data/server.crt` on the **primary** node + match the contents of `~gitlab-psql/.postgresql/root.crt` on the **secondary** node. + +1. Configure PostgreSQL to enable FDW support: + + This step is similar to how we configured the **primary** instance. + We need to enable this, to enable FDW support, even if using a single node. + + Edit `/etc/gitlab/gitlab.rb` and add the following, replacing the IP + addresses with addresses appropriate to your network configuration: + + ```ruby + ## + ## Geo Secondary role + ## - configure dependent flags automatically to enable Geo + ## + roles ['geo_secondary_role'] + + ## + ## Secondary address + ## - replace '198.51.100.2' with the public or VPC address of your Geo secondary node + ## + postgresql['listen_address'] = '198.51.100.2' + postgresql['md5_auth_cidr_addresses'] = ['198.51.100.2/32'] + + ## + ## Database credentials password (defined previously in primary node) + ## - replicate same values here as defined in primary node + ## + postgresql['sql_user_password'] = 'fca0b89a972d69f00eb3ec98a5838484' + gitlab_rails['db_password'] = 'mypassword' + + ## + ## Enable FDW support for the Geo Tracking Database (improves performance) + ## + geo_secondary['db_fdw'] = true + ``` + + For external PostgreSQL instances, see [additional instructions](external_database.md). + If you bring a former **primary** node back online to serve as a **secondary** node, then you also need to remove `roles ['geo_primary_role']` or `geo_primary_role['enable'] = true`. + +1. Reconfigure GitLab for the changes to take effect: + + ```sh + gitlab-ctl reconfigure + ``` + +1. Restart PostgreSQL for the IP change to take effect and reconfigure again: + + ```sh + gitlab-ctl restart postgresql + gitlab-ctl reconfigure + ``` + + This last reconfigure will provision the FDW configuration and enable it. + +### Step 3. Initiate the replication process + +Below we provide a script that connects the database on the **secondary** node to +the database on the **primary** node, replicates the database, and creates the +needed files for streaming replication. + +The directories used are the defaults that are set up in Omnibus. If you have +changed any defaults or are using a source installation, configure it as you +see fit replacing the directories and paths. + +CAUTION: **Warning:** +Make sure to run this on the **secondary** server as it removes all PostgreSQL's +data before running `pg_basebackup`. + +1. SSH into your GitLab **secondary** server and login as root: + + ```sh + sudo -i + ``` + +1. Choose a database-friendly name to use for your **secondary** node to + use as the replication slot name. For example, if your domain is + `secondary.geo.example.com`, you may use `secondary_example` as the slot + name as shown in the commands below. + +1. Execute the command below to start a backup/restore and begin the replication + CAUTION: **Warning:** Each Geo **secondary** node must have its own unique replication slot name. + Using the same slot name between two secondaries will break PostgreSQL replication. + + ```sh + gitlab-ctl replicate-geo-database --slot-name=secondary_example --host=198.51.100.1 + ``` + + When prompted, enter the _plaintext_ password you set up for the `gitlab_replicator` + user in the first step. + + This command also takes a number of additional options. You can use `--help` + to list them all, but here are a couple of tips: + - If PostgreSQL is listening on a non-standard port, add `--port=` as well. + - If your database is too large to be transferred in 30 minutes, you will need + to increase the timeout, e.g., `--backup-timeout=3600` if you expect the + initial replication to take under an hour. + - Pass `--sslmode=disable` to skip PostgreSQL TLS authentication altogether + (e.g., you know the network path is secure, or you are using a site-to-site + VPN). This is **not** safe over the public Internet! + - You can read more details about each `sslmode` in the + [PostgreSQL documentation][pg-docs-ssl]; + the instructions above are carefully written to ensure protection against + both passive eavesdroppers and active "man-in-the-middle" attackers. + - Change the `--slot-name` to the name of the replication slot + to be used on the **primary** database. The script will attempt to create the + replication slot automatically if it does not exist. + - If you're repurposing an old server into a Geo **secondary** node, you'll need to + add `--force` to the command line. + - When not in a production machine you can disable backup step if you + really sure this is what you want by adding `--skip-backup` + +The replication process is now complete. + +## PGBouncer support (optional) + +[PGBouncer](http://pgbouncer.github.io/) may be used with GitLab Geo to pool +PostgreSQL connections. We recommend using PGBouncer if you use GitLab in a +high-availability configuration with a cluster of nodes supporting a Geo +**primary** node and another cluster of nodes supporting a Geo **secondary** node. For more +information, see the [Omnibus HA](https://docs.gitlab.com/ee/administration/high_availability/database.html#configure-using-omnibus-for-high-availability) +documentation. + +For a Geo **secondary** node to work properly with PGBouncer in front of the database, +it will need a separate read-only user to make [PostgreSQL FDW queries][FDW] +work: + +1. On the **primary** Geo database, enter the PostgreSQL on the console as an + admin user. If you are using an Omnibus-managed database, log onto the **primary** + node that is running the PostgreSQL database: + + ```sh + sudo -u gitlab-psql /opt/gitlab/embedded/bin/psql -h /var/opt/gitlab/postgresql gitlabhq_production + ``` + +1. Then create the read-only user: + + ```sql + -- NOTE: Use the password defined earlier + CREATE USER gitlab_geo_fdw WITH password 'mypassword'; + GRANT CONNECT ON DATABASE gitlabhq_production to gitlab_geo_fdw; + GRANT USAGE ON SCHEMA public TO gitlab_geo_fdw; + GRANT SELECT ON ALL TABLES IN SCHEMA public TO gitlab_geo_fdw; + GRANT SELECT ON ALL SEQUENCES IN SCHEMA public TO gitlab_geo_fdw; + + -- Tables created by "gitlab" should be made read-only for "gitlab_geo_fdw" + -- automatically. + ALTER DEFAULT PRIVILEGES FOR USER gitlab IN SCHEMA public GRANT SELECT ON TABLES TO gitlab_geo_fdw; + ALTER DEFAULT PRIVILEGES FOR USER gitlab IN SCHEMA public GRANT SELECT ON SEQUENCES TO gitlab_geo_fdw; + ``` + +1. On the **secondary** nodes, change `/etc/gitlab/gitlab.rb`: + + ``` + geo_postgresql['fdw_external_user'] = 'gitlab_geo_fdw' + ``` + +1. Save the file and reconfigure GitLab for the changes to be applied: + + ```sh + gitlab-ctl reconfigure + ``` + +## MySQL replication + +MySQL replication is not supported for Geo. + +## Troubleshooting + +Read the [troubleshooting document](troubleshooting.md). + +[replication-slots-article]: https://medium.com/@tk512/replication-slots-in-postgresql-b4b03d277c75 +[pgback]: http://www.postgresql.org/docs/9.2/static/app-pgbasebackup.html +[replication user]:https://wiki.postgresql.org/wiki/Streaming_Replication +[FDW]: https://www.postgresql.org/docs/9.6/static/postgres-fdw.html +[toc]: index.md#using-omnibus-gitlab +[rake-maintenance]: ../../raketasks/maintenance.md +[pg-docs-ssl]: https://www.postgresql.org/docs/9.6/static/libpq-ssl.html#LIBPQ-SSL-PROTECTION +[pg-docs-runtime-conn]: https://www.postgresql.org/docs/9.6/static/runtime-config-connection.html +[pg-docs-runtime-replication]: https://www.postgresql.org/docs/9.6/static/runtime-config-replication.html diff --git a/doc/administration/geo/replication/database_source.md b/doc/administration/geo/replication/database_source.md new file mode 100644 index 00000000000..a6b990ebc43 --- /dev/null +++ b/doc/administration/geo/replication/database_source.md @@ -0,0 +1,431 @@ +# Geo database replication (source) + +NOTE: **Note:** +This documentation applies to GitLab source installations. In GitLab 11.5, this documentation was deprecated and will be removed in a future release. +Please consider [migrating to GitLab Omnibus install](https://docs.gitlab.com/omnibus/update/convert_to_omnibus.html). For installations +using the Omnibus GitLab packages, follow the +[**database replication for Omnibus GitLab**][database] guide. + +NOTE: **Note:** +The stages of the setup process must be completed in the documented order. +Before attempting the steps in this stage, [complete all prior stages](index.md#using-gitlab-installed-from-source-deprecated). + +This document describes the minimal steps you have to take in order to +replicate your **primary** GitLab database to a **secondary** node's database. You may +have to change some values according to your database setup, how big it is, etc. + +You are encouraged to first read through all the steps before executing them +in your testing/production environment. + +## PostgreSQL replication + +The GitLab **primary** node where the write operations happen will connect to +**primary** database server, and the **secondary** ones which are read-only will +connect to **secondary** database servers (which are read-only too). + +NOTE: **Note:** +In many databases' documentation, you will see "**primary**" being referenced as "master" +and "**secondary**" as either "slave" or "standby" server (read-only). + +We recommend using [PostgreSQL replication slots][replication-slots-article] +to ensure the **primary** node retains all the data necessary for the secondaries to +recover. See below for more details. + +The following guide assumes that: + +- You are using PostgreSQL 9.6 or later which includes the + [`pg_basebackup` tool][pgback] and improved [Foreign Data Wrapper][FDW] support. +- You have a **primary** node already set up (the GitLab server you are + replicating from), running PostgreSQL 9.6 or later, and + you have a new **secondary** server set up with the same versions of the OS, + PostgreSQL, and GitLab on all nodes. +- The IP of the **primary** server for our examples is `198.51.100.1`, whereas the + **secondary** node's IP is `198.51.100.2`. Note that the **primary** and **secondary** servers + **must** be able to communicate over these addresses. These IP addresses can either + be public or private. + +CAUTION: **Warning:** +Geo works with streaming replication. Logical replication is not supported at this time. +There is an [issue where support is being discussed](https://gitlab.com/gitlab-org/gitlab-ee/issues/7420). + +### Step 1. Configure the **primary** server + +1. SSH into your GitLab **primary** server and login as root: + + ```sh + sudo -i + ``` + +1. Add this node as the Geo **primary** by running: + + ```sh + bundle exec rake geo:set_primary_node + ``` + +1. Create a [replication user] named `gitlab_replicator`: + + ```sql + --- Create a new user 'replicator' + CREATE USER gitlab_replicator; + + --- Set/change a password and grants replication privilege + ALTER USER gitlab_replicator WITH REPLICATION ENCRYPTED PASSWORD 'replicationpasswordhere'; + ``` + +1. Make sure your the `gitlab` database user has a password defined: + + ```sh + sudo -u postgres psql -d template1 -c "ALTER USER gitlab WITH ENCRYPTED PASSWORD 'mydatabasepassword';" + ``` + +1. Edit the content of `database.yml` in `production:` and add the password like the example below: + + ```yaml + # + # PRODUCTION + # + production: + adapter: postgresql + encoding: unicode + database: gitlabhq_production + pool: 10 + username: gitlab + password: mydatabasepassword + host: /var/opt/gitlab/geo-postgresql + ``` + +1. Set up TLS support for the PostgreSQL **primary** server: + + CAUTION: **Warning**: + Only skip this step if you **know** that PostgreSQL traffic + between the **primary** and **secondary** nodes will be secured through some other + means, e.g., a known-safe physical network path or a site-to-site VPN that + you have configured. + + If you are replicating your database across the open Internet, it is + **essential** that the connection is TLS-secured. Correctly configured, this + provides protection against both passive eavesdroppers and active + "man-in-the-middle" attackers. + + To generate a self-signed certificate and key, run this command: + + ```sh + openssl req -nodes -batch -x509 -newkey rsa:4096 -keyout server.key -out server.crt -days 3650 + ``` + + This will create two files - `server.key` and `server.crt` - that you can + use for authentication. + + Copy them to the correct location for your PostgreSQL installation: + + ```sh + # Copying a self-signed certificate and key + install -o postgres -g postgres -m 0400 -T server.crt ~postgres/9.x/main/data/server.crt + install -o postgres -g postgres -m 0400 -T server.key ~postgres/9.x/main/data/server.key + ``` + + Add this configuration to `postgresql.conf`, removing any existing + configuration for `ssl_cert_file` or `ssl_key_file`: + + ``` + ssl = on + ssl_cert_file='server.crt' + ssl_key_file='server.key' + ``` + +1. Edit `postgresql.conf` to configure the **primary** server for streaming replication + (for Debian/Ubuntu that would be `/etc/postgresql/9.x/main/postgresql.conf`): + + ``` + listen_address = '198.51.100.1' + wal_level = hot_standby + max_wal_senders = 5 + min_wal_size = 80MB + max_wal_size = 1GB + max_replicaton_slots = 1 # Number of Geo secondary nodes + wal_keep_segments = 10 + hot_standby = on + ``` + + NOTE: **Note**: + Be sure to set `max_replication_slots` to the number of Geo **secondary** + nodes that you may potentially have (at least 1). + + For security reasons, PostgreSQL by default only listens on the local + interface (e.g. 127.0.0.1). However, Geo needs to communicate + between the **primary** and **secondary** nodes over a common network, such as a + corporate LAN or the public Internet. For this reason, we need to + configure PostgreSQL to listen on more interfaces. + + The `listen_address` option opens PostgreSQL up to external connections + with the interface corresponding to the given IP. See [the PostgreSQL + documentation][pg-docs-runtime-conn] for more details. + + You may also want to edit the `wal_keep_segments` and `max_wal_senders` to + match your database replication requirements. Consult the + [PostgreSQL - Replication documentation][pg-docs-runtime-replication] for more information. + +1. Set the access control on the **primary** node to allow TCP connections using the + server's public IP and set the connection from the **secondary** node to require a + password. Edit `pg_hba.conf` (for Debian/Ubuntu that would be + `/etc/postgresql/9.x/main/pg_hba.conf`): + + ```sh + host all all 198.51.100.1/32 md5 + host replication gitlab_replicator 198.51.100.2/32 md5 + ``` + + Where `198.51.100.1` is the public IP address of the **primary** server, and `198.51.100.2` + the public IP address of the **secondary** one. If you want to add another + secondary, add one more row like the replication one and change the IP + address: + + ```sh + host all all 198.51.100.1/32 md5 + host replication gitlab_replicator 198.51.100.2/32 md5 + host replication gitlab_replicator 198.51.100.3/32 md5 + ``` + +1. Restart PostgreSQL for the changes to take effect. + +1. Choose a database-friendly name to use for your secondary to use as the + replication slot name. For example, if your domain is + `secondary.geo.example.com`, you may use `secondary_example` as the slot + name. + +1. Create the replication slot on the **primary** node: + + ```sh + $ sudo -u postgres psql -c "SELECT * FROM pg_create_physical_replication_slot('secondary_example');" + slot_name | xlog_position + ------------------+--------------- + secondary_example | + (1 row) + ``` + +1. Now that the PostgreSQL server is set up to accept remote connections, run + `netstat -plnt` to make sure that PostgreSQL is listening to the server's + public IP. + +### Step 2. Configure the secondary server + +Follow the first steps in ["configure the secondary server"][database-replication] and note that since you are installing from source, the username and +group listed as `gitlab-psql` in those steps should be replaced by `postgres` +instead. After completing the "Test that the `gitlab-psql` user can connect to +the **primary** node's database" step, continue here: + +1. Edit `postgresql.conf` to configure the secondary for streaming replication + (for Debian/Ubuntu that would be `/etc/postgresql/9.*/main/postgresql.conf`): + + ```sh + wal_level = hot_standby + max_wal_senders = 5 + checkpoint_segments = 10 + wal_keep_segments = 10 + hot_standby = on + ``` + +1. Restart PostgreSQL for the changes to take effect. + +#### Enable tracking database on the secondary server + +Geo secondary nodes use a tracking database to keep track of replication status +and recover automatically from some replication issues. Follow the steps below to create +the tracking database. + +1. On the secondary node, run the following command to create `database_geo.yml` with the + information of your secondary PostgreSQL instance: + + ```sh + sudo cp /home/git/gitlab/config/database_geo.yml.postgresql /home/git/gitlab/config/database_geo.yml + ``` + +1. Edit the content of `database_geo.yml` in `production:` as in the example below: + + ```yaml + # + # PRODUCTION + # + production: + adapter: postgresql + encoding: unicode + database: gitlabhq_geo_production + pool: 10 + username: gitlab_geo + # password: + host: /var/opt/gitlab/geo-postgresql + ``` + +1. Create the database `gitlabhq_geo_production` on the PostgreSQL instance of the **secondary** node. + +1. Set up the Geo tracking database: + + ```sh + bundle exec rake geo:db:migrate + ``` + +1. Configure the [PostgreSQL FDW][FDW] connection and credentials: + + Save the script below in a file, ex. `/tmp/geo_fdw.sh` and modify the connection + params to match your environment. Execute it to set up the FDW connection. + + ```sh + #!/bin/bash + + # Secondary Database connection params: + DB_HOST="/var/opt/gitlab/postgresql" # change to the public IP or VPC private IP if its an external server + DB_NAME="gitlabhq_production" + DB_USER="gitlab" + DB_PORT="5432" + + # Tracking Database connection params: + GEO_DB_HOST="/var/opt/gitlab/geo-postgresql" # change to the public IP or VPC private IP if its an external server + GEO_DB_NAME="gitlabhq_geo_production" + GEO_DB_USER="gitlab_geo" + GEO_DB_PORT="5432" + + query_exec () { + gitlab-psql -h $GEO_DB_HOST -d $GEO_DB_NAME -p $GEO_DB_PORT -c "${1}" + } + + query_exec "CREATE EXTENSION postgres_fdw;" + query_exec "CREATE SERVER gitlab_secondary FOREIGN DATA WRAPPER postgres_fdw OPTIONS (host '${DB_HOST}', dbname '${DB_NAME}', port '${DB_PORT}');" + query_exec "CREATE USER MAPPING FOR ${GEO_DB_USER} SERVER gitlab_secondary OPTIONS (user '${DB_USER}');" + query_exec "CREATE SCHEMA gitlab_secondary;" + query_exec "GRANT USAGE ON FOREIGN SERVER gitlab_secondary TO ${GEO_DB_USER};" + ``` + + And edit the content of `database_geo.yml` and to add `fdw: true` to + the `production:` block. + +### Step 3. Initiate the replication process + +Below we provide a script that connects the database on the **secondary** node to +the database on the **primary** node, replicates the database, and creates the +needed files for streaming replication. + +The directories used are the defaults for Debian/Ubuntu. If you have changed +any defaults, configure it as you see fit replacing the directories and paths. + +CAUTION: **Warning:** +Make sure to run this on the **secondary** server as it removes all PostgreSQL's +data before running `pg_basebackup`. + +1. SSH into your GitLab **secondary** server and login as root: + + ```sh + sudo -i + ``` + +1. Save the snippet below in a file, let's say `/tmp/replica.sh`. Modify the + embedded paths if necessary: + + ``` + #!/bin/bash + + PORT="5432" + USER="gitlab_replicator" + echo --------------------------------------------------------------- + echo WARNING: Make sure this script is run from the secondary server + echo --------------------------------------------------------------- + echo + echo Enter the IP or FQDN of the primary PostgreSQL server + read HOST + echo Enter the password for $USER@$HOST + read -s PASSWORD + echo Enter the required sslmode + read SSLMODE + + echo Stopping PostgreSQL and all GitLab services + sudo service gitlab stop + sudo service postgresql stop + + echo Backing up postgresql.conf + sudo -u postgres mv /var/opt/gitlab/postgresql/data/postgresql.conf /var/opt/gitlab/postgresql/ + + echo Cleaning up old cluster directory + sudo -u postgres rm -rf /var/opt/gitlab/postgresql/data + + echo Starting base backup as the replicator user + echo Enter the password for $USER@$HOST + sudo -u postgres /opt/gitlab/embedded/bin/pg_basebackup -h $HOST -D /var/opt/gitlab/postgresql/data -U gitlab_replicator -v -x -P + + echo Writing recovery.conf file + sudo -u postgres bash -c "cat > /var/opt/gitlab/postgresql/data/recovery.conf <<- _EOF1_ + standby_mode = 'on' + primary_conninfo = 'host=$HOST port=$PORT user=$USER password=$PASSWORD sslmode=$SSLMODE' + _EOF1_ + " + + echo Restoring postgresql.conf + sudo -u postgres mv /var/opt/gitlab/postgresql/postgresql.conf /var/opt/gitlab/postgresql/data/ + + echo Starting PostgreSQL + sudo service postgresql start + ``` + +1. Run it with: + + ```sh + bash /tmp/replica.sh + ``` + + When prompted, enter the IP/FQDN of the **primary** node, and the password you set up + for the `gitlab_replicator` user in the first step. + + You should use `verify-ca` for the `sslmode`. You can use `disable` if you + are happy to skip PostgreSQL TLS authentication altogether (e.g., you know + the network path is secure, or you are using a site-to-site VPN). This is + **not** safe over the public Internet! + + You can read more details about each `sslmode` in the + [PostgreSQL documentation][pg-docs-ssl]; + the instructions above are carefully written to ensure protection against + both passive eavesdroppers and active "man-in-the-middle" attackers. + +The replication process is now over. + +## PGBouncer support (optional) + +1. First, enter the PostgreSQL console as an admin user. + +1. Then create the read-only user: + + ```sql + -- NOTE: Use the password defined earlier + CREATE USER gitlab_geo_fdw WITH password 'mypassword'; + GRANT CONNECT ON DATABASE gitlabhq_production to gitlab_geo_fdw; + GRANT USAGE ON SCHEMA public TO gitlab_geo_fdw; + GRANT SELECT ON ALL TABLES IN SCHEMA public TO gitlab_geo_fdw; + GRANT SELECT ON ALL SEQUENCES IN SCHEMA public TO gitlab_geo_fdw; + + -- Tables created by "gitlab" should be made read-only for "gitlab_geo_fdw" + -- automatically. + ALTER DEFAULT PRIVILEGES FOR USER gitlab IN SCHEMA public GRANT SELECT ON TABLES TO gitlab_geo_fdw; + ALTER DEFAULT PRIVILEGES FOR USER gitlab IN SCHEMA public GRANT SELECT ON SEQUENCES TO gitlab_geo_fdw; + ``` + +1. Enter the PostgreSQL console on the **secondary** tracking database and change the user mapping to this new user: + + ``` + ALTER USER MAPPING FOR gitlab_geo SERVER gitlab_secondary OPTIONS (SET user 'gitlab_geo_fdw') + ``` + +## MySQL replication + +MySQL replication is not supported for Geo. + +## Troubleshooting + +Read the [troubleshooting document](troubleshooting.md). + +[replication-slots-article]: https://medium.com/@tk512/replication-slots-in-postgresql-b4b03d277c75 +[pgback]: http://www.postgresql.org/docs/9.6/static/app-pgbasebackup.html +[replication user]:https://wiki.postgresql.org/wiki/Streaming_Replication +[FDW]: https://www.postgresql.org/docs/9.6/static/postgres-fdw.html +[database]: database.md +[add-geo-node]: configuration.md#step-3-add-the-secondary-gitlab-node +[database-replication]: database.md#step-2-configure-the-secondary-server +[pg-docs-ssl]: https://www.postgresql.org/docs/9.6/static/libpq-ssl.html#LIBPQ-SSL-PROTECTION +[pg-docs-runtime-conn]: https://www.postgresql.org/docs/9.6/static/runtime-config-connection.html +[pg-docs-runtime-replication]: https://www.postgresql.org/docs/9.6/static/runtime-config-replication.html diff --git a/doc/administration/geo/replication/docker_registry.md b/doc/administration/geo/replication/docker_registry.md new file mode 100644 index 00000000000..8a5bca1708f --- /dev/null +++ b/doc/administration/geo/replication/docker_registry.md @@ -0,0 +1,23 @@ +# Docker Registry for a secondary node + +You can set up a [Docker Registry] on your +**secondary** Geo node that mirrors the one on the **primary** Geo node. + +## Storage support + +CAUTION: **Warning:** +If you use [local storage][registry-storage] +for the Container Registry you **cannot** replicate it to a **secondary** node. + +Docker Registry currently supports a few types of storages. If you choose a +distributed storage (`azure`, `gcs`, `s3`, `swift`, or `oss`) for your Docker +Registry on the **primary** node, you can use the same storage for a **secondary** +Docker Registry as well. For more information, read the +[Load balancing considerations][registry-load-balancing] +when deploying the Registry, and how to set up the storage driver for GitLab's +integrated [Container Registry][registry-storage]. + +[ee]: https://about.gitlab.com/pricing/ +[Docker Registry]: https://docs.docker.com/registry/ +[registry-storage]: ../../container_registry.md#container-registry-storage-driver +[registry-load-balancing]: https://docs.docker.com/registry/deploying/#load-balancing-considerations diff --git a/doc/administration/geo/replication/external_database.md b/doc/administration/geo/replication/external_database.md new file mode 100644 index 00000000000..18e0c75f703 --- /dev/null +++ b/doc/administration/geo/replication/external_database.md @@ -0,0 +1,169 @@ +# Geo with external PostgreSQL instances + +This document is relevant if you are using a PostgreSQL instance that is *not +managed by Omnibus*. This includes cloud-managed instances like AWS RDS, or +manually installed and configured PostgreSQL instances. + +NOTE: **Note**: +We strongly recommend running Omnibus-managed instances as they are actively +developed and tested. We aim to be compatible with most external +(not managed by Omnibus) databases but we do not guarantee compatibility. + +## **Primary** node + +### Configure the external database to be replicated + +To set up an external database, you can either: + +- Set up streaming replication yourself (for example, in AWS RDS). +- Perform the Omnibus configuration manually as follows. + +In an Omnibus install, the +[geo_primary_role](https://docs.gitlab.com/omnibus/roles/#gitlab-geo-roles) +configures the **primary** node's database to be replicated by making changes to +`pg_hba.conf` and `postgresql.conf`. Make the following configuration changes +manually to your external database configuration: + +``` +## +## Geo Primary Role +## - pg_hba.conf +## +host replication gitlab_replicator <trusted secondary IP>/32 md5 +``` + +``` +## +## Geo Primary Role +## - postgresql.conf +## +sql_replication_user = gitlab_replicator +wal_level = hot_standby +max_wal_senders = 10 +wal_keep_segments = 50 +max_replication_slots = 1 # number of secondary instances +hot_standby = on +``` + +## **Secondary** nodes + +With Omnibus, the +[geo_secondary_role](https://docs.gitlab.com/omnibus/roles/#gitlab-geo-roles) +has three main functions: + +1. Configure the replica database. +1. Configure the tracking database. +1. Enable the Geo Log Cursor (`geo_logcursor`) (irrelevant to this doc). + +### Configure the external replica database + +To set up an external replica database, you can either: + +- Set up streaming replication yourself (for example, in AWS RDS). +- Perform the Omnibus configuration manually as follows. + +In an Omnibus install, the `geo_secondary_role` makes configuration changes to +`postgresql.conf`. Make the following configuration changes manually to your +external replica database configuration: + +``` +## +## Geo Secondary Role +## - postgresql.conf +## +wal_level = hot_standby +max_wal_senders = 10 +wal_keep_segments = 10 +hot_standby = on +``` + +### Configure the tracking database + +**Secondary** nodes use a separate PostgreSQL installation as a tracking +database to keep track of replication status and automatically recover from +potential replication issues. + +It requires an [FDW](https://www.postgresql.org/docs/9.6/static/postgres-fdw.html) +connection with the **secondary** replica database for improved performance. + +If you have an external database ready to be used as the tracking database, +follow the instructions below to use it: + +1. SSH into a GitLab **secondary** server and login as root: + + ```bash + sudo -i + ``` + +1. Edit `/etc/gitlab/gitlab.rb` with the connection params and credentials for + the machine with the PostgreSQL instance: + + ```ruby + # note this is shared between both databases, + # make sure you define the same password in both + gitlab_rails['db_password'] = 'mypassword' + + geo_secondary['db_host'] = '<change to the tracking DB public IP>' + geo_secondary['db_port'] = 5431 # change to the correct port + geo_secondary['db_fdw'] = true # enable FDW + geo_postgresql['enable'] = false # don't use internal managed instance + ``` + +1. Reconfigure GitLab for the changes to take effect: + + ```bash + gitlab-ctl reconfigure + ``` + +1. Run the tracking database migrations: + + ```bash + gitlab-rake geo:db:migrate + ``` + +1. Configure the + [PostgreSQL FDW](https://www.postgresql.org/docs/9.6/static/postgres-fdw.html) + connection and credentials: + + Save the script below in a file, ex. `/tmp/geo_fdw.sh` and modify the connection + params to match your environment. Execute it to set up the FDW connection. + + ```bash + #!/bin/bash + + # Secondary Database connection params: + DB_HOST="<change to the public IP or VPC private IP>" + DB_NAME="gitlabhq_production" + DB_USER="gitlab" + DB_PORT="5432" + + # Tracking Database connection params: + GEO_DB_HOST="<change to the public IP or VPC private IP>" + GEO_DB_NAME="gitlabhq_geo_production" + GEO_DB_USER="gitlab_geo" + GEO_DB_PORT="5432" + + query_exec () { + gitlab-psql -h $GEO_DB_HOST -d $GEO_DB_NAME -p $GEO_DB_PORT -c "${1}" + } + + query_exec "CREATE EXTENSION postgres_fdw;" + query_exec "CREATE SERVER gitlab_secondary FOREIGN DATA WRAPPER postgres_fdw OPTIONS (host '${DB_HOST}', dbname '${DB_NAME}', port '${DB_PORT}');" + query_exec "CREATE USER MAPPING FOR ${GEO_DB_USER} SERVER gitlab_secondary OPTIONS (user '${DB_USER}');" + query_exec "CREATE SCHEMA gitlab_secondary;" + query_exec "GRANT USAGE ON FOREIGN SERVER gitlab_secondary TO ${GEO_DB_USER};" + ``` + + NOTE: **Note:** The script template above uses `gitlab-psql` as it's intended to be executed from the Geo machine, + but you can change it to `psql` and run it from any machine that has access to the database. + +1. Restart GitLab: + + ```bash + gitlab-ctl restart + ``` +1. Populate the FDW tables: + + ```bash + gitlab-rake geo:db:refresh_foreign_tables + ``` diff --git a/doc/administration/geo/replication/faq.md b/doc/administration/geo/replication/faq.md new file mode 100644 index 00000000000..996542f9f2e --- /dev/null +++ b/doc/administration/geo/replication/faq.md @@ -0,0 +1,62 @@ +# Geo Frequently Asked Questions + +## What are the minimum requirements to run Geo? + +The requirements are listed [on the index page](index.md#requirements-for-running-geo) + +## How does Geo know which projects to sync? + +On each **secondary** node, there is a read-only replicated copy of the GitLab database. +A **secondary** node also has a tracking database where it stores which projects have been synced. +Geo compares the two databases to find projects that are not yet tracked. + +At the start, this tracking database is empty, so Geo will start trying to update from every project that it can see in the GitLab database. + +For each project to sync: + +1. Geo will issue a `git fetch geo --mirror` to get the latest information from the **primary** node. +If there are no changes, the sync will be fast and end quickly. Otherwise, it will pull the latest commits. +1. The **secondary** node will update the tracking database to store the fact that it has synced projects A, B, C, etc. +1. Repeat until all projects are synced. + +When someone pushes a commit to the **primary** node, it generates an event in the GitLab database that the repository has changed. +The **secondary** node sees this event, marks the project in question as dirty, and schedules the project to be resynced. + +To ensure that problems with pipelines (for example, syncs failing too many times or jobs being lost) don't permanently stop projects syncing, Geo also periodically checks the tracking database for projects that are marked as dirty. This check happens when +the number of concurrent syncs falls below `repos_max_capacity` and there are no new projects waiting to be synced. + +Geo also has a checksum feature which runs a SHA256 sum across all the Git references to the SHA values. +If the refs don't match between the **primary** node and the **secondary** node, then the **secondary** node will mark that project as dirty and try to resync it. +So even if we have an outdated tracking database, the validation should activate and find discrepancies in the repository state and resync. + +## Can I use Geo in a disaster recovery situation? + +Yes, but there are limitations to what we replicate (see +[What data is replicated to a **secondary** node?](#what-data-is-replicated-to-a-secondary-node)). + +Read the documentation for [Disaster Recovery](../disaster_recovery/index.md). + +## What data is replicated to a **secondary** node? + +We currently replicate project repositories, LFS objects, generated +attachments / avatars and the whole database. This means user accounts, +issues, merge requests, groups, project data, etc., will be available for +query. + +## Can I git push to a **secondary** node? + +Yes! Pushing directly to a **secondary** node (for both HTTP and SSH, including git-lfs) was [introduced](https://about.gitlab.com/2018/09/22/gitlab-11-3-released/) in [GitLab Premium](https://about.gitlab.com/pricing/#self-managed) 11.3. + +## How long does it take to have a commit replicated to a **secondary** node? + +All replication operations are asynchronous and are queued to be dispatched. Therefore, it depends on a lot of +factors including the amount of traffic, how big your commit is, the +connectivity between your nodes, your hardware, etc. + +## What if the SSH server runs at a different port? + +That's totally fine. We use HTTP(s) to fetch repository changes from the **primary** node to all **secondary** nodes. + +## Is this possible to set up a Docker Registry for a **secondary** node that mirrors the one on the **primary** node? + +Yes. See [Docker Registry for a **secondary** node](docker_registry.md). diff --git a/doc/administration/geo/replication/high_availability.md b/doc/administration/geo/replication/high_availability.md new file mode 100644 index 00000000000..9ff7f5bfefa --- /dev/null +++ b/doc/administration/geo/replication/high_availability.md @@ -0,0 +1,228 @@ +# Geo High Availability + +This document describes a minimal reference architecture for running Geo +in a high availability configuration. If your HA setup differs from the one +described, it is possible to adapt these instructions to your needs. + +## Architecture overview + + + +_[diagram source - gitlab employees only][diagram-source]_ + +The topology above assumes that the **primary** and **secondary** Geo clusters +are located in two separate locations, on their own virtual network +with private IP addresses. The network is configured such that all machines within +one geographic location can communicate with each other using their private IP addresses. +The IP addresses given are examples and may be different depending on the +network topology of your deployment. + +The only external way to access the two Geo deployments is by HTTPS at +`gitlab.us.example.com` and `gitlab.eu.example.com` in the example above. + +NOTE: **Note:** +The **primary** and **secondary** Geo deployments must be able to communicate to each other over HTTPS. + +## Redis and PostgreSQL High Availability + +The **primary** and **secondary** Redis and PostgreSQL should be configured +for high availability. Because of the additional complexity involved +in setting up this configuration for PostgreSQL and Redis, +it is not covered by this Geo HA documentation. + +For more information about setting up a highly available PostgreSQL cluster and Redis cluster using the omnibus package see the high availability documentation for +[PostgreSQL](../../high_availability/database.md) and +[Redis](../../high_availability/redis.md), respectively. + +NOTE: **Note:** +It is possible to use cloud hosted services for PostgreSQL and Redis, but this is beyond the scope of this document. + +## Prerequisites: A working GitLab HA cluster + +This cluster will serve as the **primary** node. Use the +[GitLab HA documentation](../../high_availability/README.md) to set this up. + +## Configure the GitLab cluster to be the **primary** node + +The following steps enable a GitLab cluster to serve as the **primary** node. + +### Step 1: Configure the **primary** frontend servers + +1. Edit `/etc/gitlab/gitlab.rb` and add the following: + + ```ruby + ## + ## Enable the Geo primary role + ## + roles ['geo_primary_role'] + + ## + ## Disable automatic migrations + ## + gitlab_rails['auto_migrate'] = false + ``` + +After making these changes, [reconfigure GitLab][gitlab-reconfigure] so the changes take effect. + +NOTE: **Note:** PostgreSQL and Redis should have already been disabled on the +application servers, and connections from the application servers to those +services on the backend servers configured, during normal GitLab HA set up. See +high availability configuration documentation for +[PostgreSQL](../../high_availability/database.md#configuring-the-application-nodes) +and [Redis](../../high_availability/redis.md#example-configuration-for-the-gitlab-application). + +The **primary** database will require modification later, as part of +[step 2](#step-2-configure-the-main-read-only-replica-postgresql-database-on-the-secondary-node). + +## Configure a **secondary** node + +A **secondary** cluster is similar to any other GitLab HA cluster, with two +major differences: + +* The main PostgreSQL database is a read-only replica of the **primary** node's + PostgreSQL database. +* There is also a single PostgreSQL database for the **secondary** cluster, + called the "tracking database", which tracks the synchronization state of + various resources. + +Therefore, we will set up the HA components one-by-one, and include deviations +from the normal HA setup. + +### Step 1: Configure the Redis and NFS services on the **secondary** node + +Configure the following services, again using the non-Geo high availability +documentation: + +* [Configuring Redis for GitLab HA](../../high_availability/redis.md) for high + availability. +* [NFS](../../high_availability/nfs.md) which will store data that is + synchronized from the **primary** node. + +### Step 2: Configure the main read-only replica PostgreSQL database on the **secondary** node + +NOTE: **Note:** The following documentation assumes the database will be run on +only a single machine, rather than as a PostgreSQL cluster. + +Configure the [**secondary** database](database.md) as a read-only replica of +the **primary** database. + +If using an external PostgreSQL instance, refer also to +[Geo with external PostgreSQL instances](external_database.md). + +### Step 3: Configure the tracking database on the **secondary** node + +NOTE: **Note:** This documentation assumes the tracking database will be run on +only a single machine, rather than as a PostgreSQL cluster. + +Configure the tracking database. + +1. Edit `/etc/gitlab/gitlab.rb` in the tracking database machine, and add the + following: + + ```ruby + ## + ## Enable the Geo secondary tracking database + ## + geo_postgresql['enable'] = true + geo_postgresql['ha'] = true + ``` + +After making these changes [Reconfigure GitLab][gitlab-reconfigure] so the changes take effect. + +If using an external PostgreSQL instance, refer also to +[Geo with external PostgreSQL instances](external_database.md). + +### Step 4: Configure the frontend application servers on the **secondary** node + +In the architecture overview, there are two machines running the GitLab +application services. These services are enabled selectively in the +configuration. + +Configure the application servers following +[Configuring GitLab for HA](../../high_availability/gitlab.md), then make the +following modifications: + +1. Edit `/etc/gitlab/gitlab.rb` on each application server in the **secondary** + cluster, and add the following: + + ```ruby + ## + ## Enable the Geo secondary role + ## + roles ['geo_secondary_role', 'application_role'] + + ## + ## Disable automatic migrations + ## + gitlab_rails['auto_migrate'] = false + + ## + ## Configure the connection to the tracking DB. And disable application + ## servers from running tracking databases. + ## + geo_secondary['db_host'] = '10.1.4.1' + geo_secondary['db_password'] = 'plaintext Geo tracking DB password' + geo_postgresql['enable'] = false + + ## + ## Configure connection to the streaming replica database, if you haven't + ## already + ## + gitlab_rails['db_host'] = '10.1.3.1' + gitlab_rails['db_password'] = 'plaintext DB password' + + ## + ## Configure connection to Redis, if you haven't already + ## + gitlab_rails['redis_host'] = '10.1.2.1' + gitlab_rails['redis_password'] = 'Redis password' + + ## + ## If you are using custom users not managed by Omnibus, you need to specify + ## UIDs and GIDs like below, and ensure they match between servers in a + ## cluster to avoid permissions issues + ## + user['uid'] = 9000 + user['gid'] = 9000 + web_server['uid'] = 9001 + web_server['gid'] = 9001 + registry['uid'] = 9002 + registry['gid'] = 9002 + ``` +NOTE: **Note:** +If you had set up PostgreSQL cluster using the omnibus package and you had set +up `postgresql['sql_user_password'] = 'md5 digest of secret'` setting, keep in +mind that `gitlab_rails['db_password']` and `geo_secondary['db_password']` +mentioned above contains the plaintext passwords. This is used to let the Rails +servers connect to the databases. + +NOTE: **Note:** +Make sure that current node IP is listed in `postgresql['md5_auth_cidr_addresses']` setting of your remote database. + +After making these changes [Reconfigure GitLab][gitlab-reconfigure] so the changes take effect. + +On the secondary the following GitLab frontend services will be enabled: + +* geo-logcursor +* gitlab-pages +* gitlab-workhorse +* logrotate +* nginx +* registry +* remote-syslog +* sidekiq +* unicorn + +Verify these services by running `sudo gitlab-ctl status` on the frontend +application servers. + +### Step 5: Set up the LoadBalancer for the **secondary** node + +In this topology, a load balancer is required at each geographic location to +route traffic to the application servers. + +See [Load Balancer for GitLab HA](../../high_availability/load_balancer.md) for +more information. + +[diagram-source]: https://docs.google.com/drawings/d/1z0VlizKiLNXVVVaERFwgsIOuEgjcUqDTWPdQYsE7Z4c/edit +[gitlab-reconfigure]: ../../restart_gitlab.md#omnibus-gitlab-reconfigure diff --git a/doc/administration/geo/replication/img/geo_architecture.png b/doc/administration/geo/replication/img/geo_architecture.png Binary files differnew file mode 100644 index 00000000000..d318cd5d0f4 --- /dev/null +++ b/doc/administration/geo/replication/img/geo_architecture.png diff --git a/doc/administration/geo/replication/img/geo_node_dashboard.png b/doc/administration/geo/replication/img/geo_node_dashboard.png Binary files differnew file mode 100644 index 00000000000..99792d0770d --- /dev/null +++ b/doc/administration/geo/replication/img/geo_node_dashboard.png diff --git a/doc/administration/geo/replication/img/geo_node_healthcheck.png b/doc/administration/geo/replication/img/geo_node_healthcheck.png Binary files differnew file mode 100644 index 00000000000..33a31f7ab49 --- /dev/null +++ b/doc/administration/geo/replication/img/geo_node_healthcheck.png diff --git a/doc/administration/geo/replication/img/geo_overview.png b/doc/administration/geo/replication/img/geo_overview.png Binary files differnew file mode 100644 index 00000000000..01c1615212c --- /dev/null +++ b/doc/administration/geo/replication/img/geo_overview.png diff --git a/doc/administration/geo/replication/index.md b/doc/administration/geo/replication/index.md new file mode 100644 index 00000000000..2e7aa90f80e --- /dev/null +++ b/doc/administration/geo/replication/index.md @@ -0,0 +1,309 @@ +# Geo Replication **[PREMIUM ONLY]** + +Geo is the solution for widely distributed development teams. + +## Overview + +Fetching large repositories can take a long time for teams located far from a single GitLab instance. + +Geo provides local, read-only instances of your GitLab instances, reducing the time it takes to clone and fetch large repositories and speeding up development. + +> - Geo is part of [GitLab Premium](https://about.gitlab.com/pricing/#self-managed). +> - Introduced in GitLab Enterprise Edition 8.9. +> - We recommend you use: +> - At least GitLab Enterprise Edition 10.0 for basic Geo features. +> - The latest version for a better experience. +> - Make sure that all nodes run the same GitLab version. +> - Geo requires PostgreSQL 9.6 and Git 2.9, in addition to GitLab's usual [minimum requirements](../../../install/requirements.md). +> - Using Geo in combination with [High Availability](../../high_availability/README.md) is considered **Generally Available** (GA) in GitLab [GitLab Premium](https://about.gitlab.com/pricing/) 10.4. + +For a video introduction to Geo, see [Introduction to GitLab Geo - GitLab Features](https://www.youtube.com/watch?v=-HDLxSjEh6w). + +CAUTION: **Caution:** +Geo undergoes significant changes from release to release. Upgrades **are** supported and [documented](#updating-geo), but you should ensure that you're using the right version of the documentation for your installation. + +To make sure you're using the right version of the documentation, navigate to [the source version of this page on GitLab.com](https://gitlab.com/gitlab-org/gitlab-ee/blob/master/doc/administration/geo/replication/index.md) and choose the appropriate release from the **Switch branch/tag** dropdown. For example, [`v11.2.3-ee`](https://gitlab.com/gitlab-org/gitlab-ee/blob/v11.2.3-ee/doc/administration/geo/replication/index.md). + +## Use cases + +Implementing Geo provides the following benefits: + +- Reduce from minutes to seconds the time taken for your distributed developers to clone and fetch large repositories and projects. +- Enable all of your developers to contribute ideas and work in parallel, no matter where they are. +- Balance the load between your **primary** and **secondary** nodes, or offload your automated tests to a **secondary** node. + +In addition, it: + +- Can be used for cloning and fetching projects, in addition to reading any data available in the GitLab web interface (see [current limitations](#current-limitations)). +- Overcomes slow connections between distant offices, saving time by improving speed for distributed teams. +- Helps reducing the loading time for automated tasks, custom integrations, and internal workflows. +- Can quickly fail over to a **secondary** node in a [disaster recovery](../disaster_recovery/index.md) scenario. +- Allows [planned failover](../disaster_recovery/planned_failover.md) to a **secondary** node. + +Geo provides: + +- Read-only **secondary** nodes: Maintain one **primary** GitLab node while still enabling read-only **secondary** nodes for each of your distributed teams. +- Authentication system hooks: **Secondary** nodes receives all authentication data (like user accounts and logins) from the **primary** instance. +- An intuitive UI: **Secondary** nodes utilize the same web interface your team has grown accustomed to. In addition, there are visual notifications that block write operations and make it clear that a user is on a **secondary** node. + +## How it works + +Your Geo instance can be used for cloning and fetching projects, in addition to reading any data. This will make working with large repositories over large distances much faster. + + + +When Geo is enabled, the: + +- Original instance is known as the **primary** node. +- Replicated read-only nodes are known as **secondary** nodes. + +Keep in mind that: + +- **Secondary** nodes talk to the **primary** node to: + - Get user data for logins (API). + - Replicate repositories, LFS Objects, and Attachments (HTTPS + JWT). +- Since GitLab Premium 10.0, the **primary** node no longer talks to **secondary** nodes to notify for changes (API). +- Pushing directly to a **secondary** node (for both HTTP and SSH, including git-lfs) was [introduced](https://about.gitlab.com/2018/09/22/gitlab-11-3-released/) in [GitLab Premium](https://about.gitlab.com/pricing/#self-managed) 11.3. +- There are [limitations](#current-limitations) in the current implementation. + +### Architecture + +The following diagram illustrates the underlying architecture of Geo. + + + +In this diagram: + +- There is the **primary** node and the details of one **secondary** node. +- Writes to the database can only be performed on the **primary** node. A **secondary** node receives database + updates via PostgreSQL streaming replication. +- If present, the [LDAP server](#ldap) should be configured to replicate for [Disaster Recovery](../disaster_recovery/index.md) scenarios. +- A **secondary** node performs different type of synchronizations against the **primary** node, using a special + authorization protected by JWT: + - Repositories are cloned/updated via Git over HTTPS. + - Attachments, LFS objects, and other files are downloaded via HTTPS using a private API endpoint. + +From the perspective of a user performing Git operations: + +- The **primary** node behaves as a full read-write GitLab instance. +- **Secondary** nodes are read-only but proxy Git push operations to the **primary** node. This makes **secondary** nodes appear to support push operations themselves. + +To simplify the diagram, some necessary components are omitted. Note that: + +- Git over SSH requires [`gitlab-shell`](https://gitlab.com/gitlab-org/gitlab-shell) and OpenSSH. +- Git over HTTPS required [`gitlab-workhorse`](https://gitlab.com/gitlab-org/gitlab-workhorse). + +Note that a **secondary** node needs two different PostgreSQL databases: + +- A read-only database instance that streams data from the main GitLab database. +- [Another database instance](#geo-tracking-database) used internally by the **secondary** node to record what data has been replicated. + +In **secondary** nodes, there is an additional daemon: [Geo Log Cursor](#geo-log-cursor). + +## Requirements for running Geo + +The following are required to run Geo: + +- An operating system that supports OpenSSH 6.9+ (needed for + [fast lookup of authorized SSH keys in the database](../../operations/fast_ssh_key_lookup.md)) + The following operating systems are known to ship with a current version of OpenSSH: + - [CentOS](https://www.centos.org) 7.4+ + - [Ubuntu](https://www.ubuntu.com) 16.04+ +- PostgreSQL 9.6+ with [FDW](https://www.postgresql.org/docs/9.6/postgres-fdw.html) support and [Streaming Replication](https://wiki.postgresql.org/wiki/Streaming_Replication) +- Git 2.9+ + +### Firewall rules + +The following table lists basic ports that must be open between the **primary** and **secondary** nodes for Geo. + +| **Primary** node | **Secondary** node | Protocol | +|:-----------------|:-------------------|:-------------| +| 80 | 80 | HTTP | +| 443 | 443 | TCP or HTTPS | +| 22 | 22 | TCP | +| 5432 | | PostgreSQL | + +See the full list of ports used by GitLab in [Package defaults](https://docs.gitlab.com/omnibus/package-information/defaults.html) + +NOTE: **Note:** +[Web terminal](../../../ci/environments.md#web-terminals) support requires your load balancer to correctly handle WebSocket connections. +When using HTTP or HTTPS proxying, your load balancer must be configured to pass through the `Connection` and `Upgrade` hop-by-hop headers. See the [web terminal](../../integration/terminal.md) integration guide for more details. + +NOTE: **Note:** +When using HTTPS protocol for port 443, you will need to add an SSL certificate to the load balancers. +If you wish to terminate SSL at the GitLab application server instead, use TCP protocol. + +### LDAP + +We recommend that if you use LDAP on your **primary** node, you also set up secondary LDAP servers on each **secondary** node. Otherwise, users will not be able to perform Git operations over HTTP(s) on the **secondary** node using HTTP Basic Authentication. However, Git via SSH and personal access tokens will still work. + +NOTE: **Note:** +It is possible for all **secondary** nodes to share an LDAP server, but additional latency can be an issue. Also, consider what LDAP server will be available in a [disaster recovery](../disaster_recovery/index.md) scenario if a **secondary** node is promoted to be a **primary** node. + +Check for instructions on how to set up replication in your LDAP service. Instructions will be different depending on the software or service used. For example, OpenLDAP provides [these instructions](https://www.openldap.org/doc/admin24/replication.html). + +### Geo Tracking Database + +The tracking database instance is used as metadata to control what needs to be updated on the disk of the local instance. For example: + +- Download new assets. +- Fetch new LFS Objects. +- Fetch changes from a repository that has recently been updated. + +Because the replicated database instance is read-only, we need this additional database instance for each **secondary** node. +The tracking database requires the `postgres_fdw` extension. + +### Geo Log Cursor + +This daemon: + +- Reads a log of events replicated by the **primary** node to the **secondary** database instance. +- Updates the Geo Tracking Database instance with changes that need to be executed. + +When something is marked to be updated in the tracking database instance, asynchronous jobs running on the **secondary** node will execute the required operations and update the state. + +This new architecture allows GitLab to be resilient to connectivity issues between the nodes. It doesn't matter how long the **secondary** node is disconnected from the **primary** node as it will be able to replay all the events in the correct order and become synchronized with the **primary** node again. + +## Setup instructions + +These instructions assume you have a working instance of GitLab. They guide you through: + +1. Making your existing instance the **primary** node. +1. Adding **secondary** nodes. + +CAUTION: **Caution:** +The steps below should be followed in the order they appear. **Make sure the GitLab version is the same on all nodes.** + +### Using Omnibus GitLab + +If you installed GitLab using the Omnibus packages (highly recommended): + +1. [Install GitLab Enterprise Edition](https://about.gitlab.com/installation/) on the server that will serve as the **secondary** node. Do not create an account or log in to the new **secondary** node. +1. [Upload the GitLab License](../../../user/admin_area/license.md) on the **primary** node to unlock Geo. The license must be for [GitLab Premium](https://about.gitlab.com/pricing/) or higher. +1. [Set up the database replication](database.md) (`primary (read-write) <-> secondary (read-only)` topology). +1. [Configure fast lookup of authorized SSH keys in the database](../../operations/fast_ssh_key_lookup.md). This step is required and needs to be done on **both** the **primary** and **secondary** nodes. +1. [Configure GitLab](configuration.md) to set the **primary** and **secondary** nodes. +1. Optional: [Configure a secondary LDAP server](../../auth/ldap.md) for the **secondary** node. See [notes on LDAP](#ldap). +1. [Follow the "Using a Geo Server" guide](using_a_geo_server.md). + +### Using GitLab installed from source (Deprecated) + +NOTE: **Note:** +In GitLab 11.5, support for using Geo in GitLab source installations was deprecated and will be removed in a future release. Please consider [migrating to GitLab Omnibus install](https://docs.gitlab.com/omnibus/update/convert_to_omnibus.html). + +If you installed GitLab from source: + +1. [Install GitLab Enterprise Edition](../../../install/installation.md) on the server that will serve as the **secondary** node. Do not create an account or log in to the new **secondary** node. +1. [Upload the GitLab License](../../../user/admin_area/license.md) on the **primary** node to unlock Geo. The license must be for [GitLab Premium](https://about.gitlab.com/pricing/) or higher. +1. [Set up the database replication](database_source.md) (`primary (read-write) <-> secondary (read-only)` topology). +1. [Configure fast lookup of authorized SSH keys in the database](../../operations/fast_ssh_key_lookup.md). Do this step for **both** **primary** and **secondary** nodes. +1. [Configure GitLab](configuration_source.md) to set the **primary** and **secondary** nodes. +1. [Follow the "Using a Geo Server" guide](using_a_geo_server.md). + +## Post-installation documentation + +After installing GitLab on the **secondary** nodes and performing the initial configuration, see the following documentation for post-installation information. + +### Configuring Geo + +For information on configuring Geo, see: + +- [Geo configuration (GitLab Omnibus)](configuration.md). +- [Geo configuration (source)](configuration_source.md). Configuring Geo in GitLab source installations was **deprecated** in GitLab 11.5. + +### Updating Geo + +For information on how to update your Geo nodes to the latest GitLab version, see [Updating the Geo nodes](updating_the_geo_nodes.md). + +### Configuring Geo high availability + +For information on configuring Geo for high availability, see [Geo High Availability](high_availability.md). + +### Configuring Geo with Object Storage + +For information on configuring Geo with object storage, see [Geo with Object storage](object_storage.md). + +### Disaster Recovery + +For information on using Geo in disaster recovery situations to mitigate data-loss and restore services, see [Disaster Recovery](../disaster_recovery/index.md). + +### Replicating the Container Registry + +For more information on how to replicate the Container Registry, see [Docker Registry for a **secondary** node](docker_registry.md). + +### Security Review + +For more information on Geo security, see [Geo security review](security_review.md). + +### Tuning Geo + +For more information on tuning Geo, see [Tuning Geo](tuning.md). + +## Remove Geo node + +For more information on removing a Geo node, see [Removing **secondary** Geo nodes](remove_geo_node.md). + +## Current limitations + +CAUTION: **Caution:** +This list of limitations only reflects the latest version of GitLab. If you are using an older version, extra limitations may be in place. + +- Pushing directly to a **secondary** node redirects (for HTTP) or proxies (for SSH) the request to the **primary** node instead of [handling it directly](https://gitlab.com/gitlab-org/gitlab-ee/issues/1381), except when using Git over HTTP with credentials embedded within the URI. For example, `https://user:password@secondary.tld`. +- The **primary** node has to be online for OAuth login to happen. Existing sessions and Git are not affected. +- The installation takes multiple manual steps that together can take about an hour depending on circumstances. We are working on improving this experience. See [gitlab-org/omnibus-gitlab#2978](https://gitlab.com/gitlab-org/omnibus-gitlab/issues/2978) for details. +- Real-time updates of issues/merge requests (for example, via long polling) doesn't work on the **secondary** node. +- [Selective synchronization](configuration.md#selective-synchronization) applies only to files and repositories. Other datasets are replicated to the **secondary** node in full, making it inappropriate for use as an access control mechanism. +- Object pools for forked project deduplication work only on the **primary** node, and are duplicated on the **secondary** node. +- [External merge request diffs](../../merge_request_diffs.md) will not be replicated if they are on-disk, and viewing merge requests will fail. However, external MR diffs in object storage **are** supported. The default configuration (in-database) does work. + +### Limitations on replication + +Only the following items are replicated to the **secondary** node: + +- All database content. For example, snippets, epics, issues, merge requests, groups, and project metadata. +- Project repositories. +- Project wiki repositories. +- User uploads. For example, attachments to issues, merge requests, epics, and avatars. +- CI job artifacts and traces. + +DANGER: **DANGER** +Data not on this list is unavailable on the **secondary** node. Failing over without manually replicating data not on this list will cause the data to be **lost**. + +### Examples of data not replicated + +Take special note that these examples of GitLab features are both: + +- Commonly used. +- **Not** replicated by Geo at present. + +Examples include: + +- [Elasticsearch integration](../../../integration/elasticsearch.md). +- [Container Registry](../../container_registry.md). [Object Storage](object_storage.md) can mitigate this. +- [GitLab Pages](../../pages/index.md). +- [Mattermost integration](https://docs.gitlab.com/omnibus/gitlab-mattermost/). + +CAUTION: **Caution:** +If you wish to use them on a **secondary** node, or to execute a failover successfully, you will need to replicate their data using some other means. + +## Frequently Asked Questions + +For answers to common questions, see the [Geo FAQ](faq.md). + +## Log files + +Since GitLab 9.5, Geo stores structured log messages in a `geo.log` file. For Omnibus installations, this file is at `/var/log/gitlab/gitlab-rails/geo.log`. + +This file contains information about when Geo attempts to sync repositories and files. Each line in the file contains a separate JSON entry that can be ingested into Elasticsearch, Splunk, etc. + +For example: + +```json +{"severity":"INFO","time":"2017-08-06T05:40:16.104Z","message":"Repository update","project_id":1,"source":"repository","resync_repository":true,"resync_wiki":true,"class":"Gitlab::Geo::LogCursor::Daemon","cursor_delay_s":0.038} +``` + +This message shows that Geo detected that a repository update was needed for project `1`. + +## Troubleshooting + +For troubleshooting steps, see [Geo Troubleshooting](troubleshooting.md). diff --git a/doc/administration/geo/replication/object_storage.md b/doc/administration/geo/replication/object_storage.md new file mode 100644 index 00000000000..adc298e2682 --- /dev/null +++ b/doc/administration/geo/replication/object_storage.md @@ -0,0 +1,43 @@ +# Geo with Object storage + +Geo can be used in combination with Object Storage (AWS S3, or +other compatible object storage). + +## Configuration + +At this time it is required that if object storage is enabled on the +**primary** node, it must also be enabled on each **secondary** node. + +**Secondary** nodes can use the same storage bucket as the **primary** node, or +they can use a replicated storage bucket. At this time GitLab does not +take care of content replication in object storage. + +For LFS, follow the documentation to +[set up LFS object storage](../../../workflow/lfs/lfs_administration.md#storing-lfs-objects-in-remote-object-storage). + +For CI job artifacts, there is similar documentation to configure +[jobs artifact object storage](../../job_artifacts.md#using-object-storage) + +For user uploads, there is similar documentation to configure [upload object storage](../../uploads.md#using-object-storage-core-only) + +You should enable and configure object storage on both **primary** and **secondary** +nodes. Migrating existing data to object storage should be performed on the +**primary** node only. **Secondary** nodes will automatically notice that the migrated +files are now in object storage. + +## Replication + +When using Amazon S3, you can use +[CRR](https://docs.aws.amazon.com/AmazonS3/latest/dev/crr.html) to +have automatic replication between the bucket used by the **primary** node and +the bucket used by **secondary** nodes. + +If you are using Google Cloud Storage, consider using +[Multi-Regional Storage](https://cloud.google.com/storage/docs/storage-classes#multi-regional). +Or you can use the [Storage Transfer Service](https://cloud.google.com/storage/transfer/), +although this only supports daily synchronization. + +For manual synchronization, or scheduled by `cron`, please have a look at: + +- [`s3cmd sync`](http://s3tools.org/s3cmd-sync) +- [`gsutil rsync`](https://cloud.google.com/storage/docs/gsutil/commands/rsync) diff --git a/doc/administration/geo/replication/remove_geo_node.md b/doc/administration/geo/replication/remove_geo_node.md new file mode 100644 index 00000000000..ba5664246b2 --- /dev/null +++ b/doc/administration/geo/replication/remove_geo_node.md @@ -0,0 +1,50 @@ +# Removing secondary Geo nodes + +**Secondary** nodes can be removed from the Geo cluster using the Geo admin page of the **primary** node. To remove a **secondary** node: + +1. Navigate to **Admin Area > Geo** (`/admin/geo/nodes`). +1. Click the **Remove** button for the **secondary** node you want to remove. +1. Confirm by clicking **Remove** when the prompt appears. + +Once removed from the Geo admin page, you must stop and uninstall the **secondary** node: + +1. On the **secondary** node, stop GitLab: + + ```bash + sudo gitlab-ctl stop + ``` +1. On the **secondary** node, uninstall GitLab: + + ```bash + # Stop gitlab and remove its supervision process + sudo gitlab-ctl uninstall + + # Debian/Ubuntu + sudo dpkg --remove gitlab-ee + + # Redhat/Centos + sudo rpm --erase gitlab-ee + ``` + +Once GitLab has been uninstalled from the **secondary** node, the replication slot must be dropped from the **primary** node's database as follows: + +1. On the **primary** node, start a PostgreSQL console session: + + ```bash + sudo gitlab-psql + ``` + + NOTE: **Note:** + Using `gitlab-rails dbconsole` will not work, because managing replication slots requires superuser permissions. + +1. Find the name of the relevant replication slot. This is the slot that is specified with `--slot-name` when running the replicate command: `gitlab-ctl replicate-geo-database`. + + ```sql + SELECT * FROM pg_replication_slots; + ``` + +1. Remove the replication slot for the **secondary** node: + + ```sql + SELECT pg_drop_replication_slot('<name_of_slot>'); + ``` diff --git a/doc/administration/geo/replication/security_review.md b/doc/administration/geo/replication/security_review.md new file mode 100644 index 00000000000..f77527ae8a7 --- /dev/null +++ b/doc/administration/geo/replication/security_review.md @@ -0,0 +1,287 @@ +# Geo security review (Q&A) + +The following security review of the Geo feature set focuses on security +aspects of the feature as they apply to customers running their own GitLab +instances. The review questions are based in part on the [application security architecture](https://www.owasp.org/index.php/Application_Security_Architecture_Cheat_Sheet) +questions from [owasp.org](https://www.owasp.org). + +## Business Model + +### What geographic areas does the application service? + +- This varies by customer. Geo allows customers to deploy to multiple areas, + and they get to choose where they are. +- Region and node selection is entirely manual. + +## Data Essentials + +### What data does the application receive, produce, and process? + +- Geo streams almost all data held by a GitLab instance between sites. This + includes full database replication, most files (user-uploaded attachments, + etc) and repository + wiki data. In a typical configuration, this will + happen across the public Internet, and be TLS-encrypted. +- PostgreSQL replication is TLS-encrypted. +- See also: [only TLSv1.2 should be supported](https://gitlab.com/gitlab-org/omnibus-gitlab/issues/2948) + +### How can the data be classified into categories according to its sensitivity? + +- GitLab’s model of sensitivity is centered around public vs. internal vs. + private projects. Geo replicates them all indiscriminately. “Selective sync” + exists for files and repositories (but not database content), which would permit + only less-sensitive projects to be replicated to a **secondary** node if desired. +- See also: [developing a data classification policy](https://gitlab.com/gitlab-com/security/issues/4). + +### What data backup and retention requirements have been defined for the application? + +- Geo is designed to provide replication of a certain subset of the application + data. It is part of the solution, rather than part of the problem. + +## End-Users + +### Who are the application's end‐users? + +- **Secondary** nodes are created in regions that are distant (in terms of + Internet latency) from the main GitLab installation (the **primary** node). They are + intended to be used by anyone who would ordinarily use the **primary** node, who finds + that the **secondary** node is closer to them (in terms of Internet latency). + +### How do the end‐users interact with the application? + +- **Secondary** nodes provide all the interfaces a **primary** node does + (notably a HTTP/HTTPS web application, and HTTP/HTTPS or SSH git repository + access), but is constrained to read-only activities. The principal use case is + envisioned to be cloning git repositories from the **secondary** node in favor of the + **primary** node, but end-users may use the GitLab web interface to view projects, + issues, merge requests, snippets, etc. + +### What security expectations do the end‐users have? + +- The replication process must be secure. It would typically be unacceptable to + transmit the entire database contents or all files and repositories across the + public Internet in plaintext, for instance. +- **Secondary** nodes must have the same access controls over its content as the + **primary** node - unauthenticated users must not be able to gain access to privileged + information on the **primary** node by querying the **secondary** node. +- Attackers must not be able to impersonate the **secondary** node to the **primary** node, and + thus gain access to privileged information. + +## Administrators + +### Who has administrative capabilities in the application? + +- Nothing Geo-specific. Any user where `admin: true` is set in the database is + considered an admin with super-user privileges. +- See also: [more granular access control](https://gitlab.com/gitlab-org/gitlab-ce/issues/32730) + (not geo-specific) +- Much of Geo’s integration (database replication, for instance) must be + configured with the application, typically by system administrators. + +### What administrative capabilities does the application offer? + +- **Secondary** nodes may be added, modified, or removed by users with + administrative access. +- The replication process may be controlled (start/stop) via the Sidekiq + administrative controls. + +## Network + +### What details regarding routing, switching, firewalling, and load‐balancing have been defined? + +- Geo requires the **primary** node and **secondary** node to be able to communicate with each + other across a TCP/IP network. In particular, the **secondary** nodes must be able to + access HTTP/HTTPS and PostgreSQL services on the **primary** node. + +### What core network devices support the application? + +- Varies from customer to customer. + +### What network performance requirements exist? + +- Maximum replication speeds between **primary** node and **secondary** node is limited by the + available bandwidth between sites. No hard requirements exist - time to complete + replication (and ability to keep up with changes on the **primary** node) is a function + of the size of the data set, tolerance for latency, and available network + capacity. + +### What private and public network links support the application? + +- Customers choose their own networks. As sites are intended to be + geographically separated, it is envisioned that replication traffic will pass + over the public Internet in a typical deployment, but this is not a requirement. + +## Systems + +### What operating systems support the application? + +- Geo imposes no additional restrictions on operating system (see the + [GitLab installation](https://about.gitlab.com/installation/) page for more + details), however we recommend using the operating systems listed in the [Geo documentation](index.md#requirements-for-running-geo). + +### What details regarding required OS components and lock‐down needs have been defined? + +- The recommended installation method (Omnibus) packages most components itself. + A from-source installation method exists. Both are documented at + <https://docs.gitlab.com/ee/administration/geo/replication/index.html> +- There are significant dependencies on the system-installed OpenSSH daemon (Geo + requires users to set up custom authentication methods) and the omnibus or + system-provided PostgreSQL daemon (it must be configured to listen on TCP, + additional users and replication slots must be added, etc). +- The process for dealing with security updates (for example, if there is a + significant vulnerability in OpenSSH or other services, and the customer + wants to patch those services on the OS) is identical to the non-Geo + situation: security updates to OpenSSH would be provided to the user via the + usual distribution channels. Geo introduces no delay there. + +## Infrastructure Monitoring + +### What network and system performance monitoring requirements have been defined? + +- None specific to Geo. + +### What mechanisms exist to detect malicious code or compromised application components? + +- None specific to Geo. + +### What network and system security monitoring requirements have been defined? + +- None specific to Geo. + +## Virtualization and Externalization + +### What aspects of the application lend themselves to virtualization? + +- All. + +## What virtualization requirements have been defined for the application? + +- Nothing Geo-specific, but everything in GitLab needs to have full + functionality in such an environment. + +### What aspects of the product may or may not be hosted via the cloud computing model? + +- GitLab is “cloud native” and this applies to Geo as much as to the rest of the + product. Deployment in clouds is a common and supported scenario. + +## If applicable, what approach(es) to cloud computing will be taken (Managed Hosting versus "Pure" Cloud, a "full machine" approach such as AWS-EC2 versus a "hosted database" approach such as AWS-RDS and Azure, etc)? + +- To be decided by our customers, according to their operational needs. + +## Environment + +### What frameworks and programming languages have been used to create the application? + +- Ruby on Rails, Ruby. + +### What process, code, or infrastructure dependencies have been defined for the application? + +- Nothing specific to Geo. + +### What databases and application servers support the application? + +- PostgreSQL >= 9.6, Redis, Sidekiq, Unicorn. + +### How will database connection strings, encryption keys, and other sensitive components be stored, accessed, and protected from unauthorized detection? + +- There are some Geo-specific values. Some are shared secrets which must be + securely transmitted from the **primary** node to the **secondary** node at setup time. Our + documentation recommends transmitting them from the **primary** node to the system + administrator via SSH, and then back out to the **secondary** node in the same manner. + In particular, this includes the PostgreSQL replication credentials and a secret + key (`db_key_base`) which is used to decrypt certain columns in the database. + The `db_key_base` secret is stored unencrypted on the filesystem, in + `/etc/gitlab/gitlab-secrets.json`, along with a number of other secrets. There is + no at-rest protection for them. + +## Data Processing + +### What data entry paths does the application support? + +- Data is entered via the web application exposed by GitLab itself. Some data is + also entered using system administration commands on the GitLab servers (e.g., + `gitlab-ctl set-primary-node`). +- **Secondary** nodes also receive inputs via PostgreSQL streaming replication from the **primary** node. + +### What data output paths does the application support? + +- **Primary** nodes output via PostgreSQL streaming replication to the **secondary** node. + Otherwise, principally via the web application exposed by GitLab itself, and via + SSH `git clone` operations initiated by the end-user. + +### How does data flow across the application's internal components? + +- **Secondary** nodes and **primary** nodes interact via HTTP/HTTPS (secured with JSON web + tokens) and via PostgreSQL streaming replication. +- Within a **primary** node or **secondary** node, the SSOT is the filesystem and the database + (including Geo tracking database on **secondary** node). The various internal components + are orchestrated to make alterations to these stores. + +### What data input validation requirements have been defined? + +- **Secondary** nodes must have a faithful replication of the **primary** node’s data. + +### What data does the application store and how? + +- Git repositories and files, tracking information related to the them, and the GitLab database contents. + +### What data is or may need to be encrypted and what key management requirements have been defined? + +- Neither **primary** nodes or **secondary** nodes encrypt Git repository or filesystem data at + rest. A subset of database columns are encrypted at rest using the `db_otp_key`. +- A static secret shared across all hosts in a GitLab deployment. +- In transit, data should be encrypted, although the application does permit + communication to proceed unencrypted. The two main transits are the **secondary** node’s + replication process for PostgreSQL, and for git repositories/files. Both should + be protected using TLS, with the keys for that managed via Omnibus per existing + configuration for end-user access to GitLab. + +### What capabilities exist to detect the leakage of sensitive data? + +- Comprehensive system logs exist, tracking every connection to GitLab and PostgreSQL. + +### What encryption requirements have been defined for data in transit - including transmission over WAN, LAN, SecureFTP, or publicly accessible protocols such as http: and https:? + +- Data must have the option to be encrypted in transit, and be secure against + both passive and active attack (e.g., MITM attacks should not be possible). + +## Access + +### What user privilege levels does the application support? + +- Geo adds one type of privilege: **secondary** nodes can access a special Geo API to + download files over HTTP/HTTPS, and to clone repositories using HTTP/HTTPS. + +### What user identification and authentication requirements have been defined? + +- **Secondary** nodes identify to Geo **primary** nodes via OAuth or JWT authentication + based on the shared database (HTTP access) or a PostgreSQL replication user (for + database replication). The database replication also requires IP-based access + controls to be defined. + +### What user authorization requirements have been defined? + +- **Secondary** nodes must only be able to *read* data. They are not currently able to mutate data on the **primary** node. + +### What session management requirements have been defined? + +- Geo JWTs are defined to last for only two minutes before needing to be regenerated. +- Geo JWTs are generated for one of the following specific scopes: + - Geo API access. + - Git access. + - LFS and File ID. + - Upload and File ID. + - Job Artifact and File ID. +- Geo JWTs scopes are not enforced for Git Access yet, but will be in a future version (currently scheduled for GitLab 11.10). + +### What access requirements have been defined for URI and Service calls? + +- **Secondary** nodes make many calls to the **primary** node's API. This is how file + replication proceeds, for instance. This endpoint is only accessible with a JWT token. +- The **primary** node also makes calls to the **secondary** node to get status information. + +## Application Monitoring + +### What application auditing requirements have been defined? How are audit and debug logs accessed, stored, and secured? + +- Structured JSON log is written to the filesystem, and can also be ingested + into a Kibana installation for further analysis. diff --git a/doc/administration/geo/replication/troubleshooting.md b/doc/administration/geo/replication/troubleshooting.md new file mode 100644 index 00000000000..6fea03cc8ec --- /dev/null +++ b/doc/administration/geo/replication/troubleshooting.md @@ -0,0 +1,384 @@ +# Geo Troubleshooting + +NOTE: **Note:** +This list is an attempt to document all the moving parts that can go wrong. +We are working into getting all this steps verified automatically in a +rake task in the future. + +Setting up Geo requires careful attention to details and sometimes it's easy to +miss a step. Here is a list of questions you should ask to try to detect +what you need to fix (all commands and path locations are for Omnibus installs): + +## First check the health of the **secondary** node + +Visit the **primary** node's **Admin Area > Geo** (`/admin/geo/nodes`) in +your browser. We perform the following health checks on each **secondary** node +to help identify if something is wrong: + +- Is the node running? +- Is the node's secondary database configured for streaming replication? +- Is the node's secondary tracking database configured? +- Is the node's secondary tracking database connected? +- Is the node's secondary tracking database up-to-date? + + + +There is also an option to check the status of the **secondary** node by running a special rake task: + +```sh +sudo gitlab-rake geo:status +``` + +## Is Postgres replication working? + +### Are my nodes pointing to the correct database instance? + +You should make sure your **primary** Geo node points to the instance with +writing permissions. + +Any **secondary** nodes should point only to read-only instances. + +### Can Geo detect my current node correctly? + +Geo uses the defined node from the **Admin Area > Geo** screen, and tries to match +it with the value defined in the `/etc/gitlab/gitlab.rb` configuration file. +The relevant line looks like: `external_url "http://gitlab.example.com"`. + +To check if the node on the current machine is correctly detected type: + +```sh +sudo gitlab-rails runner "puts Gitlab::Geo.current_node.inspect" +``` + +and expect something like: + +``` +#<GeoNode id: 2, schema: "https", host: "gitlab.example.com", port: 443, relative_url_root: "", primary: false, ...> +``` + +By running the command above, `primary` should be `true` when executed in +the **primary** node, and `false` on any **secondary** node. + +## How do I fix the message, "ERROR: replication slots can only be used if max_replication_slots > 0"? + +This means that the `max_replication_slots` PostgreSQL variable needs to +be set on the **primary** database. In GitLab 9.4, we have made this setting +default to 1. You may need to increase this value if you have more +**secondary** nodes. Be sure to restart PostgreSQL for this to take +effect. See the [PostgreSQL replication +setup][database-pg-replication] guide for more details. + +## How do I fix the message, "FATAL: could not start WAL streaming: ERROR: replication slot "geo_secondary_my_domain_com" does not exist"? + +This occurs when PostgreSQL does not have a replication slot for the +**secondary** node by that name. You may want to rerun the [replication +process](database.md) on the **secondary** node . + +## How do I fix the message, "Command exceeded allowed execution time" when setting up replication? + +This may happen while [initiating the replication process][database-start-replication] on the **secondary** node, +and indicates that your initial dataset is too large to be replicated in the default timeout (30 minutes). + +Re-run `gitlab-ctl replicate-geo-database`, but include a larger value for +`--backup-timeout`: + +```sh +sudo gitlab-ctl replicate-geo-database --host=primary.geo.example.com --slot-name=secondary_geo_example_com --backup-timeout=21600 +``` + +This will give the initial replication up to six hours to complete, rather than +the default thirty minutes. Adjust as required for your installation. + +## How do I fix the message, "PANIC: could not write to file 'pg_xlog/xlogtemp.123': No space left on device" + +Determine if you have any unused replication slots in the **primary** database. This can cause large amounts of +log data to build up in `pg_xlog`. Removing the unused slots can reduce the amount of space used in the `pg_xlog`. + +1. Start a PostgreSQL console session: + + ```sh + sudo gitlab-psql gitlabhq_production + ``` + + > Note that using `gitlab-rails dbconsole` will not work, because managing replication slots requires superuser permissions. + +1. View your replication slots with: + + ```sql + SELECT * FROM pg_replication_slots; + ``` + +Slots where `active` is `f` are not active. + +- When this slot should be active, because you have a **secondary** node configured using that slot, + log in to that **secondary** node and check the PostgreSQL logs why the replication is not running. + +- If you are no longer using the slot (e.g. you no longer have Geo enabled), you can remove it with in the + PostgreSQL console session: + + ```sql + SELECT pg_drop_replication_slot('name_of_extra_slot'); + ``` + +## Very large repositories never successfully synchronize on the **secondary** node + +GitLab places a timeout on all repository clones, including project imports +and Geo synchronization operations. If a fresh `git clone` of a repository +on the primary takes more than a few minutes, you may be affected by this. +To increase the timeout, add the following line to `/etc/gitlab/gitlab.rb` +on the **secondary** node: + +```ruby +gitlab_rails['gitlab_shell_git_timeout'] = 10800 +``` + +Then reconfigure GitLab: + +```sh +sudo gitlab-ctl reconfigure +``` + +This will increase the timeout to three hours (10800 seconds). Choose a time +long enough to accommodate a full clone of your largest repositories. + +## How to reset Geo **secondary** node replication + +If you get a **secondary** node in a broken state and want to reset the replication state, +to start again from scratch, there are a few steps that can help you: + +1. Stop Sidekiq and the Geo LogCursor + + It's possible to make Sidekiq stop gracefully, but making it stop getting new jobs and + wait until the current jobs to finish processing. + + You need to send a **SIGTSTP** kill signal for the first phase and them a **SIGTERM** + when all jobs have finished. Otherwise just use the `gitlab-ctl stop` commands. + + ```sh + gitlab-ctl status sidekiq + # run: sidekiq: (pid 10180) <- this is the PID you will use + kill -TSTP 10180 # change to the correct PID + + gitlab-ctl stop sidekiq + gitlab-ctl stop geo-logcursor + ``` + + You can watch sidekiq logs to know when sidekiq jobs processing have finished: + + ```sh + gitlab-ctl tail sidekiq + ``` + +1. Rename repository storage folders and create new ones + + ```sh + mv /var/opt/gitlab/git-data/repositories /var/opt/gitlab/git-data/repositories.old + mkdir -p /var/opt/gitlab/git-data/repositories + chown git:git /var/opt/gitlab/git-data/repositories + ``` + + TIP: **Tip** + You may want to remove the `/var/opt/gitlab/git-data/repositories.old` in the future + as soon as you confirmed that you don't need it anymore, to save disk space. + +1. _(Optional)_ Rename other data folders and create new ones + + CAUTION: **Caution**: + You may still have files on the **secondary** node that have been removed from **primary** node but + removal have not been reflected. If you skip this step, they will never be removed + from this Geo node. + + Any uploaded content like file attachments, avatars or LFS objects are stored in a + subfolder in one of the two paths below: + + 1. /var/opt/gitlab/gitlab-rails/shared + 1. /var/opt/gitlab/gitlab-rails/uploads + + To rename all of them: + + ```sh + gitlab-ctl stop + + mv /var/opt/gitlab/gitlab-rails/shared /var/opt/gitlab/gitlab-rails/shared.old + mkdir -p /var/opt/gitlab/gitlab-rails/shared + + mv /var/opt/gitlab/gitlab-rails/uploads /var/opt/gitlab/gitlab-rails/uploads.old + mkdir -p /var/opt/gitlab/gitlab-rails/uploads + ``` + + Reconfigure in order to recreate the folders and make sure permissions and ownership + are correctly + + ```sh + gitlab-ctl reconfigure + ``` + +1. Reset the Tracking Database + + ```sh + gitlab-rake geo:db:reset + ``` + +1. Restart previously stopped services + + ```sh + gitlab-ctl start + ``` + +## How do I fix a "Foreign Data Wrapper (FDW) is not configured" error? + +When setting up Geo, you might see this warning in the `gitlab-rake +gitlab:geo:check` output: + +``` +GitLab Geo tracking database Foreign Data Wrapper schema is up-to-date? ... foreign data wrapper is not configured +``` + +There are a few key points to remember: + +1. The FDW settings are configured on the Geo **tracking** database. +1. The configured foreign server enables a login to the Geo +**secondary**, read-only database. + +By default, the Geo secondary and tracking database are running on the +same host on different ports. That is, 5432 and 5431 respectively. + +### Checking configuration + +NOTE: **Note:** +The following steps are for Omnibus installs only. Using Geo with source-based installs [is deprecated](index.md#using-gitlab-installed-from-source-deprecated). + +To check the configuration: + +1. Enter the database console: + + ```sh + gitlab-geo-psql + ``` + +1. Check whether any tables are present. If everything is working, you +should see something like this: + + ```sql + gitlabhq_geo_production=# SELECT * from information_schema.foreign_tables; + foreign_table_catalog | foreign_table_schema | foreign_table_name | foreign_server_catalog | foreign_server_n + ame + -------------------------+----------------------+-------------------------------------------------+-------------------------+----------------- + ---- + gitlabhq_geo_production | gitlab_secondary | abuse_reports | gitlabhq_geo_production | gitlab_secondary + gitlabhq_geo_production | gitlab_secondary | appearances | gitlabhq_geo_production | gitlab_secondary + gitlabhq_geo_production | gitlab_secondary | application_setting_terms | gitlabhq_geo_production | gitlab_secondary + gitlabhq_geo_production | gitlab_secondary | application_settings | gitlabhq_geo_production | gitlab_secondary + <snip> + ``` + + However, if the query returns with `0 rows`, then continue onto the next steps. + +1. Check that the foreign server mapping is correct via `\des+`. The + results should look something like this: + + ```sql + gitlabhq_geo_production=# \des+ + List of foreign servers + -[ RECORD 1 ]--------+------------------------------------------------------------ + Name | gitlab_secondary + Owner | gitlab-psql + Foreign-data wrapper | postgres_fdw + Access privileges | "gitlab-psql"=U/"gitlab-psql" + + | gitlab_geo=U/"gitlab-psql" + Type | + Version | + FDW Options | (host '0.0.0.0', port '5432', dbname 'gitlabhq_production') + Description | + ``` + + NOTE: **Note:** Pay particular attention to the host and port under + FDW options. That configuration should point to the Geo secondary + database. + + If you need to experiment with changing the host or password, the + following queries demonstrate how: + + ```sql + ALTER SERVER gitlab_secondary OPTIONS (SET host 'my-new-host'); + ALTER SERVER gitlab_secondary OPTIONS (SET port 5432); + ``` + + If you change the host and/or port, you will also have to adjust the + following settings in `/etc/gitlab/gitlab.rb` and run `gitlab-ctl + reconfigure`: + + - `gitlab_rails['db_host']` + - `gitlab_rails['db_port']` + +1. Check that the user mapping is configured properly via `\deu+`: + + ```sql + gitlabhq_geo_production=# \deu+ + List of user mappings + Server | User name | FDW Options + ------------------+------------+-------------------------------------------------------------------------------- + gitlab_secondary | gitlab_geo | ("user" 'gitlab', password 'YOUR-PASSWORD-HERE') + (1 row) + ``` + + Make sure the password is correct. You can test that logins work by running `psql`: + + ```sh + # Connect to the tracking database as the `gitlab_geo` user + sudo -u git /opt/gitlab/embedded/bin/psql -h /var/opt/gitlab/geo-postgresql -p 5431 -U gitlab_geo -W -d gitlabhq_geo_production + ``` + + If you need to correct the password, the following query shows how: + + ```sql + ALTER USER MAPPING FOR gitlab_geo SERVER gitlab_secondary OPTIONS (SET password 'my-new-password'); + ``` + + If you change the user or password, you will also have to adjust the + following settings in `/etc/gitlab/gitlab.rb` and run `gitlab-ctl + reconfigure`: + + - `gitlab_rails['db_username']` + - `gitlab_rails['db_password']` + + If you are using [PgBouncer in front of the secondary + database](database.md#pgbouncer-support-optional), be sure to update + the following settings: + + - `geo_postgresql['fdw_external_user']` + - `geo_postgresql['fdw_external_password']` + +### Manual reload of FDW schema + +If you're still unable to get FDW working, you may want to try a manual +reload of the FDW schema. To manually reload the FDW schema: + +1. On the node running the Geo tracking database, enter the PostgreSQL console via + the `gitlab_geo` user: + + ```sh + sudo -u git /opt/gitlab/embedded/bin/psql -h /var/opt/gitlab/geo-postgresql -p 5431 -U gitlab_geo -W -d gitlabhq_geo_production + ``` + + Be sure to adjust the port and hostname for your configuration. You + may be asked to enter a password. + +1. Reload the schema via: + + ```sql + DROP SCHEMA IF EXISTS gitlab_secondary CASCADE; + CREATE SCHEMA gitlab_secondary; + GRANT USAGE ON FOREIGN SERVER gitlab_secondary TO gitlab_geo; + IMPORT FOREIGN SCHEMA public FROM SERVER gitlab_secondary INTO gitlab_secondary; + ``` + +1. Test that queries work: + + ```sql + SELECT * from information_schema.foreign_tables; + SELECT * FROM gitlab_secondary.projects limit 1; + ``` + +[database-start-replication]: database.md#step-3-initiate-the-replication-process +[database-pg-replication]: database.md#postgresql-replication diff --git a/doc/administration/geo/replication/tuning.md b/doc/administration/geo/replication/tuning.md new file mode 100644 index 00000000000..246e56ba43b --- /dev/null +++ b/doc/administration/geo/replication/tuning.md @@ -0,0 +1,19 @@ +# Tuning Geo + +## Changing the sync capacity values + +In the Geo admin page (`/admin/geo/nodes`), there are several variables that +can be tuned to improve performance of Geo: + +- Repository sync capacity. +- File sync capacity. + +Increasing these values will increase the number of jobs that are scheduled. +However, this may not lead to more downloads in parallel unless the number of +available Sidekiq threads is also increased. For example, if repository sync +capacity is increased from 25 to 50, you may also want to increase the number +of Sidekiq threads from 25 to 50. See the [Sidekiq concurrency +documentation][sidekiq-concurrency] +for more details. + +[sidekiq-concurrency]: ../../operations/extra_sidekiq_processes.html#concurrency diff --git a/doc/administration/geo/replication/updating_the_geo_nodes.md b/doc/administration/geo/replication/updating_the_geo_nodes.md new file mode 100644 index 00000000000..a6af8716228 --- /dev/null +++ b/doc/administration/geo/replication/updating_the_geo_nodes.md @@ -0,0 +1,404 @@ +# Updating the Geo nodes + +Depending on which version of Geo you are updating to/from, there may be +different steps. + +## General update steps + +In order to update the Geo nodes when a new GitLab version is released, +all you need to do is update GitLab itself: + +1. Log into each node (**primary** and **secondary** nodes). +1. [Update GitLab][update]. +1. [Update tracking database on **secondary** node](#update-tracking-database-on-secondary-node) when + the tracking database is enabled. +1. [Test](#check-status-after-updating) **primary** and **secondary** nodes, and check version in each. + +## Upgrading to GitLab 10.8 + +Before 10.8, broadcast messages would not propagate without flushing the cache on the **secondary** nodes. This has been fixed in 10.8, but requires one last cache flush on each **secondary** node: + +```sh +sudo gitlab-rake cache:clear +``` + +## Upgrading to GitLab 10.6 + +In 10.4, we started to recommend that you define a password for database user (`gitlab`). + +We now require this change as we use this password to enable the Foreign Data Wrapper, as a way to optimize +the Geo Tracking Database. We are also improving security by disabling the use of **trust** +authentication method. + +1. **[primary]** Login to your **primary** node and run: + + ```sh + gitlab-ctl pg-password-md5 gitlab + # Enter password: mypassword + # Confirm password: mypassword + # fca0b89a972d69f00eb3ec98a5838484 + ``` + + Copy the generated hash and edit `/etc/gitlab/gitlab.rb`: + + ```ruby + # Fill with the hash generated by `gitlab-ctl pg-password-md5 gitlab` + postgresql['sql_user_password'] = 'fca0b89a972d69f00eb3ec98a5838484' + + # Every node that runs Unicorn or Sidekiq needs to have the database + # password specified as below. If you have a high-availability setup, this + # must be present in all application nodes. + gitlab_rails['db_password'] = 'mypassword' + ``` + + Still in the configuration file, locate and remove the `trust_auth_cidr_address`: + + ```ruby + postgresql['trust_auth_cidr_addresses'] = ['127.0.0.1/32','1.2.3.4/32'] # <- Remove this + ``` + +1. **[primary]** Reconfigure and restart: + + ```sh + sudo gitlab-ctl reconfigure + sudo gitlab-ctl restart + ``` + +1. **[secondary]** Login to all **secondary** nodes and edit `/etc/gitlab/gitlab.rb`: + + ```ruby + # Fill with the hash generated by `gitlab-ctl pg-password-md5 gitlab` + postgresql['sql_user_password'] = 'fca0b89a972d69f00eb3ec98a5838484' + + # Every node that runs Unicorn or Sidekiq needs to have the database + # password specified as below. If you have a high-availability setup, this + # must be present in all application nodes. + gitlab_rails['db_password'] = 'mypassword' + + # Enable Foreign Data Wrapper + geo_secondary['db_fdw'] = true + + # Secondary address + # - replace '5.6.7.8' with the secondary public or VPC address + postgresql['md5_auth_cidr_addresses'] = ['5.6.7.8/32'] + ``` + + Still in the configuration file, locate and remove the `trust_auth_cidr_address`: + + ```ruby + postgresql['trust_auth_cidr_addresses'] = ['127.0.0.1/32','5.6.7.8/32'] # <- Remove this + ``` + +1. **[secondary]** Reconfigure and restart: + + ```sh + sudo gitlab-ctl reconfigure + sudo gitlab-ctl restart + ``` + +## Upgrading to GitLab 10.5 + +For Geo Disaster Recovery to work with minimum downtime, your **secondary** node +should use the same set of secrets as the **primary** node. However, setup instructions +prior to the 10.5 release only synchronized the `db_key_base` secret. + +To rectify this error on existing installations, you should **overwrite** the +contents of `/etc/gitlab/gitlab-secrets.json` on each **secondary** node with the +contents of `/etc/gitlab/gitlab-secrets.json` on the **primary** node, then run the +following command on each **secondary** node: + +```sh +sudo gitlab-ctl reconfigure +``` + +If you do not perform this step, you may find that two-factor authentication +[is broken following DR](../disaster_recovery/index.html#i-followed-the-disaster-recovery-instructions-and-now-two-factor-auth-is-broken). + +To prevent SSH requests to the newly promoted **primary** node from failing +due to SSH host key mismatch when updating the **primary** node domain's DNS record +you should perform the step to [Manually replicate **primary** SSH host keys](configuration.md#step-2-manually-replicate-the-primary-nodes-ssh-host-keys) in each +**secondary** node. + +## Upgrading to GitLab 10.4 + +There are no Geo-specific steps to take! + +## Upgrading to GitLab 10.3 + +### Support for SSH repository synchronization removed + +In GitLab 10.2, synchronizing secondaries over SSH was deprecated. In 10.3, +support is removed entirely. All installations will switch to the HTTP/HTTPS +cloning method instead. Before upgrading, ensure that all your Geo nodes are +configured to use this method and that it works for your installation. In +particular, ensure that [Git access over HTTP/HTTPS is enabled](configuration.md#step-6-enable-git-access-over-httphttps). + +Synchronizing repositories over the public Internet using HTTP is insecure, so +you should ensure that you have HTTPS configured before upgrading. Note that +file synchronization is **also** insecure in these cases! + +## Upgrading to GitLab 10.2 + +### Secure PostgreSQL replication + +Support for TLS-secured PostgreSQL replication has been added. If you are +currently using PostgreSQL replication across the open internet without an +external means of securing the connection (e.g., a site-to-site VPN), then you +should immediately reconfigure your **primary** and **secondary** PostgreSQL instances +according to the [updated instructions][database]. + +If you *are* securing the connections externally and wish to continue doing so, +ensure you include the new option `--sslmode=prefer` in future invocations of +`gitlab-ctl replicate-geo-database`. + +### HTTPS repository sync + +Support for replicating repositories and wikis over HTTP/HTTPS has been added. +Replicating over SSH has been deprecated, and support for this option will be +removed in a future release. + +To switch to HTTP/HTTPS replication, log into the **primary** node as an admin and visit +**Admin Area > Geo** (`/admin/geo/nodes`). For each **secondary** node listed, +press the "Edit" button, change the "Repository cloning" setting from +"SSH (deprecated)" to "HTTP/HTTPS", and press "Save changes". This should take +effect immediately. + +Any new secondaries should be created using HTTP/HTTPS replication - this is the +default setting. + +After you've verified that HTTP/HTTPS replication is working, you should remove +the now-unused SSH keys from your secondaries, as they may cause problems if the +**secondary** node if ever promoted to a **primary** node: + +1. **[secondary]** Login to **all** your **secondary** nodes and run: + + ```ruby + sudo -u git -H rm ~git/.ssh/id_rsa ~git/.ssh/id_rsa.pub + ``` + +### Hashed Storage + +CAUTION: **Warning:** +Hashed storage is in **Alpha**. It is considered experimental and not +production-ready. See [Hashed Storage] for more detail. + +If you previously enabled Hashed Storage and migrated all your existing +projects to Hashed Storage, disabling hashed storage will not migrate projects +to their previous project based storage path. As such, once enabled and +migrated we recommend leaving Hashed Storage enabled. + +## Upgrading to GitLab 10.1 + +CAUTION: **Warning:** +Hashed storage is in **Alpha**. It is considered experimental and not +production-ready. See [Hashed Storage] for more detail. + +[Hashed storage] was introduced in GitLab 10.0, and a [migration path][hashed-migration] +for existing repositories was added in GitLab 10.1. + +## Upgrading to GitLab 10.0 + +Since GitLab 10.0, we require all **Geo** systems to [use SSH key lookups via +the database][ssh-fast-lookup] to avoid having to maintain consistency of the +`authorized_keys` file for SSH access. Failing to do this will prevent users +from being able to clone via SSH. + +Note that in older versions of Geo, attachments downloaded on the **secondary** +nodes would be saved to the wrong directory. We recommend that you do the +following to clean this up. + +On the **secondary** Geo nodes, run as root: + +```sh +mv /var/opt/gitlab/gitlab-rails/working /var/opt/gitlab/gitlab-rails/working.old +mkdir /var/opt/gitlab/gitlab-rails/working +chmod 700 /var/opt/gitlab/gitlab-rails/working +chown git:git /var/opt/gitlab/gitlab-rails/working +``` + +You may delete `/var/opt/gitlab/gitlab-rails/working.old` any time. + +Once this is done, we advise restarting GitLab on the **secondary** nodes for the +new working directory to be used: + +```sh +sudo gitlab-ctl restart +``` + +## Upgrading from GitLab 9.3 or older + +If you started running Geo on GitLab 9.3 or older, we recommend that you +resync your **secondary** PostgreSQL databases to use replication slots. If you +started using Geo with GitLab 9.4 or 10.x, no further action should be +required because replication slots are used by default. However, if you +started with GitLab 9.3 and upgraded later, you should still follow the +instructions below. + +When in doubt, it does not hurt to do a resync. The easiest way to do this in +Omnibus is the following: + + 1. Make sure you have Omnibus GitLab on the **primary** server. + 1. Run `gitlab-ctl reconfigure` and `gitlab-ctl restart postgresql`. This will enable replication slots on the **primary** database. + 1. Check the steps about defining `postgresql['sql_user_password']`, `gitlab_rails['db_password']`. + 1. Make sure `postgresql['max_replication_slots']` matches the number of **secondary** Geo nodes locations. + 1. Install GitLab on the **secondary** server. + 1. Re-run the [database replication process][database-replication]. + +## Special update notes for 9.0.x + +> **IMPORTANT**: +With GitLab 9.0, the PostgreSQL version is upgraded to 9.6 and manual steps are +required in order to update the **secondary** nodes and keep the Streaming +Replication working. Downtime is required, so plan ahead. + +The following steps apply only if you upgrade from a 8.17 GitLab version to +9.0+. For previous versions, update to GitLab 8.17 first before attempting to +upgrade to 9.0+. + +--- + +Make sure to follow the steps in the exact order as they appear below and pay +extra attention in what node (either **primary** or **secondary**) you execute them! Each step +is prepended with the relevant node for better clarity: + +1. **[secondary]** Login to **all** your **secondary** nodes and stop all services: + + ```ruby + sudo gitlab-ctl stop + ``` + +1. **[secondary]** Make a backup of the `recovery.conf` file on **all** + **secondary** nodes to preserve PostgreSQL's credentials: + + ```sh + sudo cp /var/opt/gitlab/postgresql/data/recovery.conf /var/opt/gitlab/ + ``` + +1. **[primary]** Update the **primary** node to GitLab 9.0 following the + [regular update docs][update]. At the end of the update, the **primary** node + will be running with PostgreSQL 9.6. + +1. **[primary]** To prevent a de-synchronization of the repository replication, + stop all services except `postgresql` as we will use it to re-initialize the + **secondary** node's database: + + ```sh + sudo gitlab-ctl stop + sudo gitlab-ctl start postgresql + ``` + +1. **[secondary]** Run the following steps on each of the **secondary** nodes: + + 1. **[secondary]** Stop all services: + + ```sh + sudo gitlab-ctl stop + ``` + + 1. **[secondary]** Prevent running database migrations: + + ```sh + sudo touch /etc/gitlab/skip-auto-migrations + ``` + + 1. **[secondary]** Move the old database to another directory: + + ```sh + sudo mv /var/opt/gitlab/postgresql{,.bak} + ``` + + 1. **[secondary]** Update to GitLab 9.0 following the [regular update docs][update]. + At the end of the update, the node will be running with PostgreSQL 9.6. + + 1. **[secondary]** Make sure all services are up: + + ```sh + sudo gitlab-ctl start + ``` + + 1. **[secondary]** Reconfigure GitLab: + + ```sh + sudo gitlab-ctl reconfigure + ``` + + 1. **[secondary]** Run the PostgreSQL upgrade command: + + ```sh + sudo gitlab-ctl pg-upgrade + ``` + + 1. **[secondary]** See the stored credentials for the database that you will + need to re-initialize the replication: + + ```sh + sudo grep -s primary_conninfo /var/opt/gitlab/recovery.conf + ``` + + 1. **[secondary]** Create the `replica.sh` script as described in the + [database configuration document][database-source-replication]. + + 1. **[secondary]** Run the recovery script using the credentials from the + previous step: + + ```sh + sudo bash /tmp/replica.sh + ``` + + 1. **[secondary]** Reconfigure GitLab: + + ```sh + sudo gitlab-ctl reconfigure + ``` + + 1. **[secondary]** Start all services: + + ```sh + sudo gitlab-ctl start + ``` + + 1. **[secondary]** Repeat the steps for the remaining **secondary** nodes. + +1. **[primary]** After all **secondary** nodes are updated, start all services in + **primary** node: + + ```sh + sudo gitlab-ctl start + ``` + +## Check status after updating + +Now that the update process is complete, you may want to check whether +everything is working correctly: + +1. Run the Geo raketask on all nodes, everything should be green: + + ```sh + sudo gitlab-rake gitlab:geo:check + ``` + +1. Check the **primary** node's Geo dashboard for any errors. +1. Test the data replication by pushing code to the **primary** node and see if it + is received by **secondary** nodes. + +## Update tracking database on **secondary** node + +After updating a **secondary** node, you might need to run migrations on +the tracking database. The tracking database was added in GitLab 9.1, +and it is required since 10.0. + +1. Run database migrations on tracking database: + + ```sh + sudo gitlab-rake geo:db:migrate + ``` + +1. Repeat this step for each **secondary** node. + +[update]: ../../../update/README.md +[database]: database.md +[database-replication]: database.md#step-3-initiate-the-replication-process +[database-source-replication]: database_source.md#step-3-initiate-the-replication-process +[Hashed Storage]: ../../repository_storage_types.md +[hashed-migration]: ../../raketasks/storage.md +[ssh-fast-lookup]: ../../operations/fast_ssh_key_lookup.md diff --git a/doc/administration/geo/replication/using_a_geo_server.md b/doc/administration/geo/replication/using_a_geo_server.md new file mode 100644 index 00000000000..c8bfcf01fa4 --- /dev/null +++ b/doc/administration/geo/replication/using_a_geo_server.md @@ -0,0 +1,18 @@ +[//]: # (Please update EE::GitLab::GeoGitAccess::GEO_SERVER_DOCS_URL if this file is moved) + +# Using a Geo Server + +After you set up the [database replication and configure the Geo nodes][req], use your closest GitLab node as you would a normal standalone GitLab instance. + +Pushing directly to a **secondary** node (for both HTTP, SSH including git-lfs) was [introduced](https://about.gitlab.com/2018/09/22/gitlab-11-3-released/) in [GitLab Premium](https://about.gitlab.com/pricing/#self-managed) 11.3. + +Example of the output you will see when pushing to a **secondary** node: + +```bash +$ git push +> GitLab: You're pushing to a Geo secondary. +> GitLab: We'll help you by proxying this request to the primary: ssh://git@primary.geo/user/repo.git +Everything up-to-date +``` + +[req]: index.md#setup-instructions diff --git a/doc/administration/high_availability/README.md b/doc/administration/high_availability/README.md index 49fe80fb2a6..7736c4fe6f3 100644 --- a/doc/administration/high_availability/README.md +++ b/doc/administration/high_availability/README.md @@ -1,4 +1,4 @@ -# High Availability +# Scaling and High Availability GitLab supports several different types of clustering and high-availability. The solution you choose will be based on the level of scalability and @@ -13,51 +13,173 @@ of Git, developers can still commit code locally even when GitLab is not available. However, some GitLab features such as the issue tracker and Continuous Integration are not available when GitLab is down. -**Keep in mind that all Highly Available solutions come with a trade-off between +**Keep in mind that all highly-available solutions come with a trade-off between cost/complexity and uptime**. The more uptime you want, the more complex the solution. And the more complex the solution, the more work is involved in setting up and maintaining it. High availability is not free and every HA solution should balance the costs against the benefits. -## Architecture - -There are two kinds of setups: - -- active/active -- active/passive - -### Active/Active - -This architecture scales easily because all application servers handle -user requests simultaneously. The database, Redis, and GitLab application are -all deployed on separate servers. The configuration is **only** highly-available -if the database, Redis and storage are also configured as such. - -Follow the steps below to configure an active/active setup: +There are many options when choosing a highly-available GitLab architecture. We +recommend engaging with GitLab Support to choose the best architecture for your +use-case. This page contains some various options and guidelines based on +experience with GitLab.com and Enterprise Edition on-premises customers. + +For a detailed insight into how GitLab scales and configures GitLab.com, you can +watch [this 1 hour Q&A](https://www.youtube.com/watch?v=uCU8jdYzpac) +with [John Northrup](https://gitlab.com/northrup), one of our infrastructure +engineers, and live questions coming in from some of our customers. + +## GitLab Components + +The following components need to be considered for a scaled or highly-available +environment. In many cases components can be combined on the same nodes to reduce +complexity. + +- Unicorn/Workhorse - Web-requests (UI, API, Git over HTTP) +- Sidekiq - Asynchronous/Background jobs +- PostgreSQL - Database + - Consul - Database service discovery and health checks/failover + - PGBouncer - Database pool manager +- Redis - Key/Value store (User sessions, cache, queue for Sidekiq) + - Sentinel - Redis health check/failover manager +- Gitaly - Provides high-level RPC access to Git repositories + +## Scalable Architecture Examples + +When an organization reaches a certain threshold it will be necessary to scale +the GitLab instance. Still, true high availability may not be necessary. There +are options for scaling GitLab instances relatively easily without incurring the +infrastructure and maintenance costs of full high availability. + +### Basic Scaling + +This is the simplest form of scaling and will work for the majority of +cases. Backend components such as PostgreSQL, Redis and storage are offloaded +to their own nodes while the remaining GitLab components all run on 2 or more +application nodes. + +This form of scaling also works well in a cloud environment when it is more +cost-effective to deploy several small nodes rather than a single +larger one. + +- 1 PostgreSQL node +- 1 Redis node +- 2 or more GitLab application nodes (Unicorn, Workhorse, Sidekiq) +- 1 NFS/Gitaly storage server + +#### Installation Instructions + +Complete the following installation steps in order. A link at the end of each +section will bring you back to the Scalable Architecture Examples section so +you can continue with the next step. + +1. [PostgreSQL](./database.md#postgresql-in-a-scaled-environment) +1. [Redis](./redis.md#redis-in-a-scaled-environment) +1. [Gitaly](./gitaly.md) (recommended) or [NFS](./nfs.md) +1. [GitLab application nodes](./gitlab.md) + +### Full Scaling + +For very large installations it may be necessary to further split components +for maximum scalability. In a fully-scaled architecture the application node +is split into separate Sidekiq and Unicorn/Workhorse nodes. One indication that +this architecture is required is if Sidekiq queues begin to periodically increase +in size, indicating that there is contention or not enough resources. + +- 1 PostgreSQL node +- 1 Redis node +- 2 or more GitLab application nodes (Unicorn, Workhorse) +- 2 or more Sidekiq nodes +- 2 or more NFS/Gitaly storage servers + +## High Availability Architecture Examples + +When organizations require scaling *and* high availability the following +architectures can be utilized. As the introduction section at the top of this +page mentions, there is a tradeoff between cost/complexity and uptime. Be sure +this complexity is absolutely required before taking the step into full +high availability. + +For all examples below, we recommend running Consul and Redis Sentinel on +dedicated nodes. If Consul is running on PostgreSQL nodes or Sentinel on +Redis nodes there is a potential that high resource usage by PostgreSQL or +Redis could prevent communication between the other Consul and Sentinel nodes. +This may lead to the other nodes believing a failure has occurred and automated +failover is necessary. Isolating them from the services they monitor reduces +the chances of split-brain. + +The examples below do not really address high availability of NFS. Some enterprises +have access to NFS appliances that manage availability. This is the best case +scenario. In the future, GitLab may offer a more user-friendly solution to +[GitLab HA Storage](https://gitlab.com/gitlab-org/omnibus-gitlab/issues/2472). + +There are many options in between each of these examples. Work with GitLab Support +to understand the best starting point for your workload and adapt from there. + +### Horizontal + +This is the simplest form of high availability and scaling. It requires the +fewest number of individual servers (virtual or physical) but does have some +trade-offs and limits. + +This architecture will work well for many GitLab customers. Larger customers +may begin to notice certain events cause contention/high load - for example, +cloning many large repositories with binary files, high API usage, a large +number of enqueued Sidekiq jobs, etc. If this happens you should consider +moving to a hybrid or fully distributed architecture depending on what is causing +the contention. + +- 3 PostgreSQL nodes +- 2 Redis nodes +- 3 Consul/Sentinel nodes +- 2 or more GitLab application nodes (Unicorn, Workhorse, Sidekiq, PGBouncer) +- 1 NFS/Gitaly server + + + +### Hybrid + +In this architecture, certain components are split on dedicated nodes so high +resource usage of one component does not interfere with others. In larger +environments this is a good architecture to consider if you foresee or do have +contention due to certain workloads. + +- 3 PostgreSQL nodes +- 2 Redis nodes +- 3 Consul/Sentinel nodes +- 2 or more Sidekiq nodes +- 2 or more Web nodes (Unicorn, Workhorse, PGBouncer) +- 1 or more NFS/Gitaly servers + + + +### Fully Distributed + +This architecture scales to hundreds of thousands of users and projects and is +the basis of the GitLab.com architecture. While this scales well it also comes +with the added complexity of many more nodes to configure, manage and monitor. + +- 3 PostgreSQL nodes +- 4 or more Redis nodes (2 separate clusters for persistent and cache data) +- 3 Consul nodes +- 3 Sentinel nodes +- Multiple dedicated Sidekiq nodes (Split into real-time, best effort, ASAP, + CI Pipeline and Pull Mirror sets) +- 2 or more Git nodes (Git over SSH/Git over HTTP) +- 2 or more API nodes (All requests to `/api`) +- 2 or more Web nodes (All other web requests) +- 2 or more NFS/Gitaly servers + + + +The following pages outline the steps necessary to configure each component +separately: 1. [Configure the database](database.md) 1. [Configure Redis](redis.md) 1. [Configure Redis for GitLab source installations](redis_source.md) 1. [Configure NFS](nfs.md) + 1. [NFS Client and Host setup](nfs_host_client_setup.md) 1. [Configure the GitLab application servers](gitlab.md) 1. [Configure the load balancers](load_balancer.md) - - -### Active/Passive - -For pure high-availability/failover with no scaling you can use an -active/passive configuration. This utilizes DRBD (Distributed Replicated -Block Device) to keep all data in sync. DRBD requires a low latency link to -remain in sync. It is not advisable to attempt to run DRBD between data centers -or in different cloud availability zones. - -> **Note:** GitLab recommends against choosing this HA method because of the - complexity of managing DRBD and crafting automatic failover. This is - *compatible* with GitLab, but not officially *supported*. If you are - an EE customer, support will help you with GitLab related problems, but if the - root cause is identified as DRBD, we will not troubleshoot further. - -Components/Servers Required: 2 servers/virtual machines (one active/one passive) - - diff --git a/doc/administration/high_availability/alpha_database.md b/doc/administration/high_availability/alpha_database.md new file mode 100644 index 00000000000..7bf20be60e6 --- /dev/null +++ b/doc/administration/high_availability/alpha_database.md @@ -0,0 +1,6 @@ +--- +redirect_to: 'database.md' +--- + +This documentation has been moved to the main +[database documentation](database.md#configure_using_omnibus_for_high_availability). diff --git a/doc/administration/high_availability/consul.md b/doc/administration/high_availability/consul.md new file mode 100644 index 00000000000..056b7fc15d9 --- /dev/null +++ b/doc/administration/high_availability/consul.md @@ -0,0 +1,105 @@ +# Working with the bundled Consul service **[PREMIUM ONLY]** + +## Overview + +As part of its High Availability stack, GitLab Premium includes a bundled version of [Consul](http://consul.io) that can be managed through `/etc/gitlab/gitlab.rb`. + +A Consul cluster consists of multiple server agents, as well as client agents that run on other nodes which need to talk to the consul cluster. + +## Operations + +### Checking cluster membership + +To see which nodes are part of the cluster, run the following on any member in the cluster +``` +# /opt/gitlab/embedded/bin/consul members +Node Address Status Type Build Protocol DC +consul-b XX.XX.X.Y:8301 alive server 0.9.0 2 gitlab_consul +consul-c XX.XX.X.Y:8301 alive server 0.9.0 2 gitlab_consul +consul-c XX.XX.X.Y:8301 alive server 0.9.0 2 gitlab_consul +db-a XX.XX.X.Y:8301 alive client 0.9.0 2 gitlab_consul +db-b XX.XX.X.Y:8301 alive client 0.9.0 2 gitlab_consul +``` + +Ideally all nodes will have a `Status` of `alive`. + +### Restarting the server cluster + +**Note**: This section only applies to server agents. It is safe to restart client agents whenever needed. + +If it is necessary to restart the server cluster, it is important to do this in a controlled fashion in order to maintain quorum. If quorum is lost, you will need to follow the consul [outage recovery](#outage-recovery) process to recover the cluster. + +To be safe, we recommend you only restart one server agent at a time to ensure the cluster remains intact. + +For larger clusters, it is possible to restart multiple agents at a time. See the [Consul consensus document](https://www.consul.io/docs/internals/consensus.html#deployment-table) for how many failures it can tolerate. This will be the number of simulateneous restarts it can sustain. + +## Troubleshooting + +### Consul server agents unable to communicate + +By default, the server agents will attempt to [bind](https://www.consul.io/docs/agent/options.html#_bind) to '0.0.0.0', but they will advertise the first private IP address on the node for other agents to communicate with them. If the other nodes cannot communicate with a node on this address, then the cluster will have a failed status. + +You will see messages like the following in `gitlab-ctl tail consul` output if you are running into this issue: + +``` +2017-09-25_19:53:39.90821 2017/09/25 19:53:39 [WARN] raft: no known peers, aborting election +2017-09-25_19:53:41.74356 2017/09/25 19:53:41 [ERR] agent: failed to sync remote state: No cluster leader +``` + + +To fix this: + +1. Pick an address on each node that all of the other nodes can reach this node through. +1. Update your `/etc/gitlab/gitlab.rb` + + ```ruby + consul['configuration'] = { + ... + bind_addr: 'IP ADDRESS' + } + ``` +1. Run `gitlab-ctl reconfigure` + +If you still see the errors, you may have to [erase the consul database and reinitialize](#recreate-from-scratch) on the affected node. + +### Consul agents do not start - Multiple private IPs + +In the case that a node has multiple private IPs the agent be confused as to which of the private addresses to advertise, and then immediately exit on start. + +You will see messages like the following in `gitlab-ctl tail consul` output if you are running into this issue: + +``` +2017-11-09_17:41:45.52876 ==> Starting Consul agent... +2017-11-09_17:41:45.53057 ==> Error creating agent: Failed to get advertise address: Multiple private IPs found. Please configure one. +``` + +To fix this: + +1. Pick an address on the node that all of the other nodes can reach this node through. +1. Update your `/etc/gitlab/gitlab.rb` + + ```ruby + consul['configuration'] = { + ... + bind_addr: 'IP ADDRESS' + } + ``` +1. Run `gitlab-ctl reconfigure` + +### Outage recovery + +If you lost enough server agents in the cluster to break quorum, then the cluster is considered failed, and it will not function without manual intervenetion. + +#### Recreate from scratch +By default, GitLab does not store anything in the consul cluster that cannot be recreated. To erase the consul database and reinitialize + +``` +# gitlab-ctl stop consul +# rm -rf /var/opt/gitlab/consul/data +# gitlab-ctl start consul +``` + +After this, the cluster should start back up, and the server agents rejoin. Shortly after that, the client agents should rejoin as well. + +#### Recover a failed cluster +If you have taken advantage of consul to store other data, and want to restore the failed cluster, please follow the [Consul guide](https://www.consul.io/docs/guides/outage.html) to recover a failed cluster. diff --git a/doc/administration/high_availability/database.md b/doc/administration/high_availability/database.md index c1eeb40b98f..5c725f00b79 100644 --- a/doc/administration/high_availability/database.md +++ b/doc/administration/high_availability/database.md @@ -1,11 +1,6 @@ -# Configuring a Database for GitLab HA +# Configuring PostgreSQL for Scaling and High Availability -You can choose to install and manage a database server (PostgreSQL/MySQL) -yourself, or you can use GitLab Omnibus packages to help. GitLab recommends -PostgreSQL. This is the database that will be installed if you use the -Omnibus package to manage your database. - -## Configure your own database server +## Provide your own PostgreSQL instance **[CORE ONLY]** If you're hosting GitLab on a cloud provider, you can optionally use a managed service for PostgreSQL. For example, AWS offers a managed Relational @@ -20,91 +15,1147 @@ If you use a cloud-managed service, or provide your own PostgreSQL: 1. Configure the GitLab application servers with the appropriate details. This step is covered in [Configuring GitLab for HA](gitlab.md). -## Configure using Omnibus +## PostgreSQL in a Scaled Environment -1. Download/install GitLab Omnibus using **steps 1 and 2** from - [GitLab downloads](https://about.gitlab.com/downloads). Do not complete other - steps on the download page. -1. Create/edit `/etc/gitlab/gitlab.rb` and use the following configuration. - Be sure to change the `external_url` to match your eventual GitLab front-end - URL. If there is a directive listed below that you do not see in the configuration, be sure to add it. +This section is relevant for [Scaled Architecture](./README.md#scalable-architecture-examples) +environments including [Basic Scaling](./README.md#basic-scaling) and +[Full Scaling](./README.md#full-scaling). - ```ruby - external_url 'https://gitlab.example.com' +### Provide your own PostgreSQL instance **[CORE ONLY]** + +If you want to use your own deployed PostgreSQL instance(s), +see [Provide your own PostgreSQL instance](#provide-your-own-postgresql-instance-core-only) +for more details. However, you can use the GitLab Omnibus package to easily +deploy the bundled PostgreSQL. + +### Standalone PostgreSQL using GitLab Omnibus **[CORE ONLY]** + +1. SSH into the PostgreSQL server. +1. [Download/install](https://about.gitlab.com/installation) the Omnibus GitLab + package you want using **steps 1 and 2** from the GitLab downloads page. + - Do not complete any other steps on the download page. +1. Generate a password hash for PostgreSQL. This assumes you will use the default + username of `gitlab` (recommended). The command will request a password + and confirmation. Use the value that is output by this command in the next + step as the value of `POSTGRESQL_PASSWORD_HASH`. + ```sh + sudo gitlab-ctl pg-password-md5 gitlab + ``` + +1. Edit `/etc/gitlab/gitlab.rb` and add the contents below, updating placeholder + values appropriately. + + - `POSTGRESQL_PASSWORD_HASH` - The value output from the previous step + - `APPLICATION_SERVER_IP_BLOCKS` - A space delimited list of IP subnets or IP + addresses of the GitLab application servers that will connect to the + database. Example: `%w(123.123.123.123/32 123.123.123.234/32)` + + ```ruby # Disable all components except PostgreSQL roles ['postgres_role'] + repmgr['enable'] = false + consul['enable'] = false + prometheus['enable'] = false + alertmanager['enable'] = false + pgbouncer_exporter['enable'] = false + redis_exporter['enable'] = false + gitlab_monitor['enable'] = false + + postgresql['listen_address'] = '0.0.0.0' + postgresql['port'] = 5432 + + # Replace POSTGRESQL_PASSWORD_HASH with a generated md5 value + postgresql['sql_user_password'] = 'POSTGRESQL_PASSWORD_HASH' + + # Replace XXX.XXX.XXX.XXX/YY with Network Address + # ???? + postgresql['trust_auth_cidr_addresses'] = %w(APPLICATION_SERVER_IP_BLOCKS) + + # Disable automatic database migrations + gitlab_rails['auto_migrate'] = false + ``` + + NOTE: **Note:** The role `postgres_role` was introduced with GitLab 10.3 + +1. [Reconfigure GitLab] for the changes to take effect. +1. Note the PostgreSQL node's IP address or hostname, port, and + plain text password. These will be necessary when configuring the GitLab + application servers later. + +Advanced configuration options are supported and can be added if +needed. + +Continue configuration of other components by going +[back to Scaled Architectures](./README.md#scalable-architecture-examples) + +## PostgreSQL with High Availability + +This section is relevant for [High Availability Architecture](./README.md#high-availability-architecture-examples) +environments including [Horizontal](./README.md#horizontal), +[Hybrid](./README.md#hybrid), and +[Fully Distributed](./README.md#fully-distributed). + +### Provide your own PostgreSQL instance **[CORE ONLY]** + +If you want to use your own deployed PostgreSQL instance(s), +see [Provide your own PostgreSQL instance](#provide-your-own-postgresql-instance-core-only) +for more details. However, you can use the GitLab Omnibus package to easily +deploy the bundled PostgreSQL. + +### High Availability with GitLab Omnibus **[PREMIUM ONLY]** + +> Important notes: +> - This document will focus only on configuration supported with [GitLab Premium](https://about.gitlab.com/pricing/), using the Omnibus GitLab package. +> - If you are a Community Edition or Starter user, consider using a cloud hosted solution. +> - This document will not cover installations from source. +> +> - If HA setup is not what you were looking for, see the [database configuration document](http://docs.gitlab.com/omnibus/settings/database.html) +> for the Omnibus GitLab packages. + +> Please read this document fully before attempting to configure PostgreSQL HA +> for GitLab. +> +> This configuration is GA in EE 10.2. + +The recommended configuration for a PostgreSQL HA requires: + +- A minimum of three database nodes + - Each node will run the following services: + - `PostgreSQL` - The database itself + - `repmgrd` - A service to monitor, and handle failover in case of a failure + - `Consul` agent - Used for service discovery, to alert other nodes when failover occurs +- A minimum of three `Consul` server nodes +- A minimum of one `pgbouncer` service node + +You also need to take into consideration the underlying network topology, +making sure you have redundant connectivity between all Database and GitLab instances, +otherwise the networks will become a single point of failure. + +#### Architecture + + + +Database nodes run two services with PostgreSQL: + +- Repmgrd. Monitors the cluster and handles failover when issues with the master occur. The failover consists of: + - Selecting a new master for the cluster. + - Promoting the new node to master. + - Instructing remaining servers to follow the new master node. + + On failure, the old master node is automatically evicted from the cluster, and should be rejoined manually once recovered. +- Consul. Monitors the status of each node in the database cluster and tracks its health in a service definition on the consul cluster. + +Alongside pgbouncer, there is a consul agent that watches the status of the PostgreSQL service. If that status changes, consul runs a script which updates the configuration and reloads pgbouncer + +##### Connection flow + +Each service in the package comes with a set of [default ports](https://docs.gitlab.com/omnibus/package-information/defaults.html#ports). You may need to make specific firewall rules for the connections listed below: + +- Application servers connect to [PgBouncer default port](https://docs.gitlab.com/omnibus/package-information/defaults.html#pgbouncer) +- PgBouncer connects to the primary database servers [PostgreSQL default port](https://docs.gitlab.com/omnibus/package-information/defaults.html#postgresql) +- Repmgr connects to the database servers [PostgreSQL default port](https://docs.gitlab.com/omnibus/package-information/defaults.html#postgresql) +- Postgres secondaries connect to the primary database servers [PostgreSQL default port](https://docs.gitlab.com/omnibus/package-information/defaults.html#postgresql) +- Consul servers and agents connect to each others [Consul default ports](https://docs.gitlab.com/omnibus/package-information/defaults.html#consul) + +#### Required information + +Before proceeding with configuration, you will need to collect all the necessary +information. + +##### Network information + +PostgreSQL does not listen on any network interface by default. It needs to know +which IP address to listen on in order to be accessible to other services. +Similarly, PostgreSQL access is controlled based on the network source. + +This is why you will need: + +> IP address of each nodes network interface +> - This can be set to `0.0.0.0` to listen on all interfaces. It cannot +> be set to the loopack address `127.0.0.1` +> +> Network Address +> - This can be in subnet (i.e. `192.168.0.0/255.255.255.0`) or CIDR (i.e. +> `192.168.0.0/24`) form. + +##### User information + +Various services require different configuration to secure +the communication as well as information required for running the service. +Bellow you will find details on each service and the minimum required +information you need to provide. + +##### Consul information + +When using default setup, minimum configuration requires: + +- `CONSUL_USERNAME`. Defaults to `gitlab-consul` +- `CONSUL_DATABASE_PASSWORD`. Password for the database user. +- `CONSUL_PASSWORD_HASH`. This is a hash generated out of consul username/password pair. + Can be generated with: + + ```sh + sudo gitlab-ctl pg-password-md5 CONSUL_USERNAME + ``` + +- `CONSUL_SERVER_NODES`. The IP addresses or DNS records of the Consul server nodes. + +Few notes on the service itself: + +- The service runs under a system account, by default `gitlab-consul`. + - If you are using a different username, you will have to specify it. We +will refer to it with `CONSUL_USERNAME`, +- There will be a database user created with read only access to the repmgr +database +- Passwords will be stored in the following locations: + - `/etc/gitlab/gitlab.rb`: hashed + - `/var/opt/gitlab/pgbouncer/pg_auth`: hashed + - `/var/opt/gitlab/gitlab-consul/.pgpass`: plaintext + +##### PostgreSQL information + +When configuring PostgreSQL, we will set `max_wal_senders` to one more than +the number of database nodes in the cluster. +This is used to prevent replication from using up all of the +available database connections. + +> Note: +> - In this document we are assuming 3 database nodes, which makes this configuration: + +``` +postgresql['max_wal_senders'] = 4 +``` + +As previously mentioned, you'll have to prepare the network subnets that will +be allowed to authenticate with the database. +You'll also need to supply the IP addresses or DNS records of Consul +server nodes. + +We will need the following password information for the application's database user: + +- `POSTGRESQL_USERNAME`. Defaults to `gitlab` +- `POSTGRESQL_USER_PASSWORD`. The password for the database user +- `POSTGRESQL_PASSWORD_HASH`. This is a hash generated out of the username/password pair. + Can be generated with: + + ```sh + sudo gitlab-ctl pg-password-md5 POSTGRESQL_USERNAME + ``` + +##### Pgbouncer information + +When using default setup, minimum configuration requires: + +- `PGBOUNCER_USERNAME`. Defaults to `pgbouncer` +- `PGBOUNCER_PASSWORD`. This is a password for pgbouncer service. +- `PGBOUNCER_PASSWORD_HASH`. This is a hash generated out of pgbouncer username/password pair. + Can be generated with: + + ```sh + sudo gitlab-ctl pg-password-md5 PGBOUNCER_USERNAME + ``` + +- `PGBOUNCER_NODE`, is the IP address or a FQDN of the node running Pgbouncer. + +Few notes on the service itself: + +- The service runs as the same system account as the database + - In the package, this is by default `gitlab-psql` +- If you use a non-default user account for Pgbouncer service (by default `pgbouncer`), you will have to specify this username. We will refer to this requirement with `PGBOUNCER_USERNAME`. +- The service will have a regular database user account generated for it + - This defaults to `repmgr` +- Passwords will be stored in the following locations: + - `/etc/gitlab/gitlab.rb`: hashed, and in plain text + - `/var/opt/gitlab/pgbouncer/pg_auth`: hashed + +##### Repmgr information + +When using default setup, you will only have to prepare the network subnets that will +be allowed to authenticate with the service. + +Few notes on the service itself: + +- The service runs under the same system account as the database + - In the package, this is by default `gitlab-psql` +- The service will have a superuser database user account generated for it + - This defaults to `gitlab_repmgr` + +#### Installing Omnibus GitLab + +First, make sure to [download/install](https://about.gitlab.com/installation) +GitLab Omnibus **on each node**. + +Make sure you install the necessary dependencies from step 1, +add GitLab package repository from step 2. +When installing the GitLab package, do not supply `EXTERNAL_URL` value. + +#### Configuring the Consul nodes + +On each Consul node perform the following: + +1. Make sure you collect [`CONSUL_SERVER_NODES`](#consul-information) before executing the next step. + +1. Edit `/etc/gitlab/gitlab.rb` replacing values noted in the `# START user configuration` section: + + ```ruby + # Disable all components except Consul + roles ['consul_role'] + + # START user configuration + # Replace placeholders: + # + # Y.Y.Y.Y consul1.gitlab.example.com Z.Z.Z.Z + # with the addresses gathered for CONSUL_SERVER_NODES + consul['configuration'] = { + server: true, + retry_join: %w(Y.Y.Y.Y consul1.gitlab.example.com Z.Z.Z.Z) + } + + # Disable auto migrations + gitlab_rails['auto_migrate'] = false + # + # END user configuration + ``` + + > `consul_role` was introduced with GitLab 10.3 + +1. [Reconfigure GitLab] for the changes to take effect. + +##### Consul Checkpoint + +Before moving on, make sure Consul is configured correctly. Run the following +command to verify all server nodes are communicating: + +``` +/opt/gitlab/embedded/bin/consul members +``` + +The output should be similar to: + +``` +Node Address Status Type Build Protocol DC +CONSUL_NODE_ONE XXX.XXX.XXX.YYY:8301 alive server 0.9.2 2 gitlab_consul +CONSUL_NODE_TWO XXX.XXX.XXX.YYY:8301 alive server 0.9.2 2 gitlab_consul +CONSUL_NODE_THREE XXX.XXX.XXX.YYY:8301 alive server 0.9.2 2 gitlab_consul +``` + +If any of the nodes isn't `alive` or if any of the three nodes are missing, +check the [Troubleshooting section](#troubleshooting) before proceeding. + +#### Configuring the Database nodes + +1. Make sure you collect [`CONSUL_SERVER_NODES`](#consul-information), [`PGBOUNCER_PASSWORD_HASH`](#pgbouncer-information), [`POSTGRESQL_PASSWORD_HASH`](#postgresql-information), the [number of db nodes](#postgresql-information), and the [network address](#network-information) before executing the next step. + +1. On the master database node, edit `/etc/gitlab/gitlab.rb` replacing values noted in the `# START user configuration` section: + + ```ruby + # Disable all components except PostgreSQL and Repmgr and Consul + roles ['postgres_role'] # PostgreSQL configuration - gitlab_rails['db_password'] = 'DB password' - postgresql['md5_auth_cidr_addresses'] = ['0.0.0.0/0'] postgresql['listen_address'] = '0.0.0.0' + postgresql['hot_standby'] = 'on' + postgresql['wal_level'] = 'replica' + postgresql['shared_preload_libraries'] = 'repmgr_funcs' # Disable automatic database migrations gitlab_rails['auto_migrate'] = false + + # Configure the consul agent + consul['services'] = %w(postgresql) + + # START user configuration + # Please set the real values as explained in Required Information section + # + # Replace PGBOUNCER_PASSWORD_HASH with a generated md5 value + postgresql['pgbouncer_user_password'] = 'PGBOUNCER_PASSWORD_HASH' + # Replace POSTGRESQL_PASSWORD_HASH with a generated md5 value + postgresql['sql_user_password'] = 'POSTGRESQL_PASSWORD_HASH' + # Replace X with value of number of db nodes + 1 + postgresql['max_wal_senders'] = X + + # Replace XXX.XXX.XXX.XXX/YY with Network Address + postgresql['trust_auth_cidr_addresses'] = %w(XXX.XXX.XXX.XXX/YY) + repmgr['trust_auth_cidr_addresses'] = %w(127.0.0.1/32 XXX.XXX.XXX.XXX/YY) + + # Replace placeholders: + # + # Y.Y.Y.Y consul1.gitlab.example.com Z.Z.Z.Z + # with the addresses gathered for CONSUL_SERVER_NODES + consul['configuration'] = { + retry_join: %w(Y.Y.Y.Y consul1.gitlab.example.com Z.Z.Z.Z) + } + # + # END user configuration ``` -1. Run `sudo gitlab-ctl reconfigure` to install and configure PostgreSQL. + > `postgres_role` was introduced with GitLab 10.3 + +1. On secondary nodes, add all the configuration specified above for primary node + to `/etc/gitlab/gitlab.rb`. In addition, append the following configuration + to inform gitlab-ctl that they are standby nodes initially and it need not + attempt to register them as primary node + ``` + # HA setting to specify if a node should attempt to be master on initialization + repmgr['master_on_initialization'] = false + ``` - > **Note**: This `reconfigure` step will result in some errors. - That's OK - don't be alarmed. +1. [Reconfigure GitLab] for te changes to take effect. + +> Please note: +> - If you want your database to listen on a specific interface, change the config: +> `postgresql['listen_address'] = '0.0.0.0'` +> - If your Pgbouncer service runs under a different user account, +> you also need to specify: `postgresql['pgbouncer_user'] = PGBOUNCER_USERNAME` in +> your configuration + +##### Database nodes post-configuration + +###### Primary node + +Select one node as a primary node. 1. Open a database prompt: + ```sh + gitlab-psql -d gitlabhq_production ``` - su - gitlab-psql - /bin/bash - psql -h /var/opt/gitlab/postgresql -d template1 - # Output: +1. Enable the `pg_trgm` extension: - psql (9.2.15) - Type "help" for help. + ```sh + CREATE EXTENSION pg_trgm; + ``` + +1. Exit the database prompt by typing `\q` and Enter. + +1. Verify the cluster is initialized with one node: - template1=# + ```sh + gitlab-ctl repmgr cluster show ``` -1. Run the following command at the database prompt and you will be asked to - enter the new password for the PostgreSQL superuser. + The output should be similar to the following: ``` - \password + Role | Name | Upstream | Connection String + ----------+----------|----------|---------------------------------------- + * master | HOSTNAME | | host=HOSTNAME user=gitlab_repmgr dbname=gitlab_repmgr + ``` - # Output: +1. Note down the hostname/ip in the connection string: `host=HOSTNAME`. We will + refer to the hostname in the next section as `MASTER_NODE_NAME`. If the value + is not an IP address, it will need to be a resolvable name (via DNS or + `/etc/hosts`) - Enter new password: - Enter it again: + +###### Secondary nodes + +1. Set up the repmgr standby: + + ```sh + gitlab-ctl repmgr standby setup MASTER_NODE_NAME ``` -1. Similarly, set the password for the `gitlab` database user. Use the same - password that you specified in the `/etc/gitlab/gitlab.rb` file for - `gitlab_rails['db_password']`. + Do note that this will remove the existing data on the node. The command + has a wait time. + The output should be similar to the following: + + ```console + # gitlab-ctl repmgr standby setup MASTER_NODE_NAME + Doing this will delete the entire contents of /var/opt/gitlab/postgresql/data + If this is not what you want, hit Ctrl-C now to exit + To skip waiting, rerun with the -w option + Sleeping for 30 seconds + Stopping the database + Removing the data + Cloning the data + Starting the database + Registering the node with the cluster + ok: run: repmgrd: (pid 19068) 0s ``` - \password gitlab - # Output: +1. Verify the node now appears in the cluster: - Enter new password: - Enter it again: + ```sh + gitlab-ctl repmgr cluster show ``` -1. Exit from editing `template1` prompt by typing `\q` and Enter. -1. Enable the `pg_trgm` extension within the `gitlabhq_production` database: - + + The output should be similar to the following: + + ``` + Role | Name | Upstream | Connection String + ----------+---------|-----------|------------------------------------------------ + * master | MASTER | | host=MASTER_NODE_NAME user=gitlab_repmgr dbname=gitlab_repmgr + standby | STANDBY | MASTER | host=STANDBY_HOSTNAME user=gitlab_repmgr dbname=gitlab_repmgr + ``` + +Repeat the above steps on all secondary nodes. + +##### Database checkpoint + +Before moving on, make sure the databases are configured correctly. Run the +following command on the **primary** node to verify that replication is working +properly: + +``` +gitlab-ctl repmgr cluster show +``` + +The output should be similar to: + +``` +Role | Name | Upstream | Connection String +----------+--------------|--------------|-------------------------------------------------------------------- +* master | MASTER | | host=MASTER port=5432 user=gitlab_repmgr dbname=gitlab_repmgr + standby | STANDBY | MASTER | host=STANDBY port=5432 user=gitlab_repmgr dbname=gitlab_repmgr +``` + +If the 'Role' column for any node says "FAILED", check the +[Troubleshooting section](#troubleshooting) before proceeding. + +Also, check that the check master command works successfully on each node: + +``` +su - gitlab-consul +gitlab-ctl repmgr-check-master || echo 'This node is a standby repmgr node' +``` + +This command relies on exit codes to tell Consul whether a particular node is a master +or secondary. The most important thing here is that this command does not produce errors. +If there are errors it's most likely due to incorrect `gitlab-consul` database user permissions. +Check the [Troubleshooting section](#troubleshooting) before proceeding. + +#### Configuring the Pgbouncer node + +1. Make sure you collect [`CONSUL_SERVER_NODES`](#consul-information), [`CONSUL_PASSWORD_HASH`](#consul-information), and [`PGBOUNCER_PASSWORD_HASH`](#pgbouncer-information) before executing the next step. + +1. Edit `/etc/gitlab/gitlab.rb` replacing values noted in the `# START user configuration` section: + + ```ruby + # Disable all components except Pgbouncer and Consul agent + roles ['pgbouncer_role'] + + # Configure Pgbouncer + pgbouncer['admin_users'] = %w(pgbouncer gitlab-consul) + + # Configure Consul agent + consul['watchers'] = %w(postgresql) + + # START user configuration + # Please set the real values as explained in Required Information section + # Replace CONSUL_PASSWORD_HASH with with a generated md5 value + # Replace PGBOUNCER_PASSWORD_HASH with with a generated md5 value + pgbouncer['users'] = { + 'gitlab-consul': { + password: 'CONSUL_PASSWORD_HASH' + }, + 'pgbouncer': { + password: 'PGBOUNCER_PASSWORD_HASH' + } + } + # Replace placeholders: + # + # Y.Y.Y.Y consul1.gitlab.example.com Z.Z.Z.Z + # with the addresses gathered for CONSUL_SERVER_NODES + consul['configuration'] = { + retry_join: %w(Y.Y.Y.Y consul1.gitlab.example.com Z.Z.Z.Z) + } + # + # END user configuration ``` + + > `pgbouncer_role` was introduced with GitLab 10.3 + +1. [Reconfigure GitLab] for the changes to take effect. + +1. Create a `.pgpass` file so Consule is able to + reload pgbouncer. Enter the `PGBOUNCER_PASSWORD` twice when asked: + + ```sh + gitlab-ctl write-pgpass --host 127.0.0.1 --database pgbouncer --user pgbouncer --hostuser gitlab-consul + ``` + +##### PGBouncer Checkpoint + +1. Ensure the node is talking to the current master: + + ```sh + gitlab-ctl pgb-console # You will be prompted for PGBOUNCER_PASSWORD + ``` + + If there is an error `psql: ERROR: Auth failed` after typing in the + password, ensure you previously generated the MD5 password hashes with the correct + format. The correct format is to concatenate the password and the username: + `PASSWORDUSERNAME`. For example, `Sup3rS3cr3tpgbouncer` would be the text + needed to generate an MD5 password hash for the `pgbouncer` user. + +1. Once the console prompt is available, run the following queries: + + ```sh + show databases ; show clients ; + ``` + + The output should be similar to the following: + + ``` + name | host | port | database | force_user | pool_size | reserve_pool | pool_mode | max_connections | current_connections + ---------------------+-------------+------+---------------------+------------+-----------+--------------+-----------+-----------------+--------------------- + gitlabhq_production | MASTER_HOST | 5432 | gitlabhq_production | | 20 | 0 | | 0 | 0 + pgbouncer | | 6432 | pgbouncer | pgbouncer | 2 | 0 | statement | 0 | 0 + (2 rows) + + type | user | database | state | addr | port | local_addr | local_port | connect_time | request_time | ptr | link | remote_pid | tls + ------+-----------+---------------------+---------+----------------+-------+------------+------------+---------------------+---------------------+-----------+------+------------+----- + C | pgbouncer | pgbouncer | active | 127.0.0.1 | 56846 | 127.0.0.1 | 6432 | 2017-08-21 18:09:59 | 2017-08-21 18:10:48 | 0x22b3880 | | 0 | + (2 rows) + ``` + +#### Configuring the Application nodes + +These will be the nodes running the `gitlab-rails` service. You may have other +attributes set, but the following need to be set. + +1. Edit `/etc/gitlab/gitlab.rb`: + + ```ruby + # Disable PostgreSQL on the application node + postgresql['enable'] = false + + gitlab_rails['db_host'] = 'PGBOUNCER_NODE' + gitlab_rails['db_port'] = 6432 + gitlab_rails['db_password'] = 'POSTGRESQL_USER_PASSWORD' + gitlab_rails['auto_migrate'] = false + ``` + +1. [Reconfigure GitLab] for the changes to take effect. + +##### Application node post-configuration + +Ensure that all migrations ran: + +```sh +gitlab-rake gitlab:db:configure +``` + +> **Note**: If you encounter a `rake aborted!` error stating that PGBouncer is failing to connect to +PostgreSQL it may be that your PGBouncer node's IP address is missing from +PostgreSQL's `trust_auth_cidr_addresses` in `gitlab.rb` on your database nodes. See +[PGBouncer error `ERROR: pgbouncer cannot connect to server`](#pgbouncer-error-error-pgbouncer-cannot-connect-to-server) +in the Troubleshooting section before proceeding. + +##### Ensure GitLab is running + +At this point, your GitLab instance should be up and running. Verify you are +able to login, and create issues and merge requests. If you have troubles check +the [Troubleshooting section](#troubleshooting). + +#### Example configuration + +Here we'll show you some fully expanded example configurations. + +##### Example recommended setup + +This example uses 3 consul servers, 3 postgresql servers, and 1 application node. + +We start with all servers on the same 10.6.0.0/16 private network range, they +can connect to each freely other on those addresses. + +Here is a list and description of each machine and the assigned IP: + +* `10.6.0.11`: Consul 1 +* `10.6.0.12`: Consul 2 +* `10.6.0.13`: Consul 3 +* `10.6.0.21`: PostgreSQL master +* `10.6.0.22`: PostgreSQL secondary +* `10.6.0.23`: PostgreSQL secondary +* `10.6.0.31`: GitLab application + +All passwords are set to `toomanysecrets`, please do not use this password or derived hashes. + +The external_url for GitLab is `http://gitlab.example.com` + +Please note that after the initial configuration, if a failover occurs, the PostgresSQL master will change to one of the available secondaries until it is failed back. + +##### Example recommended setup for Consul servers + +On each server edit `/etc/gitlab/gitlab.rb`: + +```ruby +# Disable all components except Consul +roles ['consul_role'] + +consul['configuration'] = { + server: true, + retry_join: %w(10.6.0.11 10.6.0.12 10.6.0.13) +} +``` + +[Reconfigure Omnibus GitLab][reconfigure GitLab] for the changes to take effect. + +##### Example recommended setup for PostgreSQL servers + +###### Primary node + +On primary node edit `/etc/gitlab/gitlab.rb`: + +```ruby +# Disable all components except PostgreSQL and Repmgr and Consul +roles ['postgres_role'] + +# PostgreSQL configuration +postgresql['listen_address'] = '0.0.0.0' +postgresql['hot_standby'] = 'on' +postgresql['wal_level'] = 'replica' +postgresql['shared_preload_libraries'] = 'repmgr_funcs' + +# Disable automatic database migrations +gitlab_rails['auto_migrate'] = false + +# Configure the consul agent +consul['services'] = %w(postgresql) + +postgresql['pgbouncer_user_password'] = '771a8625958a529132abe6f1a4acb19c' +postgresql['sql_user_password'] = '450409b85a0223a214b5fb1484f34d0f' +postgresql['max_wal_senders'] = 4 + +postgresql['trust_auth_cidr_addresses'] = %w(10.6.0.0/16) +repmgr['trust_auth_cidr_addresses'] = %w(10.6.0.0/16) + +consul['configuration'] = { + retry_join: %w(10.6.0.11 10.6.0.12 10.6.0.13) +} +``` + +[Reconfigure Omnibus GitLab][reconfigure GitLab] for the changes to take effect. + +###### Secondary nodes + +On secondary nodes, edit `/etc/gitlab/gitlab.rb` and add all the configuration +added to primary node, noted above. In addition, append the following +configuration + +``` +# HA setting to specify if a node should attempt to be master on initialization +repmgr['master_on_initialization'] = false +``` + +[Reconfigure Omnibus GitLab][reconfigure GitLab] for the changes to take effect. + +##### Example recommended setup for application server + +On the server edit `/etc/gitlab/gitlab.rb`: + +```ruby +external_url 'http://gitlab.example.com' + +gitlab_rails['db_host'] = '127.0.0.1' +gitlab_rails['db_port'] = 6432 +gitlab_rails['db_password'] = 'toomanysecrets' +gitlab_rails['auto_migrate'] = false + +postgresql['enable'] = false +pgbouncer['enable'] = true +consul['enable'] = true + +# Configure Pgbouncer +pgbouncer['admin_users'] = %w(pgbouncer gitlab-consul) + +# Configure Consul agent +consul['watchers'] = %w(postgresql) + +pgbouncer['users'] = { + 'gitlab-consul': { + password: '5e0e3263571e3704ad655076301d6ebe' + }, + 'pgbouncer': { + password: '771a8625958a529132abe6f1a4acb19c' + } +} + +consul['configuration'] = { + retry_join: %w(10.6.0.11 10.6.0.12 10.6.0.13) +} +``` + +[Reconfigure Omnibus GitLab][reconfigure GitLab] for the changes to take effect. + +##### Example recommended setup manual steps + +After deploying the configuration follow these steps: + +1. On `10.6.0.21`, our primary database + + Enable the `pg_trgm` extension + + ```sh gitlab-psql -d gitlabhq_production - + ``` + + ``` CREATE EXTENSION pg_trgm; + ``` + +1. On `10.6.0.22`, our first standby database - # Output: + Make this node a standby of the primary - CREATE EXTENSION + ```sh + gitlab-ctl repmgr standby setup 10.6.0.21 ``` -1. Exit the database prompt by typing `\q` and Enter. -1. Exit the `gitlab-psql` user by running `exit` twice. -1. Run `sudo gitlab-ctl reconfigure` a final time. -1. Configure the GitLab application servers with the appropriate details. - This step is covered in [Configuring GitLab for HA](gitlab.md). + +1. On `10.6.0.23`, our second standby database + + Make this node a standby of the primary + + ```sh + gitlab-ctl repmgr standby setup 10.6.0.21 + ``` + +1. On `10.6.0.31`, our application server + + Set gitlab-consul's pgbouncer password to `toomanysecrets` + + ```sh + gitlab-ctl write-pgpass --host 127.0.0.1 --database pgbouncer --user pgbouncer --hostuser gitlab-consul + ``` + + Run database migrations + + ```sh + gitlab-rake gitlab:db:configure + ``` + +#### Example minimal setup + +This example uses 3 postgresql servers, and 1 application node. + +It differs from the [recommended setup](#example-recommended-setup) by moving the consul servers into the same servers we use for PostgreSQL. +The trade-off is between reducing server counts, against the increased operational complexity of needing to deal with postgres [failover](#failover-procedure) and [restore](#restore-procedure) procedures in addition to [consul outage recovery](consul.md#outage-recovery) on the same set of machines. + +In this example we start with all servers on the same 10.6.0.0/16 private network range, they can connect to each freely other on those addresses. + +Here is a list and description of each machine and the assigned IP: + +* `10.6.0.21`: PostgreSQL master +* `10.6.0.22`: PostgreSQL secondary +* `10.6.0.23`: PostgreSQL secondary +* `10.6.0.31`: GitLab application + +All passwords are set to `toomanysecrets`, please do not use this password or derived hashes. + +The external_url for GitLab is `http://gitlab.example.com` + +Please note that after the initial configuration, if a failover occurs, the PostgresSQL master will change to one of the available secondaries until it is failed back. + +##### Example minimal configuration for database servers + +##### Primary node +On primary database node edit `/etc/gitlab/gitlab.rb`: + +```ruby +# Disable all components except PostgreSQL, Repmgr, and Consul +roles ['postgres_role'] + +# PostgreSQL configuration +postgresql['listen_address'] = '0.0.0.0' +postgresql['hot_standby'] = 'on' +postgresql['wal_level'] = 'replica' +postgresql['shared_preload_libraries'] = 'repmgr_funcs' + +# Disable automatic database migrations +gitlab_rails['auto_migrate'] = false + +# Configure the consul agent +consul['services'] = %w(postgresql) + +postgresql['pgbouncer_user_password'] = '771a8625958a529132abe6f1a4acb19c' +postgresql['sql_user_password'] = '450409b85a0223a214b5fb1484f34d0f' +postgresql['max_wal_senders'] = 4 + +postgresql['trust_auth_cidr_addresses'] = %w(10.6.0.0/16) +repmgr['trust_auth_cidr_addresses'] = %w(10.6.0.0/16) + +consul['configuration'] = { + server: true, + retry_join: %w(10.6.0.21 10.6.0.22 10.6.0.23) +} +``` + +[Reconfigure Omnibus GitLab][reconfigure GitLab] for the changes to take effect. + +###### Secondary nodes + +On secondary nodes, edit `/etc/gitlab/gitlab.rb` and add all the information added +to primary node, noted above. In addition, append the following configuration + +``` +# HA setting to specify if a node should attempt to be master on initialization +repmgr['master_on_initialization'] = false +``` + +##### Example minimal configuration for application server + +On the server edit `/etc/gitlab/gitlab.rb`: + +```ruby +external_url 'http://gitlab.example.com' + +gitlab_rails['db_host'] = '127.0.0.1' +gitlab_rails['db_port'] = 6432 +gitlab_rails['db_password'] = 'toomanysecrets' +gitlab_rails['auto_migrate'] = false + +postgresql['enable'] = false +pgbouncer['enable'] = true +consul['enable'] = true + +# Configure Pgbouncer +pgbouncer['admin_users'] = %w(pgbouncer gitlab-consul) + +# Configure Consul agent +consul['watchers'] = %w(postgresql) + +pgbouncer['users'] = { + 'gitlab-consul': { + password: '5e0e3263571e3704ad655076301d6ebe' + }, + 'pgbouncer': { + password: '771a8625958a529132abe6f1a4acb19c' + } +} + +consul['configuration'] = { + retry_join: %w(10.6.0.21 10.6.0.22 10.6.0.23) +} +``` + +[Reconfigure Omnibus GitLab][reconfigure GitLab] for the changes to take effect. + +##### Example minimal setup manual steps + +The manual steps for this configuration are the same as for the [example recommended setup](#example-recommended-setup-manual-steps). + +#### Failover procedure + +By default, if the master database fails, `repmgrd` should promote one of the +standby nodes to master automatically, and consul will update pgbouncer with +the new master. + +If you need to failover manually, you have two options: + +**Shutdown the current master database** + +Run: + +```sh +gitlab-ctl stop postgresql +``` + +The automated failover process will see this and failover to one of the +standby nodes. + +**Or perform a manual failover** + +1. Ensure the old master node is not still active. +1. Login to the server that should become the new master and run: + + ```sh + gitlab-ctl repmgr standby promote + ``` + +1. If there are any other standby servers in the cluster, have them follow + the new master server: + + ```sh + gitlab-ctl repmgr standby follow NEW_MASTER + ``` + +#### Restore procedure + +If a node fails, it can be removed from the cluster, or added back as a standby +after it has been restored to service. + +- If you want to remove the node from the cluster, on any other node in the + cluster, run: + + ```sh + gitlab-ctl repmgr standby unregister --node=X + ``` + + where X is the value of node in `repmgr.conf` on the old server. + + To find this, you can use: + + ```sh + awk -F = '$1 == "node" { print $2 }' /var/opt/gitlab/postgresql/repmgr.conf + ``` + + It will output something like: + + ``` + 959789412 + ``` + + Then you will use this id to unregister the node: + + ```sh + gitlab-ctl repmgr standby unregister --node=959789412 + ``` + +- To add the node as a standby server: + + ```sh + gitlab-ctl repmgr standby follow NEW_MASTER + gitlab-ctl restart repmgrd + ``` + + CAUTION: **Warning:** When the server is brought back online, and before + you switch it to a standby node, repmgr will report that there are two masters. + If there are any clients that are still attempting to write to the old master, + this will cause a split, and the old master will need to be resynced from + scratch by performing a `gitlab-ctl repmgr standby setup NEW_MASTER`. + +#### Alternate configurations + +##### Database authorization + +By default, we give any host on the database network the permission to perform +repmgr operations using PostgreSQL's `trust` method. If you do not want this +level of trust, there are alternatives. + +You can trust only the specific nodes that will be database clusters, or you +can require md5 authentication. + +##### Trust specific addresses + +If you know the IP address, or FQDN of all database and pgbouncer nodes in the +cluster, you can trust only those nodes. + +In `/etc/gitlab/gitlab.rb` on all of the database nodes, set +`repmgr['trust_auth_cidr_addresses']` to an array of strings containing all of +the addresses. + +If setting to a node's FQDN, they must have a corresponding PTR record in DNS. +If setting to a node's IP address, specify it as `XXX.XXX.XXX.XXX/32`. + +For example: + +```ruby +repmgr['trust_auth_cidr_addresses'] = %w(192.168.1.44/32 db2.example.com) +``` + + +##### MD5 Authentication + +If you are running on an untrusted network, repmgr can use md5 authentication +with a [.pgpass file](https://www.postgresql.org/docs/9.6/static/libpq-pgpass.html) +to authenticate. + +You can specify by IP address, FQDN, or by subnet, using the same format as in +the previous section: + +1. On the current master node, create a password for the `gitlab` and + `gitlab_repmgr` user: + + ```sh + gitlab-psql -d template1 + template1=# \password gitlab_repmgr + Enter password: **** + Confirm password: **** + template1=# \password gitlab + ``` + +1. On each database node: + + 1. Edit `/etc/gitlab/gitlab.rb`: + 1. Ensure `repmgr['trust_auth_cidr_addresses']` is **not** set + 1. Set `postgresql['md5_auth_cidr_addresses']` to the desired value + 1. Set `postgresql['sql_replication_user'] = 'gitlab_repmgr'` + 1. Reconfigure with `gitlab-ctl reconfigure` + 1. Restart postgresql with `gitlab-ctl restart postgresql` + + 1. Create a `.pgpass` file. Enter the `gitlab_repmgr` password twice to + when asked: + + ```sh + gitlab-ctl write-pgpass --user gitlab_repmgr --hostuser gitlab-psql --database '*' + ``` + +1. On each pgbouncer node, edit `/etc/gitlab/gitlab.rb`: + 1. Ensure `gitlab_rails['db_password']` is set to the plaintext password for + the `gitlab` database user + 1. [Reconfigure GitLab] for the changes to take effect + +## Troubleshooting + +#### Consul and PostgreSQL changes not taking effect. + +Due to the potential impacts, `gitlab-ctl reconfigure` only reloads Consul and PostgreSQL, it will not restart the services. However, not all changes can be activated by reloading. + +To restart either service, run `gitlab-ctl restart SERVICE` + +For PostgreSQL, it is usually safe to restart the master node by default. Automatic failover defaults to a 1 minute timeout. Provided the database returns before then, nothing else needs to be done. To be safe, you can stop `repmgrd` on the standby nodes first with `gitlab-ctl stop repmgrd`, then start afterwards with `gitlab-ctl start repmgrd`. + +On the consul server nodes, it is important to restart the consul service in a controlled fashion. Read our [consul documentation](consul.md#restarting-the-server-cluster) for instructions on how to restart the service. + +#### `gitlab-ctl repmgr-check-master` command produces errors + +If this command displays errors about database permissions it is likely that something failed during +install, resulting in the `gitlab-consul` database user getting incorrect permissions. Follow these +steps to fix the problem: + +1. On the master database node, connect to the database prompt - `gitlab-psql -d template1` +1. Delete the `gitlab-consul` user - `DROP USER "gitlab-consul";` +1. Exit the database prompt - `\q` +1. [Reconfigure GitLab] and the user will be re-added with the proper permissions. +1. Change to the `gitlab-consul` user - `su - gitlab-consul` +1. Try the check command again - `gitlab-ctl repmgr-check-master`. + +Now there should not be errors. If errors still occur then there is another problem. + +#### PGBouncer error `ERROR: pgbouncer cannot connect to server` + +You may get this error when running `gitlab-rake gitlab:db:configure` or you +may see the error in the PGBouncer log file. + +``` +PG::ConnectionBad: ERROR: pgbouncer cannot connect to server +``` + +The problem may be that your PGBouncer node's IP address is not included in the +`trust_auth_cidr_addresses` setting in `/etc/gitlab/gitlab.rb` on the database nodes. + +You can confirm that this is the issue by checking the PostgreSQL log on the master +database node. If you see the following error then `trust_auth_cidr_addresses` +is the problem. + +``` +2018-03-29_13:59:12.11776 FATAL: no pg_hba.conf entry for host "123.123.123.123", user "pgbouncer", database "gitlabhq_production", SSL off +``` + +To fix the problem, add the IP address to `/etc/gitlab/gitlab.rb`. + +``` +postgresql['trust_auth_cidr_addresses'] = %w(123.123.123.123/32 <other_cidrs>) +``` + +[Reconfigure GitLab] for the changes to take effect. + +#### Issues with other components + +If you're running into an issue with a component not outlined here, be sure to check the troubleshooting section of their specific documentation page. + +- [Consul](consul.md#troubleshooting) +- [PostgreSQL](http://docs.gitlab.com/omnibus/settings/database.html#troubleshooting) +- [GitLab application](gitlab.md#troubleshooting) + +## Configure using Omnibus + +**Note**: We recommend that you follow the instructions here for a full [PostgreSQL cluster](#configure-using-omnibus-for-high-availability). +If you are reading this section due to an old bookmark, you can find that old documentation [in the repository](https://gitlab.com/gitlab-org/gitlab-ce/blob/v10.1.4/doc/administration/high_availability/database.md#configure-using-omnibus). --- @@ -114,3 +1165,6 @@ Read more on high-availability configuration: 1. [Configure NFS](nfs.md) 1. [Configure the GitLab application servers](gitlab.md) 1. [Configure the load balancers](load_balancer.md) +1. [Manage the bundled Consul cluster](consul.md) + +[reconfigure GitLab]: ../restart_gitlab.md#omnibus-gitlab-reconfigure diff --git a/doc/administration/high_availability/gitaly.md b/doc/administration/high_availability/gitaly.md new file mode 100644 index 00000000000..8004eea2208 --- /dev/null +++ b/doc/administration/high_availability/gitaly.md @@ -0,0 +1,91 @@ +# Configuring Gitaly for Scaled and High Availability + +Gitaly does not yet support full high availability. However, Gitaly is quite +stable and is in use on GitLab.com. Scaled and highly available GitLab environments +should consider using Gitaly on a separate node. + +See the [Gitaly HA Epic](https://gitlab.com/groups/gitlab-org/-/epics/289) to +track plans and progress toward high availability support. + +This document is relevant for [Scaled Architecture](./README.md#scalable-architecture-examples) +environments and [High Availability Architecture](./README.md#high-availability-architecture-examples). + +## Running Gitaly on its own server + +Starting with GitLab 11.4, Gitaly is a replacement for NFS except +when the [Elastic Search indexer](https://gitlab.com/gitlab-org/gitlab-elasticsearch-indexer) +is used. + +NOTE: **Note:** While Gitaly can be used as a replacement for NFS, we do not recommend using EFS as it may impact GitLab's performance. Please review the [relevant documentation](nfs.md#avoid-using-awss-elastic-file-system-efs) for more details. + +NOTE: **Note:** Gitaly network traffic is unencrypted so we recommend a firewall to +restrict access to your Gitaly server. + +The steps below are the minimum necessary to configure a Gitaly server with +Omnibus: + +1. SSH into the Gitaly server. +1. [Download/install](https://about.gitlab.com/installation) the Omnibus GitLab + package you want using **steps 1 and 2** from the GitLab downloads page. + - Do not complete any other steps on the download page. + +1. Edit `/etc/gitlab/gitlab.rb` and add the contents: + + Gitaly must trigger some callbacks to GitLab via GitLab Shell. As a result, + the GitLab Shell secret must be the same between the other GitLab servers and + the Gitaly server. The easiest way to accomplish this is to copy `/etc/gitlab/gitlab-secrets.json` + from an existing GitLab server to the Gitaly server. Without this shared secret, + Git operations in GitLab will result in an API error. + + > **NOTE:** In most or all cases the storage paths below end in `repositories` which is + different than `path` in `git_data_dirs` of Omnibus installations. Check the + directory layout on your Gitaly server to be sure. + + ```ruby + # Enable Gitaly + gitaly['enable'] = true + + ## Disable all other services + sidekiq['enable'] = false + gitlab_workhorse['enable'] = false + unicorn['enable'] = false + postgresql['enable'] = false + nginx['enable'] = false + prometheus['enable'] = false + alertmanager['enable'] = false + pgbouncer_exporter['enable'] = false + redis_exporter['enable'] = false + gitlab_monitor['enable'] = false + gitaly['enable'] = false + + # Prevent database connections during 'gitlab-ctl reconfigure' + gitlab_rails['rake_cache_clear'] = false + gitlab_rails['auto_migrate'] = false + + # Configure the gitlab-shell API callback URL. Without this, `git push` will + # fail. This can be your 'front door' GitLab URL or an internal load + # balancer. + gitlab_rails['internal_api_url'] = 'https://gitlab.example.com' + + # Make Gitaly accept connections on all network interfaces. You must use + # firewalls to restrict access to this address/port. + gitaly['listen_addr'] = "0.0.0.0:8075" + gitaly['auth_token'] = 'abc123secret' + + gitaly['storage'] = [ + { 'name' => 'default', 'path' => '/mnt/gitlab/default/repositories' }, + { 'name' => 'storage1', 'path' => '/mnt/gitlab/storage1/repositories' }, + ] + + # To use tls for gitaly you need to add + gitaly['tls_listen_addr'] = "0.0.0.0:9999" + gitaly['certificate_path'] = "path/to/cert.pem" + gitaly['key_path'] = "path/to/key.pem" + ``` + +Again, reconfigure (Omnibus) or restart (source). + +Continue configuration of other components by going back to: + +- [Scaled Architectures](./README.md#scalable-architecture-examples) +- [High Availability Architectures](./README.md#high-availability-architecture-examples) diff --git a/doc/administration/high_availability/gitlab.md b/doc/administration/high_availability/gitlab.md index d95c3acec54..888426ece5c 100644 --- a/doc/administration/high_availability/gitlab.md +++ b/doc/administration/high_availability/gitlab.md @@ -1,8 +1,4 @@ -# Configuring GitLab for HA - -Assuming you have already configured a [database](database.md), [Redis](redis.md), and [NFS](nfs.md), you can -configure the GitLab application server(s) now. Complete the steps below -for each GitLab application server in your environment. +# Configuring GitLab Scaling and High Availability > **Note:** There is some additional configuration near the bottom for additional GitLab application servers. It's important to read and understand diff --git a/doc/administration/high_availability/nfs_host_client_setup.md b/doc/administration/high_availability/nfs_host_client_setup.md new file mode 100644 index 00000000000..fce27332f23 --- /dev/null +++ b/doc/administration/high_availability/nfs_host_client_setup.md @@ -0,0 +1,136 @@ +# Configuring NFS for GitLab HA + +Setting up NFS for a GitLab HA setup allows all applications nodes in a cluster +to share the same files and maintain data consistency. Application nodes in an HA +setup act as clients while the NFS server plays host. + +> Note: The instructions provided in this documentation allow for setting a quick +proof of concept but will leave NFS as potential single point of failure and +therefore not recommended for use in production. Explore options such as [Pacemaker +and Corosync](http://clusterlabs.org/) for highly available NFS in production. + +Below are instructions for setting up an application node(client) in an HA cluster +to read from and write to a central NFS server(host). + +NOTE: **Note:** +Using EFS may negatively impact performance. Please review the [relevant documentation](nfs.md#avoid-using-awss-elastic-file-system-efs) for additional details. + +## NFS Server Setup + +> Follow the instructions below to set up and configure your NFS server. + +### Step 1 - Install NFS Server on Host + +Installing the nfs-kernel-server package allows you to share directories with the clients running the GitLab application. + +```sh +apt-get update +apt-get install nfs-kernel-server +``` + +### Step 2 - Export Host's Home Directory to Client + +In this setup we will share the home directory on the host with the client. Edit the exports file as below to share the host's home directory with the client. If you have multiple clients running GitLab you must enter the client IP addresses in line in the `/etc/exports` file. + +```text +#/etc/exports for one client +/home <client-ip-address>(rw,sync,no_root_squash,no_subtree_check) + +#/etc/exports for three clients +/home <client-ip-address>(rw,sync,no_root_squash,no_subtree_check) <client-2-ip-address>(rw,sync,no_root_squash,no_subtree_check) <client-3-ip-address>(rw,sync,no_root_squash,no_subtree_check) +``` + +Restart the NFS server after making changes to the `exports` file for the changes +to take effect. + +```sh +systemctl restart nfs-kernel-server +``` + +NOTE: **Note:** +You may need to update your server's firewall. See the [firewall section](#nfs-in-a-firewalled-environment) at the end of this guide. + +## Client/ GitLab application node Setup + +> Follow the instructions below to connect any GitLab rails application node running +inside your HA environment to the NFS server configured above. + +### Step 1 - Install NFS Common on Client + +The nfs-common provides NFS functionality without installing server components which +we don't need running on the application nodes. + +```sh +apt-get update +apt-get install nfs-common +``` + +### Step 2 - Create Mount Points on Client + +Create a directroy on the client that we can mount the shared directory from the host. +Please note that if your mount point directory contains any files they will be hidden +once the remote shares are mounted. An empty/new directory on the client is recommended +for this purpose. + +```sh +mkdir -p /nfs/home +``` + +Confirm that the mount point works by mounting it on the client and checking that +it is mounted with the command below: + +```sh +mount <host_ip_address>:/home +df -h +``` + +### Step 3 - Set up Automatic Mounts on Boot + +Edit `/etc/fstab` on client as below to mount the remote shares automatically at boot. +Note that GitLab requires advisory file locking, which is only supported natively in +NFS version 4. NFSv3 also supports locking as long as Linux Kernel 2.6.5+ is used. +We recommend using version 4 and do not specifically test NFSv3. + +```text +#/etc/fstab +165.227.159.85:/home /nfs/home nfs4 defaults,soft,rsize=1048576,wsize=1048576,noatime,nofail,lookupcache=positive 0 2 +``` + +Reboot the client and confirm that the mount point is mounted automatically. + +### Step 4 - Set up GitLab to Use NFS mounts + +When using the default Omnibus configuration you will need to share 5 data locations +between all GitLab cluster nodes. No other locations should be shared. Changing the +default file locations in `gitlab.rb` on the client allows you to have one main mount +point and have all the required locations as subdirectories to use the NFS mount for +git-data. + +```text +git_data_dirs({"default" => "/nfs/home/var/opt/gitlab-data/git-data"}) +user['home'] = '/nfs/home/var/opt/gitlab-data/home' +gitlab_rails['uploads_directory'] = '/nfs/home/var/opt/gitlab-data/uploads' +gitlab_rails['shared_path'] = '/nfs/home/var/opt/gitlab-data/shared' +gitlab_ci['builds_directory'] = '/nfs/home/var/opt/gitlab-data/builds' +``` + +Save the changes in `gitlab.rb` and run `gitlab-ctl reconfigure`. + +## NFS in a Firewalled Environment + +If the traffic between your NFS server and NFS client(s) is subject to port filtering +by a firewall, then you will need to reconfigure that firewall to allow NFS communication. + +[This guide from TDLP](http://tldp.org/HOWTO/NFS-HOWTO/security.html#FIREWALLS) +covers the basics of using NFS in a firewalled environment. Additionally, we encourage you to +search for and review the specific documentation for your OS/distro and your firewall software. + +Example for Ubuntu: + +Check that NFS traffic from the client is allowed by the firewall on the host by running +the command: `sudo ufw status`. If it's being blocked, then you can allow traffic from a specific +client with the command below. + +```sh +sudo ufw allow from <client-ip-address> to any port nfs +``` diff --git a/doc/administration/high_availability/pg_ha_architecture.png b/doc/administration/high_availability/pg_ha_architecture.png Binary files differnew file mode 100644 index 00000000000..ef870f652ae --- /dev/null +++ b/doc/administration/high_availability/pg_ha_architecture.png diff --git a/doc/administration/high_availability/pgbouncer.md b/doc/administration/high_availability/pgbouncer.md new file mode 100644 index 00000000000..c2c57f8e16d --- /dev/null +++ b/doc/administration/high_availability/pgbouncer.md @@ -0,0 +1,131 @@ +# Working with the bundle Pgbouncer service + +## Overview + +As part of its High Availability stack, GitLab Premium includes a bundled version of [Pgbouncer](https://pgbouncer.github.io/) that can be managed through `/etc/gitlab/gitlab.rb`. + +In a High Availability setup, Pgbouncer is used to seamlessly migrate database connections between servers in a failover scenario. + +Additionally, it can be used in a non-HA setup to pool connections, speeding up response time while reducing resource usage. + +It is recommended to run pgbouncer alongside the `gitlab-rails` service, or on its own dedicated node in a cluster. + +## Operations + +### Running Pgbouncer as part of an HA GitLab installation +See our [HA documentation for PostgreSQL](database.md) for information on running pgbouncer as part of a HA setup + +### Running Pgbouncer as part of a non-HA GitLab installation + +1. Generate PGBOUNCER_USER_PASSWORD_HASH with the command `gitlab-ctl pg-password-md5 pgbouncer` + +1. Generate SQL_USER_PASSWORD_HASH with the command `gitlab-ctl pg-password-md5 gitlab`. We'll also need to enter the plaintext SQL_USER_PASSWORD later + +1. On your database node, ensure the following is set in your `/etc/gitlab/gitlab.rb` + + ```ruby + postgresql['pgbouncer_user_password'] = 'PGBOUNCER_USER_PASSWORD_HASH' + postgresql['sql_user_password'] = 'SQL_USER_PASSWORD_HASH' + postgresql['listen_address'] = 'XX.XX.XX.Y' # Where XX.XX.XX.Y is the ip address on the node postgresql should listen on + postgresql['md5_auth_cidr_addresses'] = %w(AA.AA.AA.B/32) # Where AA.AA.AA.B is the IP address of the pgbouncer node + ``` + +1. Run `gitlab-ctl reconfigure` + + **Note:** If the database was already running, it will need to be restarted after reconfigure by running `gitlab-ctl restart postgresql`. + +1. On the node you are running pgbouncer on, make sure the following is set in `/etc/gitlab/gitlab.rb` + + ```ruby + pgbouncer['enable'] = true + pgbouncer['databases'] = { + gitlabhq_production: { + host: 'DATABASE_HOST', + user: 'pgbouncer', + password: 'PGBOUNCER_USER_PASSWORD_HASH' + } + } + ``` + +1. Run `gitlab-ctl reconfigure` + +1. On the node running unicorn, make sure the following is set in `/etc/gitlab/gitlab.rb` + + ```ruby + gitlab_rails['db_host'] = 'PGBOUNCER_HOST' + gitlab_rails['db_port'] = '6432' + gitlab_rails['db_password'] = 'SQL_USER_PASSWORD' + ``` + +1. Run `gitlab-ctl reconfigure` + +1. At this point, your instance should connect to the database through pgbouncer. If you are having issues, see the [Troubleshooting](#troubleshooting) section + +### Interacting with pgbouncer + +#### Administrative console + +As part of omnibus-gitlab, we provide a command `gitlab-ctl pgb-console` to automatically connect to the pgbouncer administrative console. Please see the [pgbouncer documentation](https://pgbouncer.github.io/usage.html#admin-console) for detailed instructions on how to interact with the console. + +To start a session, run + +```shell +# gitlab-ctl pgb-console +Password for user pgbouncer: +psql (9.6.8, server 1.7.2/bouncer) +Type "help" for help. + +pgbouncer=# +``` + +The password you will be prompted for is the PGBOUNCER_USER_PASSWORD + +To get some basic information about the instance, run +```shell +pgbouncer=# show databases; show clients; show servers; + name | host | port | database | force_user | pool_size | reserve_pool | pool_mode | max_connections | current_connections +---------------------+-----------+------+---------------------+------------+-----------+--------------+-----------+-----------------+--------------------- + gitlabhq_production | 127.0.0.1 | 5432 | gitlabhq_production | | 100 | 5 | | 0 | 1 + pgbouncer | | 6432 | pgbouncer | pgbouncer | 2 | 0 | statement | 0 | 0 +(2 rows) + + type | user | database | state | addr | port | local_addr | local_port | connect_time | request_time | ptr | link +| remote_pid | tls +------+-----------+---------------------+--------+-----------+-------+------------+------------+---------------------+---------------------+-----------+------ ++------------+----- + C | gitlab | gitlabhq_production | active | 127.0.0.1 | 44590 | 127.0.0.1 | 6432 | 2018-04-24 22:13:10 | 2018-04-24 22:17:10 | 0x12444c0 | +| 0 | + C | gitlab | gitlabhq_production | active | 127.0.0.1 | 44592 | 127.0.0.1 | 6432 | 2018-04-24 22:13:10 | 2018-04-24 22:17:10 | 0x12447c0 | +| 0 | + C | gitlab | gitlabhq_production | active | 127.0.0.1 | 44594 | 127.0.0.1 | 6432 | 2018-04-24 22:13:10 | 2018-04-24 22:17:10 | 0x1244940 | +| 0 | + C | gitlab | gitlabhq_production | active | 127.0.0.1 | 44706 | 127.0.0.1 | 6432 | 2018-04-24 22:14:22 | 2018-04-24 22:16:31 | 0x1244ac0 | +| 0 | + C | gitlab | gitlabhq_production | active | 127.0.0.1 | 44708 | 127.0.0.1 | 6432 | 2018-04-24 22:14:22 | 2018-04-24 22:15:15 | 0x1244c40 | +| 0 | + C | gitlab | gitlabhq_production | active | 127.0.0.1 | 44794 | 127.0.0.1 | 6432 | 2018-04-24 22:15:15 | 2018-04-24 22:15:15 | 0x1244dc0 | +| 0 | + C | gitlab | gitlabhq_production | active | 127.0.0.1 | 44798 | 127.0.0.1 | 6432 | 2018-04-24 22:15:15 | 2018-04-24 22:16:31 | 0x1244f40 | +| 0 | + C | pgbouncer | pgbouncer | active | 127.0.0.1 | 44660 | 127.0.0.1 | 6432 | 2018-04-24 22:13:51 | 2018-04-24 22:17:12 | 0x1244640 | +| 0 | +(8 rows) + + type | user | database | state | addr | port | local_addr | local_port | connect_time | request_time | ptr | link | rem +ote_pid | tls +------+--------+---------------------+-------+-----------+------+------------+------------+---------------------+---------------------+-----------+------+---- +--------+----- + S | gitlab | gitlabhq_production | idle | 127.0.0.1 | 5432 | 127.0.0.1 | 35646 | 2018-04-24 22:15:15 | 2018-04-24 22:17:10 | 0x124dca0 | | + 19980 | +(1 row) +``` + +## Troubleshooting + +In case you are experiencing any issues connecting through pgbouncer, the first place to check is always the logs: + +```shell +# gitlab-ctl tail pgbouncer +``` + +Additionally, you can check the output from `show databases` in the [Administrative console](#administrative-console). In the output, you would expect to see values in the `host` field for the `gitlabhq_production` database. Additionally, `current_connections` should be greater than 1. diff --git a/doc/administration/high_availability/redis.md b/doc/administration/high_availability/redis.md index a52bc5c3b02..953edb51bc8 100644 --- a/doc/administration/high_availability/redis.md +++ b/doc/administration/high_availability/redis.md @@ -1,6 +1,103 @@ -# Configuring Redis for GitLab HA +# Configuring Redis for Scaling and High Availability -> Experimental Redis Sentinel support was [Introduced][ce-1877] in GitLab 8.11. +## Provide your own Redis instance **[CORE ONLY]** + +The following are the requirements for providing your own Redis instance: + +- Redis version 2.8 or higher. Version 3.2 or higher is recommend as this is + what ships with the GitLab Omnibus package. +- Standalone Redis or Redis high availability with Sentinel are supported. Redis + Cluster is not supported. +- Managed Redis from cloud providers such as AWS Elasticache will work. If these + services support high availability, be sure it is not the Redis Cluster type. + +Note the Redis node's IP address or hostname, port, and password (if required). +These will be necessary when configuring the GitLab application servers later. + +## Redis in a Scaled Environment + +This section is relevant for [Scaled Architecture](./README.md#scalable-architecture-examples) +environments including [Basic Scaling](./README.md#basic-scaling) and +[Full Scaling](./README.md#full-scaling). + +### Provide your own Redis instance **[CORE ONLY]** + +If you want to use your own deployed Redis instance(s), +see [Provide your own Redis instance](#provide-your-own-redis-instance-core-only) +for more details. However, you can use the GitLab Omnibus package to easily +deploy the bundled Redis. + +### Standalone Redis using GitLab Omnibus **[CORE ONLY]** + +The GitLab Omnibus package can be used to configure a standalone Redis server. +In this configuration Redis is not highly available, and represents a single +point of failure. However, in a scaled environment the objective is to allow +the environment to handle more users or to increase throughput. Redis itself +is generally stable and can handle many requests so it is an acceptable +trade off to have only a single instance. See [Scaling and High Availability](./README.md) +for an overview of GitLab scaling and high availability options. + +The steps below are the minimum necessary to configure a Redis server with +Omnibus: + +1. SSH into the Redis server. +1. [Download/install](https://about.gitlab.com/installation) the Omnibus GitLab + package you want using **steps 1 and 2** from the GitLab downloads page. + - Do not complete any other steps on the download page. + +1. Edit `/etc/gitlab/gitlab.rb` and add the contents: + + ```ruby + ## Enable Redis + redis['enable'] = true + + ## Disable all other services + sidekiq['enable'] = false + gitlab_workhorse['enable'] = false + unicorn['enable'] = false + postgresql['enable'] = false + nginx['enable'] = false + prometheus['enable'] = false + alertmanager['enable'] = false + pgbouncer_exporter['enable'] = false + gitlab_monitor['enable'] = false + gitaly['enable'] = false + + redis['bind'] = '0.0.0.0' + redis['port'] = '6379' + redis['password'] = 'SECRET_PASSWORD_HERE' + + gitlab_rails['auto_migrate'] = false + ``` + +1. [Reconfigure Omnibus GitLab][reconfigure] for the changes to take effect. +1. Note the Redis node's IP address or hostname, port, and + Redis password. These will be necessary when configuring the GitLab + application servers later. + +Advanced configuration options are supported and can be added if +needed. + +Continue configuration of other components by going +[back to Scaled Architectures](./README.md#scalable-architecture-examples) + +## Redis with High Availability + +This section is relevant for [High Availability Architecture](./README.md#high-availability-architecture-examples) +environments including [Horizontal](./README.md#horizontal), +[Hybrid](./README.md#hybrid), and +[Fully Distributed](./README.md#fully-distributed). + +### Provide your own Redis instance **[CORE ONLY]** + +If you want to use your own deployed Redis instance(s), +see [Provide your own Redis instance](#provide-your-own-redis-instance-core-only) +for more details. However, you can use the GitLab Omnibus package to easily +deploy the bundled Redis. + +### High Availability with GitLab Omnibus **[PREMIUM ONLY]** + +> Experimental Redis Sentinel support was [introduced in GitLab 8.11][ce-1877]. Starting with 8.14, Redis Sentinel is no longer experimental. If you've used it with versions `< 8.14` before, please check the updated documentation here. @@ -52,8 +149,6 @@ failure. Make sure that you read this document once as a whole before configuring the components below. -### High Availability with Sentinel - > **Notes:** > - Starting with GitLab `8.11`, you can configure a list of Redis Sentinel > servers that will monitor a group of Redis servers to provide failover support. @@ -267,10 +362,9 @@ The prerequisites for a HA Redis setup are the following: 1. Edit `/etc/gitlab/gitlab.rb` and add the contents: ```ruby - # Enable the master role and disable all other services in the machine - # (you can still enable Sentinel). - redis_master_role['enable'] = true - + # Specify server role as 'redis_master_role' + roles ['redis_master_role'] + # IP address pointing to a local IP that the other machines can reach to. # You can also set bind to '0.0.0.0' which listen in all interfaces. # If you really need to bind to an external accessible IP, make @@ -284,6 +378,7 @@ The prerequisites for a HA Redis setup are the following: # Set up password authentication for Redis (use the same password in all nodes). redis['password'] = 'redis-password-goes-here' ``` + 1. Only the primary GitLab application server should handle migrations. To prevent database migrations from running on upgrade, add the following @@ -295,6 +390,10 @@ The prerequisites for a HA Redis setup are the following: 1. [Reconfigure Omnibus GitLab][reconfigure] for the changes to take effect. +> Note: You can specify multiple roles like sentinel and redis as: +> roles ['redis_sentinel_role', 'redis_master_role']. Read more about high +> availability roles at https://docs.gitlab.com/omnibus/roles/ + ### Step 2. Configuring the slave Redis instances 1. SSH into the **slave** Redis server. @@ -307,11 +406,9 @@ The prerequisites for a HA Redis setup are the following: 1. Edit `/etc/gitlab/gitlab.rb` and add the contents: ```ruby - # Enable the slave role and disable all other services in the machine - # (you can still enable Sentinel). This will also set automatically - # `redis['master'] = false`. - redis_slave_role['enable'] = true - + # Specify server role as 'redis_slave_role' + roles ['redis_slave_role'] + # IP address pointing to a local IP that the other machines can reach to. # You can also set bind to '0.0.0.0' which listen in all interfaces. # If you really need to bind to an external accessible IP, make @@ -333,17 +430,19 @@ The prerequisites for a HA Redis setup are the following: #redis['master_port'] = 6379 ``` -1. To prevent database migrations from running on upgrade, run: +1. To prevent reconfigure from running automatically on upgrade, run: ``` sudo touch /etc/gitlab/skip-auto-reconfigure ``` - Only the primary GitLab application server should handle migrations. - 1. [Reconfigure Omnibus GitLab][reconfigure] for the changes to take effect. 1. Go through the steps again for all the other slave nodes. +> Note: You can specify multiple roles like sentinel and redis as: +> roles ['redis_sentinel_role', 'redis_slave_role']. Read more about high +> availability roles at https://docs.gitlab.com/omnibus/roles/ + --- These values don't have to be changed again in `/etc/gitlab/gitlab.rb` after @@ -397,13 +496,13 @@ multiple machines with the Sentinel daemon. be duplicate below): ```ruby - redis_sentinel_role['enable'] = true + roles ['redis_sentinel_role'] # Must be the same in every sentinel node redis['master_name'] = 'gitlab-redis' # The same password for Redis authentication you set up for the master node. - redis['password'] = 'redis-password-goes-here' + redis['master_password'] = 'redis-password-goes-here' # The IP of the master Redis node. redis['master_ip'] = '10.0.0.1' @@ -570,8 +669,7 @@ or a failover promotes a different **Master** node. In `/etc/gitlab/gitlab.rb`: ```ruby -redis_master_role['enable'] = true -redis_sentinel_role['enable'] = true +roles ['redis_sentinel_role', 'redis_master_role'] redis['bind'] = '10.0.0.1' redis['port'] = 6379 redis['password'] = 'redis-password-goes-here' @@ -593,8 +691,7 @@ sentinel['quorum'] = 2 In `/etc/gitlab/gitlab.rb`: ```ruby -redis_slave_role['enable'] = true -redis_sentinel_role['enable'] = true +roles ['redis_sentinel_role', 'redis_slave_role'] redis['bind'] = '10.0.0.2' redis['port'] = 6379 redis['password'] = 'redis-password-goes-here' @@ -616,8 +713,7 @@ sentinel['quorum'] = 2 In `/etc/gitlab/gitlab.rb`: ```ruby -redis_slave_role['enable'] = true -redis_sentinel_role['enable'] = true +roles ['redis_sentinel_role', 'redis_slave_role'] redis['bind'] = '10.0.0.3' redis['port'] = 6379 redis['password'] = 'redis-password-goes-here' @@ -640,7 +736,7 @@ In `/etc/gitlab/gitlab.rb`: ```ruby redis['master_name'] = 'gitlab-redis' -redis['password'] = 'redis-password-goes-here' +redis['master_password'] = 'redis-password-goes-here' gitlab_rails['redis_sentinels'] = [ {'host' => '10.0.0.1', 'port' => 26379}, {'host' => '10.0.0.2', 'port' => 26379}, @@ -761,15 +857,11 @@ Before proceeding with the troubleshooting below, check your firewall rules: ### Troubleshooting Redis replication You can check if everything is correct by connecting to each server using -`redis-cli` application, and sending the `INFO` command. +`redis-cli` application, and sending the `info replication` command as below. -If authentication was correctly defined, it should fail with: -`NOAUTH Authentication required` error. Try to authenticate with the -previous defined password with `AUTH redis-password-goes-here` and -try the `INFO` command again. - -Look for the `# Replication` section where you should see some important -information like the `role` of the server. +``` +/opt/gitlab/embedded/bin/redis-cli -a <redis-password> info replication +``` When connected to a `master` redis, you will see the number of connected `slaves`, and a list of each with connection details: @@ -839,7 +931,7 @@ To make sure your configuration is correct: 1. Run in the console: ```ruby - redis = Redis.new(Gitlab::Redis.params) + redis = Redis.new(Gitlab::Redis::SharedState.params) redis.info ``` diff --git a/doc/administration/img/db_load_balancing_postgres_stats.png b/doc/administration/img/db_load_balancing_postgres_stats.png Binary files differnew file mode 100644 index 00000000000..8b311616e7b --- /dev/null +++ b/doc/administration/img/db_load_balancing_postgres_stats.png diff --git a/doc/administration/img/high_availability/fully-distributed.png b/doc/administration/img/high_availability/fully-distributed.png Binary files differnew file mode 100644 index 00000000000..ad23207134e --- /dev/null +++ b/doc/administration/img/high_availability/fully-distributed.png diff --git a/doc/administration/img/high_availability/geo-ha-diagram.png b/doc/administration/img/high_availability/geo-ha-diagram.png Binary files differnew file mode 100644 index 00000000000..da5d612827c --- /dev/null +++ b/doc/administration/img/high_availability/geo-ha-diagram.png diff --git a/doc/administration/img/high_availability/horizontal.png b/doc/administration/img/high_availability/horizontal.png Binary files differnew file mode 100644 index 00000000000..c3bd489d96f --- /dev/null +++ b/doc/administration/img/high_availability/horizontal.png diff --git a/doc/administration/img/high_availability/hybrid.png b/doc/administration/img/high_availability/hybrid.png Binary files differnew file mode 100644 index 00000000000..7d4a56bf0ea --- /dev/null +++ b/doc/administration/img/high_availability/hybrid.png diff --git a/doc/administration/img/instance_review_button.png b/doc/administration/img/instance_review_button.png Binary files differnew file mode 100644 index 00000000000..b7604d7c7e5 --- /dev/null +++ b/doc/administration/img/instance_review_button.png diff --git a/doc/administration/incoming_email.md b/doc/administration/incoming_email.md index 658b2f55d30..4de54e03c8c 100644 --- a/doc/administration/incoming_email.md +++ b/doc/administration/incoming_email.md @@ -10,6 +10,8 @@ GitLab has several features based on receiving incoming emails: - [New merge request by email](../user/project/merge_requests/index.md#create-new-merge-requests-by-email): allow GitLab users to create a new merge request by sending an email to a user-specific email address. +- [Service Desk](../user/project/service_desk.md): provide e-mail support to + your customers through GitLab. ## Requirements diff --git a/doc/administration/index.md b/doc/administration/index.md index b723edfc78f..d208d9a736a 100644 --- a/doc/administration/index.md +++ b/doc/administration/index.md @@ -32,8 +32,14 @@ Learn how to install, configure, update, and maintain your GitLab instance. ### Installing GitLab - [Install](../install/README.md): Requirements, directory structures, and installation methods. + - [Database load balancing](database_load_balancing.md): Distribute database queries among multiple database servers. **[STARTER ONLY]** + - [Omnibus support for external MySQL DB](https://docs.gitlab.com/omnibus/settings/database.html#using-a-mysql-database-management-server-enterprise-edition-only): Omnibus package supports configuring an external MySQL database. **[STARTER ONLY]** + - [Omnibus support for log forwarding](https://docs.gitlab.com/omnibus/settings/logs.html#udp-log-shipping-gitlab-enterprise-edition-only) **[STARTER ONLY]** - [High Availability](high_availability/README.md): Configure multiple servers for scaling or high availability. - [High Availability on AWS](../university/high-availability/aws/README.md): Set up GitLab HA on Amazon AWS. +- [Geo](geo/replication/index.md): Replicate your GitLab instance to other geographic locations as a read-only fully operational version. **[PREMIUM ONLY]** +- [Disaster Recovery](geo/disaster_recovery/index.md): Quickly fail-over to a different site with minimal effort in a disaster situation. **[PREMIUM ONLY]** +- [Pivotal Tile](../install/pivotal/index.md): Deploy GitLab as a pre-configured appliance using Ops Manager (BOSH) for Pivotal Cloud Foundry. **[PREMIUM ONLY]** ### Configuring GitLab @@ -43,8 +49,8 @@ Learn how to install, configure, update, and maintain your GitLab instance. - [Usage statistics, version check, and usage ping](../user/admin_area/settings/usage_statistics.md): Enable or disable information about your instance to be sent to GitLab, Inc. - [Polling](polling.md): Configure how often the GitLab UI polls for updates. - [GitLab Pages configuration](pages/index.md): Enable and configure GitLab Pages. -- [GitLab Pages configuration for GitLab source installations](pages/source.md): Enable and configure GitLab Pages on - [source installations](../install/installation.md#installation-from-source). +- [GitLab Pages configuration for GitLab source installations](pages/source.md): Enable and configure GitLab Pages on [source installations](../install/installation.md#installation-from-source). +- [Uploads configuration](uploads.md): Configure GitLab uploads storage. - [Environment variables](environment_variables.md): Supported environment variables that can be used to override their defaults values in order to configure GitLab. - [Plugins](plugins.md): With custom plugins, GitLab administrators can introduce custom integrations without modifying GitLab's source code. - [Enforcing Terms of Service](../user/admin_area/settings/terms.md) @@ -53,6 +59,9 @@ Learn how to install, configure, update, and maintain your GitLab instance. - [Diff limits](../user/admin_area/diff_limits.md): Configure the diff rendering size limits of branch comparison pages. - [Merge request diffs storage](merge_request_diffs.md): Configure merge requests diffs external storage. - [Broadcast Messages](../user/admin_area/broadcast_messages.md): Send messages to GitLab users through the UI. +- [Elasticsearch](../integration/elasticsearch.md): Enable Elasticsearch to empower GitLab's Advanced Global Search. Useful when you deal with a huge amount of data. **[STARTER ONLY]** +- [External Classification Policy Authorization](../user/admin_area/settings/external_authorization.md) **[PREMIUM ONLY]** +- [Upload a license](../user/admin_area/license.md): Upload a license to unlock features that are in paid tiers of GitLab. **[STARTER ONLY]** - [Admin Area](../user/admin_area/index.md): for self-managed instance-wide configuration and maintenance. #### Customizing GitLab's appearance @@ -62,6 +71,7 @@ Learn how to install, configure, update, and maintain your GitLab instance. - [Branded login page](../customization/branded_login_page.md): Customize the login page with your own logo, title, and description. - [Welcome message](../customization/welcome_message.md): Add a custom welcome message to the sign-in page. - ["New Project" page](../customization/new_project_page.md): Customize the text to be displayed on the page that opens whenever your users create a new project. +- [Additional custom email text](../user/admin_area/settings/email.md#custom-additional-text): Add additional custom text to emails sent from GitLab. **[PREMIUM ONLY]** ### Maintaining GitLab @@ -95,7 +105,16 @@ Learn how to install, configure, update, and maintain your GitLab instance. - [Libravatar](../customization/libravatar.md): Use Libravatar instead of Gravatar for user avatars. - [Sign-up restrictions](../user/admin_area/settings/sign_up_restrictions.md): block email addresses of specific domains, or whitelist only specific domains. - [Access restrictions](../user/admin_area/settings/visibility_and_access_controls.md#enabled-git-access-protocols): Define which Git access protocols can be used to talk to GitLab (SSH, HTTP, HTTPS). -- [Authentication and Authorization](auth/README.md): Configure external authentication with LDAP, SAML, CAS and additional providers. See also other [authentication](../topics/authentication/index.md#gitlab-administrators) topics (for example, enforcing 2FA). +- [Authentication and Authorization](auth/README.md): Configure external authentication with LDAP, SAML, CAS and additional providers. + - [Sync LDAP](auth/ldap-ee.md) **[STARTER ONLY]** + - [Kerberos authentication](../integration/kerberos.md) **[STARTER ONLY]** + - See also other [authentication](../topics/authentication/index.md#gitlab-administrators) topics (for example, enforcing 2FA). +- [Email users](../tools/email.md): Email GitLab users from within GitLab. **[STARTER ONLY]** +- [User Cohorts](../user/admin_area/user_cohorts.md): Display the monthly cohorts of new users and their activities over time. +- [Audit logs and events](audit_events.md): View the changes made within the GitLab server for: + - Groups and projects. **[STARTER]** + - Instances. **[PREMIUM ONLY]** +- [Auditor users](auditor_users.md): Users with read-only access to all projects, groups, and other resources on the GitLab instance. **[PREMIUM ONLY]** - [Incoming email](incoming_email.md): Configure incoming emails to allow users to [reply by email], create [issues by email] and [merge requests by email], and to enable [Service Desk]. @@ -107,6 +126,7 @@ Learn how to install, configure, update, and maintain your GitLab instance. [reply by email]: reply_by_email.md [issues by email]: ../user/project/issues/create_new_issue.md#new-issue-via-email [merge requests by email]: ../user/project/merge_requests/index.md#create-new-merge-requests-by-email +[Service Desk]: ../user/project/service_desk.md ## Project settings @@ -115,13 +135,15 @@ Learn how to install, configure, update, and maintain your GitLab instance. - [Gitaly](gitaly/index.md): Configuring Gitaly, GitLab's Git repository storage service. - [Default labels](../user/admin_area/labels.html): Create labels that will be automatically added to every new project. - [Restrict the use of public or internal projects](../public_access/public_access.md#restricting-the-use-of-public-or-internal-projects): Restrict the use of visibility levels for users when they create a project or a snippet. -- [Custom project templates](https://docs.gitlab.com/ee/user/admin_area/custom_project_templates.html): Configure a set of projects to be used as custom templates when creating a new project. **[PREMIUM ONLY]** +- [Custom project templates](../user/admin_area/custom_project_templates.md): Configure a set of projects to be used as custom templates when creating a new project. **[PREMIUM ONLY]** +- [Packages](packages.md): Enable GitLab to act as a Maven repository or NPM registry. **[PREMIUM ONLY]** ### Repository settings - [Repository checks](repository_checks.md): Periodic Git repository checks. - [Repository storage paths](repository_storage_paths.md): Manage the paths used to store repositories. - [Repository storage rake tasks](raketasks/storage.md): A collection of rake tasks to list and migrate existing projects and attachments associated with it from Legacy storage to Hashed storage. +- [Limit repository size](../user/admin_area/settings/account_and_limit_settings.md): Set a hard limit for your repositories' size. **[STARTER ONLY]** ## Continuous Integration settings @@ -158,6 +180,10 @@ Learn how to install, configure, update, and maintain your GitLab instance. - [Request Profiling](monitoring/performance/request_profiling.md): Get a detailed profile on slow requests. - [Performance Bar](monitoring/performance/performance_bar.md): Get performance information for the current page. +## Analytics + +- [Pseudonymizer](pseudonymizer.md): Export data from GitLab's database to CSV files in a secure way. + ## Troubleshooting - [Debugging tips](troubleshooting/debug.md): Tips to debug problems when things go wrong diff --git a/doc/administration/instance_review.md b/doc/administration/instance_review.md new file mode 100644 index 00000000000..5781ce4150c --- /dev/null +++ b/doc/administration/instance_review.md @@ -0,0 +1,17 @@ +# Instance Review + +> [Introduced][6995] in [GitLab Core][ee] 11.3. + +If you are running a medium size instance of GitLab Core edition you are qualified for a free Instance Review. You can find the button in the User menu. + + + +When you click the button you will be redirected to a form with prefilled data obtained from your instance. + +Once you submit the data to GitLab Inc. you can see the initial report. + +Additionally you will be contacted by our team for further review which should help you to improve your usage of GitLab. + +[6995]: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/6995 +[ee]: https://about.gitlab.com/pricing/ + diff --git a/doc/administration/maven_packages.md b/doc/administration/maven_packages.md new file mode 100644 index 00000000000..d8551f64ece --- /dev/null +++ b/doc/administration/maven_packages.md @@ -0,0 +1,5 @@ +--- +redirect_to: 'packages.md' +--- + +This document was moved to [another location](packages.md). diff --git a/doc/administration/maven_repository.md b/doc/administration/maven_repository.md new file mode 100644 index 00000000000..d8551f64ece --- /dev/null +++ b/doc/administration/maven_repository.md @@ -0,0 +1,5 @@ +--- +redirect_to: 'packages.md' +--- + +This document was moved to [another location](packages.md). diff --git a/doc/administration/monitoring/performance/img/request_profiling_token.png b/doc/administration/monitoring/performance/img/request_profiling_token.png Binary files differindex 9f3dd7f08ca..ee819fcb437 100644 --- a/doc/administration/monitoring/performance/img/request_profiling_token.png +++ b/doc/administration/monitoring/performance/img/request_profiling_token.png diff --git a/doc/administration/monitoring/prometheus/gitlab_metrics.md b/doc/administration/monitoring/prometheus/gitlab_metrics.md index 3bfcc9a289e..da7094338a0 100644 --- a/doc/administration/monitoring/prometheus/gitlab_metrics.md +++ b/doc/administration/monitoring/prometheus/gitlab_metrics.md @@ -48,6 +48,47 @@ The following metrics are available: | unicorn_active_connections | Gauge | 11.0 | The number of active Unicorn connections (workers) | | unicorn_queued_connections | Gauge | 11.0 | The number of queued Unicorn connections | +## Sidekiq Metrics available + +Sidekiq jobs may also gather metrics, and these metrics can be accessed if the Sidekiq exporter is enabled (e.g. via +the `monitoring.sidekiq_exporter` configuration option in `gitlab.yml`. + +| Metric | Type | Since | Description | Labels | +|:-------------------------------------------- |:------- |:----- |:----------- |:------ | +| geo_db_replication_lag_seconds | Gauge | 10.2 | Database replication lag (seconds) | url +| geo_repositories | Gauge | 10.2 | Total number of repositories available on primary | url +| geo_repositories_synced | Gauge | 10.2 | Number of repositories synced on secondary | url +| geo_repositories_failed | Gauge | 10.2 | Number of repositories failed to sync on secondary | url +| geo_lfs_objects | Gauge | 10.2 | Total number of LFS objects available on primary | url +| geo_lfs_objects_synced | Gauge | 10.2 | Number of LFS objects synced on secondary | url +| geo_lfs_objects_failed | Gauge | 10.2 | Number of LFS objects failed to sync on secondary | url +| geo_attachments | Gauge | 10.2 | Total number of file attachments available on primary | url +| geo_attachments_synced | Gauge | 10.2 | Number of attachments synced on secondary | url +| geo_attachments_failed | Gauge | 10.2 | Number of attachments failed to sync on secondary | url +| geo_last_event_id | Gauge | 10.2 | Database ID of the latest event log entry on the primary | url +| geo_last_event_timestamp | Gauge | 10.2 | UNIX timestamp of the latest event log entry on the primary | url +| geo_cursor_last_event_id | Gauge | 10.2 | Last database ID of the event log processed by the secondary | url +| geo_cursor_last_event_timestamp | Gauge | 10.2 | Last UNIX timestamp of the event log processed by the secondary | url +| geo_status_failed_total | Counter | 10.2 | Number of times retrieving the status from the Geo Node failed | url +| geo_last_successful_status_check_timestamp | Gauge | 10.2 | Last timestamp when the status was successfully updated | url +| geo_lfs_objects_synced_missing_on_primary | Gauge | 10.7 | Number of LFS objects marked as synced due to the file missing on the primary | url +| geo_job_artifacts_synced_missing_on_primary | Gauge | 10.7 | Number of job artifacts marked as synced due to the file missing on the primary | url +| geo_attachments_synced_missing_on_primary | Gauge | 10.7 | Number of attachments marked as synced due to the file missing on the primary | url +| geo_repositories_checksummed_count | Gauge | 10.7 | Number of repositories checksummed on primary | url +| geo_repositories_checksum_failed_count | Gauge | 10.7 | Number of repositories failed to calculate the checksum on primary | url +| geo_wikis_checksummed_count | Gauge | 10.7 | Number of wikis checksummed on primary | url +| geo_wikis_checksum_failed_count | Gauge | 10.7 | Number of wikis failed to calculate the checksum on primary | url +| geo_repositories_verified_count | Gauge | 10.7 | Number of repositories verified on secondary | url +| geo_repositories_verification_failed_count | Gauge | 10.7 | Number of repositories failed to verify on secondary | url +| geo_repositories_checksum_mismatch_count | Gauge | 10.7 | Number of repositories that checksum mismatch on secondary | url +| geo_wikis_verified_count | Gauge | 10.7 | Number of wikis verified on secondary | url +| geo_wikis_verification_failed_count | Gauge | 10.7 | Number of wikis failed to verify on secondary | url +| geo_wikis_checksum_mismatch_count | Gauge | 10.7 | Number of wikis that checksum mismatch on secondary | url +| geo_repositories_checked_count | Gauge | 11.1 | Number of repositories that have been checked via `git fsck` | url +| geo_repositories_checked_failed_count | Gauge | 11.1 | Number of repositories that have a failure from `git fsck` | url +| geo_repositories_retrying_verification_count | Gauge | 11.2 | Number of repositories verification failures that Geo is actively trying to correct on secondary | url +| geo_wikis_retrying_verification_count | Gauge | 11.2 | Number of wikis verification failures that Geo is actively trying to correct on secondary | url + ### Ruby metrics Some basic Ruby runtime metrics are available: diff --git a/doc/administration/monitoring/prometheus/index.md b/doc/administration/monitoring/prometheus/index.md index 20d7ef9bb74..095f126f4b2 100644 --- a/doc/administration/monitoring/prometheus/index.md +++ b/doc/administration/monitoring/prometheus/index.md @@ -202,6 +202,12 @@ The Postgres exporter allows you to measure various PostgreSQL metrics. [➔ Read more about the Postgres exporter.](postgres_exporter.md) +### PgBouncer exporter + +The PgBouncer exporter allows you to measure various PgBouncer metrics. + +[➔ Read more about the PgBouncer exporter.](pgbouncer_exporter.md) + ### GitLab monitor exporter The GitLab monitor exporter allows you to measure various GitLab metrics, pulled from Redis and the database. diff --git a/doc/administration/monitoring/prometheus/pgbouncer_exporter.md b/doc/administration/monitoring/prometheus/pgbouncer_exporter.md new file mode 100644 index 00000000000..d76834fdbea --- /dev/null +++ b/doc/administration/monitoring/prometheus/pgbouncer_exporter.md @@ -0,0 +1,34 @@ +# PgBouncer exporter + +>**Note:** +Available since [Omnibus GitLab 11.0][2493]. For installations from source +you'll have to install and configure it yourself. + +The [PgBouncer exporter] allows you to measure various PgBouncer metrics. + +To enable the PgBouncer exporter: + +1. [Enable Prometheus](index.md#configuring-prometheus) +1. Edit `/etc/gitlab/gitlab.rb` +1. Add or find and uncomment the following line, making sure it's set to `true`: + + ```ruby + pgbouncer_exporter['enable'] = true + ``` + +1. Save the file and [reconfigure GitLab][reconfigure] for the changes to + take effect. + +Prometheus will now automatically begin collecting performance data from +the PgBouncer exporter exposed under `localhost:9188`. + +The PgBouncer exporter will also be enabled by default if the [pgbouncer_role][postgres roles] +is enabled. + +[← Back to the main Prometheus page](index.md) + +[2493]: https://gitlab.com/gitlab-org/omnibus-gitlab/merge_requests/2493 +[PgBouncer exporter]: https://github.com/stanhu/pgbouncer_exporter +[postgres roles]: https://docs.gitlab.com/omnibus/roles/#postgres-roles +[prometheus]: https://prometheus.io +[reconfigure]: ../../restart_gitlab.md#omnibus-gitlab-reconfigure diff --git a/doc/administration/npm_registry.md b/doc/administration/npm_registry.md new file mode 100644 index 00000000000..d8551f64ece --- /dev/null +++ b/doc/administration/npm_registry.md @@ -0,0 +1,5 @@ +--- +redirect_to: 'packages.md' +--- + +This document was moved to [another location](packages.md). diff --git a/doc/administration/operations/cleaning_up_redis_sessions.md b/doc/administration/operations/cleaning_up_redis_sessions.md index b45ca99fd80..20c19445404 100644 --- a/doc/administration/operations/cleaning_up_redis_sessions.md +++ b/doc/administration/operations/cleaning_up_redis_sessions.md @@ -27,7 +27,7 @@ rcli() { # This example works for Omnibus installations of GitLab 7.3 or newer. For an # installation from source you will have to change the socket path and the # path to redis-cli. - sudo /opt/gitlab/embedded/bin/redis-cli -s /var/opt/gitlab/redis/redis.socket "$@" + sudo /opt/gitlab/embedded/bin/redis-cli -s /var/opt/gitlab/redis/redis.shared_state.socket "$@" } # test the new shell function; the response should be PONG diff --git a/doc/administration/operations/extra_sidekiq_processes.md b/doc/administration/operations/extra_sidekiq_processes.md new file mode 100644 index 00000000000..ee7c474bec5 --- /dev/null +++ b/doc/administration/operations/extra_sidekiq_processes.md @@ -0,0 +1,130 @@ +# Extra Sidekiq Processes + +GitLab Enterprise Edition allows one to start an extra set of Sidekiq processes +besides the default one. These processes can be used to consume a dedicated set +of queues. This can be used to ensure certain queues always have dedicated +workers, no matter the amount of jobs that need to be processed. + +## Starting Extra Processes + +Starting extra Sidekiq processes can be done using the command +`bin/sidekiq-cluster`. This command takes arguments using the following syntax: + +```bash +sidekiq-cluster [QUEUE,QUEUE,...] [QUEUE, ...] +``` + +Each separate argument denotes a group of queues that have to be processed by a +Sidekiq process. Multiple queues can be processed by the same process by +separating them with a comma instead of a space. + +Instead of a queue, a queue namespace can also be provided, to have the process +automatically listen on all queues in that namespace without needing to +explicitly list all the queue names. For more information about queue namespaces, +see the relevant section in the +[Sidekiq style guide](../../development/sidekiq_style_guide.md#queue-namespaces). + +For example, say you want to start 2 extra processes: one to process the +"process_commit" queue, and one to process the "post_receive" queue. This can be +done as follows: + +```bash +sidekiq-cluster process_commit post_receive +``` + +If you instead want to start one process processing both queues you'd use the +following syntax: + +```bash +sidekiq-cluster process_commit,post_receive +``` + +If you want to have one Sidekiq process process the "process_commit" and +"post_receive" queues, and one process to process the "gitlab_shell" queue, +you'd use the following: + +```bash +sidekiq-cluster process_commit,post_receive gitlab_shell +``` + +## Concurrency + +Each process started using `sidekiq-cluster` starts with a number of threads +that equals the number of queues, plus one spare thread. For example, a process +that processes "process_commit" and "post_receive" will use 3 threads in total. + +## Monitoring + +The `sidekiq-cluster` command will not terminate once it has started the desired +amount of Sidekiq processes. Instead the process will continue running and +forward any signals to the child processes. This makes it easy to stop all +Sidekiq processes as you simply send a signal to the `sidekiq-cluster` process, +instead of having to send it to the individual processes. + +If the `sidekiq-cluster` process crashes or is SIGKILL'd the child processes +will terminate themselves after a few seconds. This ensures you don't end up +with zombie Sidekiq processes. + +All of this makes monitoring the processes fairly easy. Simply hook up +`sidekiq-cluster` to your supervisor of choice (e.g. runit) and you're good to +go. + +If a child process died the `sidekiq-cluster` command will signal all remaining +process to terminate, then terminate itself. This removes the need for +`sidekiq-cluster` to re-implement complex process monitoring/restarting code. +Instead you should make sure your supervisor restarts the `sidekiq-cluster` +process whenever necessary. + +## PID Files + +The `sidekiq-cluster` command can store its PID in a file. By default no PID +file is written, but this can be changed by passing the `--pidfile` option to +`sidekiq-cluster`. For example: + +```bash +sidekiq-cluster --pidfile /var/run/gitlab/sidekiq_cluster.pid process_commit +``` + +Keep in mind that the PID file will contain the PID of the `sidekiq-cluster` +command, and not the PID(s) of the started Sidekiq processes. + +## Environment + +The Rails environment can be set by passing the `--environment` flag to the +`sidekiq-cluster` command, or by setting `RAILS_ENV` to a non-empty value. The +default value is "development". + +## All Queues With Exceptions + +You're able to run all queues in `sidekiq_queues.yml` file on a single or +multiple processes with exceptions using the `--negate` flag. + +For example, say you want to run a single process for all queues, +except "process_commit" and "post_receive". You can do so by executing: + +```bash +sidekiq-cluster process_commit,post_receive --negate +``` + +For multiple processes of all queues (except "process_commit" and "post_receive"): + +```bash +sidekiq-cluster process_commit,post_receive process_commit,post_receive --negate +``` + +## Limiting Concurrency + +By default, `sidekiq-cluster` will spin up extra Sidekiq processes that use +one thread per queue up to a maximum of 50. If you wish to change the cap, use +the `-m N` option. For example, this would cap the maximum number of threads to 1: + +```bash +sidekiq-cluster process_commit,post_receive -m 1 +``` + +For each queue group, the concurrency factor will be set to min(number of +queues, N). Setting the value to 0 will disable the limit. + +Note that each thread requires a Redis connection, so adding threads may +increase Redis latency and potentially cause client timeouts. See the [Sidekiq +documentation about Redis](https://github.com/mperham/sidekiq/wiki/Using-Redis) for more details. diff --git a/doc/administration/operations/fast_ssh_key_lookup.md b/doc/administration/operations/fast_ssh_key_lookup.md index c293df3fc57..6ba5768ebfd 100644 --- a/doc/administration/operations/fast_ssh_key_lookup.md +++ b/doc/administration/operations/fast_ssh_key_lookup.md @@ -30,6 +30,19 @@ instructions will break installations using older versions of OpenSSH, such as those included with CentOS 6 as of September 2017. If you want to use this feature for CentOS 6, follow [the instructions on how to build and install a custom OpenSSH package](#compiling-a-custom-version-of-openssh-for-centos-6) before continuing. +## Fast lookup is required for Geo + +By default, GitLab manages an `authorized_keys` file, which contains all the +public SSH keys for users allowed to access GitLab. However, to maintain a +single source of truth, [Geo](../../gitlab-geo/README.md) needs to be configured to perform SSH fingerprint +lookups via database lookup. + +As part of [setting up Geo](../geo/replication/index.md#setup-instructions), +you will be required to follow the steps outlined below for both the primary and +secondary nodes, but note that the `Write to "authorized keys" file` checkbox +only needs to be unchecked on the primary node since it will be reflected +automatically on the secondary if database replication is working. + ## Setting up fast lookup via GitLab Shell GitLab Shell provides a way to authorize SSH users via a fast, indexed lookup diff --git a/doc/administration/operations/index.md b/doc/administration/operations/index.md index 32f36d68c50..df795a48169 100644 --- a/doc/administration/operations/index.md +++ b/doc/administration/operations/index.md @@ -11,6 +11,7 @@ Keep your GitLab instance up and running smoothly. by GitLab to another file system or another server. - [Sidekiq MemoryKiller](sidekiq_memory_killer.md): Configure Sidekiq MemoryKiller to restart Sidekiq. +- [Extra Sidekiq operations](extra_sidekiq_processes.md): Configure an extra set of Sidekiq processes to ensure certain queues always have dedicated workers, no matter the amount of jobs that need to be processed. **[STARTER ONLY]** - [Unicorn](unicorn.md): Understand Unicorn and unicorn-worker-killer. - Speed up SSH operations by [Authorizing SSH users via a fast, indexed lookup to the GitLab database](fast_ssh_key_lookup.md), and/or diff --git a/doc/administration/packages.md b/doc/administration/packages.md new file mode 100644 index 00000000000..4d60c0c7638 --- /dev/null +++ b/doc/administration/packages.md @@ -0,0 +1,174 @@ +# GitLab Packages administration **[PREMIUM ONLY]** + +GitLab Packages allows organizations to utilize GitLab as a private repository +for a variety of common package managers. Users are able to build and publish +packages, which can be easily consumed as a dependency in downstream projects. + +The Packages feature allows GitLab to act as a repository for the following: + +| Software repository | Description | Available in GitLab version | +| ------------------- | ----------- | --------------------------- | +| [Maven Repository](../user/project/packages/maven_repository.md) | The GitLab Maven Repository enables every project in GitLab to have its own space to store [Maven](https://maven.apache.org/) packages. | 11.3+ | +| [NPM Registry](../user/project/packages/npm_registry.md) | The GitLab NPM Registry enables every project in GitLab to have its own space to store [NPM](https://www.npmjs.com/) packages. | 11.7+ | + +Don't you see your package management system supported yet? +Please consider contributing +to GitLab. This [development documentation](../development/packages.md) will guide you through the process. + +## Enabling the Packages feature + +NOTE: **Note:** +After the Packages feature is enabled, the repositories are available for +for all new projects by default. To enable it for existing projects, users will +have to explicitly do so in the project's settings. + +To enable the Packages feature: + +**Omnibus GitLab installations** + +1. Edit `/etc/gitlab/gitlab.rb` and add the following line: + + ```ruby + gitlab_rails['packages_enabled'] = true + ``` + +1. Save the file and [reconfigure GitLab][] for the changes to take effect. + +**Installations from source** + +1. After the installation is complete, you will have to configure the `packages` + section in `config/gitlab.yml`. Set to `true` to enable it: + + ```yaml + packages: + enabled: true + ``` +1. [Restart GitLab] for the changes to take effect. + +## Changing the storage path + +By default, the packages are stored locally, but you can change the default +local location or even use object storage. + +### Changing the local storage path + +The packages for Omnibus GitLab installations are stored under +`/var/opt/gitlab/gitlab-rails/shared/packages/` and for source +installations under `shared/packages/` (relative to the git homedir). +To change the local storage path: + +**Omnibus GitLab installations** + +1. Edit `/etc/gitlab/gitlab.rb` and add the following line: + + ```ruby + gitlab_rails['packages_storage_path'] = "/mnt/packages" + ``` + +1. Save the file and [reconfigure GitLab][] for the changes to take effect. + +**Installations from source** + +1. Edit the `packages` section in `config/gitlab.yml`: + + ```yaml + packages: + enabled: true + storage_path: shared/packages + ``` +1. [Restart GitLab] for the changes to take effect. + +### Using object storage + +Instead of relying on the local storage, you can use an object storage to +upload packages: + +**Omnibus GitLab installations** + +1. Edit `/etc/gitlab/gitlab.rb` and add the following lines (uncomment where + necessary): + + ```ruby + gitlab_rails['packages_enabled'] = true + gitlab_rails['packages_storage_path'] = "/var/opt/gitlab/gitlab-rails/shared/packages" + gitlab_rails['packages_object_store_enabled'] = true + gitlab_rails['packages_object_store_remote_directory'] = "packages" # The bucket name. + gitlab_rails['packages_object_store_direct_upload'] = false # Use Object Storage directly for uploads instead of background uploads if enabled (Default: false). + gitlab_rails['packages_object_store_background_upload'] = true # Temporary option to limit automatic upload (Default: true). + gitlab_rails['packages_object_store_proxy_download'] = false # Passthrough all downloads via GitLab instead of using Redirects to Object Storage. + gitlab_rails['packages_object_store_connection'] = { + ## + ## If the provider is AWS S3, uncomment the following + ## + #'provider' => 'AWS', + #'region' => 'eu-west-1', + #'aws_access_key_id' => 'AWS_ACCESS_KEY_ID', + #'aws_secret_access_key' => 'AWS_SECRET_ACCESS_KEY', + ## + ## If the provider is other than AWS (an S3-compatible one), uncomment the following + ## + #'host' => 's3.amazonaws.com', + #'aws_signature_version' => 4 # For creation of signed URLs. Set to 2 if provider does not support v4. + #'endpoint' => 'https://s3.amazonaws.com' # Useful for S3-compliant services such as DigitalOcean Spaces. + #'path_style' => false # If true, use 'host/bucket_name/object' instead of 'bucket_name.host/object'. + } + ``` + +1. Save the file and [reconfigure GitLab][] for the changes to take effect. + +**Installations from source** + +1. Edit the `packages` section in `config/gitlab.yml` (uncomment where necessary): + + ```yaml + packages: + enabled: true + ## + ## The location where build packages are stored (default: shared/packages). + ## + #storage_path: shared/packages + object_store: + enabled: false + remote_directory: packages # The bucket name. + #direct_upload: false # Use Object Storage directly for uploads instead of background uploads if enabled (Default: false). + #background_upload: true # Temporary option to limit automatic upload (Default: true). + #proxy_download: false # Passthrough all downloads via GitLab instead of using Redirects to Object Storage. + connection: + ## + ## If the provider is AWS S3, uncomment the following + ## + #provider: AWS + #region: us-east-1 + #aws_access_key_id: AWS_ACCESS_KEY_ID + #aws_secret_access_key: AWS_SECRET_ACCESS_KEY + ## + ## If the provider is other than AWS (an S3-compatible one), uncomment the following + ## + #host: 's3.amazonaws.com' # default: s3.amazonaws.com. + #aws_signature_version: 4 # For creation of signed URLs. Set to 2 if provider does not support v4. + #endpoint: 'https://s3.amazonaws.com' # Useful for S3-compliant services such as DigitalOcean Spaces. + #path_style: false # If true, use 'host/bucket_name/object' instead of 'bucket_name.host/object'. + ``` + +1. [Restart GitLab] for the changes to take effect. + +### Migrating local packages to object storage + +After [configuring the object storage](#using-object-storage), you may use the +following task to migrate existing packages from the local storage to the remote one. +The processing will be done in a background worker and requires **no downtime**. + +For Omnibus GitLab: + +```sh +sudo gitlab-rake "gitlab:packages:migrate" +``` + +For installations from source: + +```bash +RAILS_ENV=production sudo -u git -H bundle exec rake gitlab:packages:migrate +``` + +[reconfigure gitlab]: restart_gitlab.md#omnibus-gitlab-reconfigure "How to reconfigure Omnibus GitLab" +[restart gitlab]: restart_gitlab.md#omnibus-gitlab-reconfigure "How to reconfigure Omnibus GitLab" diff --git a/doc/administration/pseudonymizer.md b/doc/administration/pseudonymizer.md new file mode 100644 index 00000000000..0ad937052bd --- /dev/null +++ b/doc/administration/pseudonymizer.md @@ -0,0 +1,103 @@ +# Pseudonymizer + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/5532) in [GitLab Ultimate][ee] 11.1. + +As GitLab's database hosts sensitive information, using it unfiltered for analytics +implies high security requirements. To help alleviate this constraint, the Pseudonymizer +service is used to export GitLab's data in a pseudonymized way. + +CAUTION: **Warning:** +This process is not impervious. If the source data is available, it's possible for +a user to correlate data to the pseudonymized version. + +The Pseudonymizer currently uses `HMAC(SHA256)` to mutate fields that shouldn't +be textually exported. This ensures that: + +- the end-user of the data source cannot infer/revert the pseudonymized fields +- the referential integrity is maintained + +## Configuration + +To configure the pseudonymizer, you need to: + +- Provide a manifest file that describes which fields should be included or + pseudonymized ([example `manifest.yml` file](https://gitlab.com/gitlab-org/gitlab-ee/tree/master/config/pseudonymizer.yml)). + A default manifest is provided with the GitLab installation. Using a relative file path will be resolved from the Rails root. + Alternatively, you can use an absolute file path. +- Use an object storage and specify the connection parameters in the `pseudonymizer.upload.connection` configuration option. + +**For Omnibus installations:** + +1. Edit `/etc/gitlab/gitlab.rb` and add the following lines by replacing with + the values you want: + + ```ruby + gitlab_rails['pseudonymizer_manifest'] = 'config/pseudonymizer.yml' + gitlab_rails['pseudonymizer_upload_remote_directory'] = 'gitlab-elt' # bucket name + gitlab_rails['pseudonymizer_upload_connection'] = { + 'provider' => 'AWS', + 'region' => 'eu-central-1', + 'aws_access_key_id' => 'AWS_ACCESS_KEY_ID', + 'aws_secret_access_key' => 'AWS_SECRET_ACCESS_KEY' + } + ``` + + NOTE: **Note:** + If you are using AWS IAM profiles, be sure to omit the AWS access key and secret access key/value pairs. + + ```ruby + gitlab_rails['pseudonymizer_upload_connection'] = { + 'provider' => 'AWS', + 'region' => 'eu-central-1', + 'use_iam_profile' => true + } + ``` + +1. Save the file and [reconfigure GitLab](restart_gitlab.md#omnibus-gitlab-reconfigure) + for the changes to take effect. + +--- + +**For installations from source:** + +1. Edit `/home/git/gitlab/config/gitlab.yml` and add or amend the following + lines: + + ```yaml + pseudonymizer: + manifest: config/pseudonymizer.yml + upload: + remote_directory: 'gitlab-elt' # bucket name + connection: + provider: AWS + aws_access_key_id: AWS_ACCESS_KEY_ID + aws_secret_access_key: AWS_SECRET_ACCESS_KEY + region: eu-central-1 + ``` + +1. Save the file and [restart GitLab](restart_gitlab.md#installations-from-source) + for the changes to take effect. + +## Usage + +You can optionally run the pseudonymizer using the following environment variables: + +- `PSEUDONYMIZER_OUTPUT_DIR` - where to store the output CSV files (defaults to `/tmp`) +- `PSEUDONYMIZER_BATCH` - the batch size when querying the DB (defaults to `100000`) + +```bash +## Omnibus +sudo gitlab-rake gitlab:db:pseudonymizer + +## Source +sudo -u git -H bundle exec rake gitlab:db:pseudonymizer RAILS_ENV=production +``` + +This will produce some CSV files that might be very large, so make sure the +`PSEUDONYMIZER_OUTPUT_DIR` has sufficient space. As a rule of thumb, at least +10% of the database size is recommended. + +After the pseudonymizer has run, the output CSV files should be uploaded to the +configured object storage and deleted from the local disk. + +[ee]: https://about.gitlab.com/pricing/ diff --git a/doc/administration/raketasks/geo.md b/doc/administration/raketasks/geo.md new file mode 100644 index 00000000000..60bec0fd868 --- /dev/null +++ b/doc/administration/raketasks/geo.md @@ -0,0 +1,57 @@ +# Geo Rake Tasks + +## Git housekeeping + +There are few tasks you can run to schedule a git housekeeping to start at the +next repository sync in a **Secondary node**: + +### Incremental Repack + +This is equivalent of running `git repack -d` on a _bare_ repository. + +**Omnibus Installation** + +``` +sudo gitlab-rake geo:git:housekeeping:incremental_repack +``` + +**Source Installation** + +```bash +sudo -u git -H bundle exec rake geo:git:housekeeping:incremental_repack RAILS_ENV=production +``` + +### Full Repack + +This is equivalent of running `git repack -d -A --pack-kept-objects` on a +_bare_ repository which will optionally, write a reachability bitmap index +when this is enabled in GitLab. + +**Omnibus Installation** + +``` +sudo gitlab-rake geo:git:housekeeping:full_repack +``` + +**Source Installation** + +```bash +sudo -u git -H bundle exec rake geo:git:housekeeping:full_repack RAILS_ENV=production +``` + +### GC + +This is equivalent of running `git gc` on a _bare_ repository, optionally writing +a reachability bitmap index when this is enabled in GitLab. + +**Omnibus Installation** + +``` +sudo gitlab-rake geo:git:housekeeping:gc +``` + +**Source Installation** + +```bash +sudo -u git -H bundle exec rake geo:git:housekeeping:gc RAILS_ENV=production +``` diff --git a/doc/administration/raketasks/storage.md b/doc/administration/raketasks/storage.md index 7ad38abe4f5..d0e6540d067 100644 --- a/doc/administration/raketasks/storage.md +++ b/doc/administration/raketasks/storage.md @@ -42,6 +42,9 @@ If you find it necessary, you can run this migration script again to schedule mi Any error or warning will be logged in the sidekiq's log file. +NOTE: **Note:** +If Geo is enabled, each project that is successfully migrated generates an event to replicate the changes on any **secondary** nodes. + You only need the `gitlab:storage:migrate_to_hashed` rake task to migrate your repositories, but we have additional commands below that helps you inspect projects and attachments in both legacy and hashed storage. diff --git a/doc/administration/repository_storage_types.md b/doc/administration/repository_storage_types.md index 4934aaf39f7..7249dbf5897 100644 --- a/doc/administration/repository_storage_types.md +++ b/doc/administration/repository_storage_types.md @@ -29,15 +29,15 @@ Any change in the URL will need to be reflected on disk (when groups / users or projects are renamed). This can add a lot of load in big installations, especially if using any type of network based filesystem. -For GitLab Geo in particular: Geo does work with legacy storage, but in some +CAUTION: **Caution:** +For Geo in particular: Geo does work with legacy storage, but in some edge cases due to race conditions it can lead to errors when a project is renamed multiple times in short succession, or a project is deleted and recreated under the same name very quickly. We expect these race events to be rare, and we have not observed a race condition side-effect happening yet. - This pattern also exists in other objects stored in GitLab, like issue Attachments, GitLab Pages artifacts, Docker Containers for the integrated -Registry, etc. +Registry, etc. Hashed storage is a requirement for Geo. ## Hashed Storage @@ -87,7 +87,7 @@ The rollback has to be performed in the reverse order. To get into "Legacy" stat you need to rollback Attachments first, then Project. Also note that if Geo is enabled, after the migration was triggered, an event is generated -to replicate the operation on any Secondary node. That means the on disk changes will also +to replicate the operation on any **secondary** node. That means the [on disk changes](#project) will need to be performed on these nodes as well. Database changes will propagate without issues. You must make sure the migration event was already processed or otherwise it may migrate diff --git a/doc/analytics/README.md b/doc/analytics/README.md new file mode 100644 index 00000000000..bfb15f6c4f3 --- /dev/null +++ b/doc/analytics/README.md @@ -0,0 +1,5 @@ +--- +redirect_to: '../user/group/index.md#user-contribution-analysis-starter' +--- + +This document was moved to [another location](../user/group/index.md#user-contribution-analysis-starter) diff --git a/doc/analytics/contribution_analytics.md b/doc/analytics/contribution_analytics.md new file mode 100644 index 00000000000..e36f55071a4 --- /dev/null +++ b/doc/analytics/contribution_analytics.md @@ -0,0 +1,5 @@ +--- +redirect_to: '../user/group/contribution_analytics/index.md' +--- + +This document was moved to [another location](../user/group/contribution_analytics/index.md). diff --git a/doc/api/README.md b/doc/api/README.md index 7ec7955c596..37e44a5fb97 100644 --- a/doc/api/README.md +++ b/doc/api/README.md @@ -21,72 +21,82 @@ See also: The following API resources are available in the project context: -| Resource | Available endpoints | -|:------------------------------------------------------|:--------------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| [Access requests](access_requests.md) | `/projects/:id/access_requests` (also available for groups) | -| [Award emoji](award_emoji.md) | `/projects/:id/issues/.../award_emoji`, `/projects/:id/merge_requests/.../award_emoji`, `/projects/:id/snippets/.../award_emoji` | -| [Branches](branches.md) | `/projects/:id/repository/branches/`, `/projects/:id/repository/merged_branches` | -| [Commits](commits.md) | `/projects/:id/repository/commits`, `/projects/:id/statuses` | -| [Container Registry](container_registry.md) | `/projects/:id/registry/repositories` | -| [Custom attributes](custom_attributes.md) | `/projects/:id/custom_attributes` (also available for groups and users) | -| [Deploy keys](deploy_keys.md) | `/projects/:id/deploy_keys` (also available standalone) | -| [Deployments](deployments.md) | `/projects/:id/deployments` | -| [Discussions](discussions.md) (threaded comments) | `/projects/:id/issues/.../discussions`, `/projects/:id/snippets/.../discussions`, `/projects/:id/merge_requests/.../discussions`, `/projects/:id/commits/.../discussions` | -| [Environments](environments.md) | `/projects/:id/environments` | -| [Events](events.md) | `/projects/:id/events` (also available for users and standalone) | -| [Issues](issues.md) | `/projects/:id/issues` (also available for groups and standalone) | -| [Issue boards](boards.md) | `/projects/:id/boards` | -| [Jobs](jobs.md) | `/projects/:id/jobs`, `/projects/:id/pipelines/.../jobs` | -| [Labels](labels.md) | `/projects/:id/labels` | -| [Members](members.md) | `/projects/:id/members` (also available for groups) | -| [Merge requests](merge_requests.md) | `/projects/:id/merge_requests` (also available for groups and standalone) | -| [Notes](notes.md) (comments) | `/projects/:id/issues/.../notes`, `/projects/:id/snippets/.../notes`, `/projects/:id/merge_requests/.../notes` | -| [Notification settings](notification_settings.md) | `/projects/:id/notification_settings` (also available for groups and standalone) | -| [Pages domains](pages_domains.md) | `/projects/:id/pages` (also available standalone) | -| [Pipelines](pipelines.md) | `/projects/:id/pipelines` | -| [Pipeline schedules](pipeline_schedules.md) | `/projects/:id/pipeline_schedules` | -| [Pipeline triggers](pipeline_triggers.md) | `/projects/:id/triggers` | -| [Projects](projects.md) including setting Webhooks | `/projects`, `/projects/:id/hooks` (also available for users) | -| [Project badges](project_badges.md) | `/projects/:id/badges` | -| [Project clusters](project_clusters.md) | `/projects/:id/clusters` | -| [Project-level variables](project_level_variables.md) | `/projects/:id/variables` | -| [Project import/export](project_import_export.md) | `/projects/:id/export`, `/projects/import`, `/projects/:id/import` | -| [Project milestones](milestones.md) | `/projects/:id/milestones` | -| [Project snippets](project_snippets.md) | `/projects/:id/snippets` | -| [Project templates](project_templates.md) | `/projects/:id/templates` | -| [Protected branches](protected_branches.md) | `/projects/:id/protected_branches` | -| [Protected tags](protected_tags.md) | `/projects/:id/protected_tags` | -| [Releases](releases/index.md) | `/projects/:id/releases` | -| [Release links](releases/links.md) | `/projects/:id/releases/.../assets/links` | -| [Repositories](repositories.md) | `/projects/:id/repository` | -| [Repository files](repository_files.md) | `/projects/:id/repository/files` | -| [Repository submodules](repository_submodules.md) | `/projects/:id/repository/submodules` | -| [Resource label events](resource_label_events.md) | `/projects/:id/issues/.../resource_label_events`, `/projects/:id/merge_requests/.../resource_label_events` | -| [Runners](runners.md) | `/projects/:id/runners` (also available standalone) | -| [Search](search.md) | `/projects/:id/search` (also available for groups and standalone) | -| [Services](services.md) | `/projects/:id/services` | -| [Tags](tags.md) | `/projects/:id/repository/tags` | -| [Wikis](wikis.md) | `/projects/:id/wikis` | +| Resource | Available endpoints | +|:--------------------------------------------------------------------|:------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| [Access requests](access_requests.md) | `/projects/:id/access_requests` (also available for groups) | +| [Award emoji](award_emoji.md) | `/projects/:id/issues/.../award_emoji`, `/projects/:id/merge_requests/.../award_emoji`, `/projects/:id/snippets/.../award_emoji` | +| [Branches](branches.md) | `/projects/:id/repository/branches/`, `/projects/:id/repository/merged_branches` | +| [Commits](commits.md) | `/projects/:id/repository/commits`, `/projects/:id/statuses` | +| [Container Registry](container_registry.md) | `/projects/:id/registry/repositories` | +| [Custom attributes](custom_attributes.md) | `/projects/:id/custom_attributes` (also available for groups and users) | +| [Deploy keys](deploy_keys.md) | `/projects/:id/deploy_keys` (also available standalone) | +| [Deployments](deployments.md) | `/projects/:id/deployments` | +| [Discussions](discussions.md) (threaded comments) | `/projects/:id/issues/.../discussions`, `/projects/:id/snippets/.../discussions`, `/projects/:id/merge_requests/.../discussions`, `/projects/:id/commits/.../discussions` (also available for groups) | +| [Environments](environments.md) | `/projects/:id/environments` | +| [Events](events.md) | `/projects/:id/events` (also available for users and standalone) | +| [Issues](issues.md) | `/projects/:id/issues` (also available for groups and standalone) | +| [Issue boards](boards.md) | `/projects/:id/boards` | +| [Issue links](issue_links.md) **[STARTER]** | `/projects/:id/issues/.../links` | +| [Jobs](jobs.md) | `/projects/:id/jobs`, `/projects/:id/pipelines/.../jobs` | +| [Labels](labels.md) | `/projects/:id/labels` | +| [Managed licenses](managed_licenses.md) **[ULTIMATE]** | `/projects/:id/managed_licenses` | +| [Members](members.md) | `/projects/:id/members` (also available for groups) | +| [Merge request approvals](merge_request_approvals.md) **[STARTER]** | `/projects/:id/approvals`, `/projects/:id/merge_requests/.../approvals` | +| [Merge requests](merge_requests.md) | `/projects/:id/merge_requests` (also available for groups and standalone) | +| [Notes](notes.md) (comments) | `/projects/:id/issues/.../notes`, `/projects/:id/snippets/.../notes`, `/projects/:id/merge_requests/.../notes` (also available for groups) | +| [Notification settings](notification_settings.md) | `/projects/:id/notification_settings` (also available for groups and standalone) | +| [Packages](packages.md) **[PREMIUM]** | `/projects/:id/packages` | +| [Pages domains](pages_domains.md) | `/projects/:id/pages` (also available standalone) | +| [Pipelines](pipelines.md) | `/projects/:id/pipelines` | +| [Pipeline schedules](pipeline_schedules.md) | `/projects/:id/pipeline_schedules` | +| [Pipeline triggers](pipeline_triggers.md) | `/projects/:id/triggers` | +| [Projects](projects.md) including setting Webhooks | `/projects`, `/projects/:id/hooks` (also available for users) | +| [Project badges](project_badges.md) | `/projects/:id/badges` | +| [Project clusters](project_clusters.md) | `/projects/:id/clusters` | +| [Project-level variables](project_level_variables.md) | `/projects/:id/variables` | +| [Project import/export](project_import_export.md) | `/projects/:id/export`, `/projects/import`, `/projects/:id/import` | +| [Project milestones](milestones.md) | `/projects/:id/milestones` | +| [Project snippets](project_snippets.md) | `/projects/:id/snippets` | +| [Project templates](project_templates.md) | `/projects/:id/templates` | +| [Protected branches](protected_branches.md) | `/projects/:id/protected_branches` | +| [Protected tags](protected_tags.md) | `/projects/:id/protected_tags` | +| [Releases](releases/index.md) | `/projects/:id/releases` | +| [Release links](releases/links.md) | `/projects/:id/releases/.../assets/links` | +| [Repositories](repositories.md) | `/projects/:id/repository` | +| [Repository files](repository_files.md) | `/projects/:id/repository/files` | +| [Repository submodules](repository_submodules.md) | `/projects/:id/repository/submodules` | +| [Resource label events](resource_label_events.md) | `/projects/:id/issues/.../resource_label_events`, `/projects/:id/merge_requests/.../resource_label_events` (also available for groups) | +| [Runners](runners.md) | `/projects/:id/runners` (also available standalone) | +| [Search](search.md) | `/projects/:id/search` (also available for groups and standalone) | +| [Services](services.md) | `/projects/:id/services` | +| [Tags](tags.md) | `/projects/:id/repository/tags` | +| [Wikis](wikis.md) | `/projects/:id/wikis` | ### Group resources The following API resources are available in the group context: -| Resource | Available endpoints | -|:--------------------------------------------------|:---------------------------------------------------------------------------------| -| [Access requests](access_requests.md) | `/groups/:id/access_requests/` (also available for projects) | -| [Custom attributes](custom_attributes.md) | `/groups/:id/custom_attributes` (also available for projects and users) | -| [Groups](groups.md) | `/groups`, `/groups/.../subgroups` | -| [Group badges](group_badges.md) | `/groups/:id/badges` | -| [Group issue boards](group_boards.md) | `/groups/:id/boards` | -| [Group labels](group_labels.md) | `/groups/:id/labels` | -| [Group-level variables](group_level_variables.md) | `/groups/:id/variables` | -| [Group milestones](group_milestones.md) | `/groups/:id/milestones` | -| [Issues](issues.md) | `/groups/:id/issues` (also available for projects and standalone) | -| [Members](members.md) | `/groups/:id/members` (also available for projects) | -| [Merge requests](merge_requests.md) | `/groups/:id/merge_requests` (also available for projects and standalone) | -| [Notification settings](notification_settings.md) | `/groups/:id/notification_settings` (also available for projects and standalone) | -| [Search](search.md) | `/groups/:id/search` (also available for projects and standalone) | +| Resource | Available endpoints | +|:-----------------------------------------------------------------|:---------------------------------------------------------------------------------| +| [Access requests](access_requests.md) | `/groups/:id/access_requests/` (also available for projects) | +| [Custom attributes](custom_attributes.md) | `/groups/:id/custom_attributes` (also available for projects and users) | +| [Discussions](discussions.md) (threaded comments) **[ULTIMATE]** | `/groups/:id/epics/.../discussions` (also available for projects) | +| [Epic issues](epic_issues.md) **[ULTIMATE]** | `/groups/:id/epics/.../issues` | +| [Epic links](epic_links.md) **[ULTIMATE]** | `/groups/:id/epics/.../epics` | +| [Epics](epics.md) **[ULTIMATE]** | `/groups/:id/epics` | +| [Groups](groups.md) | `/groups`, `/groups/.../subgroups` | +| [Group badges](group_badges.md) | `/groups/:id/badges` | +| [Group issue boards](group_boards.md) | `/groups/:id/boards` | +| [Group labels](group_labels.md) | `/groups/:id/labels` | +| [Group-level variables](group_level_variables.md) | `/groups/:id/variables` | +| [Group milestones](group_milestones.md) | `/groups/:id/milestones` | +| [Issues](issues.md) | `/groups/:id/issues` (also available for projects and standalone) | +| [Members](members.md) | `/groups/:id/members` (also available for projects) | +| [Merge requests](merge_requests.md) | `/groups/:id/merge_requests` (also available for projects and standalone) | +| [Notes](notes.md) (comments) | `/groups/:id/epics/.../notes` (also available for projects) | +| [Notification settings](notification_settings.md) | `/groups/:id/notification_settings` (also available for projects and standalone) | +| [Resource label events](resource_label_events.md) | `/groups/:id/epics/.../resource_label_events` (also available for projects) | +| [Search](search.md) | `/groups/:id/search` (also available for projects and standalone) | ### Standalone resources @@ -102,9 +112,11 @@ The following API resources are available outside of project and group contexts | [Deploy keys](deploy_keys.md) | `/deploy_keys` (also available for projects) | | [Events](events.md) | `/events`, `/users/:id/events` (also available for projects) | | [Feature flags](features.md) | `/features` | +| [Geo Nodes](geo_nodes.md) **[PREMIUM ONLY]** | `/geo_nodes` | | [Import repository from GitHub](import.md) | `/import/github` | | [Issues](issues.md) | `/issues` (also available for groups and projects) | | [Keys](keys.md) | `/keys` | +| [License](license.md) **[CORE ONLY]** | `/license` | | [Markdown](markdown.md) | `/markdown` | | [Merge requests](merge_requests.md) | `/merge_requests` (also available for groups and projects) | | [Namespaces](namespaces.md) | `/namespaces` | diff --git a/doc/api/boards.md b/doc/api/boards.md index 28c73db6b98..37111f870d6 100644 --- a/doc/api/boards.md +++ b/doc/api/boards.md @@ -141,6 +141,173 @@ Example response: } ``` +## Create a board **[STARTER]** + +Creates a board. + +``` +POST /projects/:id/boards +``` + +| Attribute | Type | Required | Description | +| --------- | ---- | -------- | ----------- | +| `id` | integer/string | yes | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding) owned by the authenticated user | +| `name` | string | yes | The name of the new board | + +```bash +curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" https://gitlab.example.com/api/v4/projects/5/boards?name=newboard +``` + +Example response: + +```json + { + "id": 1, + "project": { + "id": 5, + "name": "Diaspora Project Site", + "name_with_namespace": "Diaspora / Diaspora Project Site", + "path": "diaspora-project-site", + "path_with_namespace": "diaspora/diaspora-project-site", + "http_url_to_repo": "http://example.com/diaspora/diaspora-project-site.git", + "web_url": "http://example.com/diaspora/diaspora-project-site" + }, + "name": "newboard", + "milestone": { + "id": 12 + "title": "10.0" + }, + "lists" : [ + { + "id" : 1, + "label" : { + "name" : "Testing", + "color" : "#F0AD4E", + "description" : null + }, + "position" : 1 + }, + { + "id" : 2, + "label" : { + "name" : "Ready", + "color" : "#FF0000", + "description" : null + }, + "position" : 2 + }, + { + "id" : 3, + "label" : { + "name" : "Production", + "color" : "#FF5F00", + "description" : null + }, + "position" : 3 + } + ] + } +``` + +## Update a board **[STARTER]** + +> [Introduced][ee-5954] in GitLab 11.1. + +Updates a board. + +``` +PUT /projects/:id/boards/:board_id +``` + +| Attribute | Type | Required | Description | +| ------------------- | -------------- | -------- | ----------- | +| `id` | integer/string | yes | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding) owned by the authenticated user | +| `board_id` | integer | yes | The ID of a board | +| `name` | string | no | The new name of the board | +| `assignee_id` | integer | no | The assignee the board should be scoped to | +| `milestone_id` | integer | no | The milestone the board should be scoped to | +| `labels` | string | no | Comma-separated list of label names which the board should be scoped to | +| `weight` | integer | no | The weight range from 0 to 9, to which the board should be scoped to | + + +```bash +curl --request PUT --header "PRIVATE-TOKEN: <your_access_token>" https://gitlab.example.com/api/v4/projects/5/boards/1?name=new_name&milestone_id=43&assignee_id=1&labels=Doing&weight=4 +``` + +Example response: + +```json + { + "id": 1, + "project": { + "id": 5, + "name": "Diaspora Project Site", + "name_with_namespace": "Diaspora / Diaspora Project Site", + "path": "diaspora-project-site", + "path_with_namespace": "diaspora/diaspora-project-site", + "created_at": "2018-07-03T05:48:49.982Z", + "default_branch": null, + "tag_list": [], + "ssh_url_to_repo": "ssh://user@example.com/diaspora/diaspora-project-site.git", + "http_url_to_repo": "http://example.com/diaspora/diaspora-project-site.git", + "web_url": "http://example.com/diaspora/diaspora-project-site", + "readme_url": null, + "avatar_url": null, + "star_count": 0, + "forks_count": 0, + "last_activity_at": "2018-07-03T05:48:49.982Z" + }, + "lists": [], + "name": "new_name", + "group": null, + "milestone": { + "id": 43, + "iid": 1, + "project_id": 15, + "title": "Milestone 1", + "description": "Milestone 1 desc", + "state": "active", + "created_at": "2018-07-03T06:36:42.618Z", + "updated_at": "2018-07-03T06:36:42.618Z", + "due_date": null, + "start_date": null, + "web_url": "http://example.com/root/board1/milestones/1" + }, + "assignee": { + "id": 1, + "name": "Administrator", + "username": "root", + "state": "active", + "avatar_url": "https://www.gravatar.com/avatar/e64c7d89f26bd1972efa854d13d7dd61?s=80&d=identicon", + "web_url": "http://example.com/root" + }, + "labels": [{ + "id": 10, + "name": "Doing", + "color": "#5CB85C", + "description": null + }], + "weight": 4 + } +``` + +## Delete a board **[STARTER]** + +Deletes a board. + +``` +DELETE /projects/:id/boards/:board_id +``` + +| Attribute | Type | Required | Description | +| --------- | ---- | -------- | ----------- | +| `id` | integer/string | yes | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding) owned by the authenticated user | +| `board_id` | integer | yes | The ID of a board | + +```bash +curl --request DELETE --header "PRIVATE-TOKEN: <your_access_token>" https://gitlab.example.com/api/v4/projects/5/boards/1 +``` + ## List board lists Get a list of the board's lists. @@ -237,7 +404,14 @@ POST /projects/:id/boards/:board_id/lists | --------- | ---- | -------- | ----------- | | `id` | integer/string | yes | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding) owned by the authenticated user | | `board_id` | integer | yes | The ID of a board | -| `label_id` | integer | yes | The ID of a label | +| `label_id` | integer | no | The ID of a label | +| `assignee_id` | integer | no | The ID of a user | +| `milestone_id` | integer | no | The ID of a milestone | + +>**Note**: Label, assignee and milestone arguments are mutually exclusive, +that is, only one of them are accepted in a request. +Check the [Issue Board docs](../user/project/issue_board.md) for more +information regarding the required license for each list type. ```bash curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" https://gitlab.example.com/api/v4/projects/5/boards/1/lists?label_id=5 @@ -307,3 +481,5 @@ DELETE /projects/:id/boards/:board_id/lists/:list_id ```bash curl --request DELETE --header "PRIVATE-TOKEN: <your_access_token>" https://gitlab.example.com/api/v4/projects/5/boards/1/lists/1 ``` + +[ee-5954]: https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/5954 diff --git a/doc/api/discussions.md b/doc/api/discussions.md index 7d68d0ae744..9dec7858d9e 100644 --- a/doc/api/discussions.md +++ b/doc/api/discussions.md @@ -1,6 +1,6 @@ # Discussions API -Discussions are set of related notes on snippets, issues, merge requests or commits. +Discussions are set of related notes on snippets, issues, epics, merge requests or commits. ## Issues @@ -414,6 +414,213 @@ Parameters: curl --request DELETE --header "PRIVATE-TOKEN: <your_access_token>" https://gitlab.example.com/api/v4/projects/5/snippets/11/discussions/636 ``` +## Epics **[ULTIMATE]** + +### List group epic discussions + +Gets a list of all discussions for a single epic. + +``` +GET /groups/:id/epics/:epic_id/discussions +``` + +| Attribute | Type | Required | Description | +| ------------------- | ---------------- | ---------- | ------------ | +| `id` | integer/string | yes | The ID or [URL-encoded path of the group](README.md#namespaced-path-encoding) | +| `epic_id` | integer | yes | The ID of an epic | + +```json +[ + { + "id": "6a9c1750b37d513a43987b574953fceb50b03ce7", + "individual_note": false, + "notes": [ + { + "id": 1126, + "type": "DiscussionNote", + "body": "discussion text", + "attachment": null, + "author": { + "id": 1, + "name": "root", + "username": "root", + "state": "active", + "avatar_url": "https://www.gravatar.com/avatar/00afb8fb6ab07c3ee3e9c1f38777e2f4?s=80&d=identicon", + "web_url": "http://localhost:3000/root" + }, + "created_at": "2018-03-03T21:54:39.668Z", + "updated_at": "2018-03-03T21:54:39.668Z", + "system": false, + "noteable_id": 3, + "noteable_type": "Epic", + "noteable_id": null, + "resolvable": false + }, + { + "id": 1129, + "type": "DiscussionNote", + "body": "reply to the discussion", + "attachment": null, + "author": { + "id": 1, + "name": "root", + "username": "root", + "state": "active", + "avatar_url": "https://www.gravatar.com/avatar/00afb8fb6ab07c3ee3e9c1f38777e2f4?s=80&d=identicon", + "web_url": "http://localhost:3000/root" + }, + "created_at": "2018-03-04T13:38:02.127Z", + "updated_at": "2018-03-04T13:38:02.127Z", + "system": false, + "noteable_id": 3, + "noteable_type": "Epic", + "noteable_id": null, + "resolvable": false + } + ] + }, + { + "id": "87805b7c09016a7058e91bdbe7b29d1f284a39e6", + "individual_note": true, + "notes": [ + { + "id": 1128, + "type": null, + "body": "a single comment", + "attachment": null, + "author": { + "id": 1, + "name": "root", + "username": "root", + "state": "active", + "avatar_url": "https://www.gravatar.com/avatar/00afb8fb6ab07c3ee3e9c1f38777e2f4?s=80&d=identicon", + "web_url": "http://localhost:3000/root" + }, + "created_at": "2018-03-04T09:17:22.520Z", + "updated_at": "2018-03-04T09:17:22.520Z", + "system": false, + "noteable_id": 3, + "noteable_type": "Epic", + "noteable_id": null, + "resolvable": false + } + ] + } +] +``` + +```bash +curl --request GET --header "PRIVATE-TOKEN: <your_access_token>" https://gitlab.example.com/api/v4/groups/5/epics/11/discussions +``` + +### Get single epic discussion + +Returns a single discussion for a specific group epic + +``` +GET /groups/:id/epics/:epic_id/discussions/:discussion_id +``` + +Parameters: + +| Attribute | Type | Required | Description | +| --------------- | -------------- | -------- | ----------- | +| `id` | integer/string | yes | The ID or [URL-encoded path of the group](README.md#namespaced-path-encoding) | +| `epic_id` | integer | yes | The ID of an epic | +| `discussion_id` | integer | yes | The ID of a discussion | + +```bash +curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" https://gitlab.example.com/api/v4/groups/5/epics/11/discussions/6a9c1750b37d513a43987b574953fceb50b03ce7 +``` + +### Create new epic discussion + +Creates a new discussion to a single group epic. This is similar to creating +a note but but another comments (replies) can be added to it later. + +``` +POST /groups/:id/epics/:epic_id/discussions +``` + +Parameters: + +| Attribute | Type | Required | Description | +| --------------- | -------------- | -------- | ----------- | +| `id` | integer/string | yes | The ID or [URL-encoded path of the group](README.md#namespaced-path-encoding) | +| `epic_id` | integer | yes | The ID of an epic | +| `body` | string | yes | The content of a discussion | +| `created_at` | string | no | Date time string, ISO 8601 formatted, e.g. 2016-03-11T03:45:40Z (requires admin or project/group owner rights) | + +```bash +curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" https://gitlab.example.com/api/v4/groups/5/epics/11/discussions?body=comment +``` + +### Add note to existing epic discussion + +Adds a new note to the discussion. + +``` +POST /groups/:id/epics/:epic_id/discussions/:discussion_id/notes +``` + +Parameters: + +| Attribute | Type | Required | Description | +| --------------- | -------------- | -------- | ----------- | +| `id` | integer/string | yes | The ID or [URL-encoded path of the group](README.md#namespaced-path-encoding) | +| `epic_id` | integer | yes | The ID of an epic | +| `discussion_id` | integer | yes | The ID of a discussion | +| `note_id` | integer | yes | The ID of a discussion note | +| `body` | string | yes | The content of a discussion | +| `created_at` | string | no | Date time string, ISO 8601 formatted, e.g. 2016-03-11T03:45:40Z (requires admin or project/group owner rights) | + +```bash +curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" https://gitlab.example.com/api/v4/groups/5/epics/11/discussions/6a9c1750b37d513a43987b574953fceb50b03ce7/notes?body=comment +``` + +### Modify existing epic discussion note + +Modify existing discussion note of an epic. + +``` +PUT /groups/:id/epics/:epic_id/discussions/:discussion_id/notes/:note_id +``` + +Parameters: + +| Attribute | Type | Required | Description | +| --------------- | -------------- | -------- | ----------- | +| `id` | integer/string | yes | The ID or [URL-encoded path of the group](README.md#namespaced-path-encoding) | +| `epic_id` | integer | yes | The ID of an epic | +| `discussion_id` | integer | yes | The ID of a discussion | +| `note_id` | integer | yes | The ID of a discussion note | +| `body` | string | yes | The content of a discussion | + +```bash +curl --request PUT --header "PRIVATE-TOKEN: <your_access_token>" https://gitlab.example.com/api/v4/groups/5/epics/11/discussions/6a9c1750b37d513a43987b574953fceb50b03ce7/notes/1108?body=comment +``` + +### Delete an epic discussion note + +Deletes an existing discussion note of an epic. + +``` +DELETE /groups/:id/epics/:epic_id/discussions/:discussion_id/notes/:note_id +``` + +Parameters: + +| Attribute | Type | Required | Description | +| --------------- | -------------- | -------- | ----------- | +| `id` | integer/string | yes | The ID or [URL-encoded path of the group](README.md#namespaced-path-encoding) | +| `epic_id` | integer | yes | The ID of an epic | +| `discussion_id` | integer | yes | The ID of a discussion | +| `note_id` | integer | yes | The ID of a discussion note | + +```bash +curl --request DELETE --header "PRIVATE-TOKEN: <your_access_token>" https://gitlab.example.com/api/v4/groups/5/epics/11/discussions/636 +``` + ## Merge requests ### List project merge request discussions diff --git a/doc/api/epic_issues.md b/doc/api/epic_issues.md new file mode 100644 index 00000000000..438a3361dcc --- /dev/null +++ b/doc/api/epic_issues.md @@ -0,0 +1,412 @@ +# Epic Issues API **[ULTIMATE]** + +Every API call to epic_issues must be authenticated. + +If a user is not a member of a group and the group is private, a `GET` request on that group will result to a `404` status code. + +Epics are available only in Ultimate. If epics feature is not available a `403` status code will be returned. + +## List issues for an epic +Gets all issues that are assigned to an epic and the authenticated user has access to. + +``` +GET /groups/:id/epics/:epic_iid/issues +``` + +| Attribute | Type | Required | Description | +| ------------------- | ---------------- | ---------- | ---------------------------------------------------------------------------------------| +| `id` | integer/string | yes | The ID or [URL-encoded path of the group](README.md#namespaced-path-encoding) owned by the authenticated user | +| `epic_iid` | integer/string | yes | The internal ID of the epic. | + +```bash +curl --header "PRIVATE-TOKEN: <your_access_token>" https://gitlab.example.com/api/v4/groups/1/epics/5/issues/ +``` + +Example response: + +```json +[ + { + "id": 76, + "iid": 6, + "project_id": 8, + "title" : "Consequatur vero maxime deserunt laboriosam est voluptas dolorem.", + "description" : "Ratione dolores corrupti mollitia soluta quia.", + "state": "opened", + "created_at": "2017-11-15T13:39:24.670Z", + "updated_at": "2018-01-04T10:49:19.506Z", + "closed_at": null, + "labels": [], + "milestone": { + "id": 38, + "iid": 3, + "project_id": 8, + "title": "v2.0", + "description": "In tempore culpa inventore quo accusantium.", + "state": "closed", + "created_at": "2017-11-15T13:39:13.825Z", + "updated_at": "2017-11-15T13:39:13.825Z", + "due_date": null, + "start_date": null + }, + "assignees": [{ + "id": 7, + "name": "Pamella Huel", + "username": "arnita", + "state": "active", + "avatar_url": "http://www.gravatar.com/avatar/a2f5c6fcef64c9c69cb8779cb292be1b?s=80&d=identicon", + "web_url": "http://localhost:3001/arnita" + }], + "assignee": { + "id": 7, + "name": "Pamella Huel", + "username": "arnita", + "state": "active", + "avatar_url": "http://www.gravatar.com/avatar/a2f5c6fcef64c9c69cb8779cb292be1b?s=80&d=identicon", + "web_url": "http://localhost:3001/arnita" + }, + "author": { + "id": 13, + "name": "Michell Johns", + "username": "chris_hahn", + "state": "active", + "avatar_url": "http://www.gravatar.com/avatar/30e3b2122ccd6b8e45e8e14a3ffb58fc?s=80&d=identicon", + "web_url": "http://localhost:3001/chris_hahn" + }, + "user_notes_count": 8, + "upvotes": 0, + "downvotes": 0, + "due_date": null, + "confidential": false, + "weight": null, + "discussion_locked": null, + "web_url": "http://localhost:3001/h5bp/html5-boilerplate/issues/6", + "time_stats": { + "time_estimate": 0, + "total_time_spent": 0, + "human_time_estimate": null, + "human_total_time_spent": null + }, + "_links":{ + "self": "http://localhost:3001/api/v4/projects/8/issues/6", + "notes": "http://localhost:3001/api/v4/projects/8/issues/6/notes", + "award_emoji": "http://localhost:3001/api/v4/projects/8/issues/6/award_emoji", + "project": "http://localhost:3001/api/v4/projects/8" + }, + "subscribed": true, + "epic_issue_id": 2 + } +] +``` + +**Note**: `assignee` column is deprecated, now we show it as a single-sized array `assignees` to conform to the GitLab EE API. + +## Assign an issue to the epic + +Creates an epic - issue association. If the issue in question belongs to another epic it is unassigned from that epic. + +``` +POST /groups/:id/epics/:epic_iid/issues/:issue_id +``` + +| Attribute | Type | Required | Description | +| ------------------- | ---------------- | ---------- | ---------------------------------------------------------------------------------------| +| `id` | integer/string | yes | The ID or [URL-encoded path of the group](README.md#namespaced-path-encoding) owned by the authenticated user | +| `epic_iid` | integer/string | yes | The internal ID of the epic. | +| `issue_id` | integer/string | yes | The ID of the issue. | + +```bash +curl --header POST "PRIVATE-TOKEN: <your_access_token>" https://gitlab.example.com/api/v4/groups/1/epics/5/issues/55 +``` + +Example response: + +```json +{ + "id": 11, + "epic": { + "id": 30, + "iid": 5, + "title": "Ea cupiditate dolores ut vero consequatur quasi veniam voluptatem et non.", + "description": "Molestias dolorem eos vitae expedita impedit necessitatibus quo voluptatum.", + "author": { + "id": 7, + "name": "Pamella Huel", + "username": "arnita", + "state": "active", + "avatar_url": "http://www.gravatar.com/avatar/a2f5c6fcef64c9c69cb8779cb292be1b?s=80&d=identicon", + "web_url": "http://localhost:3001/arnita" + }, + "start_date": null, + "end_date": null + }, + "issue": { + "id": 55, + "iid": 13, + "project_id": 8, + "title": "Beatae laborum voluptatem voluptate eligendi ex accusamus.", + "description": "Quam veritatis debitis omnis aliquam sit.", + "state": "opened", + "created_at": "2017-11-05T13:59:12.782Z", + "updated_at": "2018-01-05T10:33:03.900Z", + "closed_at": null, + "labels": [], + "milestone": { + "id": 48, + "iid": 6, + "project_id": 8, + "title": "Sprint - Sed sed maxime temporibus ipsa ullam qui sit.", + "description": "Quos veritatis qui expedita sunt deleniti accusamus.", + "state": "active", + "created_at": "2017-11-05T13:59:12.445Z", + "updated_at": "2017-11-05T13:59:12.445Z", + "due_date": "2017-11-13", + "start_date": "2017-11-05" + }, + "assignees": [{ + "id": 10, + "name": "Lu Mayer", + "username": "kam", + "state": "active", + "avatar_url": "http://www.gravatar.com/avatar/018729e129a6f31c80a6327a30196823?s=80&d=identicon", + "web_url": "http://localhost:3001/kam" + }], + "assignee": { + "id": 10, + "name": "Lu Mayer", + "username": "kam", + "state": "active", + "avatar_url": "http://www.gravatar.com/avatar/018729e129a6f31c80a6327a30196823?s=80&d=identicon", + "web_url": "http://localhost:3001/kam" + }, + "author": { + "id": 25, + "name": "User 3", + "username": "user3", + "state": "active", + "avatar_url": "http://www.gravatar.com/avatar/97d6d9441ff85fdc730e02a6068d267b?s=80&d=identicon", + "web_url": "http://localhost:3001/user3" + }, + "user_notes_count": 0, + "upvotes": 0, + "downvotes": 0, + "due_date": null, + "confidential": false, + "weight": null, + "discussion_locked": null, + "web_url": "http://localhost:3001/h5bp/html5-boilerplate/issues/13", + "time_stats": { + "time_estimate": 0, + "total_time_spent": 0, + "human_time_estimate": null, + "human_total_time_spent": null + } + } +} +``` + +**Note**: `assignee` column is deprecated, now we show it as a single-sized array `assignees` to conform to the GitLab EE API. + +## Remove an issue from the epic + +Removes an epic - issue association. + +``` +DELETE /groups/:id/epics/:epic_iid/issues/:epic_issue_id +``` + +| Attribute | Type | Required | Description | +| ------------------- | ---------------- | ---------- | -----------------------------------------------------------------------------------------------------| +| `id` | integer/string | yes | The ID or [URL-encoded path of the group](README.md#namespaced-path-encoding) owned by the authenticated user | +| `epic_iid` | integer/string | yes | The internal ID of the epic. | +| `epic_issue_id` | integer/string | yes | The ID of the issue - epic association. | + +```bash +curl --header DELETE "PRIVATE-TOKEN: <your_access_token>" https://gitlab.example.com/api/v4/groups/1/epics/5/issues/11 +``` + +Example response: + +```json +{ + "id": 11, + "epic": { + "id": 30, + "iid": 5, + "title": "Ea cupiditate dolores ut vero consequatur quasi veniam voluptatem et non.", + "description": "Molestias dolorem eos vitae expedita impedit necessitatibus quo voluptatum.", + "author": { + "id": 7, + "name": "Pamella Huel", + "username": "arnita", + "state": "active", + "avatar_url": "http://www.gravatar.com/avatar/a2f5c6fcef64c9c69cb8779cb292be1b?s=80&d=identicon", + "web_url": "http://localhost:3001/arnita" + }, + "start_date": null, + "end_date": null + }, + "issue": { + "id": 223, + "iid": 13, + "project_id": 8, + "title": "Beatae laborum voluptatem voluptate eligendi ex accusamus.", + "description": "Quam veritatis debitis omnis aliquam sit.", + "state": "opened", + "created_at": "2017-11-05T13:59:12.782Z", + "updated_at": "2018-01-05T10:33:03.900Z", + "closed_at": null, + "labels": [], + "milestone": { + "id": 48, + "iid": 6, + "project_id": 8, + "title": "Sprint - Sed sed maxime temporibus ipsa ullam qui sit.", + "description": "Quos veritatis qui expedita sunt deleniti accusamus.", + "state": "active", + "created_at": "2017-11-05T13:59:12.445Z", + "updated_at": "2017-11-05T13:59:12.445Z", + "due_date": "2017-11-13", + "start_date": "2017-11-05" + }, + "assignees": [{ + "id": 10, + "name": "Lu Mayer", + "username": "kam", + "state": "active", + "avatar_url": "http://www.gravatar.com/avatar/018729e129a6f31c80a6327a30196823?s=80&d=identicon", + "web_url": "http://localhost:3001/kam" + }], + "assignee": { + "id": 10, + "name": "Lu Mayer", + "username": "kam", + "state": "active", + "avatar_url": "http://www.gravatar.com/avatar/018729e129a6f31c80a6327a30196823?s=80&d=identicon", + "web_url": "http://localhost:3001/kam" + }, + "author": { + "id": 25, + "name": "User 3", + "username": "user3", + "state": "active", + "avatar_url": "http://www.gravatar.com/avatar/97d6d9441ff85fdc730e02a6068d267b?s=80&d=identicon", + "web_url": "http://localhost:3001/user3" + }, + "user_notes_count": 0, + "upvotes": 0, + "downvotes": 0, + "due_date": null, + "confidential": false, + "weight": null, + "discussion_locked": null, + "web_url": "http://localhost:3001/h5bp/html5-boilerplate/issues/13", + "time_stats": { + "time_estimate": 0, + "total_time_spent": 0, + "human_time_estimate": null, + "human_total_time_spent": null + } + } +} +``` + +**Note**: `assignee` column is deprecated, now we show it as a single-sized array `assignees` to conform to the GitLab EE API. + +## Update epic - issue association + +Updates an epic - issue association. + +``` +PUT /groups/:id/epics/:epic_iid/issues/:epic_issue_id +``` + +| Attribute | Type | Required | Description | +| ------------------- | ---------------- | ---------- | -----------------------------------------------------------------------------------------------------| +| `id` | integer/string | yes | The ID or [URL-encoded path of the group](README.md#namespaced-path-encoding) owned by the authenticated user | +| `epic_iid` | integer/string | yes | The internal ID of the epic. | +| `epic_issue_id` | integer/string | yes | The ID of the issue - epic association. | +| `move_before_id` | integer/string | no | The ID of the issue - epic association that should be placed before the link in the question. | +| `move_after_id` | integer/string | no | The ID of the issue - epic association that should be placed after the link in the question. | + +```bash +curl --header PUT "PRIVATE-TOKEN: <your_access_token>" https://gitlab.example.com/api/v4/groups/1/epics/5/issues/11?move_before_id=20 +``` + +Example response: + +```json +[ + { + "id": 30, + "iid": 6, + "project_id": 8, + "title" : "Consequatur vero maxime deserunt laboriosam est voluptas dolorem.", + "description" : "Ratione dolores corrupti mollitia soluta quia.", + "state": "opened", + "created_at": "2017-11-15T13:39:24.670Z", + "updated_at": "2018-01-04T10:49:19.506Z", + "closed_at": null, + "labels": [], + "milestone": { + "id": 38, + "iid": 3, + "project_id": 8, + "title": "v2.0", + "description": "In tempore culpa inventore quo accusantium.", + "state": "closed", + "created_at": "2017-11-15T13:39:13.825Z", + "updated_at": "2017-11-15T13:39:13.825Z", + "due_date": null, + "start_date": null + }, + "assignees": [{ + "id": 7, + "name": "Pamella Huel", + "username": "arnita", + "state": "active", + "avatar_url": "http://www.gravatar.com/avatar/a2f5c6fcef64c9c69cb8779cb292be1b?s=80&d=identicon", + "web_url": "http://localhost:3001/arnita" + }], + "assignee": { + "id": 7, + "name": "Pamella Huel", + "username": "arnita", + "state": "active", + "avatar_url": "http://www.gravatar.com/avatar/a2f5c6fcef64c9c69cb8779cb292be1b?s=80&d=identicon", + "web_url": "http://localhost:3001/arnita" + }, + "author": { + "id": 13, + "name": "Michell Johns", + "username": "chris_hahn", + "state": "active", + "avatar_url": "http://www.gravatar.com/avatar/30e3b2122ccd6b8e45e8e14a3ffb58fc?s=80&d=identicon", + "web_url": "http://localhost:3001/chris_hahn" + }, + "user_notes_count": 8, + "upvotes": 0, + "downvotes": 0, + "due_date": null, + "confidential": false, + "weight": null, + "discussion_locked": null, + "web_url": "http://localhost:3001/h5bp/html5-boilerplate/issues/6", + "time_stats": { + "time_estimate": 0, + "total_time_spent": 0, + "human_time_estimate": null, + "human_total_time_spent": null + }, + "_links":{ + "self": "http://localhost:3001/api/v4/projects/8/issues/6", + "notes": "http://localhost:3001/api/v4/projects/8/issues/6/notes", + "award_emoji": "http://localhost:3001/api/v4/projects/8/issues/6/award_emoji", + "project": "http://localhost:3001/api/v4/projects/8" + }, + "subscribed": true, + "epic_issue_id": 11, + "relative_position": 55 + } +] +``` diff --git a/doc/api/epic_links.md b/doc/api/epic_links.md new file mode 100644 index 00000000000..850aa7e7824 --- /dev/null +++ b/doc/api/epic_links.md @@ -0,0 +1,218 @@ +# Epic Links API **[ULTIMATE]** + +>**Note:** +> This endpoint was [introduced](https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/9188) in GitLab 11.8. + +Manages parent-child [epic relationships](../user/group/epics/index.md#multi-level-child-epics). + +Every API call to `epic_links` must be authenticated. + +If a user is not a member of a group and the group is private, a `GET` request on that group will result to a `404` status code. + +Epics are available only in the [Ultimate/Gold tier](https://about.gitlab.com/pricing/). If the epics feature is not available, a `403` status code will be returned. + +## List epics related to a given epic +Gets all child epics of an epic. + +``` +GET /groups/:id/epics/:epic_iid/epics +``` + +| Attribute | Type | Required | Description | +| ---------- | -------------- | -------- | ------------------------------------------------------------------------------------------------------------- | +| `id` | integer/string | yes | The ID or [URL-encoded path of the group](README.md#namespaced-path-encoding) owned by the authenticated user | +| `epic_iid` | integer | yes | The internal ID of the epic. | + +```bash +curl --header "PRIVATE-TOKEN: <your_access_token>" https://gitlab.example.com/api/v4/groups/1/epics/5/epics/ +``` + +Example response: + +```json +[ + { + "id": 29, + "iid": 6, + "group_id": 1, + "parent_id": 5, + "title": "Accusamus iste et ullam ratione voluptatem omnis debitis dolor est.", + "description": "Molestias dolorem eos vitae expedita impedit necessitatibus quo voluptatum.", + "author": { + "id": 10, + "name": "Lu Mayer", + "username": "kam", + "state": "active", + "avatar_url": "http://www.gravatar.com/avatar/018729e129a6f31c80a6327a30196823?s=80&d=identicon", + "web_url": "http://localhost:3001/kam" + }, + "start_date": null, + "start_date_is_fixed": false, + "start_date_fixed": null, + "start_date_from_milestones": null, + "end_date": "2018-07-31", + "due_date": "2018-07-31", + "due_date_is_fixed": false, + "due_date_fixed": null, + "due_date_from_milestones": "2018-07-31", + "created_at": "2018-07-17T13:36:22.770Z", + "updated_at": "2018-07-18T12:22:05.239Z", + "labels": [] + } +] +``` + +## Assign a child epic + +Creates an association between two epics, designating one as the parent epic and the other as the child epic. A parent epic can have multiple child epics. If the new child epic already belonged to another epic, it is unassigned from that previous parent. + +``` +POST /groups/:id/epics/:epic_iid/epics +``` + +| Attribute | Type | Required | Description | +| --------------- | -------------- | -------- | ------------------------------------------------------------------------------------------------------------------ | +| `id` | integer/string | yes | The ID or [URL-encoded path of the group](README.md#namespaced-path-encoding) owned by the authenticated user | +| `epic_iid` | integer | yes | The internal ID of the epic. | +| `child_epic_id` | integer | yes | The global ID of the child epic. Internal ID can't be used because they can conflict with epics from other groups. | + +```bash +curl --header POST "PRIVATE-TOKEN: <your_access_token>" https://gitlab.example.com/api/v4/groups/1/epics/5/epics/6 +``` + +Example response: + +```json +{ + "id": 6, + "iid": 38, + "group_id": 1, + "parent_id": 5 + "title": "Accusamus iste et ullam ratione voluptatem omnis debitis dolor est.", + "description": "Molestias dolorem eos vitae expedita impedit necessitatibus quo voluptatum.", + "author": { + "id": 10, + "name": "Lu Mayer", + "username": "kam", + "state": "active", + "avatar_url": "http://www.gravatar.com/avatar/018729e129a6f31c80a6327a30196823?s=80&d=identicon", + "web_url": "http://localhost:3001/kam" + }, + "start_date": null, + "start_date_is_fixed": false, + "start_date_fixed": null, + "start_date_from_milestones": null, + "end_date": "2018-07-31", + "due_date": "2018-07-31", + "due_date_is_fixed": false, + "due_date_fixed": null, + "due_date_from_milestones": "2018-07-31", + "created_at": "2018-07-17T13:36:22.770Z", + "updated_at": "2018-07-18T12:22:05.239Z", + "labels": [] +} +``` + +## Re-order a child epic + +``` +PUT /groups/:id/epics/:epic_iid/epics/:child_epic_id +``` + +| Attribute | Type | Required | Description | +| ---------------- | -------------- | -------- | ------------------------------------------------------------------------------------------------------------------ | +| `id` | integer/string | yes | The ID or [URL-encoded path of the group](README.md#namespaced-path-encoding) owned by the authenticated user. | +| `epic_iid` | integer | yes | The internal ID of the epic. | +| `child_epic_id` | integer | yes | The global ID of the child epic. Internal ID can't be used because they can conflict with epics from other groups. | +| `move_before_id` | integer | no | The global ID of a sibling epic that should be placed before the child epic. | +| `move_after_id` | integer | no | The global ID of a sibling epic that should be placed after the child epic. | + +```bash +curl --header PUT "PRIVATE-TOKEN: <your_access_token>" https://gitlab.example.com/api/v4/groups/1/epics/4/epics/5 +``` + +Example response: + +```json +[ + { + "id": 29, + "iid": 6, + "group_id": 1, + "parent_id": 5, + "title": "Accusamus iste et ullam ratione voluptatem omnis debitis dolor est.", + "description": "Molestias dolorem eos vitae expedita impedit necessitatibus quo voluptatum.", + "author": { + "id": 10, + "name": "Lu Mayer", + "username": "kam", + "state": "active", + "avatar_url": "http://www.gravatar.com/avatar/018729e129a6f31c80a6327a30196823?s=80&d=identicon", + "web_url": "http://localhost:3001/kam" + }, + "start_date": null, + "start_date_is_fixed": false, + "start_date_fixed": null, + "start_date_from_milestones": null, + "end_date": "2018-07-31", + "due_date": "2018-07-31", + "due_date_is_fixed": false, + "due_date_fixed": null, + "due_date_from_milestones": "2018-07-31", + "created_at": "2018-07-17T13:36:22.770Z", + "updated_at": "2018-07-18T12:22:05.239Z", + "labels": [] + } +] +``` + +## Unassign a child epic + +Unassigns a child epic from a parent epic. + +``` +DELETE /groups/:id/epics/:epic_iid/epics/:child_epic_id +``` + +| Attribute | Type | Required | Description | +| --------------- | -------------- | -------- | ------------------------------------------------------------------------------------------------------------------ | +| `id` | integer/string | yes | The ID or [URL-encoded path of the group](README.md#namespaced-path-encoding) owned by the authenticated user. | +| `epic_iid` | integer | yes | The internal ID of the epic. | +| `child_epic_id` | integer | yes | The global ID of the child epic. Internal ID can't be used because they can conflict with epics from other groups. | + +```bash +curl --header DELETE "PRIVATE-TOKEN: <your_access_token>" https://gitlab.example.com/api/v4/groups/1/epics/4/epics/5 +``` + +Example response: + +```json +{ + "id": 5, + "iid": 38, + "group_id": 1, + "parent_id": null, + "title": "Accusamus iste et ullam ratione voluptatem omnis debitis dolor est.", + "description": "Molestias dolorem eos vitae expedita impedit necessitatibus quo voluptatum.", + "author": { + "id": 10, + "name": "Lu Mayer", + "username": "kam", + "state": "active", + "avatar_url": "http://www.gravatar.com/avatar/018729e129a6f31c80a6327a30196823?s=80&d=identicon", + "web_url": "http://localhost:3001/kam" + }, + "start_date": null, + "start_date_is_fixed": false, + "start_date_fixed": null, + "start_date_from_milestones": null, + "end_date": "2018-07-31", + "due_date": "2018-07-31", + "due_date_is_fixed": false, + "due_date_fixed": null, + "due_date_from_milestones": "2018-07-31", + "created_at": "2018-07-17T13:36:22.770Z", + "updated_at": "2018-07-18T12:22:05.239Z", + "labels": [] +} +``` diff --git a/doc/api/epics.md b/doc/api/epics.md new file mode 100644 index 00000000000..c7bf04f0043 --- /dev/null +++ b/doc/api/epics.md @@ -0,0 +1,351 @@ +# Epics API **[ULTIMATE]** + +Every API call to epic must be authenticated. + +If a user is not a member of a group and the group is private, a `GET` request on that group will result to a `404` status code. + +If epics feature is not available a `403` status code will be returned. + +## Epic issues API + +The [epic issues API](epic_issues.md) allows you to interact with issues associated with an epic. + +# Milestone dates integration + +> [Introduced][ee-6448] in GitLab 11.3. + +Since start date and due date can be dynamically sourced from related issue milestones, when user has edit permission, additional fields will be shown. These include two boolean fields `start_date_is_fixed` and `due_date_is_fixed`, and four date fields `start_date_fixed`, `start_date_from_milestones`, `due_date_fixed` and `due_date_from_milestones`. + +`end_date` has been deprecated in favor of `due_date`. + +## Epics pagination + +By default, `GET` requests return 20 results at a time because the API results +are paginated. + +Read more on [pagination](README.md#pagination). + +## List epics for a group + +Gets all epics of the requested group and its subgroups. + +``` +GET /groups/:id/epics +GET /groups/:id/epics?author_id=5 +GET /groups/:id/epics?labels=bug,reproduced +GET /groups/:id/epics?state=opened +``` + +| Attribute | Type | Required | Description | +| ------------------- | ---------------- | ---------- | ---------------------------------------------------------------------------------------| +| `id` | integer/string | yes | The ID or [URL-encoded path of the group](README.md#namespaced-path-encoding) owned by the authenticated user | +| `author_id` | integer | no | Return epics created by the given user `id` | +| `labels` | string | no | Return epics matching a comma separated list of labels names. Label names from the epic group or a parent group can be used | +| `order_by` | string | no | Return epics ordered by `created_at` or `updated_at` fields. Default is `created_at` | +| `sort` | string | no | Return epics sorted in `asc` or `desc` order. Default is `desc` | +| `search` | string | no | Search epics against their `title` and `description` | +| `state` | string | no | Search epics against their `state`, possible filters: `opened`, `closed` and `all`, default: `all` | + +```bash +curl --header "PRIVATE-TOKEN: <your_access_token>" https://gitlab.example.com/api/v4/groups/1/epics +``` + +Example response: + +```json +[ + { + "id": 29, + "iid": 4, + "group_id": 7, + "title": "Accusamus iste et ullam ratione voluptatem omnis debitis dolor est.", + "description": "Molestias dolorem eos vitae expedita impedit necessitatibus quo voluptatum.", + "author": { + "id": 10, + "name": "Lu Mayer", + "username": "kam", + "state": "active", + "avatar_url": "http://www.gravatar.com/avatar/018729e129a6f31c80a6327a30196823?s=80&d=identicon", + "web_url": "http://localhost:3001/kam" + }, + "start_date": null, + "start_date_is_fixed": false, + "start_date_fixed": null, + "start_date_from_milestones": null, + "end_date": "2018-07-31", + "due_date": "2018-07-31", + "due_date_is_fixed": false, + "due_date_fixed": null, + "due_date_from_milestones": "2018-07-31", + "created_at": "2018-07-17T13:36:22.770Z", + "updated_at": "2018-07-18T12:22:05.239Z", + "labels": [], + "upvotes": 4, + "downvotes": 0 + } +] +``` + +## Single epic + +Gets a single epic + +``` +GET /groups/:id/epics/:epic_iid +``` + +| Attribute | Type | Required | Description | +| ------------------- | ---------------- | ---------- | ---------------------------------------------------------------------------------------| +| `id` | integer/string | yes | The ID or [URL-encoded path of the group](README.md#namespaced-path-encoding) owned by the authenticated user | +| `epic_iid` | integer/string | yes | The internal ID of the epic. | + +```bash +curl --header "PRIVATE-TOKEN: <your_access_token>" https://gitlab.example.com/api/v4/groups/1/epics/5 +``` + +Example response: + +```json +{ + "id": 30, + "iid": 5, + "group_id": 7, + "title": "Ea cupiditate dolores ut vero consequatur quasi veniam voluptatem et non.", + "description": "Molestias dolorem eos vitae expedita impedit necessitatibus quo voluptatum.", + "author":{ + "id": 7, + "name": "Pamella Huel", + "username": "arnita", + "state": "active", + "avatar_url": "http://www.gravatar.com/avatar/a2f5c6fcef64c9c69cb8779cb292be1b?s=80&d=identicon", + "web_url": "http://localhost:3001/arnita" + }, + "start_date": null, + "start_date_is_fixed": false, + "start_date_fixed": null, + "start_date_from_milestones": null, + "end_date": "2018-07-31", + "due_date": "2018-07-31", + "due_date_is_fixed": false, + "due_date_fixed": null, + "due_date_from_milestones": "2018-07-31", + "created_at": "2018-07-17T13:36:22.770Z", + "updated_at": "2018-07-18T12:22:05.239Z", + "labels": [], + "upvotes": 4, + "downvotes": 0 +} +``` + +## New epic + +Creates a new epic. + +NOTE: **Note:** +Starting with GitLab [11.3][ee-6448], `start_date` and `end_date` should no longer be assigned +directly, as they now represent composite values. You can configure it via the `*_is_fixed` and +`*_fixed` fields instead. + +``` +POST /groups/:id/epics +``` + +| Attribute | Type | Required | Description | +| ------------------- | ---------------- | ---------- | ---------------------------------------------------------------------------------------| +| `id` | integer/string | yes | The ID or [URL-encoded path of the group](README.md#namespaced-path-encoding) owned by the authenticated user | +| `title` | string | yes | The title of the epic | +| `labels` | string | no | The comma separated list of labels | +| `description` | string | no | The description of the epic | +| `start_date_is_fixed` | boolean | no | Whether start date should be sourced from `start_date_fixed` or from milestones (since 11.3) | +| `start_date_fixed` | string | no | The fixed start date of an epic (since 11.3) | +| `due_date_is_fixed` | boolean | no | Whether due date should be sourced from `due_date_fixed` or from milestones (since 11.3) | +| `due_date_fixed` | string | no | The fixed due date of an epic (since 11.3) | + +```bash +curl --header POST "PRIVATE-TOKEN: <your_access_token>" https://gitlab.example.com/api/v4/groups/1/epics?title=Epic&description=Epic%20description +``` + +Example response: + +```json +{ + "id": 33, + "iid": 6, + "group_id": 7, + "title": "Epic", + "description": "Epic description", + "author": { + "name" : "Alexandra Bashirian", + "avatar_url" : null, + "state" : "active", + "web_url" : "https://gitlab.example.com/eileen.lowe", + "id" : 18, + "username" : "eileen.lowe" + }, + "start_date": null, + "start_date_is_fixed": false, + "start_date_fixed": null, + "start_date_from_milestones": null, + "end_date": "2018-07-31", + "due_date": "2018-07-31", + "due_date_is_fixed": false, + "due_date_fixed": null, + "due_date_from_milestones": "2018-07-31", + "created_at": "2018-07-17T13:36:22.770Z", + "updated_at": "2018-07-18T12:22:05.239Z", + "labels": [], + "upvotes": 4, + "downvotes": 0 +} +``` + +## Update epic + +Updates an epic. + +NOTE: **Note:** +Starting with GitLab [11.3][ee-6448], `start_date` and `end_date` should no longer be assigned +directly, as they now represent composite values. You can configure it via the `*_is_fixed` and +`*_fixed` fields instead. + +``` +PUT /groups/:id/epics/:epic_iid +``` + +| Attribute | Type | Required | Description | +| ------------------- | ---------------- | ---------- | ---------------------------------------------------------------------------------------| +| `id` | integer/string | yes | The ID or [URL-encoded path of the group](README.md#namespaced-path-encoding) owned by the authenticated user | +| `epic_iid` | integer/string | yes | The internal ID of the epic | +| `title` | string | no | The title of an epic | +| `description` | string | no | The description of an epic | +| `labels` | string | no | The comma separated list of labels | +| `start_date_is_fixed` | boolean | no | Whether start date should be sourced from `start_date_fixed` or from milestones (since 11.3) | +| `start_date_fixed` | string | no | The fixed start date of an epic (since 11.3) | +| `due_date_is_fixed` | boolean | no | Whether due date should be sourced from `due_date_fixed` or from milestones (since 11.3) | +| `due_date_fixed` | string | no | The fixed due date of an epic (since 11.3) | +| `state_event` | string | no | State event for an epic. Set `close` to close the epic and `reopen` to reopen it (since 11.4) | + +```bash +curl --header PUT "PRIVATE-TOKEN: <your_access_token>" https://gitlab.example.com/api/v4/groups/1/epics/5?title=New%20Title +``` + +Example response: + +```json +{ + "id": 33, + "iid": 6, + "group_id": 7, + "title": "New Title", + "description": "Epic description", + "author": { + "name" : "Alexandra Bashirian", + "avatar_url" : null, + "state" : "active", + "web_url" : "https://gitlab.example.com/eileen.lowe", + "id" : 18, + "username" : "eileen.lowe" + }, + "start_date": null, + "start_date_is_fixed": false, + "start_date_fixed": null, + "start_date_from_milestones": null, + "end_date": "2018-07-31", + "due_date": "2018-07-31", + "due_date_is_fixed": false, + "due_date_fixed": null, + "due_date_from_milestones": "2018-07-31", + "created_at": "2018-07-17T13:36:22.770Z", + "updated_at": "2018-07-18T12:22:05.239Z", + "labels": [], + "upvotes": 4, + "downvotes": 0 +} +``` + +## Delete epic + +Deletes an epic + +``` +DELETE /groups/:id/epics/:epic_iid +``` + +| Attribute | Type | Required | Description | +| ------------------- | ---------------- | ---------- | ---------------------------------------------------------------------------------------| +| `id` | integer/string | yes | The ID or [URL-encoded path of the group](README.md#namespaced-path-encoding) owned by the authenticated user | +| `epic_iid` | integer/string | yes | The internal ID of the epic. | + +```bash +curl --header DELETE "PRIVATE-TOKEN: <your_access_token>" https://gitlab.example.com/api/v4/groups/1/epics/5?title=New%20Title +``` + +## Create a todo + +Manually creates a todo for the current user on an epic. If +there already exists a todo for the user on that epic, status code `304` is +returned. + +``` +POST /groups/:id/epics/:epic_iid/todo +``` + +| Attribute | Type | Required | Description | +|-------------|---------|----------|--------------------------------------| +| `id` | integer/string | yes | The ID or [URL-encoded path of the group](README.md#namespaced-path-encoding) owned by the authenticated user | +| `epic_iid ` | integer | yes | The internal ID of a group's epic | + +```bash +curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" https://gitlab.example.com/api/v4/groups/1/epics/5/todo +``` + +Example response: + +```json +{ + "id": 112, + "group": { + "id": 1, + "name": "Gitlab", + "path": "gitlab", + "kind": "group", + "full_path": "base/gitlab", + "parent_id": null + }, + "author": { + "name": "Administrator", + "username": "root", + "id": 1, + "state": "active", + "avatar_url": "http://www.gravatar.com/avatar/e64c7d89f26bd1972efa854d13d7dd61?s=80&d=identicon", + "web_url": "https://gitlab.example.com/root" + }, + "action_name": "marked", + "target_type": "epic", + "target": { + "id": 30, + "iid": 5, + "group_id": 1, + "title": "Ea cupiditate dolores ut vero consequatur quasi veniam voluptatem et non.", + "description": "Molestias dolorem eos vitae expedita impedit necessitatibus quo voluptatum.", + "author":{ + "id": 7, + "name": "Pamella Huel", + "username": "arnita", + "state": "active", + "avatar_url": "http://www.gravatar.com/avatar/a2f5c6fcef64c9c69cb8779cb292be1b?s=80&d=identicon", + "web_url": "http://localhost:3001/arnita" + }, + "start_date": null, + "end_date": null, + "created_at": "2018-01-21T06:21:13.165Z", + "updated_at": "2018-01-22T12:41:41.166Z" + }, + "target_url": "https://gitlab.example.com/groups/epics/5", + "body": "Vel voluptas atque dicta mollitia adipisci qui at.", + "state": "pending", + "created_at": "2016-07-01T11:09:13.992Z" +} +``` + +[ee-6448]: https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/6448 diff --git a/doc/api/geo_nodes.md b/doc/api/geo_nodes.md new file mode 100644 index 00000000000..d5d16aa0a89 --- /dev/null +++ b/doc/api/geo_nodes.md @@ -0,0 +1,399 @@ +# Geo Nodes API **[PREMIUM ONLY]** + +In order to interact with Geo node endpoints, you need to authenticate yourself +as an admin. + +## Retrieve configuration about all Geo nodes + +``` +GET /geo_nodes +``` + +```bash +curl --header "PRIVATE-TOKEN: <your_access_token>" https://gitlab.example.com/api/v4/geo_nodes +``` + +Example response: + +```json +[ + { + "id": 1, + "url": "https://primary.example.com/", + "primary": true, + "enabled": true, + "current": true, + "files_max_capacity": 10, + "repos_max_capacity": 25, + "verification_max_capacity": 100, + "clone_protocol": "http" + }, + { + "id": 2, + "url": "https://secondary.example.com/", + "alternate_url": "https://alternate.example.com/", + "primary": false, + "enabled": true, + "current": false, + "files_max_capacity": 10, + "repos_max_capacity": 25, + "verification_max_capacity": 100, + "clone_protocol": "http" + } +] +``` + +## Retrieve configuration about a specific Geo node + +``` +GET /geo_nodes/:id +``` + +```bash +curl --header "PRIVATE-TOKEN: <your_access_token>" https://gitlab.example.com/api/v4/geo_nodes/1 +``` + +Example response: + +```json +{ + "id": 1, + "url": "https://primary.example.com/", + "primary": true, + "enabled": true, + "current": true, + "files_max_capacity": 10, + "repos_max_capacity": 25, + "verification_max_capacity": 100, + "clone_protocol": "http" +} +``` + +## Edit a Geo node + +Updates settings of an existing Geo node. + +_This can only be run against a primary Geo node._ + +``` +PUT /geo_nodes/:id +``` + +| Attribute | Type | Required | Description | +|----------------------|---------|-----------|---------------------------------------------------------------------------| +| `id` | integer | yes | The ID of the Geo node. | +| `enabled` | boolean | no | Flag indicating if the Geo node is enabled. | +| `url` | string | no | The URL to connect to the Geo node. | +| `alternate_url` | string | no | Allows users to log in to the secondary at an alternate URL (required for OAuth) | +| `files_max_capacity` | integer | no | Control the maximum concurrency of LFS/attachment backfill for this secondary node. | +| `repos_max_capacity` | integer | no | Control the maximum concurrency of repository backfill for this secondary node. | +| `verification_max_capacity` | integer | no | Control the maximum concurrency of verification for this node. | + +Example response: + +```json +{ + "id": 1, + "url": "https://secondary.example.com/", + "alternate_url": "https://alternate.example.com/", + "primary": false, + "enabled": true, + "current": true, + "files_max_capacity": 10, + "repos_max_capacity": 25, + "verification_max_capacity": 100, + "clone_protocol": "http" +} +``` + +## Delete a Geo node + +Removes the Geo node. + +NOTE: **Note:** +Only a Geo primary node will accept this request. + +``` +DELETE /geo_nodes/:id +``` + +| Attribute | Type | Required | Description | +|-----------|---------|----------|-------------------------| +| `id` | integer | yes | The ID of the Geo node. | + +## Repair a Geo node + +To repair the OAuth authentication of a Geo node. + +_This can only be run against a primary Geo node._ + +``` +POST /geo_nodes/:id/repair +``` + +Example response: + +```json +{ + "id": 1, + "url": "https://primary.example.com/", + "primary": true, + "enabled": true, + "current": true, + "files_max_capacity": 10, + "repos_max_capacity": 25, + "verification_max_capacity": 100, + "clone_protocol": "http" +} +``` + +## Retrieve status about all Geo nodes + +``` +GET /geo_nodes/status +``` + +```bash +curl --header "PRIVATE-TOKEN: <your_access_token>" https://gitlab.example.com/api/v4/geo_nodes/status +``` + +Example response: + +```json +[ + { + "geo_node_id": 1, + "healthy": true, + "health": "Healthy", + "health_status": "Healthy", + "missing_oauth_application": false, + "attachments_count": 1, + "attachments_synced_count": nil, + "attachments_failed_count": nil, + "attachments_synced_missing_on_primary_count": 0, + "attachments_synced_in_percentage": "0.00%", + "db_replication_lag_seconds": nil, + "lfs_objects_count": 0, + "lfs_objects_synced_count": nil, + "lfs_objects_failed_count": nil, + "lfs_objects_synced_missing_on_primary_count": 0, + "lfs_objects_synced_in_percentage": "0.00%", + "job_artifacts_count": 2, + "job_artifacts_synced_count": nil, + "job_artifacts_failed_count": nil, + "job_artifacts_synced_missing_on_primary_count": 0, + "job_artifacts_synced_in_percentage": "0.00%", + "repositories_count": 41, + "projects_count": 41, + "repositories_failed_count": nil, + "repositories_synced_count": nil, + "repositories_synced_in_percentage": "0.00%", + "wikis_count": 41, + "wikis_failed_count": nil, + "wikis_synced_count": nil, + "wikis_synced_in_percentage": "0.00%", + "replication_slots_count": 1, + "replication_slots_used_count": 1, + "replication_slots_used_in_percentage": "100.00%", + "replication_slots_max_retained_wal_bytes": 0, + "repositories_checked_count": 20, + "repositories_checked_failed_count": 20, + "repositories_checked_in_percentage": "100.00%", + "repositories_checksummed_count": 20, + "repositories_checksum_failed_count": 5, + "repositories_checksummed_in_percentage": "48.78%", + "wikis_checksummed_count": 10, + "wikis_checksum_failed_count": 3, + "wikis_checksummed_in_percentage": "24.39%", + "repositories_verified_count": 20, + "repositories_verification_failed_count": 5, + "repositories_verified_in_percentage": "48.78%", + "repositories_checksum_mismatch_count": 3, + "wikis_verified_count": 10, + "wikis_verification_failed_count": 3, + "wikis_verified_in_percentage": "24.39%", + "wikis_checksum_mismatch_count": 1, + "repositories_retrying_verification_count": 1, + "wikis_retrying_verification_count": 3, + "repositories_checked_count": 7, + "repositories_checked_failed_count": 2, + "repositories_checked_in_percentage": "17.07%", + "last_event_id": 23, + "last_event_timestamp": 1509681166, + "cursor_last_event_id": nil, + "cursor_last_event_timestamp": 0, + "last_successful_status_check_timestamp": 1510125024, + "version": "10.3.0", + "revision": "33d33a096a", + }, + { + "geo_node_id": 2, + "healthy": true, + "health": "Healthy", + "health_status": "Healthy", + "missing_oauth_application": false, + "attachments_count": 1, + "attachments_synced_count": 1, + "attachments_failed_count": 0, + "attachments_synced_missing_on_primary_count": 0, + "attachments_synced_in_percentage": "100.00%", + "db_replication_lag_seconds": 0, + "lfs_objects_count": 0, + "lfs_objects_synced_count": 0, + "lfs_objects_failed_count": 0, + "lfs_objects_synced_missing_on_primary_count": 0, + "lfs_objects_synced_in_percentage": "0.00%", + "job_artifacts_count": 2, + "job_artifacts_synced_count": 1, + "job_artifacts_failed_count": 1, + "job_artifacts_synced_missing_on_primary_count": 0, + "job_artifacts_synced_in_percentage": "50.00%", + "repositories_count": 41, + "projects_count": 41, + "repositories_failed_count": 1, + "repositories_synced_count": 40, + "repositories_synced_in_percentage": "97.56%", + "wikis_count": 41, + "wikis_failed_count": 0, + "wikis_synced_count": 41, + "wikis_synced_in_percentage": "100.00%", + "replication_slots_count": nil, + "replication_slots_used_count": nil, + "replication_slots_used_in_percentage": "0.00%", + "replication_slots_max_retained_wal_bytes": nil, + "repositories_checksummed_count": 20, + "repositories_checksum_failed_count": 5, + "repositories_checksummed_in_percentage": "48.78%", + "wikis_checksummed_count": 10, + "wikis_checksum_failed_count": 3, + "wikis_checksummed_in_percentage": "24.39%", + "repositories_verified_count": 20, + "repositories_verification_failed_count": 5, + "repositories_verified_in_percentage": "48.78%", + "repositories_checksum_mismatch_count": 3, + "wikis_verified_count": 10, + "wikis_verification_failed_count": 3, + "wikis_verified_in_percentage": "24.39%", + "wikis_checksum_mismatch_count": 1, + "repositories_retrying_verification_count": 4, + "wikis_retrying_verification_count": 2, + "repositories_checked_count": 5, + "repositories_checked_failed_count": 1, + "repositories_checked_in_percentage": "12.20%", + "last_event_id": 23, + "last_event_timestamp": 1509681166, + "cursor_last_event_id": 23, + "cursor_last_event_timestamp": 1509681166, + "last_successful_status_check_timestamp": 1510125024, + "version": "10.3.0", + "revision": "33d33a096a" + } +] +``` + +Note: fields `wikis_count` and `repositories_count` are deprecated and will be deleted soon. Please use `projects_count` instead. + +## Retrieve status about a specific Geo node + +``` +GET /geo_nodes/:id/status +``` + +```bash +curl --header "PRIVATE-TOKEN: <your_access_token>" https://gitlab.example.com/api/v4/geo_nodes/2/status +``` + +Example response: + +```json +{ + "geo_node_id": 2, + "healthy": true, + "health": "Healthy", + "health_status": "Healthy", + "missing_oauth_application": false, + "attachments_count": 1, + "attachments_synced_count": 1, + "attachments_failed_count": 0, + "attachments_synced_missing_on_primary_count": 0, + "attachments_synced_in_percentage": "100.00%", + "db_replication_lag_seconds": 0, + "lfs_objects_count": 0, + "lfs_objects_synced_count": 0, + "lfs_objects_failed_count": 0, + "lfs_objects_synced_missing_on_primary_count": 0, + "lfs_objects_synced_in_percentage": "0.00%", + "job_artifacts_count": 2, + "job_artifacts_synced_count": 1, + "job_artifacts_failed_count": 1, + "job_artifacts_synced_missing_on_primary_count": 0, + "job_artifacts_synced_in_percentage": "50.00%", + "repositories_count": 41, + "projects_count": 41, + "repositories_failed_count": 1, + "repositories_synced_count": 40, + "repositories_synced_in_percentage": "97.56%", + "wikis_count": 41, + "wikis_failed_count": 0, + "wikis_synced_count": 41, + "wikis_synced_in_percentage": "100.00%", + "replication_slots_count": nil, + "replication_slots_used_count": nil, + "replication_slots_used_in_percentage": "0.00%", + "replication_slots_max_retained_wal_bytes": nil, + "last_event_id": 23, + "last_event_timestamp": 1509681166, + "cursor_last_event_id": 23, + "cursor_last_event_timestamp": 1509681166, + "last_successful_status_check_timestamp": 1510125268, + "version": "10.3.0", + "revision": "33d33a096a" +} +``` + +Note: The `health_status` parameter can only be in an "Healthy" or "Unhealthy" state, while the `health` parameter can be empty, "Healthy", or contain the actual error message. + +Note: Fields `wikis_count` and `repositories_count` are deprecated and will be deleted soon. Please use `projects_count` instead. + +## Retrieve project sync or verification failures that occurred on the current node + +This only works on a secondary node. + +``` +GET /geo_nodes/current/failures +``` + +| Attribute | Type | Required | Description | +| --------- | ---- | -------- | ----------- | +| `type` | string | no | Type of failed objects (`repository`/`wiki`) | +| `failure_type` | string | no | Type of failures (`sync`/`checksum_mismatch`/`verification`) | + +This endpoint uses [Pagination](README.md#pagination). + +```bash +curl --header "PRIVATE-TOKEN: <your_access_token>" https://gitlab.example.com/api/v4/geo_nodes/current/failures +``` + +Example response: + +```json +[ + { + "project_id": 3, + "last_repository_synced_at": "2017-10-31 14:25:55 UTC", + "last_repository_successful_sync_at": "2017-10-31 14:26:04 UTC", + "last_wiki_synced_at": "2017-10-31 14:26:04 UTC", + "last_wiki_successful_sync_at": "2017-10-31 14:26:11 UTC", + "repository_retry_count": null, + "wiki_retry_count": 1, + "last_repository_sync_failure": null, + "last_wiki_sync_failure": "Error syncing Wiki repository", + "last_repository_verification_failure": "", + "last_wiki_verification_failure": "", + "repository_verification_checksum_sha": "da39a3ee5e6b4b0d32e5bfef9a601890afd80709", + "wiki_verification_checksum_sha": "da39a3ee5e6b4b0d3255bfef9ef0189aafd80709", + "repository_checksum_mismatch": false, + "wiki_checksum_mismatch": false + } +] +``` diff --git a/doc/api/group_boards.md b/doc/api/group_boards.md index 9b0ac23b41c..5cb4e46179b 100644 --- a/doc/api/group_boards.md +++ b/doc/api/group_boards.md @@ -27,7 +27,41 @@ Example response: [ { "id": 1, - "group_id": 5, + "name:": "group issue board", + "group": { + "id": 5, + "name": "Documentcloud", + "path": "documentcloud", + "owner_id": null, + "created_at": "2018-05-07T06:52:45.788Z", + "updated_at": "2018-07-03T06:48:17.005Z", + "description": "Consequatur aut a aperiam ut.", + "avatar": { + "url": null + }, + "membership_lock": false, + "share_with_group_lock": false, + "visibility_level": 20, + "request_access_enabled": false, + "ldap_sync_status": "ready", + "ldap_sync_error": null, + "ldap_sync_last_update_at": null, + "ldap_sync_last_successful_update_at": null, + "ldap_sync_last_sync_at": null, + "lfs_enabled": null, + "parent_id": null, + "shared_runners_minutes_limit": null, + "repository_size_limit": null, + "require_two_factor_authentication": false, + "two_factor_grace_period": 48, + "plan_id": null, + "project_creation_level": 2, + "runners_token": "rgeeL-nv4wa9YdRvuMid" + }, + "milestone": { + "id": 12 + "title": "10.0" + }, "lists" : [ { "id" : 1, @@ -83,7 +117,41 @@ Example response: ```json { "id": 1, - "group_id": 5, + "name:": "group issue board", + "group": { + "id": 5, + "name": "Documentcloud", + "path": "documentcloud", + "owner_id": null, + "created_at": "2018-05-07T06:52:45.788Z", + "updated_at": "2018-07-03T06:48:17.005Z", + "description": "Consequatur aut a aperiam ut.", + "avatar": { + "url": null + }, + "membership_lock": false, + "share_with_group_lock": false, + "visibility_level": 20, + "request_access_enabled": false, + "ldap_sync_status": "ready", + "ldap_sync_error": null, + "ldap_sync_last_update_at": null, + "ldap_sync_last_successful_update_at": null, + "ldap_sync_last_sync_at": null, + "lfs_enabled": null, + "parent_id": null, + "shared_runners_minutes_limit": null, + "repository_size_limit": null, + "require_two_factor_authentication": false, + "two_factor_grace_period": 48, + "plan_id": null, + "project_creation_level": 2, + "runners_token": "rgeeL-nv4wa9YdRvuMid" + }, + "milestone": { + "id": 12 + "title": "10.0" + }, "lists" : [ { "id" : 1, @@ -116,6 +184,206 @@ Example response: } ``` +## Create a board + +Creates a board. + +``` +POST /groups/:id/boards +``` + +| Attribute | Type | Required | Description | +| --------- | ---- | -------- | ----------- | +| `id` | integer/string | yes | The ID or [URL-encoded path of the group](README.md#namespaced-path-encoding) owned by the authenticated user | +| `name` | string | yes | The name of the new board | + +```bash +curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" https://gitlab.example.com/api/v4/groups/5/boards?name=newboard +``` + +Example response: + +```json + { + "id": 1, + "name": "newboard", + "group": { + "id": 5, + "name": "Documentcloud", + "path": "documentcloud", + "owner_id": null, + "created_at": "2018-05-07T06:52:45.788Z", + "updated_at": "2018-07-03T06:48:17.005Z", + "description": "Consequatur aut a aperiam ut.", + "avatar": { + "url": null + }, + "membership_lock": false, + "share_with_group_lock": false, + "visibility_level": 20, + "request_access_enabled": false, + "ldap_sync_status": "ready", + "ldap_sync_error": null, + "ldap_sync_last_update_at": null, + "ldap_sync_last_successful_update_at": null, + "ldap_sync_last_sync_at": null, + "lfs_enabled": null, + "parent_id": null, + "shared_runners_minutes_limit": null, + "repository_size_limit": null, + "require_two_factor_authentication": false, + "two_factor_grace_period": 48, + "plan_id": null, + "project_creation_level": 2, + "runners_token": "rgeeL-nv4wa9YdRvuMid" + }, + "milestone": { + "id": 12 + "title": "10.0" + }, + "lists" : [ + { + "id" : 1, + "label" : { + "name" : "Testing", + "color" : "#F0AD4E", + "description" : null + }, + "position" : 1 + }, + { + "id" : 2, + "label" : { + "name" : "Ready", + "color" : "#FF0000", + "description" : null + }, + "position" : 2 + }, + { + "id" : 3, + "label" : { + "name" : "Production", + "color" : "#FF5F00", + "description" : null + }, + "position" : 3 + } + ] + } +``` + +## Update a board + +> [Introduced][ee-5954] in GitLab 11.1. + +Updates a board. + +``` +PUT /groups/:id/boards/:board_id +``` + +| Attribute | Type | Required | Description | +| ------------------- | -------------- | -------- | ----------- | +| `id` | integer/string | yes | The ID or [URL-encoded path of the group](README.md#namespaced-path-encoding) owned by the authenticated user | +| `board_id` | integer | yes | The ID of a board | +| `name` | string | no | The new name of the board | +| `assignee_id` | integer | no | The assignee the board should be scoped to | +| `milestone_id` | integer | no | The milestone the board should be scoped to | +| `labels` | string | no | Comma-separated list of label names which the board should be scoped to | +| `weight` | integer | no | The weight range from 0 to 9, to which the board should be scoped to | + + +```bash +curl --request PUT --header "PRIVATE-TOKEN: <your_access_token>" https://gitlab.example.com/api/v4/groups/5/boards/1?name=new_name&milestone_id=44&assignee_id=1&labels=GroupLabel&weight=4 +``` + +Example response: + +```json + { + "id": 1, + "project": null, + "lists": [], + "name": "new_name", + "group": { + "id": 5, + "name": "Documentcloud", + "path": "documentcloud", + "owner_id": null, + "created_at": "2018-05-07T06:52:45.788Z", + "updated_at": "2018-07-03T06:48:17.005Z", + "description": "Consequatur aut a aperiam ut.", + "avatar": { + "url": null + }, + "membership_lock": false, + "share_with_group_lock": false, + "visibility_level": 20, + "request_access_enabled": false, + "ldap_sync_status": "ready", + "ldap_sync_error": null, + "ldap_sync_last_update_at": null, + "ldap_sync_last_successful_update_at": null, + "ldap_sync_last_sync_at": null, + "lfs_enabled": null, + "parent_id": null, + "shared_runners_minutes_limit": null, + "repository_size_limit": null, + "require_two_factor_authentication": false, + "two_factor_grace_period": 48, + "plan_id": null, + "project_creation_level": 2, + "runners_token": "rgeeL-nv3wa6YdRvuMid" + }, + "milestone": { + "id": 44, + "iid": 1, + "group_id": 5, + "title": "Group Milestone", + "description": "Group Milestone Desc", + "state": "active", + "created_at": "2018-07-03T07:15:19.271Z", + "updated_at": "2018-07-03T07:15:19.271Z", + "due_date": null, + "start_date": null, + "web_url": "http://example.com/groups/documentcloud/-/milestones/1" + }, + "assignee": { + "id": 1, + "name": "Administrator", + "username": "root", + "state": "active", + "avatar_url": "https://www.gravatar.com/avatar/e64c7d89f26bd1972efa854d13d7dd61?s=80&d=identicon", + "web_url": "http://example.com/root" + }, + "labels": [{ + "id": 11, + "name": "GroupLabel", + "color": "#428BCA", + "description": "" + }], + "weight": 4 + } +``` + +## Delete a board + +Deletes a board. + +``` +DELETE /groups/:id/boards/:board_id +``` + +| Attribute | Type | Required | Description | +| --------- | ---- | -------- | ----------- | +| `id` | integer/string | yes | The ID or [URL-encoded path of the group](README.md#namespaced-path-encoding) owned by the authenticated user | +| `board_id` | integer | yes | The ID of a board | + +```bash +curl --request DELETE --header "PRIVATE-TOKEN: <your_access_token>" https://gitlab.example.com/api/v4/groups/5/boards/1 +``` + ## List board lists Get a list of the board's lists. @@ -282,3 +550,5 @@ DELETE /groups/:id/boards/:board_id/lists/:list_id ```bash curl --request DELETE --header "PRIVATE-TOKEN: <your_access_token>" https://gitlab.example.com/api/v4/groups/5/boards/1/lists/1 ``` + +[ee-5954]: https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/5954 diff --git a/doc/api/groups.md b/doc/api/groups.md index 907b443d355..3be0937eaca 100644 --- a/doc/api/groups.md +++ b/doc/api/groups.md @@ -239,6 +239,7 @@ Example response: "full_path": "twitter", "file_template_project_id": 1, "parent_id": null, + "shared_runners_minutes_limit": 133, "projects": [ { "id": 7, @@ -418,6 +419,7 @@ Parameters: | `lfs_enabled` | boolean | no | Enable/disable Large File Storage (LFS) for the projects in this group | | `request_access_enabled` | boolean | no | Allow users to request member access. | | `parent_id` | integer | no | The parent group id for creating nested group. | +| `shared_runners_minutes_limit` | integer | no | (admin-only) Pipeline minutes quota for this group. | ## Transfer project to group @@ -448,10 +450,13 @@ PUT /groups/:id | `name` | string | no | The name of the group | | `path` | string | no | The path of the group | | `description` | string | no | The description of the group | +| `membership_lock` | boolean | no | Prevent adding new members to project membership within this group | +| `share_with_group_lock` | boolean | no | Prevent sharing a project with another group within this group | | `visibility` | string | no | The visibility level of the group. Can be `private`, `internal`, or `public`. | | `lfs_enabled` (optional) | boolean | no | Enable/disable Large File Storage (LFS) for the projects in this group | | `request_access_enabled` | boolean | no | Allow users to request member access. | | `file_template_project_id` | integer | no | **(Premium)** The ID of a project to load custom file templates from | +| `shared_runners_minutes_limit` | integer | no | (admin-only) Pipeline minutes quota for this group | ```bash curl --request PUT --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/groups/5?name=Experimental" @@ -519,7 +524,7 @@ Example response: ## Remove group -Removes group with all projects inside. +Removes group with all projects inside. Only available to group owners and administrators. ``` DELETE /groups/:id @@ -551,10 +556,62 @@ GET /groups?search=foobar ] ``` +## Sync group with LDAP + +Syncs the group with its linked LDAP group. Only available to group owners and administrators. + +``` +POST /groups/:id/ldap_sync +``` + +Parameters: + +- `id` (required) - The ID or path of a user group + ## Group members Please consult the [Group Members](members.md) documentation. +### Add LDAP group link + +Adds LDAP group link + +``` +POST /groups/:id/ldap_group_links +``` + +Parameters: + +- `id` (required) - The ID of a group +- `cn` (required) - The CN of a LDAP group +- `group_access` (required) - Minimum access level for members of the LDAP group +- `provider` (required) - LDAP provider for the LDAP group + +### Delete LDAP group link + +Deletes a LDAP group link + +``` +DELETE /groups/:id/ldap_group_links/:cn +``` + +Parameters: + +- `id` (required) - The ID of a group +- `cn` (required) - The CN of a LDAP group + +Deletes a LDAP group link for a specific LDAP provider + +``` +DELETE /groups/:id/ldap_group_links/:provider/:cn +``` + +Parameters: + +- `id` (required) - The ID of a group +- `cn` (required) - The CN of a LDAP group +- `provider` (required) - Name of a LDAP provider + ## Namespaces in groups By default, groups only get 20 namespaces at a time because the API results are paginated. diff --git a/doc/api/issue_links.md b/doc/api/issue_links.md new file mode 100644 index 00000000000..1c7db6a8e4c --- /dev/null +++ b/doc/api/issue_links.md @@ -0,0 +1,215 @@ +# Issue links API **[STARTER]** + +## List issue relations + +Get a list of related issues of a given issue, sorted by the relationship creation datetime (ascending). +Issues will be filtered according to the user authorizations. + +``` +GET /projects/:id/issues/:issue_iid/links +``` + +Parameters: + +| Attribute | Type | Required | Description | +|-------------|---------|----------|--------------------------------------| +| `id` | integer/string | yes | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding) owned by the authenticated user | +| `issue_iid` | integer | yes | The internal ID of a project's issue | + +```json +[ + { + "id" : 84, + "iid" : 14, + "issue_link_id": 1 + "project_id" : 4, + "created_at" : "2016-01-07T12:44:33.959Z", + "title" : "Issues with auth", + "state" : "opened", + "assignees" : [], + "assignee" : null, + "labels" : [ + "bug" + ], + "author" : { + "name" : "Alexandra Bashirian", + "avatar_url" : null, + "state" : "active", + "web_url" : "https://gitlab.example.com/eileen.lowe", + "id" : 18, + "username" : "eileen.lowe" + }, + "description" : null, + "updated_at" : "2016-01-07T12:44:33.959Z", + "milestone" : null, + "subscribed" : true, + "user_notes_count": 0, + "due_date": null, + "web_url": "http://example.com/example/example/issues/14", + "confidential": false, + "weight": null, + } +] +``` + +## Create an issue link + +Creates a two-way relation between two issues. User must be allowed to update both issues in order to succeed. + +``` +POST /projects/:id/issues/:issue_iid/links +``` + +| Attribute | Type | Required | Description | +|-------------|---------|----------|--------------------------------------| +| `id` | integer/string | yes | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding) owned by the authenticated user | +| `issue_iid` | integer | yes | The internal ID of a project's issue | +| `target_project_id` | integer/string | yes | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding) of a target project | +| `target_issue_iid` | integer/string | yes | The internal ID of a target project's issue | + + +```json +{ + "source_issue" : { + "id" : 83, + "iid" : 11, + "project_id" : 4, + "created_at" : "2016-01-07T12:44:33.959Z", + "title" : "Issues with auth", + "state" : "opened", + "assignees" : [], + "assignee" : null, + "labels" : [ + "bug" + ], + "author" : { + "name" : "Alexandra Bashirian", + "avatar_url" : null, + "state" : "active", + "web_url" : "https://gitlab.example.com/eileen.lowe", + "id" : 18, + "username" : "eileen.lowe" + }, + "description" : null, + "updated_at" : "2016-01-07T12:44:33.959Z", + "milestone" : null, + "subscribed" : true, + "user_notes_count": 0, + "due_date": null, + "web_url": "http://example.com/example/example/issues/11", + "confidential": false, + "weight": null, + }, + "target_issue" : { + "id" : 84, + "iid" : 14, + "project_id" : 4, + "created_at" : "2016-01-07T12:44:33.959Z", + "title" : "Issues with auth", + "state" : "opened", + "assignees" : [], + "assignee" : null, + "labels" : [ + "bug" + ], + "author" : { + "name" : "Alexandra Bashirian", + "avatar_url" : null, + "state" : "active", + "web_url" : "https://gitlab.example.com/eileen.lowe", + "id" : 18, + "username" : "eileen.lowe" + }, + "description" : null, + "updated_at" : "2016-01-07T12:44:33.959Z", + "milestone" : null, + "subscribed" : true, + "user_notes_count": 0, + "due_date": null, + "web_url": "http://example.com/example/example/issues/14", + "confidential": false, + "weight": null, + } +} +``` + +## Delete an issue link + +Deletes an issue link, thus removes the two-way relationship. + +``` +DELETE /projects/:id/issues/:issue_iid/links/:issue_link_id +``` + + +| Attribute | Type | Required | Description | +|-------------|---------|----------|--------------------------------------| +| `id` | integer/string | yes | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding) owned by the authenticated user | +| `issue_iid` | integer | yes | The internal ID of a project's issue | +| `issue_link_id` | integer/string | yes | The ID of an issue relationship | + + +```json +{ + "source_issue" : { + "id" : 83, + "iid" : 11, + "project_id" : 4, + "created_at" : "2016-01-07T12:44:33.959Z", + "title" : "Issues with auth", + "state" : "opened", + "assignees" : [], + "assignee" : null, + "labels" : [ + "bug" + ], + "author" : { + "name" : "Alexandra Bashirian", + "avatar_url" : null, + "state" : "active", + "web_url" : "https://gitlab.example.com/eileen.lowe", + "id" : 18, + "username" : "eileen.lowe" + }, + "description" : null, + "updated_at" : "2016-01-07T12:44:33.959Z", + "milestone" : null, + "subscribed" : true, + "user_notes_count": 0, + "due_date": null, + "web_url": "http://example.com/example/example/issues/11", + "confidential": false, + "weight": null, + }, + "target_issue" : { + "id" : 84, + "iid" : 14, + "project_id" : 4, + "created_at" : "2016-01-07T12:44:33.959Z", + "title" : "Issues with auth", + "state" : "opened", + "assignees" : [], + "assignee" : null, + "labels" : [ + "bug" + ], + "author" : { + "name" : "Alexandra Bashirian", + "avatar_url" : null, + "state" : "active", + "web_url" : "https://gitlab.example.com/eileen.lowe", + "id" : 18, + "username" : "eileen.lowe" + }, + "description" : null, + "updated_at" : "2016-01-07T12:44:33.959Z", + "milestone" : null, + "subscribed" : true, + "user_notes_count": 0, + "due_date": null, + "web_url": "http://example.com/example/example/issues/14", + "confidential": false, + "weight": null, + } +} +``` diff --git a/doc/api/issues.md b/doc/api/issues.md index cb5789e76b7..b5790fb994b 100644 --- a/doc/api/issues.md +++ b/doc/api/issues.md @@ -44,6 +44,7 @@ GET /issues?confidential=true | `author_id` | integer | no | Return issues created by the given user `id`. Combine with `scope=all` or `scope=assigned_to_me`. _([Introduced][ce-13004] in GitLab 9.5)_ | | `assignee_id` | integer | no | Return issues assigned to the given user `id`. `None` returns unassigned issues. `Any` returns issues with an assignee. _([Introduced][ce-13004] in GitLab 9.5)_ | | `my_reaction_emoji` | string | no | Return issues reacted by the authenticated user by the given `emoji`. `None` returns issues not given a reaction. `Any` returns issues given at least one reaction. _([Introduced][ce-14016] in GitLab 10.0)_ | +| `weight` | integer | no | Return issues with the specified `weight`. `None` returns issues with no weight assigned. `Any` returns issues with a weight assigned. | | `iids[]` | Array[integer] | no | Return only the issues having the given `iid` | | `order_by` | string | no | Return issues ordered by `created_at` or `updated_at` fields. Default is `created_at` | | `sort` | string | no | Return issues sorted in `asc` or `desc` order. Default is `desc` | @@ -116,14 +117,15 @@ Example response: "user_notes_count": 1, "due_date": "2016-07-22", "web_url": "http://example.com/example/example/issues/6", + "confidential": false, + "weight": null, + "discussion_locked": false, "time_stats": { "time_estimate": 0, "total_time_spent": 0, "human_time_estimate": null, "human_total_time_spent": null }, - "confidential": false, - "discussion_locked": false } ] ``` @@ -164,6 +166,7 @@ GET /groups/:id/issues?confidential=true | `author_id` | integer | no | Return issues created by the given user `id` _([Introduced][ce-13004] in GitLab 9.5)_ | | `assignee_id` | integer | no | Return issues assigned to the given user `id`. `None` returns unassigned issues. `Any` returns issues with an assignee. _([Introduced][ce-13004] in GitLab 9.5)_ | | `my_reaction_emoji` | string | no | Return issues reacted by the authenticated user by the given `emoji`. `None` returns issues not given a reaction. `Any` returns issues given at least one reaction. _([Introduced][ce-14016] in GitLab 10.0)_ | +| `weight` | integer | no | Return issues with the specified `weight`. `None` returns issues with no weight assigned. `Any` returns issues with a weight assigned. | | `order_by` | string | no | Return issues ordered by `created_at` or `updated_at` fields. Default is `created_at` | | `sort` | string | no | Return issues sorted in `asc` or `desc` order. Default is `desc` | | `search` | string | no | Search group issues against their `title` and `description` | @@ -234,14 +237,15 @@ Example response: "user_notes_count": 1, "due_date": null, "web_url": "http://example.com/example/example/issues/1", + "confidential": false, + "weight": null, + "discussion_locked": false, "time_stats": { "time_estimate": 0, "total_time_spent": 0, "human_time_estimate": null, "human_total_time_spent": null }, - "confidential": false, - "discussion_locked": false } ] ``` @@ -282,6 +286,7 @@ GET /projects/:id/issues?confidential=true | `author_id` | integer | no | Return issues created by the given user `id` _([Introduced][ce-13004] in GitLab 9.5)_ | | `assignee_id` | integer | no | Return issues assigned to the given user `id`. `None` returns unassigned issues. `Any` returns issues with an assignee. _([Introduced][ce-13004] in GitLab 9.5)_ | | `my_reaction_emoji` | string | no | Return issues reacted by the authenticated user by the given `emoji`. `None` returns issues not given a reaction. `Any` returns issues given at least one reaction. _([Introduced][ce-14016] in GitLab 10.0)_ | +| `weight` | integer | no | Return issues with the specified `weight`. `None` returns issues with no weight assigned. `Any` returns issues with a weight assigned. | | `order_by` | string | no | Return issues ordered by `created_at` or `updated_at` fields. Default is `created_at` | | `sort` | string | no | Return issues sorted in `asc` or `desc` order. Default is `desc` | | `search` | string | no | Search project issues against their `title` and `description` | @@ -360,14 +365,15 @@ Example response: "user_notes_count": 1, "due_date": "2016-07-22", "web_url": "http://example.com/example/example/issues/1", + "confidential": false, + "weight": null, + "discussion_locked": false, "time_stats": { "time_estimate": 0, "total_time_spent": 0, "human_time_estimate": null, "human_total_time_spent": null }, - "confidential": false, - "discussion_locked": false } ] ``` @@ -458,6 +464,7 @@ Example response: "human_total_time_spent": null }, "confidential": false, + "weight": null, "discussion_locked": false, "_links": { "self": "http://example.com/api/v4/projects/1/issues/2", @@ -484,16 +491,17 @@ POST /projects/:id/issues |-------------------------------------------|----------------|----------|--------------| | `id` | integer/string | yes | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding) owned by the authenticated user | | `iid` | integer/string | no | The internal ID of the project's issue (requires admin or project owner rights) | -| `title` | string | yes | The title of an issue | -| `description` | string | no | The description of an issue | -| `confidential` | boolean | no | Set an issue to be confidential. Default is `false`. | -| `assignee_ids` | Array[integer] | no | The ID of the users to assign issue | -| `milestone_id` | integer | no | The global ID of a milestone to assign issue | -| `labels` | string | no | Comma-separated label names for an issue | +| `title` | string | yes | The title of an issue | +| `description` | string | no | The description of an issue | +| `confidential` | boolean | no | Set an issue to be confidential. Default is `false`. | +| `assignee_ids` | Array[integer] | no | The ID of a user to assign issue | +| `milestone_id` | integer | no | The global ID of a milestone to assign issue | +| `labels` | string | no | Comma-separated label names for an issue | | `created_at` | string | no | Date time string, ISO 8601 formatted, e.g. `2016-03-11T03:45:40Z` (requires admin or project/group owner rights) | -| `due_date` | string | no | Date time string in the format YEAR-MONTH-DAY, e.g. `2016-03-11` | -| `merge_request_to_resolve_discussions_of` | integer | no | The IID of a merge request in which to resolve all issues. This will fill the issue with a default description and mark all discussions as resolved. When passing a description or title, these values will take precedence over the default values.| -| `discussion_to_resolve` | string | no | The ID of a discussion to resolve. This will fill in the issue with a default description and mark the discussion as resolved. Use in combination with `merge_request_to_resolve_discussions_of`. | +| `due_date` | string | no | Date time string in the format YEAR-MONTH-DAY, e.g. `2016-03-11` | +| `merge_request_to_resolve_discussions_of` | integer | no | The IID of a merge request in which to resolve all issues. This will fill the issue with a default description and mark all discussions as resolved. When passing a description or title, these values will take precedence over the default values.| +| `discussion_to_resolve` | string | no | The ID of a discussion to resolve. This will fill in the issue with a default description and mark the discussion as resolved. Use in combination with `merge_request_to_resolve_discussions_of`. | +| `weight` | integer | no | The weight of the issue. Valid values are greater than or equal to 0. | ```bash curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" https://gitlab.example.com/api/v4/projects/4/issues?title=Issues%20with%20auth&labels=bug @@ -541,6 +549,7 @@ Example response: "human_total_time_spent": null }, "confidential": false, + "weight": null, "discussion_locked": false, "_links": { "self": "http://example.com/api/v4/projects/1/issues/2", @@ -577,6 +586,7 @@ PUT /projects/:id/issues/:issue_iid | `state_event` | string | no | The state event of an issue. Set `close` to close the issue and `reopen` to reopen it | | `updated_at` | string | no | Date time string, ISO 8601 formatted, e.g. `2016-03-11T03:45:40Z` (requires admin or project owner rights) | | `due_date` | string | no | Date time string in the format YEAR-MONTH-DAY, e.g. `2016-03-11` | +| `weight` | integer | no | The weight of the issue. Valid values are greater than or equal to 0. 0 | | `discussion_locked` | boolean | no | Flag indicating if the issue's discussion is locked. If the discussion is locked only project members can add or edit comments. | ```bash @@ -632,6 +642,7 @@ Example response: "human_total_time_spent": null }, "confidential": false, + "weight": null, "discussion_locked": false, "_links": { "self": "http://example.com/api/v4/projects/1/issues/2", @@ -738,6 +749,7 @@ Example response: "human_total_time_spent": null }, "confidential": false, + "weight": null, "discussion_locked": false, "_links": { "self": "http://example.com/api/v4/projects/1/issues/2", @@ -823,6 +835,7 @@ Example response: "human_total_time_spent": null }, "confidential": false, + "weight": null, "discussion_locked": false, "_links": { "self": "http://example.com/api/v4/projects/1/issues/2", @@ -993,6 +1006,7 @@ Example response: "due_date": null, "web_url": "http://example.com/example/example/issues/110", "confidential": false, + "weight": null, "discussion_locked": false }, "target_url": "https://gitlab.example.com/gitlab-org/gitlab-ci/issues/10", diff --git a/doc/api/jobs.md b/doc/api/jobs.md index 877cd99723a..e9dd6811685 100644 --- a/doc/api/jobs.md +++ b/doc/api/jobs.md @@ -341,30 +341,58 @@ Example of response > **Notes**: > > - [Introduced][ce-2893] in GitLab 8.5. +> - The use of `CI_JOB_TOKEN` in the artifacts download API was [introduced][ee-2346] +> in [GitLab Premium][ee] 9.5. -Get job artifacts of a project. +Get the job's artifacts zipped archive of a project. ``` GET /projects/:id/jobs/:job_id/artifacts ``` -| Attribute | Type | Required | Description | -|-----------|----------------|----------|------------------------------------------------------------------------------------------------------------------| -| `id` | integer/string | yes | ID or [URL-encoded path of the project](README.md#namespaced-path-encoding) owned by the authenticated user. | -| `job_id` | integer | yes | ID of a job. | +| Attribute | Type | Required | Description | +|-------------|----------------|----------|-------------------------------------------------------------------------------------------------------------------------------------------------| +| `id` | integer/string | yes | ID or [URL-encoded path of the project](README.md#namespaced-path-encoding) owned by the authenticated user. | +| `job_id` | integer | yes | ID of a job. | +| `job_token` **[PREMIUM]** | string | no | To be used with [triggers] for multi-project pipelines. It should be invoked only inside `.gitlab-ci.yml`. Its value is always `$CI_JOB_TOKEN`. | -Example requests: +Example request using the `PRIVATE-TOKEN` header: ```sh -curl --location --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/1/jobs/8/artifacts" +curl --output artifacts.zip --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/1/jobs/42/artifacts" ``` +To use this in a [`script` definition](../ci/yaml/README.md#script) inside +`.gitlab-ci.yml`, you can use either: + +- The `JOB-TOKEN` header with the GitLab-provided `CI_JOB_TOKEN` variable. + For example, the following job will download the artifacts of the job with ID + `42`. Note that the command is wrapped into single quotes since it contains a + colon (`:`): + + ```yaml + artifact_download: + stage: test + script: + - 'curl --location --output artifacts.zip --header "JOB-TOKEN: $CI_JOB_TOKEN" "https://gitlab.example.com/api/v4/projects/1/jobs/42/artifacts"' + ``` + +- Or the `job_token` attribute with the GitLab-provided `CI_JOB_TOKEN` variable. + For example, the following job will download the artifacts of the job with ID `42`: + + ```yaml + artifact_download: + stage: test + script: + - 'curl --location --output artifacts.zip "https://gitlab.example.com/api/v4/projects/1/jobs/42/artifacts?job_token=$CI_JOB_TOKEN"' + ``` + Possible response status codes: | Status | Description | |-----------|---------------------------------| -| 200 | Serves the artifacts file | -| 404 | Build not found or no artifacts | +| 200 | Serves the artifacts file. | +| 404 | Build not found or no artifacts.| [ce-2893]: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/2893 @@ -373,9 +401,13 @@ Possible response status codes: > **Notes**: > > - [Introduced][ce-5347] in GitLab 8.10. +> - The use of `CI_JOB_TOKEN` in the artifacts download API was [introduced][ee-2346] +> in [GitLab Premium][ee] 9.5. -Download the artifacts archive from the given reference name and job provided the -job finished successfully. +Download the artifacts zipped archive from the given reference name and job, +provided the job finished successfully. This is the same as +[getting the job's artifacts](#get-job-artifacts), but by defining the job's +name instead of its ID. ``` GET /projects/:id/jobs/artifacts/:ref_name/download?job=name @@ -383,24 +415,51 @@ GET /projects/:id/jobs/artifacts/:ref_name/download?job=name Parameters -| Attribute | Type | Required | Description | -|------------|----------------|----------|------------------------------------------------------------------------------------------------------------------| -| `id` | integer/string | yes | ID or [URL-encoded path of the project](README.md#namespaced-path-encoding) owned by the authenticated user. | -| `ref_name` | string | yes | Branch or tag name in repository. HEAD or SHA references are not supported. | -| `job` | string | yes | The name of the job. | +| Attribute | Type | Required | Description | +|-------------|----------------|----------|-------------------------------------------------------------------------------------------------------------------------------------------------| +| `id` | integer/string | yes | ID or [URL-encoded path of the project](README.md#namespaced-path-encoding) owned by the authenticated user. | +| `ref_name` | string | yes | Branch or tag name in repository. HEAD or SHA references are not supported. | +| `job` | string | yes | The name of the job. | +| `job_token` **[PREMIUM]** | string | no | To be used with [triggers] for multi-project pipelines. It should be invoked only inside `.gitlab-ci.yml`. Its value is always `$CI_JOB_TOKEN`. | -Example requests: +Example request using the `PRIVATE-TOKEN` header: ```sh curl --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/1/jobs/artifacts/master/download?job=test" ``` +To use this in a [`script` definition](../ci/yaml/README.md#script) inside +`.gitlab-ci.yml`, you can use either: + +- The `JOB-TOKEN` header with the GitLab-provided `CI_JOB_TOKEN` variable. + For example, the following job will download the artifacts of the `test` job + of the `master` branch. Note that the command is wrapped into single quotes + since it contains a colon (`:`): + + ```yaml + artifact_download: + stage: test + script: + - 'curl --location --output artifacts.zip --header "JOB-TOKEN: $CI_JOB_TOKEN" "https://gitlab.example.com/api/v4/projects/$CI_PROJECT_ID/jobs/artifacts/master/download?job=test"' + ``` + +- Or the `job_token` attribute with the GitLab-provided `CI_JOB_TOKEN` variable. + For example, the following job will download the artifacts of the `test` job + of the `master` branch: + + ```yaml + artifact_download: + stage: test + script: + - 'curl --location --output artifacts.zip "https://gitlab.example.com/api/v4/projects/$CI_PROJECT_ID/jobs/artifacts/master/download?job=test&job_token=$CI_JOB_TOKEN"' + ``` + Possible response status codes: | Status | Description | |-----------|---------------------------------| -| 200 | Serves the artifacts file | -| 404 | Build not found or no artifacts | +| 200 | Serves the artifacts file. | +| 404 | Build not found or no artifacts.| [ce-5347]: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/5347 @@ -409,7 +468,7 @@ Possible response status codes: > Introduced in GitLab 10.0 Download a single artifact file from a job with a specified ID from within -the job's artifacts archive. The file is extracted from the archive and +the job's artifacts zipped archive. The file is extracted from the archive and streamed to the client. ``` @@ -773,3 +832,7 @@ Example of response "user": null } ``` + +[ee]: https://about.gitlab.com/pricing/ +[ee-2346]: https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/2346 +[triggers]: ../ci/triggers/README.md#when-a-pipeline-depends-on-the-artifacts-of-another-pipeline-premium diff --git a/doc/api/license.md b/doc/api/license.md new file mode 100644 index 00000000000..2a8de64bdbf --- /dev/null +++ b/doc/api/license.md @@ -0,0 +1,176 @@ +# License **[CORE ONLY]** + +In order to interact with license endpoints, you need to authenticate yourself +as an admin. + +## Retrieve information about the current license + +``` +GET /license +``` + +```json +{ + "id": 2, + "plan": "gold", + "created_at": "2018-02-27T23:21:58.674Z", + "starts_at": "2018-01-27", + "expires_at": "2022-01-27", + "historical_max": 300, + "expired": false, + "overage": 200, + "user_limit": 100, + "active_users": 300, + "licensee": { + "Name": "John Doe1" + }, + "add_ons": { + "GitLab_FileLocks": 1, + "GitLab_Auditor_User": 1 + } +} +``` + +## Retrieve information about all licenses + +``` +GET /licenses +``` + +```json +[ + { + "id": 1, + "plan": "silver", + "created_at": "2018-02-27T23:21:58.674Z", + "starts_at": "2018-01-27", + "expires_at": "2022-01-27", + "historical_max": 300, + "expired": false, + "overage": 200, + "user_limit": 100, + "licensee": { + "Name": "John Doe1" + }, + "add_ons": { + "GitLab_FileLocks": 1, + "GitLab_Auditor_User": 1 + } + }, + { + "id": 2, + "plan": "gold", + "created_at": "2018-02-27T23:21:58.674Z", + "starts_at": "2018-01-27", + "expires_at": "2022-01-27", + "historical_max": 300, + "expired": false, + "overage": 200, + "user_limit": 100, + "licensee": { + "Name": "Doe John" + }, + "add_ons": { + "GitLab_FileLocks": 1, + } + } +] +``` + +Overage is the difference between the number of active users and the licensed number of users. +This is calculated differently depending on whether the license has expired or not. + +- If the license has expired, it uses the historical maximum active user count (`historical_max`). +- If the license has not expired, it uses the current active users count. + +Returns: + +- `200 OK` with response containing the licenses in JSON format. This will be an empty JSON array if there are no licenses. +- `403 Forbidden` if the current user in not permitted to read the licenses. + +## Add a new license + +``` +POST /license +``` + +| Attribute | Type | Required | Description | +| --------- | ---- | -------- | ----------- | +| `license` | string | yes | The license string | + +```bash +curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/license?license=eyJkYXRhIjoiMHM5Q...S01Udz09XG4ifQ==" +``` + +Example response: + +```json +{ + "id": 1, + "plan": "gold", + "created_at": "2018-02-27T23:21:58.674Z", + "starts_at": "2018-01-27", + "expires_at": "2022-01-27", + "historical_max": 300, + "expired": false, + "overage": 200, + "user_limit": 100, + "active_users": 300, + "licensee": { + "Name": "John Doe1" + }, + "add_ons": { + "GitLab_FileLocks": 1, + "GitLab_Auditor_User": 1 + } +} +``` + +Returns: + +- `201 Created` if the license is successfully added. +- `400 Bad Request` if the license couldn't be added, with an error message explaining the reason. + + +## Delete a license + +``` +DELETE /license/:id +``` + +| Attribute | Type | Required | Description | +| --------- | ---- | -------- | ----------- | +| `id` | integer | yes | ID of the GitLab license. | + +```bash +curl --request DELETE --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/license/:id" +``` + +Example response: + +```json +{ + "id": 2, + "plan": "gold", + "created_at": "2018-02-27T23:21:58.674Z", + "starts_at": "2018-01-27", + "expires_at": "2022-01-27", + "historical_max": 300, + "expired": false, + "overage": 200, + "user_limit": 100, + "licensee": { + "Name": "John Doe" + }, + "add_ons": { + "GitLab_FileLocks": 1, + "GitLab_Auditor_User": 1 + } +} +``` + +Returns: + +- `204 No Content` if the license is successfully deleted. +- `403 Forbidden` if the current user in not permitted to delete the license. +- `404 Not Found` if the license to delete could not be found. diff --git a/doc/api/license_templates.md b/doc/api/license_templates.md new file mode 100644 index 00000000000..1b68af9ce31 --- /dev/null +++ b/doc/api/license_templates.md @@ -0,0 +1,5 @@ +--- +redirect_to: 'templates/licenses.md' +--- + +This document was moved to [another location](templates/licenses.md). diff --git a/doc/api/managed_licenses.md b/doc/api/managed_licenses.md new file mode 100644 index 00000000000..47b193111b6 --- /dev/null +++ b/doc/api/managed_licenses.md @@ -0,0 +1,136 @@ +# Managed Licenses API **[ULTIMATE]** + +## List managed licenses + +Get all managed licenses for a given project. + +``` +GET /projects/:id/managed_licenses +``` + +| Attribute | Type | Required | Description | +| --------- | ------- | -------- | --------------------- | +| `id` | integer/string | yes | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding) | + +```bash +curl --header "PRIVATE-TOKEN: <your_access_token>" https://gitlab.example.com/api/v4/projects/1/managed_licenses +``` + +Example response: + +```json +[ + { + "id": 1, + "name": "MIT", + "approval_status": "approved" + }, + { + "id": 3, + "name": "ISC", + "approval_status": "blacklisted" + } +] +``` + +## Show an existing managed license + +Shows an existing managed license. + +``` +GET /projects/:id/managed_licenses/:managed_license_id +``` + +| Attribute | Type | Required | Description | +| --------------- | ------- | --------------------------------- | ------------------------------- | +| `id` | integer/string | yes | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding) owned by the authenticated user | +| `managed_license_id` | integer/string | yes | The ID or URL-encoded name of the license belonging to the project | + +```bash +curl --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/1/managed_licenses/6" +``` + +Example response: + +```json +{ + "id": 1, + "name": "MIT", + "approval_status": "blacklisted" +} +``` + +## Create a new managed license + +Creates a new managed license for the given project with the given name and approval status. + +``` +POST /projects/:id/managed_licenses +``` + +| Attribute | Type | Required | Description | +| ------------- | ------- | -------- | ---------------------------- | +| `id` | integer/string | yes | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding) owned by the authenticated user | +| `name` | string | yes | The name of the managed license | +| `approval_status` | string | yes | The approval status. "approved" or "blacklisted" | + +```bash +curl --data "name=MIT&approval_status=blacklisted" --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/1/managed_licenses" +``` + +Example response: + +```json +{ + "id": 1, + "name": "MIT", + "approval_status": "approved" +} +``` + +## Delete a managed license + +Deletes a managed license with a given id. + +``` +DELETE /projects/:id/managed_licenses/:managed_license_id +``` + +| Attribute | Type | Required | Description | +| --------- | ------- | -------- | --------------------- | +| `id` | integer/string | yes | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding) owned by the authenticated user | +| `managed_license_id` | integer/string | yes | The ID or URL-encoded name of the license belonging to the project | + +```bash +curl --request DELETE --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/1/managed_licenses/4" +``` + +When successful, it replies with an HTTP 204 response. + +## Edit an existing managed license + +Updates an existing managed license with a new approval status. + +``` +PATCH /projects/:id/managed_licenses/:managed_license_id +``` + +| Attribute | Type | Required | Description | +| --------------- | ------- | --------------------------------- | ------------------------------- | +| `id` | integer/string | yes | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding) owned by the authenticated user | +| `managed_license_id` | integer/string | yes | The ID or URL-encoded name of the license belonging to the project | +| `approval_status` | string | yes | The approval status. "approved" or "blacklisted" | + +```bash +curl --request PATCH --data "approval_status=blacklisted" --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/1/managed_licenses/6" +``` + +Example response: + +```json +{ + "id": 1, + "name": "MIT", + "approval_status": "blacklisted" +} +``` diff --git a/doc/api/merge_request_approvals.md b/doc/api/merge_request_approvals.md new file mode 100644 index 00000000000..d0a338dbdb9 --- /dev/null +++ b/doc/api/merge_request_approvals.md @@ -0,0 +1,422 @@ +# Merge request approvals API **[STARTER]** + +Configuration for approvals on all Merge Requests (MR) in the project. Must be authenticated for all endpoints. + +## Project-level MR approvals + +### Get Configuration + +>**Note:** This API endpoint is only available on 10.6 Starter and above. + +You can request information about a project's approval configuration using the +following endpoint: + +``` +GET /projects/:id/approvals +``` + +**Parameters:** + +| Attribute | Type | Required | Description | +| --------- | ------- | -------- | ------------------- | +| `id` | integer | yes | The ID of a project | + +```json +{ + "approvers": [ + { + "user": { + "id": 5, + "name": "John Doe6", + "username": "user5", + "state":"active","avatar_url":"https://www.gravatar.com/avatar/4aea8cf834ed91844a2da4ff7ae6b491?s=80\u0026d=identicon","web_url":"http://localhost/user5" + } + } + ], + "approver_groups": [ + { + "group": { + "id": 1, + "name": "group1", + "path": "group1", + "description": "", + "visibility": "public", + "lfs_enabled": false, + "avatar_url": null, + "web_url": "http://localhost/groups/group1", + "request_access_enabled": false, + "full_name": "group1", + "full_path": "group1", + "parent_id": null, + "ldap_cn": null, + "ldap_access": null + } + } + ], + "approvals_before_merge": 2, + "reset_approvals_on_push": true, + "disable_overriding_approvers_per_merge_request": false +} +``` + +### Change configuration + +>**Note:** This API endpoint is only available on 10.6 Starter and above. + +If you are allowed to, you can change approval configuration using the following +endpoint: + +``` +POST /projects/:id/approvals +``` + +**Parameters:** + +| Attribute | Type | Required | Description | +| ------------------------------------------------ | ------- | -------- | ---------------------------------------------------------- | +| `id` | integer | yes | The ID of a project | +| `approvals_before_merge` | integer | no | How many approvals are required before an MR can be merged | +| `reset_approvals_on_push` | boolean | no | Reset approvals on a new push | +| `disable_overriding_approvers_per_merge_request` | boolean | no | Allow/Disallow overriding approvers per MR | + +```json +{ + "approvers": [ + { + "user": { + "id": 5, + "name": "John Doe6", + "username": "user5", + "state":"active","avatar_url":"https://www.gravatar.com/avatar/4aea8cf834ed91844a2da4ff7ae6b491?s=80\u0026d=identicon","web_url":"http://localhost/user5" + } + } + ], + "approver_groups": [ + { + "group": { + "id": 1, + "name": "group1", + "path": "group1", + "description": "", + "visibility": "public", + "lfs_enabled": false, + "avatar_url": null, + "web_url": "http://localhost/groups/group1", + "request_access_enabled": false, + "full_name": "group1", + "full_path": "group1", + "parent_id": null, + "ldap_cn": null, + "ldap_access": null + } + } + ], + "approvals_before_merge": 2, + "reset_approvals_on_push": true, + "disable_overriding_approvers_per_merge_request": false +} +``` + +### Change allowed approvers + +>**Note:** This API endpoint is only available on 10.6 Starter and above. + +If you are allowed to, you can change approvers and approver groups using +the following endpoint: + +``` +PUT /projects/:id/approvers +``` + +**Important:** Approvers and groups not in the request will be **removed** + +**Parameters:** + +| Attribute | Type | Required | Description | +| -------------------- | ------- | -------- | --------------------------------------------------- | +| `id` | integer | yes | The ID of a project | +| `approver_ids` | Array | yes | An array of User IDs that can approve MRs | +| `approver_group_ids` | Array | yes | An array of Group IDs whose members can approve MRs | + +```json +{ + "approvers": [ + { + "user": { + "id": 5, + "name": "John Doe6", + "username": "user5", + "state":"active","avatar_url":"https://www.gravatar.com/avatar/4aea8cf834ed91844a2da4ff7ae6b491?s=80\u0026d=identicon","web_url":"http://localhost/user5" + } + } + ], + "approver_groups": [ + { + "group": { + "id": 1, + "name": "group1", + "path": "group1", + "description": "", + "visibility": "public", + "lfs_enabled": false, + "avatar_url": null, + "web_url": "http://localhost/groups/group1", + "request_access_enabled": false, + "full_name": "group1", + "full_path": "group1", + "parent_id": null, + "ldap_cn": null, + "ldap_access": null + } + } + ], + "approvals_before_merge": 2, + "reset_approvals_on_push": true, + "disable_overriding_approvers_per_merge_request": false +} +``` + + +## Merge Request-level MR approvals + +Configuration for approvals on a specific Merge Request. Must be authenticated for all endpoints. + +### Get Configuration + +>**Note:** This API endpoint is only available on 8.9 Starter and above. + +You can request information about a merge request's approval status using the +following endpoint: + +``` +GET /projects/:id/merge_requests/:merge_request_iid/approvals +``` + +**Parameters:** + +| Attribute | Type | Required | Description | +|---------------------|---------|----------|---------------------| +| `id` | integer | yes | The ID of a project | +| `merge_request_iid` | integer | yes | The IID of MR | + +```json +{ + "id": 5, + "iid": 5, + "project_id": 1, + "title": "Approvals API", + "description": "Test", + "state": "opened", + "created_at": "2016-06-08T00:19:52.638Z", + "updated_at": "2016-06-08T21:20:42.470Z", + "merge_status": "cannot_be_merged", + "approvals_required": 2, + "approvals_left": 1, + "approved_by": [ + { + "user": { + "name": "Administrator", + "username": "root", + "id": 1, + "state": "active", + "avatar_url": "http://www.gravatar.com/avatar/e64c7d89f26bd1972efa854d13d7dd61?s=80\u0026d=identicon", + "web_url": "http://localhost:3000/u/root" + } + } + ], + "approvers": [], + "approver_groups": [] +} +``` + +### Change approval configuration + +>**Note:** This API endpoint is only available on 10.6 Starter and above. + +If you are allowed to, you can change `approvals_required` using the following +endpoint: + +``` +POST /projects/:id/merge_requests/:merge_request_iid/approvals +``` + +**Parameters:** + +| Attribute | Type | Required | Description | +|----------------------|---------|----------|--------------------------------------------| +| `id` | integer | yes | The ID of a project | +| `merge_request_iid` | integer | yes | The IID of MR | +| `approvals_required` | integer | yes | Approvals required before MR can be merged | + + +```json +{ + "id": 5, + "iid": 5, + "project_id": 1, + "title": "Approvals API", + "description": "Test", + "state": "opened", + "created_at": "2016-06-08T00:19:52.638Z", + "updated_at": "2016-06-08T21:20:42.470Z", + "merge_status": "cannot_be_merged", + "approvals_required": 2, + "approvals_left": 2, + "approved_by": [], + "approvers": [], + "approver_groups": [] +} +``` + +### Change allowed approvers for Merge Request + +>**Note:** This API endpoint is only available on 10.6 Starter and above. + +If you are allowed to, you can change approvers and approver groups using +the following endpoint: + +``` +PUT /projects/:id/merge_requests/:merge_request_iid/approvers +``` + +**Important:** Approvers and groups not in the request will be **removed** + +**Parameters:** + +| Attribute | Type | Required | Description | +|----------------------|---------|----------|-----------------------------------------------------------| +| `id` | integer | yes | The ID of a project | +| `merge_request_iid` | integer | yes | The IID of MR | +| `approver_ids` | Array | yes | An array of User IDs that can approve the MR | +| `approver_group_ids` | Array | yes | An array of Group IDs whose members can approve the MR | + +```json +{ + "id": 5, + "iid": 5, + "project_id": 1, + "title": "Approvals API", + "description": "Test", + "state": "opened", + "created_at": "2016-06-08T00:19:52.638Z", + "updated_at": "2016-06-08T21:20:42.470Z", + "merge_status": "cannot_be_merged", + "approvals_required": 2, + "approvals_left": 2, + "approved_by": [], + "approvers": [ + { + "user": { + "name": "Administrator", + "username": "root", + "id": 1, + "state": "active", + "avatar_url": "http://www.gravatar.com/avatar/e64c7d89f26bd1972efa854d13d7dd61?s=80\u0026d=identicon", + "web_url": "http://localhost:3000/u/root" + } + } + ], + "approver_groups": [ + { + "group": { + "id": 5, + "name": "group1", + "path": "group1", + "description": "", + "visibility": "public", + "lfs_enabled": false, + "avatar_url": null, + "web_url": "http://localhost/groups/group1", + "request_access_enabled": false, + "full_name": "group1", + "full_path": "group1", + "parent_id": null, + "ldap_cn": null, + "ldap_access": null + } + } + ] +} +``` + +## Approve Merge Request + +>**Note:** This API endpoint is only available on 8.9 Starter and above. + +If you are allowed to, you can approve a merge request using the following +endpoint: + +``` +POST /projects/:id/merge_requests/:merge_request_iid/approve +``` + +**Parameters:** + +| Attribute | Type | Required | Description | +|---------------------|---------|----------|---------------------| +| `id` | integer | yes | The ID of a project | +| `merge_request_iid` | integer | yes | The IID of MR | +| `sha` | string | no | The HEAD of the MR | + +The `sha` parameter works in the same way as +when [accepting a merge request](merge_requests.md#accept-mr): if it is passed, then it must +match the current HEAD of the merge request for the approval to be added. If it +does not match, the response code will be `409`. + +```json +{ + "id": 5, + "iid": 5, + "project_id": 1, + "title": "Approvals API", + "description": "Test", + "state": "opened", + "created_at": "2016-06-08T00:19:52.638Z", + "updated_at": "2016-06-09T21:32:14.105Z", + "merge_status": "can_be_merged", + "approvals_required": 2, + "approvals_left": 0, + "approved_by": [ + { + "user": { + "name": "Administrator", + "username": "root", + "id": 1, + "state": "active", + "avatar_url": "http://www.gravatar.com/avatar/e64c7d89f26bd1972efa854d13d7dd61?s=80\u0026d=identicon", + "web_url": "http://localhost:3000/u/root" + } + }, + { + "user": { + "name": "Nico Cartwright", + "username": "ryley", + "id": 2, + "state": "active", + "avatar_url": "http://www.gravatar.com/avatar/cf7ad14b34162a76d593e3affca2adca?s=80\u0026d=identicon", + "web_url": "http://localhost:3000/u/ryley" + } + } + ], + "approvers": [], + "approver_groups": [] +} +``` + +## Unapprove Merge Request + +>**Note:** This API endpoint is only available on 9.0 Starter and above. + +If you did approve a merge request, you can unapprove it using the following +endpoint: + +``` +POST /projects/:id/merge_requests/:merge_request_iid/unapprove +``` + +**Parameters:** + +| Attribute | Type | Required | Description | +|---------------------|---------|----------|---------------------| +| `id` | integer | yes | The ID of a project | +| `merge_request_iid` | integer | yes | The IID of MR | diff --git a/doc/api/merge_requests.md b/doc/api/merge_requests.md index ed4b6281acc..023133eaee1 100644 --- a/doc/api/merge_requests.md +++ b/doc/api/merge_requests.md @@ -29,27 +29,28 @@ GET /merge_requests?search=foo&in=title Parameters: -| Attribute | Type | Required | Description | -| ------------------- | -------- | -------- | ---------------------------------------------------------------------------------------------------------------------- | -| `state` | string | no | Return all merge requests or just those that are `opened`, `closed`, `locked`, or `merged` | -| `order_by` | string | no | Return requests ordered by `created_at` or `updated_at` fields. Default is `created_at` | -| `sort` | string | no | Return requests sorted in `asc` or `desc` order. Default is `desc` | -| `milestone` | string | no | Return merge requests for a specific milestone. `None` returns merge requests with no milestone. `Any` returns merge requests that have an assigned milestone. | -| `view` | string | no | If `simple`, returns the `iid`, URL, title, description, and basic state of merge request | -| `labels` | string | no | Return merge requests matching a comma separated list of labels. `None` lists all merge requests with no labels. `Any` lists all merge requests with at least one label. `No+Label` (Deprecated) lists all merge requests with no labels. Predefined names are case-insensitive. | -| `created_after` | datetime | no | Return merge requests created on or after the given time | -| `created_before` | datetime | no | Return merge requests created on or before the given time | -| `updated_after` | datetime | no | Return merge requests updated on or after the given time | -| `updated_before` | datetime | no | Return merge requests updated on or before the given time | -| `scope` | string | no | Return merge requests for the given scope: `created_by_me`, `assigned_to_me` or `all`. Defaults to `created_by_me`<br> For versions before 11.0, use the now deprecated `created-by-me` or `assigned-to-me` scopes instead. | -| `author_id` | integer | no | Returns merge requests created by the given user `id`. Combine with `scope=all` or `scope=assigned_to_me` | -| `assignee_id` | integer | no | Returns merge requests assigned to the given user `id`. `None` returns unassigned merge requests. `Any` returns merge requests with an assignee. | -| `my_reaction_emoji` | string | no | Return merge requests reacted by the authenticated user by the given `emoji`. `None` returns issues not given a reaction. `Any` returns issues given at least one reaction. _([Introduced][ce-14016] in GitLab 10.0)_ | -| `source_branch` | string | no | Return merge requests with the given source branch | -| `target_branch` | string | no | Return merge requests with the given target branch | -| `search` | string | no | Search merge requests against their `title` and `description` | -| `in` | string | no | Modify the scope of the `search` attribute. `title`, `description`, or a string joining them with comma. Default is `title,description` | -| `wip` | string | no | Filter merge requests against their `wip` status. `yes` to return *only* WIP merge requests, `no` to return *non* WIP merge requests | +| Attribute | Type | Required | Description | +| ------------------- | -------- | -------- | ---------------------------------------------------------------------------------------------------------------------- | +| `state` | string | no | Return all merge requests or just those that are `opened`, `closed`, `locked`, or `merged` | +| `order_by` | string | no | Return requests ordered by `created_at` or `updated_at` fields. Default is `created_at` | +| `sort` | string | no | Return requests sorted in `asc` or `desc` order. Default is `desc` | +| `milestone` | string | no | Return merge requests for a specific milestone. `None` returns merge requests with no milestone. `Any` returns merge requests that have an assigned milestone. | +| `view` | string | no | If `simple`, returns the `iid`, URL, title, description, and basic state of merge request | +| `labels` | string | no | Return merge requests matching a comma separated list of labels. `None` lists all merge requests with no labels. `Any` lists all merge requests with at least one label. `No+Label` (Deprecated) lists all merge requests with no labels. Predefined names are case-insensitive. | +| `created_after` | datetime | no | Return merge requests created on or after the given time | +| `created_before` | datetime | no | Return merge requests created on or before the given time | +| `updated_after` | datetime | no | Return merge requests updated on or after the given time | +| `updated_before` | datetime | no | Return merge requests updated on or before the given time | +| `scope` | string | no | Return merge requests for the given scope: `created_by_me`, `assigned_to_me` or `all`. Defaults to `created_by_me`<br> For versions before 11.0, use the now deprecated `created-by-me` or `assigned-to-me` scopes instead. | +| `author_id` | integer | no | Returns merge requests created by the given user `id`. Combine with `scope=all` or `scope=assigned_to_me` | +| `assignee_id` | integer | no | Returns merge requests assigned to the given user `id`. `None` returns unassigned merge requests. `Any` returns merge requests with an assignee. | +| `approver_ids` | Array[integer] | no | Returns merge requests which have specified all the users with the given `id`s as individual approvers. `None` returns merge requests without approvers. `Any` returns merge requests with an approver. | +| `my_reaction_emoji` | string | no | Return merge requests reacted by the authenticated user by the given `emoji`. `None` returns issues not given a reaction. `Any` returns issues given at least one reaction. _([Introduced][ce-14016] in GitLab 10.0)_ | +| `source_branch` | string | no | Return merge requests with the given source branch | +| `target_branch` | string | no | Return merge requests with the given target branch | +| `search` | string | no | Search merge requests against their `title` and `description` | +| `in` | string | no | Modify the scope of the `search` attribute. `title`, `description`, or a string joining them with comma. Default is `title,description` | +| `wip` | string | no | Filter merge requests against their `wip` status. `yes` to return *only* WIP merge requests, `no` to return *non* WIP merge requests | ```json [ @@ -130,7 +131,8 @@ Parameters: "human_time_estimate": null, "human_total_time_spent": null }, - "squash": false + "squash": false, + "approvals_before_merge": null } ] ``` @@ -179,6 +181,7 @@ Parameters: | `scope` | string | no | Return merge requests for the given scope: `created_by_me`, `assigned_to_me` or `all`.<br> For versions before 11.0, use the now deprecated `created-by-me` or `assigned-to-me` scopes instead.<br> _([Introduced][ce-13060] in GitLab 9.5. [Changed to snake_case][ce-18935] in GitLab 11.0)_ | | `author_id` | integer | no | Returns merge requests created by the given user `id` _([Introduced][ce-13060] in GitLab 9.5)_ | | `assignee_id` | integer | no | Returns merge requests assigned to the given user `id`. `None` returns unassigned merge requests. `Any` returns merge requests with an assignee. _([Introduced][ce-13060] in GitLab 9.5)_ | +| `approver_ids` | Array[integer] | no | Returns merge requests which have specified all the users with the given `id`s as individual approvers. `None` returns merge requests without approvers. `Any` returns merge requests with an approver. | | `my_reaction_emoji` | string | no | Return merge requests reacted by the authenticated user by the given `emoji`. `None` returns issues not given a reaction. `Any` returns issues given at least one reaction. _([Introduced][ce-14016] in GitLab 10.0)_ | | `source_branch` | string | no | Return merge requests with the given source branch | | `target_branch` | string | no | Return merge requests with the given target branch | @@ -264,7 +267,8 @@ Parameters: "human_time_estimate": null, "human_total_time_spent": null }, - "squash": false + "squash": false, + "approvals_before_merge": null } ] ``` @@ -304,6 +308,7 @@ Parameters: | `scope` | string | no | Return merge requests for the given scope: `created_by_me`, `assigned_to_me` or `all`.<br> | | `author_id` | integer | no | Returns merge requests created by the given user `id` _([Introduced][ce-13060] in GitLab 9.5)_ | | `assignee_id` | integer | no | Returns merge requests assigned to the given user `id`. `None` returns unassigned merge requests. `Any` returns merge requests with an assignee. _([Introduced][ce-13060] in GitLab 9.5)_ | +| `approver_ids` | Array[integer] | no | Returns merge requests which have specified all the users with the given `id`s as individual approvers. `None` returns merge requests without approvers. `Any` returns merge requests with an approver. | | `my_reaction_emoji` | string | no | Return merge requests reacted by the authenticated user by the given `emoji`. `None` returns issues not given a reaction. `Any` returns issues given at least one reaction. _([Introduced][ce-14016] in GitLab 10.0)_ | | `source_branch` | string | no | Return merge requests with the given source branch | | `target_branch` | string | no | Return merge requests with the given target branch | @@ -386,7 +391,8 @@ Parameters: "human_time_estimate": null, "human_total_time_spent": null }, - "squash": false + "squash": false, + "approvals_before_merge": null } ] ``` @@ -513,7 +519,8 @@ Parameters: "start_sha": "c380d3acebd181f13629a25d2e2acca46ffe1e00" }, "diverged_commits_count": 2, - "rebase_in_progress": false + "rebase_in_progress": false, + "approvals_before_merge": null } ``` @@ -727,6 +734,15 @@ POST /projects/:id/merge_requests | `allow_maintainer_to_push` | boolean | no | Deprecated, see allow_collaboration | | `squash` | boolean | no | Squash commits into a single commit when merging | +If `approvals_before_merge` is not provided, it inherits the value from the +target project. If it is provided, then the following conditions must hold in +order for it to take effect: + +1. The target project's `approvals_before_merge` must be greater than zero. (A + value of zero disables approvals for that project.) +2. The provided value of `approvals_before_merge` must be greater than the + target project's `approvals_before_merge`. + ```json { "id": 1, @@ -824,7 +840,8 @@ POST /projects/:id/merge_requests "head_sha": "2be7ddb704c7b6b83732fdd5b9f09d5a397b5f8f", "start_sha": "c380d3acebd181f13629a25d2e2acca46ffe1e00" }, - "diverged_commits_count": 2 + "diverged_commits_count": 2, + "approvals_before_merge": null } ``` @@ -952,7 +969,8 @@ Must include at least one non-required attribute from above. "head_sha": "2be7ddb704c7b6b83732fdd5b9f09d5a397b5f8f", "start_sha": "c380d3acebd181f13629a25d2e2acca46ffe1e00" }, - "diverged_commits_count": 2 + "diverged_commits_count": 2, + "approvals_before_merge": null } ``` @@ -1097,7 +1115,8 @@ Parameters: "head_sha": "2be7ddb704c7b6b83732fdd5b9f09d5a397b5f8f", "start_sha": "c380d3acebd181f13629a25d2e2acca46ffe1e00" }, - "diverged_commits_count": 2 + "diverged_commits_count": 2, + "approvals_before_merge": null } ``` @@ -1247,7 +1266,8 @@ Parameters: "head_sha": "2be7ddb704c7b6b83732fdd5b9f09d5a397b5f8f", "start_sha": "c380d3acebd181f13629a25d2e2acca46ffe1e00" }, - "diverged_commits_count": 2 + "diverged_commits_count": 2, + "approvals_before_merge": null } ``` @@ -1502,7 +1522,8 @@ Example response: "head_sha": "2be7ddb704c7b6b83732fdd5b9f09d5a397b5f8f", "start_sha": "c380d3acebd181f13629a25d2e2acca46ffe1e00" }, - "diverged_commits_count": 2 + "diverged_commits_count": 2, + "approvals_before_merge": null } ``` @@ -1623,7 +1644,8 @@ Example response: "head_sha": "2be7ddb704c7b6b83732fdd5b9f09d5a397b5f8f", "start_sha": "c380d3acebd181f13629a25d2e2acca46ffe1e00" }, - "diverged_commits_count": 2 + "diverged_commits_count": 2, + "approvals_before_merge": null } ``` @@ -1984,3 +2006,7 @@ Example response: [ce-14016]: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/14016 [ce-15454]: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/15454 [ce-18935]: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/18935 + +## Approvals + +For approvals, please see [Merge Request Approvals](merge_request_approvals.md) diff --git a/doc/api/namespaces.md b/doc/api/namespaces.md index b8bc4c40124..c6f08fdce34 100644 --- a/doc/api/namespaces.md +++ b/doc/api/namespaces.md @@ -37,10 +37,11 @@ Example response: "id": 2, "name": "group1", "path": "group1", - "kind": "group", + "kind": "group" "full_path": "group1", "parent_id": null, - "members_count_with_descendants": 2 + "members_count_with_descendants": 2, + "plan": "bronze" }, { "id": 3, @@ -54,7 +55,7 @@ Example response: ] ``` -**Note**: `members_count_with_descendants` are presented only for group maintainers/owners. +**Note**: `members_count_with_descendants` and `plan` are presented only for group maintainers/owners. ## Search for namespace diff --git a/doc/api/notes.md b/doc/api/notes.md index dd4e18b14e6..b01823b0f0f 100644 --- a/doc/api/notes.md +++ b/doc/api/notes.md @@ -1,6 +1,6 @@ # Notes API -Notes are comments on snippets, issues or merge requests. +Notes are comments on snippets, issues, merge requests or epics. ## Issues @@ -381,3 +381,126 @@ Parameters: ```bash curl --request DELETE --header "PRIVATE-TOKEN: <your_access_token>" https://gitlab.example.com/api/v4/projects/5/merge_requests/7/notes/1602 ``` + +## Epics **[ULTIMATE]** + +### List all epic notes + +Gets a list of all notes for a single epic. Epic notes are comments users can post to an epic. + +``` +GET /groups/:id/epics/:epic_id/notes +GET /groups/:id/epics/:epic_id/notes?sort=asc&order_by=updated_at +``` + +| Attribute | Type | Required | Description | +| ------------------- | ---------------- | ---------- | ----------- | +| `id` | integer/string | yes | The ID or [URL-encoded path of the group](README.md#namespaced-path-encoding) | +| `epic_id` | integer | yes | The ID of a group epic | +| `sort` | string | no | Return epic notes sorted in `asc` or `desc` order. Default is `desc` | +| `order_by` | string | no | Return epic notes ordered by `created_at` or `updated_at` fields. Default is `created_at` | + +```bash +curl --request GET --header "PRIVATE-TOKEN: <your_access_token>" https://gitlab.example.com/api/v4/groups/5/epics/11/notes +``` + +### Get single epic note + +Returns a single note for a given epic. + +``` +GET /groups/:id/epics/:epic_id/notes/:note_id +``` + +Parameters: + +| Attribute | Type | Required | Description | +| --------- | -------------- | -------- | ----------- | +| `id` | integer/string | yes | The ID or [URL-encoded path of the group](README.md#namespaced-path-encoding) | +| `epic_id` | integer | yes | The ID of an epic | +| `note_id` | integer | yes | The ID of a note | + +```json +{ + "id": 52, + "title": "Epic", + "file_name": "epic.rb", + "author": { + "id": 1, + "username": "pipin", + "email": "admin@example.com", + "name": "Pip", + "state": "active", + "created_at": "2013-09-30T13:46:01Z" + }, + "expires_at": null, + "updated_at": "2013-10-02T07:34:20Z", + "created_at": "2013-10-02T07:34:20Z" +} +``` + +```bash +curl --request GET --header "PRIVATE-TOKEN: <your_access_token>" https://gitlab.example.com/api/v4/groups/5/epics/11/notes/1 +``` + +### Create new epic note + +Creates a new note for a single epic. Epic notes are comments users can post to an epic. +If you create a note where the body only contains an Award Emoji, you'll receive this object back. + +``` +POST /groups/:id/epics/:epic_id/notes +``` + +Parameters: + +| Attribute | Type | Required | Description | +| --------- | -------------- | -------- | ----------- | +| `id` | integer/string | yes | The ID or [URL-encoded path of the group](README.md#namespaced-path-encoding) | +| `epic_id` | integer | yes | The ID of an epic | +| `body` | string | yes | The content of a note | + +```bash +curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" https://gitlab.example.com/api/v4/projects/5/snippet/11/notes?body=note +``` + +### Modify existing epic note + +Modify existing note of an epic. + +``` +PUT /groups/:id/epics/:epic_id/notes/:note_id +``` + +Parameters: + +| Attribute | Type | Required | Description | +| --------- | -------------- | -------- | ----------- | +| `id` | integer/string | yes | The ID or [URL-encoded path of the group](README.md#namespaced-path-encoding) | +| `epic_id` | integer | yes | The ID of an epic | +| `note_id` | integer | yes | The ID of a note | +| `body` | string | yes | The content of a note | + +```bash +curl --request PUT --header "PRIVATE-TOKEN: <your_access_token>" https://gitlab.example.com/api/v4/projects/5/snippet/11/notes?body=note +``` + +### Delete an epic note + +Deletes an existing note of an epic. + +``` +DELETE /groups/:id/epics/:epic_id/notes/:note_id +``` + +Parameters: + +| Attribute | Type | Required | Description | +| --------- | -------------- | -------- | ----------- | +| `id` | integer/string | yes | The ID or [URL-encoded path of the group](README.md#namespaced-path-encoding) | +| `epic_id` | integer | yes | The ID of an epic | +| `note_id` | integer | yes | The ID of a note | + +```bash +curl --request DELETE --header "PRIVATE-TOKEN: <your_access_token>" https://gitlab.example.com/api/v4/groups/5/epics/52/notes/1659 +``` diff --git a/doc/api/notification_settings.md b/doc/api/notification_settings.md index e21e73c7dac..80b110e0552 100644 --- a/doc/api/notification_settings.md +++ b/doc/api/notification_settings.md @@ -32,6 +32,7 @@ reassign_merge_request merge_merge_request failed_pipeline success_pipeline +new_epic ``` ## Global notification settings @@ -85,6 +86,7 @@ curl --request PUT --header "PRIVATE-TOKEN: <your_access_token>" https://gitlab. | `merge_merge_request` | boolean | no | Enable/disable this notification | | `failed_pipeline` | boolean | no | Enable/disable this notification | | `success_pipeline` | boolean | no | Enable/disable this notification | +| `new_epic` | boolean | no | Enable/disable this notification ([Introduced][ee-6626] in 11.3) | Example response: @@ -153,6 +155,7 @@ curl --request PUT --header "PRIVATE-TOKEN: <your_access_token>" https://gitlab. | `merge_merge_request` | boolean | no | Enable/disable this notification | | `failed_pipeline` | boolean | no | Enable/disable this notification | | `success_pipeline` | boolean | no | Enable/disable this notification | +| `new_epic` | boolean | no | Enable/disable this notification ([Introduced][ee-6626] in 11.3) | Example responses: @@ -177,9 +180,11 @@ Example responses: "reassign_merge_request": false, "merge_merge_request": false, "failed_pipeline": false, - "success_pipeline": false + "success_pipeline": false, + "new_epic": false } } ``` [ce-5632]: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/5632 +[ee-6626]: https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/6626 diff --git a/doc/api/packages.md b/doc/api/packages.md new file mode 100644 index 00000000000..618e5c3056a --- /dev/null +++ b/doc/api/packages.md @@ -0,0 +1,152 @@ +# Packages API **[PREMIUM]** + +This is the API docs of [GitLab Packages](../administration/packages.md). + +## List project packages + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/9259) in GitLab 11.8. + +Get a list of project packages. Both Maven and NPM packages are included in results. +When accessed without authentication, only packages of public projects are returned. + +``` +GET /projects/:id/packages +``` + +| Attribute | Type | Required | Description | +| --------- | ---- | -------- | ----------- | +| `id` | integer/string | yes | ID or [URL-encoded path of the project](README.md#namespaced-path-encoding) | + +```bash +curl --header "PRIVATE-TOKEN: <your_access_token>" https://gitlab.example.com/api/v4/projects/:id/packages +``` + +Example response: + +```json +[ + { + "id": 1, + "name": "com/mycompany/my-app", + "version": "1.0-SNAPSHOT", + "package_type": "maven" + }, + { + "id": 2, + "name": "@foo/bar", + "version": "1.0.3", + "package_type": "npm" + } +] +``` + +By default, the `GET` request will return 20 results, since the API is [paginated](README.md#pagination). + +## Get a project package + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/9667) in GitLab 11.9. + +Get a single project package. + +``` +GET /projects/:id/packages/:package_id +``` + +| Attribute | Type | Required | Description | +| --------- | ---- | -------- | ----------- | +| `id` | integer/string | yes | ID or [URL-encoded path of the project](README.md#namespaced-path-encoding). | +| `package_id` | integer | yes | ID of a package. | + +```bash +curl --header "PRIVATE-TOKEN: <your_access_token>" https://gitlab.example.com/api/v4/projects/:id/packages/:package_id +``` + +Example response: + +```json +{ + "id": 1, + "name": "com/mycompany/my-app", + "version": "1.0-SNAPSHOT", + "package_type": "maven" +} +``` + +## List package files + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/9305) in GitLab 11.8. + +Get a list of package files of a single package. + +``` +GET /projects/:id/packages/:package_id/package_files +``` + +| Attribute | Type | Required | Description | +| --------- | ---- | -------- | ----------- | +| `id` | integer/string | yes | ID or [URL-encoded path of the project](README.md#namespaced-path-encoding) | +| `package_id` | integer | yes | ID of a package. | + +```bash +curl --header "PRIVATE-TOKEN: <your_access_token>" https://gitlab.example.com/api/v4/projects/1/packages/4/package_files +``` + +Example response: + +```json +[ + { + "id": 25, + "package_id": 4, + "created_at": "2018-11-07T15:25:52.199Z", + "file_name": "my-app-1.5-20181107.152550-1.jar", + "size": 2421, + "file_md5": "58e6a45a629910c6ff99145a688971ac", + "file_sha1": "ebd193463d3915d7e22219f52740056dfd26cbfe" + }, + { + "id": 26, + "package_id": 4, + "created_at": "2018-11-07T15:25:56.776Z", + "file_name": "my-app-1.5-20181107.152550-1.pom", + "size": 1122, + "file_md5": "d90f11d851e17c5513586b4a7e98f1b2", + "file_sha1": "9608d068fe88aff85781811a42f32d97feb440b5" + }, + { + "id": 27, + "package_id": 4, + "created_at": "2018-11-07T15:26:00.556Z", + "file_name": "maven-metadata.xml", + "size": 767, + "file_md5": "6dfd0cce1203145a927fef5e3a1c650c", + "file_sha1": "d25932de56052d320a8ac156f745ece73f6a8cd2" + } +] +``` + +By default, the `GET` request will return 20 results, since the API is [paginated](README.md#pagination). + +## Delete a project package + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/9623) in GitLab 11.9. + +Deletes a project package. + +``` +DELETE /projects/:id/packages/:package_id +``` + +| Attribute | Type | Required | Description | +| --------- | ---- | -------- | ----------- | +| `id` | integer/string | yes | ID or [URL-encoded path of the project](README.md#namespaced-path-encoding) | +| `package_id` | integer | yes | ID of a package. | + +```bash +curl --request DELETE --header "PRIVATE-TOKEN: <your_access_token>" https://gitlab.example.com/api/v4/projects/:id/packages/:package_id +``` + +Can return the following status codes: + +- `204 No Content`, if the package was deleted successfully. +- `404 Not Found`, if the package was not found. diff --git a/doc/api/project_clusters.md b/doc/api/project_clusters.md index 8efb98fe1fc..3dbfa0a995c 100644 --- a/doc/api/project_clusters.md +++ b/doc/api/project_clusters.md @@ -163,6 +163,7 @@ Parameters: | `platform_kubernetes_attributes[ca_cert]` | String | no | TLS certificate (needed if API is using a self-signed TLS certificate | | `platform_kubernetes_attributes[namespace]` | String | no | The unique namespace related to the project | | `platform_kubernetes_attributes[authorization_type]` | String | no | The cluster authorization type: `rbac`, `abac` or `unknown_authorization`. Defaults to `rbac`. | +| `environment_scope` | String | no | The associated environment to the cluster. Defaults to `*` | Example request: @@ -251,6 +252,7 @@ Parameters: | `platform_kubernetes_attributes[token]` | String | no | The token to authenticate against Kubernetes | | `platform_kubernetes_attributes[ca_cert]` | String | no | TLS certificate (needed if API is using a self-signed TLS certificate | | `platform_kubernetes_attributes[namespace]` | String | no | The unique namespace related to the project | +| `environment_scope` | String | no | The associated environment to the cluster | NOTE: **Note:** `name`, `api_url`, `ca_cert` and `token` can only be updated if the cluster was added diff --git a/doc/api/project_level_variables.md b/doc/api/project_level_variables.md index 438bebe62f5..6ac5448ae15 100644 --- a/doc/api/project_level_variables.md +++ b/doc/api/project_level_variables.md @@ -67,6 +67,7 @@ POST /projects/:id/variables | `key` | string | yes | The `key` of a variable; must have no more than 255 characters; only `A-Z`, `a-z`, `0-9`, and `_` are allowed | | `value` | string | yes | The `value` of a variable | | `protected` | boolean | no | Whether the variable is protected | +| `environment_scope` | string | no | The `environment_scope` of the variable | ``` curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/1/variables" --form "key=NEW_VARIABLE" --form "value=new value" @@ -76,7 +77,8 @@ curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" "https://gitla { "key": "NEW_VARIABLE", "value": "new value", - "protected": false + "protected": false, + "environment_scope": "*" } ``` @@ -94,6 +96,7 @@ PUT /projects/:id/variables/:key | `key` | string | yes | The `key` of a variable | | `value` | string | yes | The `value` of a variable | | `protected` | boolean | no | Whether the variable is protected | +| `environment_scope` | string | no | The `environment_scope` of the variable | ``` curl --request PUT --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/1/variables/NEW_VARIABLE" --form "value=updated value" @@ -103,7 +106,8 @@ curl --request PUT --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab { "key": "NEW_VARIABLE", "value": "updated value", - "protected": true + "protected": true, + "environment_scope": "*" } ``` diff --git a/doc/api/project_templates.md b/doc/api/project_templates.md index 3b5b12c8da3..0a94a8d47ae 100644 --- a/doc/api/project_templates.md +++ b/doc/api/project_templates.md @@ -15,7 +15,7 @@ templates are also available from this API endpoint. Support will be added for [Issue and Merge Request templates](../user/project/description_templates.md) in a future release. -Support for [Group-level file templates](../user/group/index.md#group-level-file-templates-premium) +Support for [Group-level file templates](../user/group/index.md#group-file-templates-premium) **[PREMIUM]** was [added](https://gitlab.com/gitlab-org/gitlab-ee/issues/5987) in GitLab 11.5 diff --git a/doc/api/projects.md b/doc/api/projects.md index 0a950352ecf..d9ad84c397f 100644 --- a/doc/api/projects.md +++ b/doc/api/projects.md @@ -14,7 +14,7 @@ Values for the project visibility level are: The project can be cloned by any logged in user. - `public`: - The project can be cloned without any authentication. + The project can be accessed without any authentication. ## Project merge method @@ -149,6 +149,7 @@ When the user is authenticated and `simple` is not set this returns something li "only_allow_merge_if_all_discussions_are_resolved": false, "request_access_enabled": false, "merge_method": "merge", + "approvals_before_merge": 0, "statistics": { "commit_count": 37, "storage_size": 1038090, @@ -230,6 +231,7 @@ When the user is authenticated and `simple` is not set this returns something li "only_allow_merge_if_all_discussions_are_resolved": false, "request_access_enabled": false, "merge_method": "merge", + "approvals_before_merge": 0, "statistics": { "commit_count": 12, "storage_size": 2066080, @@ -537,11 +539,13 @@ GET /projects/:id "group_access_level": 10 } ], + "repository_storage": "default", "only_allow_merge_if_pipeline_succeeds": false, "only_allow_merge_if_all_discussions_are_resolved": false, "printing_merge_requests_link_enabled": true, "request_access_enabled": false, "merge_method": "merge", + "approvals_before_merge": 0, "statistics": { "commit_count": 37, "storage_size": 1038090, @@ -660,6 +664,7 @@ POST /projects | `name` | string | yes if path is not provided | The name of the new project. Equals path if not provided. | | `path` | string | yes if name is not provided | Repository name for new project. Generated based on name if not provided (generated lowercased with dashes). | | `namespace_id` | integer | no | Namespace for the new project (defaults to the current user's namespace) | +| `default_branch` | string | no | `master` by default | | `description` | string | no | Short project description | | `issues_enabled` | boolean | no | Enable issues for this project | | `merge_requests_enabled` | boolean | no | Enable merge requests for this project | @@ -681,8 +686,16 @@ POST /projects | `avatar` | mixed | no | Image file for avatar of the project | | `printing_merge_request_link_enabled` | boolean | no | Show link to create/view merge request when pushing from the command line | | `ci_config_path` | string | no | The path to CI config file | +| `repository_storage` | string | no | Which storage shard the repository is on. Available only to admins | +| `approvals_before_merge` | integer | no | How many approvers should approve merge request by default | +| `mirror` | boolean | no | Enables pull mirroring in a project | +| `mirror_trigger_builds` | boolean | no | Pull mirroring triggers builds | | `initialize_with_readme` | boolean | no | `false` by default | +>**Note**: If your HTTP repository is not publicly accessible, +add authentication information to the URL: `https://username:password@gitlab.company.com/group/project.git` +where `password` is a public access key with the `api` scope enabled. + ## Create project for user Creates a new project owned by the specified user. Available only for admins. @@ -696,7 +709,6 @@ POST /projects/user/:user_id | `user_id` | integer | yes | The user ID of the project owner | | `name` | string | yes | The name of the new project | | `path` | string | no | Custom repository name for new project. By default generated based on name | -| `default_branch` | string | no | `master` by default | | `namespace_id` | integer | no | Namespace for the new project (defaults to the current user's namespace) | | `description` | string | no | Short project description | | `issues_enabled` | boolean | no | Enable issues for this project | @@ -719,6 +731,15 @@ POST /projects/user/:user_id | `avatar` | mixed | no | Image file for avatar of the project | | `printing_merge_request_link_enabled` | boolean | no | Show link to create/view merge request when pushing from the command line | | `ci_config_path` | string | no | The path to CI config file | +| `repository_storage` | string | no | Which storage shard the repository is on. Available only to admins | +| `approvals_before_merge` | integer | no | How many approvers should approve merge request by default | +| `external_authorization_classification_label` | string | no | The classification label for the project | +| `mirror` | boolean | no | Enables pull mirroring in a project | +| `mirror_trigger_builds` | boolean | no | Pull mirroring triggers builds | + +>**Note**: If your HTTP repository is not publicly accessible, +add authentication information to the URL: `https://username:password@gitlab.company.com/group/project.git` +where `password` is a public access key with the `api` scope enabled. ## Edit project @@ -754,6 +775,19 @@ PUT /projects/:id | `tag_list` | array | no | The list of tags for a project; put array of tags, that should be finally assigned to a project | | `avatar` | mixed | no | Image file for avatar of the project | | `ci_config_path` | string | no | The path to CI config file | +| `repository_storage` | string | no | Which storage shard the repository is on. Available only to admins | +| `approvals_before_merge` | integer | no | How many approvers should approve merge request by default | +| `external_authorization_classification_label` | string | no | The classification label for the project | +| `mirror` | boolean | no | Enables pull mirroring in a project | +| `mirror_user_id` | integer | no | User responsible for all the activity surrounding a pull mirror event | +| `mirror_trigger_builds` | boolean | no | Pull mirroring triggers builds | +| `only_mirror_protected_branches` | boolean | no | Only mirror protected branches | +| `mirror_overwrites_diverged_branches` | boolean | no | Pull mirror overwrites diverged branches | +| `packages_enabled` | boolean | no | Enable or disable packages repository feature | + +>**Note**: If your HTTP repository is not publicly accessible, +add authentication information to the URL: `https://username:password@gitlab.company.com/group/project.git` +where `password` is a public access key with the `api` scope enabled. ## Fork project @@ -1531,9 +1565,96 @@ POST /projects/:id/housekeeping | Attribute | Type | Required | Description | | --------- | ---- | -------- | ----------- | +| `id` | integer/string | yes | The ID of the project or NAMESPACE/PROJECT_NAME | + +## Push Rules **[STARTER]** + +### Get project push rules + +Get the push rules of a project. + +``` +GET /projects/:id/push_rule +``` + +| Attribute | Type | Required | Description | +| --------- | ---- | -------- | ----------- | +| `id` | integer/string | yes | The ID of the project or NAMESPACE/PROJECT_NAME | + +```json +{ + "id": 1, + "project_id": 3, + "commit_message_regex": "Fixes \d +\", + "branch_name_regex": "", + "deny_delete_tag": false, + "created_at": "2012-10-12T17:04:47Z", + "member_check": false, + "prevent_secrets": false, + "author_email_regex": "", + "file_name_regex": "", + "max_file_size": 5 +} +``` + +### Add project push rule + +Adds a push rule to a specified project. + +``` +POST /projects/:id/push_rule +``` + +| Attribute | Type | Required | Description | +| --------- | ---- | -------- | ----------- | +| `id` | integer/string | yes | The ID of the project or NAMESPACE/PROJECT_NAME | +| `deny_delete_tag` | boolean | no | Deny deleting a tag | +| `member_check` | boolean | no | Restrict commits by author (email) to existing GitLab users | +| `prevent_secrets` | boolean | no | GitLab will reject any files that are likely to contain secrets | +| `commit_message_regex` | string | no | All commit messages must match this, e.g. `Fixed \d+\..*` | +| `branch_name_regex` | string | no | All branch names must match this, e.g. `(feature|hotfix)\/*` | +| `author_email_regex` | string | no | All commit author emails must match this, e.g. `@my-company.com$` | +| `file_name_regex` | string | no | All commited filenames must **not** match this, e.g. `(jar|exe)$` | +| `max_file_size` | integer | no | Maximum file size (MB) | + +### Edit project push rule + +Edits a push rule for a specified project. + +``` +PUT /projects/:id/push_rule +``` + +| Attribute | Type | Required | Description | +| --------- | ---- | -------- | ----------- | +| `id` | integer/string | yes | The ID of the project or NAMESPACE/PROJECT_NAME | +| `deny_delete_tag` | boolean | no | Deny deleting a tag | +| `member_check` | boolean | no | Restrict commits by author (email) to existing GitLab users | +| `prevent_secrets` | boolean | no | GitLab will reject any files that are likely to contain secrets | +| `commit_message_regex` | string | no | All commit messages must match this, e.g. `Fixed \d+\..*` | +| `branch_name_regex` | string | no | All branch names must match this, e.g. `(feature|hotfix)\/*` | +| `author_email_regex` | string | no | All commit author emails must match this, e.g. `@my-company.com$` | +| `file_name_regex` | string | no | All commited filenames must **not** match this, e.g. `(jar|exe)$` | +| `max_file_size` | integer | no | Maximum file size (MB) | + +### Delete project push rule + +> Introduced in GitLab 9.0. + +Removes a push rule from a project. This is an idempotent method and can be called multiple times. +Either the push rule is available or not. + +``` +DELETE /projects/:id/push_rule +``` + +| Attribute | Type | Required | Description | +| --------- | ---- | -------- | ----------- | | `id` | integer/string | yes | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding) | -### Transfer a project to a new namespace +## Transfer a project to a new namespace + +> Introduced in GitLab 11.1. ``` PUT /projects/:id/transfer @@ -1555,6 +1676,22 @@ Read more in the [Project import/export](project_import_export.md) documentation Read more in the [Project members](members.md) documentation. +## Start the pull mirroring process for a Project **[STARTER]** + +> Introduced in [GitLab Starter](https://about.gitlab.com/pricing) 10.3. + +``` +POST /projects/:id/mirror/pull +``` + +| Attribute | Type | Required | Description | +| --------- | ---- | -------- | ----------- | +| `id` | integer/string | yes | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding) | + +```bash +curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" https://gitlab.example.com/api/v4/projects/:id/mirror/pull +``` + ## Project badges Read more in the [Project Badges](project_badges.md) documentation. diff --git a/doc/api/protected_branches.md b/doc/api/protected_branches.md index a261bb75be5..44980c7c649 100644 --- a/doc/api/protected_branches.md +++ b/doc/api/protected_branches.md @@ -10,6 +10,7 @@ The access levels are defined in the `ProtectedRefAccess.allowed_access_levels` 0 => No access 30 => Developer access 40 => Maintainer access +60 => Admin access ``` ## List protected branches @@ -37,13 +38,17 @@ Example response: "push_access_levels": [ { "access_level": 40, + "user_id": null, + "group_id": null, "access_level_description": "Maintainers" } ], "merge_access_levels": [ { - "access_level": 40, - "access_level_description": "Maintainers" + "access_level": null, + "user_id": null, + "group_id": 1234, + "access_level_description": "Example Merge Group" } ] }, @@ -76,13 +81,17 @@ Example response: "push_access_levels": [ { "access_level": 40, + "user_id": null, + "group_id": null, "access_level_description": "Maintainers" } ], "merge_access_levels": [ { - "access_level": 40, - "access_level_description": "Maintainers" + "access_level": null, + "user_id": null, + "group_id": 1234, + "access_level_description": "Example Merge Group" } ] } @@ -98,7 +107,7 @@ POST /projects/:id/protected_branches ``` ```bash -curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" 'https://gitlab.example.com/api/v4/projects/5/protected_branches?name=*-stable&push_access_level=30&merge_access_level=30' +curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" 'https://gitlab.example.com/api/v4/projects/5/protected_branches?name=*-stable&push_access_level=30&merge_access_level=30&unprotect_access_level=40' ``` | Attribute | Type | Required | Description | @@ -107,6 +116,11 @@ curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" 'https://gitla | `name` | string | yes | The name of the branch or wildcard | | `push_access_level` | string | no | Access levels allowed to push (defaults: `40`, maintainer access level) | | `merge_access_level` | string | no | Access levels allowed to merge (defaults: `40`, maintainer access level) | +| `unprotect_access_level` | string | no | Access levels allowed to unprotect (defaults: `40`, maintainer access level) | +| `allowed_to_push` | array | no | Array of access levels allowed to push, with each described by a hash | +| `allowed_to_merge` | array | no | Array of access levels allowed to merge, with each described by a hash | +| `allowed_to_unprotect` | array | no | Array of access levels allowed to unprotect, with each described by a hash | + Example response: @@ -116,13 +130,64 @@ Example response: "push_access_levels": [ { "access_level": 30, + "user_id": null, + "group_id": null, "access_level_description": "Developers + Maintainers" } ], "merge_access_levels": [ { "access_level": 30, + "user_id": null, + "group_id": null, "access_level_description": "Developers + Maintainers" + ], + "unprotect_access_levels": [ + { + "access_level": 40, + "user_id": null, + "group_id": null, + "access_level_description": "Maintainers" + } + ] +} +``` + +### Example with user / group level access + +Elements in the `allowed_to_push` / `allowed_to_merge` / `allowed_to_unprotect` array should take the +form `{user_id: integer}`, `{group_id: integer}` or `{access_level: integer}`. Each user must have access to the project and each group must [have this project shared](../user/project/members/share_project_with_groups.md). These access levels allow [more granular control over protected branch access](../user/project/protected_branches.md#restricting-push-and-merge-access-to-certain-users-starter) and were [added to the API in ][ee-3516] in GitLab 10.3 EE. + +```bash +curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" 'https://gitlab.example.com/api/v4/projects/5/protected_branches?name=*-stable&allowed_to_push%5B%5D%5Buser_id%5D=1' +``` + +Example response: + +```json +{ + "name":"*-stable", + "push_access_levels": [ + { + "access_level":null, + "user_id":1, + "group_id":null, + "access_level_description":"Administrator" + } + ], + "merge_access_levels": [ + { + "access_level":40, + "user_id":null, + "group_id":null, + "access_level_description":"Maintainers" + ], + "unprotect_access_levels": [ + { + "access_level":40, + "user_id":null, + "group_id":null, + "access_level_description":"Maintainers" } ] } @@ -144,3 +209,5 @@ curl --request DELETE --header "PRIVATE-TOKEN: <your_access_token>" 'https://git | --------- | ---- | -------- | ----------- | | `id` | integer/string | yes | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding) owned by the authenticated user | | `name` | string | yes | The name of the branch | + +[ee-3516]: https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/3516 "ProtectedBranches API handles per user/group granularity" diff --git a/doc/api/resource_label_events.md b/doc/api/resource_label_events.md index e1f9ffa9472..f0a7ac4e41d 100644 --- a/doc/api/resource_label_events.md +++ b/doc/api/resource_label_events.md @@ -88,6 +88,92 @@ Parameters: curl --request GET --header "PRIVATE-TOKEN: <your_access_token>" https://gitlab.example.com/api/v4/projects/5/issues/11/resource_label_events/1 ``` +## Epics **[ULTIMATE]** + +### List group epic label events + +Gets a list of all label events for a single epic. + +``` +GET /groups/:id/epics/:epic_id/resource_label_events +``` + +| Attribute | Type | Required | Description | +| ------------------- | ---------------- | ---------- | ------------ | +| `id` | integer/string | yes | The ID or [URL-encoded path of the group](README.md#namespaced-path-encoding) | +| `epic_id` | integer | yes | The ID of an epic | + +```json +[ + { + "id": 106, + "user": { + "id": 1, + "name": "Administrator", + "username": "root", + "state": "active", + "avatar_url": "https://www.gravatar.com/avatar/e64c7d89f26bd1972efa854d13d7dd61?s=80&d=identicon", + "web_url": "http://gitlab.example.com/root" + }, + "created_at": "2018-08-19T11:43:01.746Z", + "resource_type": "Epic", + "resource_id": 33, + "label": { + "id": 73, + "name": "a1", + "color": "#34495E", + "description": "" + }, + "action": "add" + }, + { + "id": 107, + "user": { + "id": 1, + "name": "Administrator", + "username": "root", + "state": "active", + "avatar_url": "https://www.gravatar.com/avatar/e64c7d89f26bd1972efa854d13d7dd61?s=80&d=identicon", + "web_url": "http://gitlab.example.com/root" + }, + "created_at": "2018-08-19T11:43:01.746Z", + "resource_type": "Epic", + "resource_id": 33, + "label": { + "id": 37, + "name": "glabel2", + "color": "#A8D695", + "description": "" + }, + "action": "add" + } +] +``` + +```bash +curl --request GET --header "PRIVATE-TOKEN: <your_access_token>" https://gitlab.example.com/api/v4/groups/1/epics/11/resource_label_events +``` + +### Get single epic label event + +Returns a single label event for a specific group epic + +``` +GET /groups/:id/epics/:epic_id/resource_label_events/:resource_label_event_id +``` + +Parameters: + +| Attribute | Type | Required | Description | +| --------------- | -------------- | -------- | ----------- | +| `id` | integer/string | yes | The ID or [URL-encoded path of the group](README.md#namespaced-path-encoding) | +| `epic_id` | integer | yes | The ID of an epic | +| `resource_label_event_id` | integer | yes | The ID of a label event | + +```bash +curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" https://gitlab.example.com/api/v4/groups/1/epics/11/resource_label_events/107 +``` + ## Merge requests ### List project merge request label events diff --git a/doc/api/search.md b/doc/api/search.md index aa601648b2c..f9c0f0b70bf 100644 --- a/doc/api/search.md +++ b/doc/api/search.md @@ -19,6 +19,8 @@ GET /search Search the expression within the specified scope. Currently these scopes are supported: projects, issues, merge_requests, milestones, snippet_titles, snippet_blobs. +If Elasticsearch is enabled additional scopes available are blobs, wiki_blobs and commits. Find more about [the feature](../integration/elasticsearch.md). + The response depends on the requested scope. ### Scope: projects @@ -281,6 +283,98 @@ Example response: ] ``` +### Scope: wiki_blobs + +This scope is available only if [Elasticsearch](../integration/elasticsearch.md) is enabled. + +```bash +curl --request GET --header "PRIVATE-TOKEN: <your_access_token>" https://gitlab.example.com/api/v4/search?scope=wiki_blobs&search=bye +``` + +Example response: + +```json + +[ + { + "basename": "home", + "data": "hello\n\nand bye\n\nend", + "filename": "home.md", + "id": null, + "ref": "master", + "startline": 5, + "project_id": 6 + } +] +``` + +### Scope: commits + +This scope is available only if [Elasticsearch](../integration/elasticsearch.md) is enabled. + +```bash +curl --request GET --header "PRIVATE-TOKEN: <your_access_token>" https://gitlab.example.com/api/v4/search?scope=commits&search=bye +``` + +Example response: + +```json + +[ + { + "id": "4109c2d872d5fdb1ed057400d103766aaea97f98", + "short_id": "4109c2d8", + "title": "goodbye $.browser", + "created_at": "2013-02-18T22:02:54.000Z", + "parent_ids": [ + "59d05353ab575bcc2aa958fe1782e93297de64c9" + ], + "message": "goodbye $.browser\n", + "author_name": "angus croll", + "author_email": "anguscroll@gmail.com", + "authored_date": "2013-02-18T22:02:54.000Z", + "committer_name": "angus croll", + "committer_email": "anguscroll@gmail.com", + "committed_date": "2013-02-18T22:02:54.000Z", + "project_id": 6 + } +] +``` + +### Scope: blobs + +This scope is available only if [Elasticsearch](../integration/elasticsearch.md) is enabled. + +Filters are available for this scope: +- filename +- path +- extension + +to use a filter simply include it in your query like so: `a query filename:some_name*`. + +You may use wildcards (`*`) to use glob matching. + +```bash +curl --request GET --header "PRIVATE-TOKEN: <your_access_token>" https://gitlab.example.com/api/v4/search?scope=blobs&search=installation +``` + +Example response: + +```json + +[ + { + "basename": "README", + "data": "```\n\n## Installation\n\nQuick start using the [pre-built", + "filename": "README.md", + "id": null, + "ref": "master", + "startline": 46, + "project_id": 6 + } +] +``` + ## Group Search API Search within the specified group. @@ -299,6 +393,8 @@ GET /groups/:id/search Search the expression within the specified scope. Currently these scopes are supported: projects, issues, merge_requests, milestones. +If Elasticsearch is enabled additional scopes available are blobs, wiki_blobs and commits. Find more about [the feature](../integration/elasticsearch.md). + The response depends on the requested scope. ### Scope: projects @@ -499,6 +595,98 @@ Example response: ] ``` +### Scope: wiki_blobs + +This scope is available only if [Elasticsearch](../integration/elasticsearch.md) is enabled. + +```bash +curl --request GET --header "PRIVATE-TOKEN: <your_access_token>" https://gitlab.example.com/api/v4/groups/6/search?scope=wiki_blobs&search=bye +``` + +Example response: + +```json + +[ + { + "basename": "home", + "data": "hello\n\nand bye\n\nend", + "filename": "home.md", + "id": null, + "ref": "master", + "startline": 5, + "project_id": 6 + } +] +``` + +### Scope: commits + +This scope is available only if [Elasticsearch](../integration/elasticsearch.md) is enabled. + +```bash +curl --request GET --header "PRIVATE-TOKEN: <your_access_token>" https://gitlab.example.com/api/v4/groups/6/search?scope=commits&search=bye +``` + +Example response: + +```json + +[ + { + "id": "4109c2d872d5fdb1ed057400d103766aaea97f98", + "short_id": "4109c2d8", + "title": "goodbye $.browser", + "created_at": "2013-02-18T22:02:54.000Z", + "parent_ids": [ + "59d05353ab575bcc2aa958fe1782e93297de64c9" + ], + "message": "goodbye $.browser\n", + "author_name": "angus croll", + "author_email": "anguscroll@gmail.com", + "authored_date": "2013-02-18T22:02:54.000Z", + "committer_name": "angus croll", + "committer_email": "anguscroll@gmail.com", + "committed_date": "2013-02-18T22:02:54.000Z", + "project_id": 6 + } +] +``` + +### Scope: blobs + +This scope is available only if [Elasticsearch](../integration/elasticsearch.md) is enabled. + +Filters are available for this scope: +- filename +- path +- extension + +to use a filter simply include it in your query like so: `a query filename:some_name*`. + +You may use wildcards (`*`) to use glob matching. + +```bash +curl --request GET --header "PRIVATE-TOKEN: <your_access_token>" https://gitlab.example.com/api/v4/groups/6/search?scope=blobs&search=installation +``` + +Example response: + +```json + +[ + { + "basename": "README", + "data": "```\n\n## Installation\n\nQuick start using the [pre-built", + "filename": "README.md", + "id": null, + "ref": "master", + "startline": 46, + "project_id": 6 + } +] +``` + ## Project Search API Search within the specified project. diff --git a/doc/api/services.md b/doc/api/services.md index c44f5cc5781..a8117dfb51f 100644 --- a/doc/api/services.md +++ b/doc/api/services.md @@ -1062,6 +1062,77 @@ Get JetBrains TeamCity CI service settings for a project. GET /projects/:id/services/teamcity ``` +## Jenkins CI + +A continuous integration and build server + +### Create/Edit Jenkins CI service + +Set Jenkins CI service for a project. + + +``` +PUT /projects/:id/services/jenkins +``` + +Parameters: + +- `jenkins_url` (**required**) - Jenkins URL like http://jenkins.example.com +- `project_name` (**required**) - The URL-friendly project name. Example: my_project_name +- `username` (optional) - A user with access to the Jenkins server, if applicable +- `password` (optional) - The password of the user + +### Delete Jenkins CI service + +Delete Jenkins CI service for a project. + +``` +DELETE /projects/:id/services/jenkins +``` + +### Get Jenkins CI service settings + +Get Jenkins CI service settings for a project. + +``` +GET /projects/:id/services/jenkins +``` + + +## Jenkins CI (Deprecated) Service + +A continuous integration and build server + +### Create/Edit Jenkins CI (Deprecated) service + +Set Jenkins CI (Deprecated) service for a project. + +``` +PUT /projects/:id/services/jenkins-deprecated +``` + +Parameters: + +- `project_url` (**required**) - Jenkins project URL like http://jenkins.example.com/job/my-project/ +- `multiproject_enabled` (optional) - Multi-project mode is configured in Jenkins GitLab Hook plugin +- `pass_unstable` (optional) - Unstable builds will be treated as passing + +### Delete Jenkins CI (Deprecated) service + +Delete Jenkins CI (Deprecated) service for a project. + +``` +DELETE /projects/:id/services/jenkins-deprecated +``` + +### Get Jenkins CI (Deprecated) service settings + +Get Jenkins CI (Deprecated) service settings for a project. + +``` +GET /projects/:id/services/jenkins-deprecated +``` + [jira-doc]: ../user/project/integrations/jira.md [old-jira-api]: https://gitlab.com/gitlab-org/gitlab-ce/blob/8-13-stable/doc/api/services.md#jira diff --git a/doc/api/settings.md b/doc/api/settings.md index c2a1f7feefd..1b7c6517dc5 100644 --- a/doc/api/settings.md +++ b/doc/api/settings.md @@ -63,6 +63,7 @@ Example response: "performance_bar_allowed_group_id": 42, "instance_statistics_visibility_private": false, "user_show_add_ssh_key_message": true, + "file_template_project_id": 1, "local_markdown_version": 0 } ``` @@ -111,6 +112,10 @@ Example response: "plantuml_url": null, "terminal_max_session_time": 0, "polling_interval_multiplier": 1.0, + "external_authorization_service_enabled": true, + "external_authorization_service_url": "https://authorize.me", + "external_authorization_service_default_label": "default", + "external_authorization_service_timeout": 0.5, "rsa_key_restriction": 0, "dsa_key_restriction": 0, "ecdsa_key_restriction": 0, @@ -121,6 +126,7 @@ Example response: "performance_bar_allowed_group_id": 42, "instance_statistics_visibility_private": false, "user_show_add_ssh_key_message": true, + "file_template_project_id": 1, "local_markdown_version": 0 } ``` @@ -138,10 +144,12 @@ are listed in the descriptions of the relevant settings. | `after_sign_up_text` | string | no | Text shown to the user after signing up | | `akismet_api_key` | string | required by: `akismet_enabled` | API key for akismet spam protection. | | `akismet_enabled` | boolean | no | (**If enabled, requires:** `akismet_api_key`) Enable or disable akismet spam protection. | +| `allow_group_owners_to_manage_ldap` | boolean | no | **(Premium)** Set to `true` to allow group owners to manage LDAP | | `allow_local_requests_from_hooks_and_services` | boolean | no | Allow requests to the local network from hooks and services. | | `authorized_keys_enabled` | boolean | no | By default, we write to the `authorized_keys` file to support Git over SSH without additional configuration. GitLab can be optimized to authenticate SSH keys via the database file. Only disable this if you have configured your OpenSSH server to use the AuthorizedKeysCommand. | | `auto_devops_domain` | string | no | Specify a domain to use by default for every project's Auto Review Apps and Auto Deploy stages. | | `auto_devops_enabled` | boolean | no | Enable Auto DevOps for projects by default. It will automatically build, test, and deploy applications based on a predefined CI/CD configuration. | +| `check_namespace_plan` | boolean | no | **(Premium)** Enabling this will make only licensed EE features available to projects if the project namespace's plan includes the feature or if the project is public. | | `clientside_sentry_dsn` | string | required by: `clientside_sentry_enabled` | Clientside Sentry Data Source Name. | | `clientside_sentry_enabled` | boolean | no | (**If enabled, requires:** `clientside_sentry_dsn`) Enable Sentry error reporting for the client side. | | `container_registry_token_expire_delay` | integer | no | Container Registry token duration in minutes. | @@ -158,10 +166,28 @@ are listed in the descriptions of the relevant settings. | `dsa_key_restriction` | integer | no | The minimum allowed bit length of an uploaded DSA key. Default is `0` (no restriction). `-1` disables DSA keys. | | `ecdsa_key_restriction` | integer | no | The minimum allowed curve size (in bits) of an uploaded ECDSA key. Default is `0` (no restriction). `-1` disables ECDSA keys. | | `ed25519_key_restriction` | integer | no | The minimum allowed curve size (in bits) of an uploaded ED25519 key. Default is `0` (no restriction). `-1` disables ED25519 keys. | +| `elasticsearch_aws` | boolean | no | **(Premium)** Enable the use of AWS hosted Elasticsearch | +| `elasticsearch_aws_access_key` | string | no | **(Premium)** AWS IAM access key | +| `elasticsearch_aws_region` | string | no | **(Premium)** The AWS region the elasticsearch domain is configured | +| `elasticsearch_aws_secret_access_key` | string | no | **(Premium)** AWS IAM secret access key | +| `elasticsearch_experimental_indexer` | boolean | no | **(Premium)** Use the experimental elasticsearch indexer. More info: https://gitlab.com/gitlab-org/gitlab-elasticsearch-indexer | +| `elasticsearch_indexing` | boolean | no | **(Premium)** Enable Elasticsearch indexing | +| `elasticsearch_search` | boolean | no | **(Premium)** Enable Elasticsearch search | +| `elasticsearch_url` | string | no | **(Premium)** The url to use for connecting to Elasticsearch. Use a comma-separated list to support cluster (e.g., "http://localhost:9200, http://localhost:9201"). If your Elasticsearch instance is password protected, pass the `username:password` in the URL (e.g., `http://<username>:<password>@<elastic_host>:9200/`). | +| `email_additional_text` | string | no | **(Premium)** Additional text added to the bottom of every email for legal/auditing/compliance reasons | | `email_author_in_body` | boolean | no | Some email servers do not support overriding the email sender name. Enable this option to include the name of the author of the issue, merge request or comment in the email body instead. | | `enabled_git_access_protocol` | string | no | Enabled protocols for Git access. Allowed values are: `ssh`, `http`, and `nil` to allow both protocols. | | `enforce_terms` | boolean | no | (**If enabled, requires:** `terms`) Enforce application ToS to all users. | -| `first_day_of_week` | integer | no | Start day of the week for calendar views and date pickers. Valid values are `0` (default) for Sunday, `1` for Monday, and `6` for Saturday. | +| `external_auth_client_cert` | string | no | **(Premium)** (**If enabled, requires:** `external_auth_client_key`) The certificate to use to authenticate with the external authorization service | +| `external_auth_client_key` | string | required by: `external_auth_client_cert` | **(Premium)** Private key for the certificate when authentication is required for the external authorization service, this is encrypted when stored | +| `external_auth_client_key_pass` | string | no | **(Premium)** Passphrase to use for the private key when authenticating with the external service this is encrypted when stored | +| `external_authorization_service_enabled` | boolean | no | **(Premium)** (**If enabled, requires:** `external_authorization_service_default_label`, `external_authorization_service_timeout` and `external_authorization_service_url` ) Enable using an external authorization service for accessing projects | +| `external_authorization_service_default_label` | string | required by: `external_authorization_service_enabled` | **(Premium)** The default classification label to use when requesting authorization and no classification label has been specified on the project | +| `external_authorization_service_timeout` | float | required by: `external_authorization_service_enabled` | **(Premium)** The timeout after which an authorization request is aborted, in seconds. When a request times out, access is denied to the user. (min: 0.001, max: 10, step: 0.001) | +| `external_authorization_service_url` | string | required by: `external_authorization_service_enabled` | **(Premium)** URL to which authorization requests will be directed | +| `file_template_project_id` | integer | no | **(Premium)** The ID of a project to load custom file templates from | +| `first_day_of_week` | integer | no | Start day of the week for calendar views and date pickers. Valid values are `0` (default) for Sunday, `1` for Monday, and `6` for Saturday. | +| `geo_status_timeout` | integer | no | **(Premium)** The amount of seconds after which a request to get a secondary node status will time out. | | `gitaly_timeout_default` | integer | no | Default Gitaly timeout, in seconds. This timeout is not enforced for git fetch/push operations or Sidekiq jobs. Set to `0` to disable timeouts. | | `gitaly_timeout_fast` | integer | no | Gitaly fast operation timeout, in seconds. Some Gitaly operations are expected to be fast. If they exceed this threshold, there may be a problem with a storage shard and 'failing fast' can help maintain the stability of the GitLab instance. Set to `0` to disable timeouts. | | `gitaly_timeout_medium` | integer | no | Medium Gitaly timeout, in seconds. This should be a value between the Fast and the Default timeout. Set to `0` to disable timeouts. | @@ -170,6 +196,7 @@ are listed in the descriptions of the relevant settings. | `help_page_hide_commercial_content` | boolean | no | Hide marketing-related entries from help. | | `help_page_support_url` | string | no | Alternate support URL for help page. | | `help_page_text` | string | no | Custom text displayed on the help page. | +| `help_text` | string | no | **(Premium)** GitLab server administrator information | | `hide_third_party_offers` | boolean | no | Do not display offers from third parties within GitLab. | | `home_page_url` | string | no | Redirect to this URL when not logged in. | | `housekeeping_bitmaps_enabled` | boolean | required by: `housekeeping_enabled` | Enable Git pack file bitmap creation. | @@ -192,6 +219,9 @@ are listed in the descriptions of the relevant settings. | `metrics_sample_interval` | integer | required by: `metrics_enabled` | The sampling interval in seconds. | | `metrics_timeout` | integer | required by: `metrics_enabled` | The amount of seconds after which InfluxDB will time out. | | `mirror_available` | boolean | no | Allow mirrors to be set up for projects. If disabled, only admins will be able to set up mirrors in projects. | +| `mirror_capacity_threshold` | integer | no | **(Premium)** Minimum capacity to be available before scheduling more mirrors preemptively | +| `mirror_max_capacity` | integer | no | **(Premium)** Maximum number of mirrors that can be synchronizing at the same time. | +| `mirror_max_delay` | integer | no | **(Premium)** Maximum time (in minutes) between updates that a mirror can have when scheduled to synchronize. | | `pages_domain_verification_enabled` | boolean | no | Require users to prove ownership of custom domains. Domain verification is an essential security measure for public GitLab sites. Users are required to demonstrate they control a domain before it is enabled. | | `password_authentication_enabled_for_git` | boolean | no | Enable authentication for Git over HTTP(S) via a GitLab account password. Default is `true`. | | `password_authentication_enabled_for_web` | boolean | no | Enable authentication for the web interface via a GitLab account password. Default is `true`. | @@ -203,10 +233,12 @@ are listed in the descriptions of the relevant settings. | `polling_interval_multiplier` | decimal | no | Interval multiplier used by endpoints that perform polling. Set to `0` to disable polling. | | `project_export_enabled` | boolean | no | Enable project export. | | `prometheus_metrics_enabled` | boolean | no | Enable prometheus metrics. | +| `pseudonymizer_enabled` | boolean | no | **(Premium)** When enabled, GitLab will run a background job that will produce pseudonymized CSVs of the GitLab database that will be uploaded to your configured object storage directory. | `recaptcha_enabled` | boolean | no | (**If enabled, requires:** `recaptcha_private_key` and `recaptcha_site_key`) Enable recaptcha. | | `recaptcha_private_key` | string | required by: `recaptcha_enabled` | Private key for recaptcha. | | `recaptcha_site_key` | string | required by: `recaptcha_enabled` | Site key for recaptcha. | | `repository_checks_enabled` | boolean | no | GitLab will periodically run `git fsck` in all project and wiki repositories to look for silent disk corruption issues. | +| `repository_size_limit` | integer | no | **(Premium)** Size limit per repository (MB) | | `repository_storages` | array of strings | no | A list of names of enabled storage paths, taken from `gitlab.yml`. New projects will be created in one of these stores, chosen at random. | | `require_two_factor_authentication` | boolean | no | (**If enabled, requires:** `two_factor_grace_period`) Require all users to set up Two-factor authentication. | | `restricted_visibility_levels` | array of strings | no | Selected levels cannot be used by non-admin users for groups, projects or snippets. Can take `private`, `internal` and `public` as a parameter. Default is `null` which means there is no restriction. | @@ -215,11 +247,16 @@ are listed in the descriptions of the relevant settings. | `sentry_dsn` | string | required by: `sentry_enabled` | Sentry Data Source Name. | | `sentry_enabled` | boolean | no | (**If enabled, requires:** `sentry_dsn`) Sentry is an error reporting and logging tool which is currently not shipped with GitLab, available at <https://sentry.io>. | | `session_expire_delay` | integer | no | Session duration in minutes. GitLab restart is required to apply changes | -| `shared_runners_enabled` | boolean | no | (**If enabled, requires:** `shared_runners_text`) Enable shared runners for new projects. | +| `shared_runners_enabled` | boolean | no | (**If enabled, requires:** `shared_runners_text` and `shared_runners_minutes`) Enable shared runners for new projects. | +| `shared_runners_minutes` | integer | required by: `shared_runners_enabled` | **(Premium)** Set the maximum number of pipeline minutes that a group can use on shared Runners per month. | | `shared_runners_text` | string | required by: `shared_runners_enabled` | Shared runners text. | | `sign_in_text` | string | no | Text on the login page. | | `signin_enabled` | string | no | (Deprecated: Use `password_authentication_enabled_for_web` instead) Flag indicating if password authentication is enabled for the web interface. | | `signup_enabled` | boolean | no | Enable registration. Default is `true`. | +| `slack_app_enabled` | boolean | no | **(Premium)** (**If enabled, requires:** `slack_app_id`, `slack_app_secret` and `slack_app_secret`) Enable Slack app. | +| `slack_app_id` | string | required by: `slack_app_enabled` | **(Premium)** The app id of the Slack-app. | +| `slack_app_secret` | string | required by: `slack_app_enabled` | **(Premium)** The app secret of the Slack-app. | +| `slack_app_verification_token` | string | required by: `slack_app_enabled` | **(Premium)** The verification token of the Slack-app. | | `terminal_max_session_time` | integer | no | Maximum time for web terminal websocket connection (in seconds). Set to `0` for unlimited time. | | `terms` | text | required by: `enforce_terms` | (**Required by:** `enforce_terms`) Markdown content for the ToS. | | `throttle_authenticated_api_enabled` | boolean | no | (**If enabled, requires:** `throttle_authenticated_api_period_in_seconds` and `throttle_authenticated_api_requests_per_period`) Enable authenticated API request rate limit. Helps reduce request volume (e.g. from crawlers or abusive bots). | diff --git a/doc/api/users.md b/doc/api/users.md index b0977810120..f406435e640 100644 --- a/doc/api/users.md +++ b/doc/api/users.md @@ -2,6 +2,8 @@ ## List users +Active users = Total accounts - Blocked users + Get a list of users. This function takes pagination parameters `page` and `per_page` to restrict the list of users. @@ -256,7 +258,8 @@ Parameters: "can_create_project": true, "two_factor_enabled": true, "external": false, - "private_profile": false + "private_profile": false, + "shared_runners_minutes_limit": 133 } ``` @@ -298,6 +301,7 @@ Parameters: - `external` (optional) - Flags the user as external - true or false(default) - `avatar` (optional) - Image file for user's avatar - `private_profile` (optional) - User's profile is private - true or false +- `shared_runners_minutes_limit` (optional) - Pipeline minutes quota for this user ## User modification @@ -328,6 +332,7 @@ Parameters: - `can_create_group` (optional) - User can create groups - true or false - `skip_reconfirmation` (optional) - Skip reconfirmation - true or false (default) - `external` (optional) - Flags the user as external - true or false(default) +- `shared_runners_minutes_limit` (optional) - Pipeline minutes quota for this user - `avatar` (optional) - Image file for user's avatar - `private_profile` (optional) - User's profile is private - true or false @@ -1149,8 +1154,6 @@ settings page. POST /users/:user_id/impersonation_tokens ``` -Parameters: - | Attribute | Type | Required | Description | | --------- | ---- | -------- | ----------- | | `user_id` | integer | yes | The ID of the user | diff --git a/doc/articles/how_to_configure_ldap_gitlab_ee/index.md b/doc/articles/how_to_configure_ldap_gitlab_ee/index.md new file mode 100644 index 00000000000..0afbf746d2c --- /dev/null +++ b/doc/articles/how_to_configure_ldap_gitlab_ee/index.md @@ -0,0 +1 @@ +This document was moved to [another location](../../administration/auth/how_to_configure_ldap_gitlab_ee/index.md). diff --git a/doc/ci/README.md b/doc/ci/README.md index 0ff12f16d9d..bd29824781b 100644 --- a/doc/ci/README.md +++ b/doc/ci/README.md @@ -48,7 +48,7 @@ into more features: | [Introduction to pipelines and jobs](pipelines.md) | Provides an overview of GitLab CI/CD and jobs. | | [CI/CD Variables](variables/README.md) | How environment variables can be configured and made available in pipelines. | | [Where variables can be used](variables/where_variables_can_be_used.md) | A deeper look into where and how CI/CD variables can be used. | -| [User](../user/permissions.md#gitlab-cicd-permissions) and [job](../user/permissions.md#job-permissions) permissions | Learn about the access levels a user can have for performing certain CI actions. | +| [User](../user/permissions.md#gitlab-cicd-permissions) and [job](../user/permissions.md#job-permissions) permissions | Learn about the access levels a user can have for performing certain CI actions. | | [Configuring GitLab Runners](runners/README.md) | Documentation for configuring [GitLab Runner](https://docs.gitlab.com/runner/). | | [Introduction to environments and deployments](environments.md) | Learn how to separate your jobs into environments and use them for different purposes like testing, building and, deploying. | | [Job artifacts](../user/project/pipelines/job_artifacts.md) | Learn about the output of jobs. | @@ -61,6 +61,10 @@ into more features: | [Connecting GitLab with a Kubernetes cluster](../user/project/clusters/index.md) | Integrate one or more Kubernetes clusters to your project. | | [ChatOps](chatops/README.md) | Trigger CI jobs from chat, with results sent back to the channel. | | [Interactive web terminals](interactive_web_terminal/index.md) | Open an interactive web terminal to debug the running jobs. | +| [Review Apps](review_apps/index.md) | Configure GitLab CI/CD to preview code changes in a per-branch basis. | +| [Deploy Boards](../user/project/deploy_boards.md) **[PREMIUM]** | Check the current health and status of each CI/CD environment running on Kubernetes. | +| [GitLab CI/CD for external repositories](ci_cd_for_external_repos/index.md) **[PREMIUM]** | Get the benefits of GitLab CI/CD combined with repositories in GitHub and BitBucket Cloud. | +| [Protected environments](environments/protected_environments.md) **[PREMIUM]** | Ensure that only people with the right privileges can deploy to an environment. | ### GitLab Pages diff --git a/doc/ci/ci_cd_for_external_repos/bitbucket_integration.md b/doc/ci/ci_cd_for_external_repos/bitbucket_integration.md new file mode 100644 index 00000000000..b20bf1ced7e --- /dev/null +++ b/doc/ci/ci_cd_for_external_repos/bitbucket_integration.md @@ -0,0 +1,147 @@ +# Using GitLab CI/CD with a Bitbucket Cloud repository **[PREMIUM]** + +GitLab CI/CD can be used with Bitbucket Cloud by creating a +[CI/CD project](../../user/project/ci_cd_for_external_repo.md) and connecting +your Git repository via URL. + +1. In GitLab create a **CI/CD for external repo**, select **Repo by URL** and + create the project. + +  + + GitLab will import the repository and enable [Pull Mirroring][pull-mirroring]. + +1. In GitLab create a + [Personal Access Token](../../user/profile/personal_access_tokens.md) + with `api` scope. This will be used to authenticate requests from the web + hook that will be created in Bitbucket to notify GitLab of new commits. + +1. In Bitbucket from **Settings > Webhooks** create a new web hook to notify + GitLab of new commits. + + The web hook URL should be set to the GitLab API to trigger pull mirroring, + using the Personal Access Token we just generated for authentication. + + ``` + https://gitlab.com/api/v4/projects/<NAMESPACE>%2F<PROJECT>/mirror/pull?private_token=<PERSONAL_ACCESS_TOKEN> + ``` + + The web hook Trigger should be set to 'Repository Push'. + +  + + After saving, test the web hook by pushing a change to your Bitbucket + repository. + +1. In Bitbucket create an **App Password** from **Bitbucket Settings > App + Passwords** to authenticate the build status script setting commit build + statuses in Bitbucket. Repository write permissions are required. + +  + +1. In GitLab from **Settings > CI/CD > Environment variables** add variables to allow + communication with Bitbucket via the Bitbucket API. + + `BITBUCKET_ACCESS_TOKEN`: the Bitbucket app password created above + + `BITBUCKET_USERNAME`: the username of the Bitbucket account + + `BITBUCKET_NAMESPACE`: set this if your GitLab and Bitbucket namespaces differ + + `BITBUCKET_REPOSITORY`: set this if your GitLab and Bitbucket project names differ + +1. In Bitbucket add a script to push the pipeline status to Bitbucket. + + > Note: changes made in GitLab will be overwritten by any changes made + upstream in Bitbucket. + + Create a file `build_status` and insert the script below and run + `chmod +x build_status` in your terminal to make the script executable. + + ```bash + #!/usr/bin/env bash + + # Push GitLab CI/CD build status to Bitbucket Cloud + + if [ -z "$BITBUCKET_ACCESS_TOKEN" ]; then + echo "ERROR: BITBUCKET_ACCESS_TOKEN is not set" + exit 1 + fi + if [ -z "$BITBUCKET_USERNAME" ]; then + echo "ERROR: BITBUCKET_USERNAME is not set" + exit 1 + fi + if [ -z "$BITBUCKET_NAMESPACE" ]; then + echo "Setting BITBUCKET_NAMESPACE to $CI_PROJECT_NAMESPACE" + BITBUCKET_NAMESPACE=$CI_PROJECT_NAMESPACE + fi + if [ -z "$BITBUCKET_REPOSITORY" ]; then + echo "Setting BITBUCKET_REPOSITORY to $CI_PROJECT_NAME" + BITBUCKET_REPOSITORY=$CI_PROJECT_NAME + fi + + BITBUCKET_API_ROOT="https://api.bitbucket.org/2.0" + BITBUCKET_STATUS_API="$BITBUCKET_API_ROOT/repositories/$BITBUCKET_NAMESPACE/$BITBUCKET_REPOSITORY/commit/$CI_COMMIT_SHA/statuses/build" + BITBUCKET_KEY="ci/gitlab-ci/$CI_JOB_NAME" + + case "$BUILD_STATUS" in + running) + BITBUCKET_STATE="INPROGRESS" + BITBUCKET_DESCRIPTION="The build is running!" + ;; + passed) + BITBUCKET_STATE="SUCCESSFUL" + BITBUCKET_DESCRIPTION="The build passed!" + ;; + failed) + BITBUCKET_STATE="FAILED" + BITBUCKET_DESCRIPTION="The build failed." + ;; + esac + + echo "Pushing status to $BITBUCKET_STATUS_API..." + curl --request POST $BITBUCKET_STATUS_API \ + --user $BITBUCKET_USERNAME:$BITBUCKET_ACCESS_TOKEN \ + --header "Content-Type:application/json" \ + --silent \ + --data "{ \"state\": \"$BITBUCKET_STATE\", \"key\": \"$BITBUCKET_KEY\", \"description\": + \"$BITBUCKET_DESCRIPTION\",\"url\": \"$CI_PROJECT_URL/-/jobs/$CI_JOB_ID\" }" + ``` + +1. Still in Bitbucket, create a `.gitlab-ci.yml` file to use the script to push + pipeline success and failures to Bitbucket. + + ``` + stages: + - test + - ci_status + + unit-tests: + script: + - echo "Success. Add your tests!" + + success: + stage: ci_status + before_script: + - "" + after_script: + - "" + script: + - BUILD_STATUS=passed BUILD_KEY=push ./build_status + when: on_success + + failure: + stage: ci_status + before_script: + - "" + after_script: + - "" + script: + - BUILD_STATUS=failed BUILD_KEY=push ./build_status + when: on_failure + ``` + +GitLab is now configured to mirror changes from Bitbucket, run CI/CD pipelines +configured in `.gitlab-ci.yml` and push the status to Bitbucket. + +[pull-mirroring]: ../../workflow/repository_mirroring.md#pulling-from-a-remote-repository-starter diff --git a/doc/ci/ci_cd_for_external_repos/github_integration.md b/doc/ci/ci_cd_for_external_repos/github_integration.md new file mode 100644 index 00000000000..1a3bb87514f --- /dev/null +++ b/doc/ci/ci_cd_for_external_repos/github_integration.md @@ -0,0 +1,111 @@ +# Using GitLab CI/CD with a GitHub repository **[PREMIUM]** + +GitLab CI/CD can be used with **GitHub.com** and **GitHub Enterprise** by +creating a [CI/CD project](../../user/project/ci_cd_for_external_repo.md) to connect your GitHub repository to +GitLab. + +NOTE: **Note:** +To use **GitHub Enterprise** with **GitLab.com** you should use the +manual method. + +## Connect with GitHub integration + +If the [GitHub integration](../../integration/github.md) has been enabled by your GitLab +administrator: + +NOTE: **Note:** +Due to a 10-token limitation on the [GitHub OAuth Implementation](https://developer.github.com/apps/building-oauth-apps/authorizing-oauth-apps/#creating-multiple-tokens-for-oauth-apps), +if you import more than 10 times, your oldest imported project's token will be +revoked. See issue [#9147](https://gitlab.com/gitlab-org/gitlab-ee/issues/9147) +for more information. + +1. In GitLab create a **CI/CD for external repo** project and select + **GitHub**. + +  + +1. Once authenticated, you will be redirected to a list of your repositories to + connect. Click **Connect** to select the repository. + +  + +1. In GitHub, add a `.gitlab-ci.yml` to [configure GitLab CI/CD](../quick_start/README.md). + +GitLab will import the project, enable [Pull Mirroring](../../workflow/repository_mirroring.md#pulling-from-a-remote-repository-starter), enable +[GitHub project integration](../../user/project/integrations/github.md), and create a web hook +on GitHub to notify GitLab of new commits. + +## Connect with Personal Access Token + +NOTE: **Note:** Personal access tokens can only be used to connect GitHub.com +repositories to GitLab. + +If you are not using the [GitHub integration](../../integration/github.md), you can +still perform a one-off authorization with GitHub to grant GitLab access your +repositories: + +1. Open https://github.com/settings/tokens/new to create a **Personal Access + Token**. This token with be used to access your repository and push commit + statuses to GitHub. + + The `repo` and `admin:repo_hook` should be enable to allow GitLab access to + your project, update commit statuses, and create a web hook to notify + GitLab of new commits. + +1. In GitLab create a **CI/CD for external repo** project and select + **GitHub**. + +  + +1. Paste the token into the **Personal access token** field and click **List + Repositories**. Click **Connect** to select the repository. + +1. In GitHub, add a `.gitlab-ci.yml` to [configure GitLab CI/CD](../quick_start/README.md). + +GitLab will import the project, enable [Pull Mirroring](../../workflow/repository_mirroring.md#pulling-from-a-remote-repository-starter), enable +[GitHub project integration](../../user/project/integrations/github.md), and create a web hook +on GitHub to notify GitLab of new commits. + +## Connect manually + +If the [GitHub integration](../../integration/github.md) is not enabled, or is enabled +for a different GitHub instance, you GitLab CI/CD can be manually enabled for +your repository. + +1. In GitHub open https://github.com/settings/tokens/new create a **Personal + Access Token.** GitLab will use this token to access your repository and + push commit statuses. + + Enter a **Token description** and update the scope to allow: + + `repo` so that GitLab can access your project and update commit statuses + +1. In GitLab create a **CI/CD project** using the Git URL option and the HTTPS + URL for your GitHub repository. If your project is private, use the personal + access token you just created for authentication. + + GitLab will automatically configure polling-based pull mirroring. + +1. Still in GitLab, enable the [GitHub project integration](../../user/project/integrations/github.md) + from **Settings > Integrations.** + + Check the **Active** checkbox to enable the integration, paste your + personal access token and HTTPS repository URL into the form, and **Save.** + +1. Still in GitLab create a **Personal Access Token** with `API` scope to + authenticate the GitHub web hook notifying GitLab of new commits. + +1. In GitHub from **Settings > Webhooks** create a web hook to notify GitLab of + new commits. + + The web hook URL should be set to the GitLab API to + [trigger pull mirroring](../../api/projects.md#start-the-pull-mirroring-process-for-a-project-starter), + using the GitLab personal access token we just created. + + ``` + https://gitlab.com/api/v4/projects/<NAMESPACE>%2F<PROJECT>/mirror/pull?private_token=<PERSONAL_ACCESS_TOKEN> + ``` + +  + +1. In GitHub add a `.gitlab-ci.yml` to configure GitLab CI/CD. diff --git a/doc/ci/ci_cd_for_external_repos/img/bitbucket_app_password.png b/doc/ci/ci_cd_for_external_repos/img/bitbucket_app_password.png Binary files differnew file mode 100644 index 00000000000..ac5be3c3058 --- /dev/null +++ b/doc/ci/ci_cd_for_external_repos/img/bitbucket_app_password.png diff --git a/doc/ci/ci_cd_for_external_repos/img/bitbucket_webhook.png b/doc/ci/ci_cd_for_external_repos/img/bitbucket_webhook.png Binary files differnew file mode 100644 index 00000000000..0a3476d9035 --- /dev/null +++ b/doc/ci/ci_cd_for_external_repos/img/bitbucket_webhook.png diff --git a/doc/ci/ci_cd_for_external_repos/img/ci_cd_for_external_repo.png b/doc/ci/ci_cd_for_external_repos/img/ci_cd_for_external_repo.png Binary files differnew file mode 100644 index 00000000000..f068688146b --- /dev/null +++ b/doc/ci/ci_cd_for_external_repos/img/ci_cd_for_external_repo.png diff --git a/doc/ci/ci_cd_for_external_repos/img/external_repository.png b/doc/ci/ci_cd_for_external_repos/img/external_repository.png Binary files differnew file mode 100644 index 00000000000..b850d91f56b --- /dev/null +++ b/doc/ci/ci_cd_for_external_repos/img/external_repository.png diff --git a/doc/ci/ci_cd_for_external_repos/img/github_omniauth.png b/doc/ci/ci_cd_for_external_repos/img/github_omniauth.png Binary files differnew file mode 100644 index 00000000000..71a3a057a41 --- /dev/null +++ b/doc/ci/ci_cd_for_external_repos/img/github_omniauth.png diff --git a/doc/ci/ci_cd_for_external_repos/img/github_omniauth_list.png b/doc/ci/ci_cd_for_external_repos/img/github_omniauth_list.png Binary files differnew file mode 100644 index 00000000000..3f2059504f5 --- /dev/null +++ b/doc/ci/ci_cd_for_external_repos/img/github_omniauth_list.png diff --git a/doc/ci/ci_cd_for_external_repos/img/github_push_webhook.png b/doc/ci/ci_cd_for_external_repos/img/github_push_webhook.png Binary files differnew file mode 100644 index 00000000000..e8c17d664e1 --- /dev/null +++ b/doc/ci/ci_cd_for_external_repos/img/github_push_webhook.png diff --git a/doc/ci/ci_cd_for_external_repos/img/github_repo_list.png b/doc/ci/ci_cd_for_external_repos/img/github_repo_list.png Binary files differnew file mode 100644 index 00000000000..73579dd3cf1 --- /dev/null +++ b/doc/ci/ci_cd_for_external_repos/img/github_repo_list.png diff --git a/doc/ci/ci_cd_for_external_repos/index.md b/doc/ci/ci_cd_for_external_repos/index.md new file mode 100644 index 00000000000..e57be551bef --- /dev/null +++ b/doc/ci/ci_cd_for_external_repos/index.md @@ -0,0 +1,30 @@ +# GitLab CI/CD for external repositories **[PREMIUM]** + +NOTE: **Note:** +To [promote its release](https://about.gitlab.com/2018/03/22/gitlab-10-6-released/#gitlab-cicd-for-external-repos), GitLab.com users are receiving this feature for free through March 22, 2019. + +>[Introduced][ee-4642] in [GitLab Premium][eep] 10.6. + +GitLab CI/CD can be used with GitHub or any other Git server. +Instead of moving your entire project to GitLab, you can connect your +external repository to get the benefits of GitLab CI/CD. + +- [GitHub](github_integration.md) +- [Bitbucket Cloud](bitbucket_integration.md) + +Connecting an external repository will set up [repository mirroring][mirroring] +and create a lightweight project where issues, merge requests, wiki, and +snippets disabled. These features +[can be re-enabled later][settings]. + +1. From your GitLab dashboard click **New project** +1. Switch to the **CI/CD for external repo** tab +1. Choose **GitHub** or **Repo by URL** +1. The next steps are similar to the [import flow](../../user/project/import/index.md) + + + +[ee-4642]: https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/4642 +[eep]: https://about.gitlab.com/pricing/ +[mirroring]: ../../workflow/repository_mirroring.md +[settings]: ../../user/project/settings/index.md#sharing-and-permissions diff --git a/doc/ci/environments.md b/doc/ci/environments.md index fe66f7e3c28..3327445724d 100644 --- a/doc/ci/environments.md +++ b/doc/ci/environments.md @@ -598,6 +598,38 @@ fetch line: fetch = +refs/environments/*:refs/remotes/origin/environments/* ``` +## Scoping environments with specs **[PREMIUM]** + +Some GitLab [Enterprise Edition](https://about.gitlab.com/pricing/) features can behave differently for each [Environment](#introduction-to-environments-and-deployments). +For example, you can [create a secret variable to be injected only into a production environment](variables/README.md#limiting-environment-scopes-of-variables-premium). +In most cases, these features use the _environment specs_ mechanism, which offers +an efficient way to implement scoping within each environment group. + +Let's say there are four environments: + +- `production` +- `staging` +- `review/feature-1` +- `review/feature-2` + +Each environment can be matched with the following environment spec: + +| Environment Spec | `production` | `staging` | `review/feature-1` | `review/feature-2` | +| ---------------- | ---------- | ------- | ---------------- | ---------------- | +| * | Matched | Matched | Matched | Matched | +| production | Matched | | | | +| staging | | Matched | | | +| review/* | | | Matched | Matched | +| review/feature-1 | | | Matched | | + +As you can see, you can use specific matching for selecting a particular environment, +and also use wildcard matching (`*`) for selecting a particular environment group, +such as [Review apps](review_apps/index.md) (`review/*`). + +NOTE: **Note:** +The most _specific_ spec takes precedence over the other wildcard matching. +In this case, `review/feature-1` spec takes precedence over `review/*` and `*` specs. + ## Limitations 1. You are limited to use only the [CI predefined variables][variables] in the @@ -611,6 +643,7 @@ Below are some links you may find interesting: - [The `.gitlab-ci.yml` definition of environments](yaml/README.md#environment) - [A blog post on Deployments & Environments](https://about.gitlab.com/2016/08/26/ci-deployment-and-environments/) - [Review Apps - Use dynamic environments to deploy your code for every branch](review_apps/index.md) +- [Deploy Boards for your applications running on Kubernetes](../user/project/deploy_boards.md) [Pipelines]: pipelines.md [jobs]: yaml/README.md#jobs diff --git a/doc/ci/environments/img/protected_environments_form.png b/doc/ci/environments/img/protected_environments_form.png Binary files differnew file mode 100644 index 00000000000..7c04903a264 --- /dev/null +++ b/doc/ci/environments/img/protected_environments_form.png diff --git a/doc/ci/environments/img/protected_environments_list.png b/doc/ci/environments/img/protected_environments_list.png Binary files differnew file mode 100644 index 00000000000..a1b9cc1e587 --- /dev/null +++ b/doc/ci/environments/img/protected_environments_list.png diff --git a/doc/ci/environments/protected_environments.md b/doc/ci/environments/protected_environments.md new file mode 100644 index 00000000000..23df3f84039 --- /dev/null +++ b/doc/ci/environments/protected_environments.md @@ -0,0 +1,48 @@ +# Protected Environments **[PREMIUM]** + +> [Introduced][6303] in [GitLab Premium][ee] 11.3. + +## Overview + +[Environments](../environments.md) can be used for different scopes, some of +them are just for testing while others are for production. As deploy jobs could +be raised by different users with different roles, it is very important that +specific environments are "protected" to avoid unauthorized people to affect them. + +By default, a protected environment does one thing: it ensures that only people +with the right privileges can deploy to it, thus keeping it safe. + +NOTE: **Note**: +A GitLab admin is always allowed to use environments, even if they are protected. + +To protect, update, or unprotect an environment, you need to have at least +[Maintainer permissions](../../user/permissions.md). + +## Configuring protected environments + +To protect an environment: + +1. Navigate to your project's **Settings ➔ CI/CD**. +1. Scroll to find the **Protected Environments** section. +1. From the **Environment** dropdown menu, select the environment you want to protect and + click **Protect**. +1. In the "Allowed to Deploy" dropdown menu, you can select the role and/or the + users and/or the groups you want to have deploy access. There are some + considerations to have in mind: + - There are two roles to choose from: + - **Maintainers**: will allow access to all maintainers in the project. + - **Developers**: will allow access to all maintainers and all developers in the project. + - You can only select groups that are associated with the project. + - Only users that have at least Developer permission level will appear on + the "Allowed to Deploy" dropdown menu. + +1. Once done, the protected environment will appear in the "Protected Environments" + list. + +Maintainers can update existing protected environments at any time +by changing the access on "Allowed to Deploy" dropdown menu. Similarly, +to unprotect a protected environment, Maintainers need to click the +**Unprotect** button of the respective environment. + +[ee]: https://about.gitlab.com/pricing/ +[6303]: https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/6303 diff --git a/doc/ci/examples/README.md b/doc/ci/examples/README.md index a1c997d1de6..f33c641bbea 100644 --- a/doc/ci/examples/README.md +++ b/doc/ci/examples/README.md @@ -18,30 +18,30 @@ Examples are available in several forms. As a collection of: The following table lists examples for different use cases: -| Use case | Resource | -|:-----------------------------------------------|:-----------------------------------------------------------------------------------------------------------------------------------| -| Browser performance testing | [Browser Performance Testing with the Sitespeed.io container](browser_performance.md). | -| Clojure | [Test a Clojure application with GitLab CI/CD](test-clojure-application.md). | -| Code quality analysis | [Analyze your project's Code Quality](code_quality.md). **[STARTER]** | -| Container scanning | [Container Scanning with GitLab CI/CD](container_scanning.md). | -| Dependency scanning | [Dependency Scanning with GitLab CI/CD](https://docs.gitlab.com/ee/ci/examples/dependency_scanning.html). **[ULTIMATE]** | -| Deployment with `dpl` | [Using `dpl` as deployment tool](deployment/README.md). | -| Dynamic application<br>security testing (DAST) | [Dynamic Application Security Testing with GitLab CI/CD](dast.md) **[ULTIMATE]** | -| Elixir | [Testing a Phoenix application with GitLab CI/CD](test_phoenix_app_with_gitlab_ci_cd/index.md). | -| Game development | [DevOps and Game Dev with GitLab CI/CD](devops_and_game_dev_with_gitlab_ci_cd/index.md). | -| GitLab Pages | See the [GitLab Pages](../../user/project/pages/index.md) documentation for a complete example. | -| Java | [Deploy a Spring Boot application to Cloud Foundry with GitLab CI/CD](deploy_spring_boot_to_cloud_foundry/index.md). | -| JUnit | [JUnit test reports](../junit_test_reports.md). | -| License management | [Dependencies license management with GitLab CI/CD](https://docs.gitlab.com/ee/ci/examples/license_management.html) **[ULTIMATE]** | -| Maven | [How to deploy Maven projects to Artifactory with GitLab CI/CD](artifactory_and_gitlab/index.md). | -| PHP | [Testing PHP projects](php.md). | -| PHP | [Running Composer and NPM scripts with deployment via SCP in GitLab CI/CD](deployment/composer-npm-deploy.md). | -| PHP | [Test and deploy Laravel applications with GitLab CI/CD and Envoy](laravel_with_gitlab_and_envoy/index.md). | -| Python | [Test and deploy a Python application with GitLab CI/CD](test-and-deploy-python-application-to-heroku.md). | -| Ruby | [Test and deploy a Ruby application with GitLab CI/CD](test-and-deploy-ruby-application-to-heroku.md). | -| Scala | [Test and deploy a Scala application to Heroku](test-scala-application.md). | -| Static application<br>security testing (SAST) | [Static Application Security Testing with GitLab CI/CD](https://docs.gitlab.com/ee/ci/examples/sast.html) **[ULTIMATE]** | -| Testing | [End-to-end testing with GitLab CI/CD and WebdriverIO](end_to_end_testing_webdriverio/index.md). | +| Use case | Resource | +|:-----------------------------------------------|:---------------------------------------------------------------------------------------------------------------------| +| Browser performance testing | [Browser Performance Testing with the Sitespeed.io container](browser_performance.md). | +| Clojure | [Test a Clojure application with GitLab CI/CD](test-clojure-application.md). | +| Code quality analysis | [Analyze your project's Code Quality](code_quality.md). **[STARTER]** | +| Container scanning | [Container Scanning with GitLab CI/CD](container_scanning.md). | +| Dependency scanning | [Dependency Scanning with GitLab CI/CD](dependency_scanning.md). **[ULTIMATE]** | +| Deployment with `dpl` | [Using `dpl` as deployment tool](deployment/README.md). | +| Dynamic application<br>security testing (DAST) | [Dynamic Application Security Testing with GitLab CI/CD](dast.md) **[ULTIMATE]** | +| Elixir | [Testing a Phoenix application with GitLab CI/CD](test_phoenix_app_with_gitlab_ci_cd/index.md). | +| Game development | [DevOps and Game Dev with GitLab CI/CD](devops_and_game_dev_with_gitlab_ci_cd/index.md). | +| GitLab Pages | See the [GitLab Pages](../../user/project/pages/index.md) documentation for a complete example. | +| Java | [Deploy a Spring Boot application to Cloud Foundry with GitLab CI/CD](deploy_spring_boot_to_cloud_foundry/index.md). | +| JUnit | [JUnit test reports](../junit_test_reports.md). | +| License management | [Dependencies license management with GitLab CI/CD](license_management.md) **[ULTIMATE]** | +| Maven | [How to deploy Maven projects to Artifactory with GitLab CI/CD](artifactory_and_gitlab/index.md). | +| PHP | [Testing PHP projects](php.md). | +| PHP | [Running Composer and NPM scripts with deployment via SCP in GitLab CI/CD](deployment/composer-npm-deploy.md). | +| PHP | [Test and deploy Laravel applications with GitLab CI/CD and Envoy](laravel_with_gitlab_and_envoy/index.md). | +| Python | [Test and deploy a Python application with GitLab CI/CD](test-and-deploy-python-application-to-heroku.md). | +| Ruby | [Test and deploy a Ruby application with GitLab CI/CD](test-and-deploy-ruby-application-to-heroku.md). | +| Scala | [Test and deploy a Scala application to Heroku](test-scala-application.md). | +| Static application<br>security testing (SAST) | [Static Application Security Testing with GitLab CI/CD](sast.md) **[ULTIMATE]** | +| Testing | [End-to-end testing with GitLab CI/CD and WebdriverIO](end_to_end_testing_webdriverio/index.md). | ### Contributing examples diff --git a/doc/ci/examples/browser_performance.md b/doc/ci/examples/browser_performance.md index b47038011de..7bec8d94bfc 100644 --- a/doc/ci/examples/browser_performance.md +++ b/doc/ci/examples/browser_performance.md @@ -56,7 +56,7 @@ provide a list of URLs to test, please consult TIP: **Tip:** For [GitLab Premium](https://about.gitlab.com/pricing/) users, key metrics are automatically extracted and shown right in the merge request widget. -[Learn more on Browser Performance Testing in merge requests](https://docs.gitlab.com/ee//user/project/merge_requests/browser_performance_testing.html). +[Learn more on Browser Performance Testing in merge requests](../../user/project/merge_requests/browser_performance_testing.md). ## Performance testing on Review Apps diff --git a/doc/ci/examples/code_quality.md b/doc/ci/examples/code_quality.md index 3e7d6e7e3f7..a6fa6979190 100644 --- a/doc/ci/examples/code_quality.md +++ b/doc/ci/examples/code_quality.md @@ -43,7 +43,7 @@ Due to implementation limitations we always take the latest Code Quality artifac TIP: **Tip:** For [GitLab Starter][ee] users, this information will be automatically extracted and shown right in the merge request widget. -[Learn more on Code Quality in merge requests](https://docs.gitlab.com/ee/user/project/merge_requests/code_quality.html). +[Learn more on Code Quality in merge requests](../../user/project/merge_requests/code_quality.html). ## Previous job definitions diff --git a/doc/ci/examples/container_scanning.md b/doc/ci/examples/container_scanning.md index e8e9c73d1b2..5ef41d498cc 100644 --- a/doc/ci/examples/container_scanning.md +++ b/doc/ci/examples/container_scanning.md @@ -62,7 +62,7 @@ in our case its named `clair-whitelist.yml`. TIP: **Tip:** For [GitLab Ultimate][ee] users, this information will be automatically extracted and shown right in the merge request widget. -[Learn more on Container Scanning in merge requests](https://docs.gitlab.com/ee/user/project/merge_requests/container_scanning.html). +[Learn more on Container Scanning in merge requests](../../user/project/merge_requests/container_scanning.html). CAUTION: **Caution:** Starting with GitLab 11.5, Container Scanning feature is licensed under the name `container_scanning`. diff --git a/doc/ci/examples/dast.md b/doc/ci/examples/dast.md index ab0ca13d2cf..52c4147e245 100644 --- a/doc/ci/examples/dast.md +++ b/doc/ci/examples/dast.md @@ -17,7 +17,7 @@ It can be very useful combined with [Review Apps](../review_apps/index.md). ## Example First, you need GitLab Runner with -[docker-in-docker executor](../docker/using_docker_build.md#use-docker-in-docker-executor). +[docker executor](https://docs.gitlab.com/runner/executors/docker.html). Once you set up the Runner, add a new job to `.gitlab-ci.yml` that generates the expected report: @@ -72,7 +72,7 @@ to learn more about authentication settings. TIP: **Tip:** For [GitLab Ultimate][ee] users, this information will be automatically extracted and shown right in the merge request widget. -[Learn more on DAST in merge requests](https://docs.gitlab.com/ee/user/project/merge_requests/dast.html). +[Learn more on DAST in merge requests](../../user/project/merge_requests/dast.md). ## Previous job definitions diff --git a/doc/ci/examples/dependency_scanning.md b/doc/ci/examples/dependency_scanning.md new file mode 100644 index 00000000000..48ddea73203 --- /dev/null +++ b/doc/ci/examples/dependency_scanning.md @@ -0,0 +1,98 @@ +# Dependency Scanning with GitLab CI/CD **[ULTIMATE]** + +CAUTION: **Caution:** +The job definition shown below is supported on GitLab 11.5 and later versions. +It also requires the GitLab Runner 11.5 or later. +For earlier versions, use the [previous job definitions](#previous-job-definitions). + +This example shows how to run Dependency Scanning on your +project's dependencies by using GitLab CI/CD. + + +First, you need GitLab Runner with +[docker-in-docker executor](../docker/using_docker_build.md#use-docker-in-docker-executor). + +Once you set up the Runner, add a new job to `.gitlab-ci.yml` that +generates the expected report: + +```yaml +dependency_scanning: + image: docker:stable + variables: + DOCKER_DRIVER: overlay2 + allow_failure: true + services: + - docker:stable-dind + script: + - export SP_VERSION=$(echo "$CI_SERVER_VERSION" | sed 's/^\([0-9]*\)\.\([0-9]*\).*/\1-\2-stable/') + - docker run + --env DEP_SCAN_DISABLE_REMOTE_CHECKS="${DEP_SCAN_DISABLE_REMOTE_CHECKS:-false}" + --volume "$PWD:/code" + --volume /var/run/docker.sock:/var/run/docker.sock + "registry.gitlab.com/gitlab-org/security-products/dependency-scanning:$SP_VERSION" /code + artifacts: + reports: + dependency_scanning: gl-dependency-scanning-report.json +``` + +The above example will create a `dependency_scanning` job in your CI/CD pipeline +and scan your dependencies for possible vulnerabilities. The report will be saved as a +[Dependency Scanning report artifact](../../ci/yaml/README.md#artifactsreportsdependency_scanning-ultimate) +that you can later download and analyze. +Due to implementation limitations we always take the latest Dependency Scanning artifact available. + +The results are sorted by the priority of the vulnerability: + +1. High +1. Medium +1. Low +1. Unknown +1. Everything else + +Behind the scenes, the [GitLab Dependency Scanning Docker image](https://gitlab.com/gitlab-org/security-products/dependency-scanning) +is used to detect the languages/package managers and in turn runs the matching scan tools. + +Some security scanners require to send a list of project dependencies to GitLab +central servers to check for vulnerabilities. To learn more about this or to +disable it, check the [GitLab Dependency Scanning documentation](https://gitlab.com/gitlab-org/security-products/dependency-scanning#remote-checks). + +TIP: **Tip:** +For [GitLab Ultimate][ee] users, this information will +be automatically extracted and shown right in the merge request widget. +[Learn more on Dependency Scanning in merge requests](../../user/project/merge_requests/dependency_scanning.md). + +## Supported languages and package managers + +See [the full list of supported languages and package managers](../../user/project/merge_requests/dependency_scanning.md#supported-languages-and-dependency-managers). + +## Previous job definitions + +CAUTION: **Caution:** +Before GitLab 11.5, Dependency Scanning job and artifact had to be named specifically +to automatically extract report data and show it in the merge request widget. +While these old job definitions are still maintained they have been deprecated +and may be removed in next major release, GitLab 12.0. +You are advised to update your current `.gitlab-ci.yml` configuration to reflect that change. + +For GitLab 11.4 and earlier, the job should look like: + +```yaml +dependency_scanning: + image: docker:stable + variables: + DOCKER_DRIVER: overlay2 + allow_failure: true + services: + - docker:stable-dind + script: + - export SP_VERSION=$(echo "$CI_SERVER_VERSION" | sed 's/^\([0-9]*\)\.\([0-9]*\).*/\1-\2-stable/') + - docker run + --env DEP_SCAN_DISABLE_REMOTE_CHECKS="${DEP_SCAN_DISABLE_REMOTE_CHECKS:-false}" + --volume "$PWD:/code" + --volume /var/run/docker.sock:/var/run/docker.sock + "registry.gitlab.com/gitlab-org/security-products/dependency-scanning:$SP_VERSION" /code + artifacts: + paths: [gl-dependency-scanning-report.json] +``` + +[ee]: https://about.gitlab.com/pricing/ diff --git a/doc/ci/examples/laravel_with_gitlab_and_envoy/img/variables_page.png b/doc/ci/examples/laravel_with_gitlab_and_envoy/img/variables_page.png Binary files differindex 4675e20ef79..edeaa011ada 100644 --- a/doc/ci/examples/laravel_with_gitlab_and_envoy/img/variables_page.png +++ b/doc/ci/examples/laravel_with_gitlab_and_envoy/img/variables_page.png diff --git a/doc/ci/examples/license_management.md b/doc/ci/examples/license_management.md new file mode 100644 index 00000000000..c45f1c0404a --- /dev/null +++ b/doc/ci/examples/license_management.md @@ -0,0 +1,100 @@ +# Dependencies license management with GitLab CI/CD **[ULTIMATE]** + +CAUTION: **Caution:** +The job definition shown below is supported on GitLab 11.5 and later versions. +It also requires the GitLab Runner 11.5 or later. +For earlier versions, use the [previous job definitions](#previous-job-definitions). + +This example shows how to run the License Management tool on your +project's dependencies by using GitLab CI/CD. + +First, you need GitLab Runner with +[docker-in-docker executor](../docker/using_docker_build.md#use-docker-in-docker-executor). + +Once you set up the Runner, add a new job to `.gitlab-ci.yml` that +generates the expected report: + +```yaml +license_management: + image: + name: "registry.gitlab.com/gitlab-org/security-products/license-management:$CI_SERVER_VERSION_MAJOR-$CI_SERVER_VERSION_MINOR-stable" + entrypoint: [""] + stage: test + allow_failure: true + script: + - /run.sh analyze . + artifacts: + reports: + license_management: gl-license-management-report.json +``` + +The above example will create a `license_management` job in your CI/CD pipeline +and scan your dependencies to find their licenses. The report will be saved as a +[License Management report artifact](../../ci/yaml/README.md#artifactsreportslicense_management-ultimate) +that you can later download and analyze. +Due to implementation limitations we always take the latest License Management artifact available. + +## Install custom project dependencies + +> Introduced in GitLab Ultimate 11.4. + +The `license_management` image already embeds many auto-detection scripts, languages, +and packages. Nevertheless, it's almost impossible to cover all cases, for all projects. +That's why sometimes it's necessary to install extra packages, or to have extra steps +in the project automated setup, like the download and installation of a certificate. +For that, a `SETUP_CMD` environment variable can be passed to the container, +with the required commands to run before license detection. + +If present, this variable will override the setup step necessary to install all the packages +of your application (ex: for a project with a `Gemfile`, the setup step will be `bundle install`). + +Example: + +```yaml +license_management: + image: + name: "registry.gitlab.com/gitlab-org/security-products/license-management:$CI_SERVER_VERSION_MAJOR-$CI_SERVER_VERSION_MINOR-stable" + entrypoint: [""] + stage: test + variables: + SETUP_CMD: ./my-custom-install-script.sh + allow_failure: true + script: + - /run.sh analyze . + artifacts: + reports: + license_management: gl-license-management-report.json +``` + +In this example, `my-custom-install-script.sh` is a shell script at the root of the project. + +TIP: **Tip:** +For [GitLab Ultimate][ee] users, this information will +be automatically extracted and shown right in the merge request widget. +[Learn more on License Management in merge requests](../../user/project/merge_requests/license_management.md). + +## Previous job definitions + +CAUTION: **Caution:** +Before GitLab 11.5, License Management job and artifact had to be named specifically +to automatically extract report data and show it in the merge request widget. +While these old job definitions are still maintained they have been deprecated +and may be removed in next major release, GitLab 12.0. +You are advised to update your current `.gitlab-ci.yml` configuration to reflect that change. + +For GitLab 11.4 and earlier, the job should look like: + +```yaml +license_management: + image: + name: "registry.gitlab.com/gitlab-org/security-products/license-management:$CI_SERVER_VERSION_MAJOR-$CI_SERVER_VERSION_MINOR-stable" + entrypoint: [""] + stage: test + allow_failure: true + script: + - /run.sh analyze . + artifacts: + paths: [gl-license-management-report.json] +``` + +[ee]: https://about.gitlab.com/pricing/ diff --git a/doc/ci/examples/sast.md b/doc/ci/examples/sast.md new file mode 100644 index 00000000000..2c9db74b9b9 --- /dev/null +++ b/doc/ci/examples/sast.md @@ -0,0 +1,95 @@ +# Static Application Security Testing with GitLab CI/CD **[ULTIMATE]** + +CAUTION: **Caution:** +The job definition shown below is supported on GitLab 11.5 and later versions. +It also requires the GitLab Runner 11.5 or later. +For earlier versions, use the [previous job definitions](#previous-job-definitions). + +This example shows how to run +[Static Application Security Testing (SAST)](https://en.wikipedia.org/wiki/Static_program_analysis) +on your project's source code by using GitLab CI/CD. + +First, you need GitLab Runner with +[docker-in-docker executor](../docker/using_docker_build.md#use-docker-in-docker-executor). + +Once you set up the Runner, add a new job to `.gitlab-ci.yml` that +generates the expected report: + +```yaml +sast: + image: docker:stable + variables: + DOCKER_DRIVER: overlay2 + allow_failure: true + services: + - docker:stable-dind + script: + - export SP_VERSION=$(echo "$CI_SERVER_VERSION" | sed 's/^\([0-9]*\)\.\([0-9]*\).*/\1-\2-stable/') + - docker run + --env SAST_CONFIDENCE_LEVEL="${SAST_CONFIDENCE_LEVEL:-3}" + --volume "$PWD:/code" + --volume /var/run/docker.sock:/var/run/docker.sock + "registry.gitlab.com/gitlab-org/security-products/sast:$SP_VERSION" /app/bin/run /code + artifacts: + reports: + sast: gl-sast-report.json +``` + +The above example will create a `sast` job in your CI/CD pipeline +and scan your dependencies for possible vulnerabilities. The report will be saved as a +[SAST report artifact](../../ci/yaml/README.md#artifactsreportssast-ultimate) +that you can later download and analyze. +Due to implementation limitations we always take the latest SAST artifact available. + +The results are sorted by the priority of the vulnerability: + +1. Critical +1. High +1. Medium +1. Low +1. Unknown +1. Everything else + +Behind the scenes, the [GitLab SAST Docker image](https://gitlab.com/gitlab-org/security-products/sast) +is used to detect the languages/frameworks and in turn runs the matching scan tools. + +TIP: **Tip:** +For [GitLab Ultimate][ee] users, this information will +be automatically extracted and shown right in the merge request widget. +[Learn more on SAST in merge requests](../../user/project/merge_requests/sast.md). + +## Supported languages and frameworks + +See [the full list of supported languages and frameworks](../../user/project/merge_requests/sast.md#supported-languages-and-frameworks). + +## Previous job definitions + +CAUTION: **Caution:** +Before GitLab 11.5, SAST job and artifact had to be named specifically +to automatically extract report data and show it in the merge request widget. +While these old job definitions are still maintained they have been deprecated +and may be removed in next major release, GitLab 12.0. +You are advised to update your current `.gitlab-ci.yml` configuration to reflect that change. + +For GitLab 11.4 and earlier, the job should look like: + +```yaml +sast: + image: docker:stable + variables: + DOCKER_DRIVER: overlay2 + allow_failure: true + services: + - docker:stable-dind + script: + - export SP_VERSION=$(echo "$CI_SERVER_VERSION" | sed 's/^\([0-9]*\)\.\([0-9]*\).*/\1-\2-stable/') + - docker run + --env SAST_CONFIDENCE_LEVEL="${SAST_CONFIDENCE_LEVEL:-3}" + --volume "$PWD:/code" + --volume /var/run/docker.sock:/var/run/docker.sock + "registry.gitlab.com/gitlab-org/security-products/sast:$SP_VERSION" /app/bin/run /code + artifacts: + paths: [gl-sast-report.json] +``` + +[ee]: https://about.gitlab.com/pricing/ diff --git a/doc/ci/img/multi_pipeline_mini_graph.gif b/doc/ci/img/multi_pipeline_mini_graph.gif Binary files differnew file mode 100644 index 00000000000..de49ba5aa12 --- /dev/null +++ b/doc/ci/img/multi_pipeline_mini_graph.gif diff --git a/doc/ci/img/multi_project_pipeline_graph.png b/doc/ci/img/multi_project_pipeline_graph.png Binary files differnew file mode 100644 index 00000000000..723a455cb4a --- /dev/null +++ b/doc/ci/img/multi_project_pipeline_graph.png diff --git a/doc/ci/img/pipelines-goal.png b/doc/ci/img/pipelines-goal.png Binary files differindex f15716d0b8f..17bd77e951f 100644 --- a/doc/ci/img/pipelines-goal.png +++ b/doc/ci/img/pipelines-goal.png diff --git a/doc/ci/img/types-of-pipelines.png b/doc/ci/img/types-of-pipelines.png Binary files differindex 829a53d5d52..4a64f4f4cf7 100644 --- a/doc/ci/img/types-of-pipelines.png +++ b/doc/ci/img/types-of-pipelines.png diff --git a/doc/ci/interactive_web_terminal/index.md b/doc/ci/interactive_web_terminal/index.md index 2a4160f62b0..7109b2ec583 100644 --- a/doc/ci/interactive_web_terminal/index.md +++ b/doc/ci/interactive_web_terminal/index.md @@ -54,3 +54,8 @@ terminal will block the job from finishing for the duration configured in close the terminal window.  + +## Interactive Web Terminals for the Web IDE **[ULTIMATE ONLY]** + +Read the Web IDE docs to learn how to run [Interactive Terminals through the Web IDE](../../user/project/web_ide/index.md). + diff --git a/doc/ci/merge_request_pipelines/img/pipeline_detail.png b/doc/ci/merge_request_pipelines/img/pipeline_detail.png Binary files differindex 6094a0975fb..90e7c449a66 100644 --- a/doc/ci/merge_request_pipelines/img/pipeline_detail.png +++ b/doc/ci/merge_request_pipelines/img/pipeline_detail.png diff --git a/doc/ci/multi_project_pipeline_graphs.md b/doc/ci/multi_project_pipeline_graphs.md new file mode 100644 index 00000000000..af10e5b6126 --- /dev/null +++ b/doc/ci/multi_project_pipeline_graphs.md @@ -0,0 +1,5 @@ +--- +redirect_to: 'multi_project_pipelines.md' +--- + +This document was moved to [another location](multi_project_pipelines.md). diff --git a/doc/ci/multi_project_pipelines.md b/doc/ci/multi_project_pipelines.md new file mode 100644 index 00000000000..1a0d3367dd8 --- /dev/null +++ b/doc/ci/multi_project_pipelines.md @@ -0,0 +1,152 @@ +# Multi-project pipelines **[PREMIUM]** + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/issues/2121) in +[GitLab Premium 9.3](https://about.gitlab.com/2017/06/22/gitlab-9-3-released/#multi-project-pipeline-graphs). + +When you set up [GitLab CI/CD](README.md) across multiple projects, you can visualize +the entire pipeline, including all cross-project inter-dependencies. + +## Overview + +GitLab CI/CD is a powerful continuous integration tool that works not only per project, but also across projects. When you +configure GitLab CI for your project, you can visualize the stages +of your [jobs](pipelines.md#jobs) on a [pipeline graph](pipelines.md#pipeline-graphs). + + + +In the Merge Request Widget, multi-project pipeline mini-graphs are displayed, +and when hovering or tapping (on touchscreen devices) they will expand and be shown adjacent to each other. + + + +Multi-project pipelines are useful for larger products that require cross-project inter-dependencies, such as those +adopting a [microservices architecture](https://about.gitlab.com/2016/08/16/trends-in-version-control-land-microservices/). + +## Use cases + +Let's assume you deploy your web app from different projects in GitLab: + +- One for the free version, which has its own pipeline that builds and tests your app +- One for the paid version add-ons, which also pass through builds and tests +- One for the documentation, which also builds, tests, and deploys with an SSG + +With Multi-Project Pipelines, you can visualize the entire pipeline, including all stages of builds and tests for the three projects. + +## Triggering multi-project pipelines through API + +When you use the [`CI_JOB_TOKEN` to trigger pipelines](triggers/README.md#ci-job-token), GitLab +recognizes the source of the job token, and thus internally ties these pipelines +together, allowing you to visualize their relationships on pipeline graphs. + +These relationships are displayed in the pipeline graph by showing inbound and +outbound connections for upstream and downstream pipeline dependencies. + +## Creating multi-project pipelines from `.gitlab-ci.yml` + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/issues/8997) in [GitLab Premium](https://about.gitlab.com/pricing/) 11.8. + +### Triggering a downstream pipeline using a bridge job + +Before GitLab 11.8, it was necessary to implement a pipeline job that was +responsible for making the API request [to trigger a pipeline](triggers/README.md#creating-cross-project-pipeline-through-API) +in a different project. + +In GitLab 11.8, GitLab provides a new CI/CD configuration syntax to make this +task easier, and avoid needing GitLab Runner for triggering cross-project +pipelines. The following illustrates configuring a bridge job: + +```yaml +rspec: + stage: test + script: bundle exec rspec + +staging: + variables: + ENVIRONMENT: staging + stage: deploy + trigger: my/deployment +``` + +In the example above, as soon as `rspec` job succeeds in the `test` stage, +the `staging` bridge job is going to be started. The initial status of this +job will be `pending`. GitLab will create a downstream pipeline in the +`my/deployment` project and, as soon as the pipeline gets created, the +`staging` job will succeed. `my/deployment` is a full path to that project. + +The user that created the upstream pipeline needs to have access rights to the +downstream project (`my/deployment` in this case). If a downstream project can +not be found, or a user does not have access rights to create pipeline there, +the `staging` job is going to be marked as _failed_. + +Caution: **Caution:** +`staging` will succeed as soon as a downstream pipeline gets created. +GitLab does not support status attribution yet, however adding first-class +`trigger` configuration syntax is ground work for implementing +[status attribution](https://gitlab.com/gitlab-org/gitlab-ce/issues/39640). + +NOTE: **Note:** +Bridge jobs do not support every configuration entry that a user can use +in the case of regular jobs. Bridge jobs will not to be picked by a Runner, +thus there is no point in adding support for `script`, for example. If a user +tries to use unsupported configuration syntax, YAML validation will fail upon +pipeline creation. + +### Specifying a downstream pipeline branch + +It is possible to specify a branch name that a downstream pipeline will use: + +```yaml +rspec: + stage: test + script: bundle exec rspec + +staging: + stage: deploy + trigger: + project: my/deployment + branch: stable-11-2 +``` + +Use a `project` keyword to specify full path to a downstream project. Use +a `branch` keyword to specify a branch name. + +GitLab will use a commit that is currently on the HEAD of the branch when +creating a downstream pipeline. + +### Passing variables to a downstream pipeline + +Sometimes you might want to pass variables to a downstream pipeline. +You can do that using the `variables` keyword, just like you would when +defining a regular job. + +```yaml +rspec: + stage: test + script: bundle exec rspec + +staging: + variables: + ENVIRONMENT: staging + stage: deploy + trigger: my/deployment +``` + +The `ENVIRONMENT` variable will be passed to every job defined in a downstream +pipeline. It will be available as an environment variable when GitLab Runner picks a job. + +### Limitations + +Because bridge jobs are a little different to regular jobs, it is not +possible to use exactly the same configuration syntax here, as one would +normally do when defining a regular job that will be picked by a runner. + +Some features are not implemented yet. For example, support for environments. + +[Configuration keywords](yaml/README.md) available for bridge jobs are: + +- `trigger` (to define a downstream pipeline trigger) +- `stage` +- `allow_failure` +- `only` and `except` +- `when` +- `extends` diff --git a/doc/ci/pipelines.md b/doc/ci/pipelines.md index 4f3106c6dc6..b1b9bd649a5 100644 --- a/doc/ci/pipelines.md +++ b/doc/ci/pipelines.md @@ -3,7 +3,7 @@ > Introduced in GitLab 8.8. NOTE: **Note:** -If you have a [mirrored repository where GitLab pulls from](https://docs.gitlab.com/ee/workflow/repository_mirroring.html#pulling-from-a-remote-repository-starter), +If you have a [mirrored repository where GitLab pulls from](../workflow/repository_mirroring.md#pulling-from-a-remote-repository-starter), you may need to enable pipeline triggering in your project's **Settings > Repository > Pull from a remote repository > Trigger pipelines for mirror updates**. @@ -231,6 +231,11 @@ by name. The order of severity is:  +### Multi-project pipelines graphs **[PREMIUM]** + +With [multi-project pipelines](multi_project_pipelines.md), +you can visualize cross-project pipelines. + ## How the pipeline duration is calculated Total running time for a given pipeline would exclude retries and pending @@ -297,9 +302,9 @@ runners will not use regular runners, they must be tagged accordingly. [manual]: yaml/README.md#whenmanual [env-manual]: environments.md#manually-deploying-to-environments [stages]: yaml/README.md#stages -[runners]: runners/README.html +[runners]: runners/README.md [pipelines settings]: ../user/project/pipelines/settings.md -[triggers]: triggers/README.md +[triggers]: triggers/README.md#ci-job-token [ce-5742]: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/5742 [ce-6242]: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/6242 [ce-7931]: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/7931 @@ -307,3 +312,5 @@ runners will not use regular runners, they must be tagged accordingly. [ce-17782]: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/17782 [ce-17814]: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/17814 [regexp]: https://gitlab.com/gitlab-org/gitlab-ce/blob/2f3dc314f42dbd79813e6251792853bc231e69dd/app/models/commit_status.rb#L99 +[eep]: https://about.gitlab.com/pricing/ "GitLab Premium" +[ee-2121]: https://gitlab.com/gitlab-org/gitlab-ee/issues/2121 diff --git a/doc/ci/quick_start/README.md b/doc/ci/quick_start/README.md index 9684cb6ed98..a1eb7043e1b 100644 --- a/doc/ci/quick_start/README.md +++ b/doc/ci/quick_start/README.md @@ -127,7 +127,7 @@ Now if you go to the **Pipelines** page you will see that the pipeline is pending. NOTE: **Note:** -If you have a [mirrored repository where GitLab pulls from](https://docs.gitlab.com/ee/workflow/repository_mirroring.html#pulling-from-a-remote-repository-starter), +If you have a [mirrored repository where GitLab pulls from](../../workflow/repository_mirroring.md#pulling-from-a-remote-repository-starter), you may need to enable pipeline triggering in your project's **Settings > Repository > Pull from a remote repository > Trigger pipelines for mirror updates**. diff --git a/doc/ci/triggers/README.md b/doc/ci/triggers/README.md index 398b017277f..6274eeee300 100644 --- a/doc/ci/triggers/README.md +++ b/doc/ci/triggers/README.md @@ -23,6 +23,63 @@ attackers can impersonate the user that exposed their trigger token publicly in their `.gitlab-ci.yml` file. Use [variables](../variables/README.md#variables) to protect trigger tokens. +### CI job token + +You can use the `CI_JOB_TOKEN` [variable][predef] (used to authenticate +with the [GitLab Container Registry][registry]) in the following cases. + +#### When used with multi-project pipelines **[PREMIUM]** + +> **Note**: +The use of `CI_JOB_TOKEN` for multi-project pipelines was [introduced][ee-2017] +in [GitLab Premium][ee] 9.3. + +This way of triggering can only be used when invoked inside `.gitlab-ci.yml`, +and it creates a dependent pipeline relation visible on the +[pipeline graph](../pipelines.md#multi-project-pipelines-graphs-premium). For example: + +```yaml +build_docs: + stage: deploy + script: + - curl --request POST --form "token=$CI_JOB_TOKEN" --form ref=master https://gitlab.example.com/api/v4/projects/9/trigger/pipeline + only: + - tags +``` + +Pipelines triggered that way also expose a special variable: +`CI_PIPELINE_SOURCE=pipeline`. + +Read more about the [pipelines trigger API][trigapi]. + +#### When a pipeline depends on the artifacts of another pipeline **[PREMIUM]** + +> The use of `CI_JOB_TOKEN` in the artifacts download API was [introduced][ee-2346] + in [GitLab Premium][ee] 9.5. + +With the introduction of dependencies between different projects, one of +them may need to access artifacts created by a previous one. This process +must be granted for authorized accesses, and it can be done using the +`CI_JOB_TOKEN` variable that identifies a specific job. For example: + +```yaml +build_submodule: + image: debian + stage: test + script: + - apt update && apt install -y unzip + - curl --location --output artifacts.zip "https://gitlab.example.com/api/v4/projects/1/jobs/artifacts/master/download?job=test&job_token=$CI_JOB_TOKEN" + - unzip artifacts.zip + only: + - tags +``` + +This allows you to use that for multi-project pipelines and download artifacts +from any project to which you have access as this follows the same principles +with the [permission model][permissions]. + +Read more about the [jobs API](../../api/jobs.md#download-the-artifacts-archive). + ## Adding a new trigger You can add a new trigger by going to your project's @@ -225,7 +282,10 @@ removed with one of the future versions of GitLab. You are advised to [take ownership](#taking-ownership-of-a-trigger) of any legacy triggers. [ee-2017]: https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/2017 +[ee-2346]: https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/2346 [ee]: https://about.gitlab.com/pricing/ [variables]: ../variables/README.md [predef]: ../variables/README.md#predefined-environment-variables [registry]: ../../user/project/container_registry.md +[permissions]: ../../user/permissions.md#job-permissions +[trigapi]: ../../api/pipeline_triggers.md diff --git a/doc/ci/triggers/img/trigger_variables.png b/doc/ci/triggers/img/trigger_variables.png Binary files differindex d273b1fe3a2..910622d74cc 100644 --- a/doc/ci/triggers/img/trigger_variables.png +++ b/doc/ci/triggers/img/trigger_variables.png diff --git a/doc/ci/variables/README.md b/doc/ci/variables/README.md index 55ecc0dccd1..46523e13538 100644 --- a/doc/ci/variables/README.md +++ b/doc/ci/variables/README.md @@ -77,7 +77,7 @@ future GitLab releases.** | **CI_JOB_MANUAL** | 8.12 | all | The flag to indicate that job was manually started | | **CI_JOB_NAME** | 9.0 | 0.5 | The name of the job as defined in `.gitlab-ci.yml` | | **CI_JOB_STAGE** | 9.0 | 0.5 | The name of the stage as defined in `.gitlab-ci.yml` | -| **CI_JOB_TOKEN** | 9.0 | 1.2 | Token used for authenticating with the [GitLab Container Registry][registry] and downloading [dependent repositories][dependent-repositories] | +| **CI_JOB_TOKEN** | 9.0 | 1.2 | Token used for authenticating with [GitLab Container Registry][registry], downloading [dependent repositories][dependent-repositories], authenticate with multi-project pipelines when [triggers][trigger-job-token] are involved, and for [downloading job artifacts][get-job-artifacts] | | **CI_JOB_URL** | 11.1 | 0.5 | Job details URL | | **CI_MERGE_REQUEST_ID** | 11.6 | all | The ID of the merge request if [the pipelines are for merge requests](../merge_request_pipelines/index.md) | | **CI_MERGE_REQUEST_IID** | 11.6 | all | The IID of the merge request if [the pipelines are for merge requests](../merge_request_pipelines/index.md) | @@ -139,6 +139,7 @@ future GitLab releases.** | **GITLAB_USER_ID** | 8.12 | all | The id of the user who started the job | | **GITLAB_USER_LOGIN** | 10.0 | all | The login username of the user who started the job | | **GITLAB_USER_NAME** | 10.0 | all | The real name of the user who started the job | +| **GITLAB_FEATURES** | 10.6 | all | The comma separated list of licensed features available for your instance and plan | | **RESTORE_CACHE_ATTEMPTS** | 8.15 | 1.9 | Number of attempts to restore the cache running a job | ## GitLab 9.0 renaming @@ -253,6 +254,25 @@ Protected variables can be added by going to your project's Once you set them, they will be available for all subsequent pipelines. +### Limiting environment scopes of variables **[PREMIUM]** + +> [Introduced][ee-2112] in [GitLab Premium](https://about.gitlab.com/pricing/) 9.4. + +You can limit the environment scope of a variable by +[defining which environments][envs] it can be available for. + +Wildcards can be used, and the default environment scope is `*` which means +any jobs will have this variable, not matter if an environment is defined or +not. + +For example, if the environment scope is `production`, then only the jobs +having the environment `production` defined would have this specific variable. +Wildcards (`*`) can be used along with the environment name, therefore if the +environment scope is `review/*` then any jobs with environment names starting +with `review/` would have that particular variable. + +To learn more about about scoping environments, see [Scoping environments with specs](../environments.md#scoping-environments-with-specs-premium). + ### Manually-specified variables > [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/issues/44059) in GitLab 10.8. @@ -632,14 +652,16 @@ Below you can find supported syntax reference: Pattern matching is case-sensitive by default. Use `i` flag modifier, like `/pattern/i` to make a pattern case-insensitive. +[ee-2112]: https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/2112 [ce-13784]: https://gitlab.com/gitlab-org/gitlab-ce/issues/13784 "Simple protection of CI variables" -[eep]: https://about.gitlab.com/pricing/ "Available only in GitLab Premium" [envs]: ../environments.md [protected branches]: ../../user/project/protected_branches.md [protected tags]: ../../user/project/protected_tags.md [shellexecutors]: https://docs.gitlab.com/runner/executors/ [triggered]: ../triggers/README.md [builds-policies]: ../yaml/README.md#only-and-except-complex +[trigger-job-token]: ../triggers/README.md#ci-job-token [gitlab-deploy-token]: ../../user/project/deploy_tokens/index.md#gitlab-deploy-token [registry]: ../../user/project/container_registry.md [dependent-repositories]: ../../user/project/new_ci_build_permissions_model.md#dependent-repositories +[get-job-artifacts]: ../../api/jobs.html#get-job-artifacts diff --git a/doc/ci/variables/where_variables_can_be_used.md b/doc/ci/variables/where_variables_can_be_used.md index ceca4af1bee..0fc9e7f67d5 100644 --- a/doc/ci/variables/where_variables_can_be_used.md +++ b/doc/ci/variables/where_variables_can_be_used.md @@ -112,3 +112,22 @@ They are: - Not supported: - For definitions where the ["Expansion place"](#gitlab-ciyml-file) is GitLab. - In the `only` and `except` [variables expressions](README.md#variables-expressions). + +## Variables with an environment scope + +Variables defined with an environment scope are supported. Given that +there is a variable `$STAGING_SECRET` defined in a scope of +`review/staging/*`, the following job that is using dynamic environments +is going to be created, based on the matching variable expression: + +```yaml +my-job: + stage: staging + environment: + name: review/$CI_JOB_STAGE/deploy + script: + - 'deploy staging' + only: + variables: + - $STAGING_SECRET == 'something' +``` diff --git a/doc/ci/yaml/README.md b/doc/ci/yaml/README.md index a44f4b62a0e..04a57011822 100644 --- a/doc/ci/yaml/README.md +++ b/doc/ci/yaml/README.md @@ -19,7 +19,7 @@ We have complete examples of configuring pipelines: - To see a large `.gitlab-ci.yml` file used in an enterprise, see the [`.gitlab-ci.yml` file for `gitlab-ce`](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/.gitlab-ci.yml). NOTE: **Note:** -If you have a [mirrored repository where GitLab pulls from](https://docs.gitlab.com/ee/workflow/repository_mirroring.html#pulling-from-a-remote-repository-starter), +If you have a [mirrored repository where GitLab pulls from](../../workflow/repository_mirroring.md#pulling-from-a-remote-repository-starter), you may need to enable pipeline triggering in your project's **Settings > Repository > Pull from a remote repository > Trigger pipelines for mirror updates**. @@ -56,7 +56,7 @@ independently from each other. Each instance of GitLab CI has an embedded debug tool called Lint, which validates the content of your `.gitlab-ci.yml` files. You can find the Lint under the page `ci/lint` of your -project namespace. For example, `http://gitlab.example.com/gitlab-org/project-123/-/ci/lint`. +project namespace. For example, `https://gitlab.example.com/gitlab-org/project-123/-/ci/lint`. ### Unavailable names for jobs @@ -1632,7 +1632,7 @@ test: from `trigger` definition is started by GitLab, a downstream pipeline gets created. -Learn more about [multi-project pipelines](https://docs.gitlab.com/ee/ci/multi_project_pipelines.html#creating-cross-project-pipelines-from-gitlab-ci-yml). +Learn more about [multi-project pipelines](../multi_project_pipelines.md#creating-cross-project-pipelines-from-gitlab-ci-yml). #### Simple `trigger` syntax diff --git a/doc/customization/branded_login_page/admin_area.png b/doc/customization/branded_login_page/admin_area.png Binary files differnew file mode 100644 index 00000000000..16dafca4d19 --- /dev/null +++ b/doc/customization/branded_login_page/admin_area.png diff --git a/doc/customization/branded_login_page/company_login_page.png b/doc/customization/branded_login_page/company_login_page.png Binary files differnew file mode 100644 index 00000000000..a98cba33ed1 --- /dev/null +++ b/doc/customization/branded_login_page/company_login_page.png diff --git a/doc/customization/issue_and_merge_request_template.md b/doc/customization/issue_and_merge_request_template.md new file mode 100644 index 00000000000..023702a91c5 --- /dev/null +++ b/doc/customization/issue_and_merge_request_template.md @@ -0,0 +1,5 @@ +--- +redirect_to: '../user/project/description_templates.md#setting-a-default-template-for-issues-and-merge-requests' +--- + +This document was moved to [description_templates](../user/project/description_templates.md#setting-a-default-template-for-issues-and-merge-requests). diff --git a/doc/customization/system_header_and_footer_messages/appearance.png b/doc/customization/system_header_and_footer_messages/appearance.png Binary files differindex fd315bb6c07..bccbff16664 100644 --- a/doc/customization/system_header_and_footer_messages/appearance.png +++ b/doc/customization/system_header_and_footer_messages/appearance.png diff --git a/doc/customization/system_header_and_footer_messages/custom_header_footer.png b/doc/customization/system_header_and_footer_messages/custom_header_footer.png Binary files differindex 691ca8a3484..15d19c87dd7 100644 --- a/doc/customization/system_header_and_footer_messages/custom_header_footer.png +++ b/doc/customization/system_header_and_footer_messages/custom_header_footer.png diff --git a/doc/customization/system_header_and_footer_messages/sign_up_custom_header_and_footer.png b/doc/customization/system_header_and_footer_messages/sign_up_custom_header_and_footer.png Binary files differindex a9c59bc9f62..51349ddf6f4 100644 --- a/doc/customization/system_header_and_footer_messages/sign_up_custom_header_and_footer.png +++ b/doc/customization/system_header_and_footer_messages/sign_up_custom_header_and_footer.png diff --git a/doc/development/README.md b/doc/development/README.md index 13646cbfe48..281ee02c975 100644 --- a/doc/development/README.md +++ b/doc/development/README.md @@ -38,6 +38,7 @@ description: 'Learn how to contribute to GitLab.' - [Sidekiq guidelines](sidekiq_style_guide.md) for working with Sidekiq workers - [Working with Gitaly](gitaly.md) - [Manage feature flags](feature_flags.md) +- [Licensed feature availability](licensed_feature_availability.md) - [View sent emails or preview mailers](emails.md) - [Shell commands](shell_commands.md) in the GitLab codebase - [`Gemfile` guidelines](gemfile.md) @@ -48,12 +49,14 @@ description: 'Learn how to contribute to GitLab.' - [How to dump production data to staging](db_dump.md) - [Working with the GitHub importer](github_importer.md) - [Import/Export development documentation](import_export.md) +- [Elasticsearch integration docs](elasticsearch.md) - [Working with Merge Request diffs](diffs.md) - [Kubernetes integration guidelines](kubernetes.md) - [Permissions](permissions.md) - [Prometheus metrics](prometheus_metrics.md) - [Guidelines for reusing abstractions](reusing_abstractions.md) - [DeclarativePolicy framework](policies.md) +- [Geo development](geo.md) ## Performance guides diff --git a/doc/development/contributing/style_guides.md b/doc/development/contributing/style_guides.md index 0eedef5e14f..7f15a5c21fa 100644 --- a/doc/development/contributing/style_guides.md +++ b/doc/development/contributing/style_guides.md @@ -21,7 +21,7 @@ of _prohibited this user from being saved due to the following errors:_ the text should be _sorry, we could not create your account because:_ 1. Code should be written in [US English][us-english] -1. [Go](../go_guide/index.md) +1. [Go][../go_guide/index.md] This is also the style used by linting tools such as [RuboCop](https://github.com/bbatsov/rubocop) and [Hound CI](https://houndci.com). diff --git a/doc/development/elasticsearch.md b/doc/development/elasticsearch.md new file mode 100644 index 00000000000..325cf347ee1 --- /dev/null +++ b/doc/development/elasticsearch.md @@ -0,0 +1,166 @@ +# Elasticsearch knowledge + +This area is to maintain a compendium of useful information when working with elasticsearch. + +Information on how to enable ElasticSearch and perform the initial indexing is kept in https://docs.gitlab.com/ee/integration/elasticsearch.html#enabling-elasticsearch + +## Initial installation on OS X + +It is recommended to use the Docker image. After installing docker you can immediately spin up an instance with + +``` +docker run --name elastic56 -p 9200:9200 -p 9300:9300 -e "discovery.type=single-node" docker.elastic.co/elasticsearch/elasticsearch:5.6.12 +``` + +and use `docker stop elastic56` and `docker start elastic56` to stop/start it. + +### Installing on the host + +We currently only support Elasticsearch [5.6 to 6.x](https://docs.gitlab.com/ee/integration/elasticsearch.html#requirements) + +Version 5.6 is available on homebrew and is the recommended version to use in order to test compatibility. + +``` +brew install elasticsearch@5.6 +``` + +There is no need to install any plugins + +## New repo indexer (beta) + +If you're interested on working with the new beta repo indexer, all you need to do is: + +- git clone git@gitlab.com:gitlab-org/gitlab-elasticsearch-indexer.git +- make +- make install + +this adds `gitlab-elasticsearch-indexer` to `$GOPATH/bin`, please make sure that is in your `$PATH`. After that GitLab will find it and you'll be able to enable it in the admin settings area. + +**note:** `make` will not recompile the executable unless you do `make clean` beforehand + +## Helpful rake tasks + +- `gitlab:elastic:test:index_size`: Tells you how much space the current index is using, as well as how many documents are in the index. +- `gitlab:elastic:test:index_size_change`: Outputs index size, reindexes, and outputs index size again. Useful when testing improvements to indexing size. + +Additionally, if you need large repos or multiple forks for testing, please consider [following these instructions](https://docs.gitlab.com/ee/development/rake_tasks.html#extra-project-seed-options) + +## How does it work? + +The ElasticSearch integration depends on an external indexer. We ship a [ruby indexer](https://gitlab.com/gitlab-org/gitlab-ee/blob/master/bin/elastic_repo_indexer) by default but are also working on an [indexer written in Go](https://gitlab.com/gitlab-org/gitlab-elasticsearch-indexer). The user must trigger the initial indexing via a rake task, but after this is done GitLab itself will trigger reindexing when required via `after_` callbacks on create, update, and destroy that are inherited from [/ee/app/models/concerns/elastic/application_search.rb](https://gitlab.com/gitlab-org/gitlab-ee/blob/master/ee/app/models/concerns/elastic/application_search.rb). + +All indexing after the initial one is done via `ElasticIndexerWorker` (sidekiq jobs). + +Search queries are generated by the concerns found in [ee/app/models/concerns/elastic](https://gitlab.com/gitlab-org/gitlab-ee/tree/master/ee/app/models/concerns/elastic). These concerns are also in charge of access control, and have been a historic source of security bugs so please pay close attention to them! + +## Existing Analyzers/Tokenizers/Filters +These are all defined in https://gitlab.com/gitlab-org/gitlab-ee/blob/master/ee/lib/elasticsearch/git/model.rb + +### Analyzers +#### `path_analyzer` +Used when indexing blobs' paths. Uses the `path_tokenizer` and the `lowercase` and `asciifolding` filters. + +Please see the `path_tokenizer` explanation below for an example. + +#### `sha_analyzer` +Used in blobs and commits. Uses the `sha_tokenizer` and the `lowercase` and `asciifolding` filters. + +Please see the `sha_tokenizer` explanation later below for an example. + +#### `code_analyzer` +Used when indexing a blob's filename and content. Uses the `whitespace` tokenizer and the filters: `code`, `edgeNGram_filter`, `lowercase`, and `asciifolding` + +The `whitespace` tokenizer was selected in order to have more control over how tokens are split. For example the string `Foo::bar(4)` needs to generate tokens like `Foo` and `bar(4)` in order to be properly searched. + +Please see the `code` filter for an explanation on how tokens are split. + +#### `code_search_analyzer` +Not directly used for indexing, but rather used to transform a search input. Uses the `whitespace` tokenizer and the `lowercase` and `asciifolding` filters. + +### Tokenizers +#### `sha_tokenizer` +This is a custom tokenizer that uses the [`edgeNGram` tokenizer](https://www.elastic.co/guide/en/elasticsearch/reference/5.5/analysis-edgengram-tokenizer.html) to allow SHAs to be searcheable by any sub-set of it (minimum of 5 chars). + +example: + +`240c29dc7e` becomes: +- `240c2` +- `240c29` +- `240c29d` +- `240c29dc` +- `240c29dc7` +- `240c29dc7e` + +#### `path_tokenizer` +This is a custom tokenizer that uses the [`path_hierarchy` tokenizer](https://www.elastic.co/guide/en/elasticsearch/reference/5.5/analysis-pathhierarchy-tokenizer.html) with `reverse: true` in order to allow searches to find paths no matter how much or how little of the path is given as input. + +example: + +`'/some/path/application.js'` becomes: +- `'/some/path/application.js'` +- `'some/path/application.js'` +- `'path/application.js'` +- `'application.js'` + +### Filters +#### `code` +Uses a [Pattern Capture token filter](https://www.elastic.co/guide/en/elasticsearch/reference/5.5/analysis-pattern-capture-tokenfilter.html) to split tokens into more easily searched versions of themselves. + +Patterns: +- `"(\\p{Ll}+|\\p{Lu}\\p{Ll}+|\\p{Lu}+)"`: captures CamelCased and lowedCameCased strings as separate tokens +- `"(\\d+)"`: extracts digits +- `"(?=([\\p{Lu}]+[\\p{L}]+))"`: captures CamelCased strings recursively. Ex: `ThisIsATest` => `[ThisIsATest, IsATest, ATest, Test]` +- `'"((?:\\"|[^"]|\\")*)"'`: captures terms inside quotes, removing the quotes +- `"'((?:\\'|[^']|\\')*)'"`: same as above, for single-quotes +- `'\.([^.]+)(?=\.|\s|\Z)'`: separate terms with periods in-between +- `'\/?([^\/]+)(?=\/|\b)'`: separate path terms `like/this/one` + +#### `edgeNGram_filter` +Uses an [Edge NGram token filter](https://www.elastic.co/guide/en/elasticsearch/reference/5.5/analysis-edgengram-tokenfilter.html) to allow inputs with only parts of a token to find the token. For example it would turn `glasses` into permutations starting with `gl` and ending with `glasses`, which would allow a search for "`glass`" to find the original token `glasses` + +## Gotchas + +- Searches can have their own analyzers. Remember to check when editing analyzers +- `Character` filters (as opposed to token filters) always replace the original character, so they're not a good choice as they can hinder exact searches + +## Troubleshooting + +### Getting "flood stage disk watermark [95%] exceeded" + +You might get an error such as + +``` +[2018-10-31T15:54:19,762][WARN ][o.e.c.r.a.DiskThresholdMonitor] [pval5Ct] + flood stage disk watermark [95%] exceeded on + [pval5Ct7SieH90t5MykM5w][pval5Ct][/usr/local/var/lib/elasticsearch/nodes/0] free: 56.2gb[3%], + all indices on this node will be marked read-only +``` + +This is because you've exceeded the disk space threshold - it thinks you don't have enough disk space left, based on the default 95% threshold. + +In addition, the `read_only_allow_delete` setting will be set to `true`. It will block indexing, `forcemerge`, etc + +``` +curl "http://localhost:9200/gitlab-development/_settings?pretty" +``` + +Add this to your `elasticsearch.yml` file: + +``` +# turn off the disk allocator +cluster.routing.allocation.disk.threshold_enabled: false +``` + +_or_ + +``` +# set your own limits +cluster.routing.allocation.disk.threshold_enabled: true +cluster.routing.allocation.disk.watermark.flood_stage: 5gb # ES 6.x only +cluster.routing.allocation.disk.watermark.low: 15gb +cluster.routing.allocation.disk.watermark.high: 10gb +``` + +Restart ElasticSearch, and the `read_only_allow_delete` will clear on it's own. + +_from "Disk-based Shard Allocation | Elasticsearch Reference" [5.6](https://www.elastic.co/guide/en/elasticsearch/reference/5.6/disk-allocator.html#disk-allocator) and [6.x](https://www.elastic.co/guide/en/elasticsearch/reference/6.x/disk-allocator.html)_ diff --git a/doc/development/geo.md b/doc/development/geo.md new file mode 100644 index 00000000000..7fd6002d3a3 --- /dev/null +++ b/doc/development/geo.md @@ -0,0 +1,417 @@ +# Geo (development) + +Geo connects GitLab instances together. One GitLab instance is +designated as a **primary** node and can be run with multiple +**secondary** nodes. Geo orchestrates quite a few components that are +described in more detail below. + +## Database replication + +Geo uses [streaming replication](#streaming-replication) to replicate +the database from the **primary** to the **secondary** nodes. This +replication gives the **secondary** nodes access to all the data saved +in the database. So users can log in on the **secondary** and read all +the issues, merge requests, etc. on the **secondary** node. + +## Repository replication + +Geo also replicates repositories. Each **secondary** node keeps track of +the state of every repository in the [tracking database](#tracking-database). + +There are a few ways a repository gets replicated by the: + +- [Repository Sync worker](#repository-sync-worker). +- [Geo Log Cursor](#geo-log-cursor). + +### Project Registry + +The `Geo::ProjectRegistry` class defines the model used to track the +state of repository replication. For each project in the main +database, one record in the tracking database is kept. + +It records the following about repositories: + +- The last time they were synced. +- The last time they were synced successfully. +- If they need to be resynced. +- When retry should be attempted. +- The number of retries. +- If and when the they were verified. + +It also stores these attributes for project wikis in dedicated columns. + +### Repository Sync worker + +The `Geo::RepositorySyncWorker` class runs periodically in the +background and it searches the `Geo::ProjectRegistry` model for +projects that need updating. Those projects can be: + +- Unsynced: Projects that have never been synced on the **secondary** + node and so do not exist yet. +- Updated recently: Projects that have a `last_repository_updated_at` + timestamp that is more recent than the `last_repository_successful_sync_at` + timestamp in the `Geo::ProjectRegistry` model. +- Manual: The admin can manually flag a repository to resync in the + [Geo admin panel](../user/admin_area/geo_nodes.md). + +When we fail to fetch a repository on the secondary `RETRIES_BEFORE_REDOWNLOAD` +times, Geo does a so-called _redownload_. It will do a clean clone +into the `@geo-temporary` directory in the root of the storage. When +it's successful, we replace the main repo with the newly cloned one. + +### Geo Log Cursor + +The [Geo Log Cursor](#geo-log-cursor) is a separate process running on +each **secondary** node. It monitors the [Geo Event Log](#geo-event-log) +and handles all of the events. When it sees an unhandled event, it +starts a background worker to handle that event, depending on the type +of event. + +When a repository receives an update, the Geo **primary** node creates +a Geo event with an associated repository updated event. The cursor +picks that up, and schedules a `Geo::ProjectSyncWorker` job which will +use the `Geo::RepositorySyncService` class and `Geo::WikiSyncService` +class to update the repository and the wiki. + +## Uploads replication + +File uploads are also being replicated to the **secondary** node. To +track the state of syncing, the `Geo::FileRegistry` model is used. + +### File Registry + +Similar to the [Project Registry](#project-registry), there is a +`Geo::FileRegistry` model that tracks the synced uploads. + +CI Job Artifacts are synced in a similar way as uploads or LFS +objects, but they are tracked by `Geo::JobArtifactRegistry` model. + +### File Download Dispatch worker + +Also similar to the [Repository Sync worker](#repository-sync-worker), +there is a `Geo::FileDownloadDispatchWorker` class that is run +periodically to sync all uploads that aren't synced to the Geo +**secondary** node yet. + +Files are copied via HTTP(s) and initiated via the +`/api/v4/geo/transfers/:type/:id` endpoint, +e.g. `/api/v4/geo/transfers/lfs/123`. + +## Authentication + +To authenticate file transfers, each `GeoNode` record has two fields: + +- A public access key (`access_key` field). +- A secret access key (`secret_access_key` field). + +The **secondary** node authenticates itself via a [JWT request](https://jwt.io/). +When the **secondary** node wishes to download a file, it sends an +HTTP request with the `Authorization` header: + +``` +Authorization: GL-Geo <access_key>:<JWT payload> +``` + +The **primary** node uses the `access_key` field to look up the +corresponding Geo **secondary** node and decrypts the JWT payload, +which contains additional information to identify the file +request. This ensures that the **secondary** node downloads the right +file for the right database ID. For example, for an LFS object, the +request must also include the SHA256 sum of the file. An example JWT +payload looks like: + +``` +{ "data": { sha256: "31806bb23580caab78040f8c45d329f5016b0115" }, iat: "1234567890" } +``` + +If the requested file matches the requested SHA256 sum, then the Geo +**primary** node sends data via the [X-Sendfile](https://www.nginx.com/resources/wiki/start/topics/examples/xsendfile/) +feature, which allows NGINX to handle the file transfer without tying +up Rails or Workhorse. + +NOTE: **Note:** +JWT requires synchronized clocks between the machines +involved, otherwise it may fail with an encryption error. + +## Using the Tracking Database + +Along with the main database that is replicated, a Geo **secondary** +node has its own separate [Tracking database](#tracking-database). + +The tracking database contains the state of the **secondary** node. + +Any database migration that needs to be run as part of an upgrade +needs to be applied to the tracking database on each **secondary** node. + +### Configuration + +The database configuration is set in [`config/database_geo.yml`](https://gitlab.com/gitlab-org/gitlab-ee/blob/master/config/database_geo.yml.postgresql). +The directory [`ee/db/geo`](https://gitlab.com/gitlab-org/gitlab-ee/tree/master/ee/db/geo) +contains the schema and migrations for this database. + +To write a migration for the database, use the `GeoMigrationGenerator`: + +``` +rails g geo_migration [args] [options] +``` + +To migrate the tracking database, run: + +``` +bundle exec rake geo:db:migrate +``` + +### Foreign Data Wrapper + +The use of [FDW](#fdw) was introduced in GitLab 10.1. + +This is useful for the [Geo Log Cursor](#geo-log-cursor) and improves +the performance of some synchronization operations. + +While FDW is available in older versions of PostgreSQL, we needed to +raise the minimum required version to 9.6 as this includes many +performance improvements to the FDW implementation. + +#### Refeshing the Foreign Tables + +Whenever the database schema changes on the **primary** node, the +**secondary** node will need to refresh its foreign tables by running +the following: + +```sh +bundle exec rake geo:db:refresh_foreign_tables +``` + +Failure to do this will prevent the **secondary** node from +functioning properly. The **secondary** node will generate error +messages, as the following PostgreSQL error: + +``` +ERROR: relation "gitlab_secondary.ci_job_artifacts" does not exist at character 323 +STATEMENT: SELECT a.attname, format_type(a.atttypid, a.atttypmod), + pg_get_expr(d.adbin, d.adrelid), a.attnotnull, a.atttypid, a.atttypmod + FROM pg_attribute a LEFT JOIN pg_attrdef d + ON a.attrelid = d.adrelid AND a.attnum = d.adnum + WHERE a.attrelid = '"gitlab_secondary"."ci_job_artifacts"'::regclass + AND a.attnum > 0 AND NOT a.attisdropped + ORDER BY a.attnum +``` + +## Finders + +Geo uses [Finders](https://gitlab.com/gitlab-org/gitlab-ee/tree/master/app/finders), +which are classes take care of the heavy lifting of looking up +projects/attachments/etc. in the tracking database and main database. + +### Finders Performance + +The Finders need to compare data from the main database with data in +the tracking database. For example, counting the number of synced +projects normally involves retrieving the project IDs from one +database and checking their state in the other database. This is slow +and requires a lot of memory. + +To overcome this, the Finders use [FDW](#fdw), or Foreign Data +Wrappers. This allows a regular `JOIN` between the main database and +the tracking database. + +## Redis + +Redis on the **secondary** node works the same as on the **primary** +node. It is used for caching, storing sessions, and other persistent +data. + +Redis data replication between **primary** and **secondary** node is +not used, so sessions etc. aren't shared between nodes. + +## Object Storage + +GitLab can optionally use Object Storage to store data it would +otherwise store on disk. These things can be: + + - LFS Objects + - CI Job Artifacts + - Uploads + +Objects that are stored in object storage, are not handled by Geo. Geo +ignores items in object storage. Either: + +- The object storage layer should take care of its own geographical + replication. +- All secondary nodes should use the same storage node. + +## Verification + +### Repository verification + +Repositories are verified with a checksum. + +The **primary** node calculates a checksum on the repository. It +basically hashes all Git refs together and stores that hash in the +`project_repository_states` table of the database. + +The **secondary** node does the same to calculate the hash of its +clone, and compares the hash with the value the **primary** node +calculated. If there is a mismatch, Geo will mark this as a mismatch +and the administrator can see this in the [Geo admin panel](../user/admin_area/geo_nodes.md). + +## Glossary + +### Primary node + +A **primary** node is the single node in a Geo setup that read-write +capabilities. It's the single source of truth and the Geo +**secondary** nodes replicate their data from there. + +In a Geo setup, there can only be one **primary** node. All +**secondary** nodes connect to that **primary**. + +### Secondary node + +A **secondary** node is a read-only replica of the **primary** node +running in a different geographical location. + +### Streaming replication + +Geo depends on the streaming replication feature of PostgreSQL. It +completely replicates the database data and the database schema. The +database replica is a read-only copy. + +Streaming replication depends on the Write Ahead Logs, or WAL. Those +logs are copied over to the replica and replayed there. + +Since streaming replication also replicates the schema, the database +migration do not need to run on the secondary nodes. + +### Tracking database + +A database on each Geo **secondary** node that keeps state for the node +on which it resides. Read more in [Using the Tracking database](#using-the-tracking-database). + +### FDW + +Foreign Data Wrapper, or FDW, is a feature built-in in PostgreSQL. It +allows data to be queried from different data sources. In Geo, it's +used to query data from different PostgreSQL instances. + +## Geo Event Log + +The Geo **primary** stores events in the `geo_event_log` table. Each +entry in the log contains a specific type of event. These type of +events include: + + - Repository Deleted event + - Repository Renamed event + - Repositories Changed event + - Repository Created event + - Hashed Storage Migrated event + - Lfs Object Deleted event + - Hashed Storage Attachments event + - Job Artifact Deleted event + - Upload Deleted event + +### Geo Log Cursor + +The process running on the **secondary** node that looks for new +`Geo::EventLog` rows. + +## Code features + +### `Gitlab::Geo` utilities + +Small utility methods related to Geo go into the +[`ee/lib/gitlab/geo.rb`](https://gitlab.com/gitlab-org/gitlab-ee/blob/master/ee/lib/gitlab/geo.rb) +file. + +Many of these methods are cached using the `RequestStore` class, to +reduce the performance impact of using the methods throughout the +codebase. + +#### Current node + +The class method `.current_node` returns the `GeoNode` record for the +current node. + +We use the `host`, `port`, and `relative_url_root` values from +`gitlab.yml` and search in the database to identify which node we are +in (see `GeoNode.current_node`). + +#### Primary or secondary + +To determine whether the current node is a **primary** node or a +**secondary** node use the `.primary?` and `.secondary?` class +methods. + +It is possible for these methods to both return `false` on a node when +the node is not enabled. See [Enablement](#enablement). + +#### Geo Database configured? + +There is also an additional gotcha when dealing with things that +happen during initialization time. In a few places, we use the +`Gitlab::Geo.geo_database_configured?` method to check if the node has +the tracking database, which only exists on the **secondary** +node. This overcomes race conditions that could happen during +bootstrapping of a new node. + +#### Enablement + +We consider Geo feature enabled when the user has a valid license with the +feature included, and they have at least one node defined at the Geo Nodes +screen. + +See `Gitlab::Geo.enabled?` and `Gitlab::Geo.license_allows?` methods. + +#### Read-only + +All Geo **secondary** nodes are read-only. + +The general principle of a [read-only database](verifying_database_capabilities.md#read-only-database) +applies to all Geo **secondary** nodes. So the +`Gitlab::Database.read_only?` method will always return `true` on a +**secondary** node. + +When some write actions are not allowed because the node is a +**secondary**, consider adding the `Gitlab::Database.read_only?` or +`Gitlab::Database.read_write?` guard, instead of `Gitlab::Geo.secondary?`. + +The database itself will already be read-only in a replicated setup, +so we don't need to take any extra step for that. + +## History of communication channel + +The communication channel has changed since first iteration, you can +check here historic decisions and why we moved to new implementations. + +### Custom code (GitLab 8.6 and earlier) + +In GitLab versions before 8.6, custom code is used to handle +notification from **primary** node to **secondary** nodes by HTTP +requests. + +### System hooks (GitLab 8.7 to 9.5) + +Later, it was decided to move away from custom code and begin using +system hooks. More people were using them, so +many would benefit from improvements made to this communication layer. + +There is a specific **internal** endpoint in our API code (Grape), +that receives all requests from this System Hooks: +`/api/v4/geo/receive_events`. + +We switch and filter from each event by the `event_name` field. + +### Geo Log Cursor (GitLab 10.0 and up) + +Since GitLab 10.0, [System Webhooks](#system-hooks-gitlab-87-to-95) are no longer +used and Geo Log Cursor is used instead. The Log Cursor traverses the +`Geo::EventLog` rows to see if there are changes since the last time +the log was checked and will handle repository updates, deletes, +changes, and renames. + +The table is within the replicated database. This has two advantages over the +old method: + +- Replication is synchronous and we preserve the order of events. +- Replication of the events happen at the same time as the changes in the + database. diff --git a/doc/development/go_guide/index.md b/doc/development/go_guide/index.md index cdc806a2d31..f2519e12705 100644 --- a/doc/development/go_guide/index.md +++ b/doc/development/go_guide/index.md @@ -40,7 +40,7 @@ of possible security breaches in our code: - SQL injections Remember to run -[SAST](https://docs.gitlab.com/ee/user/project/merge_requests/sast.html) +[SAST](../../user/project/merge_requests/sast.md) **[ULTIMATE]** on your project (or at least the [gosec analyzer](https://gitlab.com/gitlab-org/security-products/analyzers/gosec)), and to follow our [Security @@ -93,10 +93,10 @@ become available, you will be able to share job templates like this Dependencies should be kept to the minimum. The introduction of a new dependency should be argued in the merge request, as per our [Approval -Guidelines](../code_review.html#approval-guidelines). Both [License -Management](https://docs.gitlab.com/ee/user/project/merge_requests/license_management.html) +Guidelines](../code_review.md#approval-guidelines). Both [License +Management](../../user/project/merge_requests/license_management.md) **[ULTIMATE]** and [Dependency -Scanning](https://docs.gitlab.com/ee/user/project/merge_requests/dependency_scanning.html) +Scanning](../../user/project/merge_requests/dependency_scanning.md) **[ULTIMATE]** should be activated on all projects to ensure new dependencies security status and license compatibility. diff --git a/doc/development/licensed_feature_availability.md b/doc/development/licensed_feature_availability.md new file mode 100644 index 00000000000..09e3f189bc0 --- /dev/null +++ b/doc/development/licensed_feature_availability.md @@ -0,0 +1,37 @@ +# Licensed feature availability + +As of GitLab 9.4, we've been supporting a simplified version of licensed +feature availability checks via `ee/app/models/license.rb`, both for +on-premise or GitLab.com plans and features. + +## Restricting features scoped by namespaces or projects + +GitLab.com plans are persisted on user groups and namespaces, therefore, if you're adding a +feature such as [Related issues](../user/project/issues/related_issues.md) or +[Service desk](../user/project/service_desk.md), +it should be restricted on namespace scope. + +1. Add the feature symbol on `EES_FEATURES`, `EEP_FEATURES` or `EEU_FEATURES` constants in + `ee/app/models/license.rb`. Note on `ee/app/models/ee/namespace.rb` that _Bronze_ GitLab.com + features maps to on-premise _EES_, _Silver_ to _EEP_ and _Gold_ to _EEU_. +2. Check using: + +```ruby +project.feature_available?(:feature_symbol) +``` + +## Restricting global features (instance) + +However, for features such as [Geo](../gitlab-geo/README.md) and +[Load balancing](../administration/database_load_balancing.md), which cannot be restricted +to only a subset of projects or namespaces, the check will be made directly in +the instance license. + +1. Add the feature symbol on `EES_FEATURES`, `EEP_FEATURES` or `EEU_FEATURES` constants in + `ee/app/models/license.rb`. +2. Add the same feature symbol to `GLOBAL_FEATURES` +3. Check using: + +```ruby +License.feature_available?(:feature_symbol) +``` diff --git a/doc/development/packages.md b/doc/development/packages.md new file mode 100644 index 00000000000..e22144bfbb5 --- /dev/null +++ b/doc/development/packages.md @@ -0,0 +1,68 @@ +# Packages + +This document will guide you through adding another [package management system](../administration/packages.md) support to GitLab. + +See already supported package types in [Packages documentation](../administration/packages.md) + +Since GitLab packages' UI is pretty generic, it is possible to add new +package system support by solely backend changes. This guide is superficial and does +not cover the way the code should be written. However, you can find a good example +by looking at existing merge requests with Maven and NPM support: + +- [NPM registry support](https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/8673). +- [Maven repository](https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/6607). +- [Instance level endpoint for Maven repository](https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/8757) + +## General information + +The existing database model requires the following: + +- Every package belongs to a project. +- Every package file belongs to a package. +- A package can have one or more package files. +- The package model is based on storing information about the package and its version. + +## API endpoints + +Package systems work with GitLab via API. For example `ee/lib/api/npm_packages.rb` +implements API endpoints to work with NPM clients. So, the first thing to do is to +add a new `ee/lib/api/your_name_packages.rb` file with API endpoints that are +necessary to make the package system client to work. Usually that means having +endpoints like: + +- GET package information. +- GET package file content. +- PUT upload package. + +Since the packages belong to a project, it's expected to have project-level endpoint +for uploading and downloading them. For example: + +``` +GET https://gitlab.com/api/v4/projects/<your_project_id>/packages/npm/ +PUT https://gitlab.com/api/v4/projects/<your_project_id>/packages/npm/ +``` + +Group-level and instance-level endpoints are good to have but are optional. + +NOTE: **Note:** +To avoid name conflict for instance-level endpoints we use +[the package naming convention](../user/project/packages/npm_registry.html#package-naming-convention) + +## Configuration + +GitLab has a `packages` section in its configuration file (`gitlab.rb`). +It applies to all package systems supported by GitLab. Usually you don't need +to add anything there. + +Packages can be configured to use object storage, therefore your code must support it. + +## Database + +The current database model allows you to store a name and a version for each package. +Every time you upload a new package, you can either create a new record of `Package` +or add files to existing record. `PackageFile` should be able to store all file-related +information like the file `name`, `side`, `sha1`, etc. + +If there is specific data necessary to be stored for only one package system support, +consider creating a separate metadata model. See `packages_maven_metadata` table +and `Packages::MavenMetadatum` model as example for package specific data. diff --git a/doc/git_hooks/git_hooks.md b/doc/git_hooks/git_hooks.md new file mode 100644 index 00000000000..b251e58410a --- /dev/null +++ b/doc/git_hooks/git_hooks.md @@ -0,0 +1,5 @@ +--- +redirect_to: '../push_rules/push_rules.md' +--- + +This document was moved to [another location](../push_rules/push_rules.md) diff --git a/doc/gitlab-basics/create-project.md b/doc/gitlab-basics/create-project.md index 3d4259ab7d3..0dd792a89a8 100644 --- a/doc/gitlab-basics/create-project.md +++ b/doc/gitlab-basics/create-project.md @@ -16,6 +16,7 @@ To create a project in GitLab: - [Import a project](../user/project/import/index.md) from a different repository, if enabled on your GitLab instance. Contact your GitLab admin if this is unavailable. + - Run [CI/CD pipelines for external repositories](../ci/ci_cd_for_external_repos/index.md). **[PREMIUM]** ## Blank projects @@ -55,7 +56,7 @@ Built-in templates are project templates that are: To use a built-in template on the **New project** page: -1. On the **Create from template** tab. +1. On the **Create from template** tab, select the **Built-in** tab. 1. From the list of available built-in templates, click the: - **Preview** button to look at the template source itself. - **Use template** button to start creating the project. diff --git a/doc/gitlab-geo/README.md b/doc/gitlab-geo/README.md new file mode 100644 index 00000000000..30d21db7de5 --- /dev/null +++ b/doc/gitlab-geo/README.md @@ -0,0 +1,5 @@ +--- +redirect_to: '../administration/geo/replication/index.md' +--- + +This document was moved to [another location](../administration/geo/replication/index.md). diff --git a/doc/gitlab-geo/after_setup.md b/doc/gitlab-geo/after_setup.md new file mode 100644 index 00000000000..c8a7b9d1096 --- /dev/null +++ b/doc/gitlab-geo/after_setup.md @@ -0,0 +1,5 @@ +--- +redirect_to: '../administration/geo/replication/using_a_geo_server.md' +--- + +This document was moved to [another location](../administration/geo/replication/using_a_geo_server.md). diff --git a/doc/gitlab-geo/bring-primary-back.md b/doc/gitlab-geo/bring-primary-back.md new file mode 100644 index 00000000000..8c43f4d805f --- /dev/null +++ b/doc/gitlab-geo/bring-primary-back.md @@ -0,0 +1,5 @@ +--- +redirect_to: '../administration/geo/disaster_recovery/bring_primary_back.md' +--- + +This document was moved to [another location](../administration/geo/disaster_recovery/bring_primary_back.md). diff --git a/doc/gitlab-geo/configuration.md b/doc/gitlab-geo/configuration.md new file mode 100644 index 00000000000..b46a2caea4a --- /dev/null +++ b/doc/gitlab-geo/configuration.md @@ -0,0 +1,5 @@ +--- +redirect_to: '../administration/geo/replication/configuration.md' +--- + +This document was moved to [another location](../administration/geo/replication/configuration.md). diff --git a/doc/gitlab-geo/configuration_source.md b/doc/gitlab-geo/configuration_source.md new file mode 100644 index 00000000000..f1aab86fadc --- /dev/null +++ b/doc/gitlab-geo/configuration_source.md @@ -0,0 +1,5 @@ +--- +redirect_to: '../administration/geo/replication/configuration_source.md' +--- + +This document was moved to [another location](../administration/geo/replication/configuration_source.md). diff --git a/doc/gitlab-geo/database.md b/doc/gitlab-geo/database.md new file mode 100644 index 00000000000..b4156dc4ec6 --- /dev/null +++ b/doc/gitlab-geo/database.md @@ -0,0 +1,5 @@ +--- +redirect_to: '../administration/geo/replication/database.md' +--- + +This document was moved to [another location](../administration/geo/replication/database.md). diff --git a/doc/gitlab-geo/database_source.md b/doc/gitlab-geo/database_source.md new file mode 100644 index 00000000000..3392d0f02c0 --- /dev/null +++ b/doc/gitlab-geo/database_source.md @@ -0,0 +1,5 @@ +--- +redirect_to: '../administration/geo/replication/database_source.md' +--- + +This document was moved to [another location](../administration/geo/replication/database_source.md). diff --git a/doc/gitlab-geo/disaster-recovery.md b/doc/gitlab-geo/disaster-recovery.md new file mode 100644 index 00000000000..d42e815a879 --- /dev/null +++ b/doc/gitlab-geo/disaster-recovery.md @@ -0,0 +1,5 @@ +--- +redirect_to: '../administration/geo/disaster_recovery/index.md' +--- + +This document was moved to [another location](../administration/geo/disaster_recovery/index.md). diff --git a/doc/gitlab-geo/docker_registry.md b/doc/gitlab-geo/docker_registry.md new file mode 100644 index 00000000000..26a708f6845 --- /dev/null +++ b/doc/gitlab-geo/docker_registry.md @@ -0,0 +1,5 @@ +--- +redirect_to: '../administration/geo/replication/docker_registry.md' +--- + +This document was moved to [another location](../administration/geo/replication/docker_registry.md). diff --git a/doc/gitlab-geo/faq.md b/doc/gitlab-geo/faq.md new file mode 100644 index 00000000000..f1952bc7e4c --- /dev/null +++ b/doc/gitlab-geo/faq.md @@ -0,0 +1,5 @@ +--- +redirect_to: '../administration/geo/replication/faq.md' +--- + +This document was moved to [another location](../administration/geo/replication/faq.md). diff --git a/doc/gitlab-geo/ha.md b/doc/gitlab-geo/ha.md new file mode 100644 index 00000000000..23ed11eaf09 --- /dev/null +++ b/doc/gitlab-geo/ha.md @@ -0,0 +1,5 @@ +--- +redirect_to: '../administration/geo/replication/high_availability.md' +--- + +This document was moved to [another location](../administration/geo/replication/high_availability.md). diff --git a/doc/gitlab-geo/object_storage.md b/doc/gitlab-geo/object_storage.md new file mode 100644 index 00000000000..1f29b7b7e8c --- /dev/null +++ b/doc/gitlab-geo/object_storage.md @@ -0,0 +1,5 @@ +--- +redirect_to: '../administration/geo/replication/object_storage.md' +--- + +This document was moved to [another location](../administration/geo/replication/object_storage.md). diff --git a/doc/gitlab-geo/planned-failover.md b/doc/gitlab-geo/planned-failover.md new file mode 100644 index 00000000000..720b6bc9424 --- /dev/null +++ b/doc/gitlab-geo/planned-failover.md @@ -0,0 +1,5 @@ +--- +redirect_to: '../administration/geo/disaster_recovery/planned_failover.md' +--- + +This document was moved to [another location](../administration/geo/disaster_recovery/planned_failover.md). diff --git a/doc/gitlab-geo/security-review.md b/doc/gitlab-geo/security-review.md new file mode 100644 index 00000000000..a0a5b0e536c --- /dev/null +++ b/doc/gitlab-geo/security-review.md @@ -0,0 +1,5 @@ +--- +redirect_to: '../administration/geo/replication/security_review.md' +--- + +This document was moved to [another location](../administration/geo/replication/security_review.md). diff --git a/doc/gitlab-geo/ssh.md b/doc/gitlab-geo/ssh.md new file mode 100644 index 00000000000..4f8db687850 --- /dev/null +++ b/doc/gitlab-geo/ssh.md @@ -0,0 +1,5 @@ +--- +redirect_to: '../administration/operations/fast_ssh_key_lookup.md' +--- + +This document was moved to [another location](../administration/operations/fast_ssh_key_lookup.md). diff --git a/doc/gitlab-geo/troubleshooting.md b/doc/gitlab-geo/troubleshooting.md new file mode 100644 index 00000000000..25fe1372c69 --- /dev/null +++ b/doc/gitlab-geo/troubleshooting.md @@ -0,0 +1,5 @@ +--- +redirect_to: '../administration/geo/replication/troubleshooting.md' +--- + +This document was moved to [another location](../administration/geo/replication/troubleshooting.md). diff --git a/doc/gitlab-geo/tuning.md b/doc/gitlab-geo/tuning.md new file mode 100644 index 00000000000..84ac40f99db --- /dev/null +++ b/doc/gitlab-geo/tuning.md @@ -0,0 +1,5 @@ +--- +redirect_to: '../administration/geo/replication/tuning.md' +--- + +This document was moved to [another location](../administration/geo/replication/tuning.md). diff --git a/doc/gitlab-geo/updating_the_geo_nodes.md b/doc/gitlab-geo/updating_the_geo_nodes.md new file mode 100644 index 00000000000..28234ec02ed --- /dev/null +++ b/doc/gitlab-geo/updating_the_geo_nodes.md @@ -0,0 +1,5 @@ +--- +redirect_to: '../administration/geo/replication/updating_the_geo_nodes.md' +--- + +This document was moved to [another location](../administration/geo/replication/updating_the_geo_nodes.md). diff --git a/doc/gitlab-geo/using_a_geo_server.md b/doc/gitlab-geo/using_a_geo_server.md new file mode 100644 index 00000000000..c8a7b9d1096 --- /dev/null +++ b/doc/gitlab-geo/using_a_geo_server.md @@ -0,0 +1,5 @@ +--- +redirect_to: '../administration/geo/replication/using_a_geo_server.md' +--- + +This document was moved to [another location](../administration/geo/replication/using_a_geo_server.md). diff --git a/doc/img/devops-stages.png b/doc/img/devops-stages.png Binary files differindex 424bce95607..2fa5357062c 100644 --- a/doc/img/devops-stages.png +++ b/doc/img/devops-stages.png diff --git a/doc/install/ldap.md b/doc/install/ldap.md new file mode 100644 index 00000000000..a19f0342b65 --- /dev/null +++ b/doc/install/ldap.md @@ -0,0 +1,7 @@ +--- +redirect_to: '../administration/auth/ldap.md' +--- + +# GitLab LDAP integration + +This document was moved under [`administration/auth/ldap`](../administration/auth/ldap.md). diff --git a/doc/install/pivotal/index.md b/doc/install/pivotal/index.md new file mode 100644 index 00000000000..896e01ad975 --- /dev/null +++ b/doc/install/pivotal/index.md @@ -0,0 +1,45 @@ +# GitLab Pivotal Tile **[PREMIUM ONLY]** + +> Introduced in [GitLab Premium][eep] 8.2. + +Easily deploy GitLab as a pre-configured appliance using Ops Manager (BOSH) for +[Pivotal Cloud Foundry][pcf]. + +## Overview + +Enterprise admins want their development toolkit to be more customizable, more +integrated, and more secure. With Pivotal Cloud Foundry, GitLab is installed and +scales easily in a highly available environment. + +The upgrades are pain-free and well tested. All it takes is upload the new tile +and click a button to begin the upgrade process. + +## Use cases + +- You want a highly available deployment with minimal effort. Scale horizontally + as your user base grows. + +## Features + +The GitLab Pivotal Tile is based on [GitLab Premium][eep] and includes nearly all of its features. The features in Premium but _not_ supported on the Tile are: + +* PostgreSQL +* Pages +* Geo +* Registry +* Mattermost +* Subgroups +* Elasticsearch +* Service Desk +* OAuth & Kerberos Authentication + +## Installing GitLab with Pivotal + +The product information and installation documentation is hosted on Pivotal's +website: + +- [Product page](https://network.pivotal.io/products/p-gitlab/) +- [Documentation](https://docs.pivotal.io/partners/gitlab/index.html) + +[eep]: https://about.gitlab.com/pricing/ +[pcf]: https://pivotal.io/platform diff --git a/doc/install/requirements.md b/doc/install/requirements.md index 8ab7189c2a6..fa1b7407272 100644 --- a/doc/install/requirements.md +++ b/doc/install/requirements.md @@ -105,8 +105,10 @@ features of GitLab work with MySQL/MariaDB: 1. MySQL support for subgroups was [dropped with GitLab 9.3][post]. See [issue #30472][30472] for more information. -1. Geo does [not support MySQL](https://docs.gitlab.com/ee/administration/geo/replication/database.html#mysql-replication). This means no supported Disaster Recovery solution if using MySQL. **[PREMIUM ONLY]** +1. Geo does [not support MySQL](../administration/geo/replication/database.md#mysql-replication). This means no supported Disaster Recovery solution if using MySQL. **[PREMIUM ONLY]** 1. [Zero downtime migrations](../update/README.md#upgrading-without-downtime) do not work with MySQL. +1. [Database load balancing](../administration/database_load_balancing.md) is + supported only for PostgreSQL. 1. GitLab [optimizes the loading of dashboard events](https://gitlab.com/gitlab-org/gitlab-ce/issues/31806) using [PostgreSQL LATERAL JOINs](https://blog.heapanalytics.com/postgresqls-powerful-new-join-type-lateral/). 1. In general, SQL optimized for PostgreSQL may run much slower in MySQL due to differences in query planners. For example, subqueries that work well in PostgreSQL @@ -141,7 +143,17 @@ On some systems you may need to install an additional package (e.g. #### Additional requirements for GitLab Geo -If you are using [GitLab Geo](https://docs.gitlab.com/ee/development/geo.html), the [tracking database](https://docs.gitlab.com/ee/development/geo.html#geo-tracking-database) also requires the `postgres_fdw` extension. +If you are using [GitLab Geo](https://docs.gitlab.com/ee/development/geo.html): + +- We strongly recommend running Omnibus-managed instances as they are actively + developed and tested. We aim to be compatible with most external (not managed + by Omnibus) databases (for example, AWS RDS) but we do not guarantee + compatibility. +- The + [tracking database](https://docs.gitlab.com/ee/development/geo.html#geo-tracking-database) + requires the + [postgres_fdw](https://www.postgresql.org/docs/9.6/static/postgres-fdw.html) + extension. ``` CREATE EXTENSION postgres_fdw; diff --git a/doc/integration/README.md b/doc/integration/README.md index f5bc0693b84..56fe51fadc2 100644 --- a/doc/integration/README.md +++ b/doc/integration/README.md @@ -15,7 +15,9 @@ See the documentation below for details on how to configure these services. - [CAS](cas.md) Configure GitLab to sign in using CAS - [External issue tracker](external-issue-tracker.md) Redmine, JIRA, etc. - [Gmail actions buttons](gmail_action_buttons_for_gitlab.md) Adds GitLab actions to messages +- [Jenkins](jenkins.md) Integrate with the Jenkins CI - [JIRA](../user/project/integrations/jira.md) Integrate with the JIRA issue tracker +- [Kerberos](kerberos.md) Integrate with Kerberos - [LDAP](ldap.md) Set up sign in via LDAP - [OAuth2 provider](oauth_provider.md) OAuth2 application creation - [OmniAuth](omniauth.md) Sign in via Twitter, GitHub, GitLab.com, Google, Bitbucket, Facebook, Shibboleth, SAML, Crowd, Azure and Authentiq ID diff --git a/doc/integration/elasticsearch.md b/doc/integration/elasticsearch.md new file mode 100644 index 00000000000..4b81526a891 --- /dev/null +++ b/doc/integration/elasticsearch.md @@ -0,0 +1,416 @@ +# Elasticsearch integration **[STARTER ONLY]** + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/109 "Elasticsearch Merge Request") in GitLab [Starter](https://about.gitlab.com/pricing/) 8.4. Support +> for [Amazon Elasticsearch](http://docs.aws.amazon.com/elasticsearch-service/latest/developerguide/es-gsg.html) was [introduced](https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/1305) in GitLab +> [Starter](https://about.gitlab.com/pricing/) 9.0. + +This document describes how to set up Elasticsearch with GitLab. Once enabled, +you'll have the benefit of fast search response times and the advantage of two +special searches: + +- [Advanced Global Search](../user/search/advanced_global_search.md) +- [Advanced Syntax Search](../user/search/advanced_search_syntax.md) + +## Version Requirements +<!-- Please remember to update ee/lib/system_check/app/elasticsearch_check.rb if this changes --> + +| GitLab version | Elasticsearch version | +| -------------- | --------------------- | +| GitLab Enterprise Edition 8.4 - 8.17 | Elasticsearch 2.4 with [Delete By Query Plugin](https://www.elastic.co/guide/en/elasticsearch/plugins/2.4/plugins-delete-by-query.html) installed | +| GitLab Enterprise Edition 9.0 - 11.4 | Elasticsearch 5.1 - 5.5 | +| GitLab Enterprise Edition 11.5+ | Elasticsearch 5.6 - 6.x | + +## Installing Elasticsearch + +Elasticsearch is _not_ included in the Omnibus packages. You will have to +install it yourself whether you are using the Omnibus package or installed +GitLab from source. Providing detailed information on installing Elasticsearch +is out of the scope of this document. + +Once the data is added to the database or repository and [Elasticsearch is +enabled in the admin area](#enabling-elasticsearch) the search index will be +updated automatically. Elasticsearch can be installed on the same machine as +GitLab, or on a separate server, or you can use the [Amazon Elasticsearch](http://docs.aws.amazon.com/elasticsearch-service/latest/developerguide/es-gsg.html) +service. + +You can follow the steps as described in the [official web site](https://www.elastic.co/guide/en/elasticsearch/reference/current/_installation.html "Elasticsearch installation documentation") or +use the packages that are available for your OS. + +## Elasticsearch repository indexer (beta) + +In order to improve elasticsearch indexing performance GitLab has made available a [new indexer written in Go](https://gitlab.com/gitlab-org/gitlab-elasticsearch-indexer). +This will replace the included Ruby indexer in the future but should be considered beta software for now, so there may be some bugs. + +If you would like to use it, please follow the instructions below. + +### Installation + +First, we need to install some dependencies, then we'll build and install +the indexer itself. + +#### Dependencies + +This project relies on [ICU](http://site.icu-project.org/) for text encoding, +therefore we need to ensure the development packages for your platform are +installed before running `make`. + +##### Debian / Ubuntu + +To install on Debian or Ubutu, run: + +```sh +sudo apt install libicu-dev +``` + +##### Mac OSX + +To install on macOS, run: + +```sh +brew install icu4c +export PKG_CONFIG_PATH="/usr/local/opt/icu4c/lib/pkgconfig:$PKG_CONFIG_PATH" +``` + +#### Building and installing + +To build and install the indexer, run: + +```sh +make +sudo make install +``` + +The `gitlab-elasticsearch-indexer` will be installed to `/usr/local/bin`. + +You can change the installation path with the `PREFIX` env variable. +Please remember to pass the `-E` flag to `sudo` if you do so. + +Example: + +```sh +PREFIX=/usr sudo -E make install +``` + +Once installed, enable it under your instance's elasticsearch settings explained [below](#enabling-elasticsearch). + +## System Requirements + +Elasticsearch requires additional resources in excess of those documented in the +[GitLab system requirements](../install/requirements.md). These will vary by +installation size, but you should ensure **at least** an additional 8 GiB of RAM +for each Elasticsearch node, per the [official guidelines](https://www.elastic.co/guide/en/elasticsearch/guide/current/hardware.html). + +Storage requirements also vary based on installation side, but as a rule of +thumb, you should allocate the total size of your production database, **plus** +two-thirds of the total size of your git repositories. Efforts to reduce this +total are being tracked in this epic: [gitlab-org&153](https://gitlab.com/groups/gitlab-org/-/epics/153). + +## Enabling Elasticsearch + +In order to enable Elasticsearch, you need to have admin access. Go to +**Admin > Settings** and find the "Elasticsearch" section. + +The following Elasticsearch settings are available: + +| Parameter | Description | +| --------- | ----------- | +| `Elasticsearch indexing` | Enables/disables Elasticsearch indexing. You may want to enable indexing but disable search in order to give the index time to be fully completed, for example. Also keep in mind that this option doesn't have any impact on existing data, this only enables/disables background indexer which tracks data changes. So by enabling this you will not get your existing data indexed, use special rake task for that as explained in [Adding GitLab's data to the Elasticsearch index](#adding-gitlabs-data-to-the-elasticsearch-index). | +| `Use the new repository indexer (beta)` | Perform repository indexing using [GitLab Elasticsearch Indexer](https://gitlab.com/gitlab-org/gitlab-elasticsearch-indexer). | +| `Search with Elasticsearch enabled` | Enables/disables using Elasticsearch in search. | +| `URL` | The URL to use for connecting to Elasticsearch. Use a comma-separated list to support clustering (e.g., "http://host1, https://host2:9200"). If your Elasticsearch instance is password protected, pass the `username:password` in the URL (e.g., `http://<username>:<password>@<elastic_host>:9200/`). | +| `Using AWS hosted Elasticsearch with IAM credentials` | Sign your Elasticsearch requests using [AWS IAM authorization](http://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_access-keys.html) or [AWS EC2 Instance Profile Credentials](http://docs.aws.amazon.com/codedeploy/latest/userguide/getting-started-create-iam-instance-profile.html#getting-started-create-iam-instance-profile-cli). The policies must be configured to allow `es:*` actions. | +| `AWS Region` | The AWS region your Elasticsearch service is located in. | +| `AWS Access Key` | The AWS access key. | +| `AWS Secret Access Key` | The AWS secret access key. | + +## Disabling Elasticsearch + +To disable the Elasticsearch integration: + +1. Navigate to the **Admin area > Settings** +1. Find the 'Elasticsearch' section and uncheck 'Search with Elasticsearch enabled' + and 'Elasticsearch indexing' +1. Click **Save** for the changes to take effect + +## Adding GitLab's data to the Elasticsearch index + +### Indexing small instances (database size less than 500 MiB, size of repos less than 5 GiB) + +Configure Elasticsearch's host and port in **Admin > Settings**. Then create empty indexes using one of the following commands: + +```sh +# Omnibus installations +sudo gitlab-rake gitlab:elastic:create_empty_index + +# Installations from source +bundle exec rake gitlab:elastic:create_empty_index RAILS_ENV=production +``` + +Then enable Elasticsearch indexing and run repository indexing tasks: + +```sh +# Omnibus installations +sudo gitlab-rake gitlab:elastic:index + +# Installations from source +bundle exec rake gitlab:elastic:index RAILS_ENV=production +``` + +Enable Elasticsearch search. + +### Indexing large instances + +NOTE: **Note**: +After indexing the repositories asynchronously, you MUST index the database to be able to search. + +Configure Elasticsearch's host and port in **Admin > Settings**. Then create empty indexes using one of the following commands: + +```sh +# Omnibus installations +sudo gitlab-rake gitlab:elastic:create_empty_index + +# Installations from source +bundle exec rake gitlab:elastic:create_empty_index RAILS_ENV=production +``` + +Indexing large Git repositories can take a while. To speed up the process, you +can temporarily disable auto-refreshing and replicating. In our experience you can expect a 20% +time drop. We'll enable them when indexing is done. This step is optional! + +```bash +curl --request PUT localhost:9200/gitlab-production/_settings --data '{ + "index" : { + "refresh_interval" : "-1", + "number_of_replicas" : 0 + } }' +``` + +Then enable Elasticsearch indexing and run repository indexing tasks: + +```sh +# Omnibus installations +sudo gitlab-rake gitlab:elastic:index_repositories_async + +# Installations from source +bundle exec rake gitlab:elastic:index_repositories_async RAILS_ENV=production +``` + +This enqueues a number of Sidekiq jobs to index your existing repositories. +You can view the jobs in the admin panel (they are placed in the `elastic_batch_project_indexer`) +queue), or you can query indexing status using a rake task: + +```sh +# Omnibus installations +sudo gitlab-rake gitlab:elastic:index_repositories_status + +# Installations from source +bundle exec rake gitlab:elastic:index_repositories_status RAILS_ENV=production + +Indexing is 65.55% complete (6555/10000 projects) +``` + +By default, one job is created for every 300 projects. For large numbers of +projects, you may wish to increase the batch size, by setting the `BATCH` +environment variable. + +You can also run the initial indexing synchronously - this is most useful if +you have a small number of projects, or need finer-grained control over indexing +than Sidekiq permits: + +```sh +# Omnibus installations +sudo gitlab-rake gitlab:elastic:index_repositories + +# Installations from source +bundle exec rake gitlab:elastic:index_repositories RAILS_ENV=production +``` + +It might take a while depending on how big your Git repositories are. + +If you want to run several tasks in parallel (probably in separate terminal +windows) you can provide the `ID_FROM` and `ID_TO` parameters: + +```sh +# Omnibus installations +sudo gitlab-rake gitlab:elastic:index_repositories ID_FROM=1001 ID_TO=2000 + +# Installations from source +bundle exec rake gitlab:elastic:index_repositories ID_FROM=1001 ID_TO=2000 RAILS_ENV=production +``` + +Where `ID_FROM` and `ID_TO` are project IDs. Both parameters are optional. +As an example, if you have 3,000 repositories and you want to run three separate indexing tasks, you might run: + +```sh +# Omnibus installations +sudo gitlab-rake gitlab:elastic:index_repositories ID_TO=1000 +sudo gitlab-rake gitlab:elastic:index_repositories ID_FROM=1001 ID_TO=2000 +sudo gitlab-rake gitlab:elastic:index_repositories ID_FROM=2001 + +# Installations from source +bundle exec rake gitlab:elastic:index_repositories RAILS_ENV=production ID_TO=1000 +bundle exec rake gitlab:elastic:index_repositories RAILS_ENV=production ID_FROM=1001 ID_TO=2000 +bundle exec rake gitlab:elastic:index_repositories RAILS_ENV=production ID_FROM=2001 +``` + +Sometimes your repository index process `gitlab:elastic:index_repositories` or +`gitlab:elastic:index_repositories_async` can get interrupted. This may happen +for many reasons, but it's always safe to run the indexing job again - it will +skip those repositories that have already been indexed. + +As the indexer stores the last commit SHA of every indexed repository in the +database, you can run the indexer with the special parameter `UPDATE_INDEX` and +it will check every project repository again to make sure that every commit in +that repository is indexed, it can be useful in case if your index is outdated: + +```sh +# Omnibus installations +sudo gitlab-rake gitlab:elastic:index_repositories UPDATE_INDEX=true ID_TO=1000 + +# Installations from source +bundle exec rake gitlab:elastic:index_repositories UPDATE_INDEX=true ID_TO=1000 RAILS_ENV=production +``` + +You can also use the `gitlab:elastic:clear_index_status` Rake task to force the +indexer to "forget" all progresss, so retrying the indexing process from the +start. + +To index all wikis: + +```sh +# Omnibus installations +sudo gitlab-rake gitlab:elastic:index_wikis + +# Installations from source +bundle exec rake gitlab:elastic:index_wikis RAILS_ENV=production +``` + +The wiki indexer also supports the `ID_FROM` and `ID_TO` parameters if you want +to limit a project set. + +Index all database entities (Keep in mind it can take a while so consider using `screen` or `tmux`): + +```sh +# Omnibus installations +sudo gitlab-rake gitlab:elastic:index_database + +# Installations from source +bundle exec rake gitlab:elastic:index_database RAILS_ENV=production +``` + +Enable replication and refreshing again after indexing (only if you previously disabled it): + +```bash +curl --request PUT localhost:9200/gitlab-production/_settings --data '{ + "index" : { + "number_of_replicas" : 1, + "refresh_interval" : "1s" + } }' +``` + +A force merge should be called after enabling the refreshing above: + +```bash +curl --request POST 'http://localhost:9200/_forcemerge?max_num_segments=5' +``` + +Enable Elasticsearch search in **Admin > Settings**. That's it. Enjoy it! + +## Tuning + +### Deleted documents + +Whenever a change or deletion is made to an indexed GitLab object (a merge request description is changed, a file is deleted from the master branch in a repository, a project is deleted, etc), a document in the index is deleted. However, since these are "soft" deletes, the overall number of "deleted documents", and therefore wasted space, increases. Elasticsearch does intelligent merging of segments in order to remove these deleted documents. However, depending on the amount and type of activity in your GitLab installation, it's possible to see as much as 50% wasted space in the index. + +In general, we recommend simply letting Elasticseach merge and reclaim space automatically, with the default settings. From [Lucene's Handling of Deleted Documents](https://www.elastic.co/blog/lucenes-handling-of-deleted-documents "Lucene's Handling of Deleted Documents"), _"Overall, besides perhaps decreasing the maximum segment size, it is best to leave Lucene's defaults as-is and not fret too much about when deletes are reclaimed."_ + +However, some larger installations may wish to tune the merge policy settings: + +- Consider reducing the `index.merge.policy.max_merged_segment` size from the default 5 GB to maybe 2 GB or 3 GB. Merging only happens when a segment has at least 50% deletions. Smaller segment sizes will allow merging to happen more frequently. + + ```bash + curl --request PUT http://localhost:9200/gitlab-production/_settings --data '{ + "index" : { + "merge.policy.max_merged_segment": "2gb" + } + }' + ``` + +- You can also adjust `index.merge.policy.reclaim_deletes_weight`, which controls how aggressively deletions are targetd. But this can lead to costly merge decisions, so we recommend not changing this unless you understand the tradeoffs. + + ```bash + curl --request PUT http://localhost:9200/gitlab-production/_settings --data '{ + "index" : { + "merge.policy.reclaim_deletes_weight": "3.0" + } + }' + ``` + +- Do not do a [force merge](https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-forcemerge.html "Force Merge") to remove deleted documents. A warning in the [documentation](https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-forcemerge.html "Force Merge") states that this can lead to very large segments that may never get reclaimed, and can also cause significant performance or availability issues. + +## Troubleshooting + +Here are some common pitfalls and how to overcome them: + +- **I updated GitLab and now I can't find anything** + + We continuously make updates to our indexing strategies and aim to support + newer versions of Elasticsearch. When indexing changes are made, it may + be necessary for you to [reindex](#adding-gitlabs-data-to-the-elasticsearch-index) after updating GitLab. + +- **I indexed all the repositories but I can't find anything** + + Make sure you indexed all the database data [as stated above](#adding-gitlabs-data-to-the-elasticsearch-index). + +- **I indexed all the repositories but then switched elastic search servers and now I can't find anything** + + You will need to re-run all the rake tasks to re-index the database, repositories, and wikis. + +- **No new data is added to the Elasticsearch index when I push code** + + When performing the initial indexing of blobs, we lock all projects until the project finishes indexing. It could + happen that an error during the process causes one or multiple projects to remain locked. In order to unlock them, + run the `gitlab:elastic:clear_locked_projects` rake task. + +- **"Can't specify parent if no parent field has been configured"** + + If you enabled Elasticsearch before GitLab 8.12 and have not rebuilt indexes you will get + exception in lots of different cases: + + ```text + Elasticsearch::Transport::Transport::Errors::BadRequest([400] { + "error": { + "root_cause": [{ + "type": "illegal_argument_exception", + "reason": "Can't specify parent if no parent field has been configured" + }], + "type": "illegal_argument_exception", + "reason": "Can't specify parent if no parent field has been configured" + }, + "status": 400 + }): + ``` + + This is because we changed the index mapping in GitLab 8.12 and the old indexes should be removed and built from scratch again, + see details in the [8-11-to-8-12 update guide](https://gitlab.com/gitlab-org/gitlab-ee/blob/master/doc/update/8.11-to-8.12.md#11-elasticsearch-index-update-if-you-currently-use-elasticsearch). + +- Exception `Elasticsearch::Transport::Transport::Errors::BadRequest` + + If you have this exception (just like in the case above but the actual message is different) please check if you have the correct Elasticsearch version and you met the other [requirements](#system-requirements). + There is also an easy way to check it automatically with `sudo gitlab-rake gitlab:check` command. + +- Exception `Elasticsearch::Transport::Transport::Errors::RequestEntityTooLarge` + + ```text + [413] {"Message":"Request size exceeded 10485760 bytes"} + ``` + + This exception is seen when your Elasticsearch cluster is configured to reject + requests above a certain size (10MiB in this case). This corresponds to the + `http.max_content_length` setting in `elasticsearch.yml`. Increase it to a + larger size and restart your Elasticsearch cluster. + + AWS has [fixed limits](http://docs.aws.amazon.com/elasticsearch-service/latest/developerguide/aes-limits.html) + for this setting ("Maximum Size of HTTP Request Payloads"), based on the size of + the underlying instance. + diff --git a/doc/integration/img/github_app_entry.png b/doc/integration/img/github_app_entry.png Binary files differindex 0a1fe0ca65a..c736bed3dd6 100644 --- a/doc/integration/img/github_app_entry.png +++ b/doc/integration/img/github_app_entry.png diff --git a/doc/integration/img/github_register_app.png b/doc/integration/img/github_register_app.png Binary files differindex 5786b822f53..d03ca5cc2aa 100644 --- a/doc/integration/img/github_register_app.png +++ b/doc/integration/img/github_register_app.png diff --git a/doc/integration/img/jira_dev_panel_gl_setup_1.png b/doc/integration/img/jira_dev_panel_gl_setup_1.png Binary files differnew file mode 100644 index 00000000000..75279877c93 --- /dev/null +++ b/doc/integration/img/jira_dev_panel_gl_setup_1.png diff --git a/doc/integration/img/jira_dev_panel_jira_setup_1.png b/doc/integration/img/jira_dev_panel_jira_setup_1.png Binary files differnew file mode 100644 index 00000000000..5c0f594cc1d --- /dev/null +++ b/doc/integration/img/jira_dev_panel_jira_setup_1.png diff --git a/doc/integration/img/jira_dev_panel_jira_setup_2.png b/doc/integration/img/jira_dev_panel_jira_setup_2.png Binary files differnew file mode 100644 index 00000000000..a4778a00dd5 --- /dev/null +++ b/doc/integration/img/jira_dev_panel_jira_setup_2.png diff --git a/doc/integration/img/jira_dev_panel_jira_setup_3.png b/doc/integration/img/jira_dev_panel_jira_setup_3.png Binary files differnew file mode 100644 index 00000000000..4049a65f56b --- /dev/null +++ b/doc/integration/img/jira_dev_panel_jira_setup_3.png diff --git a/doc/integration/img/jira_dev_panel_jira_setup_4.png b/doc/integration/img/jira_dev_panel_jira_setup_4.png Binary files differnew file mode 100644 index 00000000000..81d84cb173d --- /dev/null +++ b/doc/integration/img/jira_dev_panel_jira_setup_4.png diff --git a/doc/integration/img/jira_dev_panel_jira_setup_5.png b/doc/integration/img/jira_dev_panel_jira_setup_5.png Binary files differnew file mode 100644 index 00000000000..73dc867d301 --- /dev/null +++ b/doc/integration/img/jira_dev_panel_jira_setup_5.png diff --git a/doc/integration/img/jira_dev_panel_manual_refresh.png b/doc/integration/img/jira_dev_panel_manual_refresh.png Binary files differnew file mode 100644 index 00000000000..dc92d533bde --- /dev/null +++ b/doc/integration/img/jira_dev_panel_manual_refresh.png diff --git a/doc/integration/jenkins.md b/doc/integration/jenkins.md new file mode 100644 index 00000000000..bd4245f3f93 --- /dev/null +++ b/doc/integration/jenkins.md @@ -0,0 +1,129 @@ +# Jenkins CI service **[STARTER]** + +>**Note:** +In GitLab 8.3, Jenkins integration using the +[GitLab Hook Plugin](https://wiki.jenkins.io/display/JENKINS/GitLab+Hook+Plugin) +was deprecated in favor of the +[GitLab Plugin](https://wiki.jenkins.io/display/JENKINS/GitLab+Plugin). +The deprecated integration has been renamed to [Jenkins CI (Deprecated)](jenkins_deprecated.md) in the +project service settings. We may remove this in a future release and recommend +using the new 'Jenkins CI' project service instead which is described in this +document. + +## Overview + +[Jenkins](https://jenkins.io/) is a great Continuous Integration tool, similar to our built-in +[GitLab CI](../ci/README.md). + +GitLab's Jenkins integration allows you to trigger a Jenkins build when you +push code to a repository, or when a merge request is created. Additionally, +it shows the pipeline status on merge requests widgets and on the project's home page. + +## Use cases + +- Suppose you are new to GitLab, and want to keep using Jenkins until you prepare +your projects to build with [GitLab CI/CD](../ci/README.md). You set up the +integration between GitLab and Jenkins, then you migrate to GitLab CI later. While +you organize yourself and your team to onboard GitLab, you keep your pipelines +running with Jenkins, but view the results in your project's repository in GitLab. +- Your team uses [Jenkins Plugins](https://plugins.jenkins.io/) for other proceedings, +therefore, you opt for keep using Jenkins to build your apps. Show the results of your +pipelines directly in GitLab. + +## Requirements + +* [Jenkins GitLab Plugin](https://wiki.jenkins.io/display/JENKINS/GitLab+Plugin) +* [Jenkins Git Plugin](https://wiki.jenkins.io/display/JENKINS/Git+Plugin) +* Git clone access for Jenkins from the GitLab repository +* GitLab API access to report build status + +## Configure GitLab users + +Create a user or choose an existing user that Jenkins will use to interact +through the GitLab API. This user will need to be a global Admin or added +as a member to each Group/Project. Developer permission is required for reporting +build status. This is because a successful build status can trigger a merge +when 'Merge when pipeline succeeds' feature is used. Some features of the GitLab +Plugin may require additional privileges. For example, there is an option to +accept a merge request if the build is successful. Using this feature would +require developer, maintainer or owner-level permission. + +Copy the private API token from **Profile Settings -> Account**. You will need this +when configuring the Jenkins server later. + +## Configure the Jenkins server + +Install [Jenkins GitLab Plugin](https://wiki.jenkins.io/display/JENKINS/GitLab+Plugin) +and [Jenkins Git Plugin](https://wiki.jenkins.io/display/JENKINS/Git+Plugin). + +Go to Manage Jenkins -> Configure System and scroll down to the 'GitLab' section. +Enter the GitLab server URL in the 'GitLab host URL' field and paste the API token +copied earlier in the 'API Token' field. + +For more information, see GitLab Plugin documentation about +[Jenkins-to-GitLab authentication](https://github.com/jenkinsci/gitlab-plugin#jenkins-to-gitlab-authentication) + + + +## Configure a Jenkins project + +Follow the GitLab Plugin documentation about [Jenkins Job Configuration](https://github.com/jenkinsci/gitlab-plugin#jenkins-job-configuration). + +NOTE: **Note:** +Be sure to include the steps about [Build status configuration](https://github.com/jenkinsci/gitlab-plugin#build-status-configuration). +The 'Publish build status to GitLab' post-build step is required to view +Jenkins build status in GitLab Merge Requests. + +## Configure a GitLab project + +Create a new GitLab project or choose an existing one. Then, go to **Integrations -> +Jenkins CI**. + +Check the 'Active' box. Select whether you want GitLab to trigger a build +on push, Merge Request creation, tag push, or any combination of these. We +recommend unchecking 'Merge Request events' unless you have a specific use-case +that requires re-building a commit when a merge request is created. With 'Push +events' selected, GitLab will build the latest commit on each push and the build +status will be displayed in the merge request. + +Enter the Jenkins URL and Project name. The project name should be URL-friendly +where spaces are replaced with underscores. To be safe, copy the project name +from the URL bar of your browser while viewing the Jenkins project. + +Optionally, enter a username and password if your Jenkins server requires +authentication. + + + +## Plugin functional overview + +GitLab does not contain a database table listing commits. Commits are always +read from the repository directly. Therefore, it is not possible to retain the +build status of a commit in GitLab. This is overcome by requesting build +information from the integrated CI tool. The CI tool is responsible for creating +and storing build status for Commits and Merge Requests. + +### Steps required to implement a similar integration + +>**Note:** +All steps are implemented using AJAX requests on the merge request page. + +1. In order to display the build status in a merge request you must create a project service in GitLab. +2. Your project service will do a (JSON) query to a URL of the CI tool with the SHA1 of the commit. +3. The project service builds this URL and payload based on project service settings and knowledge of the CI tool. +4. The response is parsed to give a response in GitLab (success/failed/pending). + +## Troubleshooting + +### Error in merge requests - "Could not connect to the CI server" + +This integration relies on Jenkins reporting the build status back to GitLab via +the [Commit Status API](../api/commits.md#commit-status). + +The error 'Could not connect to the CI server' usually means that GitLab did not +receive a build status update via the API. Either Jenkins was not properly +configured or there was an error reporting the status via the API. + +1. [Configure the Jenkins server](#configure-the-jenkins-server) for GitLab API access +2. [Configure a Jenkins project](#configure-a-jenkins-project), including the + 'Publish build status to GitLab' post-build action. diff --git a/doc/integration/jenkins_deprecated.md b/doc/integration/jenkins_deprecated.md new file mode 100644 index 00000000000..4003217704e --- /dev/null +++ b/doc/integration/jenkins_deprecated.md @@ -0,0 +1,53 @@ +# Jenkins CI (deprecated) service + +>**Note:** In GitLab 8.3, Jenkins integration using the +[GitLab Hook Plugin](https://wiki.jenkins.io/display/JENKINS/GitLab+Hook+Plugin) +was deprecated in favor of the +[GitLab Plugin](https://wiki.jenkins.io/display/JENKINS/GitLab+Plugin). +Please use documentation for the new [Jenkins CI service](jenkins.md). + +Integration includes: + +* Trigger Jenkins build after push to repo +* Show build status on Merge Request page + +Requirements: + +* [Jenkins GitLab Hook plugin](https://wiki.jenkins.io/display/JENKINS/GitLab+Hook+Plugin) +* git clone access for Jenkins from GitLab repo (via ssh key) + +## Jenkins + +1. Install [GitLab Hook plugin](https://wiki.jenkins.io/display/JENKINS/GitLab+Hook+Plugin) +2. Set up jenkins project + + + +## GitLab + +In GitLab, perform the following steps. + +### Read access to repository + +Jenkins needs read access to the GitLab repository. We already specified a +private key to use in Jenkins, now we need to add a public one to the GitLab +project. For that case we will need a Deploy key. Read the documentation on +[how to set up a Deploy key](../ssh/README.md#deploy-keys). + +### Jenkins service + +Now navigate to GitLab services page and activate Jenkins + + + +Done! Now when you push to GitLab - it will create a build for Jenkins. +And also you will be able to see merge request build status with a link to the Jenkins build. + +### Multi-project Configuration + +The GitLab Hook plugin in Jenkins supports the automatic creation of a project +for each feature branch. After configuration GitLab will trigger feature branch +builds and a corresponding project will be created in Jenkins. + +Configure the GitLab Hook plugin in Jenkins. Go to 'Manage Jenkins' and then +'Configure System'. Find the 'GitLab Web Hook' section and configure as shown below. diff --git a/doc/integration/jenkins_gitlab_plugin_config.png b/doc/integration/jenkins_gitlab_plugin_config.png Binary files differnew file mode 100644 index 00000000000..3e1f83e96dc --- /dev/null +++ b/doc/integration/jenkins_gitlab_plugin_config.png diff --git a/doc/integration/jenkins_gitlab_service.png b/doc/integration/jenkins_gitlab_service.png Binary files differnew file mode 100644 index 00000000000..682a5ae8ee2 --- /dev/null +++ b/doc/integration/jenkins_gitlab_service.png diff --git a/doc/integration/jenkins_gitlab_service_settings.png b/doc/integration/jenkins_gitlab_service_settings.png Binary files differnew file mode 100644 index 00000000000..5a12e9cb39a --- /dev/null +++ b/doc/integration/jenkins_gitlab_service_settings.png diff --git a/doc/integration/jenkins_multiproject_configuration.png b/doc/integration/jenkins_multiproject_configuration.png Binary files differnew file mode 100644 index 00000000000..6babeba4f16 --- /dev/null +++ b/doc/integration/jenkins_multiproject_configuration.png diff --git a/doc/integration/jenkins_multiproject_enabled.png b/doc/integration/jenkins_multiproject_enabled.png Binary files differnew file mode 100644 index 00000000000..b4fb12c5c3e --- /dev/null +++ b/doc/integration/jenkins_multiproject_enabled.png diff --git a/doc/integration/jenkins_project.png b/doc/integration/jenkins_project.png Binary files differnew file mode 100644 index 00000000000..126b05c8879 --- /dev/null +++ b/doc/integration/jenkins_project.png diff --git a/doc/integration/jenkins_unstable_passing.png b/doc/integration/jenkins_unstable_passing.png Binary files differnew file mode 100644 index 00000000000..569cf345943 --- /dev/null +++ b/doc/integration/jenkins_unstable_passing.png diff --git a/doc/integration/jira_development_panel.md b/doc/integration/jira_development_panel.md new file mode 100644 index 00000000000..db85f4eb5ca --- /dev/null +++ b/doc/integration/jira_development_panel.md @@ -0,0 +1,139 @@ +# GitLab Jira development panel integration **[PREMIUM]** + +> [Introduced][ee-2381] in [GitLab Premium][eep] 10.0. + +Complementary to our [existing Jira][existing-jira] project integration, you're now able to integrate +GitLab projects with [Jira Development Panel][jira-development-panel]. Both can be used +simultaneously. This works with self-hosted GitLab or GitLab.com integrated with self-hosted Jira +or cloud Jira. + +By doing this you can easily access related GitLab merge requests, branches, and commits directly from a Jira issue. + +This integration connects all GitLab projects within a top-level group or a personal namespace to projects in the Jira instance. +A top-level GitLab group is one that does not have any parent group itself. All the projects of that top-level group, +as well as projects of the top-level group's subgroups nesting down, are connected. Alternatively, you can specify +a GitLab personal namespace in the Jira configuration, which will then connect the projects in that personal namespace to Jira. + +NOTE: **Note**: +Note this is different from the [existing Jira][existing-jira] project integration, where the mapping +is one GitLab project to the entire Jira instance. + +We recommend that a GitLab group admin +or instance admin (in the case of self-hosted GitLab) set up the integration, +in order to simplify administration (subject to [limitations](#limitations)). + +TIP: **Tip:** +Create and use a single-purpose "jira" user in GitLab, so that removing +regular users won't impact your integration. + +## Requirements + +### Self-hosted GitLab + +If you are using self-hosted GitLab, make sure your GitLab instance is accessible by Jira. + +- If you are connecting to Jira Cloud, make sure your instance is accessible via the internet. +- If you are using Jira Server, make sure your instance is accessible however your network is set up. + +### GitLab.com + +There are no special requirements if you are using GitLab.com. + +## GitLab Configuration + +1. In GitLab, create a new application in order to allow Jira to connect with your GitLab account + + While logged-in, go to `Settings -> Applications`. (Click your profile avatar at + the top right, choose `Settings`, and then navigate to `Applications` from the left + navigation menu.) Use the form to create a new application. + + Enter a useful name for the `Name` field. + + For the `Redirect URI` field, enter `https://<your-gitlab-instance-domain>/login/oauth/callback`, + replacing `<your-gitlab-instance-domain>` appropriately. So for example, if you are using GitLab.com, + this would be `https://gitlab.com/login/oauth/callback`. + + NOTE: **Note**: + If using a GitLab version earlier than 11.3 the `Redirect URI` value should be `https://<your-gitlab-instance-domain>/-/jira/login/oauth/callback`. + +  + - Check `api` in the Scopes section. + +2. Click `Save application`. You will see the generated 'Application Id' and 'Secret' values. + Copy these values that you will use on the Jira configuration side. + +## Jira Configuration + +1. In Jira, from the gear menu at the top right, go to `Applications`. Navigate to `DVCS accounts` + from the left navigation menu. Click `Link GitHub account` to start creating a new integration. + (We are pretending to be GitHub in this integration until there is further platform support from Jira.) + +  + +2. Complete the form + + Select GitHub Enterprise for the `Host` field. + + For the `Team or User Account` field, enter the relative path of a top-level GitLab group that you have access to, + or the relative path of your personal namespace. + +  + + For the `Host URL` field, enter `https://<your-gitlab-instance-domain>/`, + replacing `<your-gitlab-instance-domain>` appropriately. So for example, if you are using GitLab.com, + this would be `https://gitlab.com/`. + + NOTE: **Note**: + If using a GitLab version earlier than 11.3 the `Host URL` value should be `https://<your-gitlab-instance-domain>/-/jira` + + For the `Client ID` field, use the `Application ID` value from the previous section. + + For the `Client Secret` field, use the `Secret` value from the previous section. + + Ensure that the rest of the checkboxes are checked. + +3. Click `Add` to complete and create the integration. + + Jira takes up to a few minutes to know about (import behind the scenes) all the commits and branches + for all the projects in the GitLab group you specified in the previous step. These are refreshed + every 60 minutes. + + > **Note:** + > In the future, we plan on implementating real-time integration. If you need + > to refresh the data manually, you can do this from the `Applications -> DVCS + > accounts` screen where you initially set up the integration: + > + >  + +To connect additional GitLab projects from other GitLab top-level groups (or personal namespaces), repeat the above +steps with additional Jira DVCS accounts. + +You may now refer any Jira issue by its ID in branch names, commit messages and merge request names on GitLab's side, +and you will be able to see the linked `branches`, `commits`, and `merge requests` when entering a Jira issue +(inside the Jira issue, merge requests will be called "pull requests"). + + + +Click the links to see your GitLab repository data. + + + + + +## Limitations + +- This integration is currently not supported on GitLab instances under a [relative url][relative-url] (e.g. `http://example.com/gitlab`). +- The GitLab user that you use to configure this integration must be a member of all + projects you wish to link to Jira. We're investigating removing this limitation in in [a feature proposal](https://gitlab.com/gitlab-org/gitlab-ee/issues/8902). + +## Changelog + +### 11.1 + +- [Support GitLab subgroups in Jira development panel](https://gitlab.com/gitlab-org/gitlab-ee/issues/3561) + +[existing-jira]: ../user/project/integrations/jira.md +[jira-development-panel]: https://confluence.atlassian.com/adminjiraserver070/integrating-with-development-tools-776637096.html#Integratingwithdevelopmenttools-Developmentpanelonissues +[eep]: https://about.gitlab.com/pricing/ +[ee-2381]: https://gitlab.com/gitlab-org/gitlab-ee/issues/2381 +[relative-url]: https://docs.gitlab.com/omnibus/settings/configuration.html#configuring-a-relative-url-for-gitlab diff --git a/doc/integration/kerberos.md b/doc/integration/kerberos.md new file mode 100644 index 00000000000..44117755b83 --- /dev/null +++ b/doc/integration/kerberos.md @@ -0,0 +1,315 @@ +# Kerberos integration **[STARTER ONLY]** + +GitLab can integrate with [Kerberos][kerb] as an authentication mechanism. + +## Overview + +[Kerberos][kerb] is a secure method for authenticating a request for a service in a +computer network. Kerberos was developed in the Athena Project at the +[Massachusetts Institute of Technology (MIT)][mit]. The name is taken from Greek +mythology; Kerberos was a three-headed dog who guarded the gates of Hades. + +## Use-cases + +- GitLab can be configured to allow your users to sign with their Kerberos credentials. +- You can use Kerberos to [prevent][why-kerb] anyone from intercepting or eavesdropping on the transmitted password. + +## Configuration + +For GitLab to offer Kerberos token-based authentication, perform the +following prerequisites. You still need to configure your system for +Kerberos usage, such as specifying realms. GitLab will make use of the +system's Kerberos settings. + +### GitLab keytab + +1. Create a Kerberos Service Principal for the HTTP service on your GitLab server. + If your GitLab server is `gitlab.example.com` and your Kerberos realm + `EXAMPLE.COM`, create a Service Principal `HTTP/gitlab.example.com@EXAMPLE.COM` + in your Kerberos database. +1. Create a keytab on the GitLab server for the above Service Principal, e.g. + `/etc/http.keytab`. + +The keytab is a sensitive file and must be readable by the GitLab user. Set +ownership and protect the file appropriately: + +``` +sudo chown git /etc/http.keytab +sudo chmod 0600 /etc/http.keytab +``` + +### Configure GitLab + +**Installations from source** + +>**Note:** +For source installations, make sure the `kerberos` gem group +[has been installed](../install/installation.md#install-gems). + +1. Edit the kerberos section of [gitlab.yml] to enable Kerberos ticket-based + authentication. In most cases, you only need to enable Kerberos and specify + the location of the keytab: + + ```yaml + omniauth: + enabled: true + allow_single_sign_on: ['kerberos'] + + kerberos: + # Allow the HTTP Negotiate authentication method for Git clients + enabled: true + + # Kerberos 5 keytab file. The keytab file must be readable by the GitLab user, + # and should be different from other keytabs in the system. + # (default: use default keytab from Krb5 config) + keytab: /etc/http.keytab + ``` + +1. [Restart GitLab] for the changes to take effect. + +--- + +**Omnibus package installations** + +1. Edit `/etc/gitlab/gitlab.rb`: + + ```ruby + gitlab_rails['omniauth_enabled'] = true + gitlab_rails['omniauth_allow_single_sign_on'] = ['kerberos'] + + gitlab_rails['kerberos_enabled'] = true + gitlab_rails['kerberos_keytab'] = "/etc/http.keytab" + ``` + +1. [Reconfigure GitLab] for the changes to take effect. + +--- + +GitLab will now offer the `negotiate` authentication method for signing in and +HTTP Git access, enabling Git clients that support this authentication protocol +to authenticate with Kerberos tokens. + +## Creating and linking Kerberos accounts + +The Administrative user can navigate to **Admin > Users > Example User > Identities** +and attach a Kerberos account. Existing GitLab users can go to **Profile > Account** +and attach a Kerberos account. If you want to allow users without a GitLab +account to login, you should enable the option `allow_single_sign_on` as +described in the [Configure GitLab](#configure-gitlab) section. Then, the first +time a user signs in with Kerberos credentials, GitLab will create a new GitLab +user associated with the email, which is built from the Kerberos username and +realm. User accounts will be created automatically when authentication was +successful. + +## Linking Kerberos and LDAP accounts together + +If your users log in with Kerberos, but you also have [LDAP integration](../administration/auth/ldap.md) +enabled, then your users will be automatically linked to their LDAP accounts on +first login. For this to work, some prerequisites must be met: + +The Kerberos username must match the LDAP user's UID. You can choose which LDAP +attribute is used as the UID in GitLab's [LDAP configuration](../administration/auth/ldap.md#configuration) +but for Active Directory, this should be `sAMAccountName`. + +The Kerberos realm must match the domain part of the LDAP user's Distinguished +Name. For instance, if the Kerberos realm is `AD.EXAMPLE.COM`, then the LDAP +user's Distinguished Name should end in `dc=ad,dc=example,dc=com`. + +Taken together, these rules mean that linking will only work if your users' +Kerberos usernames are of the form `foo@AD.EXAMPLE.COM` and their +LDAP Distinguished Names look like `sAMAccountName=foo,dc=ad,dc=example,dc=com`. + +## HTTP Git access + +A linked Kerberos account enables you to `git pull` and `git push` using your +Kerberos account, as well as your standard GitLab credentials. + +GitLab users with a linked Kerberos account can also `git pull` and `git push` +using Kerberos tokens, i.e., without having to send their password with each +operation. + +### HTTP Git access with Kerberos token (passwordless authentication) + +#### Support for Git before 2.4 + +Until Git version 2.4, the `git` command uses only the `negotiate` authentication +method if the HTTP server offers it, even if this method fails (such as when +the client does not have a Kerberos token). It is thus not possible to fall back +to username/password (also known as `basic`) authentication if Kerberos +authentication fails. + +For GitLab users to be able to use either `basic` or `negotiate` authentication +with older Git versions, it is possible to offer Kerberos ticket-based +authentication on a different port (e.g. 8443) while the standard port will +keep offering only `basic` authentication. + +**For source installations with HTTPS** + +1. Edit the NGINX configuration file for GitLab + (e.g., `/etc/nginx/sites-available/gitlab-ssl`) and configure NGINX to + listen to port `8443` in addition to the standard HTTPS port: + + ```conf + server { + listen 0.0.0.0:443 ssl; + listen [::]:443 ipv6only=on ssl default_server; + listen 0.0.0.0:8443 ssl; + listen [::]:8443 ipv6only=on ssl; + ``` + +1. Update the Kerberos section of [gitlab.yml]: + + ```yaml + kerberos: + # Dedicated port: Git before 2.4 does not fall back to Basic authentication if Negotiate fails. + # To support both Basic and Negotiate methods with older versions of Git, configure + # nginx to proxy GitLab on an extra port (e.g. 8443) and uncomment the following lines + # to dedicate this port to Kerberos authentication. (default: false) + use_dedicated_port: true + port: 8443 + https: true + ``` + +1. [Restart GitLab] and NGINX for the changes to take effect. + +--- + +**For Omnibus package installations** + +1. Edit `/etc/gitlab/gitlab.rb`: + + ```ruby + gitlab_rails['kerberos_use_dedicated_port'] = true + gitlab_rails['kerberos_port'] = 8443 + gitlab_rails['kerberos_https'] = true + ``` + +1. [Reconfigure GitLab] for the changes to take effect. + +--- + +After this change, all Git remote URLs will have to be updated to +`https://gitlab.example.com:8443/mygroup/myproject.git` in order to use +Kerberos ticket-based authentication. + +## Upgrading from password-based to ticket-based Kerberos sign-ins + +Prior to GitLab 8.10 Enterprise Edition, users had to submit their +Kerberos username and password to GitLab when signing in. We will +remove support for password-based Kerberos sign-ins in a future +release, so we recommend that you upgrade to ticket-based sign-ins. + +Depending on your existing GitLab configuration, the 'Sign in with: +Kerberos Spnego' button may already be visible on your GitLab sign-in +page. If not, then add the settings [described above](#configuration). + +Once you have verified that the 'Kerberos Spnego' button works +without entering any passwords, you can proceed to disable +password-based Kerberos sign-ins. To do this you need only need to +remove the OmniAuth provider named `kerberos` from your `gitlab.yml` / +`gitlab.rb` file. + +**For installations from source** + +1. Edit [gitlab.yml] and remove the `- { name: 'kerberos' }` line under omniauth + providers: + + ```yaml + omniauth: + # ... + providers: + - { name: 'kerberos' } # <-- remove this line + ``` + +1. [Restart GitLab] for the changes to take effect. + +--- + +**For Omnibus installations** + +1. Edit `/etc/gitlab/gitlab.rb` and remove the `{ "name" => "kerberos" }` line + under `gitlab_rails['omniauth_providers']`: + + ```ruby + gitlab_rails['omniauth_providers'] = [ + { "name" => "kerberos" } # <-- remove this entry + ] + ``` + +1. [Reconfigure GitLab] for the changes to take effect. + +## Support for Active Directory Kerberos environments + +When using Kerberos ticket-based authentication in an Active Directory domain, +it may be necessary to increase the maximum header size allowed by NGINX, +as extensions to the Kerberos protocol may result in HTTP authentication headers +larger than the default size of 8kB. Configure `large_client_header_buffers` +to a larger value in [the NGINX configuration][nginx]. + +## Troubleshooting + +### Unsupported GSSAPI mechanism + +With Kerberos SPNEGO authentication, the browser is expected to send a list of +mechanisms it supports to GitLab. If it doesn't support any of the mechanisms +GitLab supports, authentication will fail with a message like this in the log: + +``` +OmniauthKerberosSpnegoController: failed to process Negotiate/Kerberos authentication: gss_accept_sec_context did not return GSS_S_COMPLETE: An unsupported mechanism was requested Unknown error +``` + +This is usually seen when the browser is unable to contact the Kerberos server +directly. It will fall back to an unsupported mechanism known as +[`IAKERB`](https://k5wiki.kerberos.org/wiki/Projects/IAKERB), which tries to use +the GitLab server as an intermediary to the Kerberos server. + +If you're experiencing this error, ensure there is connectivity between the +client machine and the Kerberos server - this is a prerequisite! Traffic may be +blocked by a firewall, or the DNS records may be incorrect. + +Another failure mode occurs when the forward and reverse DNS records for the +GitLab server do not match. Often, Windows clients will work in this case, while +Linux clients will fail. They use reverse DNS while detecting the Kerberos +realm. If they get the wrong realm, then ordinary Kerberos mechanisms will fail, +so the client will fall back to attempting to negotiate `IAKERB`, leading to the +above error message. + +To fix this, ensure that the forward and reverse DNS for your GitLab server +match. So for instance, if you acces GitLab as `gitlab.example.com`, resolving +to IP address `1.2.3.4`, then `4.3.2.1.in-addr.arpa` must be a PTR record for +`gitlab.example.com`. + +Finally, it's possible that the browser or client machine lack Kerberos support +completely. Ensure that the Kerberos libraries are installed and that you can +authenticate to other Kerberos services. + +### HTTP Basic: Access denied when cloning + +```sh +remote: HTTP Basic: Access denied +fatal: Authentication failed for '<KRB5 path>' +``` + +If you are using Git v2.11 or newer and see the above error when cloning, you can +set the `http.emptyAuth` Git option to `true` to fix this: + +``` +git config --global http.emptyAuth true +``` + +See also: [Git v2.11 release notes](https://github.com/git/git/blob/master/Documentation/RelNotes/2.11.0.txt#L482-L486) + +## Helpful links + +- <https://help.ubuntu.com/community/Kerberos> +- <http://blog.manula.org/2012/04/setting-up-kerberos-server-with-debian.html> +- <http://www.roguelynn.com/words/explain-like-im-5-kerberos/> + +[gitlab.yml]: https://gitlab.com/gitlab-org/gitlab-ce/blob/master/config/gitlab.yml.example +[restart gitlab]: ../administration/restart_gitlab.md#installations-from-source +[reconfigure gitlab]: ../administration/restart_gitlab.md#omnibus-gitlab-reconfigure +[nginx]: http://nginx.org/en/docs/http/ngx_http_core_module.html#large_client_header_buffers +[kerb]: https://web.mit.edu/kerberos/ +[mit]: http://web.mit.edu/ +[why-kerb]: http://web.mit.edu/sipb/doc/working/guide/guide/node20.html +[ee]: https://about.gitlab.com/pricing/ diff --git a/doc/integration/ldap/select_group_cn.png b/doc/integration/ldap/select_group_cn.png Binary files differnew file mode 100644 index 00000000000..9a8b9599018 --- /dev/null +++ b/doc/integration/ldap/select_group_cn.png diff --git a/doc/integration/ldap/select_group_cn_admin.png b/doc/integration/ldap/select_group_cn_admin.png Binary files differnew file mode 100644 index 00000000000..a9c100e54f5 --- /dev/null +++ b/doc/integration/ldap/select_group_cn_admin.png diff --git a/doc/integration/ldap/select_group_cn_engineering.png b/doc/integration/ldap/select_group_cn_engineering.png Binary files differnew file mode 100644 index 00000000000..98160222945 --- /dev/null +++ b/doc/integration/ldap/select_group_cn_engineering.png diff --git a/doc/integration/ldap/two_linked_ldap_groups.png b/doc/integration/ldap/two_linked_ldap_groups.png Binary files differnew file mode 100644 index 00000000000..836cd23b1c0 --- /dev/null +++ b/doc/integration/ldap/two_linked_ldap_groups.png diff --git a/doc/integration/merge_request_close_jira.png b/doc/integration/merge_request_close_jira.png Binary files differnew file mode 100644 index 00000000000..d161f689667 --- /dev/null +++ b/doc/integration/merge_request_close_jira.png diff --git a/doc/integration/saml.md b/doc/integration/saml.md index 8ee07a7fcdc..8a2af311a5b 100644 --- a/doc/integration/saml.md +++ b/doc/integration/saml.md @@ -185,6 +185,78 @@ tell GitLab which groups are external via the `external_groups:` element: } } ``` +## Required groups + +>**Note:** +This setting is only available on GitLab 10.2 EE and above. + +This setting works like `External Groups` setting. Just like there, your IdP has to +pass Group Information to GitLab, you have to tell GitLab where to look or the +groups SAML response, and which group membership should be requisite for logging in. +When `required_groups` is not set or it is empty, anyone with proper authentication +will be able to use the service. + +Example: + +```yaml +{ name: 'saml', + label: 'Our SAML Provider', + groups_attribute: 'Groups', + required_groups: ['Developers', 'Managers', 'Admins'], + args: { + assertion_consumer_service_url: 'https://gitlab.example.com/users/auth/saml/callback', + idp_cert_fingerprint: '43:51:43:a1:b5:fc:8b:b7:0a:3a:a9:b1:0f:66:73:a8', + idp_sso_target_url: 'https://login.example.com/idp', + issuer: 'https://gitlab.example.com', + name_identifier_format: 'urn:oasis:names:tc:SAML:2.0:nameid-format:transient' + } } +``` + +## Admin Groups + +>**Note:** +This setting is only available on GitLab 8.8 EE and above. + +This setting works very similarly to the `External Groups` setting. The requirements +are the same, your IdP needs to pass Group information to GitLab, you need to tell +GitLab where to look for the groups in the SAML response, and which group should be +considered `admin groups`. + +```yaml +{ name: 'saml', + label: 'Our SAML Provider', + groups_attribute: 'Groups', + admin_groups: ['Managers', 'Admins'], + args: { + assertion_consumer_service_url: 'https://gitlab.example.com/users/auth/saml/callback', + idp_cert_fingerprint: '43:51:43:a1:b5:fc:8b:b7:0a:3a:a9:b1:0f:66:73:a8', + idp_sso_target_url: 'https://login.example.com/idp', + issuer: 'https://gitlab.example.com', + name_identifier_format: 'urn:oasis:names:tc:SAML:2.0:nameid-format:transient' + } } +``` + +## Auditor Groups + +>**Note:** +This setting is only available on GitLab 11.4 EE and above. + +This setting also follows the requirements documented for the `External Groups` setting. GitLab uses the Group information provided by your IdP to determine if a user should be assigned the `auditor` role. + +```yaml +{ name: 'saml', + label: 'Our SAML Provider', + groups_attribute: 'Groups', + auditor_groups: ['Auditors', 'Security'], + args: { + assertion_consumer_service_url: 'https://gitlab.example.com/users/auth/saml/callback', + idp_cert_fingerprint: '43:51:43:a1:b5:fc:8b:b7:0a:3a:a9:b1:0f:66:73:a8', + idp_sso_target_url: 'https://login.example.com/idp', + issuer: 'https://gitlab.example.com', + name_identifier_format: 'urn:oasis:names:tc:SAML:2.0:nameid-format:transient' + } } +``` + ## Bypass two factor authentication If you want some SAML authentication methods to count as 2FA on a per session basis, you can register them in the diff --git a/doc/legal/corporate_contributor_license_agreement.md b/doc/legal/corporate_contributor_license_agreement.md index e5fc7a3c85f..7f08188bd65 100644 --- a/doc/legal/corporate_contributor_license_agreement.md +++ b/doc/legal/corporate_contributor_license_agreement.md @@ -1,3 +1,29 @@ ---- -redirect_to: 'README.md' ---- +# Corporate contributor license agreement + +You accept and agree to the following terms and conditions for Your present and future Contributions submitted to GitLab B.V.. Except for the license granted herein to GitLab B.V. and recipients of software distributed by GitLab B.V., You reserve all right, title, and interest in and to Your Contributions. + +1. Definitions. + + "You" (or "Your") shall mean the copyright owner or legal entity authorized by the copyright owner that is making this Agreement with GitLab B.V.. For legal entities, the entity making a Contribution and all other entities that control, are controlled by, or are under common control with that entity are considered to be a single Contributor. For the purposes of this definition, "control" means (i) the power, direct or indirect, to cause the direction or management of such entity, whether by contract or otherwise, or (ii) ownership of fifty percent (50%) or more of the outstanding shares, or (iii) beneficial ownership of such entity. + + "Contribution" shall mean the code, documentation or other original works of authorship, including any modifications or additions to an existing work, that is submitted by You to GitLab B.V. for inclusion in, or documentation of, any of the products owned or managed by GitLab B.V. (the "Work"). For the purposes of this definition, "submitted" means any form of electronic, verbal, or written communication sent to GitLab B.V. or its representatives, including but not limited to communication on electronic mailing lists, source code control systems, and issue tracking systems that are managed by, or on behalf of, GitLab B.V. for the purpose of discussing and improving the Work, but excluding communication that is conspicuously marked or otherwise designated in writing by You as "Not a Contribution." + +2. Grant of Copyright License. + +Subject to the terms and conditions of this Agreement, You hereby grant to GitLab B.V. and to recipients of software distributed by GitLab B.V. a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable copyright license to reproduce, prepare derivative works of, publicly display, publicly perform, sublicense, and distribute Your Contributions and such derivative works. + +3. Grant of Patent License. + +Subject to the terms and conditions of this Agreement, You hereby grant to GitLab B.V. and to recipients of software distributed by GitLab B.V. a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable (except as stated in this section) patent license to make, have made, use, offer to sell, sell, import, and otherwise transfer the Work, where such license applies only to those patent claims licensable by You that are necessarily infringed by Your Contribution(s) alone or by combination of Your Contribution(s) with the Work to which such Contribution(s) was submitted. If any entity institutes patent litigation against You or any other entity (including a cross-claim or counterclaim in a lawsuit) alleging that your Contribution, or the Work to which you have contributed, constitutes direct or contributory patent infringement, then any patent licenses granted to that entity under this Agreement for that Contribution or Work shall terminate as of the date such litigation is filed. + +4. You represent that You are legally entitled to grant the above license. You represent further that each of Your employees is authorized to submit Contributions on Your behalf, but excluding employees that are designated in writing by You as "Not authorized to submit Contributions on behalf of [name of Your corporation here]." Such designations of exclusion for unauthorized employees are to be submitted via email to legal@gitlab.com. + +5. You represent that each of Your Contributions is Your original creation (see section 7 for submissions on behalf of others). + +6. You are not expected to provide support for Your Contributions, except to the extent You desire to provide support. You may provide support for free, for a fee, or not at all. Unless required by applicable law or agreed to in writing, You provide Your Contributions on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied, including, without limitation, any warranties or conditions of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A PARTICULAR PURPOSE. + +7. Should You wish to submit work that is not Your original creation, You may submit it to GitLab B.V. separately from any Contribution, identifying the complete details of its source and of any license or other restriction (including, but not limited to, related patents, trademarks, and license agreements) of which you are personally aware, and conspicuously marking the work as "Submitted on behalf of a third-party: [named here]". + +8. It is Your responsibility to notify GitLab.com when any change is required to the list of designated employees excluded from submitting Contributions on Your behalf per Section 4. Such notification should be sent via email to legal@gitlab.com. + +This text is licensed under the [Creative Commons Attribution 3.0 License](https://creativecommons.org/licenses/by/3.0/) and the original source is the Google Open Source Programs Office. diff --git a/doc/legal/individual_contributor_license_agreement.md b/doc/legal/individual_contributor_license_agreement.md index e5fc7a3c85f..59803aea080 100644 --- a/doc/legal/individual_contributor_license_agreement.md +++ b/doc/legal/individual_contributor_license_agreement.md @@ -1,3 +1,25 @@ ---- -redirect_to: 'README.md' ---- +# Individual contributor license agreement + +You accept and agree to the following terms and conditions for Your present and future Contributions submitted to GitLab B.V.. Except for the license granted herein to GitLab B.V. and recipients of software distributed by GitLab B.V., You reserve all right, title, and interest in and to Your Contributions. + +1. Definitions. + + "You" (or "Your") shall mean the copyright owner or legal entity authorized by the copyright owner that is making this Agreement with GitLab B.V.. For legal entities, the entity making a Contribution and all other entities that control, are controlled by, or are under common control with that entity are considered to be a single Contributor. For the purposes of this definition, "control" means (i) the power, direct or indirect, to cause the direction or management of such entity, whether by contract or otherwise, or (ii) ownership of fifty percent (50%) or more of the outstanding shares, or (iii) beneficial ownership of such entity. + + "Contribution" shall mean any original work of authorship, including any modifications or additions to an existing work, that is intentionally submitted by You to GitLab B.V. for inclusion in, or documentation of, any of the products owned or managed by GitLab B.V. (the "Work"). For the purposes of this definition, "submitted" means any form of electronic, verbal, or written communication sent to GitLab B.V. or its representatives, including but not limited to communication on electronic mailing lists, source code control systems, and issue tracking systems that are managed by, or on behalf of, GitLab B.V. for the purpose of discussing and improving the Work, but excluding communication that is conspicuously marked or otherwise designated in writing by You as "Not a Contribution." + +2. Grant of Copyright License. Subject to the terms and conditions of this Agreement, You hereby grant to GitLab B.V. and to recipients of software distributed by GitLab B.V. a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable copyright license to reproduce, prepare derivative works of, publicly display, publicly perform, sublicense, and distribute Your Contributions and such derivative works. + +3. Grant of Patent License. Subject to the terms and conditions of this Agreement, You hereby grant to GitLab B.V. and to recipients of software distributed by GitLab B.V. a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable (except as stated in this section) patent license to make, have made, use, offer to sell, sell, import, and otherwise transfer the Work, where such license applies only to those patent claims licensable by You that are necessarily infringed by Your Contribution(s) alone or by combination of Your Contribution(s) with the Work to which such Contribution(s) was submitted. If any entity institutes patent litigation against You or any other entity (including a cross-claim or counterclaim in a lawsuit) alleging that your Contribution, or the Work to which you have contributed, constitutes direct or contributory patent infringement, then any patent licenses granted to that entity under this Agreement for that Contribution or Work shall terminate as of the date such litigation is filed. + +4. You represent that you are legally entitled to grant the above license. If your employer(s) has rights to intellectual property that you create that includes your Contributions, you represent that you have received permission to make Contributions on behalf of that employer, that your employer has waived such rights for your Contributions to GitLab B.V., or that your employer has executed a separate Corporate CLA with GitLab B.V.. + +5. You represent that each of Your Contributions is Your original creation (see section 7 for submissions on behalf of others). You represent that Your Contribution submissions include complete details of any third-party license or other restriction (including, but not limited to, related patents and trademarks) of which you are personally aware and which are associated with any part of Your Contributions. + +6. You are not expected to provide support for Your Contributions, except to the extent You desire to provide support. You may provide support for free, for a fee, or not at all. Unless required by applicable law or agreed to in writing, You provide Your Contributions on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied, including, without limitation, any warranties or conditions of TITLE, NON- INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A PARTICULAR PURPOSE. + +7. Should You wish to submit work that is not Your original creation, You may submit it to GitLab B.V. separately from any Contribution, identifying the complete details of its source and of any license or other restriction (including, but not limited to, related patents, trademarks, and license agreements) of which you are personally aware, and conspicuously marking the work as "Submitted on behalf of a third-party: [insert_name_here]". + +8. You agree to notify GitLab B.V. of any facts or circumstances of which you become aware that would make these representations inaccurate in any respect. + +This text is licensed under the [Creative Commons Attribution 3.0 License](https://creativecommons.org/licenses/by/3.0/) and the original source is the Google Open Source Programs Office. diff --git a/doc/license/README.md b/doc/license/README.md new file mode 100644 index 00000000000..e3c1a658546 --- /dev/null +++ b/doc/license/README.md @@ -0,0 +1,5 @@ +--- +redirect_to: '../user/admin_area/license.md' +--- + +This document was moved to [user/admin_area/license](../user/admin_area/license.md). diff --git a/doc/policy/maintenance.md b/doc/policy/maintenance.md index 1d656574acd..c3e61b80ee8 100644 --- a/doc/policy/maintenance.md +++ b/doc/policy/maintenance.md @@ -82,4 +82,4 @@ Please see the table below for some examples: More information about the release procedures can be found in our [release documentation](https://gitlab.com/gitlab-org/release/docs). You may also want to read our -[Responsible Disclosure Policy](https://about.gitlab.com/security/disclosure/). +[Responsible Disclosure Policy](https://about.gitlab.com/disclosure/). diff --git a/doc/push_rules/push_rules.md b/doc/push_rules/push_rules.md new file mode 100644 index 00000000000..7654023f266 --- /dev/null +++ b/doc/push_rules/push_rules.md @@ -0,0 +1,156 @@ +# Push Rules **[STARTER]** + +Gain additional control over pushes to your repository. + +## Overview + +GitLab already offers [protected branches][protected-branches], but there are +cases when you need some specific rules like preventing git tag removal or +enforcing a special format for commit messages. + +Push rules are essentially [pre-receive Git hooks][hooks] that are easy to +enable in a user-friendly interface. They are defined globally if you are an +admin or per project so you can have different rules applied to different +projects depending on your needs. + +## Use cases + +Every push rule could have its own use case, but let's consider some examples. + +### Commit messages with a specific reference + +Let's assume you have the following requirements for your workflow: + +- every commit should reference a JIRA issue, for example: `Refactored css. Fixes JIRA-123.` +- users should not be able to remove git tags with `git push` + +All you need to do is write a simple regular expression that requires the mention +of a JIRA issue in the commit message, like `JIRA\-\d+`. + +Now when a user tries to push a commit with a message `Bugfix`, their push will +be declined. Only pushing commits with messages like `Bugfix according to JIRA-123` +will be accepted. + +### Restrict branch names + +Let's assume there's a strictly policy for branch names in your company, and +you want the branches to start with a certain name because you have different +GitLab CI jobs (`feature`, `hotfix`, `docker`, `android`, etc.) that rely on the +branch name. + +Your developers however, don't always remember that policy, so they push +various branches and CI pipelines do not work as expected. By restricting the +branch names globally in Push Rules, you can now sleep without the anxiety +of your developers' mistakes. Every branch that doesn't match your push rule +will get rejected. + +## Enabling push rules + +>**Note:** +GitLab administrators can set push rules globally under +**Admin area > Push Rules** that all new projects will inherit. You can later +override them in a project's settings. + +1. Navigate to your project's **Settings > Repository** and expand **Push Rules** +1. Set the rule you want +1. Click **Save Push Rules** for the changes to take effect + +The following options are available. + +| Push rule | GitLab version | Description | +| --------- | :------------: | ----------- | +| Removal of tags with `git push` | **Starter** 7.10 | Forbid users to remove git tags with `git push`. Tags will still be able to be deleted through the web UI. | +| Check whether author is a GitLab user | **Starter** 7.10 | Restrict commits by author (email) to existing GitLab users. | +| Check whether committer is the current authenticated user | **Premium** 10.2 | GitLab will reject any commit that was not committed by the current authenticated user | +| Check whether commit is signed through GPG | **Premium** 10.1 | Reject commit when it is not signed through GPG. Read [signing commits with GPG][signing-commits]. | +| Prevent committing secrets to Git | **Starter** 8.12 | GitLab will reject any files that are likely to contain secrets. Read [what files are forbidden](#prevent-pushing-secrets-to-the-repository). | +| Restrict by commit message | **Starter** 7.10 | Only commit messages that match this regular expression are allowed to be pushed. Leave empty to allow any commit message. Uses multiline mode, which can be disabled using `(?-m)`. | +| Restrict by commit message (negative match)| **Starter** 11.1 | Only commit messages that do not match this regular expression are allowed to be pushed. Leave empty to allow any commit message. Uses multiline mode, which can be disabled using `(?-m)`. | +| Restrict by branch name | **Starter** 9.3 | Only branch names that match this regular expression are allowed to be pushed. Leave empty to allow any branch name. | +| Restrict by commit author's email | **Starter** 7.10 | Only commit author's email that match this regular expression are allowed to be pushed. Leave empty to allow any email. | +| Prohibited file names | **Starter** 7.10 | Any committed filenames that match this regular expression are not allowed to be pushed. Leave empty to allow any filenames. | +| Maximum file size | **Starter** 7.12 | Pushes that contain added or updated files that exceed this file size (in MB) are rejected. Set to 0 to allow files of any size. | + +>**Tip:** +GitLab uses [RE2 syntax](https://github.com/google/re2/wiki/Syntax) for regular expressions in push rules. You can check your regular expressions at <https://regex-golang.appspot.com>. + +## Prevent pushing secrets to the repository + +> [Introduced][ee-385] in [GitLab Starter][ee] 8.12. + +You can turn on a predefined blacklist of files which won't be allowed to be +pushed to a repository. + +By selecting the checkbox *Prevent committing secrets to Git*, GitLab prevents +pushes to the repository when a file matches a regular expression as read from +[`files_blacklist.yml`][list] (make sure you are at the right branch +as your GitLab version when viewing this file). + +NOTE: **Note**: +Files already committed won't get restricted by this push rule. + +Below is an example list of what will be rejected by these regular expressions: + +```shell +##################### +# AWS CLI credential blobs +##################### +.aws/credentials +aws/credentials +homefolder/aws/credentials + +##################### +# Private RSA SSH keys +##################### +/ssh/id_rsa +/.ssh/personal_rsa +/config/server_rsa +id_rsa +.id_rsa + +##################### +# Private DSA SSH keys +##################### +/ssh/id_dsa +/.ssh/personal_dsa +/config/server_dsa +id_dsa +.id_dsa + +##################### +# Private ed25519 SSH keys +##################### +/ssh/id_ed25519 +/.ssh/personal_ed25519 +/config/server_ed25519 +id_ed25519 +.id_ed25519 + +##################### +# Private ECDSA SSH keys +##################### +/ssh/id_ecdsa +/.ssh/personal_ecdsa +/config/server_ecdsa +id_ecdsa +.id_ecdsa + +##################### +# Any file with .pem or .key extensions +##################### +*.pem +*.key + +##################### +# Any file ending with _history or .history extension +##################### +pry.history +bash_history +``` + +[protected-branches]: ../user/project/protected_branches.md +[signing-commits]: ../user/project/repository/gpg_signed_commits/index.md +[ee-385]: https://gitlab.com/gitlab-org/gitlab-ee/issues/385 +[list]: https://gitlab.com/gitlab-org/gitlab-ee/blob/master/ee/lib/gitlab/checks/files_blacklist.yml +[hooks]: https://git-scm.com/book/en/v2/Customizing-Git-Git-Hooks +[ee]: https://about.gitlab.com/pricing/ diff --git a/doc/raketasks/cleanup.md b/doc/raketasks/cleanup.md index e70a009323e..f5c788af578 100644 --- a/doc/raketasks/cleanup.md +++ b/doc/raketasks/cleanup.md @@ -23,6 +23,16 @@ sudo gitlab-rake gitlab:cleanup:repos bundle exec rake gitlab:cleanup:repos RAILS_ENV=production ``` +Remove old repository copies from repositories moved to another storage. + +``` +# omnibus-gitlab +sudo gitlab-rake gitlab:cleanup:moved + +# installation from source +bundle exec rake gitlab:cleanup:moved RAILS_ENV=production +``` + Clean up local project upload files if they don't exist in GitLab database. The task attempts to fix the file if it can find its project, otherwise it moves the file to a lost and found directory. diff --git a/doc/tools/email.md b/doc/tools/email.md new file mode 100644 index 00000000000..ab39206ffa4 --- /dev/null +++ b/doc/tools/email.md @@ -0,0 +1,38 @@ +# Email from GitLab **[STARTER ONLY]** + +As a GitLab administrator you can email GitLab users from within GitLab. + +## Overview + +GitLab provides a simple tool to email all users or users of a chosen group or +project right from the admin area. Users will receive the email to their primary +email address. + +## Use-cases + +- Notify your users about a new project, a new feature, or a new product launch. +- Notify your users about a new deployment, or that will be downtime expected + for a particular reason. + +## Sending emails to users from within GitLab + +1. Go to the admin area using the wrench icon in the top right corner and + navigate to **Overview > Users > Send email to users**. + +  + +1. Compose an email and choose where it will be sent (all users or users of a + chosen group or project): + +  + +## Unsubscribing from emails + +User can choose to unsubscribe from receiving emails from GitLab by following +the unsubscribe link from the email. Unsubscribing is unauthenticated in order +to keep the simplicity of this feature. + +On unsubscribe, user will receive an email notifying that unsubscribe happened. +The endpoint that provides the unsubscribe option is rate-limited. + +[ee]: https://about.gitlab.com/pricing/ diff --git a/doc/tools/email1.png b/doc/tools/email1.png Binary files differnew file mode 100644 index 00000000000..e79ccc3e9a9 --- /dev/null +++ b/doc/tools/email1.png diff --git a/doc/tools/email2.png b/doc/tools/email2.png Binary files differnew file mode 100644 index 00000000000..d073c0e42da --- /dev/null +++ b/doc/tools/email2.png diff --git a/doc/topics/authentication/index.md b/doc/topics/authentication/index.md index eef26612d5b..9a181427949 100644 --- a/doc/topics/authentication/index.md +++ b/doc/topics/authentication/index.md @@ -21,7 +21,7 @@ This page gathers all the resources for the topic **Authentication** within GitL - [Enforce Two-factor Authentication (2FA)](../../security/two_factor_authentication.md#enforce-two-factor-authentication-2fa) - **Articles:** - [How to Configure LDAP with GitLab CE](../../administration/auth/how_to_configure_ldap_gitlab_ce/index.md) - - [How to Configure LDAP with GitLab EE](https://docs.gitlab.com/ee/administration/auth/how_to_configure_ldap_gitlab_ee/index.md) + - [How to Configure LDAP with GitLab EE](../../administration/auth/how_to_configure_ldap_gitlab_ee/index.md) - [Feature Highlight: LDAP Integration](https://about.gitlab.com/2014/07/10/feature-highlight-ldap-sync/) - [Debugging LDAP](https://about.gitlab.com/handbook/support/workflows/support-engineering/ldap/debugging_ldap.html) - **Integrations:** diff --git a/doc/topics/autodevops/index.md b/doc/topics/autodevops/index.md index 24fd4d70b9f..430a4f4c09f 100644 --- a/doc/topics/autodevops/index.md +++ b/doc/topics/autodevops/index.md @@ -179,7 +179,7 @@ Those environments are tied to jobs that use [Auto Deploy](#auto-deploy), so except for the environment scope, they would also need to have a different domain they would be deployed to. This is why you need to define a separate `KUBE_INGRESS_BASE_DOMAIN` variable for all the above -[based on the environment](https://docs.gitlab.com/ee/ci/variables/index.html#limiting-environment-scopes-of-variables-premium). +[based on the environment](../../ci/variables/README.md#limiting-environment-scopes-of-variables-premium). The following table is an example of how the three different clusters would be configured. @@ -348,7 +348,7 @@ created, and is uploaded as an artifact which you can later download and check out. Any differences between the source and target branches are also -[shown in the merge request widget](https://docs.gitlab.com/ee/user/project/merge_requests/code_quality.html). +[shown in the merge request widget](../../user/project/merge_requests/code_quality.md). ### Auto SAST **[ULTIMATE]** @@ -361,7 +361,7 @@ report is created, it's uploaded as an artifact which you can later download and check out. Any security warnings are also -[shown in the merge request widget](https://docs.gitlab.com/ee//user/project/merge_requests/sast.html). +[shown in the merge request widget](../../user/project/merge_requests/sast.md). NOTE: **Note:** The Auto SAST stage will be skipped on licenses other than Ultimate. @@ -380,7 +380,7 @@ report is created, it's uploaded as an artifact which you can later download and check out. Any security warnings are also -[shown in the merge request widget](https://docs.gitlab.com/ee//user/project/merge_requests/dependency_scanning.html). +[shown in the merge request widget](../../user/project/merge_requests/dependency_scanning.md). NOTE: **Note:** The Auto Dependency Scanning stage will be skipped on licenses other than Ultimate. @@ -399,7 +399,7 @@ report is created, it's uploaded as an artifact which you can later download and check out. Any licenses are also -[shown in the merge request widget](https://docs.gitlab.com/ee//user/project/merge_requests/license_management.html). +[shown in the merge request widget](../../user/project/merge_requests/license_management.md). NOTE: **Note:** The Auto License Management stage will be skipped on licenses other than Ultimate. diff --git a/doc/university/high-availability/aws/img/reference-arch.png b/doc/university/high-availability/aws/img/reference-arch.png Binary files differnew file mode 100644 index 00000000000..63e571989e5 --- /dev/null +++ b/doc/university/high-availability/aws/img/reference-arch.png diff --git a/doc/university/high-availability/aws/img/reference-arch2.png b/doc/university/high-availability/aws/img/reference-arch2.png Binary files differindex a9cb6663103..0f8790d0f74 100644 --- a/doc/university/high-availability/aws/img/reference-arch2.png +++ b/doc/university/high-availability/aws/img/reference-arch2.png diff --git a/doc/update/10.0-ce-to-ee.md b/doc/update/10.0-ce-to-ee.md new file mode 100644 index 00000000000..10c9e21fa81 --- /dev/null +++ b/doc/update/10.0-ce-to-ee.md @@ -0,0 +1,5 @@ +--- +redirect_to: upgrading_from_ce_to_ee.md +--- + +This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/10.0-to-10.1.md b/doc/update/10.0-to-10.1.md index 8514aa13f48..10c9e21fa81 100644 --- a/doc/update/10.0-to-10.1.md +++ b/doc/update/10.0-to-10.1.md @@ -1,5 +1,5 @@ --- -redirect_to: upgrading_from_source.md +redirect_to: upgrading_from_ce_to_ee.md --- -This document was moved to [another location](upgrading_from_source.md). +This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/10.1-ce-to-ee.md b/doc/update/10.1-ce-to-ee.md new file mode 100644 index 00000000000..10c9e21fa81 --- /dev/null +++ b/doc/update/10.1-ce-to-ee.md @@ -0,0 +1,5 @@ +--- +redirect_to: upgrading_from_ce_to_ee.md +--- + +This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/10.1-to-10.2.md b/doc/update/10.1-to-10.2.md index 8514aa13f48..10c9e21fa81 100644 --- a/doc/update/10.1-to-10.2.md +++ b/doc/update/10.1-to-10.2.md @@ -1,5 +1,5 @@ --- -redirect_to: upgrading_from_source.md +redirect_to: upgrading_from_ce_to_ee.md --- -This document was moved to [another location](upgrading_from_source.md). +This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/10.2-ce-to-ee.md b/doc/update/10.2-ce-to-ee.md new file mode 100644 index 00000000000..10c9e21fa81 --- /dev/null +++ b/doc/update/10.2-ce-to-ee.md @@ -0,0 +1,5 @@ +--- +redirect_to: upgrading_from_ce_to_ee.md +--- + +This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/10.2-to-10.3.md b/doc/update/10.2-to-10.3.md index 8514aa13f48..10c9e21fa81 100644 --- a/doc/update/10.2-to-10.3.md +++ b/doc/update/10.2-to-10.3.md @@ -1,5 +1,5 @@ --- -redirect_to: upgrading_from_source.md +redirect_to: upgrading_from_ce_to_ee.md --- -This document was moved to [another location](upgrading_from_source.md). +This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/10.3-ce-to-ee.md b/doc/update/10.3-ce-to-ee.md new file mode 100644 index 00000000000..10c9e21fa81 --- /dev/null +++ b/doc/update/10.3-ce-to-ee.md @@ -0,0 +1,5 @@ +--- +redirect_to: upgrading_from_ce_to_ee.md +--- + +This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/10.3-to-10.4.md b/doc/update/10.3-to-10.4.md index 8514aa13f48..10c9e21fa81 100644 --- a/doc/update/10.3-to-10.4.md +++ b/doc/update/10.3-to-10.4.md @@ -1,5 +1,5 @@ --- -redirect_to: upgrading_from_source.md +redirect_to: upgrading_from_ce_to_ee.md --- -This document was moved to [another location](upgrading_from_source.md). +This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/10.4-ce-to-ee.md b/doc/update/10.4-ce-to-ee.md new file mode 100644 index 00000000000..10c9e21fa81 --- /dev/null +++ b/doc/update/10.4-ce-to-ee.md @@ -0,0 +1,5 @@ +--- +redirect_to: upgrading_from_ce_to_ee.md +--- + +This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/10.4-to-10.5.md b/doc/update/10.4-to-10.5.md index 8514aa13f48..10c9e21fa81 100644 --- a/doc/update/10.4-to-10.5.md +++ b/doc/update/10.4-to-10.5.md @@ -1,5 +1,5 @@ --- -redirect_to: upgrading_from_source.md +redirect_to: upgrading_from_ce_to_ee.md --- -This document was moved to [another location](upgrading_from_source.md). +This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/10.5-ce-to-ee.md b/doc/update/10.5-ce-to-ee.md new file mode 100644 index 00000000000..10c9e21fa81 --- /dev/null +++ b/doc/update/10.5-ce-to-ee.md @@ -0,0 +1,5 @@ +--- +redirect_to: upgrading_from_ce_to_ee.md +--- + +This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/10.5-to-10.6.md b/doc/update/10.5-to-10.6.md index 8514aa13f48..10c9e21fa81 100644 --- a/doc/update/10.5-to-10.6.md +++ b/doc/update/10.5-to-10.6.md @@ -1,5 +1,5 @@ --- -redirect_to: upgrading_from_source.md +redirect_to: upgrading_from_ce_to_ee.md --- -This document was moved to [another location](upgrading_from_source.md). +This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/10.6-ce-to-ee.md b/doc/update/10.6-ce-to-ee.md new file mode 100644 index 00000000000..10c9e21fa81 --- /dev/null +++ b/doc/update/10.6-ce-to-ee.md @@ -0,0 +1,5 @@ +--- +redirect_to: upgrading_from_ce_to_ee.md +--- + +This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/10.6-to-10.7.md b/doc/update/10.6-to-10.7.md index 8514aa13f48..10c9e21fa81 100644 --- a/doc/update/10.6-to-10.7.md +++ b/doc/update/10.6-to-10.7.md @@ -1,5 +1,5 @@ --- -redirect_to: upgrading_from_source.md +redirect_to: upgrading_from_ce_to_ee.md --- -This document was moved to [another location](upgrading_from_source.md). +This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/10.7-ce-to-ee.md b/doc/update/10.7-ce-to-ee.md new file mode 100644 index 00000000000..10c9e21fa81 --- /dev/null +++ b/doc/update/10.7-ce-to-ee.md @@ -0,0 +1,5 @@ +--- +redirect_to: upgrading_from_ce_to_ee.md +--- + +This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/10.7-to-10.8.md b/doc/update/10.7-to-10.8.md index 8514aa13f48..10c9e21fa81 100644 --- a/doc/update/10.7-to-10.8.md +++ b/doc/update/10.7-to-10.8.md @@ -1,5 +1,5 @@ --- -redirect_to: upgrading_from_source.md +redirect_to: upgrading_from_ce_to_ee.md --- -This document was moved to [another location](upgrading_from_source.md). +This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/10.8-ce-to-ee.md b/doc/update/10.8-ce-to-ee.md new file mode 100644 index 00000000000..10c9e21fa81 --- /dev/null +++ b/doc/update/10.8-ce-to-ee.md @@ -0,0 +1,5 @@ +--- +redirect_to: upgrading_from_ce_to_ee.md +--- + +This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/10.8-to-11.0.md b/doc/update/10.8-to-11.0.md index 8514aa13f48..10c9e21fa81 100644 --- a/doc/update/10.8-to-11.0.md +++ b/doc/update/10.8-to-11.0.md @@ -1,5 +1,5 @@ --- -redirect_to: upgrading_from_source.md +redirect_to: upgrading_from_ce_to_ee.md --- -This document was moved to [another location](upgrading_from_source.md). +This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/11.0-ce-to-ee.md b/doc/update/11.0-ce-to-ee.md new file mode 100644 index 00000000000..10c9e21fa81 --- /dev/null +++ b/doc/update/11.0-ce-to-ee.md @@ -0,0 +1,5 @@ +--- +redirect_to: upgrading_from_ce_to_ee.md +--- + +This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/11.0-to-11.1.md b/doc/update/11.0-to-11.1.md index 8514aa13f48..10c9e21fa81 100644 --- a/doc/update/11.0-to-11.1.md +++ b/doc/update/11.0-to-11.1.md @@ -1,5 +1,5 @@ --- -redirect_to: upgrading_from_source.md +redirect_to: upgrading_from_ce_to_ee.md --- -This document was moved to [another location](upgrading_from_source.md). +This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/11.1-ce-to-ee.md b/doc/update/11.1-ce-to-ee.md new file mode 100644 index 00000000000..10c9e21fa81 --- /dev/null +++ b/doc/update/11.1-ce-to-ee.md @@ -0,0 +1,5 @@ +--- +redirect_to: upgrading_from_ce_to_ee.md +--- + +This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/11.1-to-11.2.md b/doc/update/11.1-to-11.2.md index 8514aa13f48..10c9e21fa81 100644 --- a/doc/update/11.1-to-11.2.md +++ b/doc/update/11.1-to-11.2.md @@ -1,5 +1,5 @@ --- -redirect_to: upgrading_from_source.md +redirect_to: upgrading_from_ce_to_ee.md --- -This document was moved to [another location](upgrading_from_source.md). +This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/11.2-ce-to-ee.md b/doc/update/11.2-ce-to-ee.md new file mode 100644 index 00000000000..10c9e21fa81 --- /dev/null +++ b/doc/update/11.2-ce-to-ee.md @@ -0,0 +1,5 @@ +--- +redirect_to: upgrading_from_ce_to_ee.md +--- + +This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/11.2-to-11.3.md b/doc/update/11.2-to-11.3.md index 8514aa13f48..10c9e21fa81 100644 --- a/doc/update/11.2-to-11.3.md +++ b/doc/update/11.2-to-11.3.md @@ -1,5 +1,5 @@ --- -redirect_to: upgrading_from_source.md +redirect_to: upgrading_from_ce_to_ee.md --- -This document was moved to [another location](upgrading_from_source.md). +This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/11.3-ce-to-ee.md b/doc/update/11.3-ce-to-ee.md new file mode 100644 index 00000000000..10c9e21fa81 --- /dev/null +++ b/doc/update/11.3-ce-to-ee.md @@ -0,0 +1,5 @@ +--- +redirect_to: upgrading_from_ce_to_ee.md +--- + +This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/11.3-to-11.4.md b/doc/update/11.3-to-11.4.md index 8514aa13f48..10c9e21fa81 100644 --- a/doc/update/11.3-to-11.4.md +++ b/doc/update/11.3-to-11.4.md @@ -1,5 +1,5 @@ --- -redirect_to: upgrading_from_source.md +redirect_to: upgrading_from_ce_to_ee.md --- -This document was moved to [another location](upgrading_from_source.md). +This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/11.4-ce-to-ee.md b/doc/update/11.4-ce-to-ee.md new file mode 100644 index 00000000000..10c9e21fa81 --- /dev/null +++ b/doc/update/11.4-ce-to-ee.md @@ -0,0 +1,5 @@ +--- +redirect_to: upgrading_from_ce_to_ee.md +--- + +This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/11.4-to-11.5.md b/doc/update/11.4-to-11.5.md index 8514aa13f48..10c9e21fa81 100644 --- a/doc/update/11.4-to-11.5.md +++ b/doc/update/11.4-to-11.5.md @@ -1,5 +1,5 @@ --- -redirect_to: upgrading_from_source.md +redirect_to: upgrading_from_ce_to_ee.md --- -This document was moved to [another location](upgrading_from_source.md). +This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/11.5-ce-to-ee.md b/doc/update/11.5-ce-to-ee.md new file mode 100644 index 00000000000..10c9e21fa81 --- /dev/null +++ b/doc/update/11.5-ce-to-ee.md @@ -0,0 +1,5 @@ +--- +redirect_to: upgrading_from_ce_to_ee.md +--- + +This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/11.5-to-11.6.md b/doc/update/11.5-to-11.6.md index 8514aa13f48..10c9e21fa81 100644 --- a/doc/update/11.5-to-11.6.md +++ b/doc/update/11.5-to-11.6.md @@ -1,5 +1,5 @@ --- -redirect_to: upgrading_from_source.md +redirect_to: upgrading_from_ce_to_ee.md --- -This document was moved to [another location](upgrading_from_source.md). +This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/11.6-ce-to-ee.md b/doc/update/11.6-ce-to-ee.md new file mode 100644 index 00000000000..10c9e21fa81 --- /dev/null +++ b/doc/update/11.6-ce-to-ee.md @@ -0,0 +1,5 @@ +--- +redirect_to: upgrading_from_ce_to_ee.md +--- + +This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/11.6-to-11.7.md b/doc/update/11.6-to-11.7.md index 8514aa13f48..10c9e21fa81 100644 --- a/doc/update/11.6-to-11.7.md +++ b/doc/update/11.6-to-11.7.md @@ -1,5 +1,5 @@ --- -redirect_to: upgrading_from_source.md +redirect_to: upgrading_from_ce_to_ee.md --- -This document was moved to [another location](upgrading_from_source.md). +This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/11.7-ce-to-ee.md b/doc/update/11.7-ce-to-ee.md new file mode 100644 index 00000000000..10c9e21fa81 --- /dev/null +++ b/doc/update/11.7-ce-to-ee.md @@ -0,0 +1,5 @@ +--- +redirect_to: upgrading_from_ce_to_ee.md +--- + +This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/11.7-to-11.8.md b/doc/update/11.7-to-11.8.md index 8514aa13f48..10c9e21fa81 100644 --- a/doc/update/11.7-to-11.8.md +++ b/doc/update/11.7-to-11.8.md @@ -1,5 +1,5 @@ --- -redirect_to: upgrading_from_source.md +redirect_to: upgrading_from_ce_to_ee.md --- -This document was moved to [another location](upgrading_from_source.md). +This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/11.8-ce-to-ee.md b/doc/update/11.8-ce-to-ee.md new file mode 100644 index 00000000000..10c9e21fa81 --- /dev/null +++ b/doc/update/11.8-ce-to-ee.md @@ -0,0 +1,5 @@ +--- +redirect_to: upgrading_from_ce_to_ee.md +--- + +This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/2.6-to-3.0.md b/doc/update/2.6-to-3.0.md index 8514aa13f48..10c9e21fa81 100644 --- a/doc/update/2.6-to-3.0.md +++ b/doc/update/2.6-to-3.0.md @@ -1,5 +1,5 @@ --- -redirect_to: upgrading_from_source.md +redirect_to: upgrading_from_ce_to_ee.md --- -This document was moved to [another location](upgrading_from_source.md). +This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/2.9-to-3.0.md b/doc/update/2.9-to-3.0.md index 8514aa13f48..10c9e21fa81 100644 --- a/doc/update/2.9-to-3.0.md +++ b/doc/update/2.9-to-3.0.md @@ -1,5 +1,5 @@ --- -redirect_to: upgrading_from_source.md +redirect_to: upgrading_from_ce_to_ee.md --- -This document was moved to [another location](upgrading_from_source.md). +This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/3.0-to-3.1.md b/doc/update/3.0-to-3.1.md index 8514aa13f48..10c9e21fa81 100644 --- a/doc/update/3.0-to-3.1.md +++ b/doc/update/3.0-to-3.1.md @@ -1,5 +1,5 @@ --- -redirect_to: upgrading_from_source.md +redirect_to: upgrading_from_ce_to_ee.md --- -This document was moved to [another location](upgrading_from_source.md). +This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/3.1-to-4.0.md b/doc/update/3.1-to-4.0.md index 8514aa13f48..10c9e21fa81 100644 --- a/doc/update/3.1-to-4.0.md +++ b/doc/update/3.1-to-4.0.md @@ -1,5 +1,5 @@ --- -redirect_to: upgrading_from_source.md +redirect_to: upgrading_from_ce_to_ee.md --- -This document was moved to [another location](upgrading_from_source.md). +This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/4.0-to-4.1.md b/doc/update/4.0-to-4.1.md index 8514aa13f48..10c9e21fa81 100644 --- a/doc/update/4.0-to-4.1.md +++ b/doc/update/4.0-to-4.1.md @@ -1,5 +1,5 @@ --- -redirect_to: upgrading_from_source.md +redirect_to: upgrading_from_ce_to_ee.md --- -This document was moved to [another location](upgrading_from_source.md). +This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/4.1-to-4.2.md b/doc/update/4.1-to-4.2.md index 8514aa13f48..10c9e21fa81 100644 --- a/doc/update/4.1-to-4.2.md +++ b/doc/update/4.1-to-4.2.md @@ -1,5 +1,5 @@ --- -redirect_to: upgrading_from_source.md +redirect_to: upgrading_from_ce_to_ee.md --- -This document was moved to [another location](upgrading_from_source.md). +This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/4.2-to-5.0.md b/doc/update/4.2-to-5.0.md index 8514aa13f48..10c9e21fa81 100644 --- a/doc/update/4.2-to-5.0.md +++ b/doc/update/4.2-to-5.0.md @@ -1,5 +1,5 @@ --- -redirect_to: upgrading_from_source.md +redirect_to: upgrading_from_ce_to_ee.md --- -This document was moved to [another location](upgrading_from_source.md). +This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/5.0-to-5.1.md b/doc/update/5.0-to-5.1.md index 8514aa13f48..10c9e21fa81 100644 --- a/doc/update/5.0-to-5.1.md +++ b/doc/update/5.0-to-5.1.md @@ -1,5 +1,5 @@ --- -redirect_to: upgrading_from_source.md +redirect_to: upgrading_from_ce_to_ee.md --- -This document was moved to [another location](upgrading_from_source.md). +This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/5.1-to-5.2.md b/doc/update/5.1-to-5.2.md index 8514aa13f48..10c9e21fa81 100644 --- a/doc/update/5.1-to-5.2.md +++ b/doc/update/5.1-to-5.2.md @@ -1,5 +1,5 @@ --- -redirect_to: upgrading_from_source.md +redirect_to: upgrading_from_ce_to_ee.md --- -This document was moved to [another location](upgrading_from_source.md). +This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/5.1-to-5.4.md b/doc/update/5.1-to-5.4.md index 8514aa13f48..10c9e21fa81 100644 --- a/doc/update/5.1-to-5.4.md +++ b/doc/update/5.1-to-5.4.md @@ -1,5 +1,5 @@ --- -redirect_to: upgrading_from_source.md +redirect_to: upgrading_from_ce_to_ee.md --- -This document was moved to [another location](upgrading_from_source.md). +This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/5.1-to-6.0.md b/doc/update/5.1-to-6.0.md index 8514aa13f48..10c9e21fa81 100644 --- a/doc/update/5.1-to-6.0.md +++ b/doc/update/5.1-to-6.0.md @@ -1,5 +1,5 @@ --- -redirect_to: upgrading_from_source.md +redirect_to: upgrading_from_ce_to_ee.md --- -This document was moved to [another location](upgrading_from_source.md). +This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/5.2-to-5.3.md b/doc/update/5.2-to-5.3.md index 8514aa13f48..10c9e21fa81 100644 --- a/doc/update/5.2-to-5.3.md +++ b/doc/update/5.2-to-5.3.md @@ -1,5 +1,5 @@ --- -redirect_to: upgrading_from_source.md +redirect_to: upgrading_from_ce_to_ee.md --- -This document was moved to [another location](upgrading_from_source.md). +This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/5.3-to-5.4.md b/doc/update/5.3-to-5.4.md index 8514aa13f48..10c9e21fa81 100644 --- a/doc/update/5.3-to-5.4.md +++ b/doc/update/5.3-to-5.4.md @@ -1,5 +1,5 @@ --- -redirect_to: upgrading_from_source.md +redirect_to: upgrading_from_ce_to_ee.md --- -This document was moved to [another location](upgrading_from_source.md). +This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/5.4-to-6.0.md b/doc/update/5.4-to-6.0.md index 8514aa13f48..10c9e21fa81 100644 --- a/doc/update/5.4-to-6.0.md +++ b/doc/update/5.4-to-6.0.md @@ -1,5 +1,5 @@ --- -redirect_to: upgrading_from_source.md +redirect_to: upgrading_from_ce_to_ee.md --- -This document was moved to [another location](upgrading_from_source.md). +This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/6.0-ce-to-ee.md b/doc/update/6.0-ce-to-ee.md new file mode 100644 index 00000000000..10c9e21fa81 --- /dev/null +++ b/doc/update/6.0-ce-to-ee.md @@ -0,0 +1,5 @@ +--- +redirect_to: upgrading_from_ce_to_ee.md +--- + +This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/6.0-to-6.1.md b/doc/update/6.0-to-6.1.md index 8514aa13f48..10c9e21fa81 100644 --- a/doc/update/6.0-to-6.1.md +++ b/doc/update/6.0-to-6.1.md @@ -1,5 +1,5 @@ --- -redirect_to: upgrading_from_source.md +redirect_to: upgrading_from_ce_to_ee.md --- -This document was moved to [another location](upgrading_from_source.md). +This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/6.0-to-6.2.md b/doc/update/6.0-to-6.2.md new file mode 100644 index 00000000000..10c9e21fa81 --- /dev/null +++ b/doc/update/6.0-to-6.2.md @@ -0,0 +1,5 @@ +--- +redirect_to: upgrading_from_ce_to_ee.md +--- + +This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/6.1-ce-to-ee.md b/doc/update/6.1-ce-to-ee.md new file mode 100644 index 00000000000..10c9e21fa81 --- /dev/null +++ b/doc/update/6.1-ce-to-ee.md @@ -0,0 +1,5 @@ +--- +redirect_to: upgrading_from_ce_to_ee.md +--- + +This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/6.1-to-6.2.md b/doc/update/6.1-to-6.2.md index 8514aa13f48..10c9e21fa81 100644 --- a/doc/update/6.1-to-6.2.md +++ b/doc/update/6.1-to-6.2.md @@ -1,5 +1,5 @@ --- -redirect_to: upgrading_from_source.md +redirect_to: upgrading_from_ce_to_ee.md --- -This document was moved to [another location](upgrading_from_source.md). +This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/6.2-ce-to-ee.md b/doc/update/6.2-ce-to-ee.md new file mode 100644 index 00000000000..10c9e21fa81 --- /dev/null +++ b/doc/update/6.2-ce-to-ee.md @@ -0,0 +1,5 @@ +--- +redirect_to: upgrading_from_ce_to_ee.md +--- + +This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/6.2-to-6.3.md b/doc/update/6.2-to-6.3.md index 8514aa13f48..10c9e21fa81 100644 --- a/doc/update/6.2-to-6.3.md +++ b/doc/update/6.2-to-6.3.md @@ -1,5 +1,5 @@ --- -redirect_to: upgrading_from_source.md +redirect_to: upgrading_from_ce_to_ee.md --- -This document was moved to [another location](upgrading_from_source.md). +This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/6.3-ce-to-ee.md b/doc/update/6.3-ce-to-ee.md new file mode 100644 index 00000000000..10c9e21fa81 --- /dev/null +++ b/doc/update/6.3-ce-to-ee.md @@ -0,0 +1,5 @@ +--- +redirect_to: upgrading_from_ce_to_ee.md +--- + +This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/6.3-to-6.4.md b/doc/update/6.3-to-6.4.md index 8514aa13f48..10c9e21fa81 100644 --- a/doc/update/6.3-to-6.4.md +++ b/doc/update/6.3-to-6.4.md @@ -1,5 +1,5 @@ --- -redirect_to: upgrading_from_source.md +redirect_to: upgrading_from_ce_to_ee.md --- -This document was moved to [another location](upgrading_from_source.md). +This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/6.4-ce-to-ee.md b/doc/update/6.4-ce-to-ee.md new file mode 100644 index 00000000000..10c9e21fa81 --- /dev/null +++ b/doc/update/6.4-ce-to-ee.md @@ -0,0 +1,5 @@ +--- +redirect_to: upgrading_from_ce_to_ee.md +--- + +This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/6.4-to-6.5.md b/doc/update/6.4-to-6.5.md index 8514aa13f48..10c9e21fa81 100644 --- a/doc/update/6.4-to-6.5.md +++ b/doc/update/6.4-to-6.5.md @@ -1,5 +1,5 @@ --- -redirect_to: upgrading_from_source.md +redirect_to: upgrading_from_ce_to_ee.md --- -This document was moved to [another location](upgrading_from_source.md). +This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/6.5-ce-to-ee.md b/doc/update/6.5-ce-to-ee.md new file mode 100644 index 00000000000..10c9e21fa81 --- /dev/null +++ b/doc/update/6.5-ce-to-ee.md @@ -0,0 +1,5 @@ +--- +redirect_to: upgrading_from_ce_to_ee.md +--- + +This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/6.5-to-6.6.md b/doc/update/6.5-to-6.6.md index 8514aa13f48..10c9e21fa81 100644 --- a/doc/update/6.5-to-6.6.md +++ b/doc/update/6.5-to-6.6.md @@ -1,5 +1,5 @@ --- -redirect_to: upgrading_from_source.md +redirect_to: upgrading_from_ce_to_ee.md --- -This document was moved to [another location](upgrading_from_source.md). +This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/6.6-ce-to-ee.md b/doc/update/6.6-ce-to-ee.md new file mode 100644 index 00000000000..10c9e21fa81 --- /dev/null +++ b/doc/update/6.6-ce-to-ee.md @@ -0,0 +1,5 @@ +--- +redirect_to: upgrading_from_ce_to_ee.md +--- + +This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/6.6-to-6.7.md b/doc/update/6.6-to-6.7.md index 8514aa13f48..10c9e21fa81 100644 --- a/doc/update/6.6-to-6.7.md +++ b/doc/update/6.6-to-6.7.md @@ -1,5 +1,5 @@ --- -redirect_to: upgrading_from_source.md +redirect_to: upgrading_from_ce_to_ee.md --- -This document was moved to [another location](upgrading_from_source.md). +This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/6.7-ce-to-ee.md b/doc/update/6.7-ce-to-ee.md new file mode 100644 index 00000000000..10c9e21fa81 --- /dev/null +++ b/doc/update/6.7-ce-to-ee.md @@ -0,0 +1,5 @@ +--- +redirect_to: upgrading_from_ce_to_ee.md +--- + +This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/6.7-to-6.8.md b/doc/update/6.7-to-6.8.md index 8514aa13f48..10c9e21fa81 100644 --- a/doc/update/6.7-to-6.8.md +++ b/doc/update/6.7-to-6.8.md @@ -1,5 +1,5 @@ --- -redirect_to: upgrading_from_source.md +redirect_to: upgrading_from_ce_to_ee.md --- -This document was moved to [another location](upgrading_from_source.md). +This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/6.8-ce-to-ee.md b/doc/update/6.8-ce-to-ee.md new file mode 100644 index 00000000000..10c9e21fa81 --- /dev/null +++ b/doc/update/6.8-ce-to-ee.md @@ -0,0 +1,5 @@ +--- +redirect_to: upgrading_from_ce_to_ee.md +--- + +This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/6.8-to-6.9.md b/doc/update/6.8-to-6.9.md index 8514aa13f48..10c9e21fa81 100644 --- a/doc/update/6.8-to-6.9.md +++ b/doc/update/6.8-to-6.9.md @@ -1,5 +1,5 @@ --- -redirect_to: upgrading_from_source.md +redirect_to: upgrading_from_ce_to_ee.md --- -This document was moved to [another location](upgrading_from_source.md). +This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/6.9-ce-to-ee.md b/doc/update/6.9-ce-to-ee.md new file mode 100644 index 00000000000..10c9e21fa81 --- /dev/null +++ b/doc/update/6.9-ce-to-ee.md @@ -0,0 +1,5 @@ +--- +redirect_to: upgrading_from_ce_to_ee.md +--- + +This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/6.9-to-7.0.md b/doc/update/6.9-to-7.0.md index 8514aa13f48..10c9e21fa81 100644 --- a/doc/update/6.9-to-7.0.md +++ b/doc/update/6.9-to-7.0.md @@ -1,5 +1,5 @@ --- -redirect_to: upgrading_from_source.md +redirect_to: upgrading_from_ce_to_ee.md --- -This document was moved to [another location](upgrading_from_source.md). +This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/6.x-or-7.x-to-7.14.md b/doc/update/6.x-or-7.x-to-7.14.md index 8514aa13f48..10c9e21fa81 100644 --- a/doc/update/6.x-or-7.x-to-7.14.md +++ b/doc/update/6.x-or-7.x-to-7.14.md @@ -1,5 +1,5 @@ --- -redirect_to: upgrading_from_source.md +redirect_to: upgrading_from_ce_to_ee.md --- -This document was moved to [another location](upgrading_from_source.md). +This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/7.0-ce-to-ee.md b/doc/update/7.0-ce-to-ee.md new file mode 100644 index 00000000000..10c9e21fa81 --- /dev/null +++ b/doc/update/7.0-ce-to-ee.md @@ -0,0 +1,5 @@ +--- +redirect_to: upgrading_from_ce_to_ee.md +--- + +This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/7.0-to-7.1.md b/doc/update/7.0-to-7.1.md index 8514aa13f48..10c9e21fa81 100644 --- a/doc/update/7.0-to-7.1.md +++ b/doc/update/7.0-to-7.1.md @@ -1,5 +1,5 @@ --- -redirect_to: upgrading_from_source.md +redirect_to: upgrading_from_ce_to_ee.md --- -This document was moved to [another location](upgrading_from_source.md). +This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/7.1-ce-to-ee.md b/doc/update/7.1-ce-to-ee.md new file mode 100644 index 00000000000..10c9e21fa81 --- /dev/null +++ b/doc/update/7.1-ce-to-ee.md @@ -0,0 +1,5 @@ +--- +redirect_to: upgrading_from_ce_to_ee.md +--- + +This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/7.1-to-7.2.md b/doc/update/7.1-to-7.2.md index 8514aa13f48..10c9e21fa81 100644 --- a/doc/update/7.1-to-7.2.md +++ b/doc/update/7.1-to-7.2.md @@ -1,5 +1,5 @@ --- -redirect_to: upgrading_from_source.md +redirect_to: upgrading_from_ce_to_ee.md --- -This document was moved to [another location](upgrading_from_source.md). +This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/7.10-ce-to-ee.md b/doc/update/7.10-ce-to-ee.md new file mode 100644 index 00000000000..10c9e21fa81 --- /dev/null +++ b/doc/update/7.10-ce-to-ee.md @@ -0,0 +1,5 @@ +--- +redirect_to: upgrading_from_ce_to_ee.md +--- + +This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/7.10-to-7.11.md b/doc/update/7.10-to-7.11.md index 8514aa13f48..10c9e21fa81 100644 --- a/doc/update/7.10-to-7.11.md +++ b/doc/update/7.10-to-7.11.md @@ -1,5 +1,5 @@ --- -redirect_to: upgrading_from_source.md +redirect_to: upgrading_from_ce_to_ee.md --- -This document was moved to [another location](upgrading_from_source.md). +This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/7.11-ce-to-ee.md b/doc/update/7.11-ce-to-ee.md new file mode 100644 index 00000000000..10c9e21fa81 --- /dev/null +++ b/doc/update/7.11-ce-to-ee.md @@ -0,0 +1,5 @@ +--- +redirect_to: upgrading_from_ce_to_ee.md +--- + +This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/7.11-to-7.12.md b/doc/update/7.11-to-7.12.md index 8514aa13f48..10c9e21fa81 100644 --- a/doc/update/7.11-to-7.12.md +++ b/doc/update/7.11-to-7.12.md @@ -1,5 +1,5 @@ --- -redirect_to: upgrading_from_source.md +redirect_to: upgrading_from_ce_to_ee.md --- -This document was moved to [another location](upgrading_from_source.md). +This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/7.12-ce-to-ee.md b/doc/update/7.12-ce-to-ee.md new file mode 100644 index 00000000000..10c9e21fa81 --- /dev/null +++ b/doc/update/7.12-ce-to-ee.md @@ -0,0 +1,5 @@ +--- +redirect_to: upgrading_from_ce_to_ee.md +--- + +This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/7.12-to-7.13.md b/doc/update/7.12-to-7.13.md index 8514aa13f48..10c9e21fa81 100644 --- a/doc/update/7.12-to-7.13.md +++ b/doc/update/7.12-to-7.13.md @@ -1,5 +1,5 @@ --- -redirect_to: upgrading_from_source.md +redirect_to: upgrading_from_ce_to_ee.md --- -This document was moved to [another location](upgrading_from_source.md). +This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/7.13-ce-to-ee.md b/doc/update/7.13-ce-to-ee.md new file mode 100644 index 00000000000..10c9e21fa81 --- /dev/null +++ b/doc/update/7.13-ce-to-ee.md @@ -0,0 +1,5 @@ +--- +redirect_to: upgrading_from_ce_to_ee.md +--- + +This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/7.13-to-7.14.md b/doc/update/7.13-to-7.14.md index 8514aa13f48..10c9e21fa81 100644 --- a/doc/update/7.13-to-7.14.md +++ b/doc/update/7.13-to-7.14.md @@ -1,5 +1,5 @@ --- -redirect_to: upgrading_from_source.md +redirect_to: upgrading_from_ce_to_ee.md --- -This document was moved to [another location](upgrading_from_source.md). +This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/7.14-ce-to-ee.md b/doc/update/7.14-ce-to-ee.md new file mode 100644 index 00000000000..10c9e21fa81 --- /dev/null +++ b/doc/update/7.14-ce-to-ee.md @@ -0,0 +1,5 @@ +--- +redirect_to: upgrading_from_ce_to_ee.md +--- + +This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/7.14-to-8.0.md b/doc/update/7.14-to-8.0.md index 8514aa13f48..10c9e21fa81 100644 --- a/doc/update/7.14-to-8.0.md +++ b/doc/update/7.14-to-8.0.md @@ -1,5 +1,5 @@ --- -redirect_to: upgrading_from_source.md +redirect_to: upgrading_from_ce_to_ee.md --- -This document was moved to [another location](upgrading_from_source.md). +This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/7.2-to-7.3.md b/doc/update/7.2-to-7.3.md index 8514aa13f48..10c9e21fa81 100644 --- a/doc/update/7.2-to-7.3.md +++ b/doc/update/7.2-to-7.3.md @@ -1,5 +1,5 @@ --- -redirect_to: upgrading_from_source.md +redirect_to: upgrading_from_ce_to_ee.md --- -This document was moved to [another location](upgrading_from_source.md). +This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/7.3-ce-to-ee.md b/doc/update/7.3-ce-to-ee.md new file mode 100644 index 00000000000..10c9e21fa81 --- /dev/null +++ b/doc/update/7.3-ce-to-ee.md @@ -0,0 +1,5 @@ +--- +redirect_to: upgrading_from_ce_to_ee.md +--- + +This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/7.3-to-7.4.md b/doc/update/7.3-to-7.4.md index 8514aa13f48..10c9e21fa81 100644 --- a/doc/update/7.3-to-7.4.md +++ b/doc/update/7.3-to-7.4.md @@ -1,5 +1,5 @@ --- -redirect_to: upgrading_from_source.md +redirect_to: upgrading_from_ce_to_ee.md --- -This document was moved to [another location](upgrading_from_source.md). +This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/7.4-ce-to-ee.md b/doc/update/7.4-ce-to-ee.md new file mode 100644 index 00000000000..10c9e21fa81 --- /dev/null +++ b/doc/update/7.4-ce-to-ee.md @@ -0,0 +1,5 @@ +--- +redirect_to: upgrading_from_ce_to_ee.md +--- + +This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/7.4-to-7.5.md b/doc/update/7.4-to-7.5.md index 8514aa13f48..10c9e21fa81 100644 --- a/doc/update/7.4-to-7.5.md +++ b/doc/update/7.4-to-7.5.md @@ -1,5 +1,5 @@ --- -redirect_to: upgrading_from_source.md +redirect_to: upgrading_from_ce_to_ee.md --- -This document was moved to [another location](upgrading_from_source.md). +This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/7.5-ce-to-ee.md b/doc/update/7.5-ce-to-ee.md new file mode 100644 index 00000000000..10c9e21fa81 --- /dev/null +++ b/doc/update/7.5-ce-to-ee.md @@ -0,0 +1,5 @@ +--- +redirect_to: upgrading_from_ce_to_ee.md +--- + +This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/7.5-to-7.6.md b/doc/update/7.5-to-7.6.md index 8514aa13f48..10c9e21fa81 100644 --- a/doc/update/7.5-to-7.6.md +++ b/doc/update/7.5-to-7.6.md @@ -1,5 +1,5 @@ --- -redirect_to: upgrading_from_source.md +redirect_to: upgrading_from_ce_to_ee.md --- -This document was moved to [another location](upgrading_from_source.md). +This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/7.6-ce-to-ee.md b/doc/update/7.6-ce-to-ee.md new file mode 100644 index 00000000000..10c9e21fa81 --- /dev/null +++ b/doc/update/7.6-ce-to-ee.md @@ -0,0 +1,5 @@ +--- +redirect_to: upgrading_from_ce_to_ee.md +--- + +This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/7.6-to-7.7.md b/doc/update/7.6-to-7.7.md index 8514aa13f48..10c9e21fa81 100644 --- a/doc/update/7.6-to-7.7.md +++ b/doc/update/7.6-to-7.7.md @@ -1,5 +1,5 @@ --- -redirect_to: upgrading_from_source.md +redirect_to: upgrading_from_ce_to_ee.md --- -This document was moved to [another location](upgrading_from_source.md). +This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/7.7-ce-to-ee.md b/doc/update/7.7-ce-to-ee.md new file mode 100644 index 00000000000..10c9e21fa81 --- /dev/null +++ b/doc/update/7.7-ce-to-ee.md @@ -0,0 +1,5 @@ +--- +redirect_to: upgrading_from_ce_to_ee.md +--- + +This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/7.7-to-7.8.md b/doc/update/7.7-to-7.8.md index 8514aa13f48..10c9e21fa81 100644 --- a/doc/update/7.7-to-7.8.md +++ b/doc/update/7.7-to-7.8.md @@ -1,5 +1,5 @@ --- -redirect_to: upgrading_from_source.md +redirect_to: upgrading_from_ce_to_ee.md --- -This document was moved to [another location](upgrading_from_source.md). +This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/7.8-ce-to-ee.md b/doc/update/7.8-ce-to-ee.md new file mode 100644 index 00000000000..10c9e21fa81 --- /dev/null +++ b/doc/update/7.8-ce-to-ee.md @@ -0,0 +1,5 @@ +--- +redirect_to: upgrading_from_ce_to_ee.md +--- + +This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/7.8-to-7.9.md b/doc/update/7.8-to-7.9.md index 8514aa13f48..10c9e21fa81 100644 --- a/doc/update/7.8-to-7.9.md +++ b/doc/update/7.8-to-7.9.md @@ -1,5 +1,5 @@ --- -redirect_to: upgrading_from_source.md +redirect_to: upgrading_from_ce_to_ee.md --- -This document was moved to [another location](upgrading_from_source.md). +This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/7.9-ce-to-ee.md b/doc/update/7.9-ce-to-ee.md new file mode 100644 index 00000000000..10c9e21fa81 --- /dev/null +++ b/doc/update/7.9-ce-to-ee.md @@ -0,0 +1,5 @@ +--- +redirect_to: upgrading_from_ce_to_ee.md +--- + +This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/7.9-to-7.10.md b/doc/update/7.9-to-7.10.md index 8514aa13f48..10c9e21fa81 100644 --- a/doc/update/7.9-to-7.10.md +++ b/doc/update/7.9-to-7.10.md @@ -1,5 +1,5 @@ --- -redirect_to: upgrading_from_source.md +redirect_to: upgrading_from_ce_to_ee.md --- -This document was moved to [another location](upgrading_from_source.md). +This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/8.0-ce-to-ee.md b/doc/update/8.0-ce-to-ee.md new file mode 100644 index 00000000000..10c9e21fa81 --- /dev/null +++ b/doc/update/8.0-ce-to-ee.md @@ -0,0 +1,5 @@ +--- +redirect_to: upgrading_from_ce_to_ee.md +--- + +This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/8.0-to-8.1.md b/doc/update/8.0-to-8.1.md index 8514aa13f48..10c9e21fa81 100644 --- a/doc/update/8.0-to-8.1.md +++ b/doc/update/8.0-to-8.1.md @@ -1,5 +1,5 @@ --- -redirect_to: upgrading_from_source.md +redirect_to: upgrading_from_ce_to_ee.md --- -This document was moved to [another location](upgrading_from_source.md). +This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/8.1-ce-to-ee.md b/doc/update/8.1-ce-to-ee.md new file mode 100644 index 00000000000..10c9e21fa81 --- /dev/null +++ b/doc/update/8.1-ce-to-ee.md @@ -0,0 +1,5 @@ +--- +redirect_to: upgrading_from_ce_to_ee.md +--- + +This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/8.1-to-8.2.md b/doc/update/8.1-to-8.2.md index 8514aa13f48..10c9e21fa81 100644 --- a/doc/update/8.1-to-8.2.md +++ b/doc/update/8.1-to-8.2.md @@ -1,5 +1,5 @@ --- -redirect_to: upgrading_from_source.md +redirect_to: upgrading_from_ce_to_ee.md --- -This document was moved to [another location](upgrading_from_source.md). +This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/8.10-ce-to-ee.md b/doc/update/8.10-ce-to-ee.md new file mode 100644 index 00000000000..10c9e21fa81 --- /dev/null +++ b/doc/update/8.10-ce-to-ee.md @@ -0,0 +1,5 @@ +--- +redirect_to: upgrading_from_ce_to_ee.md +--- + +This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/8.10-to-8.11.md b/doc/update/8.10-to-8.11.md index 8514aa13f48..10c9e21fa81 100644 --- a/doc/update/8.10-to-8.11.md +++ b/doc/update/8.10-to-8.11.md @@ -1,5 +1,5 @@ --- -redirect_to: upgrading_from_source.md +redirect_to: upgrading_from_ce_to_ee.md --- -This document was moved to [another location](upgrading_from_source.md). +This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/8.11-ce-to-ee.md b/doc/update/8.11-ce-to-ee.md new file mode 100644 index 00000000000..10c9e21fa81 --- /dev/null +++ b/doc/update/8.11-ce-to-ee.md @@ -0,0 +1,5 @@ +--- +redirect_to: upgrading_from_ce_to_ee.md +--- + +This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/8.11-to-8.12.md b/doc/update/8.11-to-8.12.md index 8514aa13f48..10c9e21fa81 100644 --- a/doc/update/8.11-to-8.12.md +++ b/doc/update/8.11-to-8.12.md @@ -1,5 +1,5 @@ --- -redirect_to: upgrading_from_source.md +redirect_to: upgrading_from_ce_to_ee.md --- -This document was moved to [another location](upgrading_from_source.md). +This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/8.12-ce-to-ee.md b/doc/update/8.12-ce-to-ee.md new file mode 100644 index 00000000000..10c9e21fa81 --- /dev/null +++ b/doc/update/8.12-ce-to-ee.md @@ -0,0 +1,5 @@ +--- +redirect_to: upgrading_from_ce_to_ee.md +--- + +This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/8.12-to-8.13.md b/doc/update/8.12-to-8.13.md index 8514aa13f48..10c9e21fa81 100644 --- a/doc/update/8.12-to-8.13.md +++ b/doc/update/8.12-to-8.13.md @@ -1,5 +1,5 @@ --- -redirect_to: upgrading_from_source.md +redirect_to: upgrading_from_ce_to_ee.md --- -This document was moved to [another location](upgrading_from_source.md). +This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/8.13-ce-to-ee.md b/doc/update/8.13-ce-to-ee.md new file mode 100644 index 00000000000..10c9e21fa81 --- /dev/null +++ b/doc/update/8.13-ce-to-ee.md @@ -0,0 +1,5 @@ +--- +redirect_to: upgrading_from_ce_to_ee.md +--- + +This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/8.13-to-8.14.md b/doc/update/8.13-to-8.14.md index 8514aa13f48..10c9e21fa81 100644 --- a/doc/update/8.13-to-8.14.md +++ b/doc/update/8.13-to-8.14.md @@ -1,5 +1,5 @@ --- -redirect_to: upgrading_from_source.md +redirect_to: upgrading_from_ce_to_ee.md --- -This document was moved to [another location](upgrading_from_source.md). +This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/8.14-ce-to-ee.md b/doc/update/8.14-ce-to-ee.md new file mode 100644 index 00000000000..10c9e21fa81 --- /dev/null +++ b/doc/update/8.14-ce-to-ee.md @@ -0,0 +1,5 @@ +--- +redirect_to: upgrading_from_ce_to_ee.md +--- + +This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/8.14-to-8.15.md b/doc/update/8.14-to-8.15.md index 8514aa13f48..10c9e21fa81 100644 --- a/doc/update/8.14-to-8.15.md +++ b/doc/update/8.14-to-8.15.md @@ -1,5 +1,5 @@ --- -redirect_to: upgrading_from_source.md +redirect_to: upgrading_from_ce_to_ee.md --- -This document was moved to [another location](upgrading_from_source.md). +This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/8.15-ce-to-ee.md b/doc/update/8.15-ce-to-ee.md new file mode 100644 index 00000000000..10c9e21fa81 --- /dev/null +++ b/doc/update/8.15-ce-to-ee.md @@ -0,0 +1,5 @@ +--- +redirect_to: upgrading_from_ce_to_ee.md +--- + +This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/8.15-to-8.16.md b/doc/update/8.15-to-8.16.md index 8514aa13f48..10c9e21fa81 100644 --- a/doc/update/8.15-to-8.16.md +++ b/doc/update/8.15-to-8.16.md @@ -1,5 +1,5 @@ --- -redirect_to: upgrading_from_source.md +redirect_to: upgrading_from_ce_to_ee.md --- -This document was moved to [another location](upgrading_from_source.md). +This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/8.16-ce-to-ee.md b/doc/update/8.16-ce-to-ee.md new file mode 100644 index 00000000000..10c9e21fa81 --- /dev/null +++ b/doc/update/8.16-ce-to-ee.md @@ -0,0 +1,5 @@ +--- +redirect_to: upgrading_from_ce_to_ee.md +--- + +This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/8.16-to-8.17.md b/doc/update/8.16-to-8.17.md index 8514aa13f48..10c9e21fa81 100644 --- a/doc/update/8.16-to-8.17.md +++ b/doc/update/8.16-to-8.17.md @@ -1,5 +1,5 @@ --- -redirect_to: upgrading_from_source.md +redirect_to: upgrading_from_ce_to_ee.md --- -This document was moved to [another location](upgrading_from_source.md). +This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/8.17-ce-to-ee.md b/doc/update/8.17-ce-to-ee.md new file mode 100644 index 00000000000..10c9e21fa81 --- /dev/null +++ b/doc/update/8.17-ce-to-ee.md @@ -0,0 +1,5 @@ +--- +redirect_to: upgrading_from_ce_to_ee.md +--- + +This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/8.17-to-9.0.md b/doc/update/8.17-to-9.0.md index 8514aa13f48..10c9e21fa81 100644 --- a/doc/update/8.17-to-9.0.md +++ b/doc/update/8.17-to-9.0.md @@ -1,5 +1,5 @@ --- -redirect_to: upgrading_from_source.md +redirect_to: upgrading_from_ce_to_ee.md --- -This document was moved to [another location](upgrading_from_source.md). +This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/8.2-ce-to-ee.md b/doc/update/8.2-ce-to-ee.md new file mode 100644 index 00000000000..10c9e21fa81 --- /dev/null +++ b/doc/update/8.2-ce-to-ee.md @@ -0,0 +1,5 @@ +--- +redirect_to: upgrading_from_ce_to_ee.md +--- + +This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/8.2-to-8.3.md b/doc/update/8.2-to-8.3.md index 8514aa13f48..10c9e21fa81 100644 --- a/doc/update/8.2-to-8.3.md +++ b/doc/update/8.2-to-8.3.md @@ -1,5 +1,5 @@ --- -redirect_to: upgrading_from_source.md +redirect_to: upgrading_from_ce_to_ee.md --- -This document was moved to [another location](upgrading_from_source.md). +This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/8.3-ce-to-ee.md b/doc/update/8.3-ce-to-ee.md new file mode 100644 index 00000000000..10c9e21fa81 --- /dev/null +++ b/doc/update/8.3-ce-to-ee.md @@ -0,0 +1,5 @@ +--- +redirect_to: upgrading_from_ce_to_ee.md +--- + +This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/8.3-to-8.4.md b/doc/update/8.3-to-8.4.md index 8514aa13f48..10c9e21fa81 100644 --- a/doc/update/8.3-to-8.4.md +++ b/doc/update/8.3-to-8.4.md @@ -1,5 +1,5 @@ --- -redirect_to: upgrading_from_source.md +redirect_to: upgrading_from_ce_to_ee.md --- -This document was moved to [another location](upgrading_from_source.md). +This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/8.4-ce-to-ee.md b/doc/update/8.4-ce-to-ee.md new file mode 100644 index 00000000000..10c9e21fa81 --- /dev/null +++ b/doc/update/8.4-ce-to-ee.md @@ -0,0 +1,5 @@ +--- +redirect_to: upgrading_from_ce_to_ee.md +--- + +This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/8.4-to-8.5.md b/doc/update/8.4-to-8.5.md index 8514aa13f48..10c9e21fa81 100644 --- a/doc/update/8.4-to-8.5.md +++ b/doc/update/8.4-to-8.5.md @@ -1,5 +1,5 @@ --- -redirect_to: upgrading_from_source.md +redirect_to: upgrading_from_ce_to_ee.md --- -This document was moved to [another location](upgrading_from_source.md). +This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/8.5-ce-to-ee.md b/doc/update/8.5-ce-to-ee.md new file mode 100644 index 00000000000..10c9e21fa81 --- /dev/null +++ b/doc/update/8.5-ce-to-ee.md @@ -0,0 +1,5 @@ +--- +redirect_to: upgrading_from_ce_to_ee.md +--- + +This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/8.5-to-8.6.md b/doc/update/8.5-to-8.6.md index 8514aa13f48..10c9e21fa81 100644 --- a/doc/update/8.5-to-8.6.md +++ b/doc/update/8.5-to-8.6.md @@ -1,5 +1,5 @@ --- -redirect_to: upgrading_from_source.md +redirect_to: upgrading_from_ce_to_ee.md --- -This document was moved to [another location](upgrading_from_source.md). +This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/8.6-ce-to-ee.md b/doc/update/8.6-ce-to-ee.md new file mode 100644 index 00000000000..10c9e21fa81 --- /dev/null +++ b/doc/update/8.6-ce-to-ee.md @@ -0,0 +1,5 @@ +--- +redirect_to: upgrading_from_ce_to_ee.md +--- + +This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/8.6-to-8.7.md b/doc/update/8.6-to-8.7.md index 8514aa13f48..10c9e21fa81 100644 --- a/doc/update/8.6-to-8.7.md +++ b/doc/update/8.6-to-8.7.md @@ -1,5 +1,5 @@ --- -redirect_to: upgrading_from_source.md +redirect_to: upgrading_from_ce_to_ee.md --- -This document was moved to [another location](upgrading_from_source.md). +This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/8.7-ce-to-ee.md b/doc/update/8.7-ce-to-ee.md new file mode 100644 index 00000000000..10c9e21fa81 --- /dev/null +++ b/doc/update/8.7-ce-to-ee.md @@ -0,0 +1,5 @@ +--- +redirect_to: upgrading_from_ce_to_ee.md +--- + +This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/8.7-to-8.8.md b/doc/update/8.7-to-8.8.md index 8514aa13f48..10c9e21fa81 100644 --- a/doc/update/8.7-to-8.8.md +++ b/doc/update/8.7-to-8.8.md @@ -1,5 +1,5 @@ --- -redirect_to: upgrading_from_source.md +redirect_to: upgrading_from_ce_to_ee.md --- -This document was moved to [another location](upgrading_from_source.md). +This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/8.8-ce-to-ee.md b/doc/update/8.8-ce-to-ee.md new file mode 100644 index 00000000000..10c9e21fa81 --- /dev/null +++ b/doc/update/8.8-ce-to-ee.md @@ -0,0 +1,5 @@ +--- +redirect_to: upgrading_from_ce_to_ee.md +--- + +This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/8.8-to-8.9.md b/doc/update/8.8-to-8.9.md index 8514aa13f48..10c9e21fa81 100644 --- a/doc/update/8.8-to-8.9.md +++ b/doc/update/8.8-to-8.9.md @@ -1,5 +1,5 @@ --- -redirect_to: upgrading_from_source.md +redirect_to: upgrading_from_ce_to_ee.md --- -This document was moved to [another location](upgrading_from_source.md). +This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/8.9-ce-to-ee.md b/doc/update/8.9-ce-to-ee.md new file mode 100644 index 00000000000..10c9e21fa81 --- /dev/null +++ b/doc/update/8.9-ce-to-ee.md @@ -0,0 +1,5 @@ +--- +redirect_to: upgrading_from_ce_to_ee.md +--- + +This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/8.9-to-8.10.md b/doc/update/8.9-to-8.10.md index 8514aa13f48..10c9e21fa81 100644 --- a/doc/update/8.9-to-8.10.md +++ b/doc/update/8.9-to-8.10.md @@ -1,5 +1,5 @@ --- -redirect_to: upgrading_from_source.md +redirect_to: upgrading_from_ce_to_ee.md --- -This document was moved to [another location](upgrading_from_source.md). +This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/9.0-ce-to-ee.md b/doc/update/9.0-ce-to-ee.md new file mode 100644 index 00000000000..10c9e21fa81 --- /dev/null +++ b/doc/update/9.0-ce-to-ee.md @@ -0,0 +1,5 @@ +--- +redirect_to: upgrading_from_ce_to_ee.md +--- + +This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/9.0-to-9.1.md b/doc/update/9.0-to-9.1.md index 8514aa13f48..10c9e21fa81 100644 --- a/doc/update/9.0-to-9.1.md +++ b/doc/update/9.0-to-9.1.md @@ -1,5 +1,5 @@ --- -redirect_to: upgrading_from_source.md +redirect_to: upgrading_from_ce_to_ee.md --- -This document was moved to [another location](upgrading_from_source.md). +This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/9.1-ce-to-ee.md b/doc/update/9.1-ce-to-ee.md new file mode 100644 index 00000000000..10c9e21fa81 --- /dev/null +++ b/doc/update/9.1-ce-to-ee.md @@ -0,0 +1,5 @@ +--- +redirect_to: upgrading_from_ce_to_ee.md +--- + +This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/9.1-to-9.2.md b/doc/update/9.1-to-9.2.md index 8514aa13f48..10c9e21fa81 100644 --- a/doc/update/9.1-to-9.2.md +++ b/doc/update/9.1-to-9.2.md @@ -1,5 +1,5 @@ --- -redirect_to: upgrading_from_source.md +redirect_to: upgrading_from_ce_to_ee.md --- -This document was moved to [another location](upgrading_from_source.md). +This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/9.2-ce-to-ee.md b/doc/update/9.2-ce-to-ee.md new file mode 100644 index 00000000000..10c9e21fa81 --- /dev/null +++ b/doc/update/9.2-ce-to-ee.md @@ -0,0 +1,5 @@ +--- +redirect_to: upgrading_from_ce_to_ee.md +--- + +This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/9.2-to-9.3.md b/doc/update/9.2-to-9.3.md index 8514aa13f48..10c9e21fa81 100644 --- a/doc/update/9.2-to-9.3.md +++ b/doc/update/9.2-to-9.3.md @@ -1,5 +1,5 @@ --- -redirect_to: upgrading_from_source.md +redirect_to: upgrading_from_ce_to_ee.md --- -This document was moved to [another location](upgrading_from_source.md). +This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/9.3-ce-to-ee.md b/doc/update/9.3-ce-to-ee.md new file mode 100644 index 00000000000..10c9e21fa81 --- /dev/null +++ b/doc/update/9.3-ce-to-ee.md @@ -0,0 +1,5 @@ +--- +redirect_to: upgrading_from_ce_to_ee.md +--- + +This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/9.3-to-9.4.md b/doc/update/9.3-to-9.4.md index 8514aa13f48..10c9e21fa81 100644 --- a/doc/update/9.3-to-9.4.md +++ b/doc/update/9.3-to-9.4.md @@ -1,5 +1,5 @@ --- -redirect_to: upgrading_from_source.md +redirect_to: upgrading_from_ce_to_ee.md --- -This document was moved to [another location](upgrading_from_source.md). +This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/9.4-ce-to-ee.md b/doc/update/9.4-ce-to-ee.md new file mode 100644 index 00000000000..10c9e21fa81 --- /dev/null +++ b/doc/update/9.4-ce-to-ee.md @@ -0,0 +1,5 @@ +--- +redirect_to: upgrading_from_ce_to_ee.md +--- + +This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/9.4-to-9.5.md b/doc/update/9.4-to-9.5.md index 8514aa13f48..10c9e21fa81 100644 --- a/doc/update/9.4-to-9.5.md +++ b/doc/update/9.4-to-9.5.md @@ -1,5 +1,5 @@ --- -redirect_to: upgrading_from_source.md +redirect_to: upgrading_from_ce_to_ee.md --- -This document was moved to [another location](upgrading_from_source.md). +This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/9.5-ce-to-ee.md b/doc/update/9.5-ce-to-ee.md new file mode 100644 index 00000000000..10c9e21fa81 --- /dev/null +++ b/doc/update/9.5-ce-to-ee.md @@ -0,0 +1,5 @@ +--- +redirect_to: upgrading_from_ce_to_ee.md +--- + +This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/9.5-to-10.0.md b/doc/update/9.5-to-10.0.md index 8514aa13f48..10c9e21fa81 100644 --- a/doc/update/9.5-to-10.0.md +++ b/doc/update/9.5-to-10.0.md @@ -1,5 +1,5 @@ --- -redirect_to: upgrading_from_source.md +redirect_to: upgrading_from_ce_to_ee.md --- -This document was moved to [another location](upgrading_from_source.md). +This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/patch_versions.md b/doc/update/patch_versions.md index 2e8380aa5d8..385edae20eb 100644 --- a/doc/update/patch_versions.md +++ b/doc/update/patch_versions.md @@ -106,14 +106,20 @@ sudo -u git -H git checkout v$(</home/git/gitlab/GITLAB_PAGES_VERSION) sudo -u git -H make ``` -### 8. Start application +### 8. Install/Update `gitlab-elasticsearch-indexer` (optional) **[STARTER ONLY]** + +If you're interested in using GitLab's new [elasticsearch repository indexer](../integration/elasticsearch.md#elasticsearch-repository-indexer-beta) (currently in beta) +please follow the instructions on the document linked above and enable the +indexer usage in the GitLab admin settings. + +### 9. Start application ```bash sudo service gitlab start sudo service nginx restart ``` -### 9. Check application status +### 10. Check application status Check if GitLab and its environment are configured correctly: diff --git a/doc/user/admin_area/geo_nodes.md b/doc/user/admin_area/geo_nodes.md new file mode 100644 index 00000000000..262f963ce1d --- /dev/null +++ b/doc/user/admin_area/geo_nodes.md @@ -0,0 +1,58 @@ +# Geo nodes admin area **[PREMIUM ONLY]** + +For more information about setting up GitLab Geo, read the +[Geo documentation](../../gitlab-geo/README.md). + +When you're done, you can navigate to **Admin area > Geo** (`/admin/geo/nodes`). + +## Common settings + +All Geo nodes have the following settings: + +| Setting | Description | +| --------| ----------- | +| Primary | This marks a Geo Node as primary. There can be only one primary, make sure that you first add the primary node and then all the others. | +| URL | The instance's full URL, in the same way it is configured in `/etc/gitlab/gitlab.rb` (Omnibus GitLab installations) or `gitlab.yml` (source based installations). | + +The node you're reading from is indicated with a green `Current node` label, and +the primary is given a blue `Primary` label. Remember that you can only make +changes on the primary! + +## Secondary node settings + +Secondaries have a number of additional settings available: + +| Setting | Description | +|---------------------------|-------------| +| Selective synchronization | Enable Geo [selective sync](../../administration/geo/replication/configuration.md#selective-synchronization) for this secondary. | +| Repository sync capacity | Number of concurrent requests this secondary will make to the primary when backfilling repositories. | +| File sync capacity | Number of concurrent requests this secondary will make to the primary when backfilling files. | +| Alternate URL | Allows users to log in to the secondary at an alternate URL (required for OAuth) | + +## Geo backfill + +Secondaries are notified of changes to repositories and files by the primary, +and will always attempt to synchronize those changes as quickly as possible. + +Backfill is the act of populating the secondary with repositories and files that +existed *before* the secondary was added to the database. Since there may be +extremely large numbers of repositories and files, it's infeasible to attempt to +download them all at once, so GitLab places an upper limit on the concurrency of +these operations. + +How long the backfill takes is a function of the maximum concurrency, but higher +values place more strain on the primary node. From [GitLab 10.2](https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/3107), +the limits are configurable - if your primary node has lots of surplus capacity, +you can increase the values to complete backfill in a shorter time. If it's +under heavy load and backfill is reducing its availability for normal requests, +you can decrease them. + +## Multiple secondaries behind a load balancer + +Secondaries are authenticated via OAuth with the primary. For security, the +primary does not allow redirecting back to an arbitrary URL. If you want to +allow users to log in to secondaries at a common name/load balancer URL, then +this URL must be specified as the "Alternate URL" on every secondary behind it. + +Additionally, the load balancer should use sticky sessions, since users must +authenticate each first request to each secondary. diff --git a/doc/user/admin_area/img/admin_wrench.png b/doc/user/admin_area/img/admin_wrench.png Binary files differnew file mode 100644 index 00000000000..17eee143e87 --- /dev/null +++ b/doc/user/admin_area/img/admin_wrench.png diff --git a/doc/user/admin_area/img/license_admin_area.png b/doc/user/admin_area/img/license_admin_area.png Binary files differnew file mode 100644 index 00000000000..b5662b81c5e --- /dev/null +++ b/doc/user/admin_area/img/license_admin_area.png diff --git a/doc/user/admin_area/img/license_details.png b/doc/user/admin_area/img/license_details.png Binary files differnew file mode 100644 index 00000000000..1020204321f --- /dev/null +++ b/doc/user/admin_area/img/license_details.png diff --git a/doc/user/admin_area/img/license_history.png b/doc/user/admin_area/img/license_history.png Binary files differnew file mode 100644 index 00000000000..f378a52f87a --- /dev/null +++ b/doc/user/admin_area/img/license_history.png diff --git a/doc/user/admin_area/img/license_no_license_message.png b/doc/user/admin_area/img/license_no_license_message.png Binary files differnew file mode 100644 index 00000000000..87b397f7905 --- /dev/null +++ b/doc/user/admin_area/img/license_no_license_message.png diff --git a/doc/user/admin_area/img/license_upload.png b/doc/user/admin_area/img/license_upload.png Binary files differnew file mode 100644 index 00000000000..29d55175a2d --- /dev/null +++ b/doc/user/admin_area/img/license_upload.png diff --git a/doc/user/admin_area/index.md b/doc/user/admin_area/index.md index 00cea22e4e1..f627b9df119 100644 --- a/doc/user/admin_area/index.md +++ b/doc/user/admin_area/index.md @@ -14,16 +14,19 @@ Only admin users can access the Admin Area. The Admin Area is made up of the following sections: -| Section | Description | -|:------------------|:---------------------------------------------------------------------------------------------------------------------------------------------------------| -| Overview | View your GitLab Dashboard, and maintain projects, users, groups, jobs, runners, and Gitaly servers. | -| Monitoring | View GitLab system information, and information on background jobs, logs, [health checks](monitoring/health_check.md), request profiles, and audit logs. | -| Messages | Send and manage [broadcast messages](broadcast_messages.md) for your users. | -| System Hooks | Configure [system hooks](../../system_hooks/system_hooks.md) for many events. | -| Applications | Create system [OAuth applications](../../integration/oauth_provider.md) for integrations with other services. | -| Abuse Reports | Manage [abuse reports](abuse_reports.md) submitted by your users. | -| Deploy Keys | Create instance-wide [SSH deploy keys](../../ssh/README.md#deploy-keys). | -| Service Templates | Create [service templates](../project/integrations/services_templates.md) for projects. | -| Labels | Create and maintain [labels](labels.md) for your GitLab instance. | -| Appearance | Customize [GitLab's appearance](../../customization/index.md). | -| Settings | Modify the [settings](settings/index.md) for your GitLab instance. | +| Section | Description | +|:---------------------------|:---------------------------------------------------------------------------------------------------------------------------------------------------------| +| Overview | View your GitLab Dashboard, and maintain projects, users, groups, jobs, runners, and Gitaly servers. | +| Monitoring | View GitLab system information, and information on background jobs, logs, [health checks](monitoring/health_check.md), request profiles, and audit logs. | +| Messages | Send and manage [broadcast messages](broadcast_messages.md) for your users. | +| System Hooks | Configure [system hooks](../../system_hooks/system_hooks.md) for many events. | +| Applications | Create system [OAuth applications](../../integration/oauth_provider.md) for integrations with other services. | +| Abuse Reports | Manage [abuse reports](abuse_reports.md) submitted by your users. | +| License **[STARTER ONLY]** | Upload, display, and remove [licenses](license.md). | +| Push Rules **[STARTER]** | Configure pre-defined git [push rules](../../push_rules/push_rules.md) for projects. | +| Geo **[PREMIUM ONLY]** | Configure and maintain [Geo nodes](geo_nodes.md). | +| Deploy Keys | Create instance-wide [SSH deploy keys](../../ssh/README.md#deploy-keys). | +| Service Templates | Create [service templates](../project/integrations/services_templates.md) for projects. | +| Labels | Create and maintain [labels](labels.md) for your GitLab instance. | +| Appearance | Customize [GitLab's appearance](../../customization/index.md). | +| Settings | Modify the [settings](settings/index.md) for your GitLab instance. | diff --git a/doc/user/admin_area/license.md b/doc/user/admin_area/license.md new file mode 100644 index 00000000000..b81d016aa3f --- /dev/null +++ b/doc/user/admin_area/license.md @@ -0,0 +1,83 @@ +# Activate all GitLab Enterprise Edition functionality with a license **[STARTER ONLY]** + +To activate all GitLab Enterprise Edition (EE) functionality, you need to upload +a license. Once you've received your license from GitLab Inc., you can upload it +by **signing into your GitLab instance as an admin**. + +The license has the form of a base64 encoded ASCII text with a `.gitlab-license` +extension and can be obtained when you [purchase one][pricing] or when you sign +up for a [free trial]. + +NOTE: **Note:** +As of GitLab Enterprise Edition 9.4.0, a newly-installed instance without an +uploaded license will only have the Core features active. A trial license will +activate all Ultimate features, but after +[the trial expires](#what-happens-when-your-license-expires), some functionality +will be locked. + +## Uploading your license + +The very first time you visit your GitLab EE installation signed in as an admin, +you should see a note urging you to upload a license with a link that takes you +straight to the License admin area. + +Otherwise, you can: + +1. Navigate manually to the **Admin Area** by clicking the wrench icon in the menu bar. + +  + +1. And then going to the **License** tab and click on **Upload New License**. + +  + +1. If you've received a `.gitlab-license` file, you should have already downloaded + it in your local machine. You can then upload it directly by choosing the + license file and clicking the **Upload license** button. In the image below, + you can see that the selected license file is named `GitLab.gitlab-license`. + +  + + If you've received your license as plain text, you need to select the + "Enter license key" option, copy the license, paste it into the "License key" + field and click **Upload license**. + +--- + +Once the license is uploaded, all GitLab Enterprise Edition functionality +will be active until the end of the license period. When that period ends, the +instance will [fall back](#what-happens-when-your-license-expires) to Core-only +functionality. + +You can review the license details at any time in the License section of the +Admin Area. + + + +## Notification before the license expires + +One month before the license expires, a message informing when the expiration +is due to, will start appearing to GitLab admins. Make sure that you update your +license, otherwise you will miss all the paid features if it expires. + +## What happens when your license expires + +In case your license expires, GitLab will lock down some features like Git pushes, +issue creation, etc., and a message to inform of the expired license will be +presented to all admins. + +In order to get back all the previous functionality, a new license must be uploaded. +To fall back to having only the Core features active, you'll need to delete the +expired license(s). + +## License history + +It's possible to upload more than one license, but only the last one will be +taken into account. + +You can see your previous licenses' history at the bottom of the License page. + + + +[free trial]: https://about.gitlab.com/free-trial/ +[pricing]: https://about.gitlab.com/pricing/ diff --git a/doc/user/admin_area/settings/account_and_limit_settings.md b/doc/user/admin_area/settings/account_and_limit_settings.md new file mode 100644 index 00000000000..4a5bfb6b677 --- /dev/null +++ b/doc/user/admin_area/settings/account_and_limit_settings.md @@ -0,0 +1,53 @@ +# Account and limit settings + +## Repository size limit **[STARTER]** + +> [Introduced][ee-740] in [GitLab Enterprise Edition 8.12][ee-8.12]. + +Repositories within your GitLab instance can grow quickly, especially if you are +using LFS. Their size can grow exponentially and eat up your storage device quite +fast. + +In order to avoid this from happening, you can set a hard limit for your +repositories' size. This limit can be set globally, per group, or per project, +with per project limits taking the highest priority. + +There are numerous cases where you'll need to set up a limit for repository size. +For instance, consider the following workflow: + +1. Your team develops apps which demand large files to be stored in + the application repository. +1. Although you have enabled [Git LFS](../../../workflow/lfs/manage_large_binaries_with_git_lfs.html#git-lfs) + to your project, your storage has grown significantly. +1. Before you blow your storage limit up, you set up a limit of 10 GB + per repository. + +### How it works + +Only a GitLab administrator can set those limits. Setting the limit to `0` means +there are no restrictions. + +These settings can be found within: + +- Each project's settings. +- A group's settings. +- The **Size limit per repository (MB)** field in the **Account and limit** section of a GitLab instance's + settings by navigating to either: + - **Admin Area > Settings > General**. + - The path `/admin/application_settings`. + +The very first push of a new project cannot be checked for size as of now, so +the first push will allow you to upload more than the limit dictates, but every +subsequent push will be denied. LFS objects, however, can be checked on first +push and **will** be rejected if the sum of their sizes exceeds the maximum +allowed repository size. + +For more manually purging the files, read the docs on +[reducing the repository size using Git][repo-size]. + +> **Note:** +> For GitLab.com, the repository size limit is 10 GB. + +[ee-740]: https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/740 +[repo-size]: ../../project/repository/reducing_the_repo_size_using_git.md +[ee-8.12]: https://about.gitlab.com/2016/09/22/gitlab-8-12-released/#limit-project-size-ee diff --git a/doc/user/admin_area/settings/continuous_integration.md b/doc/user/admin_area/settings/continuous_integration.md index a1825581ebf..cd7daec0af6 100644 --- a/doc/user/admin_area/settings/continuous_integration.md +++ b/doc/user/admin_area/settings/continuous_integration.md @@ -10,8 +10,8 @@ You can find it in the admin area, under **Settings > Continuous Integration and To enable (or disable) [Auto DevOps](../../../topics/autodevops/index.md) for all projects: -1. Go to **Admin area > Settings > Continuous Integration and Deployment**. -1. Check (or uncheck to disable) the box that says "Default to Auto DevOps pipeline for all projects". +1. Go to **Admin area > Settings > Continuous Integration and Deployment** +1. Check (or uncheck to disable) the box that says "Default to Auto DevOps pipeline for all projects" 1. Optionally, set up the [Auto DevOps base domain](../../../topics/autodevops/index.md#auto-devops-base-domain) which is going to be used for Auto Deploy and Auto Review Apps. 1. Hit **Save changes** for the changes to take effect. @@ -24,9 +24,9 @@ If you want to disable it for a specific project, you can do so in ## Maximum artifacts size **[CORE ONLY]** -The maximum size of the [job artifacts][art-yml] can be set in the Admin area -of your GitLab instance. The value is in *MB* and the default is 100MB per job; -on GitLab.com it's [set to 1G](../../gitlab_com/index.md#gitlab-cicd). +The maximum size of the [job artifacts](../../../administration/job_artifacts.md) +can be set in the Admin area of your GitLab instance. The value is in *MB* and +the default is 100MB per job; on GitLab.com it's [set to 1G](../../gitlab_com/index.md#gitlab-cicd). To change it: @@ -50,6 +50,48 @@ This setting is set per job and can be overridden in [`.gitlab-ci.yml`](../../../ci/yaml/README.md#artifactsexpire_in). To disable the expiration, set it to `0`. The default unit is in seconds. +## Shared Runners pipeline minutes quota **[STARTER ONLY]** + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/1078) +in GitLab Starter 8.16. + +If you have enabled shared Runners for your GitLab instance, you can limit their +usage by setting a maximum number of pipeline minutes that a group can use on +shared Runners per month. Setting this to `0` (default value) will grant +unlimited pipeline minutes. While build limits are stored as minutes, the +counting is done in seconds. Usage resets on the first day of each month. +On GitLab.com, the quota is calculated based on your +[subscription plan](https://about.gitlab.com/pricing/#gitlab-com). + +To change the pipelines minutes quota: + +1. Go to **Admin area > Settings > Continuous Integration and Deployment** +1. Set the pipeline minutes quota limit. +1. Hit **Save changes** for the changes to take effect + +--- + +While the setting in the Admin area has a global effect, as an admin you can +also change each group's pipeline minutes quota to override the global value. + +1. Navigate to the **Groups** admin area and hit the **Edit** button for the + group you wish to change the pipeline minutes quota. +1. Set the pipeline minutes quota to the desired value +1. Hit **Save changes** for the changes to take effect. + +Once saved, you can see the build quota in the group admin view. +The quota can also be viewed in the project admin view if shared Runners +are enabled. + + + +When the pipeline minutes quota for a group is set to a value different than 0, +the **Pipelines quota** page is available to the group page settings list. +You can see there an overview of the pipeline minutes quota of all projects of +the group. + + + ## Archive jobs **[CORE ONLY]** Archiving jobs is useful for reducing the CI/CD footprint on the system by diff --git a/doc/user/admin_area/settings/email.md b/doc/user/admin_area/settings/email.md index 50c318a4969..f21fcf4e59f 100644 --- a/doc/user/admin_area/settings/email.md +++ b/doc/user/admin_area/settings/email.md @@ -4,6 +4,22 @@ The logo in the header of some emails can be customized, see the [logo customization section](../../../customization/branded_page_and_email_header.md). +## Custom additional text + +>[Introduced][ee-5031] in [GitLab Premium][eep] 10.7. + +The additional text will appear at the bottom of any email and can be used for +legal/auditing/compliance reasons. + +1. Go to **Admin area > Settings** (`/admin/application_settings`). +1. Under the **Email** section, change the **Additional text** field. +1. Hit **Save** for the changes to take effect. + + + +[ee-5031]: https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/5031 +[eep]: https://about.gitlab.com/pricing/ + ## Custom hostname for private commit emails > [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/22560) in GitLab 11.5. @@ -18,5 +34,5 @@ In order to change this option: 1. Hit **Save** for the changes to take effect. NOTE: **Note**: Once the hostname gets configured, every private commit email using the previous hostname, will not get -recognized by GitLab. This can directly conflict with certain [Push rules](https://docs.gitlab.com/ee/push_rules/push_rules.html) such as +recognized by GitLab. This can directly conflict with certain [Push rules](../../../push_rules/push_rules.md) such as `Check whether author is a GitLab user` and `Check whether committer is the current authenticated user`. diff --git a/doc/user/admin_area/settings/external_authorization.md b/doc/user/admin_area/settings/external_authorization.md new file mode 100644 index 00000000000..3a5f9ae5016 --- /dev/null +++ b/doc/user/admin_area/settings/external_authorization.md @@ -0,0 +1,110 @@ +# External authorization control **[PREMIUM]** + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/issues/4216) in +> [GitLab Premium](https://about.gitlab.com/pricing) 10.6. + +In highly controlled environments, it may be necessary for access policy to be +controlled by an external service that permits access based on project +classification and user access. GitLab provides a way to check project +authorization with your own defined service. + +## Overview + +Once the external service is configured and enabled, when a project is accessed, +a request is made to the external service with the user information and project +classification label assigned to the project. When the service replies with a +known response, the result is cached for 6 hours. + +If the external authorization is enabled, GitLab will further block pages and +functionality that render cross-project data. That includes: + +- most pages under Dashboard (Activity, Milestones, Snippets, Assigned merge + requests, Assigned issues, Todos) +- under a specific group (Activity, Contribution analytics, Issues, Issue boards, + Labels, Milestones, Merge requests) +- Global and Group search will be disabled + +This is to prevent performing to many requests at once to the external +authorization service. + +Whenever access is granted or denied this is logged in a logfile called +`external-policy-access-control.log`. +Read more about logs GitLab keeps in the [omnibus documentation][omnibus-log-docs]. + +## Configuration + +The external authorization service can be enabled by an admin on the GitLab's +admin area under the settings page: + + + +The available required properties are: + +- **Service URL**: The URL to make authorization requests to. When leaving the + URL blank, cross project features will remain available while still being able + to specify classification labels for projects. +- **External authorization request timeout**: The timeout after which an + authorization request is aborted. When a request times out, access is denied + to the user. +- **Client authentication certificate**: The certificate to use to authenticate + with the external authorization service. +- **Client authentication key**: Private key for the certificate when + authentication is required for the external authorization service, this is + encrypted when stored. +- **Client authentication key password**: Passphrase to use for the private key when authenticating with the external service this is encrypted when stored. +- **Default classification label**: The classification label to use when + requesting authorization if no specific label is defined on the project + +When using TLS Authentication with a self signed certificate, the CA certificate +needs to be trused by the openssl installation. When using GitLab installed using +Omnibus, learn to install a custom CA in the +[omnibus documentation][omnibus-ssl-docs]. Alternatively learn where to install +custom certificates using `openssl version -d`. + +## How it works + +When GitLab requests access, it will send a JSON POST request to the external +service with this body: + +```json +{ + "user_identifier": "jane@acme.org", + "project_classification_label": "project-label", + "user_ldap_dn": "CN=Jane Doe,CN=admin,DC=acme" +} +``` + +The `user_ldap_dn` is optional and is only sent when the user is logged in +through LDAP. + +When the external authorization service responds with a status code 200, the +user is granted access. When the external service responds with a status code +401 or 403, the user is denied access. In any case, the request is cached for 6 hours. + +When denying access, a `reason` can be optionally specified in the JSON body: + +```json +{ + "reason": "You are not allowed access to this project." +} +``` + +Any other status code than 200, 401 or 403 will also deny access to the user, but the +response will not be cached. + +If the service times out (after 500ms), a message "External Policy Server did +not respond" will be displayed. + +## Classification labels + +You can use your own classification label in the project's +**Settings > General > General project settings** page in the "Classification +label" box. When no classification label is specified on a project, the default +label defined in the [global settings](#configuration) will be used. + +The label will be shown on all project pages in the upper right corner. + + + +[omnibus-ssl-docs]: https://docs.gitlab.com/omnibus/settings/ssl.html +[omnibus-log-docs]: https://docs.gitlab.com/omnibus/settings/logs.html diff --git a/doc/user/admin_area/settings/img/admin_area_group_edit.png b/doc/user/admin_area/settings/img/admin_area_group_edit.png Binary files differnew file mode 100644 index 00000000000..c9bd2f10b36 --- /dev/null +++ b/doc/user/admin_area/settings/img/admin_area_group_edit.png diff --git a/doc/user/admin_area/settings/img/admin_area_groups.png b/doc/user/admin_area/settings/img/admin_area_groups.png Binary files differnew file mode 100644 index 00000000000..ebdee0eafdc --- /dev/null +++ b/doc/user/admin_area/settings/img/admin_area_groups.png diff --git a/doc/user/admin_area/settings/img/admin_project_quota_view.png b/doc/user/admin_area/settings/img/admin_project_quota_view.png Binary files differnew file mode 100644 index 00000000000..8320be860da --- /dev/null +++ b/doc/user/admin_area/settings/img/admin_project_quota_view.png diff --git a/doc/user/admin_area/settings/img/ci_shared_runners_build_minutes_quota.png b/doc/user/admin_area/settings/img/ci_shared_runners_build_minutes_quota.png Binary files differnew file mode 100644 index 00000000000..269a3cf1fbc --- /dev/null +++ b/doc/user/admin_area/settings/img/ci_shared_runners_build_minutes_quota.png diff --git a/doc/user/admin_area/settings/img/classification_label_on_project_page.png b/doc/user/admin_area/settings/img/classification_label_on_project_page.png Binary files differnew file mode 100644 index 00000000000..4aedb332cec --- /dev/null +++ b/doc/user/admin_area/settings/img/classification_label_on_project_page.png diff --git a/doc/user/admin_area/settings/img/email_settings.png b/doc/user/admin_area/settings/img/email_settings.png Binary files differnew file mode 100644 index 00000000000..ed0a80d10ce --- /dev/null +++ b/doc/user/admin_area/settings/img/email_settings.png diff --git a/doc/user/admin_area/settings/img/external_authorization_service_settings.png b/doc/user/admin_area/settings/img/external_authorization_service_settings.png Binary files differnew file mode 100644 index 00000000000..9b8658fd1a1 --- /dev/null +++ b/doc/user/admin_area/settings/img/external_authorization_service_settings.png diff --git a/doc/user/admin_area/settings/img/file_template_admin_area.png b/doc/user/admin_area/settings/img/file_template_admin_area.png Binary files differnew file mode 100644 index 00000000000..269d997e1d9 --- /dev/null +++ b/doc/user/admin_area/settings/img/file_template_admin_area.png diff --git a/doc/user/admin_area/settings/img/file_template_user_dropdown.png b/doc/user/admin_area/settings/img/file_template_user_dropdown.png Binary files differnew file mode 100644 index 00000000000..8c9eb49f6c9 --- /dev/null +++ b/doc/user/admin_area/settings/img/file_template_user_dropdown.png diff --git a/doc/user/admin_area/settings/img/group_pipelines_quota.png b/doc/user/admin_area/settings/img/group_pipelines_quota.png Binary files differnew file mode 100644 index 00000000000..d94b609ad6f --- /dev/null +++ b/doc/user/admin_area/settings/img/group_pipelines_quota.png diff --git a/doc/user/admin_area/settings/img/group_quota_view.png b/doc/user/admin_area/settings/img/group_quota_view.png Binary files differnew file mode 100644 index 00000000000..791bfd868e0 --- /dev/null +++ b/doc/user/admin_area/settings/img/group_quota_view.png diff --git a/doc/user/admin_area/settings/img/group_settings.png b/doc/user/admin_area/settings/img/group_settings.png Binary files differnew file mode 100644 index 00000000000..a849d9cfdc1 --- /dev/null +++ b/doc/user/admin_area/settings/img/group_settings.png diff --git a/doc/user/admin_area/settings/img/mirror_settings.png b/doc/user/admin_area/settings/img/mirror_settings.png Binary files differnew file mode 100644 index 00000000000..090db6808a7 --- /dev/null +++ b/doc/user/admin_area/settings/img/mirror_settings.png diff --git a/doc/user/admin_area/settings/index.md b/doc/user/admin_area/settings/index.md index 8358fe64f18..a92d8ecb619 100644 --- a/doc/user/admin_area/settings/index.md +++ b/doc/user/admin_area/settings/index.md @@ -6,6 +6,7 @@ instance like sign-up restrictions, account limits and quota, metrics, etc. Navigate to it by going to **Admin area > Settings**. Some of the settings include: +- [Account and limit settings](account_and_limit_settings.md) - [Continuous Integration and Deployment](continuous_integration.md) - [Email](email.md) - [Sign up restrictions](sign_up_restrictions.md) @@ -13,6 +14,7 @@ include: - [Third party offers](third_party_offers.md) - [Usage statistics](usage_statistics.md) - [Visibility and access controls](visibility_and_access_controls.md) +- [Custom templates repository](instance_template_repository.md) NOTE: **Note:** You can change the [first day of the week](../../profile/preferences.md) for the entire GitLab instance diff --git a/doc/user/admin_area/settings/instance_template_repository.md b/doc/user/admin_area/settings/instance_template_repository.md new file mode 100644 index 00000000000..4010008f694 --- /dev/null +++ b/doc/user/admin_area/settings/instance_template_repository.md @@ -0,0 +1,63 @@ +# Instance template repository **[PREMIUM ONLY]** + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/issues/5986) in +> [GitLab Premium](https://about.gitlab.com/pricing) 11.3. + +## Overview + +In hosted systems, enterprises often have a need to share their own templates +across teams. This feature allows an administrator to pick a project to be the +instance-wide collection of file templates. These templates are then exposed to +all users [via the web editor](../../project/repository/web_editor.md#template-dropdowns) +while the project remains secure. + +## Configuration + +As an administrator, navigate to **Admin area > Settings > Templates** and +select the project to serve as the custom template repository. + + + +Once a project has been selected, you can add custom templates to the repository, +and they will appear in the appropriate places in the +[frontend](../../project/repository/web_editor.md#template-dropdowns) and +[API](../../../api/settings.md). + +Templates must be added to a specific subdirectory in the repository, +corresponding to the kind of template. The following types of custom templates +are supported: + +| Type | Directory | Extension | +| :---------------: | :-----------: | :-----------: | +| `Dockerfile` | `Dockerfile` | `.dockerfile` | +| `.gitignore` | `gitignore` | `.gitignore` | +| `.gitlab-ci.yml` | `gitlab-ci` | `.yml` | +| `LICENSE` | `LICENSE` | `.txt` | + +Each template must go in its respective subdirectory, have the correct +extension and not be empty. So, the hierarchy should look like this: + +```text +|-- README.md +|-- Dockerfile + |-- custom_dockerfile.dockerfile + |-- another_dockerfile.dockerfile +|-- gitignore + |-- custom_gitignore.gitignore + |-- another_gitignore.gitignore +|-- gitlab-ci + |-- custom_gitlab-ci.yml + |-- another_gitlab-ci.yml +|-- LICENSE + |-- custom_license.txt + |-- another_license.txt +``` + +Once this is established, the list of custom templates will be included when +creating a new file and the template type is selected. These will appear at the +top of the list. + + + +If this feature is disabled or no templates are present, there will be +no "Custom" section in the selection dropdown. diff --git a/doc/user/admin_area/settings/visibility_and_access_controls.md b/doc/user/admin_area/settings/visibility_and_access_controls.md index 6a1e8004f87..4d1b7d0f252 100644 --- a/doc/user/admin_area/settings/visibility_and_access_controls.md +++ b/doc/user/admin_area/settings/visibility_and_access_controls.md @@ -49,5 +49,15 @@ block access to the server itself. The ports used for the protocol, be it SSH or HTTP, will still be accessible. What GitLab does is restrict access on the application level. +## Allow mirrors to be set up for projects + +> [Introduced][ee-3586] in GitLab 10.3. + +This option is enabled by default. By disabling it, both pull and push mirroring will no longer +work in every repository and can only be re-enabled on a per-project basis by an admin. + + + [ce-4696]: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/4696 [ce-18021]: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/18021 +[ee-3586]: https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/3586 diff --git a/doc/user/discussions/img/mr_review_resolve.png b/doc/user/discussions/img/mr_review_resolve.png Binary files differnew file mode 100644 index 00000000000..34176f3fa8e --- /dev/null +++ b/doc/user/discussions/img/mr_review_resolve.png diff --git a/doc/user/discussions/img/mr_review_resolve2.png b/doc/user/discussions/img/mr_review_resolve2.png Binary files differnew file mode 100644 index 00000000000..e4adb5f2c2d --- /dev/null +++ b/doc/user/discussions/img/mr_review_resolve2.png diff --git a/doc/user/discussions/img/mr_review_second_comment.png b/doc/user/discussions/img/mr_review_second_comment.png Binary files differnew file mode 100644 index 00000000000..920ea05ad66 --- /dev/null +++ b/doc/user/discussions/img/mr_review_second_comment.png diff --git a/doc/user/discussions/img/mr_review_second_comment_added.png b/doc/user/discussions/img/mr_review_second_comment_added.png Binary files differnew file mode 100644 index 00000000000..1fb54348547 --- /dev/null +++ b/doc/user/discussions/img/mr_review_second_comment_added.png diff --git a/doc/user/discussions/img/mr_review_start.png b/doc/user/discussions/img/mr_review_start.png Binary files differnew file mode 100644 index 00000000000..38b44bda0d2 --- /dev/null +++ b/doc/user/discussions/img/mr_review_start.png diff --git a/doc/user/discussions/img/mr_review_unresolve.png b/doc/user/discussions/img/mr_review_unresolve.png Binary files differnew file mode 100644 index 00000000000..da895ceb89f --- /dev/null +++ b/doc/user/discussions/img/mr_review_unresolve.png diff --git a/doc/user/discussions/img/mr_review_unresolve2.png b/doc/user/discussions/img/mr_review_unresolve2.png Binary files differnew file mode 100644 index 00000000000..a824b806e4a --- /dev/null +++ b/doc/user/discussions/img/mr_review_unresolve2.png diff --git a/doc/user/discussions/img/pending_review_comment.png b/doc/user/discussions/img/pending_review_comment.png Binary files differnew file mode 100644 index 00000000000..916ef5b7452 --- /dev/null +++ b/doc/user/discussions/img/pending_review_comment.png diff --git a/doc/user/discussions/img/review_comment_quickactions.png b/doc/user/discussions/img/review_comment_quickactions.png Binary files differnew file mode 100644 index 00000000000..bd9880c329a --- /dev/null +++ b/doc/user/discussions/img/review_comment_quickactions.png diff --git a/doc/user/discussions/img/review_preview.png b/doc/user/discussions/img/review_preview.png Binary files differnew file mode 100644 index 00000000000..4bf53a81b9c --- /dev/null +++ b/doc/user/discussions/img/review_preview.png diff --git a/doc/user/discussions/index.md b/doc/user/discussions/index.md index a15e1a59921..ebe4f2a286d 100644 --- a/doc/user/discussions/index.md +++ b/doc/user/discussions/index.md @@ -5,6 +5,7 @@ The ability to contribute conversationally is offered throughout GitLab. You can leave a comment in the following places: - issues +- epics - merge requests - snippets - commits @@ -278,6 +279,75 @@ edit existing comments. Non-team members are restricted from adding or editing c Additionally, locked issues and merge requests can not be reopened. +## Merge Request Reviews **[PREMIUM]** + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/4213) in GitLab 11.4. + +When looking at a Merge Request diff, you are able to start a review. +This allows you to create comments inside a Merge Request that are **only visible to you** until published, +in order to allow you to submit them all as a single action. + +### Starting a review + +In order to start a review, simply write a comment on a diff as normal under the **Changes** tab +in an MR and click on the **Start a review** button. + + + +Once a review is started, you will see any comments that are part of this review marked `Pending`. +All comments that are part of a review show two buttons: + +- **Submit review**: Submits all comments that are part of the review, making them visible to other users. +- **Add comment now**: Submits the specific comment as a regular comment instead of as part of the review. + + + +You can use [quick actions] inside review comments. The comment will show the actions that will be performed once published. + + + +To add more comments to a review, start writing a comment as normal and click the **Add to review** button. + + + +This will add the comment to the review. + + + +### Resolving/Unresolving discussions + +Review comments can also resolve/unresolve [resolvable discussions](#resolvable-discussions). +When replying to a comment, you will see a checkbox that you can click in order to resolve or unresolve +the discussion once published. + + + + +If a particular pending comment will resolve or unresolve the discussion, this will be shown on the pending +comment itself. + + + + +### Submitting a review + +If you have any comments that have not been submitted, you will see a bar at the +bottom of the screen with two buttons: + +- **Discard**: Discards all comments that have not been submitted. +- **Finish review**: Opens a list of comments ready to be submitted for review. + Clicking **Submit review** will publish all comments. Any quick actions + submitted are performed at this time. + +Alternatively, every pending comment has a button to finish the entire review. + + + +Submitting the review will send a single email to every notifiable user of the +merge request with all the comments associated to it. + +Replying to this email will, consequentially, create a new comment on the associated merge request. + ## Filtering notes > [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/issues/26723) in GitLab 11.5. diff --git a/doc/user/group/clusters/index.md b/doc/user/group/clusters/index.md index 6d67688fdff..3f2c7ae1545 100644 --- a/doc/user/group/clusters/index.md +++ b/doc/user/group/clusters/index.md @@ -75,7 +75,7 @@ The domain should have a wildcard DNS configured to the Ingress IP address. When adding more than one Kubernetes cluster to your project, you need to differentiate them with an environment scope. The environment scope associates clusters with [environments](../../../ci/environments.md) similar to how the -[environment-specific variables](https://docs.gitlab.com/ee/ci/variables/README.html#limiting-environment-scopes-of-variables-premium) +[environment-specific variables](../../../ci/variables/README.md#limiting-environment-scopes-of-variables-premium) work. While evaluating which environment matches the environment scope of a diff --git a/doc/user/group/contribution_analytics/img/group_stats_cal.png b/doc/user/group/contribution_analytics/img/group_stats_cal.png Binary files differnew file mode 100644 index 00000000000..0238c7bf088 --- /dev/null +++ b/doc/user/group/contribution_analytics/img/group_stats_cal.png diff --git a/doc/user/group/contribution_analytics/img/group_stats_graph.png b/doc/user/group/contribution_analytics/img/group_stats_graph.png Binary files differnew file mode 100644 index 00000000000..ccfd3782c6f --- /dev/null +++ b/doc/user/group/contribution_analytics/img/group_stats_graph.png diff --git a/doc/user/group/contribution_analytics/img/group_stats_table.png b/doc/user/group/contribution_analytics/img/group_stats_table.png Binary files differnew file mode 100644 index 00000000000..f1d1031fa18 --- /dev/null +++ b/doc/user/group/contribution_analytics/img/group_stats_table.png diff --git a/doc/user/group/contribution_analytics/index.md b/doc/user/group/contribution_analytics/index.md new file mode 100644 index 00000000000..a737b80c8c3 --- /dev/null +++ b/doc/user/group/contribution_analytics/index.md @@ -0,0 +1,55 @@ +# Contribution Analytics **[STARTER]** + +>**Note:** +Introduced in [GitLab Starter][ee] 8.3. + +Track your team members' activity across your organization. + +## Overview + +With Contribution Analytics you can get an overview of the activity of +issues, merge requests, and push events of your organization and its members. + +The analytics page is located at **Group > Contribution Analytics** +under the URL `/groups/<groupname>/analytics`. + +## Use cases + +- Analyze your team's contributions over a period of time and offer a bonus for the top contributors +- Identify opportunities for improveent with group members who may benefit from additional support + +## Using Contribution Analytics + +There are three main bar graphs that are deducted from the number of +contributions per group member. These contributions include push events, merge +requests and closed issues. Hovering on each bar you can see the number of +events for a specific member. + + + +## Changing the period time + +There are three periods you can choose from, 'Last week', 'Last month' and +'Last three months'. The default is 'Last week'. + +You can choose which period to display by using the dropdown calendar menu in +the upper right corner. + + + +## Sorting by different factors + +Apart from the bar graphs you can also see the contributions per group member +which are depicted in a table that can be sorted by: + +* Member name +* Number of pushed events +* Number of opened issues +* Number of closed issues +* Number of opened MRs +* Number of accepted MRs +* Number of total contributions + + + +[ee]: https://about.gitlab.com/pricing/ diff --git a/doc/user/group/epics/img/button_close_epic.png b/doc/user/group/epics/img/button_close_epic.png Binary files differnew file mode 100644 index 00000000000..aa1a889ea23 --- /dev/null +++ b/doc/user/group/epics/img/button_close_epic.png diff --git a/doc/user/group/epics/img/button_reopen_epic.png b/doc/user/group/epics/img/button_reopen_epic.png Binary files differnew file mode 100644 index 00000000000..7a953189270 --- /dev/null +++ b/doc/user/group/epics/img/button_reopen_epic.png diff --git a/doc/user/group/epics/img/containing_epic.png b/doc/user/group/epics/img/containing_epic.png Binary files differnew file mode 100644 index 00000000000..5ed693e6d6a --- /dev/null +++ b/doc/user/group/epics/img/containing_epic.png diff --git a/doc/user/group/epics/img/epic_view.png b/doc/user/group/epics/img/epic_view.png Binary files differnew file mode 100644 index 00000000000..dbda98e4351 --- /dev/null +++ b/doc/user/group/epics/img/epic_view.png diff --git a/doc/user/group/epics/img/epics_list_view.png b/doc/user/group/epics/img/epics_list_view.png Binary files differnew file mode 100644 index 00000000000..b30608d9d31 --- /dev/null +++ b/doc/user/group/epics/img/epics_list_view.png diff --git a/doc/user/group/epics/img/epics_search.png b/doc/user/group/epics/img/epics_search.png Binary files differnew file mode 100644 index 00000000000..96335527468 --- /dev/null +++ b/doc/user/group/epics/img/epics_search.png diff --git a/doc/user/group/epics/img/epics_sort.png b/doc/user/group/epics/img/epics_sort.png Binary files differnew file mode 100644 index 00000000000..b23c65fd0ef --- /dev/null +++ b/doc/user/group/epics/img/epics_sort.png diff --git a/doc/user/group/epics/index.md b/doc/user/group/epics/index.md new file mode 100644 index 00000000000..bd19f588751 --- /dev/null +++ b/doc/user/group/epics/index.md @@ -0,0 +1,197 @@ +# Epics **[ULTIMATE]** + +> Introduced in [GitLab Ultimate][ee] 10.2. + +Epics let you manage your portfolio of projects more efficiently and with less +effort by tracking groups of issues that share a theme, across projects and +milestones. + + + +## Creating an epic + +A paginated list of epics is available in each group from where you can create +a new epic. The list of epics includes also epics from all subgroups of the +selected group. From your group page: + +1. Go to **Epics** +1. Click the **New epic** button at the top right +1. Enter a descriptive title and hit **Create epic** + +Once created, you will be taken to the view for that newly-created epic where +you can change its title, description, start date, and due date. + + + +## Adding an issue to an epic + +An epic contains a list of issues and an issue can be associated with at most +one epic. When on an epic, you can add its associated issues: + +1. Click the plus icon (<kbd>+</kbd>) under the epic description. +1. Paste the link of the issue (you can hit <kbd>Spacebar</kbd> to add more than + one issues at a time). +1. Click **Add**. + +Any issue belonging to a project in the epic's group or any of the epic's +subgroups are eligible to be added. To remove an issue from an epic, click +on the <kbd>x</kbd> button in the epic's issue list. + +NOTE: **Note:** +When you add an issue or an epic to an epic that's already associated with another epic, +the issue or the epic is automatically removed from the previous epic. + +## Multi-level child epics + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/issues/8333) in GitLab Ultimate 11.7. + +Much like adding issues to an epic, an epic can have multiple child epics with +the maximum depth being 5. To add a child epic: + +1. Click the plus icon (<kbd>+</kbd>) under the epic description. +1. Paste the link of the epic. +1. Click **Add**. + +Any epic that belongs to a group or subgroup of the parent epic's group is +eligible to be added. To remove a child epic from a parent epic, +click on the <kbd>x</kbd> button in the parent epic's epic list. + +## Start date and due date + +For each of the dates in the sidebar of an epic, you can choose to either: + +- Enter a fixed value. +- Inherit a dynamic value called "From milestones". + +If you select "From milestones" for the start date, GitLab will automatically set the +date to be earliest start date across all milestones that are currently assigned +to the issues that are attached to the epic. Similarly, if you select "From milestones" +for the due date, GitLab will set it to be the latest due date across all +milestones that are currently assigned to those issues. + +These are dynamic dates in that if milestones are re-assigned to the issues, if the +milestone dates change, or if issues are added or removed from the epic, then +the re-calculation will happen immediately to set a new dynamic date. + +## Reordering issues and child epics + +Drag and drop to reorder issues and child epics. New issues and child epics added to an epic appear at the top of the list. + +## Deleting an epic + +NOTE: **Note:** +To delete an epic, you need to be an [Owner][permissions] of a group/subgroup. + +When inside a single epic view, click the **Delete** button to delete the epic. +A modal will pop-up to confirm your action. + +Deleting an epic releases all existing issues from their associated epic in the +system. + +## Closing and reopening epics + +### Using buttons + +Whenever you decide that there is no longer need for that epic, +close the epic using the close button: + + + +You can always reopen it using the reopen button. + + + +### Using quick actions + +You can close or reopen an epic using [Quick actions](../../project/quick_actions.md) + +## Navigating to an epic from an issue + +If an issue belongs to an epic, you can navigate to the containing epic with the +link in the issue sidebar. + + + +## Promoting an issue to an epic + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/issues/3777) in [GitLab Ultimate](https://about.gitlab.com/pricing/) 11.6. + +If you have [permissions](../../permissions.md) to close an issue and create an +epic in the parent group, you can promote an issue to an epic with the `/promote` +[quick action](../../project/quick_actions.md#quick-actions-for-epics-ultimate). +Only issues from projects that are in groups can be promoted. + +When the quick action is executed: + +- An epic is created in the same group as the project of the issue. +- Subscribers of the issue are notified that the epic was created. + +The following issue metadata will be copied to the epic: + +- Title, description, activity/comment thread. +- Upvotes/downvotes. +- Participants. +- Group labels that the issue already has. + +## Searching for an epic from epics list page + +> Introduced in [GitLab Ultimate][ee] 10.5. + +You can search for an epic from the list of epics using filtered search bar (similar to +that of Issues and Merge requests) based on following parameters: + +- Title or description +- Author name / username +- Labels + + + +To search, go to the list of epics and click on the field **Search or filter results...**. +It will display a dropdown menu, from which you can add an author. You can also enter plain +text to search by epic title or description. When done, press <kbd>Enter</kbd> on your +keyboard to filter the list. + +You can also sort epics list by: + +- **Created date** +- **Last updated** +- **Start date** +- **Due date** + +Each option contains a button that can toggle the order between **ascending** and **descending**. The sort option and order will be persisted to be used wherever epics are browsed including the [roadmap](../roadmap/index.md). + + + +## Permissions + +If you have access to view an epic and have access to view an issue already +added to that epic, then you can view the issue in the epic issue list. + +If you have access to edit an epic and have access to edit an issue, then you +can add the issue to or remove it from the epic. + +Note that for a given group, the visibility of all projects must be the same as +the group, or less restrictive. That means if you have access to a group's epic, +then you already have access to its projects' issues. + +You may also consult the [group permissions table][permissions]. + +[ee]: https://about.gitlab.com/pricing/ +[permissions]: ../../permissions.md#group-members-permissions + +## Thread + +- Comments: collaborate on that epic by posting comments in its thread. +These text fields also fully support +[GitLab Flavored Markdown](../../markdown.md#gitlab-flavored-markdown-gfm). + +## Comment, or start a discussion + +Once you wrote your comment, you can either: + +- Click "Comment" and your comment will be published. +- Click "Start discussion": start a thread within that epic's thread to discuss specific points. + +## Award emoji + +- You can [award an emoji](../../award_emojis.md) to that epic or its comments. diff --git a/doc/user/group/img/create_new_group_info.png b/doc/user/group/img/create_new_group_info.png Binary files differindex c2e6ed43c5b..bd724240c37 100644 --- a/doc/user/group/img/create_new_group_info.png +++ b/doc/user/group/img/create_new_group_info.png diff --git a/doc/user/group/img/group_file_template_dropdown.png b/doc/user/group/img/group_file_template_dropdown.png Binary files differnew file mode 100644 index 00000000000..d9caae1a223 --- /dev/null +++ b/doc/user/group/img/group_file_template_dropdown.png diff --git a/doc/user/group/img/group_file_template_settings.png b/doc/user/group/img/group_file_template_settings.png Binary files differnew file mode 100644 index 00000000000..ca42f7726db --- /dev/null +++ b/doc/user/group/img/group_file_template_settings.png diff --git a/doc/user/group/img/group_issue_board.png b/doc/user/group/img/group_issue_board.png Binary files differnew file mode 100644 index 00000000000..a0da74a320f --- /dev/null +++ b/doc/user/group/img/group_issue_board.png diff --git a/doc/user/group/img/member_lock.png b/doc/user/group/img/member_lock.png Binary files differnew file mode 100644 index 00000000000..3f1382e76c6 --- /dev/null +++ b/doc/user/group/img/member_lock.png diff --git a/doc/user/group/index.md b/doc/user/group/index.md index c1f50bcc593..db576dd5f23 100644 --- a/doc/user/group/index.md +++ b/doc/user/group/index.md @@ -5,10 +5,12 @@ and grant members access to several projects at once. Groups can also be nested in [subgroups](subgroups/index.md). -Find your groups by expanding the left menu and clicking **Groups**: +Find your groups by clicking **Groups** in the top navigation.  +> The groups dropdown in the top navigation was [introduced][ce-36234] in [GitLab 11.1](https://about.gitlab.com/2018/07/22/gitlab-11-1-released/#groups-dropdown-in-navigation). + The Groups page displays all groups you are a member of, how many projects it holds, how many members it has, the group visibility, and, if you have enough permissions, a link to the group settings. By clicking the last button you can leave that group. @@ -44,7 +46,7 @@ For example, consider a user named Alex: 1. Alex creates an account on GitLab.com with the username `alex`; their profile will be accessed under `https://gitlab.example.com/alex` -1. Alex creates a group for their team with the groupname `alex-team`; +1. Alex creates a group for their team with the group name `alex-team`; the group and its projects will be accessed under `https://gitlab.example.com/alex-team` 1. Alex creates a subgroup of `alex-team` with the subgroup name `marketing`; this subgroup and its projects will be accessed under `https://gitlab.example.com/alex-team/marketing` @@ -69,7 +71,7 @@ together in a single list view. You can create a group in GitLab from: -1. The Groups page: expand the left menu, click **Groups**, and click the green button **New group**: +1. The Groups page: from the top menu, click **Groups**, and click the green button **New group**:  @@ -81,11 +83,11 @@ Add the following information:  -1. Set the **Group path** which will be the **namespace** under which your projects +1. The **Group name** will populate the path automatically. Optionally, you can + change it. This is the name that will me displayed in the group views. +1. Set the **Group URL** which will be the **namespace** under which your projects will be hosted (path can contain only letters, digits, underscores, dashes and dots; it cannot start with dashes or end in dot). -1. The **Group name** will populate with the path. Optionally, you can change - it. This is the name that will display in the group views. 1. Optionally, you can add a description so that others can briefly understand what this group is about. 1. Optionally, choose an avatar for your project. @@ -152,6 +154,21 @@ There are two different ways to add a new project to a group:  +### Default project creation level **[STARTER]** + +> [Introduced][ee-2534] in [GitLab Premium][ee] 10.5. +> Brought to [GitLab Starter][ee] in 10.7. + +Group owners or administrators can set an option that will give users with the +Developer role the ability to create projects under groups. + +By default, `Developers` and `Maintainers` are allowed to create projects under a +group, but this can be changed either within the group settings for a group, or +be set globally by a GitLab administrator in the Admin area +(**Settings > Visibility and Access Controls**). + +The setting can set to "None", "Maintainers", or "Developers + Maintainers". + ## Transfer projects into groups Learn how to [transfer a project into a group](../project/settings/index.md#transferring-an-existing-project-into-another-namespace). @@ -168,6 +185,26 @@ Alternatively, you can [lock the sharing with group feature](#share-with-group-l In GitLab Enterprise Edition it is possible to manage GitLab group memberships using LDAP groups. See [the GitLab Enterprise Edition documentation](../../integration/ldap.md) for more information. +## Epics **[ULTIMATE]** + +> Introduced in [GitLab Ultimate][ee] 10.2. + +Epics let you manage your portfolio of projects more efficiently and with less +effort by tracking groups of issues that share a theme, across projects and +milestones. + +[Learn more about Epics.](epics/index.md) + +## Group Security Dashboard **[ULTIMATE]** + +Get an overview of the vulnerabilities of all the projects in a group and its subgroups. + +[Learn more about the Group Security Dashboard.](security_dashboard/index.md) + +## Insights **[ULTIMATE]** + +> Introduced in [GitLab Ultimate][ee] 11.9 behind the `insights` feature flag. + ## Transfer groups to another group From 10.5 there are two different ways to transfer a group: @@ -239,7 +276,7 @@ working together in a project. To inherit the group membership, you share the project between the two groups A and B. **Share with group lock** prevents any project within the group from being shared with another group. By doing so, you -guarantee only the right group members have access to that projects. +guarantee only the right group members have access to those projects. To enable this feature, navigate to the group settings page. Select **Share with group lock** and **Save the group**. @@ -248,17 +285,50 @@ To enable this feature, navigate to the group settings page. Select #### Member Lock **[STARTER]** -With **Member Lock** it is possible to lock membership in project to the -level of members in group. +With Member lock, it is possible to lock membership in a project to the +level of members in the group. -Learn more about [Member Lock](https://docs.gitlab.com/ee/user/group/index.html#member-lock). +Member lock lets a group owner lock down any new project membership to all the +projects within the group, allowing tighter control over project membership. -#### Group-level file templates **[PREMIUM]** +For instance, if you want to lock the group for an [Audit Event](../../administration/audit_events.md), +you enable Member lock to guarantee that membership of a project cannot be modified during that audit. -Group-level file templates allow you to share a set of templates for common file -types with every project in a group. +To enable this feature: -Learn more about [Group-level file templates](https://docs.gitlab.com/ee/user/group/index.html#group-level-file-templates-premium). +1. Navigate to the group's **Settings > General** page. +1. Expand the **Permissions, LFS, 2FA** section and select **Member lock**. +1. Click the **Save changes** button. + + + +This will disable the option for all users who previously had permissions to +operate project memberships so no new users can be added. Furthermore, any +request to add a new user to a project through API will not be possible. + +#### Group file templates **[PREMIUM]** + +Group file templates allow you to share a set of templates for common file +types with every project in a group. It is analogous to the +[instance template repository](../admin_area/settings/instance_template_repository.md) +feature, and the selected project should follow the same naming conventions as +are documented on that page. + +Only projects that are in the group may be chosen as the source of templates. +This includes projects shared with the group, but **excludes** projects in +subgroups or parent groups of the group being configured. + +This feature may be configured for subgroups as well as parent groups. A project +in a subgroup will have access to the templates for that subgroup, as well as +any parent groups. + + + +To enable this feature, navigate to the group settings page, expand the +**Templates** section, choose a project to act as the template repository, and +**Save group**. + + #### Group-level project templates **[PREMIUM]** @@ -274,3 +344,16 @@ Define project templates at a group-level by setting a group as a template sourc - **Audit Events**: view [Audit Events](https://docs.gitlab.com/ee/administration/audit_events.html#audit-events) for the group. **[STARTER ONLY]** - **Pipelines quota**: keep track of the [pipeline quota](../admin_area/settings/continuous_integration.md) for the group. + +## User contribution analysis **[STARTER]** + +With [GitLab Contribution Analytics](contribution_analytics/index.md) +you have an overview of the contributions (pushes, merge requests, +and issues) performed by your group members. + +## Issues analytics **[PREMIUM]** + +With [GitLab Issues Analytics](issues_analytics/index.md), in groups, you can see a bar chart of the number of issues created each month. + +[ee]: https://about.gitlab.com/pricing/ +[ee-2534]: https://gitlab.com/gitlab-org/gitlab-ee/issues/2534 diff --git a/doc/user/group/issues_analytics/img/issues_created_per_month.png b/doc/user/group/issues_analytics/img/issues_created_per_month.png Binary files differnew file mode 100644 index 00000000000..96f0d36c917 --- /dev/null +++ b/doc/user/group/issues_analytics/img/issues_created_per_month.png diff --git a/doc/user/group/issues_analytics/index.md b/doc/user/group/issues_analytics/index.md new file mode 100644 index 00000000000..bbfab5fcea1 --- /dev/null +++ b/doc/user/group/issues_analytics/index.md @@ -0,0 +1,16 @@ +# Issues Analytics + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/issues/7478) in [GitLab Premium](https://about.gitlab.com/pricing/) 11.5. + +GitLab by default displays a bar chart of the number of issues created each month, for the +current month, and 12 months prior, for a total of 13 months. + +You can change the total number of months displayed by setting a URL parameter. +For example, `https://gitlab.com/groups/gitlab-org/-/issues_analytics?months_back=15` +would show a total of 15 months for the chart in the GitLab.org grouop. + +The **Search or filter results...** field can be used for filtering the issues by any attribute. For example, labels, assignee, milestone, and author. + +To access the chart, navigate to a group's sidebar and select **Issues > Analytics**. + + diff --git a/doc/user/group/roadmap/img/epics_state_dropdown.png b/doc/user/group/roadmap/img/epics_state_dropdown.png Binary files differnew file mode 100644 index 00000000000..cbcc3658a54 --- /dev/null +++ b/doc/user/group/roadmap/img/epics_state_dropdown.png diff --git a/doc/user/group/roadmap/img/roadmap_timeline_months.png b/doc/user/group/roadmap/img/roadmap_timeline_months.png Binary files differnew file mode 100644 index 00000000000..5a046b21507 --- /dev/null +++ b/doc/user/group/roadmap/img/roadmap_timeline_months.png diff --git a/doc/user/group/roadmap/img/roadmap_timeline_quarters.png b/doc/user/group/roadmap/img/roadmap_timeline_quarters.png Binary files differnew file mode 100644 index 00000000000..56f428cb471 --- /dev/null +++ b/doc/user/group/roadmap/img/roadmap_timeline_quarters.png diff --git a/doc/user/group/roadmap/img/roadmap_timeline_weeks.png b/doc/user/group/roadmap/img/roadmap_timeline_weeks.png Binary files differnew file mode 100644 index 00000000000..bf4c1dc0284 --- /dev/null +++ b/doc/user/group/roadmap/img/roadmap_timeline_weeks.png diff --git a/doc/user/group/roadmap/img/roadmap_view.png b/doc/user/group/roadmap/img/roadmap_view.png Binary files differnew file mode 100644 index 00000000000..ff41a2e0441 --- /dev/null +++ b/doc/user/group/roadmap/img/roadmap_view.png diff --git a/doc/user/group/roadmap/index.md b/doc/user/group/roadmap/index.md new file mode 100644 index 00000000000..74db43f1aff --- /dev/null +++ b/doc/user/group/roadmap/index.md @@ -0,0 +1,64 @@ +# Roadmap **[ULTIMATE]** + +> Introduced in [GitLab Ultimate][ee] 10.5. + +An Epic within a group containing **Start date** and/or **Due date** +can be visualized in a form of a timeline (e.g. a Gantt chart). The Epics Roadmap page +shows such a visualization for all the epics which are under a group and/or its subgroups. + + + +A dropdown allows you to show only open or closed epics. By default, all epics are shown. + + + +Epics in the view can be sorted by: + +- **Created date** +- **Last updated** +- **Start date** +- **Due date** + +Each option contains a button that can toggle the order between **ascending** and **descending**. The sort option and order will be persisted to be used wherever epics are browsed including the [epics list view](../epics/index.md). + +## Timeline duration + +Starting with [GitLab Ultimate][ee] 11.0, Roadmap supports three different date ranges; Quarters, Months (Default) and Weeks. + +### Quarters + + + +In _Quarters_ preset, roadmap shows epics which have start or due dates _falling within_ or +_going through_ **past quarter**, **current quarter** and **next 4 quarters**, where _today_ +is shown by the vertical red line in the timeline. The sub-headers underneath the quarter name on +the timeline header represent the month of the quarter. + +### Months + + + +In _Months_ preset, roadmap shows epics which have start or due dates _falling within_ or +_going through_ **past month**, **current month** and **next 5 months**, where _today_ +is shown by the vertical red line in the timeline. The sub-headers underneath the month name on +the timeline header represent the date on starting day (Sunday) of the week. This preset is +selected by default. + +### Weeks + + + +In _Weeks_ preset, roadmap shows epics which have start or due dates _falling within_ or +_going through_ **past week**, **current week** and **next 4 weeks**, where _today_ +is shown by the vertical red line in the timeline. The sub-headers underneath the week name on +the timeline header represent the days of the week. + +## Timeline bar for an epic + +The timeline bar indicates the approximate position of an epic based on its start +and due date. If an epic doesn't have a due date, the timeline bar fades +away towards the future. Similarly, if an epic doesn't have a start date, the +timeline bar becomes more visible as it approaches the epic's due date on the +timeline. + +[ee]: https://about.gitlab.com/pricing diff --git a/doc/user/group/saml_sso/index.md b/doc/user/group/saml_sso/index.md new file mode 100644 index 00000000000..d1917894ee8 --- /dev/null +++ b/doc/user/group/saml_sso/index.md @@ -0,0 +1,57 @@ +# SAML SSO for Groups **[PREMIUM]** + +> Introduced in [GitLab Premium](https://about.gitlab.com/pricing/) 11.0. + +This allows SAML to be used for adding users to a group on GitLab.com and other instances where using [site-wide SAML](../../../integration/saml.md) is not possible. + +When using a group SAML SSO link, users should already have an account on the GitLab instance with the email address that matches the user account from the provider. + +NOTE: **Note:** SAML SSO for groups is used only as a convenient way to add users and does not sync users between providers. Group owners will still need to manage user accounts, such as removing users when necessary. + +## How to configure + +1. Navigate to the group and click Settings -> SAML SSO. +1. Configure your SAML server using the **Assertion consumer service URL** and **Issuer**. See [your identity provider's documentation](#providers) for more details. +1. Configure required assertions using the table below. +1. Find the SSO URL from your Identity Provider and enter it on GitLab. +1. Find and enter the fingerprint for the SAML token signing certificate. + +## NameID + +GitLab.com uses the SAML NameID to identify users, so it must be present in the SAML response and unique to the user. + +The value should be something that will never change for that user, such as a unique ID or username. Email could also be used as the NameID, but only if it can be guaranteed to never change. + +## Assertions + +| Field | Supported keys | Notes | +|-|----------------|-------------| +| Email | `email`, `mail` | (required) | +| Full Name | `name` | | +| First Name | `first_name`, `firstname`, `firstName` | | +| Last Name | `last_name`, `lastname`, `lastName` | | + +## Providers + +| Provider | Documentation | +|----------|---------------| +| ADFS (Active Directory Federation Services) | [Create a Relying Party Trust](https://docs.microsoft.com/en-us/windows-server/identity/ad-fs/operations/create-a-relying-party-trust) | +| Azure | [Configuring single sign-on to applications](https://docs.microsoft.com/en-us/azure/active-directory/active-directory-saas-custom-apps) | +| Auth0 | [Auth0 as Identity Provider](https://auth0.com/docs/protocols/saml/saml-idp-generic) | +| G Suite | [Set up your own custom SAML application](https://support.google.com/a/answer/6087519?hl=en) | +| JumpCloud | [Single Sign On (SSO) with GitLab](https://support.jumpcloud.com/customer/en/portal/articles/2810701-single-sign-on-sso-with-gitlab) | +| Okta | [Setting up a SAML application in Okta](https://developer.okta.com/standards/SAML/setting_up_a_saml_application_in_okta) | +| OneLogin | [Use the OneLogin SAML Test Connector](https://onelogin.service-now.com/support?id=kb_article&sys_id=93f95543db109700d5505eea4b96198f) | +| Ping Identity | [Add and configure a new SAML application](https://docs.pingidentity.com/bundle/p1_enterpriseConfigSsoSaml_cas/page/enableAppWithoutURL.html) | + +## Glossary + +| Term | Description | +|------|-------------| +| Identity Provider | The service which manages your user identities such as ADFS, Okta, Onelogin or Ping Identity. | +| Service Provider | SAML considers GitLab to be a service provider. | +| Assertion | A piece of information about a user's identity, such as their name or role. Also know as claims or attributes. | +| SSO | Single Sign On. | +| Assertion consumer service URL | The callback on GitLab where users will be redirected after successfully authenticating with the identity provider. | +| Issuer | How GitLab identifies itself to the identity provider. Also known as a "Relying party trust identifier". | +| Certificate fingerprint | Used to confirm that communications over SAML are secure by checking that the server is signing communications with the correct certificate. Also known as a certificate thumbprint. | diff --git a/doc/user/group/security_dashboard/img/dashboard.png b/doc/user/group/security_dashboard/img/dashboard.png Binary files differnew file mode 100644 index 00000000000..8785db3dd0a --- /dev/null +++ b/doc/user/group/security_dashboard/img/dashboard.png diff --git a/doc/user/group/security_dashboard/img/issue.png b/doc/user/group/security_dashboard/img/issue.png Binary files differnew file mode 100644 index 00000000000..6467201df3f --- /dev/null +++ b/doc/user/group/security_dashboard/img/issue.png diff --git a/doc/user/group/security_dashboard/img/modal.png b/doc/user/group/security_dashboard/img/modal.png Binary files differnew file mode 100644 index 00000000000..cdbaaade5dd --- /dev/null +++ b/doc/user/group/security_dashboard/img/modal.png diff --git a/doc/user/group/security_dashboard/index.md b/doc/user/group/security_dashboard/index.md new file mode 100644 index 00000000000..79b75a70998 --- /dev/null +++ b/doc/user/group/security_dashboard/index.md @@ -0,0 +1,118 @@ +--- +description: "The Group Security Dashboard gives an overview of the vulnerabilities of all the projects in a group and its subgroups." +--- + +# Group Security Dashboard **[ULTIMATE]** + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/issues/6709) in + [GitLab Ultimate](https://about.gitlab.com/pricing) 11.5. + +The Group Security Dashboard gives an overview of the vulnerabilities of all the +projects in a group and its subgroups. + +## Overview + +To use the Group Security Dashboard, you need a group that has at least one +project with [Static Application Security Testing](../../project/merge_requests/sast.md) or [Dependency Scanning](../../project/merge_requests/dependency_scanning.md) +enabled. + +The Dashboard is a good place to get an overview of the security vulnerabilities in your projects. +You can also drill down into a vulnerability and get extra information, see which +project it comes from, the file it's in, and various metadata to help you analyze +the risk. You can also action these vulnerabilities by creating an issue for them, or by dismissing them. + +Having your vulnerabilities in GitLab allows you to keep track of them and action them, all in the same application. + +## Use cases + +You want to measure how secure your projects are without having to look into +each one separately. + +## Supported features + +The group security dashboard supports [SAST](../../project/merge_requests/sast.md), and [Dependency Scanning](../../project/merge_requests/dependency_scanning.md) reports. + +## Requirements + +To use the group security dashboard: + +1. At least one project inside a group must be configured with + [Static Application Security Testing](../../project/merge_requests/sast.md), or [Dependency Scanning](../../project/merge_requests/dependency_scanning.md). +2. The configured jobs must use the [new `reports` syntax](../../../ci/yaml/README.md#artifactsreports) (see an [example job](../../../ci/examples/sast.md)). +3. [GitLab Runner](https://docs.gitlab.com/runner/) 11.5 or above must be used to execute the jobs. + +## Keeping the dashboard up to date + +Vulnerabilities are spotted during CI/CD pipelines, so having up-to-date results +depends on how often security jobs are run. + +In order to have the latest results displayed in the dashboard, you can +[schedule a daily pipeline](../../project/pipelines/schedules.md), so reports +are created even if no code change happens. + +## Viewing the vulnerabilities + +First, navigate to the Security Dashboard found under your group's +**Overview > Security Dashboard**. + +Once you're on the dashboard, at the top you should see a series of filters for: + +- Severity +- Report type +- Project + +Selecting one or more of these will filter +the results in the sectons below. The first section is an overview of all the +vulnerabilities, grouped by severity. Underneath these overviews is a timeline +chart that shows how many open vulnerabilities you had at various points in time. +You can hover over the chart to get more details about the open vulnerabilities +at that time. + + + +Finally, there is a list of all the vulnerabilities in the group, sorted by severity. +In that list, you can see the severity of the vulnerability, its name, its +confidence (likelihood of the vulnerability to be a positive one), and the project +it's from. + +If you hover over a row, there will appear some actions you can take: + +- "More info" +- "Create issue" +- "Dismiss vulnerability" + +### Getting more information for a vulnerability + +Clicking the "More info" button opens a modal with more information about the +selected vulnerability where you can get a better description, as well as the +file it came from, and a possible solution. You get access to the +["Dismiss vulnerability"](#dismissing-a-vulnerability) and +["Create issue"](#creating-an-issue-for-a-vulnerability) buttons inside this +modal as well. + + + +### Creating an issue for a vulnerability + +You can create an issue for a vulnerability by selecting the "Create issue" +button from the action buttons to the right of a vulnerability row. +This will create an issue on the project this vulnerability came from and pre-fill +it with some useful information. + +Once the issue is created, you will be redirected to it so you can edit, assign, +or comment on it. Upon returning to the dashboard you'll see that the vulnerability +will now have an associated issue next to the name. + + + +You can get the same result if you select the **Create issue** button from inside +the "More info" modal. + +### Dismissing a vulnerability + +You can also dismiss vulnerabilities by clicking the "Dismiss vulnerability" button. +This will dismiss the vulnerability and re-render it to reflect its dismissed state. +If you wish to undo this dismissal, you can click the "Undo dismiss" button. + +You can get the same behaviour if you dismiss a vulnerability from within the +"More info" modal. diff --git a/doc/user/img/color_inline_colorchip_render_gfm.png b/doc/user/img/color_inline_colorchip_render_gfm.png Binary files differindex fed8ca5c34b..6f93dbeeb10 100644 --- a/doc/user/img/color_inline_colorchip_render_gfm.png +++ b/doc/user/img/color_inline_colorchip_render_gfm.png diff --git a/doc/user/img/task_list_ordered_render_gfm.png b/doc/user/img/task_list_ordered_render_gfm.png Binary files differindex 0905a8378be..98ec791e958 100644 --- a/doc/user/img/task_list_ordered_render_gfm.png +++ b/doc/user/img/task_list_ordered_render_gfm.png diff --git a/doc/user/img/unordered_check_list_render_gfm.png b/doc/user/img/unordered_check_list_render_gfm.png Binary files differindex ccdeab6e62c..2ce0fb95645 100644 --- a/doc/user/img/unordered_check_list_render_gfm.png +++ b/doc/user/img/unordered_check_list_render_gfm.png diff --git a/doc/user/index.md b/doc/user/index.md index b84879601ff..1b38a0eb466 100644 --- a/doc/user/index.md +++ b/doc/user/index.md @@ -64,6 +64,7 @@ With GitLab Enterprise Edition, you can also: - [Lock files](https://docs.gitlab.com/ee/user/project/file_lock.html) to prevent conflicts - View the current health and status of each CI environment running on Kubernetes with [Deploy Boards](https://docs.gitlab.com/ee/user/project/deploy_boards.html) - Leverage continuous delivery method with [Canary Deployments](https://docs.gitlab.com/ee/user/project/canary_deployments.html) +- Scan your code for vulnerabilities and [display them in merge requests](project/merge_requests/sast.md). You can also [integrate](project/integrations/project_services.md) GitLab with numerous third-party applications, such as Mattermost, Microsoft Teams, Trello, @@ -173,3 +174,8 @@ Learn what is [Git](../topics/git/index.md) and its best practices. ## Instance statistics See [various statistics](instance_statistics/index.md) of your GitLab instance. + +## Operations Dashboard + +See [Operations Dashboard](operations_dashboard/index.md) for a summary of each +project's operational health. diff --git a/doc/user/instance_statistics/convdev.md b/doc/user/instance_statistics/convdev.md index 2c9e0ecbf65..47a9f36071e 100644 --- a/doc/user/instance_statistics/convdev.md +++ b/doc/user/instance_statistics/convdev.md @@ -21,6 +21,7 @@ Your overall index score is an average of all your feature score percentages - t The page also provides helpful links to articles and GitLab docs, to help you improve your scores. + Usage ping data is aggregated on GitLab's servers for analysis. Your usage information is **not sent** to any other GitLab instances. If you have just started using GitLab, it may take a few weeks for data to be collected before this feature is available. diff --git a/doc/user/markdown.md b/doc/user/markdown.md index a7a87773eec..58f46a468c0 100644 --- a/doc/user/markdown.md +++ b/doc/user/markdown.md @@ -20,6 +20,7 @@ You can use GFM in the following areas: - Snippets (the snippet must be named with a `.md` extension) - Wiki pages - Markdown documents inside repositories +- Epics You can also use other rich text files in GitLab. You might have to install a dependency to do so. Please see the [`github-markup` gem readme](https://github.com/gitlabhq/markup#markups) for more information. @@ -323,6 +324,7 @@ GFM will recognize the following: | `#12345` | issue | | `!123` | merge request | | `$123` | snippet | +| `&123` | epic | | `~123` | label by ID | | `~bug` | one-word label by name | | `~"feature request"` | multi-word label by name | @@ -343,6 +345,7 @@ GFM also recognizes certain cross-project references: | `namespace/project%123` | project milestone | | `namespace/project$123` | snippet | | `namespace/project@9ba12248` | specific commit | +| `group1/subgroup&123` | epic | | `namespace/project@9ba12248...b19a04f5` | commit range comparison | | `namespace/project~"Some label"` | issues with given label | diff --git a/doc/user/operations_dashboard/img/index_change_default_dashboard.png b/doc/user/operations_dashboard/img/index_change_default_dashboard.png Binary files differnew file mode 100644 index 00000000000..bfb67296399 --- /dev/null +++ b/doc/user/operations_dashboard/img/index_change_default_dashboard.png diff --git a/doc/user/operations_dashboard/img/index_operations_dashboard_top_bar_icon.png b/doc/user/operations_dashboard/img/index_operations_dashboard_top_bar_icon.png Binary files differnew file mode 100644 index 00000000000..9d6a509ea72 --- /dev/null +++ b/doc/user/operations_dashboard/img/index_operations_dashboard_top_bar_icon.png diff --git a/doc/user/operations_dashboard/img/index_operations_dashboard_with_projects.png b/doc/user/operations_dashboard/img/index_operations_dashboard_with_projects.png Binary files differnew file mode 100644 index 00000000000..b4ae95fdcfa --- /dev/null +++ b/doc/user/operations_dashboard/img/index_operations_dashboard_with_projects.png diff --git a/doc/user/operations_dashboard/index.md b/doc/user/operations_dashboard/index.md new file mode 100644 index 00000000000..abae2302a27 --- /dev/null +++ b/doc/user/operations_dashboard/index.md @@ -0,0 +1,38 @@ +# Operations Dashboard **[ULTIMATE]** + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/issues/5781) +in [GitLab Ultimate](https://about.gitlab.com/pricing/) 11.5. + +The Operations Dashboard provides a summary of each project's operational health, +including pipeline and alert status. + +The dashboard can be accessed via the top bar, by clicking on the new +dashboard icon: + + + +The Operations Dashboard can also be made the default GitLab dashboard shown when +you sign in. To make it the default: + +1. Go to **User Settings > Preferences**. +1. In the **Behaviour** section, select **Operations Dashboard** from the **Default dashboard** dropdown. +1. Click the **Save changes** button. + + + +## Adding a project to the dashboard + +NOTE: **Note:** +For GitLab.com, the Operations Dashboard is available for free for public projects. +If your project is private, the group it belongs to must have a +[Gold](https://about.gitlab.com/pricing/) plan. + +To add a project to the dashboard: + +1. Search for a project using the **Search your projects** field. +1. Click the **Add projects** button. + +Once added, the dashboard will display the number of active alerts, +last commit, pipeline status, and when it was last deployed. + + diff --git a/doc/user/permissions.md b/doc/user/permissions.md index 833b9b66102..c5c8946ecce 100644 --- a/doc/user/permissions.md +++ b/doc/user/permissions.md @@ -36,8 +36,7 @@ The following table depicts the various user permission levels in a project. | Create confidential issue | ✓ [^1] | ✓ | ✓ | ✓ | ✓ | | View confidential issues | (✓) [^2] | ✓ | ✓ | ✓ | ✓ | | Leave comments | ✓ [^1] | ✓ | ✓ | ✓ | ✓ | -| Lock issue discussions | | ✓ | ✓ | ✓ | ✓ | -| Lock merge request discussions | | | ✓ | ✓ | ✓ | +| See related issues | ✓ | ✓ | ✓ | ✓ | ✓ | | See a list of jobs | ✓ [^3] | ✓ | ✓ | ✓ | ✓ | | See a job log | ✓ [^3] | ✓ | ✓ | ✓ | ✓ | | Download and browse job artifacts | ✓ [^3] | ✓ | ✓ | ✓ | ✓ | @@ -80,11 +79,12 @@ The following table depicts the various user permission levels in a project. | Update a container registry | | | ✓ | ✓ | ✓ | | Remove a container registry image | | | ✓ | ✓ | ✓ | | Create/edit/delete project milestones | | | ✓ | ✓ | ✓ | -| View approved/blacklisted licenses **[ULTIMATE]** | | | ✓ | ✓ | ✓ | +| View approved/blacklisted licenses **[ULTIMATE]** | ✓ | ✓ | ✓ | ✓ | ✓ | | Use security dashboard **[ULTIMATE]** | | | ✓ | ✓ | ✓ | | Dismiss vulnerability **[ULTIMATE]** | | | ✓ | ✓ | ✓ | | Apply code change suggestions | | | ✓ | ✓ | ✓ | | Use environment terminals | | | | ✓ | ✓ | +| Run Web IDE's Interactive Web Terminals **[ULTIMATE ONLY]** | | | | ✓ | ✓ | | Add new team members | | | | ✓ | ✓ | | Push to protected branches | | | | ✓ | ✓ | | Enable/disable branch protection | | | | ✓ | ✓ | @@ -250,9 +250,9 @@ Auditor users are given read-only access to all projects, groups, and other resources on the GitLab instance. An Auditor user should be able to access all projects and groups of a GitLab instance -with the permissions described on the documentation on [auditor users permissions](https://docs.gitlab.com/ee/administration/auditor_users.html#permissions-and-restrictions-of-an-auditor-user). +with the permissions described on the documentation on [auditor users permissions](../administration/auditor_users.md#permissions-and-restrictions-of-an-auditor-user). -[Read more about Auditor users.](https://docs.gitlab.com/ee/administration/auditor_users.html) +[Read more about Auditor users.](../administration/auditor_users.md) ## Project features @@ -337,7 +337,7 @@ for details about the pipelines security model. ## LDAP users permissions Since GitLab 8.15, LDAP user permissions can now be manually overridden by an admin user. -Read through the documentation on [LDAP users permissions](https://docs.gitlab.com/ee/articles/how_to_configure_ldap_gitlab_ee/index.html#updating-user-permissions-new-feature) to learn more. +Read through the documentation on [LDAP users permissions](../administration/auth/how_to_configure_ldap_gitlab_ee/index.md) to learn more. [^1]: On public and internal projects, all users are able to perform this action [^2]: Guest users can only view the confidential issues they created themselves diff --git a/doc/user/project/canary_deployments.md b/doc/user/project/canary_deployments.md new file mode 100644 index 00000000000..cce862b0911 --- /dev/null +++ b/doc/user/project/canary_deployments.md @@ -0,0 +1,71 @@ +# Canary Deployments **[PREMIUM]** + +> [Introduced][ee-1659] in [GitLab Premium][eep] 9.1. + +A popular [Continuous Integration](https://en.wikipedia.org/wiki/Continuous_integration) +strategy, where a small portion of the fleet is updated to the new version of +your application. + +## Overview + +When embracing [Continuous Delivery][cd-blog], a company needs to decide what +type of deployment strategy to use. One of the most popular strategies is canary +deployments, where a small portion of the fleet is updated to the new version +first. This subset, the canaries, then serve as the proverbial +[canary in the coal mine](https://en.wiktionary.org/wiki/canary_in_a_coal_mine). + +If there is a problem with the new version of the application, only a small +percentage of users are affected and the change can either be fixed or quickly +reverted. + +Leveraging [Kubernetes' Canary deployments][kube-canary], visualize your canary +deployments right inside the [Deploy Board], without the need to leave GitLab. + +## Use cases + +Canary deployments can be used when you want to ship features to only a portion of +your pods fleet and watch their behavior as a percentage of your user base +visits the temporarily deployed feature. If all works well, you can deploy the +feature to production knowing that it won't cause any problems. + +Canary deployments are also especially useful for backend refactors, performance +improvements, or other changes where the user interface doesn't change, but you +want to make sure the performance stays the same, or improves. Developers need +to be careful when using canaries with user-facing changes, because by default, +requests from the same user will be randomly distributed between canary and +non-canary pods, which could result in confusion or even errors. If needed, you +may want to consider [setting `service.spec.sessionAffinity` to `ClientIP` in +your Kubernetes service definitions][kube-net], but that is beyond the scope of +this document. + +## Enabling Canary Deployments + +Canary deployments require that you properly configure Deploy Boards: + +1. Follow the steps to [enable Deploy Boards](deploy_boards.md#enabling-deploy-boards). +1. To track canary deployments you need to label your Kubernetes deployments and + pods with `track: canary`. To get started quickly, you can use the [Auto Deploy] + template for canary deployments that GitLab provides. + +Depending on the deploy, the label should be either `stable` or `canary`. +Usually, `stable` and blank or missing label means the same thing, and `canary` +or any other track means canary/temporary. +This allows GitLab to discover whether deployment is stable or canary (temporary). + +Once all of the above are set up and the pipeline has run at least once, +navigate to the environments page under **Pipelines > Environments**. +As the pipeline executes Deploy Boards will clearly mark canary pods, enabling +quick and easy insight into the status of each environment and deployment. + +Canary deployments are marked with a yellow dot in the Deploy Board so that you +can easily notice them. + + + +[autodeploy]: ../../ci/autodeploy/index.md "GitLab Autodeploy" +[eep]: https://about.gitlab.com/pricing/ +[ee-1659]: https://gitlab.com/gitlab-org/gitlab-ee/issues/1659 +[kube-canary]: https://kubernetes.io/docs/concepts/cluster-administration/manage-deployment/#canary-deployments +[deploy board]: deploy_boards.md +[cd-blog]: https://about.gitlab.com/2016/08/05/continuous-integration-delivery-and-deployment-with-gitlab/ +[kube-net]: https://kubernetes.io/docs/concepts/services-networking/service/#virtual-ips-and-service-proxies diff --git a/doc/user/project/ci_cd_for_external_repo.md b/doc/user/project/ci_cd_for_external_repo.md new file mode 100644 index 00000000000..a92d3a2c308 --- /dev/null +++ b/doc/user/project/ci_cd_for_external_repo.md @@ -0,0 +1,5 @@ +--- +redirect_to: '../../ci/ci_cd_for_external_repos/index.md' +--- + +This document was moved to [another location](../../ci/ci_cd_for_external_repos/index.md). diff --git a/doc/user/project/clusters/img/k8s_cluster_monitoring.png b/doc/user/project/clusters/img/k8s_cluster_monitoring.png Binary files differnew file mode 100644 index 00000000000..d0adec4520f --- /dev/null +++ b/doc/user/project/clusters/img/k8s_cluster_monitoring.png diff --git a/doc/user/project/clusters/img/kubernetes_pod_logs.png b/doc/user/project/clusters/img/kubernetes_pod_logs.png Binary files differnew file mode 100644 index 00000000000..e664a47386a --- /dev/null +++ b/doc/user/project/clusters/img/kubernetes_pod_logs.png diff --git a/doc/user/project/clusters/img/pod_logs_deploy_board.png b/doc/user/project/clusters/img/pod_logs_deploy_board.png Binary files differnew file mode 100644 index 00000000000..7f83382968b --- /dev/null +++ b/doc/user/project/clusters/img/pod_logs_deploy_board.png diff --git a/doc/user/project/clusters/index.md b/doc/user/project/clusters/index.md index e84c3ca4bef..81e7ad4d5ac 100644 --- a/doc/user/project/clusters/index.md +++ b/doc/user/project/clusters/index.md @@ -479,7 +479,7 @@ differentiate the new cluster with the rest. When adding more than one Kubernetes cluster to your project, you need to differentiate them with an environment scope. The environment scope associates clusters with [environments](../../../ci/environments.md) similar to how the -[environment-specific variables](https://docs.gitlab.com/ee/ci/variables/README.html#limiting-environment-scopes-of-variables-premium) work. +[environment-specific variables](../../../ci/variables/README.md#limiting-environment-scopes-of-variables-premium) work. The default environment scope is `*`, which means all jobs, regardless of their environment, will use that cluster. Each scope can only be used by a single @@ -576,7 +576,7 @@ credentials. When [Prometheus is deployed](#installing-applications), GitLab will automatically monitor the cluster's health. At the top of the cluster settings page, CPU and Memory utilization is displayed, along with the total amount available. Keeping an eye on cluster resources can be important, if the cluster runs out of memory pods may be shutdown or fail to start. - + ## Enabling or disabling the Kubernetes cluster integration @@ -607,7 +607,7 @@ and add a Kubernetes cluster again. ## View Kubernetes pod logs from GitLab **[ULTIMATE]** Learn how to easily -[view the logs of running pods in connected Kubernetes clusters](https://docs.gitlab.com/ee/user/project/clusters/kubernetes_pod_logs.html). +[view the logs of running pods in connected Kubernetes clusters](kubernetes_pod_logs.md). ## What you can get with the Kubernetes integration @@ -621,7 +621,7 @@ displaying the status of the pods in the deployment. Developers and other teammates can view the progress and status of a rollout, pod by pod, in the workflow they already use without any need to access Kubernetes. -[Read more about Deploy Boards](https://docs.gitlab.com/ee/user/project/deploy_boards.html) +[Read more about Deploy Boards](../deploy_boards.md) ### Canary Deployments **[PREMIUM]** @@ -629,7 +629,7 @@ Leverage [Kubernetes' Canary deployments](https://kubernetes.io/docs/concepts/cl and visualize your canary deployments right inside the Deploy Board, without the need to leave GitLab. -[Read more about Canary Deployments](https://docs.gitlab.com/ee/user/project/canary_deployments.html) +[Read more about Canary Deployments](../canary_deployments.md) ### Kubernetes monitoring diff --git a/doc/user/project/clusters/kubernetes_pod_logs.md b/doc/user/project/clusters/kubernetes_pod_logs.md new file mode 100644 index 00000000000..e0815b5b553 --- /dev/null +++ b/doc/user/project/clusters/kubernetes_pod_logs.md @@ -0,0 +1,32 @@ +# Kubernetes Pod Logs **[ULTIMATE]** + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/issues/4752) in [GitLab Ultimate](https://about.gitlab.com/pricing/) 11.0. + +GitLab makes it easy to view the logs of running pods in [connected Kubernetes clusters](index.md). +By displaying the logs directly in GitLab, developers can avoid having to manage console tools or jump to a different interface. + +## Overview + +[Kubernetes](https://kubernetes.io) pod logs can be viewed directly within GitLab. Logs can be displayed by clicking on a specific pod from [Deploy Boards](../deploy_boards.md): + +1. Go to **Operations > Environments** and find the environment which contains the desired pod, like `production`. +1. On the **Environments** page, you should see the status of the environment's pods with [Deploy Boards](../deploy_boards.md). +1. When mousing over the list of pods, a tooltip will appear with the exact pod name and status. + +1. Click on the desired pod to bring up the logs view, which will contain the last 500 lines for that pod. Support for pods with multiple containers is coming [in a future release](https://gitlab.com/gitlab-org/gitlab-ee/issues/6502). + + +## Requirements + +There are a few key requirements in order to be able to use Pod Logs: + +1. [Create or connect a Kubernetes cluster to the project](index.md#adding-and-creating-a-new-gke-cluster-via-gitlab) +1. A GitLab Runner. If one is not available, they can be easily [deployed](index.md#installing-applications). +1. Ensure a Kubernetes label of `app: $CI_ENVIRONMENT_SLUG` is applied to the + deployments, replica sets, and pods, where `$CI_ENVIRONMENT_SLUG` is the value + of the [CI variable](../../../ci/variables/README.md#predefined-environment-variables). + This needed to determine the environment a pod belongs to + when is more than one. These resources should be + contained in the namespace defined in the Kubernetes service setting. + The [Auto DevOps](../../../topics/autodevops/index.md) `.gitlab-ci.yml` template has predefined + stages and commands to use, and automatically applies the labeling. diff --git a/doc/user/project/clusters/runbooks/img/authorize-jupyter.png b/doc/user/project/clusters/runbooks/img/authorize-jupyter.png Binary files differindex 84cce311483..f3e868de802 100644 --- a/doc/user/project/clusters/runbooks/img/authorize-jupyter.png +++ b/doc/user/project/clusters/runbooks/img/authorize-jupyter.png diff --git a/doc/user/project/clusters/runbooks/img/gitlab-variables.png b/doc/user/project/clusters/runbooks/img/gitlab-variables.png Binary files differindex 1d338f063a9..7c9c9099851 100644 --- a/doc/user/project/clusters/runbooks/img/gitlab-variables.png +++ b/doc/user/project/clusters/runbooks/img/gitlab-variables.png diff --git a/doc/user/project/clusters/runbooks/img/helm-install.png b/doc/user/project/clusters/runbooks/img/helm-install.png Binary files differindex 003e482e756..e67cf317ec1 100644 --- a/doc/user/project/clusters/runbooks/img/helm-install.png +++ b/doc/user/project/clusters/runbooks/img/helm-install.png diff --git a/doc/user/project/clusters/runbooks/img/ingress-install.png b/doc/user/project/clusters/runbooks/img/ingress-install.png Binary files differindex 7edc11d5b45..08256a65138 100644 --- a/doc/user/project/clusters/runbooks/img/ingress-install.png +++ b/doc/user/project/clusters/runbooks/img/ingress-install.png diff --git a/doc/user/project/clusters/runbooks/img/jupyterhub-install.png b/doc/user/project/clusters/runbooks/img/jupyterhub-install.png Binary files differindex 75c6028a763..784e508ff25 100644 --- a/doc/user/project/clusters/runbooks/img/jupyterhub-install.png +++ b/doc/user/project/clusters/runbooks/img/jupyterhub-install.png diff --git a/doc/user/project/clusters/runbooks/img/postgres-query.png b/doc/user/project/clusters/runbooks/img/postgres-query.png Binary files differindex 04315d54d5e..dbb5a62ebba 100644 --- a/doc/user/project/clusters/runbooks/img/postgres-query.png +++ b/doc/user/project/clusters/runbooks/img/postgres-query.png diff --git a/doc/user/project/clusters/runbooks/img/sample-runbook.png b/doc/user/project/clusters/runbooks/img/sample-runbook.png Binary files differindex 70011202bf0..545d34fd4ad 100644 --- a/doc/user/project/clusters/runbooks/img/sample-runbook.png +++ b/doc/user/project/clusters/runbooks/img/sample-runbook.png diff --git a/doc/user/project/code_owners.md b/doc/user/project/code_owners.md new file mode 100644 index 00000000000..a4937e6cf6b --- /dev/null +++ b/doc/user/project/code_owners.md @@ -0,0 +1,85 @@ +# Code Owners **[STARTER]** + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/6916) +in [GitLab Starter](https://about.gitlab.com/pricing/) 11.3. + +You can use a `CODEOWNERS` file to specify users that are responsible +for certain files in a repository. + +You can choose and add the `CODEOWNERS` file in three places: + +- to the root directory of the repository +- inside the `.gitlab/` directory +- inside the `docs/` directory + +The `CODEOWNERS` file is scoped to a branch, which means that with the +introduction of new files, the person adding the new content can +specify themselves as a code owner, all before the new changes +get merged to the default branch. + +When a file matches multiple entries in the `CODEOWNERS` file, +the users from all entries are displayed on the blob page of +the given file. + +## The syntax of Code Owners files + +Files can be specified using the same kind of patterns you would use +in the `.gitignore` file followed by the `@username` or email of one +or more users that should be owners of the file. + +The order in which the paths are defined is significant: the last +pattern that matches a given path will be used to find the code +owners. + +Starting a line with a `#` indicates a comment. This needs to be +escaped using `\#` to address files for which the name starts with a +`#`. + +Example `CODEOWNERS` file: + +``` +# This is an example code owners file, lines starting with a `#` will +# be ignored. + +# app/ @commented-rule + +# We can specifiy a default match using wildcards: +* @default-codeowner + +# Rules defined later in the file take precedence over the rules +# defined before. +# This will match all files for which the file name ends in `.rb` +*.rb @ruby-owner + +# Files with a `#` can still be accesssed by escaping the pound sign +\#file_with_pound.rb @owner-file-with-pound + +# Multiple codeowners can be specified, separated by whitespace +CODEOWNERS @multiple @owners @tab-separated + +# Both usernames or email addresses can be used to match +# users. Everything else will be ignored. For example this will +# specify `@legal` and a user with email `janedoe@gitlab.com` as the +# owner for the LICENSE file +LICENSE @legal this_does_not_match janedoe@gitlab.com + +# Ending a path in a `/` will specify the code owners for every file +# nested in that directory, on any level +/docs/ @all-docs + +# Ending a path in `/*` will specify code owners for every file in +# that directory, but not nested deeper. This will match +# `docs/index.md` but not `docs/projects/index.md` +/docs/* @root-docs + +# This will make a `lib` directory nested anywhere in the repository +# match +lib/ @lib-owner + +# This will only match a `config` directory in the root of the +# repository +/config/ @config-owner + +# If the path contains spaces, these need to be escaped like this: +path\ with\ spaces/ @space-owner +``` diff --git a/doc/user/project/deploy_boards.md b/doc/user/project/deploy_boards.md new file mode 100644 index 00000000000..01fb14f1e66 --- /dev/null +++ b/doc/user/project/deploy_boards.md @@ -0,0 +1,127 @@ +# Deploy Boards **[PREMIUM]** + +> [Introduced][ee-1589] in [GitLab Premium][ee] 9.0. + +GitLab's Deploy Boards offer a consolidated view of the current health and +status of each CI [environment] running on [Kubernetes], displaying the status +of the pods in the deployment. Developers and other teammates can view the +progress and status of a rollout, pod by pod, in the workflow they already use +without any need to access Kubernetes. + +## Overview + +With Deploy Boards you can gain more insight into deploys with benefits such as: + +- Following a deploy from the start, not just when it's done +- Watching the rollout of a build across multiple servers +- Finer state detail (Waiting, Deploying, Finished, Unknown) +- See [Canary Deployments](canary_deployments.md) + +Here's an example of a Deploy Board of the production environment. + + + +The squares represent pods in your Kubernetes cluster that are associated with +the given environment. Hovering above each square you can see the state of a +deploy rolling out. The percentage is the percent of the pods that are updated +to the latest release. + +Since Deploy Boards are tightly coupled with Kubernetes, there is some required +knowledge. In particular you should be familiar with: + +- [Kubernetes pods](https://kubernetes.io/docs/user-guide/pods) +- [Kubernetes labels](https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/) +- [Kubernetes namespaces](https://kubernetes.io/docs/user-guide/namespaces/) +- [Kubernetes canary deployments](https://kubernetes.io/docs/concepts/cluster-administration/manage-deployment/#canary-deployments) + +## Use cases + +Since the Deploy Board is a visual representation of the Kubernetes pods for a +specific environment, there are lot of uses cases. To name a few: + +- You want to promote what's running in staging, to production. You go to the + environments list, verify that what's running in staging is what you think is + running, then click on the [manual action] to deploy to production. +- You trigger a deploy, and you've got lots of containers to upgrade so you know + it'll take a while (you've also throttled your deploy to only take down X + containers at a time). But you need to tell someone when it's deployed, so you + go to the environments list, look at the production environment to see what + the progress is in real-time as each pod is rolled. +- You get a report that something is weird in production, so you look at the + production environment to see what is running, and if a deploy is ongoing or + stuck or failed. +- You've got an MR that looks good, but you want to run it on staging because + staging is set up in some way closer to production. You go to the environment + list, find the [Review App][review apps] you're interested in, and click the + manual action to deploy it to staging. + +## Enabling Deploy Boards + +To display the Deploy Boards for a specific [environment] you should: + +1. Have a Kubernetes cluster up and running. + + NOTE: **Running on OpenShift:** + If you are using OpenShift, ensure that you're using the `Deployment` resource + instead of `DeploymentConfiguration`, otherwise the Deploy Boards won't render + correctly. For more information, read the + [OpenShift docs](https://docs.openshift.com/container-platform/3.7/dev_guide/deployments/kubernetes_deployments.html#kubernetes-deployments-vs-deployment-configurations) + and [GitLab issue #4584](https://gitlab.com/gitlab-org/gitlab-ee/issues/4584). + +1. [Configure GitLab Runner][runners] with the [Docker][docker-exec] or + [Kubernetes][kube-exec] executor. +1. Configure the [Kubernetes service][kube-service] in your project for the + cluster. The Kubernetes namespace is of particular note as you will need it + for your deployment scripts (exposed by the `KUBE_NAMESPACE` env variable). +1. Ensure a Kubernetes label of `app: $CI_ENVIRONMENT_SLUG` is applied to the + deployments, replica sets, and pods, where `$CI_ENVIRONMENT_SLUG` the value + of the CI variable. This is so we can lookup the proper environment in a + cluster/namespace which may have more than one. These resources should be + contained in the namespace defined in the Kubernetes service setting. + You can use an [Autodeploy] `.gitlab-ci.yml` template which has predefined + stages and commands to use, and automatically applies the labeling. + Each project will need to have a unique namespace in Kubernetes as well. + The image below demonstrates how this is shown inside Kubernetes. + +  + +Once all of the above are set up and the pipeline has run at least once, +navigate to the environments page under **Operations > Environments**. + +Deploy Boards are visible by default. You can explicitly click +the triangle next to their respective environment name in order to hide them. +GitLab will then query Kubernetes for the state of each node (e.g., waiting, +deploying, finished, unknown), and the Deploy Board status will finally appear. + +GitLab will only display a Deploy Board for top-level environments. Foldered +environments like `review/*` (usually used for [Review Apps]) won't have a +Deploy Board attached to them. + +## Canary Deployments + +A popular CI strategy, where a small portion of the fleet is updated to the new +version of your application. + +[Read more about Canary Deployments.](canary_deployments.md) + +## Further reading + +- [GitLab Autodeploy][autodeploy] +- [GitLab CI environment variables][variables] +- [Environments and deployments][environment] +- [Kubernetes deploy example][kube-deploy] + +[ee-1589]: https://gitlab.com/gitlab-org/gitlab-ee/issues/1589 "Deploy Boards initial issue" +[ee]: https://about.gitlab.com/pricing/ "GitLab Enterprise Edition landing page" +[kube-deploy]: https://gitlab.com/gitlab-examples/kubernetes-deploy "Kubernetes deploy example project" +[kubernetes]: https://kubernetes.io "Kubernetes website" +[environment]: ../../ci/environments.md "Environments and deployments documentation" +[docker-exec]: https://docs.gitlab.com/runner/executors/docker.html "GitLab Runner Docker executor" +[kube-exec]: https://docs.gitlab.com/runner/executors/kubernetes.html "GitLab Runner Kubernetes executor" +[kube-service]: integrations/kubernetes.md "Kubernetes project service" +[review apps]: ../../ci/review_apps/index.md "Review Apps documentation" +[variables]: ../../ci/variables/README.md "GitLab CI variables" +[autodeploy]: ../../ci/autodeploy/index.md "GitLab Autodeploy" +[kube-image]: https://gitlab.com/gitlab-examples/kubernetes-deploy/container_registry "Kubernetes deploy Container Registry" +[manual action]: ../../ci/yaml/README.md#manual-actions +[runners]: ../../ci/runners/README.md diff --git a/doc/user/project/description_templates.md b/doc/user/project/description_templates.md index 08647caaf07..e230444fa67 100644 --- a/doc/user/project/description_templates.md +++ b/doc/user/project/description_templates.md @@ -2,8 +2,12 @@ >[Introduced][ce-4981] in GitLab 8.11. +We all know that a properly submitted issue is more likely to be addressed in +a timely manner by the developers of a project. + Description templates allow you to define context-specific templates for issue -and merge request description fields for your project. +and merge request description fields for your project, as well as help filter +out a lot of unnecessary noise from issues. ## Overview @@ -18,6 +22,18 @@ Description templates must be written in [Markdown](../markdown.md) and stored in your project's repository under a directory named `.gitlab`. Only the templates of the default branch will be taken into account. +## Use-cases + +- Add a template to be used in every issue for a specific project, +giving instructions and guidelines, requiring for information specific to that subject. +For example, if you have a project for tracking new blog posts, you can require the +title, outlines, author name, author social media information, etc. +- Following the previous example, you can make a template for every MR submitted +with a new blog post, requiring information about the post date, frontmatter data, +images guidelines, link to the related issue, reviewer name, etc. +- You can also create issues and merge request templates for different +stages of your workflow, e.g., feature proposal, feature improvement, bug report, etc. + ## Creating issue templates Create a new Markdown (`.md`) file inside the `.gitlab/issue_templates/` @@ -39,6 +55,32 @@ changes you made after picking the template and return it to its initial status.  +## Setting a default template for issues and merge requests **[STARTER]** + +> **Notes:** +> - This feature was introduced before [description templates](#overview) and is +> available in [GitLab Starter][products]. It can be enabled +> in the project's settings. +> - Templates for issues were [introduced][ee-28] in GitLab EE 8.1. +> - Templates for merge requests were [introduced][ee-7478ece] in GitLab EE 6.9. + +The visibility of issues and/or merge requests should be set to either "Everyone +with access" or "Only team members" in your project's **Settings** otherwise the +template text areas won't show. This is the default behavior so in most cases +you should be fine. + +Go to your project's **Settings** and fill in the "Default description template +for issues" and "Default description template for merge requests" text areas +for issues and merge requests respectively. Since GitLab issues and merge +request support [Markdown](../markdown.md), you can use special markup like +headings, lists, etc. + + + +After you add the description, hit **Save changes** for the settings to take +effect. Now, every time a new issue or merge request is created, it will be +pre-filled with the text you entered in the template(s). + ## Description template example We make use of Description Templates for Issues and Merge Requests within the GitLab Community Edition project. Please refer to the [`.gitlab` folder][gitlab-ce-templates] for some examples. @@ -93,3 +135,6 @@ Possible fixes [ce-4981]: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/4981 [gitlab-ce-templates]: https://gitlab.com/gitlab-org/gitlab-ce/tree/master/.gitlab +[ee-28]: https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/28 "Merge Request for adding issues template" +[ee-7478ece]: https://gitlab.com/gitlab-org/gitlab-ee/commit/7478ece8b48e80782b5465b96c79f85cc91d391b "Commit that introduced merge requests templates" +[products]: https://about.gitlab.com/pricing/ diff --git a/doc/user/project/file_lock.md b/doc/user/project/file_lock.md new file mode 100644 index 00000000000..541023692ea --- /dev/null +++ b/doc/user/project/file_lock.md @@ -0,0 +1,107 @@ +# File Locking **[PREMIUM]** + +> **Notes:** +> - [Introduced][ee-440] in [GitLab Premium][ee] 8.9. +> - This feature needs to have a license with the "File Lock" option enabled. If +> you are using Premium but you don't see the "Lock" button, +> ask your GitLab administrator. + +File Locking helps you avoid merge conflicts and better manage your binary files. +Lock any file or directory, make your changes, and then unlock it so another +member of the team can edit it. + +## Overview + +Working with multiple people on the same file can be a risk. Conflicts +when merging a non-text file are hard to overcome and will require a lot +of manual work to resolve. With GitLab Premium, File +Locking helps you avoid merge conflicts and better manage your binary +files by preventing everyone, except you, from modifying a specific file +or entire directory. + +## Use-cases + +The file locking feature is useful in situations when: + +- Multiple people are working on the same file and you want to avoid merge + conflicts. +- Your repository contains binary files in which situation there is no easy + way to tell the diff between yours and your colleagues' changes. +- Prevent design assets from being overwritten. + +Locked directories are locked recursively, which means that everything that +lies under them is also locked. + +## Permissions on file locking + +The user that locks a file or directory **is the only one** that can edit and +push their changes back to the repository where the locked objects are located. + +Locks can be created by any person who has [push access] to the repository; i.e., +Developer and higher level, and can be removed solely by their author and any +user with Maintainer permissions and above. + +If a file is locked and you are not the author of its locked state, a +pre-receive hook will reject your changes when you try to push. In the +following example, a user who has no permissions on the locked `.gitignore` +file will see the message below: + +```bash +Counting objects: 3, done. +Delta compression using up to 4 threads. +Compressing objects: 100% (3/3), done. +Writing objects: 100% (3/3), 320 bytes | 0 bytes/s, done. +Total 3 (delta 1), reused 0 (delta 0) +remote: GitLab: The path '.gitignore' is locked by Administrator +To https://example.com/gitlab-org/gitlab-ce.git + ! [remote rejected] master -> master (pre-receive hook declined) + error: failed to push some refs to 'https://example.com/gitlab-org/gitlab-ce.git' +``` + +Similarly, when a user that is not the author of the locked state of a file +accepts a merge request, an error message will appear stating that the file +is locked. + + + +## Locking a file or a directory + +>**Note:** +Locking only works for the default branch you have set in the project's settings +(usually `master`). + +To lock a file, navigate to the repository tree under the **Repository > Files** tab, +pick the file you want to lock and hit the "Lock" button. + + + +--- + +To lock an entire directory, look for the "Lock" link next to "History". + + + +--- + +After you lock a file or directory, it will appear as locked in the repository +view. + + + +## Unlocking a file or a directory + +To unlock a file or a directory, follow the same procedure as when you locked +them. For a detailed view of every existing lock, see the next section on +"Viewing and managing existing locks". + +## Viewing and managing existing locks + +To view or manage every existing lock, navigate to the +**Project > Repository > Locked Files** area. There, you can view all existing +locks and [remove the ones you have permission for](#permissions-on-file-locking). + + + +[ee-440]: https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/440 "File Lock" +[ee]: https://about.gitlab.com/pricing/ +[push access]: ../../user/permissions.md diff --git a/doc/user/project/img/assignee_lists.png b/doc/user/project/img/assignee_lists.png Binary files differnew file mode 100644 index 00000000000..f2660cd8f80 --- /dev/null +++ b/doc/user/project/img/assignee_lists.png diff --git a/doc/user/project/img/deploy_boards_canary_deployments.png b/doc/user/project/img/deploy_boards_canary_deployments.png Binary files differnew file mode 100644 index 00000000000..037b4a908ab --- /dev/null +++ b/doc/user/project/img/deploy_boards_canary_deployments.png diff --git a/doc/user/project/img/deploy_boards_kubernetes_label.png b/doc/user/project/img/deploy_boards_kubernetes_label.png Binary files differnew file mode 100644 index 00000000000..c6d90dc92df --- /dev/null +++ b/doc/user/project/img/deploy_boards_kubernetes_label.png diff --git a/doc/user/project/img/deploy_boards_landing_page.png b/doc/user/project/img/deploy_boards_landing_page.png Binary files differnew file mode 100644 index 00000000000..c9621a06860 --- /dev/null +++ b/doc/user/project/img/deploy_boards_landing_page.png diff --git a/doc/user/project/img/description_templates_default_settings.png b/doc/user/project/img/description_templates_default_settings.png Binary files differnew file mode 100644 index 00000000000..ab314e83d06 --- /dev/null +++ b/doc/user/project/img/description_templates_default_settings.png diff --git a/doc/user/project/img/file_lock.png b/doc/user/project/img/file_lock.png Binary files differnew file mode 100644 index 00000000000..33f96f20e91 --- /dev/null +++ b/doc/user/project/img/file_lock.png diff --git a/doc/user/project/img/file_lock_folders.png b/doc/user/project/img/file_lock_folders.png Binary files differnew file mode 100644 index 00000000000..5ff3d4d9dbd --- /dev/null +++ b/doc/user/project/img/file_lock_folders.png diff --git a/doc/user/project/img/file_lock_list.png b/doc/user/project/img/file_lock_list.png Binary files differnew file mode 100644 index 00000000000..2c276335c83 --- /dev/null +++ b/doc/user/project/img/file_lock_list.png diff --git a/doc/user/project/img/file_lock_merge_request_error_message.png b/doc/user/project/img/file_lock_merge_request_error_message.png Binary files differnew file mode 100644 index 00000000000..65bc3692da0 --- /dev/null +++ b/doc/user/project/img/file_lock_merge_request_error_message.png diff --git a/doc/user/project/img/file_lock_repository_view.png b/doc/user/project/img/file_lock_repository_view.png Binary files differnew file mode 100644 index 00000000000..8f4ef52aacd --- /dev/null +++ b/doc/user/project/img/file_lock_repository_view.png diff --git a/doc/user/project/img/labels_default.png b/doc/user/project/img/labels_default.png Binary files differindex 7a7fab611a4..221c5cf55a2 100644 --- a/doc/user/project/img/labels_default.png +++ b/doc/user/project/img/labels_default.png diff --git a/doc/user/project/img/labels_epic_sidebar.png b/doc/user/project/img/labels_epic_sidebar.png Binary files differnew file mode 100644 index 00000000000..f8162d89e9d --- /dev/null +++ b/doc/user/project/img/labels_epic_sidebar.png diff --git a/doc/user/project/img/labels_list.png b/doc/user/project/img/labels_list.png Binary files differindex 6878349fc0c..6940dde68bf 100644 --- a/doc/user/project/img/labels_list.png +++ b/doc/user/project/img/labels_list.png diff --git a/doc/user/project/img/project_security_dashboard.png b/doc/user/project/img/project_security_dashboard.png Binary files differnew file mode 100644 index 00000000000..3294e59e943 --- /dev/null +++ b/doc/user/project/img/project_security_dashboard.png diff --git a/doc/user/project/img/protected_branches_select_roles_and_users.png b/doc/user/project/img/protected_branches_select_roles_and_users.png Binary files differnew file mode 100644 index 00000000000..4f62b057cc7 --- /dev/null +++ b/doc/user/project/img/protected_branches_select_roles_and_users.png diff --git a/doc/user/project/img/protected_branches_select_roles_and_users_list.png b/doc/user/project/img/protected_branches_select_roles_and_users_list.png Binary files differnew file mode 100644 index 00000000000..519f2267496 --- /dev/null +++ b/doc/user/project/img/protected_branches_select_roles_and_users_list.png diff --git a/doc/user/project/img/service_desk_confirmation_email.png b/doc/user/project/img/service_desk_confirmation_email.png Binary files differnew file mode 100644 index 00000000000..e08dced1e38 --- /dev/null +++ b/doc/user/project/img/service_desk_confirmation_email.png diff --git a/doc/user/project/img/service_desk_disabled.png b/doc/user/project/img/service_desk_disabled.png Binary files differnew file mode 100644 index 00000000000..3ae64dcbe96 --- /dev/null +++ b/doc/user/project/img/service_desk_disabled.png diff --git a/doc/user/project/img/service_desk_enabled.png b/doc/user/project/img/service_desk_enabled.png Binary files differnew file mode 100644 index 00000000000..329348e4b52 --- /dev/null +++ b/doc/user/project/img/service_desk_enabled.png diff --git a/doc/user/project/img/service_desk_issue_tracker.png b/doc/user/project/img/service_desk_issue_tracker.png Binary files differnew file mode 100644 index 00000000000..485751fab0a --- /dev/null +++ b/doc/user/project/img/service_desk_issue_tracker.png diff --git a/doc/user/project/img/service_desk_nav_item.png b/doc/user/project/img/service_desk_nav_item.png Binary files differnew file mode 100644 index 00000000000..3420e355f67 --- /dev/null +++ b/doc/user/project/img/service_desk_nav_item.png diff --git a/doc/user/project/img/service_desk_reply.png b/doc/user/project/img/service_desk_reply.png Binary files differnew file mode 100644 index 00000000000..ae6ce31713b --- /dev/null +++ b/doc/user/project/img/service_desk_reply.png diff --git a/doc/user/project/img/service_desk_thread.png b/doc/user/project/img/service_desk_thread.png Binary files differnew file mode 100644 index 00000000000..8d8ac39cc5b --- /dev/null +++ b/doc/user/project/img/service_desk_thread.png diff --git a/doc/user/project/import/gemnasium.md b/doc/user/project/import/gemnasium.md new file mode 100644 index 00000000000..664c50e0494 --- /dev/null +++ b/doc/user/project/import/gemnasium.md @@ -0,0 +1,102 @@ +# Gemnasium **[ULTIMATE]** + +This guide describes how to migrate from Gemnasium.com to your own GitLab +instance or GitLab.com. + +## Why is Gemnasium.com closed? + +Gemnasium has been [acquired by GitLab](https://about.gitlab.com/press/releases/2018-01-30-gemnasium-acquisition.html) +in January 2018. Since May 15, 2018, the services provided by Gemnasium are no longer available. +The team behind Gemnasium has joined GitLab as the new Security Products team +and is working on a wider range of tools than just Dependency Scanning: +[SAST](../merge_requests/sast.md), +[DAST](../merge_requests/dast.md), +[Container Scanning](../merge_requests/container_scanning.md) and more. +If you want to continue monitoring your dependencies, see the +[Migrating to GitLab](#migrating-to-gitlab) section below. + +## What happened to my account? + +Your account has been automatically closed on May 15th, 2018. If you had a paid +subscription at that time, your card will be refunded on a pro rata temporis basis. +You may contact `gemnasium@gitlab.com` regarding your closed account. + +## Will my account/data be transferred to GitLab Inc.? + +All accounts and data have been deleted on May 15th, 2018. GitLab Inc. +doesn't know anything about your private data, nor your projects, and therefore +if they were vulnerable or not. GitLab Inc. takes personal information very seriously. + +## What happened to my badge? + +To avoid broken 404 images, all badges pointing to Gemnasium.com will be a +placeholder, inviting you to migrate to GitLab (and pointing to this page). + +## Migrating to GitLab + +Gemnasium has been ported and integrated directly into GitLab CI/CD. +You can still benefit from our dependency monitoring features, and it requires +some steps to migrate your projects. There is no automatic import since GitLab +doesn't know anything about any projects which existed on Gemnasium.com. +Security features are free for public (open-source) projects hosted on GitLab.com. + +### If your project is hosted on GitLab (https://gitlab.com / self-hosted) + +You're almost set! If you're already using +[Auto DevOps](../../../topics/autodevops/), you are already covered. +Otherwise, you must configure your `.gitlab-ci.yml` according to the +[dependency scanning page](../merge_requests/dependency_scanning.md). + +### If your project is hosted on GitHub (https://github.com / GitHub Enterprise) + +Since [GitLab 10.6 comes with GitHub integration](https://about.gitlab.com/features/github/), +GitLab users can now create a CI/CD project in GitLab connected to an external +GitHub.com or GitHub Enterprise repository. This will automatically prompt +GitLab CI/CD to run whenever code is pushed to GitHub and post CI/CD results +back to both GitLab and GitHub when completed. + +1. Create a new project, and select the "CI/CD for external repo" tab: + +  + +1. Use the "GitHub" button to connect your repositories. + +  + +1. Select the project(s) to be set up with GitLab CI/CD and chose "Connect". + +  + + Once the configuration is done, you may click on your new + project on GitLab. + +  + + Your project is now mirrored on GitLab, where the Runners will be able to access + your source code and run your tests. + + Optional step: If you set this up on GitLab.com, make sure the project is + public (in the project settings) if your GitHub project is public, since + the security feature is available only for [GitLab Ultimate](https://about.gitlab.com/pricing). + +1. To set up the dependency scanning job, corresponding to what Gemnasium was + doing, you must create a `.gitlab-ci.yml` file, or update it according to + the [dependency scanning docs](../merge_requests/dependency_scanning.md). + The mirroring is pull-only by default, so you may create or update the file on + GitHub: + +  + +1. Once your file has been committed, a new pipeline will be automatically + triggered if your file is valid: + +  + +1. The result of the job will be visible directly from the pipeline view: + +  + +NOTE: **Note:** +If you don't commit very often to your project, you may want to use +[scheduled pipelines](../pipelines/schedules.md) to run the job on a regular +basis. diff --git a/doc/user/project/import/img/gemnasium/connect_github.png b/doc/user/project/import/img/gemnasium/connect_github.png Binary files differnew file mode 100644 index 00000000000..5933f4d5d0a --- /dev/null +++ b/doc/user/project/import/img/gemnasium/connect_github.png diff --git a/doc/user/project/import/img/gemnasium/create_project.png b/doc/user/project/import/img/gemnasium/create_project.png Binary files differnew file mode 100644 index 00000000000..6e00bf63405 --- /dev/null +++ b/doc/user/project/import/img/gemnasium/create_project.png diff --git a/doc/user/project/import/img/gemnasium/edit_gitlab-ci.png b/doc/user/project/import/img/gemnasium/edit_gitlab-ci.png Binary files differnew file mode 100644 index 00000000000..8336c803b83 --- /dev/null +++ b/doc/user/project/import/img/gemnasium/edit_gitlab-ci.png diff --git a/doc/user/project/import/img/gemnasium/pipeline.png b/doc/user/project/import/img/gemnasium/pipeline.png Binary files differnew file mode 100644 index 00000000000..ae4d5295ffa --- /dev/null +++ b/doc/user/project/import/img/gemnasium/pipeline.png diff --git a/doc/user/project/import/img/gemnasium/project_connected.png b/doc/user/project/import/img/gemnasium/project_connected.png Binary files differnew file mode 100644 index 00000000000..8e7216c015b --- /dev/null +++ b/doc/user/project/import/img/gemnasium/project_connected.png diff --git a/doc/user/project/import/img/gemnasium/report.png b/doc/user/project/import/img/gemnasium/report.png Binary files differnew file mode 100644 index 00000000000..5c4d58662c0 --- /dev/null +++ b/doc/user/project/import/img/gemnasium/report.png diff --git a/doc/user/project/import/img/gemnasium/select_project.png b/doc/user/project/import/img/gemnasium/select_project.png Binary files differnew file mode 100644 index 00000000000..588c5ca7ce1 --- /dev/null +++ b/doc/user/project/import/img/gemnasium/select_project.png diff --git a/doc/user/project/import/index.md b/doc/user/project/import/index.md index ca02e4e9e96..50dc3df77a6 100644 --- a/doc/user/project/import/index.md +++ b/doc/user/project/import/index.md @@ -13,11 +13,14 @@ 1. [From TFS](tfs.md) 1. [From repo by URL](repo_by_url.md) 1. [By uploading a manifest file (AOSP)](manifest.md) +1. [From Gemnasium](gemnasium.md) In addition to the specific migration documentation above, you can import any Git repository via HTTP from the New Project page. Be aware that if the repository is too large the import can timeout. +There is also the option of [connecting your external repository to get CI/CD benefits](../ci_cd_for_external_repo.md). + ## Migrating from self-hosted GitLab to GitLab.com You can copy your repos by changing the remote and pushing to the new server, diff --git a/doc/user/project/index.md b/doc/user/project/index.md index 4148310dc98..cea1f3b37c4 100644 --- a/doc/user/project/index.md +++ b/doc/user/project/index.md @@ -13,7 +13,7 @@ the number of private projects you create. When you create a project in GitLab, you'll have access to a large number of [features](https://about.gitlab.com/features/): -**Issues and merge requests:** +**Repositories:** - [Issue tracker](issues/index.md): Discuss implementations with your team within issues - [Issue Boards](issue_board.md): Organize and prioritize your workflow @@ -28,6 +28,13 @@ When you create a project in GitLab, you'll have access to a large number of permission to create tags, and prevent accidental update or deletion - [Signing commits](gpg_signed_commits/index.md): use GPG to sign your commits - [Deploy tokens](deploy_tokens/index.md): Manage project-based deploy tokens that allow permanent access to the repository and Container Registry. +- [Web IDE](web_ide/index.md) + +**Issues and merge requests:** + +- [Issue tracker](issues/index.md): Discuss implementations with your team within issues + - [Issue Boards](issue_board.md): Organize and prioritize your workflow + - [Multiple Issue Boards](issue_board.md#multiple-issue-boards): Allow your teams to create their own workflows (Issue Boards) for the same project **[STARTER]** - [Merge Requests](merge_requests/index.md): Apply your branching strategy and get reviewed by your team - [Merge Request Approvals](https://docs.gitlab.com/ee/user/project/merge_requests/merge_request_approvals.html): Ask for approval before @@ -67,6 +74,8 @@ When you create a project in GitLab, you'll have access to a large number of timeout (defines the maximum amount of time in minutes that a job is able run), custom path for `.gitlab-ci.yml`, test coverage parsing, pipeline's visibility, and much more - [Kubernetes cluster integration](clusters/index.md): Connecting your GitLab project with a Kubernetes cluster + - [Feature Flags](operations/feature_flags.md): Feature flags allow you to ship a project in + different flavors by dynamically toggling certain functionality **[PREMIUM]** - [GitLab Pages](pages/index.md): Build, test, and deploy your static website with GitLab Pages @@ -75,12 +84,17 @@ When you create a project in GitLab, you'll have access to a large number of - [Wiki](wiki/index.md): document your GitLab project in an integrated Wiki. - [Snippets](../snippets.md): store, share and collaborate on code snippets. - [Cycle Analytics](cycle_analytics.md): review your development lifecycle. +- [Security Dashboard](security_dashboard.md): Security Dashboard. - [Syntax highlighting](highlighting.md): an alternative to customize your code blocks, overriding GitLab's default choice of language. - [Badges](badges.md): badges for the project overview. - [Releases](releases/index.md): a way to track deliverables in your project as snapshot in time of the source, build output, and other metadata or artifacts associated with a released version of your code. +- [Maven packages](packages/maven_repository.md): your private Maven repository in GitLab. **[PREMIUM]** +- [NPM packages](packages/npm_registry.md): your private NPM package registry in GitLab. **[PREMIUM]** +- [Code owners](code_owners.md): specify code owners for certain files **[STARTER]** +- [License Management](merge_requests/license_management.md): approve and blacklist licenses for projects. **[ULTIMATE]** ### Project integrations @@ -116,6 +130,13 @@ Read through the documentation on [project settings](settings/index.md). - [Export a project from GitLab](settings/import_export.md#exporting-a-project-and-its-data) - [Importing and exporting projects between GitLab instances](settings/import_export.md) +## CI/CD for external repositories + +Instead of importing a repository directly to GitLab, you can connect your repository +as a CI/CD project. + +Read through the documentation on [CI/CD for external repositories](../../ci/ci_cd_for_external_repos/index.md). + ## Project members Learn how to [add members to your projects](members/index.md). diff --git a/doc/user/project/integrations/github.md b/doc/user/project/integrations/github.md new file mode 100644 index 00000000000..cdb0e34fdf6 --- /dev/null +++ b/doc/user/project/integrations/github.md @@ -0,0 +1,48 @@ +# GitHub project integration **[PREMIUM]** + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/issues/3836) in GitLab Premium 10.6. + +GitLab provides an integration for updating the pipeline statuses on GitHub. +This is especially useful if using GitLab for CI/CD only. + +This project integration is separate from the [instance wide GitHub integration](../import/github.md#mirroring-and-pipeline-status-sharing) +and is automatically configured on [GitHub import](../../../integration/github.md). + + + +## Configuration + +### Complete these steps on GitHub + +This integration requires a [GitHub API token](https://help.github.com/articles/creating-a-personal-access-token-for-the-command-line/) +with `repo:status` access granted: + +1. Go to your "Personal access tokens" page at https://github.com/settings/tokens +1. Click "Generate New Token" +1. Ensure that `repo:status` is checked and click "Generate token" +1. Copy the generated token to use on GitLab + +### Complete these steps on GitLab + +1. Navigate to the project you want to configure. +1. Navigate to the [Integrations page](project_services.md#accessing-the-project-services) +1. Click "GitHub". +1. Select the "Active" checkbox. +1. Paste the token you've generated on GitHub +1. Enter the path to your project on GitHub, such as `https://github.com/username/repository` +1. Optionally check "Static status check names" checkbox to enable static status check names. +1. Save or optionally click "Test Settings". + +#### Static / dynamic status check names + +Since GitLab 11.5 it is possible to opt-in to using static status check names. + +This makes it possible to mark these status checks as _Required_ on GitHub. +If you check "Static status check names" checkbox on the integration page, your +GitLab instance host name is going to be appended to a status check name, +whereas in case of dynamic status check names, a branch name is going to be +appended. + +Dynamic status check name is a default behavior. + + diff --git a/doc/user/project/integrations/gitlab_slack_application.md b/doc/user/project/integrations/gitlab_slack_application.md new file mode 100644 index 00000000000..3dcd018f305 --- /dev/null +++ b/doc/user/project/integrations/gitlab_slack_application.md @@ -0,0 +1,65 @@ +# GitLab Slack application + +NOTE: **Note:** +The GitLab Slack application is only configurable for GitLab.com. It will **not** +work for on-premises installations where you can configure the +[Slack slash commands](slack_slash_commands.md) service instead. We're working +with Slack on making this configurable for all GitLab installations, but there's +no ETA. +It was first introduced in GitLab 9.4 and distributed to Slack App Directory in +GitLab 10.2. + +Slack provides a native application which you can enable via your project's +integrations on GitLab.com. + +## Slack App Directory + +The simplest way to enable the GitLab Slack application for your workspace is to +install the [GitLab application](https://slack-platform.slack.com/apps/A676ADMV5-gitlab) from +the [Slack App Directory](https://slack.com/apps). + +Clicking install will take you to the +[GitLab Slack application landing page](https://gitlab.com/profile/slack/edit) +where you can select a project to enable the GitLab Slack application for. + + + +## Configuration + +Alternatively, you can configure the Slack application with a project's +integration settings. + +Keep in mind that you need to have the appropriate permissions for your Slack +team in order to be able to install a new application, read more in Slack's +docs on [Adding an app to your team][slack-docs]. + +To enable GitLab's service for your Slack team: + +1. Go to your project's **Settings > Integration > Slack application** (only + visible on GitLab.com) +1. Click the "Add to Slack" button + +That's all! You can now start using the Slack slash commands. + +## Usage + +After confirming the installation, you, and everyone else in your Slack team, +can use all the [slash commands]. + +When you perform your first slash command you will be asked to authorize your +Slack user on GitLab.com. + +The only difference with the [manually configurable Slack slash commands][slack-manual] +is that all the commands should be prefixed with the `/gitlab` keyword. +We are working on making this configurable in the future. + +For example, to show the issue number `1001` under the `gitlab-org/gitlab-ce` +project, you would do: + +``` +/gitlab gitlab-org/gitlab-ce issue show 1001 +``` + +[slack-docs]: https://get.slack.help/hc/en-us/articles/202035138-Adding-apps-to-your-team +[slash commands]: ../../../integration/slash_commands.md +[slack-manual]: slack_slash_commands.md diff --git a/doc/user/project/integrations/img/github_configuration.png b/doc/user/project/integrations/img/github_configuration.png Binary files differnew file mode 100644 index 00000000000..9f2d47921c7 --- /dev/null +++ b/doc/user/project/integrations/img/github_configuration.png diff --git a/doc/user/project/integrations/img/github_status_check_pipeline_update.png b/doc/user/project/integrations/img/github_status_check_pipeline_update.png Binary files differnew file mode 100644 index 00000000000..9d02ca18497 --- /dev/null +++ b/doc/user/project/integrations/img/github_status_check_pipeline_update.png diff --git a/doc/user/project/integrations/img/gitlab_slack_app_landing_page.png b/doc/user/project/integrations/img/gitlab_slack_app_landing_page.png Binary files differnew file mode 100644 index 00000000000..57cd35c9f5d --- /dev/null +++ b/doc/user/project/integrations/img/gitlab_slack_app_landing_page.png diff --git a/doc/user/project/integrations/img/jira_api_token_menu.png b/doc/user/project/integrations/img/jira_api_token_menu.png Binary files differindex 55c8fb1bdb9..14037bd0b47 100644 --- a/doc/user/project/integrations/img/jira_api_token_menu.png +++ b/doc/user/project/integrations/img/jira_api_token_menu.png diff --git a/doc/user/project/integrations/img/jira_service_page.png b/doc/user/project/integrations/img/jira_service_page.png Binary files differindex 3a27b4df841..377b69d9d06 100644 --- a/doc/user/project/integrations/img/jira_service_page.png +++ b/doc/user/project/integrations/img/jira_service_page.png diff --git a/doc/user/project/integrations/img/prometheus_add_metric.png b/doc/user/project/integrations/img/prometheus_add_metric.png Binary files differnew file mode 100644 index 00000000000..e85670e1a13 --- /dev/null +++ b/doc/user/project/integrations/img/prometheus_add_metric.png diff --git a/doc/user/project/integrations/img/prometheus_alert.png b/doc/user/project/integrations/img/prometheus_alert.png Binary files differnew file mode 100644 index 00000000000..a37f0477fd9 --- /dev/null +++ b/doc/user/project/integrations/img/prometheus_alert.png diff --git a/doc/user/project/integrations/img/prometheus_service_alerts.png b/doc/user/project/integrations/img/prometheus_service_alerts.png Binary files differnew file mode 100644 index 00000000000..a81dfe7da14 --- /dev/null +++ b/doc/user/project/integrations/img/prometheus_service_alerts.png diff --git a/doc/user/project/integrations/project_services.md b/doc/user/project/integrations/project_services.md index e2f23827360..6e4b1ea6b11 100644 --- a/doc/user/project/integrations/project_services.md +++ b/doc/user/project/integrations/project_services.md @@ -35,9 +35,11 @@ Click on the service links to see further configuration instructions and details | [Emails on push](emails_on_push.md) | Email the commits and diff of each push to a list of recipients | | External Wiki | Replaces the link to the internal wiki with a link to an external wiki | | Flowdock | Flowdock is a collaboration web app for technical teams | +| [GitHub](github.md) | Sends pipeline notifications to GitHub | | [Hangouts Chat](hangouts_chat.md) | Receive events notifications in Google Hangouts Chat | | [Irker (IRC gateway)](irker.md) | Send IRC messages, on update, to a list of recipients through an Irker gateway | | [JIRA](jira.md) | JIRA issue tracker | +| [Jenkins](../../../integration/jenkins.md) | An extendable open source continuous integration server | | JetBrains TeamCity CI | A continuous integration and build server | | [Mattermost slash commands](mattermost_slash_commands.md) | Mattermost chat and ChatOps slash commands | | [Mattermost Notifications](mattermost.md) | Receive event notifications in Mattermost | @@ -46,6 +48,7 @@ Click on the service links to see further configuration instructions and details | Pipelines emails | Email the pipeline status to a list of recipients | | [Slack Notifications](slack.md) | Send GitLab events (e.g. issue created) to Slack as notifications | | [Slack slash commands](slack_slash_commands.md) | Use slash commands in Slack to control GitLab | +| [GitLab Slack application](gitlab_slack_application.md) | Use Slack's official application | PivotalTracker | Project Management Software (Source Commits Endpoint) | | [Prometheus](prometheus.md) | Monitor the performance of your deployed apps | | Pushover | Pushover makes it easy to get real-time notifications on your Android device, iPhone, iPad, and Desktop | diff --git a/doc/user/project/integrations/prometheus.md b/doc/user/project/integrations/prometheus.md index 43a9e24526d..57d4668adaf 100644 --- a/doc/user/project/integrations/prometheus.md +++ b/doc/user/project/integrations/prometheus.md @@ -13,7 +13,7 @@ There are two ways to set up Prometheus integration, depending on where your app - For deployments on Kubernetes, GitLab can automatically [deploy and manage Prometheus](#managed-prometheus-on-kubernetes). - For other deployment targets, simply [specify the Prometheus server](#manual-configuration-of-prometheus). -Once enabled, GitLab will automatically detect metrics from known services in the [metric library](#monitoring-cicd-environments). +Once enabled, GitLab will automatically detect metrics from known services in the [metric library](#monitoring-cicd-environments). You are also able to [add your own metrics](#adding-additional-metrics-premium) as well. ## Enabling Prometheus Integration @@ -93,6 +93,70 @@ GitLab will automatically scan the Prometheus server for metrics from known serv You can view the performance dashboard for an environment by [clicking on the monitoring button](../../../ci/environments.md#monitoring-environments). +### Adding additional metrics **[PREMIUM]** + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/3799) in [GitLab Premium](https://about.gitlab.com/pricing/) 10.6. + +Additional metrics can be monitored by adding them on the Prometheus integration page. Once saved, they will be displayed on the environment performance dashboard. + + + +A few fields are required: + +- **Name**: Chart title +- **Type**: Type of metric. Metrics of the same type will be shown together. +- **Query**: Valid [PromQL query](https://prometheus.io/docs/prometheus/latest/querying/basics/). Note, no validation is performed at this time. If the query is not valid, the dashboard will display an error. +- **Y-axis label**: Y axis title to display on the dashboard. +- **Unit label**: Query units, for example `req / sec`. Shown next to the value. + +#### Query Variables + +GitLab supports a limited set of [CI variables](../../../ci/variables/README.html) in the Prometheus query. This is particularly useful for identifying a specific environment, for example with `CI_ENVIRONMENT_SLUG`. The supported variables are: + +- CI_ENVIRONMENT_SLUG +- KUBE_NAMESPACE + +To specify a variable in a query, enclose it in curly braces with a leading percent. For example: `%{ci_environment_slug}`. + +### Setting up alerts for Prometheus metrics **[ULTIMATE]** + +#### Managed Prometheus instances + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/6590) in [GitLab Ultimate](https://about.gitlab.com/pricing/) 11.2 for [custom metrics](#adding-additional-metrics-premium), and 11.3 for [library metrics](prometheus_library/metrics.md). + +For managed Prometheus instances using auto configuration, alerts for metrics [can be configured](#adding-additional-metrics-premium) directly in the performance dashboard. + +To set an alert, click on the alarm icon in the top right corner of the metric you want to create the alert for. A dropdown +will appear, with options to set the threshold and operator. Click **Add** to save and activate the alert. + + + +If the metric exceeds the threshold of the alert for over 5 minutes, an email +will be sent to all [Maintainers and Owners](../../permissions.md#project-members-permissions) of the project. + +To remove the alert, click back on the alert icon for the desired metric, and click **Delete**. + +#### External Prometheus instances + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/issues/9258) in [GitLab Ultimate](https://about.gitlab.com/pricing/) 11.8. + +For manually configured Prometheus servers, a notify endpoint is provided to use with Prometheus webhooks. If you have manual configuration enabled, an **Alerts** section is added to **Settings > Integrations > Prometheus**. This contains the *URL* and *Authorization Key*. The **Reset Key** button will invalidate the key and generate a new one. + + + +To send GitLab alert notifications, copy the *URL* and *Authorization Key* into the [`webhook_configs`](https://prometheus.io/docs/alerting/configuration/#webhook_config) section of your Prometheus Alertmanager configuration: + +```yaml +receivers: + name: gitlab + webhook_configs: + - http_config: + bearer_token: 9e1cbfcd546896a9ea8be557caf13a76 + send_resolved: true + url: http://192.168.178.31:3001/root/manual_prometheus/prometheus/alerts/notify.json + ... +``` + ## Determining the performance impact of a merge > [Introduced][ce-10408] in GitLab 9.2. diff --git a/doc/user/project/integrations/prometheus_library/kubernetes.md b/doc/user/project/integrations/prometheus_library/kubernetes.md index 7a45c87ada0..4d523967d2a 100644 --- a/doc/user/project/integrations/prometheus_library/kubernetes.md +++ b/doc/user/project/integrations/prometheus_library/kubernetes.md @@ -1,6 +1,6 @@ # Monitoring Kubernetes -> [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/8935) in GitLab 9.0 +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/8935) in GitLab 9.0. GitLab has support for automatically detecting and monitoring Kubernetes metrics. @@ -35,3 +35,25 @@ Prometheus needs to be deployed into the cluster and configured properly in orde In order to isolate and only display relevant CPU and Memory metrics for a given environment, GitLab needs a method to detect which containers it is running. Because these metrics are tracked at the container level, traditional Kubernetes labels are not available. Instead, the [Deployment](https://kubernetes.io/docs/concepts/workloads/controllers/deployment/) or [DaemonSet](https://kubernetes.io/docs/concepts/workloads/controllers/daemonset/) name should begin with [CI_ENVIRONMENT_SLUG](../../../../ci/variables/README.md#predefined-environment-variables). It can be followed by a `-` and additional content if desired. For example, a deployment name of `review-homepage-5620p5` would match the `review/homepage` environment. + +## Displaying Canary metrics + +> Introduced in [GitLab 10.2](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/15201). + +GitLab also gathers Kubernetes metrics for [canary deployments](../../canary_deployments.md), allowing easy comparison between the current deployed version and the canary. + +These metrics expect the [Deployment](https://kubernetes.io/docs/concepts/workloads/controllers/deployment/) or [DaemonSet](https://kubernetes.io/docs/concepts/workloads/controllers/daemonset/) name to begin with `$CI_ENVIRONMENT_SLUG-canary`, to isolate the canary metrics. + +### Canary metrics supported + +- Average Memory Usage (MB) + + ``` + avg(sum(container_memory_usage_bytes{container_name!="POD",pod_name=~"^%{ci_environment_slug}-canary-(.*)",namespace="%{kube_namespace}"}) by (job)) without (job) / count(avg(container_memory_usage_bytes{container_name!="POD",pod_name=~"^%{ci_environment_slug}-canary-(.*)",namespace="%{kube_namespace}"}) without (job)) /1024/1024 + ``` + +- Average CPU Utilization (%) + + ``` + avg(sum(rate(container_cpu_usage_seconds_total{container_name!="POD",pod_name=~"^%{ci_environment_slug}-canary-(.*)",namespace="%{kube_namespace}"}[15m])) by (job)) without (job) / count(sum(rate(container_cpu_usage_seconds_total{container_name!="POD",pod_name=~"^%{ci_environment_slug}-canary-(.*)",namespace="%{kube_namespace}"}[15m])) by (pod_name)) + ``` diff --git a/doc/user/project/issues/create_new_issue.md b/doc/user/project/issues/create_new_issue.md index 40040e44d64..1153ed1474f 100644 --- a/doc/user/project/issues/create_new_issue.md +++ b/doc/user/project/issues/create_new_issue.md @@ -65,6 +65,26 @@ In GitLab 11.7, we updated the format of the generated email address. However the older format is still supported, allowing existing aliases or contacts to continue working._ +## New issue via Service Desk **[PREMIUM]** + +Enable [Service Desk](../service_desk.md) to your project and offer email support. +By doing so, when your customer sends a new email, a new issue can be created in +the appropriate project and followed up from there. + +## New issue from the group-level Issue Tracker + +Head to the Group dashboard and click "Issues" in the sidebar to visit the Issue Tracker +for all projects in your Group. Select the project you'd like to add an issue for +using the dropdown button at the top-right of the page. + + + +We'll keep track of the project you selected most recently, and use it as the default +for your next visit. This should save you a lot of time and clicks, if you mostly +create issues for the same project. + + + ## New issue via URL with prefilled fields You can link directly to the new issue page for a given project, with prefilled diff --git a/doc/user/project/issues/csv_export.md b/doc/user/project/issues/csv_export.md new file mode 100644 index 00000000000..56b94585672 --- /dev/null +++ b/doc/user/project/issues/csv_export.md @@ -0,0 +1,77 @@ +# Export Issues to CSV **[STARTER]** + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/1126) in [GitLab Starter 9.0](https://about.gitlab.com/2017/03/22/gitlab-9-0-released/#export-issues-ees-eep). + +Issues can be exported as CSV from GitLab and are sent to your default notification email as an attachment. + +## Overview + +**Export Issues to CSV** enables you and your team to export all the data collected from issues into +a **[comma-separated values](https://en.wikipedia.org/wiki/Comma-separated_values)** (CSV) file, +which stores tabular data in plain text. + +> _CSVs are a handy way of getting data from one program to another where one program cannot read the other ones normal output._ [Ref](https://www.quora.com/What-is-a-CSV-file-and-its-uses) + +CSV files can be used with any plotter or spreadsheet-based program, such as Microsoft Excel, +Open Office Calc, or Google Spreadsheets. + +## Use cases + +Among numerous use cases for exporting issues for CSV, we can name a few: + +- Make a snapshot of issues for offline analysis or to communicate with other teams who may not be in GitLab +- Create diagrams, graphs, and charts from the CSV data +- Present the data in any other format for auditing or sharing reasons +- Import the issues elsewhere to a system outside of GitLab +- Long-term issues' data analysis with multiple snapshots created along the time +- Use the long-term data to gather relevant feedback given in the issues, and improve your product based on real metrics + +## Choosing which issues to include + +From the issues page you can narrow down which issues to export using the search bar, along with the All/Open/Closed tabs. All issues returned will be exported, including those not shown on the first page. + + + +You will be asked to confirm the number of issues and email address for the export, after which the email will begin being prepared. + + + +## Sorting + +Exported issues are always sorted by `Issue ID`. + +## Format + +> **Time Estimate** and **Time Spent** columns were [introduced](https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/2627) in GitLab Starter 10.0. +> +> **Weight** and **Locked** columns were [introduced](https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/5300) in GitLab Starter 10.8. + +Data will be encoded with a comma as the column delimiter, with `"` used to quote fields if needed, and newlines to separate rows. The first row will be the headers, which are listed in the following table along with a description of the values: + + +| Column | Description | +|---------|-------------| +| Issue ID | Issue `iid` | +| URL | A link to the issue on GitLab | +| Title | Issue `title` | +| State | `Open` or `Closed` | +| Description | Issue `description` | +| Author | Full name of the issue author | +| Author Username | Username of the author, with the `@` symbol omitted | +| Assignee | Full name of the issue assignee | +| Assignee Username | Username of the author, with the `@` symbol omitted | +| Confidential | `Yes` or `No` | +| Locked | `Yes` or `No` | +| Due Date | Formated as `YYYY-MM-DD` | +| Created At (UTC) | Formated as `YYYY-MM-DD HH:MM:SS` | +| Updated At (UTC) | Formated as `YYYY-MM-DD HH:MM:SS` | +| Milestone | Title of the issue milestone | +| Weight | Issue weight | +| Labels | Title of any labels joined with a `,` | +| Time Estimate | [Time estimate](../../../workflow/time_tracking.md#estimates) in seconds | +| Time Spent | [Time spent](../../../workflow/time_tracking.md#time-spent) in seconds | + + +## Limitations + +As the issues will be sent as an email attachment, there is a limit on how much data can be exported. Currently this limit is 20MB to ensure successful delivery across a range of email providers. If this limit is reached we suggest narrowing the search before export, perhaps by exporting open and closed issues separately. diff --git a/doc/user/project/issues/img/create_issue_from_group_level_issue_tracker.png b/doc/user/project/issues/img/create_issue_from_group_level_issue_tracker.png Binary files differnew file mode 100644 index 00000000000..8996490ae63 --- /dev/null +++ b/doc/user/project/issues/img/create_issue_from_group_level_issue_tracker.png diff --git a/doc/user/project/issues/img/csv_export_button.png b/doc/user/project/issues/img/csv_export_button.png Binary files differnew file mode 100644 index 00000000000..f9fcfd71c2f --- /dev/null +++ b/doc/user/project/issues/img/csv_export_button.png diff --git a/doc/user/project/issues/img/csv_export_modal.png b/doc/user/project/issues/img/csv_export_modal.png Binary files differnew file mode 100644 index 00000000000..f988deec966 --- /dev/null +++ b/doc/user/project/issues/img/csv_export_modal.png diff --git a/doc/user/project/issues/img/multiple_assignees.gif b/doc/user/project/issues/img/multiple_assignees.gif Binary files differnew file mode 100644 index 00000000000..055a0efd9ae --- /dev/null +++ b/doc/user/project/issues/img/multiple_assignees.gif diff --git a/doc/user/project/issues/img/multiple_assignees_for_issues.png b/doc/user/project/issues/img/multiple_assignees_for_issues.png Binary files differnew file mode 100644 index 00000000000..e8ae37d427d --- /dev/null +++ b/doc/user/project/issues/img/multiple_assignees_for_issues.png diff --git a/doc/user/project/issues/img/related_issues_add.png b/doc/user/project/issues/img/related_issues_add.png Binary files differnew file mode 100644 index 00000000000..f59d2335386 --- /dev/null +++ b/doc/user/project/issues/img/related_issues_add.png diff --git a/doc/user/project/issues/img/related_issues_remove.png b/doc/user/project/issues/img/related_issues_remove.png Binary files differnew file mode 100644 index 00000000000..be2ec59e61b --- /dev/null +++ b/doc/user/project/issues/img/related_issues_remove.png diff --git a/doc/user/project/issues/img/select_project_from_group_level_issue_tracker.png b/doc/user/project/issues/img/select_project_from_group_level_issue_tracker.png Binary files differnew file mode 100644 index 00000000000..97d93620b2a --- /dev/null +++ b/doc/user/project/issues/img/select_project_from_group_level_issue_tracker.png diff --git a/doc/user/project/issues/img/similar_issues.png b/doc/user/project/issues/img/similar_issues.png Binary files differindex 0dfb5b00e02..c5b7902b2ac 100644 --- a/doc/user/project/issues/img/similar_issues.png +++ b/doc/user/project/issues/img/similar_issues.png diff --git a/doc/user/project/issues/index.md b/doc/user/project/issues/index.md index 907a305fe23..94235eb1fdd 100644 --- a/doc/user/project/issues/index.md +++ b/doc/user/project/issues/index.md @@ -34,7 +34,7 @@ your project public, open to collaboration. ### Streamline collaboration -With [Multiple Assignees for Issues](https://docs.gitlab.com/ee/user/project/issues/multiple_assignees_for_issues.html), +With [Multiple Assignees for Issues](multiple_assignees_for_issues.md), available in [GitLab Starter](https://about.gitlab.com/pricing/) you can streamline collaboration and allow shared responsibilities to be clearly displayed. All assignees are shown across your workflows and receive notifications (as they @@ -139,8 +139,19 @@ Find GitLab Issue Boards by navigating to your **Project's Dashboard** > **Issue Read through the documentation for [Issue Boards](../issue_board.md) to find out more about this feature. -With [GitLab Starter](https://about.gitlab.com/pricing/), you can also -create various boards per project with [Multiple Issue Boards](https://docs.gitlab.com/ee/user/project/issue_board.html#multiple-issue-boards). +### Export Issues to CSV **[STARTER]** + +Issues can be [exported as CSV](csv_export.md) from GitLab and are sent to your email as an attachment. + +_Exporting issues to CSV is available only in [GitLab Enterprise Edition](https://about.gitlab.com/products/)._ + +### Related Issues **[STARTER]** + +Related Issues are a bi-directional relationship between any two issues +and appear in a block below the issue description. Issues can be across groups +and projects. + +Read more about [Related Issues](related_issues.md). ### Import Issues from CSV diff --git a/doc/user/project/issues/issues_functionalities.md b/doc/user/project/issues/issues_functionalities.md index 27b9dc51760..ebbccd4162b 100644 --- a/doc/user/project/issues/issues_functionalities.md +++ b/doc/user/project/issues/issues_functionalities.md @@ -50,7 +50,7 @@ where there is shared ownership of an issue. In [GitLab Starter](https://about.gitlab.com/pricing/), you can assign multiple people to an issue. -Learn more in the [Multiple Assignees documentation](https://docs.gitlab.com/ee/user/project/issues/multiple_assignees_for_issues.html). +Learn more in the [Multiple Assignees documentation](multiple_assignees_for_issues.md). #### 4. Milestone @@ -90,7 +90,7 @@ If a label doesn't exist yet, you can click **Edit**, and it opens a dropdown me - Assign a weight. Larger values are used to indicate more effort is required to complete the issue. Only positive values or zero are allowed. -Learn more in the [Issue Weight documentation](https://docs.gitlab.com/ee/workflow/issue_weight.html). +Learn more in the [Issue Weight documentation](../../../workflow/issue_weight.md). #### 9. Participants diff --git a/doc/user/project/issues/multiple_assignees_for_issues.md b/doc/user/project/issues/multiple_assignees_for_issues.md new file mode 100644 index 00000000000..8781ebdd5b0 --- /dev/null +++ b/doc/user/project/issues/multiple_assignees_for_issues.md @@ -0,0 +1,41 @@ +# Multiple Assignees for Issues **[STARTER]** + +> **Note:** +[Introduced](https://gitlab.com/gitlab-org/gitlab-ee/issues/1904) +in [GitLab Starter 9.2](https://about.gitlab.com/2017/05/22/gitlab-9-2-released/#multiple-assignees-for-issues). + +## Overview + +In large teams, where there is shared ownership of an issue, it can be difficult +to track who is working on it, who already completed their contributions, who +didn't even start yet. + +In [GitLab Enterprise Edition](https://about.gitlab.com/pricing/), +you can also select multiple assignees to an issue, making it easier to +track, and making clearer who is accountable for it. + + + +## Use cases + +Consider a team formed by frontend developers, backend developers, +UX designers, QA testers, and a product manager working together to bring an idea to +market. + +Multiple Assignees for Issues makes collaboration smother, +and allows shared responsibilities to be clearly displayed. +All assignees are shown across your team's workflows and receive notifications (as they +would as single assignees), simplifying communication and ownership. + +Once an assignee had their work completed, they would remove themselves as assignees, making +it clear that their role is complete. + +## How it works + +From an opened issue, expand the right sidebar, locate the assignees entry, +and click on **Edit**. From the dropdown menu, select as many users as you want +to assign the issue to. + + + +An assignee can be easily removed by deselecting them from the same dropdown menu. diff --git a/doc/user/project/issues/related_issues.md b/doc/user/project/issues/related_issues.md new file mode 100644 index 00000000000..7180bd1d297 --- /dev/null +++ b/doc/user/project/issues/related_issues.md @@ -0,0 +1,37 @@ +# Related issues **[STARTER]** + +> [Introduced][ee-1797] in [GitLab Starter][ee] 9.4. + +Related issues are a bi-directional relationship between any two issues +and appear in a block below the issue description. Issues can be across groups +and projects. + +The relationship only shows up in the UI if the user can see both issues. + +## Adding a related issue + +You can relate one issue to another by clicking the related issues "+" button +in the header of the related issue block. Then input the issue reference number +or paste in a link to the issue. + +Valid references will be added to a temporary list that you can review. +When ready, click the green "Add related issues" button to submit. + + + +## Removing a related issue + +In the related issues block, click the "x" icon on the right-side of each issue +token that you wish to remove. Due to the bi-directional relationship, it +will no longer appear in either issue. + + + +Please access our [permissions] page for more information. + +Additionally, you are also able to manage related issues through [our API]. + +[ee]: https://about.gitlab.com/pricing/ +[ee-1797]: https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/1797 +[permissions]: ../../permissions.md +[Our API]: ../../../api/issue_links.md diff --git a/doc/user/project/labels.md b/doc/user/project/labels.md index 9c045dfcce4..fa073e659aa 100644 --- a/doc/user/project/labels.md +++ b/doc/user/project/labels.md @@ -8,7 +8,7 @@ Labels allow you to categorize issues or merge requests using descriptive titles In GitLab, you can create project and group labels: -- **Project labels** can be assigned to issues or merge requests in that project only. +- **Project labels** can be assigned to issues or merge requests in that project only. - **Group labels** can be assigned to any issue or merge request of any project in that group or any subgroups of the group. ## Creating labels @@ -35,6 +35,9 @@ GitLab will add the following default labels to the project: To create a **group label**, follow similar steps from above to project labels. Navigate to **Issues > Labels** in the group and create it from there. This page only shows group labels in this group. +Alternatively, you can create group labels also from Epic sidebar. Please note that the created label will belong to the immediate group to which epic belongs. + + Group labels appear in every label list page of the group's child projects. @@ -81,7 +84,7 @@ top-right: GitLab will consider the label title and description for the search. -## Filtering issues and merge requests by label +## Filtering issues, merge requests and epics by label ### Filtering in list pages @@ -89,11 +92,16 @@ From the project issue list page and the project merge request list page, you ca From the group issue list page and the group merge request list page, you can [filter](../search/index.md#issues-and-merge-requests) by both group labels (including subgroup ancestors and subgroup descendants) and project labels. +From the group epic list page, you can [filter](../search/index.md#issues-and-merge-requests) by both current group labels as well as decendent group labels. +  ### Filtering in issue boards - From [project boards](issue_board.md), you can filter by both group labels and project labels in the [search and filter bar](../search/index.md#issue-boards). +- From [group issue boards](issue_board.md#group-issue-boards), you can filter by only group labels in the [search and filter bar](../search/index.md#issue-boards). **[PREMIUM]** +- From [project boards](issue_board.md), you can filter by both group labels and project labels in the [issue board configuration](issue_board.md#board-with-configuration). **[PREMIUM]** +- From [group issue boards](issue_board.md#group-issue-boards), you can filter by only group labels in the [issue board configuration](issue_board.md#board-with-configuration). **[PREMIUM]** ## Subscribing to labels diff --git a/doc/user/project/maven_packages.md b/doc/user/project/maven_packages.md new file mode 100644 index 00000000000..48835a2dac7 --- /dev/null +++ b/doc/user/project/maven_packages.md @@ -0,0 +1,5 @@ +--- +redirect_to: 'packages/maven_repository.md' +--- + +This document was moved to [another location](packages/maven_repository.md). diff --git a/doc/user/project/merge_requests/browser_performance_testing.md b/doc/user/project/merge_requests/browser_performance_testing.md new file mode 100644 index 00000000000..65ee2e128ae --- /dev/null +++ b/doc/user/project/merge_requests/browser_performance_testing.md @@ -0,0 +1,54 @@ +# Browser Performance Testing **[PREMIUM]** + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/3507) +in [GitLab Premium](https://about.gitlab.com/pricing/) 10.3. + +## Overview + +If your application offers a web interface and you are using +[GitLab CI/CD](../../../ci/README.md), you can quickly determine the performance +impact of pending code changes. + +GitLab uses [Sitespeed.io](https://www.sitespeed.io), a free and open source +tool for measuring the performance of web sites, and has built a simple +[Sitespeed plugin](https://gitlab.com/gitlab-org/gl-performance) +which outputs the results in a file called `performance.json`. This plugin +outputs the performance score for each page that is analyzed. + +The [Sitespeed.io performance score](http://examples.sitespeed.io/6.0/2017-11-23-23-43-35/help.html) +is a composite value based on best practices, and we will be expanding support +for [additional metrics](https://gitlab.com/gitlab-org/gitlab-ee/issues/4370) +in a future release. + +Going a step further, GitLab can show the Performance report right +in the merge request widget area: + +## Use cases + +For instance, consider the following workflow: + +1. A member of the marketing team is attempting to track engagement by adding a new tool +1. With browser performance metrics, they see how their changes are impacting the usability of the page for end users +1. The metrics show that after their changes the performance score of the page has gone down +1. When looking at the detailed report, they see that the new Javascript library was included in `<head>` which affects loading page speed +1. They ask a front end developer to help them, who sets the library to load asynchronously +1. The frontend developer approves the merge request and authorizes its deployment to production + +## How it works + +First of all, you need to define a job in your `.gitlab-ci.yml` file that generates the +[Performance report artifact](../../../ci/yaml/README.md#artifactsreportsperformance-premium). +For more information on how the Performance job should look like, check the +example on [Testing Browser Performance](../../../ci/examples/browser_performance.md). + +GitLab then checks this report, compares key performance metrics for each page +between the source and target branches, and shows the information right on the merge request. + +>**Note:** +If the Performance report doesn't have anything to compare to, no information +will be displayed in the merge request area. That is the case when you add the +Performance job in your `.gitlab-ci.yml` for the very first time. +Consecutive merge requests will have something to compare to and the Performance +report will be shown properly. + + diff --git a/doc/user/project/merge_requests/code_quality.md b/doc/user/project/merge_requests/code_quality.md new file mode 100644 index 00000000000..e6811b5df5e --- /dev/null +++ b/doc/user/project/merge_requests/code_quality.md @@ -0,0 +1,82 @@ +# Code Quality **[STARTER]** + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/1984) +in [GitLab Starter](https://about.gitlab.com/pricing/) 9.3. + +## Overview + +If you are using [GitLab CI/CD](../../../ci/README.md), you can analyze your +source code quality using GitLab Code Quality. +Code Quality uses [Code Climate Engines](https://codeclimate.com), which are +free and open source. Code Quality doesn’t require a Code Climate subscription. + +Going a step further, GitLab can show the Code Quality report right +in the merge request widget area: + + + +## Use cases + +For instance, consider the following workflow: + +1. Your backend team member starts a new implementation for making certain feature in your app faster +1. With Code Quality reports, they analyze how their implementation is impacting the code quality +1. The metrics show that their code degrade the quality in 10 points +1. You ask a co-worker to help them with this modification +1. They both work on the changes until Code Quality report displays no degradations, only improvements +1. You approve the merge request and authorize its deployment to staging +1. Once verified, their changes are deployed to production + +## How it works + +First of all, you need to define a job in your `.gitlab-ci.yml` file that generates the +[Code Quality report artifact](../../../ci/yaml/README.md#artifactsreportscodequality-starter). + +The Code Quality report artifact is a subset of the +[Code Climate spec](https://github.com/codeclimate/spec/blob/master/SPEC.md#data-types). +It must be a JSON file containing an array of objects with the following properties: + +| Name | Description | +| ---------------------- | -------------------------------------------------------------------------------------- | +| `description` | A description of the code quality violation. | +| `fingerprint` | A unique fingerprint to identify the code quality violation. For example, an MD5 hash. | +| `location.path` | The relative path to the file containing the code quality violation. | +| `location.lines.begin` | The line on which the code quality violation occurred. | + +Example: + +```json +[ + { + "description": "'unused' is assigned a value but never used.", + "fingerprint": "7815696ecbf1c96e6894b779456d330e", + "location": { + "path": "lib/index.js", + "lines": { + "begin": 42 + } + } + } +] +``` + +NOTE: **Note:** +Although the Code Climate spec supports more properties, those are ignored by GitLab. + +For more information on how the Code Quality job should look like, check the +example on [analyzing a project's code quality](../../../ci/examples/code_quality.md). + +GitLab then checks this report, compares the metrics between the source and target +branches, and shows the information right on the merge request. + +CAUTION: **Caution:** +If multiple jobs in a pipeline generate a code quality artifact, only the artifact from +the last created job (the job with the largest job ID) is used. To avoid confusion, +configure only one job to generate a code quality artifact. + +NOTE: **Note:** +If the Code Quality report doesn't have anything to compare to, no information +will be displayed in the merge request area. That is the case when you add the +Code Quality job in your `.gitlab-ci.yml` for the very first time. +Consecutive merge requests will have something to compare to and the Code Quality +report will be shown properly. diff --git a/doc/user/project/merge_requests/code_quality_diff.md b/doc/user/project/merge_requests/code_quality_diff.md new file mode 100644 index 00000000000..890058eec6f --- /dev/null +++ b/doc/user/project/merge_requests/code_quality_diff.md @@ -0,0 +1,6 @@ +--- +redirect_from: 'https://docs.gitlab.com/ee/user/project/merge_requests/code_quality_diff.html' +redirect_to: 'code_quality.md' +--- + +This document was moved to [another location](code_quality.md). diff --git a/doc/user/project/merge_requests/container_scanning.md b/doc/user/project/merge_requests/container_scanning.md new file mode 100644 index 00000000000..5d94e3eef16 --- /dev/null +++ b/doc/user/project/merge_requests/container_scanning.md @@ -0,0 +1,42 @@ +# Container Scanning **[ULTIMATE]** + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/3672) +in [GitLab Ultimate](https://about.gitlab.com/pricing/) 10.4. + +> [Introduced][ee-3672] in [GitLab Ultimate][ee] 10.4. + +## Overview + +If you are using [GitLab CI/CD](../../../ci/README.md), you can analyze your Docker images for known +vulnerabilities using [Clair](https://github.com/coreos/clair), +a Vulnerability Static Analysis tool for containers. + +You can take advantage of Container Scanning by either [including the CI job](../../../ci/examples/container_scanning.md) in +your existing `.gitlab-ci.yml` file or by implicitly using +[Auto Container Scanning](../../../topics/autodevops/index.md#auto-container-scanning) +that is provided by [Auto DevOps](../../../topics/autodevops/index.md). + +Going a step further, GitLab can show the vulnerability list right in the merge +request widget area. + + + +## Use cases + +If you distribute your application with Docker, then there's a great chance +that your image is based on other Docker images that may in turn contain some +known vulnerabilities that could be exploited. + +Having an extra job in your pipeline that checks for those vulnerabilities, +and the fact that they are displayed inside a merge request, makes it very easy +to perform audits for your Docker-based apps. + +## How it works + +First of all, you need to define a job in your `.gitlab-ci.yml` file that generates the +[Container Scanning report artifact](../../../ci/yaml/README.md#artifactsreportscontainer_scanning-ultimate). +For more information on how the Container Scanning job should look like, check the +example on [Container Scanning with GitLab CI/CD](../../../ci/examples/container_scanning.md). + +GitLab then checks this report, compares the found vulnerabilities between the source and target +branches, and shows the information right on the merge request. diff --git a/doc/user/project/merge_requests/dast.md b/doc/user/project/merge_requests/dast.md new file mode 100644 index 00000000000..9560f373a70 --- /dev/null +++ b/doc/user/project/merge_requests/dast.md @@ -0,0 +1,45 @@ +# Dynamic Application Security Testing (DAST) **[ULTIMATE]** + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/issues/4348) +in [GitLab Ultimate](https://about.gitlab.com/pricing/) 10.4. + +## Overview + +Running [static checks](sast.md) on your code is the first step to detect +vulnerabilities that can put the security of your code at risk. Yet, once +deployed, your application is exposed to a new category of possible attacks, +such as cross-site scripting or broken authentication flaws. This is where +Dynamic Application Security Testing (DAST) comes into place. + +If you are using [GitLab CI/CD](../../../ci/README.md), you can analyze your running web application(s) +for known vulnerabilities using Dynamic Application Security Testing (DAST). + +You can take advantage of DAST by either [including the CI job](../../../ci/examples/dast.md) in +your existing `.gitlab-ci.yml` file or by implicitly using +[Auto DAST](../../../topics/autodevops/index.md#auto-dast-ultimate) +that is provided by [Auto DevOps](../../../topics/autodevops/index.md). + +Going a step further, GitLab can show the vulnerability list right in the merge +request widget area. + +## Use cases + +It helps you automatically find security vulnerabilities in your running web +applications while you are developing and testing your applications. + +## How it works + +First of all, you need to define a job in your `.gitlab-ci.yml` file that generates the +[DAST report artifact](../../../ci/yaml/README.md#artifactsreportsdast-ultimate). +For more information on how the DAST job should look like, check the +example on [Dynamic Application Security Testing with GitLab CI/CD](../../../ci/examples/dast.md). + +GitLab then checks this report, compares the found vulnerabilities between the source and target +branches, and shows the information right on the merge request. + + + +By clicking on one of the detected linked vulnerabilities, you will be able to +see the details and the URL(s) affected. + + diff --git a/doc/user/project/merge_requests/dependency_scanning.md b/doc/user/project/merge_requests/dependency_scanning.md new file mode 100644 index 00000000000..09916b75863 --- /dev/null +++ b/doc/user/project/merge_requests/dependency_scanning.md @@ -0,0 +1,50 @@ +# Dependency Scanning **[ULTIMATE]** + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/issues/5105) +in [GitLab Ultimate](https://about.gitlab.com/pricing/) 10.7. + +## Overview + +If you are using [GitLab CI/CD](../../../ci/README.md), you can analyze your dependencies for known +vulnerabilities using Dependency Scanning. + +You can take advantage of Dependency Scanning by either [including the CI job](../../../ci/examples/dependency_scanning.md) in +your existing `.gitlab-ci.yml` file or by implicitly using +[Auto Dependency Scanning](../../../topics/autodevops/index.md#auto-dependency-scanning-ultimate) +that is provided by [Auto DevOps](../../../topics/autodevops/index.md). + +Going a step further, GitLab can show the vulnerability list right in the merge +request widget area. + +## Use cases + +It helps you automatically find security vulnerabilities in your dependencies +while you are developing and testing your applications. E.g. your application +is using an external (open source) library which is known to be vulnerable. + +## Supported languages and dependency managers + +The following languages and dependency managers are supported. + +| Language (package managers) | Scan tool | +|-----------------------------------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------| +| JavaScript ([npm](https://www.npmjs.com/), [yarn](https://yarnpkg.com/en/)) | [gemnasium](https://gitlab.com/gitlab-org/security-products/gemnasium/general), [Retire.js](https://retirejs.github.io/retire.js) | +| Python ([pip](https://pip.pypa.io/en/stable/)) (only `requirements.txt` supported) | [gemnasium](https://gitlab.com/gitlab-org/security-products/gemnasium/general) | +| Ruby ([gem](https://rubygems.org/)) | [gemnasium](https://gitlab.com/gitlab-org/security-products/gemnasium/general), [bundler-audit](https://github.com/rubysec/bundler-audit) | +| Java ([Maven](https://maven.apache.org/)) | [gemnasium](https://gitlab.com/gitlab-org/security-products/gemnasium/general) | +| PHP ([Composer](https://getcomposer.org/)) | [gemnasium](https://gitlab.com/gitlab-org/security-products/gemnasium/general) | + +Some scanners require to send a list of project dependencies to GitLab central servers to check for vulnerabilities. To learn more about this or to disable it please +check [GitLab Dependency Scanning documentation](https://gitlab.com/gitlab-org/security-products/dependency-scanning#remote-checks). + +## How it works + +First of all, you need to define a job in your `.gitlab-ci.yml` file that generates the +[Dependency Scanning report artifact](../../../ci/yaml/README.md#artifactsreportsdependency_scanning-ultimate). +For more information on how the Dependency Scanning job should look like, check the +example on [Dependency Scanning with GitLab CI/CD](../../../ci/examples/dependency_scanning.md). + +GitLab then checks this report, compares the found vulnerabilities between the source and target +branches, and shows the information right on the merge request. + + diff --git a/doc/user/project/merge_requests/img/approvals_can_override.png b/doc/user/project/merge_requests/img/approvals_can_override.png Binary files differnew file mode 100644 index 00000000000..8d207d018e0 --- /dev/null +++ b/doc/user/project/merge_requests/img/approvals_can_override.png diff --git a/doc/user/project/merge_requests/img/approvals_premium_mr_widget.png b/doc/user/project/merge_requests/img/approvals_premium_mr_widget.png Binary files differnew file mode 100644 index 00000000000..b6dc86f312e --- /dev/null +++ b/doc/user/project/merge_requests/img/approvals_premium_mr_widget.png diff --git a/doc/user/project/merge_requests/img/approvals_premium_project_edit.png b/doc/user/project/merge_requests/img/approvals_premium_project_edit.png Binary files differnew file mode 100644 index 00000000000..b6f6188b9cd --- /dev/null +++ b/doc/user/project/merge_requests/img/approvals_premium_project_edit.png diff --git a/doc/user/project/merge_requests/img/approvals_remove_on_push.png b/doc/user/project/merge_requests/img/approvals_remove_on_push.png Binary files differnew file mode 100644 index 00000000000..73964827587 --- /dev/null +++ b/doc/user/project/merge_requests/img/approvals_remove_on_push.png diff --git a/doc/user/project/merge_requests/img/approvals_starter_project_edit.png b/doc/user/project/merge_requests/img/approvals_starter_project_edit.png Binary files differnew file mode 100644 index 00000000000..868b9d58740 --- /dev/null +++ b/doc/user/project/merge_requests/img/approvals_starter_project_edit.png diff --git a/doc/user/project/merge_requests/img/approvals_starter_project_empty.png b/doc/user/project/merge_requests/img/approvals_starter_project_empty.png Binary files differnew file mode 100644 index 00000000000..7375820224c --- /dev/null +++ b/doc/user/project/merge_requests/img/approvals_starter_project_empty.png diff --git a/doc/user/project/merge_requests/img/approve.png b/doc/user/project/merge_requests/img/approve.png Binary files differnew file mode 100644 index 00000000000..e68259ac5c2 --- /dev/null +++ b/doc/user/project/merge_requests/img/approve.png diff --git a/doc/user/project/merge_requests/img/approve_additionally.png b/doc/user/project/merge_requests/img/approve_additionally.png Binary files differnew file mode 100644 index 00000000000..3db5a9159e5 --- /dev/null +++ b/doc/user/project/merge_requests/img/approve_additionally.png diff --git a/doc/user/project/merge_requests/img/browser_performance_testing.png b/doc/user/project/merge_requests/img/browser_performance_testing.png Binary files differnew file mode 100644 index 00000000000..eea77fb8b93 --- /dev/null +++ b/doc/user/project/merge_requests/img/browser_performance_testing.png diff --git a/doc/user/project/merge_requests/img/code_quality.gif b/doc/user/project/merge_requests/img/code_quality.gif Binary files differnew file mode 100644 index 00000000000..bab921cf38b --- /dev/null +++ b/doc/user/project/merge_requests/img/code_quality.gif diff --git a/doc/user/project/merge_requests/img/comment-on-any-diff-line.png b/doc/user/project/merge_requests/img/comment-on-any-diff-line.png Binary files differindex c2455c2d1e5..5b9844bf02f 100644 --- a/doc/user/project/merge_requests/img/comment-on-any-diff-line.png +++ b/doc/user/project/merge_requests/img/comment-on-any-diff-line.png diff --git a/doc/user/project/merge_requests/img/container_scanning.png b/doc/user/project/merge_requests/img/container_scanning.png Binary files differnew file mode 100644 index 00000000000..e47f62acd9d --- /dev/null +++ b/doc/user/project/merge_requests/img/container_scanning.png diff --git a/doc/user/project/merge_requests/img/dast_all.png b/doc/user/project/merge_requests/img/dast_all.png Binary files differnew file mode 100644 index 00000000000..b6edc928dc3 --- /dev/null +++ b/doc/user/project/merge_requests/img/dast_all.png diff --git a/doc/user/project/merge_requests/img/dast_single.png b/doc/user/project/merge_requests/img/dast_single.png Binary files differnew file mode 100644 index 00000000000..26ca4bde786 --- /dev/null +++ b/doc/user/project/merge_requests/img/dast_single.png diff --git a/doc/user/project/merge_requests/img/dependency_scanning.png b/doc/user/project/merge_requests/img/dependency_scanning.png Binary files differnew file mode 100644 index 00000000000..18df356f846 --- /dev/null +++ b/doc/user/project/merge_requests/img/dependency_scanning.png diff --git a/doc/user/project/merge_requests/img/filter_approver_merge_requests.png b/doc/user/project/merge_requests/img/filter_approver_merge_requests.png Binary files differnew file mode 100644 index 00000000000..9c386391a4f --- /dev/null +++ b/doc/user/project/merge_requests/img/filter_approver_merge_requests.png diff --git a/doc/user/project/merge_requests/img/filter_wip_merge_requests.png b/doc/user/project/merge_requests/img/filter_wip_merge_requests.png Binary files differindex 81878709487..6d511efd049 100644 --- a/doc/user/project/merge_requests/img/filter_wip_merge_requests.png +++ b/doc/user/project/merge_requests/img/filter_wip_merge_requests.png diff --git a/doc/user/project/merge_requests/img/interactive_reports.png b/doc/user/project/merge_requests/img/interactive_reports.png Binary files differnew file mode 100644 index 00000000000..9f9812dc69d --- /dev/null +++ b/doc/user/project/merge_requests/img/interactive_reports.png diff --git a/doc/user/project/merge_requests/img/license_management.png b/doc/user/project/merge_requests/img/license_management.png Binary files differnew file mode 100644 index 00000000000..cdce6b5fe38 --- /dev/null +++ b/doc/user/project/merge_requests/img/license_management.png diff --git a/doc/user/project/merge_requests/img/license_management_decision.png b/doc/user/project/merge_requests/img/license_management_decision.png Binary files differnew file mode 100644 index 00000000000..0763130c375 --- /dev/null +++ b/doc/user/project/merge_requests/img/license_management_decision.png diff --git a/doc/user/project/merge_requests/img/license_management_pipeline_tab.png b/doc/user/project/merge_requests/img/license_management_pipeline_tab.png Binary files differnew file mode 100644 index 00000000000..80ffca815b9 --- /dev/null +++ b/doc/user/project/merge_requests/img/license_management_pipeline_tab.png diff --git a/doc/user/project/merge_requests/img/license_management_settings.png b/doc/user/project/merge_requests/img/license_management_settings.png Binary files differnew file mode 100644 index 00000000000..b5490e59074 --- /dev/null +++ b/doc/user/project/merge_requests/img/license_management_settings.png diff --git a/doc/user/project/merge_requests/img/remove_approval.png b/doc/user/project/merge_requests/img/remove_approval.png Binary files differnew file mode 100644 index 00000000000..6083e1745ef --- /dev/null +++ b/doc/user/project/merge_requests/img/remove_approval.png diff --git a/doc/user/project/merge_requests/img/sast.png b/doc/user/project/merge_requests/img/sast.png Binary files differnew file mode 100644 index 00000000000..2c75592c32a --- /dev/null +++ b/doc/user/project/merge_requests/img/sast.png diff --git a/doc/user/project/merge_requests/img/security_report.png b/doc/user/project/merge_requests/img/security_report.png Binary files differnew file mode 100644 index 00000000000..ba41b707238 --- /dev/null +++ b/doc/user/project/merge_requests/img/security_report.png diff --git a/doc/user/project/merge_requests/img/vulnerability_solution.png b/doc/user/project/merge_requests/img/vulnerability_solution.png Binary files differnew file mode 100644 index 00000000000..7443b9b6eea --- /dev/null +++ b/doc/user/project/merge_requests/img/vulnerability_solution.png diff --git a/doc/user/project/merge_requests/index.md b/doc/user/project/merge_requests/index.md index 593eb80e044..11d5269e386 100644 --- a/doc/user/project/merge_requests/index.md +++ b/doc/user/project/merge_requests/index.md @@ -23,7 +23,7 @@ With GitLab merge requests, you can: - Assign it to any registered user, and change the assignee how many times you need - Assign a [milestone](../../project/milestones/index.md) and track the development of a broader implementation - Organize your issues and merge requests consistently throughout the project with [labels](../../project/labels.md) -- Add a time estimation and the time spent with that merge request with [Time Tracking](../../../workflow/time_tracking.html#time-tracking) +- Add a time estimation and the time spent with that merge request with [Time Tracking](../../../workflow/time_tracking.md#time-tracking) - [Resolve merge conflicts from the UI](#resolve-conflicts) - Enable [fast-forward merge requests](#fast-forward-merge-requests) - Enable [semi-linear history merge requests](#semi-linear-history-merge-requests) as another security layer to guarantee the pipeline is passing in the target branch @@ -33,9 +33,16 @@ With GitLab merge requests, you can: With **[GitLab Enterprise Edition][ee]**, you can also: -- View the deployment process across projects with [Multi-Project Pipeline Graphs](https://docs.gitlab.com/ee/ci/multi_project_pipeline_graphs.html#multi-project-pipeline-graphs) **[PREMIUM]** -- Request [approvals](https://docs.gitlab.com/ee/user/project/merge_requests/merge_request_approvals.html) from your managers **[STARTER]** -- Analyze the impact of your changes with [Code Quality reports](https://docs.gitlab.com/ee/user/project/merge_requests/code_quality.html) **[STARTER]** +- Prepare a full review and submit it once it's ready with [Merge Request Reviews](../../discussions/index.md#merge-request-reviews-premium) **[PREMIUM]** +- View the deployment process across projects with [Multi-Project Pipelines](../../../ci/multi_project_pipelines.md) **[PREMIUM]** +- Request [approvals](merge_request_approvals.md) from your managers **[STARTER]** +- Analyze the impact of your changes with [Code Quality reports](code_quality.md) **[STARTER]** +- Manage the licenses of your dependencies with [License Management](#license-management-ultimate) **[ULTIMATE]** +- Analyze your source code for vulnerabilities with [Static Application Security Testing](sast.md) **[ULTIMATE]** +- Analyze your running web applications for vulnerabilities with [Dynamic Application Security Testing](dast.md) **[ULTIMATE]** +- Analyze your dependencies for vulnerabilities with [Dependency Scanning](dependency_scanning.md) **[ULTIMATE]** +- Analyze your Docker images for vulnerabilities with [Container Scanning](container_scanning.md) **[ULTIMATE]** +- Determine the performance impact of changes with [Browser Performance Testing](#browser-performance-testing-premium) **[PREMIUM]** ## Use cases @@ -43,19 +50,21 @@ A. Consider you are a software developer working in a team: 1. You checkout a new branch, and submit your changes through a merge request 1. You gather feedback from your team +1. You work on the implementation optimizing code with [Code Quality reports](code_quality.md) **[STARTER]** 1. You verify your changes with [JUnit test reports](../../../ci/junit_test_reports.md) in GitLab CI/CD -1. You request the approval from your manager -1. Your manager pushes a commit with his final review, [approves the merge request](https://docs.gitlab.com/ee/user/project/merge_requests/merge_request_approvals.html), and set it to [merge when pipeline succeeds](#merge-when-pipeline-succeeds) (Merge Request Approvals are available in GitLab Starter) +1. You avoid using dependencies whose license is not compatible with your project with [License Management reports](license_management.md) **[ULTIMATE]** +1. You request the [approval](#merge-request-approvals-starter) from your manager +1. Your manager pushes a commit with their final review, [approves the merge request](merge_request_approvals.md), and set it to [merge when pipeline succeeds](#merge-when-pipeline-succeeds) (Merge Request Approvals are available in GitLab Starter) 1. Your changes get deployed to production with [manual actions](../../../ci/yaml/README.md#whenmanual) for GitLab CI/CD 1. Your implementations were successfully shipped to your customer -B. Consider you're a web developer writing a webpage for your company's: +B. Consider you're a web developer writing a webpage for your company's website: 1. You checkout a new branch, and submit a new page through a merge request 1. You gather feedback from your reviewers 1. Your changes are previewed with [Review Apps](../../../ci/review_apps/index.md) 1. You request your web designers for their implementation -1. You request the [approval](https://docs.gitlab.com/ee/user/project/merge_requests/merge_request_approvals.html) from your manager **[STARTER]** +1. You request the [approval](merge_request_approvals.md) from your manager **[STARTER]** 1. Once approved, your merge request is [squashed and merged](squash_and_merge.md), and [deployed to staging with GitLab Pages](https://about.gitlab.com/2016/08/26/ci-deployment-and-environments/) 1. Your production team [cherry picks](#cherry-pick-changes) the merge commit into production @@ -159,6 +168,21 @@ in a Merge Request. To do so, click the **...** button in the gutter of the Merg  +## Perform a Review **[PREMIUM]** + +Start a review in order to create multiple comments on a diff and publish them once you're ready. +Starting a review allows you to get all your thoughts in order and ensure you haven't missed anything +before submitting all your comments. + +[Learn more about Merge Request Reviews](../../discussions/index.md#merge-request-reviews-premium) + +## Squash and merge + +GitLab allows you to squash all changes present in a merge request into a single +commit when merging, to allow for a neater commit history. + +[Learn more about squash and merge.](squash_and_merge.md) + ## Suggest changes > [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/issues/18008) in GitLab 11.6. @@ -256,6 +280,111 @@ have been marked as a **Work In Progress**. [Learn more about settings a merge request as "Work In Progress".](work_in_progress_merge_requests.md) +## Merge request approvals **[STARTER]** + +> Included in [GitLab Starter][products]. + +If you want to make sure every merge request is approved by one or more people, +you can enforce this workflow by using merge request approvals. Merge request +approvals allow you to set the number of necessary approvals and predefine a +list of approvers that will need to approve every merge request in a project. + +[Read more about merge request approvals.](merge_request_approvals.md) + +## Code Quality **[STARTER]** + +> Introduced in [GitLab Starter][products] 9.3. + +If you are using [GitLab CI][ci], you can analyze your source code quality using +the [Code Climate][cc] analyzer [Docker image][cd]. Going a step further, GitLab +can show the Code Climate report right in the merge request widget area. + +[Read more about Code Quality reports.](code_quality.md) + +## Browser Performance Testing **[PREMIUM]** + +> Introduced in [GitLab Premium][products] 10.3. + +If your application offers a web interface and you are using [GitLab CI/CD][ci], you can quickly determine the performance impact of pending code changes. GitLab uses [Sitespeed.io][sitespeed], a free and open source tool for measuring the performance of web sites, to analyze the performance of specific pages. + +GitLab runs the [Sitespeed.io container][sitespeed-container] and displays the difference in overall performance scores between the source and target branches. + +[Read more about Browser Performance Testing.](browser_performance_testing.md) + +## License Management **[ULTIMATE]** + +> Introduced in [GitLab Ultimate][products] 11.0. + +If you are using [GitLab CI/CD][ci], you can search your dependencies for their +licenses using License Management. +Going a step further, GitLab can show the licenses report right in the +merge request widget area. + +[Read more about License Management reports.](license_management.md) + +## Security reports **[ULTIMATE]** + +GitLab can scan and report any vulnerabilities found in your project. The +following security reports are available: + +- [Static Application Security Testing reports](sast.md) - Analyze your source + code for known vulnerabilities using Static Application Security Testing (SAST) + and see the security report right in your merge requests. +- [Dynamic Application Security Testing reports](dast.md) - Analyze your running + web application(s) for known vulnerabilities using Dynamic Application Security + Testing (DAST) and see the security report right in your merge requests. +- [Dependency Scanning reports](dependency_scanning.md) - Analyze your + dependencies for known vulnerabilities using Dependency Scanning and see the + security report right in your merge requests. +- [Container Scanning reports](container_scanning.md) - Analyze your Docker + images for known vulnerabilities and see the security report right in your + merge requests. + +### Interacting with security reports **[ULTIMATE]** + +> Introduced in [GitLab Ultimate][products] 10.8. + +CAUTION: **Warning:** +This feature is currently [Alpha](https://about.gitlab.com/handbook/product/#alpha-beta-ga) and while you can start using it, it may receive important changes in the future. + +Each security vulnerability in the report is actionable. Clicking on an entry, +a detailed information will pop up with two different possible options: + +- **Dismiss vulnerability** - Dismissing a vulnerability will place a <s>strikethrough</s> styling on it. +- **Create issue** - The new issue will have the title and description + pre-populated with the information from the vulnerability report and is created as [confidential](../issues/confidential_issues.md) by default. +- **Solution** - For some vulnerabilities ([Dependency Scanning](dependency_scanning.md) and [Container Scanning](container_scanning.md)) + a solution is provided for how to fix the vulnerability. + + + +You can also revert your dismissal or see the linked issue after the action has +been taken. + +### Solutions for Dependency Scanning **[ULTIMATE]** + +> Introduced in [GitLab Ultimate][products] 11.7. + +CAUTION: **Warning:** Automatic Patch creation is only available for a subset of [Dependency Scanning](dependency_scanning.md). At the moment only Node.JS projects managed with yarn are supported. + +Some Vulnerabilities can be fixed by applying a patch that is automatically generated by GitLab. To apply the fix: + +1. Download and review the patch file `remediation.patch`. +2. Ensure your local project has the same commit checked out that was used to generate the patch. +3. Run `git apply remediation.patch`. +4. Verify and commit the changes to your branch. + + + +## Live preview with Review Apps + +If you configured [Review Apps](https://about.gitlab.com/features/review-apps/) for your project, +you can preview the changes submitted to a feature-branch through a merge request +in a per-branch basis. No need to checkout the branch, install and preview locally; +all your changes will be available to preview by anyone with the Review Apps link. + +[Read more about Review Apps.](../../../ci/review_apps/index.md) + ## Merge request diff file navigation When reviewing changes in the **Changes** tab the diff can be navigated using @@ -448,5 +577,11 @@ And to check out a particular merge request: git checkout origin/merge-requests/1 ``` +[products]: https://about.gitlab.com/products/ "GitLab products page" [protected branches]: ../protected_branches.md +[ci]: ../../../ci/README.md +[cc]: https://codeclimate.com/ +[cd]: https://hub.docker.com/r/codeclimate/codeclimate/ +[sitespeed]: https://www.sitespeed.io +[sitespeed-container]: https://hub.docker.com/r/sitespeedio/sitespeed.io/ [ee]: https://about.gitlab.com/pricing/ "GitLab Enterprise Edition" diff --git a/doc/user/project/merge_requests/license_management.md b/doc/user/project/merge_requests/license_management.md new file mode 100644 index 00000000000..9bed3c0480a --- /dev/null +++ b/doc/user/project/merge_requests/license_management.md @@ -0,0 +1,102 @@ +# License Management **[ULTIMATE]** + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/issues/5483) +in [GitLab Ultimate](https://about.gitlab.com/pricing/) 11.0. + +## Overview + +If you are using [GitLab CI/CD](../../../ci/README.md), you can search your project dependencies for their licenses +using License Management. + +You can take advantage of License Management by either [including the CI job](../../../ci/examples/license_management.md) in +your existing `.gitlab-ci.yml` file or by implicitly using +[Auto License Management](../../../topics/autodevops/index.md#auto-license-management-ultimate) +that is provided by [Auto DevOps](../../../topics/autodevops/index.md). + +In addition, you can [manually approve or blacklist](#manual-license-management) licenses in the project's settings. + +Going a step further, GitLab can show the licenses list right in the merge +request widget area, highlighting the presence of licenses you don't want to use, or new +ones that need a decision. + +## Use cases + +It helps you find what licenses your project uses in its dependencies, and decide for each of then +whether to allow it or forbid it. For example, your application is using an external (open source) +library whose license is incompatible with yours. + +## Supported languages and package managers + +The following languages and package managers are supported. + +| Language | Package managers | +|------------|-------------------------------------------------------------------| +| JavaScript | [Bower](https://bower.io/), [npm](https://www.npmjs.com/) | +| Go | [Godep](https://github.com/tools/godep), go get | +| Java | [Gradle](https://gradle.org/), [Maven](https://maven.apache.org/) | +| .NET | [Nuget](https://www.nuget.org/) | +| Python | [pip](https://pip.pypa.io/en/stable/) | +| Ruby | [gem](https://rubygems.org/) | + +## How it works + +First of all, you need to define a job in your `.gitlab-ci.yml` file that generates the +[License Management report artifact](../../../ci/yaml/README.md#artifactsreportslicense_management-ultimate). +For more information on how the License Management job should look like, check the +example on [Dependencies license management with GitLab CI/CD](../../../ci/examples/license_management.md). + +GitLab then checks this report, compares the licenses between the source and target +branches, and shows the information right on the merge request. +Blacklisted licenses will be clearly visible with an `x` red icon next to them +as well as new licenses which need a decision from you. + +NOTE: **Note:** +If the license management report doesn't have anything to compare to, no information +will be displayed in the merge request area. That is the case when you add the +`license_management` job in your `.gitlab-ci.yml` for the first time. +Consecutive merge requests will have something to compare to and the license +management report will be shown properly. + + + +If you are a project or group Maintainer, you can click on a license to be given +the choice to approve it or blacklist it. + + + +### Project policies for license management + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/issues/5940) +in [GitLab Ultimate](https://about.gitlab.com/pricing/) 11.4. + +From the project's settings: + +- The list of licenses and their status can be managed. +- Licenses can be manually approved or blacklisted. + +To approve or blacklist a license: + +1. Either use the **Manage licenses** button in the merge request widget, or + navigate to the project's **Settings > CI/CD** and expand the + **License Management** section. +1. Click the **Add a license** button. +1. In the **License name** dropdown, either: + - Select one of the available licenses. You can search for licenses in the field + at the top of the list. + - Enter arbitrary text in the field at the top of the list. This will cause the text to be + added as a license name to the list. +1. Select the **Approve** or **Blacklist** radio button to approve or blacklist respectively + the selected license. + +  + +## License Management report under pipelines + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/issues/5491) +in [GitLab Ultimate](https://about.gitlab.com/pricing/) 11.2. + +From your project's left sidebar, navigate to **CI/CD > Pipelines** and click on the +pipeline ID that has a `license_management` job to see the Licenses tab with the listed +licenses (if any). + + diff --git a/doc/user/project/merge_requests/merge_request_approvals.md b/doc/user/project/merge_requests/merge_request_approvals.md new file mode 100644 index 00000000000..b91e07b9812 --- /dev/null +++ b/doc/user/project/merge_requests/merge_request_approvals.md @@ -0,0 +1,299 @@ +# Merge request approvals **[STARTER]** + +> Introduced in [GitLab Enterprise Edition 7.12](https://about.gitlab.com/2015/06/22/gitlab-7-12-released/#merge-request-approvers-ee-only). + +NOTE: **Note:** +If you are running a self-managed instance, the new interface shown on +this page will not be available unless the feature flag +`approval_rules` is enabled, which can be done from the Rails console by +instance administrators. + +Use these commands to start the Rails console: + +```sh +# Omnibus GitLab +gitlab-rails console + +# Installation from source +cd /home/git/gitlab +sudo -u git -H bin/rails console RAILS_ENV=production +``` + +Then run `Feature.enable(:approval_rules)` to enable the feature flag. + +The documentation for the older interface can be accessed +[here](/11.7/ee/user/project/merge_requests/merge_request_approvals.html). + +## Overview + +Merge request approvals enable enforced code review by requiring specified people to approve a merge request before it can be unblocked for merging. + +## Use cases + +1. Enforcing review of all code that gets merged into a repository. +2. Specifying code maintainers for an entire repository. +3. Specifying reviewers for a given proposed code change. +4. Specifying categories of reviewers, such as BE, FE, QA, DB, etc., for all proposed code changes. + +## Editing approvals + +To edit the merge request approvals: + +1. Navigate to your project's **Settings > General** and expand + **Merge request approvals**. + +  + +1. Click **Edit**. +1. Search for users or groups that will be [eligible to approve](#eligible-approvers) + merge requests and click the **Add** button to add them as approvers. Note: selecting + approvers is optional. +1. Set the minimum number of required approvals under the **No. approvals required** + box. Note: the minimum can be 0. +1. Click **Update approvers**. + +  + +The steps above are the minimum required to get approvals working in your +merge requests, but there are a couple more options available that might be +suitable to your workflow: + +- Choose whether the default settings can be + [overridden per merge request](#overriding-the-merge-request-approvals-default-settings) +- Choose whether [approvals will be reset with new pushed commits](#resetting-approvals-on-push) + +## Editing approvals **[PREMIUM]** + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/issues/1979) in [GitLab Premium](https://about.gitlab.com/pricing/) 11.8. + +For GitLab Premium, [multiple approver rules](#multiple-approval-rules-premium) can be configured. To configure the merge +request approval rules: + +1. Navigate to your project's **Settings > General** and expand **Merge request approvals**. +1. Click **Add approvers** to create a new approval rule. +1. Just like in [GitLab Starter](#editing-approvals), select the approval members and aprovals required. +1. Give the approval rule a name that describes the set of approvers selected. +1. Click **Add approvers** to submit the new rule. + +  + +## Multiple approval rules **[PREMIUM]** + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/issues/1979) in [GitLab Premium](https://about.gitlab.com/pricing/) 11.8. + +For GitLab Premium, a merge request's overall approval status is determined by a set of rules. Each rule contains: + +- A set of [eligible approvers](#eligible-approvers). +- A minimum number of approvals required. + +When an [eligible approver](#eligible-approvers) approves a merge request, it will reduce the number of approvals left for +all rules that the approver belongs to. + + + +If no approval rules are set, then the overall minimum number of approvals required can be configured. With no approval rules, +any [eligible approver](#eligible-approvers) may approve. + +## Eligible approvers + +The following can approve merge requests: + +- Users being added as approvers at project or merge request level. +- [Code owners](../code_owners.md) related to the merge request ([introduced](https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/7933) in [GitLab Starter](https://about.gitlab.com/pricing/) 11.5). + +An individual user can be added as an approver for a project if they are a member of: + +- The project. +- The project's immediate parent group. +- A group that has access to the project via a [share](../members/share_project_with_groups.md). + +A group can also be added as an approver. [In the future](https://gitlab.com/gitlab-org/gitlab-ee/issues/2048), +group approvers will be restricted. + +If a user is added as an individual approver and is also part of a group approver, +then that user is just counted once. The merge request author and users who have authored +commits in the merge request do not count as eligible approvers, +unless [self-approval] is explicitly enabled on the project settings. + +### Implicit approvers + +If the number of required approvals is greater than the number of approvers, +other users will become implicit approvers to fill the gap. +Those implicit approvers include members of the given project with Developer role or higher. + +## Adding or removing an approval + +If approvals are activated for the given project, when a user visits an open +merge request, depending on their [eligibility](#eligible-approvers), one of +the following is possible: + +- **They are not an eligible approver**: They cannot do anything with respect + to approving this merge request. + +- **They have not approved this merge request**: + + - If the required number of approvals has _not_ been yet met, they can approve + it by clicking the displayed **Approve** button. +  + - If the required number of approvals has already been met, they can still + approve it by clicking the displayed **Approve additionally** button. +  + +- **They have already approved this merge request**: They can remove their approval. + +  + +NOTE: **Note:** +The merge request author is only allowed to approve their own merge request +if [self-approval] is enabled on the project settings. + +For a given merge request, if the approval restrictions have been satisfied, +the merge request is unblocked and can be merged. +Note, that meeting the required number of approvals is a necessary, but not +sufficient condition for unblocking a merge request from being merged. There +are other conditions that may block it, such as merge conflicts, +[pending discussions](../../discussions/index.md#only-allow-merge-requests-to-be-merged-if-all-discussions-are-resolved) +or a [failed CI/CD pipeline](merge_when_pipeline_succeeds.md). + +## Code Owners approvals **[PREMIUM]** + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/issues/4418) in [GitLab Premium](https://about.gitlab.com/pricing/) 11.9. + +It is possible to require at least one approval for each entry in the +[`CODEOWNERS` file](../code_owners.md) that matches a file changed in +the merge request. To enable this feature: + +1. Navigate to your project's **Settings > General** and expand + **Merge request approvals**. +1. Tick the **Require approval from code owners** checkbox + checkbox. +1. Click **Save changes**. + +When this feature is enabled, all merge requests will need approval +from one code owner per matched rule before it can be merged. + +## Overriding the merge request approvals default settings + +> Introduced in GitLab Enterprise Edition 9.4. + +NOTE: **Note:** +If you are using GitLab Premium, things are a little different with [multiple approval rules](#multiple-approval-rules-premium). +Read the differences [in GitLab Premium when overriding merge request approvals](#overriding-merge-request-approvals-default-settings-premium). + +If approvals are [set at the project level](#editing-approvals), the +default configuration (number of required approvals and approvers) can be +overridden for each merge request in that project. + +One possible scenario would be to to assign a group of approvers at the project +level and change them later when creating or editing the merge request. + +First, you have to enable this option in the project's settings: + +1. Navigate to your project's **Settings > General** and expand + **Merge request approvals** +1. Tick the "Can override approvers and approvals required per merge request" + checkbox + +  + +1. Click **Save changes** + +NOTE: **Note:** +If approver overriding is enabled +and the project level approvers are changed after a merge request is created, +the merge request retains the previous approvers. +However, the approvers can be changed by [editing the merge request](#overriding-the-merge-request-approvals-default-settings). + +--- + +The default approval settings can now be overridden when creating a +[merge request](index.md) or by editing it after it's been created: + +1. Click **Edit** under the **Approvers** section. +1. Search for users or groups that will be [eligible to approve](#eligible-approvers) + merge requests and click the **Add** button to add them as approvers or + remove existing approvers that were set in the project's settings. +1. If you want to change the number of required approvals, set a new number + in the **No. approvals required** box. +1. Click **Update approvers**. + +There are however some restrictions: + +- The amount of required approvals, if changed, must be greater than the default + set at the project level. This ensures that you're not forced to adjust settings + when someone is unavailable for approval, yet the process is still enforced. + +NOTE: **Note:** +If you are contributing to a forked project, things are a little different. +Read what happens when the +[source and target branches are not the same](#merge-requests-with-different-source-branch-and-target-branch-projects). + +## Overriding merge request approvals default settings **[PREMIUM]** + +In GitLab Premium, when the approval rules are [set at the project level](#editing-approvals-premium), and +**Can override approvers and approvals required per merge request** is checked, there are a few more +restrictions (compared to [GitLab Starter](#overriding-the-merge-request-approvals-default-settings)): + +- Approval rules can be added to an MR with no restriction. +- For project sourced approval rules, editing and removing approvers is not allowed. +- The approvals required of all approval rules is configurable, but if a rule is backed by a project rule, then it is restricted +to the minimum approvals required set in the project's corresponding rule. + +## Resetting approvals on push + +If approvals are [set at the project level](#editing-approvals), +you can choose whether all approvals on a merge request are removed when +new commits are pushed to the source branch of the merge request: + +1. Navigate to your project's **Settings > General** and expand + **Merge request approvals** +1. Tick the "Remove all approvals in a merge request when new commits are pushed to its source branch" + checkbox + +  + +1. Click **Save changes** + +NOTE: **Note:** +Approvals do not get reset when [rebasing a merge request](fast_forward_merge.md) +from the UI. +However, approvals will be reset if the target branch is changed. + +If you want approvals to persist, independent of changes to the merge request, +turn this setting to off by unchecking the box and saving the changes. + +## Allowing merge request authors to approve their own merge requests + +You can allow merge request authors to self-approve merge requests by +enabling it [at the project level](#editing-approvals). Authors +also need to be included in the approvers list in order to be able to +approve their merge request. + +1. Navigate to your project's **Settings > General** and expand + **Merge request approvals** +1. Tick the "Enable self approval of merge requests" checkbox +1. Click **Save changes** + +## Merge requests with different source branch and target branch projects + +If the merge request source branch and target branch belong to different +projects (which happens in merge requests in forked projects), everything is +with respect to the target branch's project (typically the original project). +In particular, since the merge request in this case is part of the target +branch's project, the relevant settings are the target project's. The source +branch's project settings are not applicable. Even if you start the merge +request from the source branch's project UI, pay attention to the created merge +request itself. It belongs to the target branch's project. + +[self-approval]: #allowing-merge-request-authors-to-approve-their-own-merge-requests + +## Approver suggestions + +Approvers are suggested for merge requests based on the previous authors of the files affected by the merge request. + +## Filtering merge requests by approvers + +To filter merge requests by an individual approver, you can type (or select from +the dropdown) `approver` and select the user. + + diff --git a/doc/user/project/merge_requests/sast.md b/doc/user/project/merge_requests/sast.md new file mode 100644 index 00000000000..7ebf5ad69ac --- /dev/null +++ b/doc/user/project/merge_requests/sast.md @@ -0,0 +1,71 @@ +# Static Application Security Testing (SAST) **[ULTIMATE]** + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/issues/3775) +in [GitLab Ultimate](https://about.gitlab.com/pricing/) 10.3. + +NOTE: **4 of the top 6 attacks were application based.** +Download our whitepaper, +["A Seismic Shift in Application Security"](https://about.gitlab.com/resources/whitepaper-seismic-shift-application-security/) +to learn how to protect your organization. + +## Overview + +If you are using [GitLab CI/CD](../../../ci/README.md), you can analyze your source code for known +vulnerabilities using Static Application Security Testing (SAST). + +You can take advantage of SAST by either [including the CI job](../../../ci/examples/sast.md) in +your existing `.gitlab-ci.yml` file or by implicitly using +[Auto SAST](../../../topics/autodevops/index.md#auto-sast-ultimate) +that is provided by [Auto DevOps](../../../topics/autodevops/index.md). + +Going a step further, GitLab can show the vulnerability list right in the merge +request widget area. + +## Use cases + +- Your code has a potentially dangerous attribute in a class, or unsafe code + that can lead to unintended code execution. +- Your application is vulnerable to cross-site scripting (XSS) attacks that can + be leveraged to unauthorized access to session data + +## Supported languages and frameworks + +The following languages and frameworks are supported. + +| Language / framework | Scan tool | +|-------------------------|----------------------------------------------------------------------------------------| +| .NET | [Security Code Scan](https://security-code-scan.github.io) | +| Any | [Gitleaks](https://github.com/zricethezav/gitleaks), [TruffleHog](https://github.com/dxa4481/truffleHog) and [Diffence](https://github.com/techjacker/diffence) (secret detectors) | +| C/C++ | [Flawfinder](https://www.dwheeler.com/flawfinder/) | +| Go | [Gosec](https://github.com/securego/gosec) | +| Groovy (Ant, Gradle, Maven and SBT) | [find-sec-bugs](https://find-sec-bugs.github.io/) | +| Java (Ant, Gradle, Maven and SBT) | [find-sec-bugs](https://find-sec-bugs.github.io/) | +| JavaScript | [ESLint security plugin](https://github.com/nodesecurity/eslint-plugin-security) | +| Node.js | [NodeJsScan](https://github.com/ajinabraham/NodeJsScan) | +| PHP | [phpcs-security-audit](https://github.com/FloeDesignTechnologies/phpcs-security-audit) | +| Python | [bandit](https://github.com/PyCQA/bandit) | +| Ruby on Rails | [brakeman](https://brakemanscanner.org) | +| Scala (Ant, Gradle, Maven and SBT) | [find-sec-bugs](https://find-sec-bugs.github.io/) | +| Typescript | [TSLint Config Security](https://github.com/webschik/tslint-config-security/) | + +## How it works + +First of all, you need to define a job in your `.gitlab-ci.yml` file that generates the +[SAST report artifact](../../../ci/yaml/README.md#artifactsreportssast-ultimate). +For more information on how the SAST job should look like, check the +example on [Static Application Security Testing with GitLab CI/CD](../../../ci/examples/sast.md). + +GitLab then checks this report, compares the found vulnerabilities between the source and target +branches, and shows the information right on the merge request. + + + +## Security report under pipelines + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/issues/3776) +in [GitLab Ultimate](https://about.gitlab.com/pricing) 10.6. + +Visit any pipeline page which has a `sast` job and you will be able to see +the security report tab with the listed vulnerabilities (if any). + + diff --git a/doc/user/project/merge_requests/sast_docker.md b/doc/user/project/merge_requests/sast_docker.md new file mode 100644 index 00000000000..70b269046e5 --- /dev/null +++ b/doc/user/project/merge_requests/sast_docker.md @@ -0,0 +1,5 @@ +--- +redirect_to: 'container_scanning.md' +--- + +This document was moved to [another location](container_scanning.md). diff --git a/doc/user/project/milestones/burndown_charts.md b/doc/user/project/milestones/burndown_charts.md new file mode 100644 index 00000000000..0ad08da8ff5 --- /dev/null +++ b/doc/user/project/milestones/burndown_charts.md @@ -0,0 +1,70 @@ +# Burndown Charts **[STARTER]** + +> **Notes:** +> - [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/1540) in [GitLab Starter](https://about.gitlab.com/pricing/) 9.1 for project milestones. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/5354) in [GitLab Premium](https://about.gitlab.com/pricing/) 10.8 for group milestones. +> - [Added](https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/6495) to [GitLab Starter](https://about.gitlab.com/pricing/) 11.2 for group milestones. +> - Closed or reopened issues prior to GitLab 9.1 won't have a `closed_at` +> value, so the burndown chart considers them as closed on the milestone +> `start_date`. In that case, a warning will be displayed. + +## Overview + +Burndown Charts are visual representations of the progress of completing a milestone. + + + +At a glance, you see the current state for the completion a given milestone. +Without them, you would have to organize the data from the milestone and plot it +yourself to have the same sense of progress. + +GitLab Starter plots it for you and presents it in a clear and beautiful chart. + +For an overview, check the video demonstration on [Mapping Work Versus Time, With Burndown Charts](https://about.gitlab.com/2017/04/25/mapping-work-to-do-versus-time-with-burndown-charts/). + +## Use cases + +Burndown Charts, in general, are used for tracking and analyzing the completion of +a milestone. Therefore, their use cases are tied to the +[use you are assigning your milestone to](index.md). + +To exemplify, suppose you lead a team of developers in a large company, +and you follow this workflow: + +- Your company set the goal for the quarter to deliver 10 new features for your app + in the upcoming major release. +- You create a milestone, and remind your team to assign that milestone to every new issue + and merge request that's part of the launch of your app. +- Every week, you open the milestone, visualize the progress, identify the gaps, + and help your team to get their work done. +- Every month, you check in with your supervisor, and show the progress of that milestone + from the Burndown Chart. +- By the end of the quarter, your team successfully delivered 100% of that milestone, as + it was taken care of closely throughout the whole quarter. + +## How it works + +A Burndown Chart is available for every project or group milestone that has been attributed a **start +date** and a **due date**. + +Find your project's **Burndown Chart** under **Project > Issues > Milestones**, +and select a milestone from your current ones, while for group's, access the **Groups** dashboard, +select a group, and go through **Issues > Milestones** on the sidebar. + +NOTE: **Note:** +You're able to [promote project](https://docs.gitlab.com/ee/user/project/milestones/#promoting-project-milestones-to-group-milestones) to group milestones and still see the **Burndown Chart** for them, respecting license limitations. + +The chart indicates the project's progress throughout that milestone (for issues assigned to it). + +In particular, it shows how many issues were or are still open for a given day in the +milestone's corresponding period. + +The Burndown Chart tracks when issues were created and when they were last closed—not their full history. For each day, it takes the number of issues still open and issues created that day and subtracts the number of issues closed that day. +**Issues that were created and assigned a milestone before its start date—and remain open as of the start date—are considered as having been opened on the start date**. Therefore, when the milestone start date is changed the number of opened issues on each day may change. +Reopened issues are +considered as having been opened on the day after they were last closed. + +The Burndown Chart can also be toggled to display the cumulative open issue +weight for a given day. When using this feature, make sure issue weights have +been properly assigned, since an open issue with no weight adds zero to the +cumulative value. diff --git a/doc/user/project/milestones/img/burndown_chart.png b/doc/user/project/milestones/img/burndown_chart.png Binary files differnew file mode 100644 index 00000000000..e06b24f9907 --- /dev/null +++ b/doc/user/project/milestones/img/burndown_chart.png diff --git a/doc/user/project/milestones/index.md b/doc/user/project/milestones/index.md index e6033ca8655..0dfd87f91a5 100644 --- a/doc/user/project/milestones/index.md +++ b/doc/user/project/milestones/index.md @@ -83,7 +83,12 @@ From the project issue/merge request list pages and the group issue/merge reques ### Filtering in issue boards -From [project issue boards](../issue_board.md), you can filter by both group milestones and project milestones in the [search and filter bar](../../search/index.md#issue-boards). +- From [project issue boards](../issue_board.md), you can filter by both group milestones and project milestones in the [search and filter bar](../../search/index.md#issue-boards). +- From [group issue boards](../issue_board.md#group-issue-boards), you can filter by only group milestones in the [search and filter bar](../../search/index.md#issue-boards). **[PREMIUM]** +- From [project issue boards](../issue_board.md), you can filter by both group milestones and project milestones in the [issue board configuration](../issue_board.md#board-with-configuration). **[STARTER]** +- From [group issue boards](../issue_board.md#group-issue-boards) you can filter by only group milestones in the [issue board configuration](../issue_board.md#board-with-configuration). **[PREMIUM]** + + ### Special milestone filters @@ -108,6 +113,7 @@ Not all features in the project milestone view are available in the group milest | Start date and due date | ✓ | ✓ | | Total issue time spent | ✓ | ✓ | | Total issue weight | ✓ | | +| Burndown chart | ✓ | ✓ | The milestone view shows the title and description. @@ -118,6 +124,17 @@ These features are only available for project milestones and not group milestone - Issues assigned to the milestone are displayed in three columns: Unstarted issues, ongoing issues, and completed issues. - Merge requests assigned to the milestone are displayed in four columns: Work in progress merge requests, waiting for merge, rejected, and closed. - Participants and labels that are used in issues and merge requests that have the milestone assigned are displayed. +- [Burndown chart](#project-burndown-charts-starter). + +### Project Burndown Charts **[STARTER]** + +For project milestones in [GitLab Starter](https://about.gitlab.com/pricing), a [burndown chart](burndown_charts.md) is in the milestone view, showing the progress of completing a milestone. + + + +### Group Burndown Charts **[PREMIUM]** + +For group milestones in [GitLab Premium](https://about.gitlab.com/pricing), a [burndown chart](burndown_charts.md) is in the milestone view, showing the progress of completing a milestone. ### Milestone sidebar diff --git a/doc/user/project/operations/feature_flags.md b/doc/user/project/operations/feature_flags.md new file mode 100644 index 00000000000..a5db3d70bb9 --- /dev/null +++ b/doc/user/project/operations/feature_flags.md @@ -0,0 +1,177 @@ +# Feature Flags **[PREMIUM]** + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/11845) in GitLab 11.4. + +CAUTION: **Warning:** +This an _alpha_ feature and is subject to change at any time without +prior notice. + +Feature flags allow you to ship a project in different flavors by +dynamically toggling certain functionality. + +## Overview + +Feature Flags offer a feature toggle system for your application. They enable teams +to achieve Continuous Delivery by deploying new features to production at smaller +batches for controlled testing, separating feature delivery from customer launch. +This helps reducing risk and allows you to easily manage which features to enable. + +GitLab offers a Feature Flags interface that allows you to create, toggle and +remove feature flags. + +## How it works + +Underneath, GitLab uses [unleash](https://github.com/Unleash/unleash), a feature +toggle service. GitLab provides an API where your application can talk to and get the +list of feature flags you set in GitLab. + +The application must be configured to talk to GitLab, so that's up to the +developers to use a compatible [client library](#client-libraries) and +integrate it in their app. + +By setting a flag active or inactive via GitLab, your application will automatically +know which features to enable or disable respectively. + +## Adding a new feature flag + +To add a new feature flag: + +1. Navigate to your project's **Operations > Feature Flags**. +1. Click on the **New Feature Flag** button. +1. Give it a name. + + NOTE: **Note:** + A name can contain only lowercase letters, digits, underscores (`_`) + and dashes (`-`), must start with a letter, and cannot end with a dash (`-`) + or an underscore (`_`). + +1. Give it a description (optional, 255 characters max). +1. Define environment [specs](#define-environment-specs). If you want the flag on by default, enable the catch-all [wildcard spec (`*`)](#define-environment-specs) +1. Click **Create feature flag**. + +Once a feature flag is created, the list of existing feature flags will be presented +with ability to edit or remove them. + +To make a feature flag active or inactive, click the pencil icon to edit it, +and toggle the status for each [spec](#define-environment-specs). + + + +## Define environment specs + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/issues/8621) in GitLab 11.8. + +In general, an application is deployed to multiple environments, such as +production, staging and [review apps](../../../ci/review_apps/index.md). +For example, you may not want to enable a feature flag on production until your QA team has +first confirmed that the feature is working correctly on testing environments. + +To handle these situations, you can enable a feature flag on a particular environment +with [Environment specs](../../../ci/environments.md#scoping-environments-with-specs-premium). +You can define multiple specs per flag so that you can control your feature flag more granularly. + +To define specs for each environment: + +1. Navigate to your project's **Operations > Feature Flags**. +1. Click on the **New Feature Flag** button or edit an existing flag. +1. Set the status of the default [spec](../../../ci/environments.md#scoping-environments-with-specs-premium) (`*`). This status will be used for _all_ environments. +1. If you want to enable/disable the feature on a specific environment, create a new [spec](../../../ci/environments.md#scoping-environments-with-specs-premium) and type the environment name. +1. Set the status of the additional spec. This status takes precedence over the default spec's status since we always use the most specific match available. +1. Click **Create feature flag** or **Update feature flag**. + + + +NOTE: **NOTE** +We'd highly recommend you to use the [Environment](../../../ci/environments.md) +feature in order to quickly assess which flag is enabled per environment. + +## Integrating with your application + +In order to use Feature Flags, you need to first +[get the access credentials](#configuring-feature-flags) from GitLab and then +prepare your application and hook it with a [client library](#client-libraries). + +### Configuring Feature Flags + +To get the access credentials that your application will need to talk to GitLab: + +1. Navigate to your project's **Operations > Feature Flags**. +1. Click on the **Configure** button to see the values: + - **API URL**: URL where the client (application) connects to get a list of feature flags. + - **Instance ID**: Unique token that authorizes the retrieval of the feature flags. + - **Application name**: The name of the running environment. For instance, + if the application runs for production server, application name would be + `production` or similar. This value is used for + [the environment spec evaluation](#define-environment-specs). + +NOTE: **Note:** +The meaning of these fields might change over time. For example, we are not sure +if **Instance ID** will be single token or multiple tokens, assigned to the +**Environment**. Also, **Application name** could describe the version of +application instead of the running environment. + +### Client libraries + +GitLab currently implements a single backend that is compatible with +[Unleash](https://github.com/Unleash/unleash#client-implementations) clients. + +Unleash clients allow the developers to define in the app's code the default +values for flags. Each feature flag evaluation can express the desired +outcome in case the flag isn't present on the provided configuration file. + +Unleash currently offers a number of official SDKs for various frameworks and +a number of community contributed libraries. + +Official clients: + +- [unleash/unleash-client-java](https://github.com/unleash/unleash-client-java) +- [unleash/unleash-client-node](https://github.com/unleash/unleash-client-node) +- [unleash/unleash-client-go](https://github.com/unleash/unleash-client-go) +- [unleash/unleash-client-ruby](https://github.com/unleash/unleash-client-ruby) + +Community contributed clients: + +- [stiano/unleash-client-dotnet](https://github.com/stiano/unleash-client-dotnet) (.Net Core) +- [onybo/unleash-client-core](https://github.com/onybo/unleash-client-core) (.Net Core) +- [aes/unleash-client-python](https://github.com/aes/unleash-client-python) (Python 3) + +### Golang application example + +Here's an example of how to integrate the feature flags in a Golang application: + +```golang +package main + +import ( + "io" + "log" + "net/http" + + "github.com/Unleash/unleash-client-go" +) + +type metricsInterface struct { +} + +func init() { + unleash.Initialize( + unleash.WithUrl("https://gitlab.com/api/v4/feature_flags/unleash/42"), + unleash.WithInstanceId("29QmjsW6KngPR5JNPMWx"), + unleash.WithAppName("production"), + unleash.WithListener(&metricsInterface{}), + ) +} + +func helloServer(w http.ResponseWriter, req *http.Request) { + if unleash.IsEnabled("my_feature_name") { + io.WriteString(w, "Feature enabled\n") + } else { + io.WriteString(w, "hello, world!\n") + } +} + +func main() { + http.HandleFunc("/", helloServer) + log.Fatal(http.ListenAndServe(":8080", nil)) +} +``` diff --git a/doc/user/project/operations/img/feature_flags_list.png b/doc/user/project/operations/img/feature_flags_list.png Binary files differnew file mode 100644 index 00000000000..5313a163fec --- /dev/null +++ b/doc/user/project/operations/img/feature_flags_list.png diff --git a/doc/user/project/operations/img/specs_list.png b/doc/user/project/operations/img/specs_list.png Binary files differnew file mode 100644 index 00000000000..9630c907cfc --- /dev/null +++ b/doc/user/project/operations/img/specs_list.png diff --git a/doc/user/project/operations/index.md b/doc/user/project/operations/index.md index b0f9936be5c..0086c15c98a 100644 --- a/doc/user/project/operations/index.md +++ b/doc/user/project/operations/index.md @@ -7,5 +7,5 @@ your applications: - Deploy to different [environments](../../../ci/environments.md). - Connect your project to a [Kubernetes cluster](../clusters/index.md). - Discover and view errors generated by your applications with [Error Tracking](error_tracking.md). -- Create, toggle, and remove [Feature Flags](https://docs.gitlab.com/ee/user/project/operations/feature_flags.html). **[PREMIUM]** -- [Trace](https://docs.gitlab.com/ee/user/project/operations/tracing.html) the performance and health of a deployed application. **[ULTIMATE]** +- Create, toggle, and remove [Feature Flags](feature_flags.md). **[PREMIUM]** +- [Trace](tracing.md) the performance and health of a deployed application. **[ULTIMATE]** diff --git a/doc/user/project/operations/tracing.md b/doc/user/project/operations/tracing.md new file mode 100644 index 00000000000..0416e096cf2 --- /dev/null +++ b/doc/user/project/operations/tracing.md @@ -0,0 +1,34 @@ +# Tracing **[ULTIMATE]** + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/7903) in GitLab Ultimate 11.5. + +Tracing provides insight into the performance and health of a deployed application, +tracking each function or microservice which handles a given request. + +This makes it easy to +understand the end-to-end flow of a request, regardless of whether you are using a monolithic or distributed system. + +## Jaeger tracing + +[Jaeger](https://www.jaegertracing.io/) is an open source, end-to-end distributed +tracing system used for monitoring and troubleshooting microservices-based distributed +systems. + +### Deploying Jaeger + +To learn more about deploying Jaeger, read the official +[Getting Started documentation](https://www.jaegertracing.io/docs/latest/getting-started/). +There is an easy to use [all-in-one Docker image](https://www.jaegertracing.io/docs/latest/getting-started/#AllinoneDockerimage), +as well as deployment options for [Kubernetes](https://github.com/jaegertracing/jaeger-kubernetes) +and [OpenShift](https://github.com/jaegertracing/jaeger-openshift). + +### Enabling Jaeger + +GitLab provides an easy way to open the Jaeger UI from within your project: + +1. [Set up Jaeger](#deploying-jaeger) and configure your application using one of the + [client libraries](https://www.jaegertracing.io/docs/latest/client-libraries/). +1. Navigate to your project's **Settings > Operations** and provide the Jaeger URL. +1. Click **Save changes** for the changes to take effect. +1. You can now visit **Operations > Tracing** in your project's sidebar and + GitLab will redirect you to the configured Jaeger URL.
\ No newline at end of file diff --git a/doc/user/project/packages/img/maven_package_view.png b/doc/user/project/packages/img/maven_package_view.png Binary files differnew file mode 100644 index 00000000000..2eb4b6f76b4 --- /dev/null +++ b/doc/user/project/packages/img/maven_package_view.png diff --git a/doc/user/project/packages/img/npm_package_view.png b/doc/user/project/packages/img/npm_package_view.png Binary files differnew file mode 100644 index 00000000000..8baf7d0ef9f --- /dev/null +++ b/doc/user/project/packages/img/npm_package_view.png diff --git a/doc/user/project/packages/maven.md b/doc/user/project/packages/maven.md new file mode 100644 index 00000000000..266225fdb8d --- /dev/null +++ b/doc/user/project/packages/maven.md @@ -0,0 +1,5 @@ +--- +redirect_to: 'maven_repository.md' +--- + +This document was moved to [another location](maven_repository.md). diff --git a/doc/user/project/packages/maven_packages.md b/doc/user/project/packages/maven_packages.md new file mode 100644 index 00000000000..266225fdb8d --- /dev/null +++ b/doc/user/project/packages/maven_packages.md @@ -0,0 +1,5 @@ +--- +redirect_to: 'maven_repository.md' +--- + +This document was moved to [another location](maven_repository.md). diff --git a/doc/user/project/packages/maven_repository.md b/doc/user/project/packages/maven_repository.md new file mode 100644 index 00000000000..b62b79162a2 --- /dev/null +++ b/doc/user/project/packages/maven_repository.md @@ -0,0 +1,340 @@ +# GitLab Maven Repository **[PREMIUM]** + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/issues/5811) in + [GitLab Premium](https://about.gitlab.com/pricing/) 11.3. + +With the GitLab [Maven](https://maven.apache.org) Repository, every +project can have its own space to store its Maven artifacts. + + + +## Enabling the Maven Repository + +NOTE: **Note:** +This option is available only if your GitLab administrator has +[enabled support for the Maven repository](../../../administration/packages.md). + +After the Packages feature is enabled, the Maven Repository will be available for +all new projects by default. To enable it for existing projects, or if you want +to disable it: + +1. Navigate to your project's **Settings > General > Permissions**. +1. Find the Packages feature and enable or disable it. +1. Click on **Save changes** for the changes to take effect. + +You should then be able to see the **Packages** section on the left sidebar. +Next, you must configure your project to authorize with the GitLab Maven +repository. + +## Authenticating to the GitLab Maven Repository + +If a project is private or you want to upload Maven artifacts to GitLab, +credentials will need to be provided for authorization. Support is available for +[personal access tokens](#authenticating-with-a-personal-access-token) and +[CI job tokens](#authenticating-with-a-ci-job-token). + +### Authenticating with a personal access token + +To authenticate with a [personal access token](../../profile/personal_access_tokens.md), +add a corresponding section to your +[`settings.xml`](https://maven.apache.org/settings.html) file: + +```xml +<settings> + <servers> + <server> + <id>gitlab-maven</id> + <configuration> + <httpHeaders> + <property> + <name>Private-Token</name> + <value>REPLACE_WITH_YOUR_PERSONAL_ACCESS_TOKEN</value> + </property> + </httpHeaders> + </configuration> + </server> + </servers> +</settings> +``` + +You should now be able to upload Maven artifacts to your project. + +### Authenticating with a CI job token + +If you're using Maven with GitLab CI/CD, a CI job token can be used instead +of a personal access token. + +To authenticate with a CI job token, add a corresponding section to your +[`settings.xml`](https://maven.apache.org/settings.html) file: + +```xml +<settings> + <servers> + <server> + <id>gitlab-maven</id> + <configuration> + <httpHeaders> + <property> + <name>Job-Token</name> + <value>${env.CI_JOB_TOKEN}</value> + </property> + </httpHeaders> + </configuration> + </server> + </servers> +</settings> +``` + +You can read more on +[how to create Maven packages using GitLab CI/CD](#creating-maven-packages-with-gitlab-cicd). + +## Configuring your project to use the GitLab Maven repository URL + +To download and upload packages from GitLab, you need a `repository` and +`distributionManagement` section in your `pom.xml` file. + +Depending on your workflow and the amount of Maven packages you have, there are +3 ways you can configure your project to use the GitLab endpoint for Maven packages: + +- **Project level**: Useful when you have few Maven packages which are not under + the same GitLab group. +- **Group level**: Useful when you have many Maven packages under the same GitLab + group. +- **Instance level**: Useful when you have many Maven packages under different + GitLab groups or on their own namespace. + +NOTE: **Note:** +In all cases, you need a project specific URL for uploading a package in +the `distributionManagement` section. + +### Project level Maven endpoint + +The example below shows how the relevant `repository` section of your `pom.xml` +would look like: + +```xml +<repositories> + <repository> + <id>gitlab-maven</id> + <url>https://gitlab.com/api/v4/projects/PROJECT_ID/packages/maven</url> + </repository> +</repositories> +<distributionManagement> + <repository> + <id>gitlab-maven</id> + <url>https://gitlab.com/api/v4/projects/PROJECT_ID/packages/maven</url> + </repository> + <snapshotRepository> + <id>gitlab-maven</id> + <url>https://gitlab.com/api/v4/projects/PROJECT_ID/packages/maven</url> + </snapshotRepository> +</distributionManagement> +``` + +The `id` must be the same with what you +[defined in `settings.xml`](#authenticating-to-the-gitlab-maven-repository). + +Replace `PROJECT_ID` with your project ID which can be found on the home page +of your project. + +If you have a self-hosted GitLab installation, replace `gitlab.com` with your +domain name. + +NOTE: **Note:** +For retrieving artifacts, you can use either the +[URL encoded](../../../api/README.md#namespaced-path-encoding) path of the project +(e.g., `group%2Fproject`) or the project's ID (e.g., `42`). However, only the +project's ID can be used for uploading. + +### Group level Maven endpoint + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/8798) in GitLab Premium 11.7. + +If you rely on many packages, it might be inefficient to include the `repository` section +with a unique URL for each package. Instead, you can use the group level endpoint for +all your Maven packages stored within one GitLab group. Only packages you have access to +will be available for download. + +The group level endpoint works with any package names, which means the you +have the flexibility of naming compared to [instance level endpoint](#instance-level-maven-endpoint). +However, GitLab will not guarantee the uniqueness of the package names within +the group. You can have two projects with the same package name and package +version. As a result, GitLab will serve whichever one is more recent. + +The example below shows how the relevant `repository` section of your `pom.xml` +would look like. You still need a project specific URL for uploading a package in +the `distributionManagement` section: + +```xml +<repositories> + <repository> + <id>gitlab-maven</id> + <url>https://gitlab.com/api/v4/groups/my-group/-/packages/maven</url> + </repository> +</repositories> +<distributionManagement> + <repository> + <id>gitlab-maven</id> + <url>https://gitlab.com/api/v4/projects/PROJECT_ID/packages/maven</url> + </repository> + <snapshotRepository> + <id>gitlab-maven</id> + <url>https://gitlab.com/api/v4/projects/PROJECT_ID/packages/maven</url> + </snapshotRepository> +</distributionManagement> +``` + +The `id` must be the same with what you +[defined in `settings.xml`](#authenticating-to-the-gitlab-maven-repository). + +Replace `my-group` with your group name and `PROJECT_ID` with your project ID +which can be found on the home page of your project. + +If you have a self-hosted GitLab installation, replace `gitlab.com` with your +domain name. + +NOTE: **Note:** +For retrieving artifacts, you can use either the +[URL encoded](../../../api/README.md#namespaced-path-encoding) path of the project +(e.g., `group%2Fproject`) or the project's ID (e.g., `42`). However, only the +project's ID can be used for uploading. + +### Instance level Maven endpoint + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/8274) in GitLab Premium 11.7. + +If you rely on many packages, it might be inefficient to include the `repository` section +with a unique URL for each package. Instead, you can use the instance level endpoint for +all maven packages stored in GitLab and the packages you have access to will be available +for download. + +Note that **only packages that have the same path as the project** are exposed via +the instance level endpoint. + +| Project | Package | Instance level endpoint available | +| ------- | ------- | --------------------------------- | +| `foo/bar` | `foo/bar/1.0-SNAPSHOT` | Yes | +| `gitlab-org/gitlab-ce` | `foo/bar/1.0-SNAPSHOT` | No | +| `gitlab-org/gitlab-ce` | `gitlab-org/gitlab-ce/1.0-SNAPSHOT` | Yes | + +The example below shows how the relevant `repository` section of your `pom.xml` +would look like. You still need a project specific URL for uploading a package in +the `distributionManagement` section: + +```xml +<repositories> + <repository> + <id>gitlab-maven</id> + <url>https://gitlab.com/api/v4/packages/maven</url> + </repository> +</repositories> +<distributionManagement> + <repository> + <id>gitlab-maven</id> + <url>https://gitlab.com/api/v4/projects/PROJECT_ID/packages/maven</url> + </repository> + <snapshotRepository> + <id>gitlab-maven</id> + <url>https://gitlab.com/api/v4/projects/PROJECT_ID/packages/maven</url> + </snapshotRepository> +</distributionManagement> +``` + +The `id` must be the same with what you +[defined in `settings.xml`](#authenticating-to-the-gitlab-maven-repository). + +Replace `PROJECT_ID` with your project ID which can be found on the home page +of your project. + +If you have a self-hosted GitLab installation, replace `gitlab.com` with your +domain name. + +NOTE: **Note:** +For retrieving artifacts, you can use either the +[URL encoded](../../../api/README.md#namespaced-path-encoding) path of the project +(e.g., `group%2Fproject`) or the project's ID (e.g., `42`). However, only the +project's ID can be used for uploading. + +## Uploading packages + +Once you have set up the [authentication](#authenticating-to-the-gitlab-maven-repository) +and [configuration](#configuring-your-project-to-use-the-gitlab-maven-repository-url), +test to upload a Maven artifact from a project of yours: + +```sh +mvn deploy +``` + +You can then navigate to your project's **Packages** page and see the uploaded +artifacts or even delete them. + +## Creating Maven packages with GitLab CI/CD + +Once you have your repository configured to use the GitLab Maven Repository, +you can configure GitLab CI/CD to build new packages automatically. The example below +shows how to create a new package each time the `master` branch is updated: + +1. Create a `ci_settings.xml` file that will serve as Maven's `settings.xml` file. + Add the server section with the same id you defined in your `pom.xml` file. + For example, in our case it's `gitlab-maven`: + + ```xml + <settings xmlns="http://maven.apache.org/SETTINGS/1.1.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" + xsi:schemaLocation="http://maven.apache.org/SETTINGS/1.1.0 http://maven.apache.org/xsd/settings-1.1.0.xsd"> + <servers> + <server> + <id>gitlab-maven</id> + <configuration> + <httpHeaders> + <property> + <name>Job-Token</name> + <value>${env.CI_JOB_TOKEN}</value> + </property> + </httpHeaders> + </configuration> + </server> + </servers> + </settings> + ``` + +1. Make sure your `pom.xml` file includes the following: + + ```xml + <repositories> + <repository> + <id>gitlab-maven</id> + <url>https://gitlab.com/api/v4/projects/${env.CI_PROJECT_ID}/packages/maven</url> + </repository> + </repositories> + <distributionManagement> + <repository> + <id>gitlab-maven</id> + <url>https://gitlab.com/api/v4/projects/${env.CI_PROJECT_ID}/packages/maven</url> + </repository> + <snapshotRepository> + <id>gitlab-maven</id> + <url>https://gitlab.com/api/v4/projects/${env.CI_PROJECT_ID}/packages/maven</url> + </snapshotRepository> + </distributionManagement> + ``` + + TIP: **Tip:** + You can either let Maven utilize the CI environment variables or hardcode your project's ID. + +1. Add a `deploy` job to your `.gitlab-ci.yml` file: + + ```yaml + deploy: + image: maven:3.3.9-jdk-8 + script: + - 'mvn deploy -s ci_settings.xml' + only: + - master + ``` + +1. Push those files to your repository. + +The next time the `deploy` job runs, it will copy `ci_settings.xml` to the +user's home location (in this case the user is `root` since it runs in a +Docker container), and Maven will utilize the configured CI +[environment variables](../../../ci/variables/README.md#predefined-environment-variables). diff --git a/doc/user/project/packages/npm_registry.md b/doc/user/project/packages/npm_registry.md new file mode 100644 index 00000000000..9774e402ef7 --- /dev/null +++ b/doc/user/project/packages/npm_registry.md @@ -0,0 +1,120 @@ +# GitLab NPM Registry **[PREMIUM]** + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/issues/5934) + in [GitLab Premium](https://about.gitlab.com/pricing/) 11.7. + +With the GitLab NPM Registry, every +project can have its own space to store NPM packages. + + + +NOTE: **Note:** +Only [scoped](https://docs.npmjs.com/misc/scope) packages are supported. + + +NOTE: **Note:** +As `@group/subgroup/project` is not a valid NPM package name, publishing a package +within a subgroup is not supported yet. + +## Enabling the NPM Registry + +NOTE: **Note:** +This option is available only if your GitLab administrator has +[enabled support for the NPM registry](../../../administration/packages.md). + +After the NPM registry is enabled, it will be available for all new projects +by default. To enable it for existing projects, or if you want to disable it: + +1. Navigate to your project's **Settings > General > Permissions**. +1. Find the Packages feature and enable or disable it. +1. Click on **Save changes** for the changes to take effect. + +You should then be able to see the **Packages** section on the left sidebar. + +Before proceeding to authenticating with the GitLab NPM Registry, you should +get familiar with the package naming convention. + +## Package naming convention + +**Only packages that have the same path as the project** are supported. For + example: + +| Project | Package | Supported | +| ---------------------- | ----------------------- | --------- | +| `foo/bar` | `@foo/bar` | Yes | +| `gitlab-org/gitlab-ce` | `@gitlab-org/gitlab-ce` | Yes | +| `gitlab-org/gitlab-ce` | `@foo/bar` | No | + +Now, you can configure your project to authenticate with the GitLab NPM +Registry. + +## Authenticating to the GitLab NPM Registry + +If a project is private or you want to upload an NPM package to GitLab, +credentials will need to be provided for authentication. Support is available +only for [OAuth tokens](../../../api/oauth2.md#resource-owner-password-credentials-flow). + +CAUTION: **2FA not supported:** +Authentication for personal access tokens is not yet supported +([#9140](https://gitlab.com/gitlab-org/gitlab-ee/issues/9140)). If you have 2FA +enabled, you won't be able to authenticate to the GitLab NPM Registry. + +### Authenticating with an OAuth token + +To authenticate with an [OAuth token](../../../api/oauth2.md#resource-owner-password-credentials-flow), +add a corresponding section to your `.npmrc` file: + +```ini +; Set URL for your scoped packages. +; For example package with name `@foo/bar` will use this URL for download +@foo:registry=https://gitlab.com/api/v4/packages/npm/ + +; Add the OAuth token for the scoped packages URL. This will allow you to download +; `@foo/` packages from private projects. +//gitlab.com/api/v4/packages/npm/:_authToken=<your_oauth_token> + +; Add OAuth token for uploading to the registry. Replace <your_project_id> +; with the project you want your package to be uploaded to. +//gitlab.com/api/v4/projects/<your_project_id>/packages/npm/:_authToken=<your_oauth_token> +``` + +Replace `<your_project_id>` with your project ID which can be found on the home page +of your project and `<your_oauth_token>` with your OAuth token. + +If you have a self-hosted GitLab installation, replace `gitlab.com` with your +domain name. + +You should now be able to download and upload NPM packages to your project. + +## Uploading packages + +Before you will be able to upload a package, you need to specify the registry +for NPM. To do this, add the following section to the bottom of `package.json`: + +```json + "publishConfig": { + "@foo:registry":"https://gitlab.com/api/v4/projects/<your_project_id>/packages/npm/" + } +``` + +Replace `<your_project_id>` with your project ID, which can be found on the home +page of your project, and replace `@foo` with your own scope. + +If you have a self-hosted GitLab installation, replace `gitlab.com` with your +domain name. + +Once you have enabled it and set up [authentication](#authenticating-to-the-gitlab-npm-registry), +you can upload an NPM package to your project: + +```sh +npm publish +``` + +You can then navigate to your project's **Packages** page and see the uploaded +packages or even delete them. + +## Uploading a package with the same version twice + +If you upload a package with a same name and version twice, GitLab will show +both packages in the UI, but the GitLab NPM Registry will expose the most recent +one as it supports only one package per version for `npm install`. diff --git a/doc/user/project/pages/img/icons/click.png b/doc/user/project/pages/img/icons/click.png Binary files differindex a534ae29e0f..62a997a7591 100644 --- a/doc/user/project/pages/img/icons/click.png +++ b/doc/user/project/pages/img/icons/click.png diff --git a/doc/user/project/pages/img/icons/fork.png b/doc/user/project/pages/img/icons/fork.png Binary files differindex 8a3aa46eb37..1ae8fa722b7 100644 --- a/doc/user/project/pages/img/icons/fork.png +++ b/doc/user/project/pages/img/icons/fork.png diff --git a/doc/user/project/pages/img/icons/free.png b/doc/user/project/pages/img/icons/free.png Binary files differindex ae455033e94..c74cd90fa1a 100644 --- a/doc/user/project/pages/img/icons/free.png +++ b/doc/user/project/pages/img/icons/free.png diff --git a/doc/user/project/pages/img/icons/lock.png b/doc/user/project/pages/img/icons/lock.png Binary files differindex f4c35c84112..f7f32fded45 100644 --- a/doc/user/project/pages/img/icons/lock.png +++ b/doc/user/project/pages/img/icons/lock.png diff --git a/doc/user/project/pages/img/icons/monitor.png b/doc/user/project/pages/img/icons/monitor.png Binary files differindex 8bad059a74c..ce27b7e177e 100644 --- a/doc/user/project/pages/img/icons/monitor.png +++ b/doc/user/project/pages/img/icons/monitor.png diff --git a/doc/user/project/pages/img/icons/terminal.png b/doc/user/project/pages/img/icons/terminal.png Binary files differindex 377eeb4edc6..5a711f05c4e 100644 --- a/doc/user/project/pages/img/icons/terminal.png +++ b/doc/user/project/pages/img/icons/terminal.png diff --git a/doc/user/project/protected_branches.md b/doc/user/project/protected_branches.md index 3eb8123144f..91a85a36a53 100644 --- a/doc/user/project/protected_branches.md +++ b/doc/user/project/protected_branches.md @@ -15,12 +15,10 @@ By default, a protected branch does four simple things: - it prevents **anyone** from force pushing to the branch - it prevents **anyone** from deleting the branch -See the [Changelog](#changelog) section for changes over time. +>**Note**: +A GitLab admin is allowed to push to the protected branches. -> ->Additional functionality for GitLab Enterprise Edition: -> ->- Restrict push and merge access to [certain users][ee-restrict] +See the [Changelog](#changelog) section for changes over time. ## Configuring protected branches @@ -68,6 +66,21 @@ dropdown list in the "Already protected" area. If you don't choose any of those options while creating a protected branch, they are set to "Maintainers" by default. +## Restricting push and merge access to certain users **[STARTER]** + +> This feature was [introduced][ce-5081] in [GitLab Starter][ee] 8.11. + +With GitLab Enterprise Edition you can restrict access to protected branches +by choosing a role (Maintainers, Developers) as well as certain users. From the +dropdown menu select the role and/or the users you want to have merge or push +access. + + + +Click **Protect** and the branch will appear in the "Protected branch" list. + + + ## Wildcard protected branches > [Introduced][ce-4665] in GitLab 8.10. @@ -169,3 +182,4 @@ for details about the pipelines security model. [ce-21393]: https://gitlab.com/gitlab-org/gitlab-ce/issues/21393 [ee-restrict]: http://docs.gitlab.com/ee/user/project/protected_branches.html#restricting-push-and-merge-access-to-certain-users [perm]: ../permissions.md +[ee]: https://about.gitlab.com/pricing/ diff --git a/doc/user/project/quick_actions.md b/doc/user/project/quick_actions.md index 392e72dcc5c..4c35be39224 100644 --- a/doc/user/project/quick_actions.md +++ b/doc/user/project/quick_actions.md @@ -45,13 +45,15 @@ discussions, and descriptions: | `/remove_due_date` | Remove due date | ✓ | | | `/weight 0,1,2, ...` | Set weight **[STARTER]** | ✓ | | | `/clear_weight` | Clears weight **[STARTER]** | ✓ | | -| `/epic <group&epic | Epic URL>` | Add to epic **[ULTIMATE]** | ✓ | | +| `/epic <&epic | group&epic | Epic URL>` | Add to epic **[ULTIMATE]** | ✓ | | | `/remove_epic` | Removes from epic **[ULTIMATE]** | ✓ | | +| `/promote` | Promote issue to epic **[ULTIMATE]** | ✓ | | | `/confidential` | Make confidential | ✓ | | | `/duplicate #issue` | Mark this issue as a duplicate of another issue | ✓ | | `/move path/to/project` | Move this issue to another project | ✓ | | | `/target_branch <Local branch Name>` | Set target branch | | ✓ | | `/wip` | Toggle the Work In Progress status | | ✓ | +| `/approve` | Approve the merge request | | ✓ | | `/merge` | Merge (when pipeline succeeds) | | ✓ | | `/create_merge_request <branch name>` | Create a new merge request starting from the current issue | ✓ | | diff --git a/doc/user/project/repository/branches/img/branch_filter_search_box.png b/doc/user/project/repository/branches/img/branch_filter_search_box.png Binary files differindex 5dc7eccf189..5dc300cf24e 100644 --- a/doc/user/project/repository/branches/img/branch_filter_search_box.png +++ b/doc/user/project/repository/branches/img/branch_filter_search_box.png diff --git a/doc/user/project/repository/gpg_signed_commits/index.md b/doc/user/project/repository/gpg_signed_commits/index.md index 6495ede42e0..3260a355fdc 100644 --- a/doc/user/project/repository/gpg_signed_commits/index.md +++ b/doc/user/project/repository/gpg_signed_commits/index.md @@ -264,7 +264,7 @@ To remove a GPG key from your account: ## Rejecting commits that are not signed **[PREMIUM]** You can configure your project to reject commits that aren't GPG-signed -via [push rules](https://docs.gitlab.com/ee/push_rules/push_rules.html). +via [push rules](../../../../push_rules/push_rules.md). ## GPG signing API diff --git a/doc/user/project/repository/img/repository_cleanup.png b/doc/user/project/repository/img/repository_cleanup.png Binary files differindex bda40d3e193..e343f23ac27 100644 --- a/doc/user/project/repository/img/repository_cleanup.png +++ b/doc/user/project/repository/img/repository_cleanup.png diff --git a/doc/user/project/repository/index.md b/doc/user/project/repository/index.md index 22d912cd9d1..c6606bea10e 100644 --- a/doc/user/project/repository/index.md +++ b/doc/user/project/repository/index.md @@ -220,14 +220,9 @@ Select branches to compare using the [branch filter search box](branches/index.m Find it under your project's **Repository > Compare**. -## Locked files +## Locked files **[PREMIUM]** -> Available in [GitLab Premium](https://about.gitlab.com/pricing/). - -Lock your files to prevent any conflicting changes. - -[File Locking](https://docs.gitlab.com/ee/user/project/file_lock.html) is available only in -[GitLab Premium](https://about.gitlab.com/pricing/). +[Lock your files](../file_lock.md) to prevent any conflicting changes. ## Repository's API diff --git a/doc/user/project/security_dashboard.md b/doc/user/project/security_dashboard.md new file mode 100644 index 00000000000..b7294642150 --- /dev/null +++ b/doc/user/project/security_dashboard.md @@ -0,0 +1,35 @@ +# Project Security Dashboard **[ULTIMATE]** + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/issues/6165) in [GitLab Ultimate](https://about.gitlab.com/pricing) 11.1. + +The Security Dashboard displays the latest security reports for your project. +Use it to find and fix vulnerabilities affecting the [default branch](./repository/branches/index.md#default-branch). + + + +## How it works? + +To benefit from the Security Dashboard you must first configure the [Security Reports](merge_requests/index.md#security-reports-ultimate). + +The Security Dashboard will then list security vulnerabilities from the latest pipeline run on the default branch (e.g., `master`). + +You will also be able to interact with the reports [the same way you can on a merge request](merge_requests/index.md#interacting-with-security-reports-ultimate). + +## Keeping the Security Dashboard updated + +The Security Dashboard displays information from the results of the most recent security scan on the default branch. Security scans are performed every time the branch is updated. + +If the default branch is updated infrequently, scans are run infrequently and the information on the Security Dashboard can become outdated as new vulnerabilities are discovered. + +To ensure the information on the Security Dashboard is regularly updated, configure a [scheduled pipeline](pipelines/schedules.md) to run a daily security scan. This will update the information displayed on the Security Dashboard regardless of how often the default branch is updated. + +A daily security scan can be configured to only execute jobs that to relate to security. For more information on configuring security-related jobs, see: + +- [Static Application Security Testing](merge_requests/sast.md) and [example](../../ci/examples/sast.md). +- [Dynamic Application Security Testing](merge_requests/dast.md) and [example](../../ci/examples/dast.md). +- [Dependency Scanning](merge_requests/dependency_scanning.md) and [example](../../ci/examples/dependency_scanning.md). +- [Container Scanning](merge_requests/container_scanning.md) and [example](../../ci/examples/container_scanning.md). + +### Security scans using Auto DevOps + +When using [Auto DevOps](../../topics/autodevops/index.md), use [special environment variables](../../topics/autodevops/index.md#environment-variables) to configure daily security scans. diff --git a/doc/user/project/service_desk.md b/doc/user/project/service_desk.md new file mode 100644 index 00000000000..1a582164d03 --- /dev/null +++ b/doc/user/project/service_desk.md @@ -0,0 +1,124 @@ +# Service Desk **[PREMIUM]** + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/issues/149) in [GitLab Premium 9.1](https://about.gitlab.com/2017/04/22/gitlab-9-1-released/#service-desk-eep). + +## Overview + +Service Desk is a module that allows your team to connect directly +with any external party through email right inside of GitLab; no external tools required. +An ongoing conversation right where your software is built ensures that user feedback ends +up directly where it's needed, helping you build the right features to solve your users' +real problems. + +With Service Desk, you can provide efficient email support to your customers, who can now +email you bug reports, feature requests, or general feedback that will all end up in your +GitLab project as new issues. In turn, your team can respond straight from the project. + +As Service Desk is built right into GitLab itself, the complexity and inefficiencies +of multiple tools and external integrations are eliminated, significantly shortening +the cycle time from feedback to software update. + +For an overview, check the video demonstration on [GitLab Service Desk](https://about.gitlab.com/2017/05/09/demo-service-desk/). + +## Use cases + +For instance, let's assume you develop a game for iOS or Android. +The codebase is hosted in your GitLab instance, built and deployed +with GitLab CI. + +Here's how Service Desk will work for you: + +1. You'll provide a project-specific email address to your paying customers, who can email you directly from within the app +1. Each email they send creates an issue in the appropriate project +1. Your team members navigate to the Service Desk issue tracker, where they can see new support requests and respond inside associated issues +1. Your team communicates back and forth with the customer to understand the request +1. Your team starts working on implementing code to solve your customer's problem +1. When your team finishes the implementation, whereupon the merge request is merged and the issue is closed automatically +1. The customer will have been attended successfully via email, without having real access to your GitLab instance +1. Your team saved time by not having to leave GitLab (or setup any integrations) to follow up with your customer + +## How it works + +GitLab Service Desk is a simple way to allow people to create issues in your +GitLab instance without needing their own user account. + +It provides a unique email address for end users to create issues in a project, +and replies can be sent either through the GitLab interface or by email. End +users will only see the thread through email. + +## Configuring Service Desk + +> **Note:** +Service Desk is enabled on GitLab.com. If you're a +[Silver subscriber](https://about.gitlab.com/gitlab-com/), +you can skip the step 1 below; you only need to enable it per project. + +1. [Set up incoming email](../../administration/incoming_email.md#set-it-up) for the GitLab instance. This must + support [email sub-addressing](../../administration/incoming_email.md#email-sub-addressing). +2. Navigate to your project's **Settings** and scroll down to the **Service Desk** + section. +3. If you have the correct access and an Premium license, + you will see an option to set up Service Desk: + +  + +4. Checking that box will enable Service Desk for the project, and show a + unique email address to email issues to the project. These issues will be + [confidential](issues/confidential_issues.md), so they will only be visible to project members. + + **Warning**: this email address can be used by anyone to create an issue on + this project, whether or not they have access to your GitLab instance. + We recommend **putting this behind an alias** so that it can be changed if + needed, and **[enabling Akismet](../../integration/akismet.md)** on your GitLab instance to add spam + checking to this service. Unblocked email spam would result in many spam + issues being created, and may disrupt your GitLab service. + +  + + _In GitLab 11.7, we updated the format of the generated email address. + However the older format is still supported, allowing existing aliases + or contacts to continue working._ + + +5. Service Desk is now enabled for this project! You should be able to access it from your project's navigation **Issue submenu**: + +  + +## Using Service Desk + +### As an end user (issue creator) + +To create a Service Desk issue, an end user doesn't need to know anything about +the GitLab instance. They just send an email to the address they are given, and +receive an email back confirming receipt: + + + +This also gives the end user an option to unsubscribe. + +If they don't choose to unsubscribe, then any new comments added to the issue +will be sent as emails: + + + +And any responses they send will be displayed in the issue itself. + +### As a responder to the issue + +For responders to the issue, everything works as usual. They'll see a familiar looking +issue tracker, where they can see issues created via customer support requests and +filter and interact with them just like other GitLab issues. + + + +Messages from the end user will show as coming from the special Support Bot user, but apart from that, +you can read and write comments as you normally do: + + + +> Note that the project's visibility (private, internal, public) does not affect Service Desk. + +### Support Bot user + +Behind the scenes, Service Desk works by the special Support Bot user creating issues. This user +does not count toward the license limit count. diff --git a/doc/user/project/settings/index.md b/doc/user/project/settings/index.md index d5f4a7fd4ac..16cd1f2c14e 100644 --- a/doc/user/project/settings/index.md +++ b/doc/user/project/settings/index.md @@ -42,9 +42,9 @@ Set up your project's merge request settings:  -### Service Desk +### Service Desk **[PREMIUM]** -Enable [Service Desk](https://docs.gitlab.com/ee/user/project/service_desk.html) for your project to offer customer support. Service Desk is available in [GitLab Premium](https://about.gitlab.com/pricing/). +Enable [Service Desk](../service_desk.md) for your project to offer customer support. ### Export project @@ -128,3 +128,7 @@ namespace if needed. ### Error Tracking Configure Error Tracking to discover and view [Sentry errors within GitLab](../operations/error_tracking.md). + +### Jaeger tracing + +Add the URL of a Jaeger server to allow your users to [easily access the Jaeger UI from within GitLab](../operations/tracing.md). diff --git a/doc/user/project/web_ide/index.md b/doc/user/project/web_ide/index.md index 46a1b2bc3aa..2a2507d98a3 100644 --- a/doc/user/project/web_ide/index.md +++ b/doc/user/project/web_ide/index.md @@ -128,5 +128,123 @@ IDE. An example `package.json` is below. } ``` +## Interactive Web Terminals for the Web IDE **[ULTIMATE ONLY]** + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/issues/5426) in [GitLab Ultimate](https://about.gitlab.com/pricing/) 11.6. + +CAUTION: **Warning:** +Interactive Web Terminals for the Web IDE is currently in **Beta**. + +[Interactive web terminals](../../../ci/interactive_web_terminal/index.md) +give the user access to a terminal to interact with the Runner directly from +GitLab, including through the Web IDE. + +Only project [**maintainers**](../../permissions.md#project-members-permissions) +can run Interactive Web Terminals through the Web IDE. + +CAUTION: **Warning:** +GitLab.com [does not support Interactive Web Terminals yet](https://gitlab.com/gitlab-org/gitlab-ce/issues/52611). +Shared Runners in private instances are not supported either. + +### Runner configuration + +Some things need to be configured in the runner for the interactive web terminal +to work: + +- The Runner needs to have + [`[session_server]` configured properly](https://docs.gitlab.com/runner/configuration/advanced-configuration.html#the-session_server-section). +- If you are using a reverse proxy with your GitLab instance, web terminals need to be + [enabled](../../../administration/integration/terminal.md#enabling-and-disabling-terminal-support). **[ULTIMATE ONLY]** + +If you have the terminal open and the job has finished with its tasks, the +terminal will block the job from finishing for the duration configured in +[`[session_server].terminal_max_retention_time`](https://docs.gitlab.com/runner/configuration/advanced-configuration.html#the-session_server-section) +until you close the terminal window. + +NOTE: **Note:** Not all executors are +[supported](https://docs.gitlab.com/runner/executors/#compatibility-chart) + +### Web IDE configuration file + +In order to enable the Web IDE terminals you need to create the file +`.gitlab/.gitlab-webide.yml` inside the repository's root. This +file is fairly similar to the [CI configuration file](../../../ci/yaml/README.md) +syntax but with some restrictions: + +- No global blocks can be defined (ie: `before_script` or `after_script`) +- Only one job named `terminal` can be added to this file. +- Only the keywords `image`, `services`, `tags`, `before_script`, `script`, and +`variables` are allowed to be used to configure the job. +- To connect to the interactive terminal, the `terminal` job must be still alive +and running, otherwise the terminal won't be able to connect to the job's session. +By default the `script` keyword has the value `sleep 60` to prevent +the job from ending and giving the Web IDE enough time to connect. This means +that, if you override the default `script` value, you'll have to add a command +which would keep the job running, like `sleep`. + +In the code below there is an example of this configuration file: + +```yaml +terminal: + before_script: + - apt-get update + script: sleep 60 + variables: + RAILS_ENV: "test" + NODE_ENV: "test" +``` + +Once the terminal has started, the console will be displayed and we could access +the project repository files. + +**Important**. The terminal job is branch dependant. This means that the +configuration file used to trigger and configure the terminal will be the one in +the selected branch of the Web IDE. + +If there is no configuration file in a branch, an error message will be shown. + +### Running Interactive Terminals in the Web IDE + +If Interactive Terminals are available for the current user, the **Terminal** button +will be visible in the right sidebar of the Web IDE. Click this button to open +or close the terminal tab. + +Once open, the tab will show the **Start Web Terminal** button. This button may +be disabled if the environment is not configured correctly. If so, a status +message will describe the issue. Here are some reasons why **Start Web Terminal** +may be disabled: + +- `.gitlab/.gitlab-webide.yml` does not exist or is set up incorrectly. +- No active private runners are available for the project. + +If active, clicking the **Start Web Terminal** button will load the terminal view +and start connecting to the runner's terminal. At any time, the **Terminal** tab +can be closed and reopened and the state of the terminal will not be affected. + +When the terminal is started and is successfully connected to the runner, then the +runner's shell prompt will appear in the terminal. From here, you can enter +commands that will be executed within the runner's environment. This is similar +to running commands in a local terminal or through SSH. + +While the terminal is running, it can be stopped by clicking **Stop Terminal**. +This will disconnect the terminal and stop the runner's terminal job. From here, +click **Restart Terminal** to start a new terminal session. + +### Limitations + +Interactive Terminals is in a beta phase and will continue to be improved upon in upcoming +releases. In the meantime, please note that the user is limited to having only one +active terminal at a time. + +### Troubleshooting + +- If the terminal's text is gray and unresponsive, then the terminal has stopped + and it can no longer be used. A stopped terminal can be restarted by clicking + **Restart Terminal**. +- If the terminal displays **Connection Failure**, then the terminal could not + connect to the runner. Please try to stop and restart the terminal. If the + problem persists, double check your runner configuration. + + [ce]: https://about.gitlab.com/pricing/ [ee]: https://about.gitlab.com/pricing/ diff --git a/doc/user/search/advanced_global_search.md b/doc/user/search/advanced_global_search.md new file mode 100644 index 00000000000..f80f4183802 --- /dev/null +++ b/doc/user/search/advanced_global_search.md @@ -0,0 +1,75 @@ +# Advanced Global Search **[STARTER ONLY]** + +> - [Introduced][ee-109] in GitLab [Starter][ee] 8.4. +> - This is the user documentation. To install and configure Elasticsearch, +> visit the [administrator documentation](../../integration/elasticsearch.md). + +NOTE: **Note** +Advanced Global Search (powered by Elasticsearch) is not yet available on GitLab.com. We are working on adding it. [Follow this epic for the latest updates](https://gitlab.com/groups/gitlab-org/-/epics/153). + +Leverage Elasticsearch for faster, more advanced code search across your entire +GitLab instance. + +## Overview + +The Advanced Global Search in GitLab is a powerful search service that saves +you time. Instead of creating duplicate code and wasting time, you can +now search for code within other teams that can help your own project. + +GitLab leverages the search capabilities of [Elasticsearch] and enables it when +searching in: + +- GitLab application +- Projects +- Repositories +- Commits +- Issues +- Merge requests +- Milestones +- Notes (comments) +- Snippets +- Wiki + +## Use cases + +The Advanced Global Search can be useful in various scenarios. + +### Faster searches + +If you are dealing with huge amount of data and want to keep GitLab's search +fast, the Advanced Global Search will help you achieve that. + +### Promote innersourcing + +Your company may consist of many different developer teams each of which has +their own group where the various projects are hosted. Some of your applications +may be connected to each other, so your developers need to instantly search +throughout the GitLab instance and find the code they search for. + +## Searching globally + +Just use the search as before and GitLab will show you matching code from each +project you have access to. + + + +You can also use the [Advanced Syntax Search](advanced_search_syntax.md) which +provides some useful queries. + +>**Note:** +Elasticsearch has only data for the default branch. That means that if you go +to the repository tree and switch the branch from the default to something else, +then the "Code" tab in the search result page will be served by the regular +search even if Elasticsearch is enabled. + +[ee-1305]: https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/1305 +[aws-elastic]: http://docs.aws.amazon.com/elasticsearch-service/latest/developerguide/es-gsg.html +[aws-iam]: http://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_access-keys.html +[aws-instance-profile]: http://docs.aws.amazon.com/codedeploy/latest/userguide/getting-started-create-iam-instance-profile.html#getting-started-create-iam-instance-profile-cli +[ee-109]: https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/109 "Elasticsearch Merge Request" +[elasticsearch]: https://www.elastic.co/products/elasticsearch "Elasticsearch website" +[install]: https://www.elastic.co/guide/en/elasticsearch/reference/current/_installation.html "Elasticsearch installation documentation" +[pkg]: https://about.gitlab.com/downloads/ "Download Omnibus GitLab" +[elastic-settings]: https://www.elastic.co/guide/en/elasticsearch/reference/current/setup-configuration.html#settings "Elasticsearch configuration settings" +[ee]: https://about.gitlab.com/pricing/ +[es]: https://www.elastic.co/products/elasticsearch diff --git a/doc/user/search/advanced_search_syntax.md b/doc/user/search/advanced_search_syntax.md new file mode 100644 index 00000000000..d302cb7a809 --- /dev/null +++ b/doc/user/search/advanced_search_syntax.md @@ -0,0 +1,69 @@ +# Advanced Syntax Search **[STARTER ONLY]** + +> **Notes:** +> - Introduced in [GitLab Enterprise Starter][ee] 9.2 +> - This is the user documentation. To install and configure Elasticsearch, +> visit the [administrator documentation](../../integration/elasticsearch.md). + +NOTE: **Note** +Advanced Global Search (powered by Elasticsearch) is not yet available on GitLab.com. We are working on adding it. [Follow this epic for the latest updates](https://gitlab.com/groups/gitlab-org/-/epics/153). + +Use advanced queries for more targeted search results. + +## Overview + +The Advanced Syntax Search is a subset of the +[Advanced Global Search](advanced_global_search.md), which you can use if you +want to have more specific search results. + +## Use cases + +Let's say for example that the product you develop relies on the code of another +product that's hosted under some other group. + +Since under your GitLab instance there are hosted hundreds of different projects, +you need the search results to be as efficient as possible. You have a feeling +of what you want to find (e.g., a function name), but at the same you're also +not so sure. + +In that case, using the advanced search syntax in your query will yield much +better results. + +## Using the Advanced Syntax Search + +The Advanced Syntax Search supports fuzzy or exact search queries with prefixes, +boolean operators, and much more. + +Full details can be found in the [Elasticsearch documentation][elastic], but +here's a quick guide: + +- Searches look for all the words in a query, in any order - e.g.: searching + issues for `display bug` will return all issues matching both those words, in any order. +- To find the exact phrase (stemming still applies), use double quotes: `"display bug"` +- To find bugs not mentioning display, use `-`: `bug -display` +- To find a bug in display or sound, use `|`: `bug display | sound` +- To group terms together, use parentheses: `bug | (display +sound)` +- To match a partial word, use `*`: `bug find_by_*` +- To find a term containing one of these symbols, use `\`: `argument \-last` + +### Syntax search filters + +The Advanced Syntax Search also supports the use of filters. The available filters are: + + - filename: Filters by filename. You can use the glob (`*`) operator for fuzzy matching. + - path: Filters by path. You can use the glob (`*`) operator for fuzzy matching. + - extension: Filters by extension in the filename. Please write the extension without a leading dot. Exact match only. + +To use them, simply add them to your query in the format `<filter_name>:<value>` without + any spaces between the colon (`:`) and the value. + +Examples: + +- Finding a file with any content named `hello_world.rb`: `* filename:hello_world.rb` +- Finding a file named `hello_world` with the text `whatever` inside of it: `whatever filename:hello_world` +- Finding the text 'def create' inside files with the `.rb` extension: `def create extension:rb` +- Finding the text `sha` inside files in a folder called `encryption`: `sha path:encryption` +- Finding any file starting with `hello` containing `world` and with the `.js` extension: `world filename:hello* extension:js` + +[ee]: https://about.gitlab.com/pricing/ +[elastic]: https://www.elastic.co/guide/en/elasticsearch/reference/5.3/query-dsl-simple-query-string-query.html#_simple_query_string_syntax diff --git a/doc/user/search/img/advanced_global_search.png b/doc/user/search/img/advanced_global_search.png Binary files differnew file mode 100644 index 00000000000..4903bbb07e1 --- /dev/null +++ b/doc/user/search/img/advanced_global_search.png diff --git a/doc/user/search/img/multiple_assignees.png b/doc/user/search/img/multiple_assignees.png Binary files differnew file mode 100644 index 00000000000..5c46f3dda46 --- /dev/null +++ b/doc/user/search/img/multiple_assignees.png diff --git a/doc/user/search/index.md b/doc/user/search/index.md index 705983cce2a..9dee068bd50 100644 --- a/doc/user/search/index.md +++ b/doc/user/search/index.md @@ -84,6 +84,12 @@ You can view recent searches by clicking on the little arrow-clock icon, which i Individual filters can be removed by clicking on the filter's (x) button or backspacing. The entire search filter can be cleared by clicking on the search box's (x) button. +## Filtering with multiple filters of the same type + +Some filters can be added multiple times. These include but are not limited to assignees and labels. When you filter with these multiple filters of the same type, the AND logic is applied. For example, if you were filtering `assignee:@sam assignee:@sarah`, your results will only include entries whereby the assignees are assigned to both Sam and Sarah are returned. + + + ### Shortcut You'll also find a shortcut on the search field on the top-right of the project's dashboard to @@ -136,3 +142,18 @@ you'll be able to, besides filtering them by **Name**, **Author**, **Assignee**, and **Labels**, select multiple issues to add to a list of your choice:  + +## Advanced Global Search + +Leverage Elasticsearch for faster, more advanced code search across your entire +GitLab instance. + +[Learn how to use the Advanced Global Search.](advanced_global_search.md) + +## Advanced Syntax Search + +Use advanced queries for more targeted search results. + +[Learn how to use the Advanced Syntax Search.](advanced_search_syntax.md) + +[ee]: https://about.gitlab.com/pricing/ diff --git a/doc/workflow/README.md b/doc/workflow/README.md index 272f7807ac0..5505964a71e 100644 --- a/doc/workflow/README.md +++ b/doc/workflow/README.md @@ -13,12 +13,15 @@ comments: false - [Groups](../user/group/index.md) - Issues - The GitLab Issue Tracker is an advanced and complete tool for tracking the evolution of a new idea or the process of solving a problem. + - (EE) [Exporting Issues](../user/project/issues/csv_export.md) Export issues as a CSV, emailed as an attachment. - [Confidential issues](../user/project/issues/confidential_issues.md) - [Due date for issues](../user/project/issues/due_dates.md) - [Issue Board](../user/project/issue_board.md) - [Keyboard shortcuts](shortcuts.md) - [File finder](file_finder.md) +- (EE) [File lock](../user/project/file_lock.md) - [Labels](../user/project/labels.md) +- (EE) [Issue weight](issue_weight.md) - [Notification emails](notifications.md) - [Projects](../user/project/index.md) - [Project forking workflow](forking_workflow.md) @@ -41,6 +44,9 @@ comments: false - [Merge requests versions](../user/project/merge_requests/versions.md) - ["Work In Progress" merge requests](../user/project/merge_requests/work_in_progress_merge_requests.md) - [Fast-forward merge requests](../user/project/merge_requests/fast_forward_merge.md) + - [Merge request approvals](../user/project/merge_requests/merge_request_approvals.md) **[STARTER]** +- [Repository mirroring](repository_mirroring.md) **[STARTER]** +- [Service Desk](../user/project/service_desk.md) **[PREMIUM]** - [Manage large binaries with Git LFS](lfs/manage_large_binaries_with_git_lfs.md) - [Importing from SVN, GitHub, Bitbucket, etc](importing/README.md) - [Todos](todos.md) diff --git a/doc/workflow/ff_merge.md b/doc/workflow/ff_merge.md new file mode 100644 index 00000000000..2b747c26ca2 --- /dev/null +++ b/doc/workflow/ff_merge.md @@ -0,0 +1 @@ +This document was moved to [user/project/merge_requests/fast_forward_merge](../user/project/merge_requests/fast_forward_merge.md). diff --git a/doc/workflow/git_annex.md b/doc/workflow/git_annex.md new file mode 100644 index 00000000000..84d25951908 --- /dev/null +++ b/doc/workflow/git_annex.md @@ -0,0 +1,238 @@ +# Git annex + +> **Warning:** GitLab has [completely +removed][deprecate-annex-issue] in GitLab 9.0 (2017/03/22). +Read through the [migration guide from git-annex to git-lfs][guide]. + +The biggest limitation of Git, compared to some older centralized version +control systems, has been the maximum size of the repositories. + +The general recommendation is to not have Git repositories larger than 1GB to +preserve performance. Although GitLab has no limit (some repositories in GitLab +are over 50GB!), we subscribe to the advice to keep repositories as small as +you can. + +Not being able to version control large binaries is a big problem for many +larger organizations. +Videos, photos, audio, compiled binaries and many other types of files are too +large. As a workaround, people keep artwork-in-progress in a Dropbox folder and +only check in the final result. This results in using outdated files, not +having a complete history and increases the risk of losing work. + +This problem is solved in GitLab Enterprise Edition by integrating the +[git-annex] application. + +`git-annex` allows managing large binaries with Git without checking the +contents into Git. +You check-in only a symlink that contains the SHA-1 of the large binary. If you +need the large binary, you can sync it from the GitLab server over `rsync`, a +very fast file copying tool. + +## GitLab git-annex Configuration + +`git-annex` is disabled by default in GitLab. Below you will find the +configuration options required to enable it. + +### Requirements + +`git-annex` needs to be installed both on the server and the client side. + +For Debian-like systems (e.g., Debian, Ubuntu) this can be achieved by running: + +``` +sudo apt-get update && sudo apt-get install git-annex +``` + +For RedHat-like systems (e.g., CentOS, RHEL) this can be achieved by running: + +``` +sudo yum install epel-release && sudo yum install git-annex +``` + +### Configuration for Omnibus packages + +For omnibus-gitlab packages, only one configuration setting is needed. +The Omnibus package will internally set the correct options in all locations. + +1. In `/etc/gitlab/gitlab.rb` add the following line: + + ```ruby + gitlab_shell['git_annex_enabled'] = true + ``` + +1. Save the file and [reconfigure GitLab][] for the changes to take effect. + +### Configuration for installations from source + +There are 2 settings to enable git-annex on your GitLab server. + +One is located in `config/gitlab.yml` of the GitLab repository and the other +one is located in `config.yml` of gitlab-shell. + +1. In `config/gitlab.yml` add or edit the following lines: + + ```yaml + gitlab_shell: + git_annex_enabled: true + ``` + +1. In `config.yml` of gitlab-shell add or edit the following lines: + + ```yaml + git_annex_enabled: true + ``` + +1. Save the files and [restart GitLab][] for the changes to take effect. + +## Using GitLab git-annex + +> **Note:** +> Your Git remotes must be using the SSH protocol, not HTTP(S). + +Here is an example workflow of uploading a very large file and then checking it +into your Git repository: + +```bash +git clone git@example.com:group/project.git + +git annex init 'My Laptop' # initialize the annex project and give an optional description +cp ~/tmp/debian.iso ./ # copy a large file into the current directory +git annex add debian.iso # add the large file to git annex +git commit -am "Add Debian iso" # commit the file metadata +git annex sync --content # sync the Git repo and large file to the GitLab server +``` + +The output should look like this: + +``` +commit +On branch master +Your branch is ahead of 'origin/master' by 1 commit. + (use "git push" to publish your local commits) +nothing to commit, working tree clean +ok +pull origin +remote: Counting objects: 5, done. +remote: Compressing objects: 100% (4/4), done. +remote: Total 5 (delta 2), reused 0 (delta 0) +Unpacking objects: 100% (5/5), done. +From example.com:group/project + 497842b..5162f80 git-annex -> origin/git-annex +ok +(merging origin/git-annex into git-annex...) +(recording state in git...) +copy debian.iso (checking origin...) (to origin...) +SHA256E-s26214400--8092b3d482fb1b7a5cf28c43bc1425c8f2d380e86869c0686c49aa7b0f086ab2.iso + 26,214,400 100% 638.88kB/s 0:00:40 (xfr#1, to-chk=0/1) +ok +pull origin +ok +(recording state in git...) +push origin +Counting objects: 15, done. +Delta compression using up to 4 threads. +Compressing objects: 100% (13/13), done. +Writing objects: 100% (15/15), 1.64 KiB | 0 bytes/s, done. +Total 15 (delta 1), reused 0 (delta 0) +To example.com:group/project.git + * [new branch] git-annex -> synced/git-annex + * [new branch] master -> synced/master +ok +``` + +Your files can be found in the `master` branch, but you'll notice that there +are more branches created by the `annex sync` command. + +Git Annex will also create a new directory at `.git/annex/` and will record the +tracked files in the `.git/config` file. The files you assign to be tracked +with `git-annex` will not affect the existing `.git/config` records. The files +are turned into symbolic links that point to data in `.git/annex/objects/`. + +The `debian.iso` file in the example will contain the symbolic link: + +``` +.git/annex/objects/ZW/1k/SHA256E-s82701--6384039733b5035b559efd5a2e25a493ab6e09aabfd5162cc03f6f0ec238429d.png/SHA256E-s82701--6384039733b5035b559efd5a2e25a493ab6e09aabfd5162cc03f6f0ec238429d.iso +``` + +Use `git annex info` to retrieve the information about the local copy of your +repository. + +--- + +Downloading a single large file is also very simple: + +```bash +git clone git@gitlab.example.com:group/project.git + +git annex sync # sync Git branches but not the large file +git annex get debian.iso # download the large file +``` + +To download all files: + +```bash +git clone git@gitlab.example.com:group/project.git + +git annex sync --content # sync Git branches and download all the large files +``` + +By using `git-annex` without GitLab, anyone that can access the server can also +access the files of all projects, but GitLab Annex ensures that you can only +access files of projects you have access to (developer, maintainer, or owner role). + +## How it works + +Internally GitLab uses [GitLab Shell] to handle SSH access and this was a great +integration point for `git-annex`. +There is a setting in gitlab-shell so you can disable GitLab Annex support +if you want to. + +## Troubleshooting tips + +Differences in version of `git-annex` on the GitLab server and on local machines +can cause `git-annex` to raise unpredicted warnings and errors. + +Consult the [Annex upgrade page][annex-upgrade] for more information about +the differences between versions. You can find out which version is installed +on your server by navigating to <https://pkgs.org/download/git-annex> and +searching for your distribution. + +Although there is no general guide for `git-annex` errors, there are a few tips +on how to go around the warnings. + +### git-annex-shell: Not a git-annex or gcrypt repository. + +This warning can appear on the initial `git annex sync --content` and is caused +by differences in `git-annex-shell`. You can read more about it +[in this git-annex issue][issue]. + +One important thing to note is that despite the warning, the `sync` succeeds +and the files are pushed to the GitLab repository. + +If you get hit by this, you can run the following command inside the repository +that the warning was raised: + +``` +git config remote.origin.annex-ignore false +``` + +Consecutive runs of `git annex sync --content` **should not** produce this +warning and the output should look like this: + +``` +commit ok +pull origin +ok +pull origin +ok +push origin +``` + +[annex-upgrade]: https://git-annex.branchable.com/upgrades/ +[deprecate-annex-issue]: https://gitlab.com/gitlab-org/gitlab-ee/issues/1648 +[git-annex]: https://git-annex.branchable.com/ "git-annex website" +[gitlab shell]: https://gitlab.com/gitlab-org/gitlab-shell "GitLab Shell repository" +[guide]: lfs/migrate_from_git_annex_to_git_lfs.html +[issue]: https://git-annex.branchable.com/forum/Error_from_git-annex-shell_on_creation_of_gcrypt_special_remote/ "git-annex issue" +[reconfigure GitLab]: ../administration/restart_gitlab.md#omnibus-gitlab-reconfigure +[restart GitLab]: ../administration/restart_gitlab.md#installations-from-source diff --git a/doc/workflow/git_lfs.md b/doc/workflow/git_lfs.md new file mode 100644 index 00000000000..d7389a5c987 --- /dev/null +++ b/doc/workflow/git_lfs.md @@ -0,0 +1 @@ +This document was moved to [another location](lfs/manage_large_binaries_with_git_lfs.md). diff --git a/doc/workflow/issue_weight.md b/doc/workflow/issue_weight.md new file mode 100644 index 00000000000..267160dae2a --- /dev/null +++ b/doc/workflow/issue_weight.md @@ -0,0 +1,22 @@ +# Issue Weight **[STARTER]** + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/76) +> in [GitLab Starter](https://about.gitlab.com/pricing/) 8.3. + +When you have a lot of issues, it can be hard to get an overview. +By adding a weight to each issue, you can get a better idea of how much time, +value or complexity a given issue has or will cost. + +You can set the weight of an issue during its creation, by simply changing the +value in the dropdown menu. You can set it to a non-negative integer +value from 0, 1, 2, and so on. (The database stores a 4-byte value, so the +upper bound is essentially limitless). +You can remove weight from an issue +as well. + +This value will appear on the right sidebar of an individual issue, as well as +in the issues page next to a distinctive balance scale icon. + +As an added bonus, you can see the total sum of all issues on the milestone page. + + diff --git a/doc/workflow/issue_weight/issue.png b/doc/workflow/issue_weight/issue.png Binary files differnew file mode 100644 index 00000000000..3800b5940b8 --- /dev/null +++ b/doc/workflow/issue_weight/issue.png diff --git a/doc/workflow/lfs/images/git-annex-branches.png b/doc/workflow/lfs/images/git-annex-branches.png Binary files differnew file mode 100644 index 00000000000..3d614f68177 --- /dev/null +++ b/doc/workflow/lfs/images/git-annex-branches.png diff --git a/doc/workflow/lfs/migrate_from_git_annex_to_git_lfs.md b/doc/workflow/lfs/migrate_from_git_annex_to_git_lfs.md new file mode 100644 index 00000000000..1aeab5980a3 --- /dev/null +++ b/doc/workflow/lfs/migrate_from_git_annex_to_git_lfs.md @@ -0,0 +1,261 @@ +# Migration guide from Git Annex to Git LFS + +>**Note:** +Git Annex support [has been removed][issue-remove-annex] in GitLab Enterprise +Edition 9.0 (2017/03/22). + +Both [Git Annex][] and [Git LFS][] are tools to manage large files in Git. + +## History + +Git Annex [was introduced in GitLab Enterprise Edition 7.8][post-3], at a time +where Git LFS didn't yet exist. A few months later, GitLab brought support for +Git LFS in [GitLab 8.2][post-2] and is available for both Community and +Enterprise editions. + +## Differences between Git Annex and Git LFS + +Some items below are general differences between the two protocols and some are +ones that GitLab developed. + +- Git Annex works only through SSH, whereas Git LFS works both with SSH and HTTPS + (SSH support was added in GitLab 8.12). +- Annex files are stored in a sub-directory of the normal repositories, whereas + LFS files are stored outside of the repositories in a place you can define. +- Git Annex requires a more complex setup, but has much more options than Git + LFS. You can compare the commands each one offers by running `man git-annex` + and `man git-lfs`. +- Annex files cannot be browsed directly in GitLab's interface, whereas LFS + files can. + +## Migration steps + +>**Note:** +Since Git Annex files are stored in a sub-directory of the normal repositories +(`.git/annex/objects`) and LFS files are stored outside of the repositories, +they are not compatible as they are using a different scheme. Therefore, the +migration has to be done manually per repository. + +There are basically two steps you need to take in order to migrate from Git +Annex to Git LFS. + +### TL; DR + +If you know what you are doing and want to skip the reading, this is what you +need to do (we assume you have [git-annex enabled][annex-gitlab-use] in your +repository and that you have made backups in case something goes wrong). +Fire up a terminal, navigate to your Git repository and: + + +1. Disable `git-annex`: + + ```bash + git annex sync --content + git annex direct + git annex uninit + git annex indirect + ``` + +1. Enable `git-lfs`: + + ``` + git lfs install + git lfs track <files> + git add . + git commit -m "commit message" + git push + ``` + +### Disabling Git Annex in your repo + +Before changing anything, make sure you have a backup of your repository first. +There are a couple of ways to do that, but you can simply clone it to another +local path and maybe push it to GitLab if you want a remote backup as well. +Here you'll find a guide on +[how to back up a **git-annex** repository to an external hard drive][bkp-ext-drive]. + +Since Annex files are stored as objects with symlinks and cannot be directly +modified, we need to first remove those symlinks. + +>**Note:** +Make sure the you read about the [`direct` mode][annex-direct] as it contains +useful information that may fit in your use case. Note that `annex direct` is +deprecated in Git Annex version 6, so you may need to upgrade your repository +if the server also has Git Annex 6 installed. Read more in the +[Git Annex troubleshooting tips][annex-tips] section. + +1. Backup your repository + + ```bash + cd repository + git annex sync --content + cd .. + git clone repository repository-backup + cd repository-backup + git annex get + cd .. + ``` + +1. Use `annex direct`: + + ```bash + cd repository + git annex direct + ``` + + The output should be similar to this: + + ```bash + commit + On branch master + Your branch is up-to-date with 'origin/master'. + nothing to commit, working tree clean + ok + direct debian.iso ok + direct ok + ``` + +1. Disable Git Annex with [`annex uninit`][uninit]: + + ```bash + git annex uninit + ``` + + The output should be similar to this: + + ```bash + unannex debian.iso ok + Deleted branch git-annex (was 2534d2c). + ``` + + This will `unannex` every file in the repository, leaving the original files. + +1. Switch back to `indirect` mode: + + ```bash + git annex indirect + ``` + + The output should be similar to this: + + ```bash + (merging origin/git-annex into git-annex...) + (recording state in git...) + commit (recording state in git...) + + ok + (recording state in git...) + [master fac3194] commit before switching to indirect mode + 1 file changed, 1 deletion(-) + delete mode 120000 alpine-virt-3.4.4-x86_64.iso + ok + indirect ok + ok + ``` + +--- + +At this point, you have two options. Either add, commit and push the files +directly back to GitLab or switch to Git LFS. We will tackle the LFS switch in +the next section. + +### Enabling Git LFS in your repo + +Git LFS is enabled by default on all GitLab products (GitLab CE, GitLab EE, +GitLab.com), therefore, you don't need to do anything server-side. + +1. First, make sure you have `git-lfs` installed locally: + + ```bash + git lfs help + ``` + + If the terminal doesn't prompt you with a full response on `git-lfs` commands, + [install the Git LFS client][install-lfs] first. + +1. Inside the repo, run the following command to initiate LFS: + + ```bash + git lfs install + ``` + +1. Enable `git-lfs` for the group of files you want to track. You + can track specific files, all files containing the same extension, or an + entire directory: + + ```bash + git lfs track images/01.png # per file + git lfs track **/*.png # per extension + git lfs track images/ # per directory + ``` + + Once you do that, run `git status` and you'll see `.gitattributes` added + to your repo. It collects all file patterns that you chose to track via + `git-lfs`. + +1. Add the files, commit and push them to GitLab: + + ```bash + git add . + git commit -m "commit message" + git push + ``` + + If your remote is set up with HTTP, you will be asked to enter your login + credentials. If you have [2FA enabled][2fa], make sure to use a + [personal access token][token] instead of your password. + +## Removing the Git Annex branches + +After the migration finishes successfully, you can remove all `git-annex` +related branches from your repository. + +On GitLab, navigate to your project's **Repository ➔ Branches** and delete all +branches created by Git Annex: `git-annex`, and all under `synced/`. + + + +You can also do this on the commandline with: + + ```bash + git branch -d synced/master + git branch -d synced/git-annex + git push origin :synced/master + git push origin :synced/git-annex + git push origin :git-annex + git remote prune origin + ``` + +If there are still some Annex objects inside your repository (`.git/annex/`) +or references inside `.git/config`, run `annex uninit` again: + +```bash +git annex uninit +``` + +## Further Reading + +- (Blog Post) [Getting Started with Git FLS][post-1] +- (Blog Post) [Announcing LFS Support in GitLab][post-2] +- (Blog Post) [GitLab Annex Solves the Problem of Versioning Large Binaries with Git][post-3] +- (GitLab Docs) [Git Annex][doc-1] +- (GitLab Docs) [Git LFS][doc-2] + +[2fa]: ../../user/profile/account/two_factor_authentication.md +[token]: ../../user/profile/account/two_factor_authentication.html#personal-access-tokens +[annex-tips]: ../git_annex.html#troubleshooting-tips +[annex-direct]: https://git-annex.branchable.com/direct_mode/ +[annex-gitlab-use]: ../git_annex.md#using-gitlab-git-annex +[annex-ee]: https://docs.gitlab.com/ee/workflow/git_annex.html +[bkp-ext-drive]: https://www.thomas-krenn.com/en/wiki/Git-annex_Repository_on_an_External_Hard_Drive +[doc-1]: https://docs.gitlab.com/ee/workflow/git_annex.html +[doc-2]: https://docs.gitlab.com/ee/workflow/lfs/manage_large_binaries_with_git_lfs.html +[Git Annex]: http://git-annex.branchable.com/ +[Git LFS]: https://git-lfs.github.com/ +[install-lfs]: https://git-lfs.github.com/ +[issue-remove-annex]: https://gitlab.com/gitlab-org/gitlab-ee/issues/1648 +[lfs-track]: https://about.gitlab.com/2017/01/30/getting-started-with-git-lfs-tutorial/#tracking-files-with-lfs +[post-1]: https://about.gitlab.com/2017/01/30/getting-started-with-git-lfs-tutorial/ +[post-2]: https://about.gitlab.com/2015/11/23/announcing-git-lfs-support-in-gitlab/ +[post-3]: https://about.gitlab.com/2015/02/17/gitlab-annex-solves-the-problem-of-versioning-large-binaries-with-git/ +[uninit]: https://git-annex.branchable.com/git-annex-uninit/ diff --git a/doc/workflow/merge_request_approvals.md b/doc/workflow/merge_request_approvals.md new file mode 100644 index 00000000000..4ce1c355b0f --- /dev/null +++ b/doc/workflow/merge_request_approvals.md @@ -0,0 +1 @@ +This document was moved to [user/project/merge_requests/merge_request_approvals](../user/project/merge_requests/merge_request_approvals.md). diff --git a/doc/workflow/notifications.md b/doc/workflow/notifications.md index 6ce789998a4..924dde3bd1c 100644 --- a/doc/workflow/notifications.md +++ b/doc/workflow/notifications.md @@ -73,18 +73,18 @@ Below is the table of events users can be notified of: | Group access level changed | User | Sent when user group access level is changed | | Project moved | Project members [1] | [1] not disabled | -### Issue / Merge request events +### Issue / Epics / Merge request events In most of the below cases, the notification will be sent to: - Participants: - the author and assignee of the issue/merge request - authors of comments on the issue/merge request - - anyone mentioned by `@username` in the issue/merge request title or description - - anyone mentioned by `@username` in any of the comments on the issue/merge request + - anyone mentioned by `@username` in the issue/epic/merge request title or description + - anyone mentioned by `@username` in any of the comments on the issue/epic/merge request ...with notification level "Participating" or higher - Watchers: users with notification level "Watch" -- Subscribers: anyone who manually subscribed to the issue/merge request +- Subscribers: anyone who manually subscribed to the issue/epic/merge request - Custom: Users with notification level "custom" who turned on notifications for any of the events present in the table below | Event | Sent to | @@ -107,6 +107,9 @@ In most of the below cases, the notification will be sent to: | New comment | The above, plus anyone mentioned by `@username` in the comment, with notification level "Mention" or higher | | Failed pipeline | The author of the pipeline | | Successful pipeline | The author of the pipeline, if they have the custom notification setting for successful pipelines set | +| New epic | | +| Close epic | | +| Reopen epic | | In addition, if the title or description of an Issue or Merge Request is changed, notifications will be sent to any **new** mentions by `@username` as diff --git a/doc/workflow/rebase_before_merge.md b/doc/workflow/rebase_before_merge.md new file mode 100644 index 00000000000..9a87ee4f343 --- /dev/null +++ b/doc/workflow/rebase_before_merge.md @@ -0,0 +1 @@ +This document was moved to [another location](../user/project/merge_requests/fast_forward_merge.md). diff --git a/doc/workflow/rebase_request_widget.png b/doc/workflow/rebase_request_widget.png Binary files differnew file mode 100644 index 00000000000..cabc80d556e --- /dev/null +++ b/doc/workflow/rebase_request_widget.png diff --git a/doc/workflow/shortcuts.md b/doc/workflow/shortcuts.md index 6ed6b0bda66..6ce565323ae 100644 --- a/doc/workflow/shortcuts.md +++ b/doc/workflow/shortcuts.md @@ -85,6 +85,14 @@ You can see GitLab's keyboard shortcuts by using 'shift + ?' | <kbd>]</kbd> or <kbd>j</kbd> | Move to next file | | <kbd>[</kbd> or <kbd>k</kbd> | Move to previous file | +## Epics + +| Keyboard Shortcut | Description | +| ----------------- | ----------- | +| <kbd>r</kbd> | Reply (quoting selected text) | +| <kbd>e</kbd> | Edit description | +| <kbd>l</kbd> | Change label | + ## Wiki pages | Keyboard Shortcut | Description | diff --git a/doc/workflow/todos.md b/doc/workflow/todos.md index 830f17aa7f2..05f166ee20b 100644 --- a/doc/workflow/todos.md +++ b/doc/workflow/todos.md @@ -26,8 +26,8 @@ will still be shown in the body of the _To do_ tab. A Todo appears in your Todos dashboard when: - an issue or merge request is assigned to you, -- you are `@mentioned` in an issue or merge request, be it the description of - the issue/merge request or in a comment, +- you are `@mentioned` in an issue, merge request or epic, be it the description of + the issue/merge request/epic or in a comment, - you are `@mentioned` in a comment on a commit, - a job in the CI pipeline running for your merge request failed, but this job is not allowed to fail. @@ -63,14 +63,14 @@ for filtering; otherwise, they appear as normal. ### Manually creating a Todo -You can also add an issue or merge request to your Todos dashboard by clicking -the "Add todo" button in the issue or merge request sidebar. +You can also add an issue, merge request or epic to your Todos dashboard by clicking +the "Add todo" button in the issue, merge request or epic sidebar.  ## Marking a Todo as done -Any action to the corresponding issue or merge request will mark your Todo as +Any action to the corresponding issue, merge request or epic will mark your Todo as **Done**. Actions that dismiss Todos include: - changing the assignee @@ -84,10 +84,10 @@ Todos are personal, and they're only marked as done if the action is coming from you. If you close the issue or merge request, your Todo will automatically be marked as done. -If someone else closes, merges, or takes action on the issue or merge +If someone else closes, merges, or takes action on the issue, epic or merge request, your Todo will remain pending. This prevents other users from closing issues without you being notified. -There is just one Todo per issue or merge request, so mentioning a user a +There is just one Todo per issue, epic or merge request, so mentioning a user a hundred times in an issue will only trigger one Todo. --- @@ -97,7 +97,7 @@ corresponding **Done** button, and it will disappear from your Todo list.  -A Todo can also be marked as done from the issue or merge request sidebar using +A Todo can also be marked as done from the issue, merge request or epic sidebar using the "Mark todo as done" button.  @@ -114,7 +114,7 @@ There are four kinds of filters you can use on your Todos dashboard. | Project | Filter by project | | Group | Filter by group | | Author | Filter by the author that triggered the Todo | -| Type | Filter by issue or merge request | +| Type | Filter by issue, merge request or epic | | Action | Filter by the action that triggered the Todo | You can also filter by more than one of these at the same time. The possible Actions are `Any Action`, `Assigned`, `Mentioned`, `Added`, `Pipelines`, and `Directly Addressed`, [as described above](#what-triggers-a-todo). |
