diff options
| author | Mike Lewis <mlewis@gitlab.com> | 2019-06-07 20:13:17 +0000 |
|---|---|---|
| committer | Mike Lewis <mlewis@gitlab.com> | 2019-06-07 20:13:17 +0000 |
| commit | 99df0218f82b851b017bd0eea1b8351dc89df6ed (patch) | |
| tree | b01f884fbd1418dd5465fc1741f1620061ae8c5c /doc | |
| parent | 3eea6906747d10bea501426febaf15d2c209e06a (diff) | |
| parent | e07b2b277f79bc25cdce22ca2defba1ba80791aa (diff) | |
| download | gitlab-ce-99df0218f82b851b017bd0eea1b8351dc89df6ed.tar.gz | |
Merge branch 'master' into 'docs/fix-example-dot-net'
# Conflicts:
# doc/user/project/clusters/serverless/index.md
Diffstat (limited to 'doc')
1217 files changed, 37442 insertions, 7413 deletions
diff --git a/doc/README.md b/doc/README.md index 14e8d2edb90..3863e17c268 100644 --- a/doc/README.md +++ b/doc/README.md @@ -4,7 +4,7 @@ description: 'Learn how to use and administer GitLab, the most scalable Git-base --- <div class="display-none"> - <em>Visit <a href="https://docs.gitlab.com/ce/">docs.gitlab.com</a> for optimized + <em>Visit <a href="https://docs.gitlab.com/ee/">docs.gitlab.com</a> for optimized navigation, discoverability, and readability.</em> </div> <!-- the div above will not display on the docs site but will display on /help --> @@ -38,6 +38,7 @@ Have a look at some of our most popular documentation resources: | [GitLab CI/CD examples](ci/examples/README.md) | Get up to speed quickly with common CI/CD scenarios. | | [GitLab Container Registry](user/project/container_registry.md) | Host containers within GitLab. | | [GitLab Pages](user/project/pages/index.md) | Host static websites for your projects with GitLab. | +| [GitLab.com settings](user/gitlab_com/index.md) | Settings for [GitLab.com](#gitlabcom). | | [Kubernetes integration](user/project/clusters/index.md) | Use GitLab with Kubernetes. | | [SSH authentication](ssh/README.md) | Secure your network communications. | | [Using Docker images](ci/docker/using_docker_images.md) | Build and test your applications with Docker. | @@ -106,13 +107,18 @@ The following documentation relates to the DevOps **Plan** stage: | Plan Topics | Description | |:-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:-------------------------------------------------------------------------------------------------------------------------------------------------| +| [Burndown Charts](user/project/milestones/burndown_charts.md) **[STARTER]** | Watch your project's progress throughout a specific milestone. | | [Discussions](user/discussions/index.md) | Threads, comments, and resolvable discussions in issues, commits, and merge requests. | | [Due Dates](user/project/issues/due_dates.md) | Keep track of issue deadlines. | -| [Quick Actions](user/project/quick_actions.md) | Shortcuts for common actions on issues or merge requests, replacing the need to click buttons or use dropdowns in GitLab's UI. | +| [Epics](user/group/epics/index.md) **[ULTIMATE]** | Tracking groups of issues that share a theme. | | [Issues](user/project/issues/index.md), including [confidential issues](user/project/issues/confidential_issues.md),<br/>[issue and merge request templates](user/project/description_templates.md),<br/>and [moving issues](user/project/issues/moving_issues.md) | Project issues, restricting access to issues, create templates for submitting new issues and merge requests, and moving issues between projects. | | [Labels](user/project/labels.md) | Categorize issues or merge requests with descriptive labels. | | [Milestones](user/project/milestones/index.md) | Set milestones for delivery of issues and merge requests, with optional due date. | | [Project Issue Board](user/project/issue_board.md) | Display issues on a Scrum or Kanban board. | +| [Quick Actions](user/project/quick_actions.md) | Shortcuts for common actions on issues or merge requests, replacing the need to click buttons or use dropdowns in GitLab's UI. | +| [Related Issues](user/project/issues/related_issues.md) **[STARTER]** | Create a relationship between issues. | +| [Roadmap](user/group/roadmap/index.md) **[ULTIMATE]** | Visualize epic timelines. | +| [Service Desk](user/project/service_desk.md) **[PREMIUM]** | A simple way to allow people to create issues in your GitLab instance without needing their own user account. | | [Time Tracking](workflow/time_tracking.md) | Track time spent on issues and merge requests. | | [Todos](workflow/todos.md) | Keep track of work requiring attention with a chronological list displayed on a simple dashboard. | @@ -135,16 +141,21 @@ The following documentation relates to the DevOps **Create** stage: #### Projects and Groups -| Create Topics - Projects and Groups | Description | -|:-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:-------------------------------------------------------------------------------| -| [Create](gitlab-basics/create-project.md) and [fork](gitlab-basics/fork-project.md) projects, and<br/>[import and export<br/>projects between instances](user/project/settings/import_export.md) | Create, duplicate, and move projects. | -| [GitLab Pages](user/project/pages/index.md) | Build, test, and deploy your static website with GitLab Pages. | -| [Groups](user/group/index.md) and [Subgroups](user/group/subgroups/index.md) | Organize your projects in groups. | -| [Projects](user/project/index.md), including [project access](public_access/public_access.md)<br/>and [settings](user/project/settings/index.md) | Host source code, and control your project's visibility and set configuration. | -| [Search through GitLab](user/search/index.md) | Search for issues, merge requests, projects, groups, and todos. | -| [Snippets](user/snippets.md) | Snippets allow you to create little bits of code. | -| [Web IDE](user/project/web_ide/index.md) | Edit files within GitLab's user interface. | -| [Wikis](user/project/wiki/index.md) | Enhance your repository documentation with built-in wikis. | +| Create Topics - Projects and Groups | Description | +|:-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:-------------------------------------------------------------------------------------------------| +| [Advanced global search](user/search/advanced_global_search.md) **[STARTER]** | Leverage Elasticsearch for faster, more advanced code search across your entire GitLab instance. | +| [Advanced syntax search](user/search/advanced_search_syntax.md) **[STARTER]** | Use advanced queries for more targeted search results. | +| [Contribution analytics](user/group/contribution_analytics/index.md) **[STARTER]** | See detailed statistics of group contributors. | +| [Create](gitlab-basics/create-project.md) and [fork](gitlab-basics/fork-project.md) projects, and<br/>[import and export projects<br/>between instances](user/project/settings/import_export.md) | Create, duplicate, and move projects. | +| [File locking](user/project/file_lock.md) **[PREMIUM]** | Lock files to avoid merge conflicts. | +| [GitLab Pages](user/project/pages/index.md) | Build, test, and deploy your static website with GitLab Pages. | +| [Groups](user/group/index.md) and [Subgroups](user/group/subgroups/index.md) | Organize your projects in groups. | +| [Issues Analytics](user/group/issues_analytics/index.md) **[PREMIUM]** | Check how many issues were created per month. | +| [Projects](user/project/index.md), including [project access](public_access/public_access.md)<br/>and [settings](user/project/settings/index.md) | Host source code, and control your project's visibility and set configuration. | +| [Search through GitLab](user/search/index.md) | Search for issues, merge requests, projects, groups, and todos. | +| [Snippets](user/snippets.md) | Snippets allow you to create little bits of code. | +| [Web IDE](user/project/web_ide/index.md) | Edit files within GitLab's user interface. | +| [Wikis](user/project/wiki/index.md) | Enhance your repository documentation with built-in wikis. | <div align="right"> <a type="button" class="btn btn-default" href="#overview"> @@ -164,7 +175,9 @@ The following documentation relates to the DevOps **Create** stage: | [Files](user/project/repository/index.md#files) | Files management. | | [Jupyter Notebook files](user/project/repository/index.md#jupyter-notebook-files) | GitLab's support for `.ipynb` files. | | [Protected branches](user/project/protected_branches.md) | Use protected branches. | +| [Push rules](push_rules/push_rules.md) **[STARTER]** | Additional control over pushes to your projects. | | [Repositories](user/project/repository/index.md) | Manage source code repositories in GitLab's user interface. | +| [Repository mirroring](workflow/repository_mirroring.md) **[STARTER]** | Push to or pull from repositories outside of GitLab | | [Start a merge request](user/project/repository/web_editor.md#tips) | Start merge request when committing via GitLab's user interface. | <div align="right"> @@ -191,13 +204,14 @@ The following documentation relates to the DevOps **Create** stage: #### Integration and Automation -| Create Topics - Integration and Automation | Description | -|:------------------------------------------------------------------|:-----------------------------------------------------------------------------------------------------------------------| -| [GitLab API](api/README.md) | Integrate GitLab via a simple and powerful API. | -| [GitLab Integration](integration/README.md) | Integrate with multiple third-party services with GitLab to allow external issue trackers and external authentication. | -| [GitLab Webhooks](user/project/integrations/webhooks.md) | Let GitLab notify you when new code has been pushed to your project. | -| [Project Services](user/project/integrations/project_services.md) | Integrate a project with external services, such as CI and chat. | -| [Trello Power-Up](integration/trello_power_up.md) | Integrate with GitLab's Trello Power-Up. | +| Create Topics - Integration and Automation | Description | +|:------------------------------------------------------------------------------|:-----------------------------------------------------------------------------------------------------------------------| +| [GitLab API](api/README.md) | Integrate GitLab via a simple and powerful API. | +| [GitLab Integration](integration/README.md) | Integrate with multiple third-party services with GitLab to allow external issue trackers and external authentication. | +| [GitLab Webhooks](user/project/integrations/webhooks.md) | Let GitLab notify you when new code has been pushed to your project. | +| [JIRA Development Panel](integration/jira_development_panel.md) **[PREMIUM]** | See GitLab information in the JIRA Development Panel. | +| [Project Services](user/project/integrations/project_services.md) | Integrate a project with external services, such as CI and chat. | +| [Trello Power-Up](integration/trello_power_up.md) | Integrate with GitLab's Trello Power-Up. | <div align="right"> <a type="button" class="btn btn-default" href="#overview"> @@ -217,12 +231,14 @@ scales to run your tests faster. The following documentation relates to the DevOps **Verify** stage: -| Verify Topics | Description | -|:---------------------------------------------------|:-----------------------------------------------------------------------------| -| [GitLab CI/CD](ci/README.md) | Explore the features and capabilities of Continuous Integration with GitLab. | -| [JUnit test reports](ci/junit_test_reports.md) | Display JUnit test reports on merge requests. | -| [Pipeline Graphs](ci/pipelines.md#pipeline-graphs) | Visualize builds. | -| [Review Apps](ci/review_apps/index.md) | Preview changes to your application right from a merge request. | +| Verify Topics | Description | +|:----------------------------------------------------------------------------------|:--------------------------------------------------------------------------------------------------------| +| [Code Quality reports](user/project/merge_requests/code_quality.md) **[STARTER]** | Analyze source code quality. | +| [GitLab CI/CD](ci/README.md) | Explore the features and capabilities of Continuous Integration with GitLab. | +| [JUnit test reports](ci/junit_test_reports.md) | Display JUnit test reports on merge requests. | +| [Multi-project pipelines](ci/multi_project_pipelines.md) **[PREMIUM]** | Visualize entire pipelines that span multiple projects, including all cross-project inter-dependencies. | +| [Pipeline Graphs](ci/pipelines.md#visualizing-pipelines) | Visualize builds. | +| [Review Apps](ci/review_apps/index.md) | Preview changes to your application right from a merge request. | <div align="right"> <a type="button" class="btn btn-default" href="#overview"> @@ -241,6 +257,7 @@ The following documentation relates to the DevOps **Package** stage: | Package Topics | Description | |:----------------------------------------------------------------|:-------------------------------------------------------| | [GitLab Container Registry](user/project/container_registry.md) | Learn how to use GitLab's built-in Container Registry. | +| [GitLab Packages](administration/packages.md) **[PREMIUM]** | Use GitLab as an NPM registry or Maven repository. | <div align="right"> <a type="button" class="btn btn-default" href="#overview"> @@ -256,14 +273,17 @@ confidently and securely with GitLab’s built-in Continuous Delivery and Deploy The following documentation relates to the DevOps **Release** stage: -| Release Topics | Description | -|:------------------------------------------------------------|:---------------------------------------------------------------------------------------------| -| [Auto Deploy](topics/autodevops/index.md#auto-deploy) | Configure GitLab for the deployment of your application. | -| [Environments and deployments](ci/environments.md) | With environments, you can control the continuous deployment of your software within GitLab. | -| [GitLab CI/CD](ci/README.md) | Explore the features and capabilities of Continuous Deployment and Delivery with GitLab. | -| [GitLab Pages](user/project/pages/index.md) | Build, test, and deploy a static site directly from GitLab. | -| [Protected Runners](ci/runners/README.md#protected-runners) | Select Runners to only pick jobs for protected branches and tags. | -| [Scheduled Pipelines](user/project/pipelines/schedules.md) | Execute pipelines on a schedule. | +| Release Topics | Description | +|:------------------------------------------------------------------------------------------------------------------------------------|:--------------------------------------------------------------------------------------------------------------------------------------| +| [Auto Deploy](topics/autodevops/index.md#auto-deploy) | Configure GitLab for the deployment of your application. | +| [Canary Deployments](user/project/canary_deployments.md) **[PREMIUM]** | Employ a popular CI strategy where a small portion of the fleet is updated to the new version first. | +| [Deploy Boards](user/project/deploy_boards.md) **[PREMIUM]** | View the current health and status of each CI environment running on Kubernetes, displaying the status of the pods in the deployment. | +| [Environments and deployments](ci/environments.md) | With environments, you can control the continuous deployment of your software within GitLab. | +| [Environment-specific variables](ci/variables/README.md#limiting-environment-scopes-of-environment-variables-premium) **[PREMIUM]** | Limit scope of variables to specific environments. | +| [GitLab CI/CD](ci/README.md) | Explore the features and capabilities of Continuous Deployment and Delivery with GitLab. | +| [GitLab Pages](user/project/pages/index.md) | Build, test, and deploy a static site directly from GitLab. | +| [Protected Runners](ci/runners/README.md#protected-runners) | Select Runners to only pick jobs for protected branches and tags. | +| [Scheduled Pipelines](user/project/pipelines/schedules.md) | Execute pipelines on a schedule. | <div align="right"> <a type="button" class="btn btn-default" href="#overview"> @@ -287,7 +307,9 @@ The following documentation relates to the DevOps **Configure** stage: | [GitLab ChatOps](ci/chatops/README.md) | Interact with CI/CD jobs through chat services. | | [Installing Applications](user/project/clusters/index.md#installing-applications) | Deploy Helm, Ingress, and Prometheus on Kubernetes. | | [Mattermost slash commands](user/project/integrations/mattermost_slash_commands.md) | Enable and use slash commands from within Mattermost. | -| [Protected variables](ci/variables/README.md#protected-variables) | Restrict variables to protected branches and tags. | +| [Multiple Kubernetes Clusters](user/project/clusters/index.md#multiple-kubernetes-clusters-premium) **[PREMIUM]** | Associate more than one Kubernetes clusters to your project. | +| [Protected variables](ci/variables/README.md#protected-environment-variables) | Restrict variables to protected branches and tags. | +| [Serverless](user/project/clusters/serverless/index.md) | Run serverless workloads on Kubernetes. | | [Slack slash commands](user/project/integrations/slack_slash_commands.md) | Enable and use slash commands from within Slack. | <div align="right"> @@ -321,16 +343,23 @@ The following documentation relates to the DevOps **Monitor** stage: ### Secure -GitLab can help you secure your applications from within your development lifecycle. +Check your application for security vulnerabilities that may lead to unauthorized access, +data leaks, and denial of services. GitLab will perform static and dynamic tests on the +code of your application, looking for known flaws and report them in the merge request +so you can fix them before merging. Security teams can use dashboards to get a +high-level view on projects and groups, and start remediation processes when needed. The following documentation relates to the DevOps **Secure** stage: -| Monitor Topics | Description | -|:----------------------------------------------------------------|:-----------------------------------------------------------------------------------------------------------| -| [Container Scanning example](ci/examples/container_scanning.md) | `.gitlab-ci.yml` example of using Clair and clair-scanner to scan docker images for known vulnerabilities. | - -NOTE: **Note:** -Viewing [Container Scanning reports](https://docs.gitlab.com/ee/user/project/merge_requests/container_scanning.html) within merge requests requires [GitLab Ultimate](https://about.gitlab.com/pricing/). +| Secure Topics | Description | +|:------------------------------------------------------------------------------------------------------|:-----------------------------------------------------------------------| +| [Container Scanning](user/application_security/container_scanning/index.md) **[ULTIMATE]** | Use Clair to scan docker images for known vulnerabilities. | +| [Dependency Scanning](user/application_security/dependency_scanning/index.md) **[ULTIMATE]** | Analyze your dependencies for known vulnerabilities. | +| [Dynamic Application Security Testing (DAST)](user/application_security/dast/index.md) **[ULTIMATE]** | Analyze running web applications for known vulnerabilities. | +| [Group Security Dashboard](user/application_security/security_dashboard/index.md) **[ULTIMATE]** | View vulnerabilities in all the projects in a group and its subgroups. | +| [License Management](user/application_security/license_management/index.md) **[ULTIMATE]** | Search your project's dependencies for their licenses. | +| [Project Security Dashboard](user/application_security/security_dashboard/index.md) **[ULTIMATE]** | View the latest security reports for your project. | +| [Static Application Security Testing (SAST)](user/application_security/sast/index.md) **[ULTIMATE]** | Analyze source code for known vulnerabilities. | ## Subscribe to GitLab @@ -340,6 +369,8 @@ There are two ways to use GitLab: - [GitLab.com](#gitlabcom): GitLab's SaaS offering. You don't need to install anything to use GitLab.com, you only need to [sign up](https://gitlab.com/users/sign_in) and start using GitLab straight away. +For more information on managing your subscription and [Customers Portal](https://customers.gitlab.com) account, please see [Getting Started with Subscriptions](getting-started/subscription.md). + The following sections outline tiers and features within GitLab self-managed and GitLab.com. <div align="right"> @@ -391,6 +422,12 @@ GitLab.com subscriptions grant access to the same features available in GitLab self-managed, **except [administration](administration/index.md) tools and settings**. +GitLab.com allows you to apply your subscription to a group or your personal user. + +When applied to a **group**, the group, all subgroups, and all projects under the selected group on GitLab.com will have the features of the associated plan. It is recommended to go with a group plan when managing projects and users of an organization. + +When associated with a **personal userspace** instead, all projects will have features with the subscription applied, but as it is not a group, group features will not be available. + TIP: **Tip:** To support the open source community and encourage the development of open source projects, GitLab grants access to **Gold** features for all GitLab.com **public** projects, regardless of the subscription. @@ -416,7 +453,7 @@ We have the following documentation to rapidly uplift your GitLab knowledge: | Topic | Description | |:-----------------------------------------------------------------------------------------------------------------------|:---------------------------------------------------------------| -| [GitLab Basics](gitlab-basics/README.md) | Start working on the command line and with GitLab. | +| [GitLab basics guides](gitlab-basics/README.md) | Start working on the command line and with GitLab. | | [GitLab Workflow](workflow/README.md) and [overview](https://about.gitlab.com/2016/10/25/gitlab-workflow-an-overview/) | Enhance your workflow with the best of GitLab Workflow. | | [Get started with GitLab CI/CD](ci/quick_start/README.md) | Quickly implement GitLab CI/CD. | | [Auto DevOps](topics/autodevops/index.md) | Learn more about GitLab's Auto DevOps. | diff --git a/doc/administration/audit_events.md b/doc/administration/audit_events.md new file mode 100644 index 00000000000..d7a2e13b53e --- /dev/null +++ b/doc/administration/audit_events.md @@ -0,0 +1,116 @@ +--- +last_updated: 2019-02-04 +--- + +# Audit Events **[STARTER]** + +GitLab offers a way to view the changes made within the GitLab server for owners and administrators on a [paid plan][ee]. + +GitLab system administrators can also take advantage of the logs located on the +filesystem, see [the logs system documentation](logs.md) for more details. + +## Overview + +**Audit Events** is a tool for GitLab owners and administrators to be +able to track important events such as who performed certain actions and the +time they happened. These actions could be, for example, change a user +permission level, who added a new user, or who removed a user. + +## Use-cases + +- Check who was the person who changed the permission level of a particular + user for a project in GitLab. +- Use it to track which users have access to a certain group of projects + in GitLab, and who gave them that permission level. + +## List of events + +There are two kinds of events logged: + +- Events scoped to the group or project, used by group / project managers + to look up who made what change. +- Instance events scoped to the whole GitLab instance, used by your Compliance team to + perform formal audits. + +### Group events **[STARTER]** + +NOTE: **Note:** +You need Owner [permissions] to view the group Audit Events page. + +To view a group's audit events, navigate to **Group > Settings > Audit Events**. +From there, you can see the following actions: + +- Group name/path changed +- Group repository size limit changed +- Group created/deleted +- Group changed visibility +- User was added to group and with which [permissions] +- Permissions changes of a user assigned to a group +- Removed user from group +- Project added to group and with which visibility level +- Project removed from group +- [Project shared with group](../user/project/members/share_project_with_groups.md) + and with which [permissions] +- Removal of a previously shared group with a project +- LFS enabled/disabled +- Shared runners minutes limit changed +- Membership lock enabled/disabled +- Request access enabled/disabled +- 2FA enforcement/grace period changed +- Roles allowed to create project changed + +### Project events **[STARTER]** + +NOTE: **Note:** +You need Maintainer [permissions] or higher to view the project Audit Events page. + +To view a project's audit events, navigate to **Project > Settings > Audit Events**. +From there, you can see the following actions: + +- Added/removed deploy keys +- Project created/deleted/renamed/moved(transferred)/changed path +- Project changed visibility level +- User was added to project and with which [permissions] +- Permission changes of a user assigned to a project +- User was removed from project + +### Instance events **[PREMIUM ONLY]** + +> [Introduced][ee-2336] in [GitLab Premium][ee] 9.3. + +Server-wide audit logging introduces the ability to observe user actions across +the entire instance of your GitLab server, making it easy to understand who +changed what and when for audit purposes. + +To view the server-wide admin log, visit **Admin Area > Monitoring > Audit Log**. + +In addition to the group and project events, the following user actions are also +recorded: + +- Failed Logins +- Sign-in events and the authentication type (standard, LDAP, OmniAuth, etc.) +- Added SSH key +- Added/removed email +- Changed password +- Ask for password reset +- Grant OAuth access + +It is possible to filter particular actions by choosing an audit data type from +the filter drop-down. You can further filter by specific group, project or user +(for authentication events). + + + +### Missing events + +Some events are not being tracked in Audit Events. Please see the following +epics for more detail on which events are not being tracked and our progress +on adding these events into GitLab: + +- [Project settings and activity](https://gitlab.com/groups/gitlab-org/-/epics/474) +- [Group settings and activity](https://gitlab.com/groups/gitlab-org/-/epics/475) +- [Instance-level settings and activity](https://gitlab.com/groups/gitlab-org/-/epics/476) + +[ee-2336]: https://gitlab.com/gitlab-org/gitlab-ee/issues/2336 +[ee]: https://about.gitlab.com/pricing/ +[permissions]: ../user/permissions.md diff --git a/doc/administration/audit_log.png b/doc/administration/audit_log.png Binary files differnew file mode 100644 index 00000000000..d4f4c2abf38 --- /dev/null +++ b/doc/administration/audit_log.png diff --git a/doc/administration/auditor_access_form.png b/doc/administration/auditor_access_form.png Binary files differnew file mode 100644 index 00000000000..c179a7d3b0a --- /dev/null +++ b/doc/administration/auditor_access_form.png diff --git a/doc/administration/auditor_users.md b/doc/administration/auditor_users.md new file mode 100644 index 00000000000..ef8c8197d6d --- /dev/null +++ b/doc/administration/auditor_users.md @@ -0,0 +1,88 @@ +# Auditor users **[PREMIUM ONLY]** + +>[Introduced][ee-998] in [GitLab Premium][eep] 8.17. + +Auditor users are given read-only access to all projects, groups, and other +resources on the GitLab instance. + +## Overview + +Auditor users can have full access to their own resources (projects, groups, +snippets, etc.), and read-only access to **all** other resources, except the +Admin area. To put another way, they are just regular users (who can be added +to projects, create personal snippets, create milestones on their groups, etc.) +who also happen to have read-only access to all projects on the system that +they haven't been explicitly [given access][permissions] to. + +The Auditor role is _not_ a read-only version of the Admin role. Auditor users +will not be able to access the project/group settings pages, or the Admin Area. + +To sum up, assuming you have logged-in as an Auditor user: + +- For a project the Auditor is not member of, the Auditor should have + read-only access. If the project is public or internal, they would have the + same access as the users that are not members of that project/group. +- For a project the Auditor owns, the Auditor should have full access to + everything. +- For a project the Auditor has been added to as a member, the Auditor should + have the same access as the [permissions] they were given to. For example, if + they were added as a Developer, they could then push commits or comment on + issues. +- The Auditor cannot view the Admin area, or perform any admin actions. + +For more information about what an Auditor can or can't do, see the +[Permissions and restrictions of an Auditor user](#permissions-and-restrictions-of-an-auditor-user) +section. + +## Use cases + +1. Your compliance department wants to run tests against the entire GitLab base + to ensure users are complying with password, credit card, and other sensitive + data policies. With Auditor users, this can be achieved very easily without + resulting to tactics like giving a user admin rights or having to use the API + to add them to all projects. +1. If particular users need visibility or access to most of all projects in + your GitLab instance, instead of manually adding the user to all projects, + you can simply create an Auditor user and share the credentials with those + that you want to grant access to. + +## Adding an Auditor user + +1. Create a new user or edit an existing one by navigating to + **Admin Area > Users**. You will find the option of the access level under + the 'Access' section. + +  + +1. Click **Save changes** or **Create user** for the changes to take effect. + +To revoke the Auditor permissions from a user, simply make them a Regular user +following the same steps as above. + +## Permissions and restrictions of an Auditor user + +An Auditor user should be able to access all projects and groups of a GitLab +instance, with the following permissions/restrictions: + +- Has read-only access to the API +- Can access projects that are: + - Private + - Public + - Internal +- Can read all files in a repository +- Can read issues / MRs +- Can read project snippets +- Cannot be Admin and Auditor at the same time +- Cannot access the Admin area +- In a group / project they're not a member of: + - Cannot access project settings + - Cannot access group settings + - Cannot commit to repository + - Cannot create / comment on issues / MRs + - Cannot create/modify files from the Web UI + - Cannot merge a merge request + - Cannot create project snippets + +[ee-998]: https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/998 +[eep]: https://about.gitlab.com/pricing/ +[permissions]: ../user/permissions.md diff --git a/doc/administration/auth/README.md b/doc/administration/auth/README.md index 54be7b616cc..e215a0df6ec 100644 --- a/doc/administration/auth/README.md +++ b/doc/administration/auth/README.md @@ -9,9 +9,11 @@ providers. - [LDAP](ldap.md) Includes Active Directory, Apple Open Directory, Open LDAP, and 389 Server + - [LDAP for GitLab EE](ldap-ee.md): LDAP additions to GitLab Enterprise Editions **[STARTER ONLY]** - [OmniAuth](../../integration/omniauth.md) Sign in via Twitter, GitHub, GitLab.com, Google, Bitbucket, Facebook, Shibboleth, Crowd, Azure, Authentiq ID, and JWT - [CAS](../../integration/cas.md) Configure GitLab to sign in using CAS - [SAML](../../integration/saml.md) Configure GitLab as a SAML 2.0 Service Provider - [Okta](okta.md) Configure GitLab to sign in using Okta - [Authentiq](authentiq.md): Enable the Authentiq OmniAuth provider for passwordless authentication +- [Smartcard](smartcard.md) Smartcard authentication **[PREMIUM ONLY]** diff --git a/doc/administration/auth/google_secure_ldap.md b/doc/administration/auth/google_secure_ldap.md new file mode 100644 index 00000000000..760af0cfd1a --- /dev/null +++ b/doc/administration/auth/google_secure_ldap.md @@ -0,0 +1,207 @@ +# Google Secure LDAP **[CORE ONLY]** + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/issues/46391) in GitLab 11.9. + +[Google Cloud Identity](https://cloud.google.com/identity/) provides a Secure +LDAP service that can be configured with GitLab for authentication and group sync. + +Secure LDAP requires a slightly different configuration than standard LDAP servers. +The steps below cover: + +- Configuring the Secure LDAP Client in the Google Admin console. +- Required GitLab configuration. + +## Configuring Google LDAP client + +1. Navigate to https://admin.google.com and sign in as a GSuite domain administrator. + +1. Go to **Apps > LDAP > Add Client**. + +1. Provide an `LDAP client name` and an optional `Description`. Any descriptive + values are acceptable. For example, the name could be 'GitLab' and the + description could be 'GitLab LDAP Client'. Click the **Continue** button. + +  + +1. Set **Access Permission** according to your needs. You must choose either + 'Entire domain (GitLab)' or 'Selected organizational units' for both 'Verify user + credentials' and 'Read user information'. Select 'Add LDAP Client' + + TIP: **Tip:** If you plan to use GitLab [LDAP Group Sync](ldap-ee.md#group-sync) + , turn on 'Read group information'. + +  + +1. Download the generated certificate. This is required for GitLab to + communicate with the Google Secure LDAP service. Save the downloaded certificates + for later use. After downloading, click the **Continue to Client Details** button. + +1. Expand the **Service Status** section and turn the LDAP client 'ON for everyone'. + After selecting 'Save', click on the 'Service Status' bar again to collapse + and return to the rest of the settings. + +1. Expand the **Authentication** section and choose 'Generate New Credentials'. + Copy/note these credentials for later use. After selecting 'Close', click + on the 'Authentication' bar again to collapse and return to the rest of the settings. + +Now the Google Secure LDAP Client configuration is finished. The screenshot below +shows an example of the final settings. Continue on to configure GitLab. + + + +## Configuring GitLab + +Edit GitLab configuration, inserting the access credentials and certificate +obtained earlier. + +The following are the configuration keys that need to be modified using the +values obtained during the LDAP client configuration earlier: + +- `bind_dn`: The access credentials username +- `password`: The access credentials password +- `cert`: The `.crt` file text from the downloaded certificate bundle +- `key`: The `.key` file text from the downloaded certificate bundle + +**For Omnibus installations** + +1. Edit `/etc/gitlab/gitlab.rb`: + + ```ruby + gitlab_rails['ldap_enabled'] = true + gitlab_rails['ldap_servers'] = YAML.load <<-EOS # remember to close this block with 'EOS' below + main: # 'main' is the GitLab 'provider ID' of this LDAP server + label: 'Google Secure LDAP' + + host: 'ldap.google.com' + port: 636 + uid: 'uid' + bind_dn: 'DizzyHorse' + password: 'd6V5H8nhMUW9AuDP25abXeLd' + encryption: 'simple_tls' + verify_certificates: true + + tls_options: + cert: | + -----BEGIN CERTIFICATE----- + MIIDbDCCAlSgAwIBAgIGAWlzxiIfMA0GCSqGSIb3DQEBCwUAMHcxFDASBgNVBAoTC0dvb2dsZSBJ + bmMuMRYwFAYDVQQHEw1Nb3VudGFpbiBWaWV3MRQwEgYDVQQDEwtMREFQIENsaWVudDEPMA0GA1UE + CxMGR1N1aXRlMQswCQYDVQQGEwJVUzETMBEGA1UECBMKQ2FsaWZvcm5pYTAeFw0xOTAzMTIyMTE5 + MThaFw0yMjAzMTEyMTE5MThaMHcxFDASBgNVBAoTC0dvb2dsZSBJbmMuMRYwFAYDVQQHEw1Nb3Vu + dGFpbiBWaWV3MRQwEgYDVQQDEwtMREFQIENsaWVudDEPMA0GA1UECxMGR1N1aXRlMQswCQYDVQQG + EwJVUzETMBEGA1UECBMKQ2FsaWZvcm5pYTCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEB + ALOTy4aC38dyjESk6N8fRsKk8DN23ZX/GaNFL5OUmmA1KWzrvVC881OzNdtGm3vNOIxr9clteEG/ + tQwsmsJvQT5U+GkBt+tGKF/zm7zueHUYqTP7Pg5pxAnAei90qkIRFi17ulObyRHPYv1BbCt8pxNB + 4fG/gAXkFbCNxwh1eiQXXRTfruasCZ4/mHfX7MVm8JmWU9uAVIOLW+DSWOFhrDQduJdGBXJOyC2r + Gqoeg9+tkBmNH/jjxpnEkFW8q7io9DdOUqqNgoidA1h9vpKTs3084sy2DOgUvKN9uXWx14uxIyYU + Y1DnDy0wczcsuRt7l+EgtCEgpsLiLJQbKW+JS1UCAwEAATANBgkqhkiG9w0BAQsFAAOCAQEAf60J + yazhbHkDKIH2gFxfm7QLhhnqsmafvl4WP7JqZt0u0KdnvbDPfokdkM87yfbKJU1MTI86M36wEC+1 + P6bzklKz7kXbzAD4GggksAzxsEE64OWHC+Y64Tkxq2NiZTw/76POkcg9StiIXjG0ZcebHub9+Ux/ + rTncip92nDuvgEM7lbPFKRIS/YMhLCk09B/U0F6XLsf1yYjyf5miUTDikPkov23b/YGfpc8kh6hq + 1kqdi6a1cYPP34eAhtRhMqcZU9qezpJF6s9EeN/3YFfKzLODFSsVToBRAdZgGHzj//SAtLyQTD4n + KCSvK1UmaMxNaZyTHg8JnMf0ZuRpv26iSg== + -----END CERTIFICATE----- + + key: | + -----BEGIN PRIVATE KEY----- + MIIEvQIBADANBgkqhkiG9w0BAQEFAASCBKcwggSjAgEAAoIBAQCzk8uGgt/HcoxEpOjfH0bCpPAz + dt2V/xmjRS+TlJpgNSls671QvPNTszXbRpt7zTiMa/XJbXhBv7UMLJrCb0E+VPhpAbfrRihf85u8 + 7nh1GKkz+z4OacQJwHovdKpCERYte7pTm8kRz2L9QWwrfKcTQeHxv4AF5BWwjccIdXokF10U367m + rAmeP5h31+zFZvCZllPbgFSDi1vg0ljhYaw0HbiXRgVyTsgtqxqqHoPfrZAZjR/448aZxJBVvKu4 + qPQ3TlKqjYKInQNYfb6Sk7N9POLMtgzoFLyjfbl1sdeLsSMmFGNQ5w8tMHM3LLkbe5fhILQhIKbC + 4iyUGylviUtVAgMBAAECggEAIPb0CQy0RJoX+q/lGbRVmnyJpYDf+115WNnl+mrwjdGkeZyqw4v0 + BPzkWYzUFP1esJRO6buBNFybQRFdFW0z5lvVv/zzRKq71aVUBPInxaMRyHuJ8D5lIL8nDtgVOwyE + 7DOGyDtURUMzMjdUwoTe7K+O6QBU4X/1pVPZYgmissYSMmt68LiP8k0p601F4+r5xOi/QEy44aVp + aOJZBUOisKB8BmUXZqmQ4Cy05vU9Xi1rLyzkn9s7fxnZ+JO6Sd1r0Thm1mE0yuPgxkDBh/b4f3/2 + GsQNKKKCiij/6TfkjnBi8ZvWR44LnKpu760g/K7psVNrKwqJG6C/8RAcgISWQQKBgQDop7BaKGhK + 1QMJJ/vnlyYFTucfGLn6bM//pzTys5Gop0tpcfX/Hf6a6Dd+zBhmC3tBmhr80XOX/PiyAIbc0lOI + 31rafZuD/oVx5mlIySWX35EqS14LXmdVs/5vOhsInNgNiE+EPFf1L9YZgG/zA7OUBmqtTeYIPDVC + 7ViJcydItQKBgQDFmK0H0IA6W4opGQo+zQKhefooqZ+RDk9IIZMPOAtnvOM7y3rSVrfsSjzYVuMS + w/RP/vs7rwhaZejnCZ8/7uIqwg4sdUBRzZYR3PRNFeheW+BPZvb+2keRCGzOs7xkbF1mu54qtYTa + HZGZj1OsD83AoMwVLcdLDgO1kw32dkS8IQKBgFRdgoifAHqqVah7VFB9se7Y1tyi5cXWsXI+Wufr + j9U9nQ4GojK52LqpnH4hWnOelDqMvF6TQTyLIk/B+yWWK26Ft/dk9wDdSdystd8L+dLh4k0Y+Whb + +lLMq2YABw+PeJUnqdYE38xsZVHoDjBsVjFGRmbDybeQxauYT7PACy3FAoGBAK2+k9bdNQMbXp7I + j8OszHVkJdz/WXlY1cmdDAxDwXOUGVKIlxTAf7TbiijILZ5gg0Cb+hj+zR9/oI0WXtr+mAv02jWp + W8cSOLS4TnBBpTLjIpdu+BwbnvYeLF6MmEjNKEufCXKQbaLEgTQ/XNlchBSuzwSIXkbWqdhM1+gx + EjtBAoGARAdMIiDMPWIIZg3nNnFebbmtBP0qiBsYohQZ+6i/8s/vautEHBEN6Q0brIU/goo+nTHc + t9VaOkzjCmAJSLPUanuBC8pdYgLu5J20NXUZLD9AE/2bBT3OpezKcdYeI2jqoc1qlWHlNtVtdqQ2 + AcZSFJQjdg5BTyvdEDhaYUKGdRw= + -----END PRIVATE KEY----- + EOS + ``` + +1. Save the file and [reconfigure] GitLab for the changes to take effect. + +--- + +**For installations from source** + +1. Edit `config/gitlab.yml`: + + ```yaml + ldap: + enabled: true + servers: + main: # 'main' is the GitLab 'provider ID' of this LDAP server + label: 'Google Secure LDAP' + + host: 'ldap.google.com' + port: 636 + uid: 'uid' + bind_dn: 'DizzyHorse' + password: 'd6V5H8nhMUW9AuDP25abXeLd' + encryption: 'simple_tls' + verify_certificates: true + + tls_options: + cert: | + -----BEGIN CERTIFICATE----- + MIIDbDCCAlSgAwIBAgIGAWlzxiIfMA0GCSqGSIb3DQEBCwUAMHcxFDASBgNVBAoTC0dvb2dsZSBJ + bmMuMRYwFAYDVQQHEw1Nb3VudGFpbiBWaWV3MRQwEgYDVQQDEwtMREFQIENsaWVudDEPMA0GA1UE + CxMGR1N1aXRlMQswCQYDVQQGEwJVUzETMBEGA1UECBMKQ2FsaWZvcm5pYTAeFw0xOTAzMTIyMTE5 + MThaFw0yMjAzMTEyMTE5MThaMHcxFDASBgNVBAoTC0dvb2dsZSBJbmMuMRYwFAYDVQQHEw1Nb3Vu + dGFpbiBWaWV3MRQwEgYDVQQDEwtMREFQIENsaWVudDEPMA0GA1UECxMGR1N1aXRlMQswCQYDVQQG + EwJVUzETMBEGA1UECBMKQ2FsaWZvcm5pYTCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEB + ALOTy4aC38dyjESk6N8fRsKk8DN23ZX/GaNFL5OUmmA1KWzrvVC881OzNdtGm3vNOIxr9clteEG/ + tQwsmsJvQT5U+GkBt+tGKF/zm7zueHUYqTP7Pg5pxAnAei90qkIRFi17ulObyRHPYv1BbCt8pxNB + 4fG/gAXkFbCNxwh1eiQXXRTfruasCZ4/mHfX7MVm8JmWU9uAVIOLW+DSWOFhrDQduJdGBXJOyC2r + Gqoeg9+tkBmNH/jjxpnEkFW8q7io9DdOUqqNgoidA1h9vpKTs3084sy2DOgUvKN9uXWx14uxIyYU + Y1DnDy0wczcsuRt7l+EgtCEgpsLiLJQbKW+JS1UCAwEAATANBgkqhkiG9w0BAQsFAAOCAQEAf60J + yazhbHkDKIH2gFxfm7QLhhnqsmafvl4WP7JqZt0u0KdnvbDPfokdkM87yfbKJU1MTI86M36wEC+1 + P6bzklKz7kXbzAD4GggksAzxsEE64OWHC+Y64Tkxq2NiZTw/76POkcg9StiIXjG0ZcebHub9+Ux/ + rTncip92nDuvgEM7lbPFKRIS/YMhLCk09B/U0F6XLsf1yYjyf5miUTDikPkov23b/YGfpc8kh6hq + 1kqdi6a1cYPP34eAhtRhMqcZU9qezpJF6s9EeN/3YFfKzLODFSsVToBRAdZgGHzj//SAtLyQTD4n + KCSvK1UmaMxNaZyTHg8JnMf0ZuRpv26iSg== + -----END CERTIFICATE----- + + key: | + -----BEGIN PRIVATE KEY----- + MIIEvQIBADANBgkqhkiG9w0BAQEFAASCBKcwggSjAgEAAoIBAQCzk8uGgt/HcoxEpOjfH0bCpPAz + dt2V/xmjRS+TlJpgNSls671QvPNTszXbRpt7zTiMa/XJbXhBv7UMLJrCb0E+VPhpAbfrRihf85u8 + 7nh1GKkz+z4OacQJwHovdKpCERYte7pTm8kRz2L9QWwrfKcTQeHxv4AF5BWwjccIdXokF10U367m + rAmeP5h31+zFZvCZllPbgFSDi1vg0ljhYaw0HbiXRgVyTsgtqxqqHoPfrZAZjR/448aZxJBVvKu4 + qPQ3TlKqjYKInQNYfb6Sk7N9POLMtgzoFLyjfbl1sdeLsSMmFGNQ5w8tMHM3LLkbe5fhILQhIKbC + 4iyUGylviUtVAgMBAAECggEAIPb0CQy0RJoX+q/lGbRVmnyJpYDf+115WNnl+mrwjdGkeZyqw4v0 + BPzkWYzUFP1esJRO6buBNFybQRFdFW0z5lvVv/zzRKq71aVUBPInxaMRyHuJ8D5lIL8nDtgVOwyE + 7DOGyDtURUMzMjdUwoTe7K+O6QBU4X/1pVPZYgmissYSMmt68LiP8k0p601F4+r5xOi/QEy44aVp + aOJZBUOisKB8BmUXZqmQ4Cy05vU9Xi1rLyzkn9s7fxnZ+JO6Sd1r0Thm1mE0yuPgxkDBh/b4f3/2 + GsQNKKKCiij/6TfkjnBi8ZvWR44LnKpu760g/K7psVNrKwqJG6C/8RAcgISWQQKBgQDop7BaKGhK + 1QMJJ/vnlyYFTucfGLn6bM//pzTys5Gop0tpcfX/Hf6a6Dd+zBhmC3tBmhr80XOX/PiyAIbc0lOI + 31rafZuD/oVx5mlIySWX35EqS14LXmdVs/5vOhsInNgNiE+EPFf1L9YZgG/zA7OUBmqtTeYIPDVC + 7ViJcydItQKBgQDFmK0H0IA6W4opGQo+zQKhefooqZ+RDk9IIZMPOAtnvOM7y3rSVrfsSjzYVuMS + w/RP/vs7rwhaZejnCZ8/7uIqwg4sdUBRzZYR3PRNFeheW+BPZvb+2keRCGzOs7xkbF1mu54qtYTa + HZGZj1OsD83AoMwVLcdLDgO1kw32dkS8IQKBgFRdgoifAHqqVah7VFB9se7Y1tyi5cXWsXI+Wufr + j9U9nQ4GojK52LqpnH4hWnOelDqMvF6TQTyLIk/B+yWWK26Ft/dk9wDdSdystd8L+dLh4k0Y+Whb + +lLMq2YABw+PeJUnqdYE38xsZVHoDjBsVjFGRmbDybeQxauYT7PACy3FAoGBAK2+k9bdNQMbXp7I + j8OszHVkJdz/WXlY1cmdDAxDwXOUGVKIlxTAf7TbiijILZ5gg0Cb+hj+zR9/oI0WXtr+mAv02jWp + W8cSOLS4TnBBpTLjIpdu+BwbnvYeLF6MmEjNKEufCXKQbaLEgTQ/XNlchBSuzwSIXkbWqdhM1+gx + EjtBAoGARAdMIiDMPWIIZg3nNnFebbmtBP0qiBsYohQZ+6i/8s/vautEHBEN6Q0brIU/goo+nTHc + t9VaOkzjCmAJSLPUanuBC8pdYgLu5J20NXUZLD9AE/2bBT3OpezKcdYeI2jqoc1qlWHlNtVtdqQ2 + AcZSFJQjdg5BTyvdEDhaYUKGdRw= + -----END PRIVATE KEY----- + ``` + +1. Save the file and [restart] GitLab for the changes to take effect. + + +[reconfigure]: ../restart_gitlab.md#omnibus-gitlab-reconfigure +[restart]: ../restart_gitlab.md#installations-from-source diff --git a/doc/administration/auth/how_to_configure_ldap_gitlab_ce/index.md b/doc/administration/auth/how_to_configure_ldap_gitlab_ce/index.md index 0d03b481881..1f67e8f5744 100644 --- a/doc/administration/auth/how_to_configure_ldap_gitlab_ce/index.md +++ b/doc/administration/auth/how_to_configure_ldap_gitlab_ce/index.md @@ -14,7 +14,7 @@ Managing a large number of users in GitLab can become a burden for system admini In this guide we will focus on configuring GitLab with Active Directory. [Active Directory](https://en.wikipedia.org/wiki/Active_Directory) is a popular LDAP compatible directory service provided by Microsoft, included in all modern Windows Server operating systems. -GitLab has supported LDAP integration since [version 2.2](https://about.gitlab.com/2012/02/22/gitlab-version-2-2/). With GitLab LDAP [group syncing](#group-syncing-ee) being added to GitLab Enterprise Edition in [version 6.0](https://about.gitlab.com/2013/08/20/gitlab-6-dot-0-released/). LDAP integration has become one of the most popular features in GitLab. +GitLab has supported LDAP integration since [version 2.2](https://about.gitlab.com/2012/02/22/gitlab-version-2-2/). With GitLab LDAP [group syncing](../how_to_configure_ldap_gitlab_ee/index.html#group-sync) being added to GitLab Enterprise Edition in [version 6.0](https://about.gitlab.com/2013/08/20/gitlab-6-dot-0-released/). LDAP integration has become one of the most popular features in GitLab. ## Getting started @@ -111,7 +111,7 @@ The initial configuration of LDAP in GitLab requires changes to the `gitlab.rb` The two Active Directory specific values are `active_directory: true` and `uid: 'sAMAccountName'`. `sAMAccountName` is an attribute returned by Active Directory used for GitLab usernames. See the example output from `ldapsearch` for a full list of attributes a "person" object (user) has in **AD** - [`ldapsearch` example](#using-ldapsearch-unix) -> Both group_base and admin_group configuration options are only available in GitLab Enterprise Edition. See [GitLab EE - LDAP Features](#gitlab-enterprise-edition---ldap-features) +> Both group_base and admin_group configuration options are only available in GitLab Enterprise Edition. See [GitLab EE - LDAP Features](../how_to_configure_ldap_gitlab_ee/index.html#gitlab-enterprise-edition---ldap-features) **[STARTER ONLY]** ### Example `gitlab.rb` LDAP @@ -145,7 +145,7 @@ Securing LDAP (enabling LDAPS) on Windows Server 2012 involves installing a vali > By default a LDAP service listens for connections on TCP and UDP port 389. LDAPS (LDAP over SSL) listens on port 636 -### Testing you AD server +### Testing your AD server #### Using **AdFind** (Windows) @@ -267,4 +267,4 @@ have extended functionalities with LDAP, such as: - Updating user permissions - Multiple LDAP servers -Read through the article on [LDAP for GitLab EE](https://docs.gitlab.com/ee/administration/auth/how_to_configure_ldap_gitlab_ee/) for an overview. +Read through the article on [LDAP for GitLab EE](../how_to_configure_ldap_gitlab_ee/index.md) **[STARTER ONLY]** for an overview. diff --git a/doc/administration/auth/how_to_configure_ldap_gitlab_ee/img/admin_group.png b/doc/administration/auth/how_to_configure_ldap_gitlab_ee/img/admin_group.png Binary files differnew file mode 100644 index 00000000000..9896379d669 --- /dev/null +++ b/doc/administration/auth/how_to_configure_ldap_gitlab_ee/img/admin_group.png diff --git a/doc/administration/auth/how_to_configure_ldap_gitlab_ee/img/group_link_final.png b/doc/administration/auth/how_to_configure_ldap_gitlab_ee/img/group_link_final.png Binary files differnew file mode 100644 index 00000000000..21fb5a7d0ce --- /dev/null +++ b/doc/administration/auth/how_to_configure_ldap_gitlab_ee/img/group_link_final.png diff --git a/doc/administration/auth/how_to_configure_ldap_gitlab_ee/img/group_linking.gif b/doc/administration/auth/how_to_configure_ldap_gitlab_ee/img/group_linking.gif Binary files differnew file mode 100644 index 00000000000..d35cf55804f --- /dev/null +++ b/doc/administration/auth/how_to_configure_ldap_gitlab_ee/img/group_linking.gif diff --git a/doc/administration/auth/how_to_configure_ldap_gitlab_ee/img/manual_permissions.gif b/doc/administration/auth/how_to_configure_ldap_gitlab_ee/img/manual_permissions.gif Binary files differnew file mode 100644 index 00000000000..29b28df1cbd --- /dev/null +++ b/doc/administration/auth/how_to_configure_ldap_gitlab_ee/img/manual_permissions.gif diff --git a/doc/administration/auth/how_to_configure_ldap_gitlab_ee/img/multi_login.gif b/doc/administration/auth/how_to_configure_ldap_gitlab_ee/img/multi_login.gif Binary files differnew file mode 100644 index 00000000000..d317add9837 --- /dev/null +++ b/doc/administration/auth/how_to_configure_ldap_gitlab_ee/img/multi_login.gif diff --git a/doc/administration/auth/how_to_configure_ldap_gitlab_ee/index.md b/doc/administration/auth/how_to_configure_ldap_gitlab_ee/index.md new file mode 100644 index 00000000000..4d82a7370bb --- /dev/null +++ b/doc/administration/auth/how_to_configure_ldap_gitlab_ee/index.md @@ -0,0 +1,119 @@ +--- +author: Chris Wilson +author_gitlab: MrChrisW +level: intermediary +article_type: admin guide +date: 2017-05-03 +--- + +# How to configure LDAP with GitLab EE **[STARTER ONLY]** + +## Introduction + +The present article follows [How to Configure LDAP with GitLab CE](../how_to_configure_ldap_gitlab_ce/index.md). Make sure to read through it before moving forward. + +## GitLab Enterprise Edition - LDAP features + +[GitLab Enterprise Edition (EE)](https://about.gitlab.com/pricing/) has a number of advantages when it comes to integrating with Active Directory (LDAP): + +- [Administrator Sync](../ldap-ee.md#administrator-sync): As an extension of group sync, you can automatically manage your global GitLab administrators. Specify a group CN for `admin_group` and all members of the LDAP group will be given administrator privileges. +- [Group Sync](#group-sync): This allows GitLab group membership to be automatically updated based on LDAP group members. +- [Multiple LDAP servers](#multiple-ldap-servers): The ability to configure multiple LDAP servers. This is useful if an organization has different LDAP servers within departments. This is not designed for failover. We're working on [supporting LDAP failover](https://gitlab.com/gitlab-org/gitlab-ee/issues/139) in GitLab. + +- Daily user synchronization: Once a day, GitLab will run a synchronization to check and update GitLab users against LDAP. This process updates all user details automatically. + +On the following section, you'll find a description for each of these features. Read through [LDAP GitLab EE docs](../ldap-ee.md) for complementary information. + + + +All members of the group `Global Admins` will be given **administrator** access to GitLab, allowing them to view the `/admin` dashboard. + +### Group Sync + +Group syncing allows AD (LDAP) groups to be mapped to GitLab groups. This provides more control over per-group user management. To configure group syncing edit the `group_base` **DN** (`'OU=Global Groups,OU=GitLab INT,DC=GitLab,DC=org'`). This **OU** contains all groups that will be associated with [GitLab groups](../../../user/group/index.md). + +#### Creating group links - example + +As an example, let's suppose we have a "UKGov" GitLab group, which deals with confidential government information. Therefore, it is important that users of this group are given the correct permissions to projects contained within the group. Granular group permissions can be applied based on the AD group. + +**UK Developers** of our "UKGov" group are given **"developer"** permissions. + +_The developer permission allows the development staff to effectively manage all project code, issues, and merge requests._ + +**UK Support** staff of our "UKGov" group are given **"reporter"** permissions. + +_The reporter permission allows support staff to manage issues, labels, and review project code._ + +**US People Ops** of our "UKGov" group are given **"guest"** permissions. + + + +> Guest permissions allows people ops staff to review and lodge new issues while allowing no read or write access to project code or [confidential issues](../../../user/project/issues/confidential_issues.md#permissions-and-access-to-confidential-issues) created by other users. + +See the [permission list](../../../user/permissions.md) for complementary info. + +#### Group permissions - example + +Considering the previous example, our staff will have +access to our GitLab instance with the following structure: + + + +Using this permission structure in our example allows only UK staff access to sensitive information stored in the projects code, while still allowing other teams to work effectively. As all permissions are controlled via AD groups new users can be quickly added to existing groups. New group members will then automatically inherit the required permissions. + +> [More information](../ldap-ee.md#group-sync) on group syncing. + +### Updating user permissions - new feature + +Since GitLab [v8.15](https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/822) LDAP user permissions can now be manually overridden by an admin user. To override a user's permissions visit the groups **Members** page and select **Edit permissions**. + + + +### Multiple LDAP servers + +GitLab EE can support multiple LDAP servers. Simply configure another server in the `gitlab.rb` file within the `ldap_servers` block. In the example below we configure a new secondary server with the label **GitLab Secondary AD**. This is shown on the GitLab login screen. Large enterprises often utilize multiple LDAP servers for segregating organizational departments. + + + +Considering the example illustrated on the image above, +our `gitlab.rb` configuration would look like: + +```ruby +gitlab_rails['ldap_enabled'] = true +gitlab_rails['ldap_servers'] = { +'main' => { + 'label' => 'GitLab AD', + 'host' => 'ad.example.org', + 'port' => 636, + 'uid' => 'sAMAccountName', + 'method' => 'ssl', + 'bind_dn' => 'CN=GitLabSRV,CN=Users,DC=GitLab,DC=org', + 'password' => 'Password1', + 'active_directory' => true, + 'base' => 'OU=GitLab INT,DC=GitLab,DC=org', + 'group_base' => 'OU=Global Groups,OU=GitLab INT,DC=GitLab,DC=org', + 'admin_group' => 'Global Admins' + }, + +'secondary' => { + 'label' => 'GitLab Secondary AD', + 'host' => 'ad-secondary.example.net', + 'port' => 636, + 'uid' => 'sAMAccountName', + 'method' => 'ssl', + 'bind_dn' => 'CN=GitLabSRV,CN=Users,DC=GitLab,DC=com', + 'password' => 'Password1', + 'active_directory' => true, + 'base' => 'OU=GitLab Secondary,DC=GitLab,DC=com', + 'group_base' => 'OU=Global Groups,OU=GitLab INT,DC=GitLab,DC=com', + 'admin_group' => 'Global Admins' + } +} +``` + +## Conclusion + +Integration of GitLab with Active Directory (LDAP) reduces the complexity of user management. +It has the advantage of improving user permission controls, whilst easing the deployment of GitLab into an existing [IT environment](https://www.techopedia.com/definition/29199/it-infrastructure). GitLab EE offers advanced group management and multiple LDAP servers. + +With the assistance of the [GitLab Support](https://about.gitlab.com/support) team, setting up GitLab with an existing AD/LDAP solution will be a smooth and painless process. diff --git a/doc/administration/auth/img/google_secure_ldap_add_step_1.png b/doc/administration/auth/img/google_secure_ldap_add_step_1.png Binary files differnew file mode 100644 index 00000000000..fd254443d75 --- /dev/null +++ b/doc/administration/auth/img/google_secure_ldap_add_step_1.png diff --git a/doc/administration/auth/img/google_secure_ldap_add_step_2.png b/doc/administration/auth/img/google_secure_ldap_add_step_2.png Binary files differnew file mode 100644 index 00000000000..611a21ae03c --- /dev/null +++ b/doc/administration/auth/img/google_secure_ldap_add_step_2.png diff --git a/doc/administration/auth/img/google_secure_ldap_client_settings.png b/doc/administration/auth/img/google_secure_ldap_client_settings.png Binary files differnew file mode 100644 index 00000000000..3c0b3f3d4bd --- /dev/null +++ b/doc/administration/auth/img/google_secure_ldap_client_settings.png diff --git a/doc/administration/auth/ldap-ee.md b/doc/administration/auth/ldap-ee.md new file mode 100644 index 00000000000..30095d35705 --- /dev/null +++ b/doc/administration/auth/ldap-ee.md @@ -0,0 +1,557 @@ +# LDAP Additions in GitLab EE **[STARTER ONLY]** + +This is a continuation of the main [LDAP documentation](ldap.md), detailing LDAP +features specific to GitLab Enterprise Edition Starter, Premium and Ultimate. + +## Overview + +[LDAP](https://en.wikipedia.org/wiki/Lightweight_Directory_Access_Protocol) +stands for **Lightweight Directory Access Protocol**, which +is a standard application protocol for +accessing and maintaining distributed directory information services +over an Internet Protocol (IP) network. + +GitLab integrates with LDAP to support **user authentication**. This integration +works with most LDAP-compliant directory servers, including Microsoft Active +Directory, Apple Open Directory, Open LDAP, and 389 Server. +**GitLab Enterprise Edition** includes enhanced integration, including group +membership syncing. + +## Use cases + +- User sync: Once a day, GitLab will update users against LDAP +- Group sync: Once an hour, GitLab will update group membership + based on LDAP group members + +## Multiple LDAP servers + +With GitLab Enterprise Edition Starter, you can configure multiple LDAP servers +that your GitLab instance will connect to. + +To add another LDAP server, you can start by duplicating the settings under +[the main configuration](ldap.md#configuration) and edit them to match the +additional LDAP server. + +Be sure to choose a different provider ID made of letters a-z and numbers 0-9. +This ID will be stored in the database so that GitLab can remember which LDAP +server a user belongs to. + +## User sync + +Once per day, GitLab will run a worker to check and update GitLab +users against LDAP. + +The process will execute the following access checks: + +1. Ensure the user is still present in LDAP +1. If the LDAP server is Active Directory, ensure the user is active (not + blocked/disabled state). This will only be checked if + `active_directory: true` is set in the LDAP configuration [^1] + +The user will be set to `ldap_blocked` state in GitLab if the above conditions +fail. This means the user will not be able to login or push/pull code. + +The process will also update the following user information: + +1. Email address +1. If `sync_ssh_keys` is set, SSH public keys +1. If Kerberos is enabled, Kerberos identity + +NOTE: **Note:** +The LDAP sync process updates existing users while new users will +be created on first sign in. + +## Group Sync + +If your LDAP supports the `memberof` property, when the user signs in for the +first time GitLab will trigger a sync for groups the user should be a member of. +That way they don't need to wait for the hourly sync to be granted +access to their groups and projects. + +In GitLab Premium, we can also add a GitLab group to sync with one or multiple LDAP groups or we can +also add a filter. The filter must comply with the syntax defined in [RFC 2254](https://tools.ietf.org/search/rfc2254). + +A group sync process will run every hour on the hour, and `group_base` must be set +in LDAP configuration for LDAP synchronizations based on group CN to work. This allows +GitLab group membership to be automatically updated based on LDAP group members. + +The `group_base` configuration should be a base LDAP 'container', such as an +'organization' or 'organizational unit', that contains LDAP groups that should +be available to GitLab. For example, `group_base` could be +`ou=groups,dc=example,dc=com`. In the config file it will look like the +following. + +**Omnibus configuration** + +1. Edit `/etc/gitlab/gitlab.rb`: + + ```ruby + gitlab_rails['ldap_servers'] = YAML.load <<-EOS + main: + ## snip... + ## + ## Base where we can search for groups + ## + ## Ex. ou=groups,dc=gitlab,dc=example + ## + ## + group_base: ou=groups,dc=example,dc=com + EOS + ``` + +1. [Reconfigure GitLab][reconfigure] for the changes to take effect. + +**Source configuration** + +1. Edit `/home/git/gitlab/config/gitlab.yml`: + + ```yaml + production: + ldap: + servers: + main: + # snip... + group_base: ou=groups,dc=example,dc=com + ``` + +1. [Restart GitLab][restart] for the changes to take effect. + +--- + +To take advantage of group sync, group owners or maintainers will need to create an +LDAP group link in their group **Settings > LDAP Groups** page. Multiple LDAP +groups and/or filters can be linked with a single GitLab group. When the link is +created, an access level/role is specified (Guest, Reporter, Developer, Maintainer, +or Owner). + +## Administrator sync + +As an extension of group sync, you can automatically manage your global GitLab +administrators. Specify a group CN for `admin_group` and all members of the +LDAP group will be given administrator privileges. The configuration will look +like the following. + +NOTE: **Note:** +Administrators will not be synced unless `group_base` is also +specified alongside `admin_group`. Also, only specify the CN of the admin +group, as opposed to the full DN. + +**Omnibus configuration** + +1. Edit `/etc/gitlab/gitlab.rb`: + + ```ruby + gitlab_rails['ldap_servers'] = YAML.load <<-EOS + main: + ## snip... + ## + ## Base where we can search for groups + ## + ## Ex. ou=groups,dc=gitlab,dc=example + ## + ## + group_base: ou=groups,dc=example,dc=com + + ## + ## The CN of a group containing GitLab administrators + ## + ## Ex. administrators + ## + ## Note: Not `cn=administrators` or the full DN + ## + ## + admin_group: my_admin_group + + EOS + ``` + +1. [Reconfigure GitLab][reconfigure] for the changes to take effect. + +**Source configuration** + +1. Edit `/home/git/gitlab/config/gitlab.yml`: + + ```yaml + production: + ldap: + servers: + main: + # snip... + group_base: ou=groups,dc=example,dc=com + admin_group: my_admin_group + ``` + +1. [Restart GitLab][restart] for the changes to take effect. + +## Adjusting LDAP user sync schedule + +> Introduced in GitLab Enterprise Edition Starter. + +NOTE: **Note:** +These are cron formatted values. You can use a crontab generator to create +these values, for example http://www.crontabgenerator.com/. + +By default, GitLab will run a worker once per day at 01:30 a.m. server time to +check and update GitLab users against LDAP. + +You can manually configure LDAP user sync times by setting the +following configuration values. The example below shows how to set LDAP user +sync to run once every 12 hours at the top of the hour. + +**Omnibus installations** + +1. Edit `/etc/gitlab/gitlab.rb`: + + ```ruby + gitlab_rails['ldap_sync_worker_cron'] = "0 */12 * * *" + ``` + +1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. + +**Source installations** + +1. Edit `config/gitlab.yaml`: + + ```yaml + cron_jobs: + ldap_sync_worker_cron: + "0 */12 * * *" + ``` + +1. [Restart GitLab](../restart_gitlab.md#installations-from-source) for the changes to take effect. + +## Adjusting LDAP group sync schedule + +NOTE: **Note:** +These are cron formatted values. You can use a crontab generator to create +these values, for example http://www.crontabgenerator.com/. + +By default, GitLab will run a group sync process every hour, on the hour. + +CAUTION: **Important:** +It's recommended that you do not run too short intervals as this +could lead to multiple syncs running concurrently. This is primarily a concern +for installations with a large number of LDAP users. Please review the +[LDAP group sync benchmark metrics](#benchmarks) to see how +your installation compares before proceeding. + +You can manually configure LDAP group sync times by setting the +following configuration values. The example below shows how to set group +sync to run once every 2 hours at the top of the hour. + +**Omnibus installations** + +1. Edit `/etc/gitlab/gitlab.rb`: + + ```ruby + gitlab_rails['ldap_group_sync_worker_cron'] = "0 */2 * * * *" + ``` + +1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. + +**Source installations** + +1. Edit `config/gitlab.yaml`: + + ```yaml + cron_jobs: + ldap_group_sync_worker_cron: + "*/30 * * * *" + ``` + +1. [Restart GitLab](../restart_gitlab.md#installations-from-source) for the changes to take effect. + +## External groups + +> Introduced in GitLab Enterprise Edition Starter 8.9. + +Using the `external_groups` setting will allow you to mark all users belonging +to these groups as [external users](../../user/permissions.md#external-users-permissions). +Group membership is checked periodically through the `LdapGroupSync` background +task. + +**Omnibus configuration** + +1. Edit `/etc/gitlab/gitlab.rb`: + + ```ruby + gitlab_rails['ldap_servers'] = YAML.load <<-EOS + main: + ## snip... + ## + ## An array of CNs of groups containing users that should be considered external + ## + ## Ex. ['interns', 'contractors'] + ## + ## Note: Not `cn=interns` or the full DN + ## + external_groups: ['interns', 'contractors'] + EOS + ``` + +1. [Reconfigure GitLab][reconfigure] for the changes to take effect. + +**Source configuration** + +1. Edit `config/gitlab.yaml`: + + ```yaml + production: + ldap: + servers: + main: + # snip... + external_groups: ['interns', 'contractors'] + ``` + +1. [Restart GitLab][restart] for the changes to take effect. + +## Group sync technical details + +There is a lot going on with group sync 'under the hood'. This section +outlines what LDAP queries are executed and what behavior you can expect +from group sync. + +Group member access will be downgraded from a higher level if their LDAP group +membership changes. For example, if a user has 'Owner' rights in a group and the +next group sync reveals they should only have 'Developer' privileges, their +access will be adjusted accordingly. The only exception is if the user is the +*last* owner in a group. Groups need at least one owner to fulfill +administrative duties. + +### Supported LDAP group types/attributes + +GitLab supports LDAP groups that use member attributes `member`, `submember`, +`uniquemember`, `memberof` and `memberuid`. This means group sync supports, at +least, LDAP groups with object class `groupOfNames`, `posixGroup`, and +`groupOfUniqueName`. Other object classes should work fine as long as members +are defined as one of the mentioned attributes. This also means GitLab supports +Microsoft Active Directory, Apple Open Directory, Open LDAP, and 389 Server. +Other LDAP servers should work, too. + +Active Directory also supports nested groups. Group sync will recursively +resolve membership if `active_directory: true` is set in the configuration file. + +> **Note:** Nested group membership will only be resolved if the nested group + also falls within the configured `group_base`. For example, if GitLab sees a + nested group with DN `cn=nested_group,ou=special_groups,dc=example,dc=com` but + the configured `group_base` is `ou=groups,dc=example,dc=com`, `cn=nested_group` + will be ignored. + +### Queries + +- Each LDAP group is queried a maximum of one time with base `group_base` and + filter `(cn=<cn_from_group_link>)`. +- If the LDAP group has the `memberuid` attribute, GitLab will execute another + LDAP query per member to obtain each user's full DN. These queries are + executed with base `base`, scope 'base object', and a filter depending on + whether `user_filter` is set. Filter may be `(uid=<uid_from_group>)` or a + joining of `user_filter`. + +### Benchmarks + +Group sync was written to be as performant as possible. Data is cached, database +queries are optimized, and LDAP queries are minimized. The last benchmark run +revealed the following metrics: + +For 20,000 LDAP users, 11,000 LDAP groups and 1,000 GitLab groups with 10 +LDAP group links each: + +- Initial sync (no existing members assigned in GitLab) took 1.8 hours +- Subsequent syncs (checking membership, no writes) took 15 minutes + +These metrics are meant to provide a baseline and performance may vary based on +any number of factors. This was a pretty extreme benchmark and most instances will +not have near this many users or groups. Disk speed, database performance, +network and LDAP server response time will affect these metrics. + +## Troubleshooting + +### Referral error + +If you see `LDAP search error: Referral` in the logs, or when troubleshooting +LDAP Group Sync, this error may indicate a configuration problem. The LDAP +configuration `/etc/gitlab/gitlab.rb` (Omnibus) or `config/gitlab.yml` (source) +is in YAML format and is sensitive to indentation. Check that `group_base` and +`admin_group` configuration keys are indented 2 spaces past the server +identifier. The default identifier is `main` and an example snippet looks like +the following: + +```yaml +main: # 'main' is the GitLab 'provider ID' of this LDAP server + label: 'LDAP' + host: 'ldap.example.com' + ... + group_base: 'cn=my_group,ou=groups,dc=example,dc=com' + admin_group: 'my_admin_group' +``` + +[reconfigure]: ../restart_gitlab.md#omnibus-gitlab-reconfigure +[restart]: ../restart_gitlab.md#installations-from-source + +[^1]: In Active Directory, a user is marked as disabled/blocked if the user + account control attribute (`userAccountControl:1.2.840.113556.1.4.803`) + has bit 2 set. See https://ctogonewild.com/2009/09/03/bitmask-searches-in-ldap/ + for more information. + +### User DN has changed + +When an LDAP user is created in GitLab, their LDAP DN is stored for later reference. + +If GitLab cannot find a user by their DN, it will attempt to fallback +to finding the user by their email. If the lookup is successful, GitLab will +update the stored DN to the new value. + +### User is not being added to a group + +Sometimes you may think a particular user should be added to a GitLab group via +LDAP group sync, but for some reason it's not happening. There are several +things to check to debug the situation. + +- Ensure LDAP configuration has a `group_base` specified. This configuration is + required for group sync to work properly. +- Ensure the correct LDAP group link is added to the GitLab group. Check group + links by visiting the GitLab group, then **Settings dropdown -> LDAP groups**. +- Check that the user has an LDAP identity + 1. Sign in to GitLab as an administrator user. + 1. Navigate to **Admin area -> Users**. + 1. Search for the user + 1. Open the user, by clicking on their name. Do not click 'Edit'. + 1. Navigate to the **Identities** tab. There should be an LDAP identity with + an LDAP DN as the 'Identifier'. + +If all of the above looks good, jump in to a little more advanced debugging. +Often, the best way to learn more about why group sync is behaving a certain +way is to enable debug logging. There is verbose output that details every +step of the sync. + +1. Start a Rails console + + ```bash + # For Omnibus installations + sudo gitlab-rails console + + # For installations from source + sudo -u git -H bundle exec rails console production + ``` +1. Set the log level to debug (only for this session): + + ```ruby + Rails.logger.level = Logger::DEBUG + ``` +1. Choose a GitLab group to test with. This group should have an LDAP group link + already configured. If the output is `nil`, the group could not be found. + If a bunch of group attributes are output, your group was found successfully. + + ```ruby + group = Group.find_by(name: 'my_group') + + # Output + => #<Group:0x007fe825196558 id: 1234, name: "my_group"...> + ``` +1. Run a group sync for this particular group. + + ```ruby + EE::Gitlab::Auth::LDAP::Sync::Group.execute_all_providers(group) + ``` +1. Look through the output of the sync. See [example log output](#example-log-output) + below for more information about the output. +1. If you still aren't able to see why the user isn't being added, query the + LDAP group directly to see what members are listed. Still in the Rails console, + run the following query: + + ```ruby + adapter = Gitlab::Auth::LDAP::Adapter.new('ldapmain') # If `main` is the LDAP provider + ldap_group = EE::Gitlab::Auth::LDAP::Group.find_by_cn('group_cn_here', adapter) + + # Output + => #<EE::Gitlab::Auth::LDAP::Group:0x007fcbdd0bb6d8 + ``` +1. Query the LDAP group's member DNs and see if the user's DN is in the list. + One of the DNs here should match the 'Identifier' from the LDAP identity + checked earlier. If it doesn't, the user does not appear to be in the LDAP + group. + + ```ruby + ldap_group.member_dns + + # Output + => ["uid=john,ou=people,dc=example,dc=com", "uid=mary,ou=people,dc=example,dc=com"] + ``` +1. Some LDAP servers don't store members by DN. Rather, they use UIDs instead. + If you didn't see results from the last query, try querying by UIDs instead. + + ```ruby + ldap_group.member_uids + + # Output + => ['john','mary'] + ``` + +#### Example log output + +The output of the last command will be very verbose, but contains lots of +helpful information. For the most part you can ignore log entries that are SQL +statements. + +Indicates the point where syncing actually begins: + +```bash +Started syncing all providers for 'my_group' group +``` + +The follow entry shows an array of all user DNs GitLab sees in the LDAP server. +Note that these are the users for a single LDAP group, not a GitLab group. If +you have multiple LDAP groups linked to this GitLab group, you will see multiple +log entries like this - one for each LDAP group. If you don't see an LDAP user +DN in this log entry, LDAP is not returning the user when we do the lookup. +Verify the user is actually in the LDAP group. + +```bash +Members in 'ldap_group_1' LDAP group: ["uid=john0,ou=people,dc=example,dc=com", +"uid=mary0,ou=people,dc=example,dc=com", "uid=john1,ou=people,dc=example,dc=com", +"uid=mary1,ou=people,dc=example,dc=com", "uid=john2,ou=people,dc=example,dc=com", +"uid=mary2,ou=people,dc=example,dc=com", "uid=john3,ou=people,dc=example,dc=com", +"uid=mary3,ou=people,dc=example,dc=com", "uid=john4,ou=people,dc=example,dc=com", +"uid=mary4,ou=people,dc=example,dc=com"] +``` + +Shortly after each of the above entries, you will see a hash of resolved member +access levels. This hash represents all user DNs GitLab thinks should have +access to this group, and at which access level (role). This hash is additive, +and more DNs may be added, or existing entries modified, based on additional +LDAP group lookups. The very last occurrence of this entry should indicate +exactly which users GitLab believes should be added to the group. + +> **Note:** 10 is 'Guest', 20 is 'Reporter', 30 is 'Developer', 40 is 'Maintainer' + and 50 is 'Owner' + +```bash +Resolved 'my_group' group member access: {"uid=john0,ou=people,dc=example,dc=com"=>30, +"uid=mary0,ou=people,dc=example,dc=com"=>30, "uid=john1,ou=people,dc=example,dc=com"=>30, +"uid=mary1,ou=people,dc=example,dc=com"=>30, "uid=john2,ou=people,dc=example,dc=com"=>30, +"uid=mary2,ou=people,dc=example,dc=com"=>30, "uid=john3,ou=people,dc=example,dc=com"=>30, +"uid=mary3,ou=people,dc=example,dc=com"=>30, "uid=john4,ou=people,dc=example,dc=com"=>30, +"uid=mary4,ou=people,dc=example,dc=com"=>30} +``` + +It's not uncommon to see warnings like the following. These indicate that GitLab +would have added the user to a group, but the user could not be found in GitLab. +Usually this is not a cause for concern. + +If you think a particular user should already exist in GitLab, but you're seeing +this entry, it could be due to a mismatched DN stored in GitLab. See +[User DN has changed](#User-DN-has-changed) to update the user's LDAP identity. + +```bash +User with DN `uid=john0,ou=people,dc=example,dc=com` should have access +to 'my_group' group but there is no user in GitLab with that +identity. Membership will be updated once the user signs in for +the first time. +``` + +Finally, the following entry says syncing has finished for this group: + +```bash +Finished syncing all providers for 'my_group' group +``` diff --git a/doc/administration/auth/ldap.md b/doc/administration/auth/ldap.md index d5d0d99ac24..54279897e04 100644 --- a/doc/administration/auth/ldap.md +++ b/doc/administration/auth/ldap.md @@ -14,7 +14,7 @@ including group membership syncing as well as multiple LDAP servers support. The information on this page is relevant for both GitLab CE and EE. For more details about EE-specific LDAP features, see the -[LDAP Enterprise Edition documentation](https://docs.gitlab.com/ee/administration/auth/ldap-ee.html). +[LDAP Enterprise Edition documentation](ldap-ee.md). **[STARTER ONLY]** ## Security @@ -30,16 +30,16 @@ the LDAP server. ### User deletion -If a user is deleted from the LDAP server, they will be blocked in GitLab, as +If a user is deleted from the LDAP server, they will be blocked in GitLab as well. Users will be immediately blocked from logging in. However, there is an -LDAP check cache time (sync time) of one hour (see note). This means users that +LDAP check cache time of one hour (see note) which means users that are already logged in or are using Git over SSH will still be able to access GitLab for up to one hour. Manually block the user in the GitLab Admin area to immediately block all access. NOTE: **Note**: GitLab Enterprise Edition Starter supports a -[configurable sync time](https://docs.gitlab.com/ee/administration/auth/ldap-ee.html#adjusting-ldap-user-and-group-sync-schedules), +[configurable sync time](ldap-ee.md#adjusting-ldap-user-sync-schedule), with a default of one hour. ## Git password authentication @@ -48,6 +48,14 @@ LDAP-enabled users can always authenticate with Git using their GitLab username or email and LDAP password, even if password authentication for Git is disabled in the application settings. +## Google Secure LDAP **[CORE ONLY]** + +> Introduced in GitLab 11.9. + +[Google Cloud Identity](https://cloud.google.com/identity/) provides a Secure +LDAP service that can be configured with GitLab for authentication and group sync. +See [Google Secure LDAP](google_secure_ldap.md) for detailed configuration instructions. + ## Configuration NOTE: **Note**: @@ -56,6 +64,7 @@ to connect to one GitLab server. For a complete guide on configuring LDAP with GitLab Community Edition, please check the admin guide [How to configure LDAP with GitLab CE](how_to_configure_ldap_gitlab_ce/index.md). +For GitLab Enterprise Editions, see also [How to configure LDAP with GitLab EE](how_to_configure_ldap_gitlab_ee/index.md). **[STARTER ONLY]** To enable LDAP integration you need to add your LDAP server settings in `/etc/gitlab/gitlab.rb` or `/home/git/gitlab/config/gitlab.yml` for Omnibus @@ -81,6 +90,9 @@ library. `tls` corresponds to StartTLS, not to be confused with regular TLS. Normally, if you specify `ssl` it will be on port 636, while `tls` (StartTLS) would be on port 389. `plain` also operates on port 389. +NOTE: **Note:** +LDAP users must have an email address set, regardless of whether it is used to log in. + **Omnibus configuration** ```ruby @@ -136,14 +148,54 @@ main: ## verify_certificates: true - ## - ## Specifies the SSL version for OpenSSL to use, if the OpenSSL default - ## is not appropriate. - ## - ## Example: 'TLSv1_1' - ## - ## - ssl_version: '' + # OpenSSL::SSL::SSLContext options. + tls_options: + # Specifies the path to a file containing a PEM-format CA certificate, + # e.g. if you need to use an internal CA. + # + # Example: '/etc/ca.pem' + # + ca_file: '' + + # Specifies the SSL version for OpenSSL to use, if the OpenSSL default + # is not appropriate. + # + # Example: 'TLSv1_1' + # + ssl_version: '' + + # Specific SSL ciphers to use in communication with LDAP servers. + # + # Example: 'ALL:!EXPORT:!LOW:!aNULL:!eNULL:!SSLv2' + ciphers: '' + + # Client certificate + # + # Example: + # cert: | + # -----BEGIN CERTIFICATE----- + # MIIDbDCCAlSgAwIBAgIGAWkJxLmKMA0GCSqGSIb3DQEBCwUAMHcxFDASBgNVBAoTC0dvb2dsZSBJ + # bmMuMRYwFAYDVQQHEw1Nb3VudGFpbiBWaWV3MRQwEgYDVQQDEwtMREFQIENsaWVudDEPMA0GA1UE + # CxMGR1N1aXRlMQswCQYDVQQGEwJVUzETMBEGA1UECBMKQ2FsaWZvcm5pYTAeFw0xOTAyMjAwNzE4 + # rntnF4d+0dd7zP3jrWkbdtoqjLDT/5D7NYRmVCD5vizV98FJ5//PIHbD1gL3a9b2MPAc6k7NV8tl + # ... + # 4SbuJPAiJxC1LQ0t39dR6oMCAMab3hXQqhL56LrR6cRBp6Mtlphv7alu9xb/x51y2x+g2zWtsf80 + # Jrv/vKMsIh/sAyuogb7hqMtp55ecnKxceg== + # -----END CERTIFICATE ----- + cert: '' + + # Client private key + # key: | + # -----BEGIN PRIVATE KEY----- + # MIIEvQIBADANBgkqhkiG9w0BAQEFAASCBKcwggSjAgEAAoIBAQC3DmJtLRmJGY4xU1QtI3yjvxO6 + # bNuyE4z1NF6Xn7VSbcAaQtavWQ6GZi5uukMo+W5DHVtEkgDwh92ySZMuJdJogFbNvJvHAayheCdN + # 7mCQ2UUT9jGXIbmksUn9QMeJVXTZjgJWJzPXToeUdinx9G7+lpVa62UATEd1gaI3oyL72WmpDy/C + # rntnF4d+0dd7zP3jrWkbdtoqjLDT/5D7NYRmVCD5vizV98FJ5//PIHbD1gL3a9b2MPAc6k7NV8tl + # ... + # +9IhSYX+XIg7BZOVDeYqlPfxRvQh8vy3qjt/KUihmEPioAjLaGiihs1Fk5ctLk9A2hIUyP+sEQv9 + # l6RG+a/mW+0rCWn8JAd464Ps9hE= + # -----END PRIVATE KEY----- + key: '' ## ## Set a timeout, in seconds, for LDAP queries. This helps avoid blocking @@ -337,7 +389,7 @@ group, you can use the following syntax: Find more information about this "LDAP_MATCHING_RULE_IN_CHAIN" filter at <https://docs.microsoft.com/en-us/windows/desktop/ADSI/search-filter-syntax>. Support for nested members in the user filter should not be confused with -[group sync nested groups support (EE only)](https://docs.gitlab.com/ee/administration/auth/ldap-ee.html#supported-ldap-group-types-attributes). +[group sync nested groups support](ldap-ee.md#supported-ldap-group-typesattributes). **[STARTER ONLY]** Please note that GitLab does not support the custom filter syntax used by omniauth-ldap. diff --git a/doc/administration/auth/oidc.md b/doc/administration/auth/oidc.md new file mode 100644 index 00000000000..df4f22aa3e7 --- /dev/null +++ b/doc/administration/auth/oidc.md @@ -0,0 +1,140 @@ +# OpenID Connect OmniAuth provider + +GitLab can use [OpenID Connect](https://openid.net/specs/openid-connect-core-1_0.html) as an OmniAuth provider. + +To enable the OpenID Connect OmniAuth provider, you must register your application with an OpenID Connect provider. +The OpenID Connect will provide you with a client details and secret for you to use. + +1. On your GitLab server, open the configuration file. + + For Omnibus GitLab: + + ```sh + sudo editor /etc/gitlab/gitlab.rb + ``` + + For installations from source: + + ```sh + cd /home/git/gitlab + sudo -u git -H editor config/gitlab.yml + ``` + + See [Initial OmniAuth Configuration](../../integration/omniauth.md#initial-omniauth-configuration) for initial settings. + +1. Add the provider configuration. + + For Omnibus GitLab: + + ```ruby + gitlab_rails['omniauth_providers'] = [ + { 'name' => 'openid_connect', + 'label' => '<your_oidc_label>', + 'args' => { + "name' => 'openid_connect', + 'scope' => ['openid','profile'], + 'response_type' => 'code', + 'issuer' => '<your_oidc_url>', + 'discovery' => true, + 'client_auth_method' => 'query', + 'uid_field' => '<uid_field>', + 'client_options' => { + 'identifier' => '<your_oidc_client_id>', + 'secret' => '<your_oidc_client_secret>', + 'redirect_uri' => '<your_gitlab_url>/users/auth/openid_connect/callback' + } + } + } + ] + ``` + + For installation from source: + + ```yaml + - { name: 'openid_connect', + label: '<your_oidc_label>', + args: { + name: 'openid_connect', + scope: ['openid','profile'], + response_type: 'code', + issuer: '<your_oidc_url>', + discovery: true, + client_auth_method: 'query', + uid_field: '<uid_field>', + client_options: { + identifier: '<your_oidc_client_id>', + secret: '<your_oidc_client_secret>', + redirect_uri: '<your_gitlab_url>/users/auth/openid_connect/callback' + } + } + } + ``` + + > **Note:** + > + > - For more information on each configuration option refer to + the [OmniAuth OpenID Connect usage documentation](https://github.com/m0n9oose/omniauth_openid_connect#usage) and + the [OpenID Connect Core 1.0 specification](https://openid.net/specs/openid-connect-core-1_0.html). + +1. For the configuration above, change the values for the provider to match your OpenID Connect client setup. Use the following as a guide: + - `<your_oidc_label>` is the label that will be displayed on the login page. + - `<your_oidc_url>` (optional) is the URL that points to the OpenID Connect provider. For example, `https://example.com/auth/realms/your-realm`. + If this value is not provided, the URL is constructed from the `client_options` in the following format: `<client_options.scheme>://<client_options.host>:<client_options.port>`. + - If `discovery` is set to `true`, the OpenID Connect provider will try to auto discover the client options using `<your_oidc_url>/.well-known/openid-configuration`. Defaults to `false`. + - `<uid_field>` (optional) is the field name from the `user_info` details that will be used as `uid` value. For example, `preferred_username`. + If this value is not provided or the field with the configured value is missing from the `user_info` details, the `uid` will use the `sub` field. + - `client_options` are the OpenID Connect client-specific options. Specifically: + + - `identifier` is the client identifier as configured in the OpenID Connect service provider. + - `secret` is the client secret as configured in the OpenID Connect service provider. + - `redirect_uri` is the GitLab URL to redirect the user to after successful login. For example, `http://example.com/users/auth/openid_connect/callback`. + - `end_session_endpoint` (optional) is the URL to the endpoint that end the session (logout). Can be provided if auto-discovery disabled or unsuccessful. + + The following `client_options` are optional unless auto-discovery is disabled or unsuccessful: + + - `authorization_endpoint` is the URL to the endpoint that authorizes the end user. + - `token_endpoint` is the URL to the endpoint that provides Access Token. + - `userinfo_endpoint` is the URL to the endpoint that provides the user information. + - `jwks_uri` is the URL to the endpoint where the Token signer publishes its keys. + +1. Save the configuration file. +1. [Reconfigure](../restart_gitlab.md#omnibus-gitlab-reconfigure) or [restart GitLab](../restart_gitlab.md#installations-from-source) + for the changes to take effect if you installed GitLab via Omnibus or from source respectively. + +On the sign in page, there should now be an OpenID Connect icon below the regular sign in form. +Click the icon to begin the authentication process. The OpenID Connect provider will ask the user to +sign in and authorize the GitLab application (if confirmation required by the client). If everything goes well, the user +will be redirected to GitLab and will be signed in. + +## Example configurations + +The following configurations illustrate how to set up OpenID with +different providers with Omnibus GitLab. + +### Google + +See the [Google +documentation](https://developers.google.com/identity/protocols/OpenIDConnect) +for more details: + +```ruby + gitlab_rails['omniauth_providers'] = [ + { + 'name' => 'openid_connect', + 'label' => 'Google OpenID', + 'args' => { + 'name' => 'openid_connect', + 'scope' => ['openid', 'profile', 'email'], + 'response_type' => 'code', + 'issuer' => 'https://accounts.google.com', + 'client_auth_method' => 'query', + 'discovery' => true, + 'uid_field' => 'preferred_username', + 'client_options' => { + 'identifier' => '<YOUR PROJECT CLIENT ID>', + 'secret' => '<YOUR PROJECT CLIENT SECRET>', + 'redirect_uri' => 'https://example.com/users/auth/openid_connect/callback', + } + } + } +``` diff --git a/doc/administration/auth/okta.md b/doc/administration/auth/okta.md index 3136923fa96..aa4e1b0d2e0 100644 --- a/doc/administration/auth/okta.md +++ b/doc/administration/auth/okta.md @@ -1,6 +1,6 @@ # Okta SSO provider -Okta is a [Single Sign-on provider][okta-sso] that can be used to authenticate +Okta is a [Single Sign-on provider](https://www.okta.com/products/single-sign-on/) that can be used to authenticate with GitLab. The following documentation enables Okta as a SAML provider. @@ -92,18 +92,23 @@ Now that the Okta app is configured, it's time to enable it in GitLab. 1. Add the provider configuration. >**Notes:** + > >- Change the value for `assertion_consumer_service_url` to match the HTTPS endpoint of GitLab (append `users/auth/saml/callback` to the HTTPS URL of your GitLab installation to generate the correct value). + > >- To get the `idp_cert_fingerprint` fingerprint, first download the certificate from the Okta app you registered and then run: `openssl x509 -in okta.cert -noout -fingerprint`. Substitute `okta.cert` with the location of your certificate. + > >- Change the value of `idp_sso_target_url`, with the value of the **Identity Provider Single Sign-On URL** from the step when you configured the Okta app. - >- Change the value of `issuer` to a unique name, which will identify the application + > + >- Change the value of `issuer` to the value of the **Audience Restriction** from your Okta app configuration. This will identify GitLab to the IdP. + > >- Leave `name_identifier_format` as-is. **For Omnibus GitLab installations** @@ -140,7 +145,7 @@ Now that the Okta app is configured, it's time to enable it in GitLab. } ``` -1. [Reconfigure][reconf] or [restart] GitLab for Omnibus and installations +1. [Reconfigure](../restart_gitlab.md#omnibus-gitlab-reconfigure) or [restart](../restart_gitlab.md#installations-from-source) GitLab for Omnibus and installations from source respectively for the changes to take effect. You might want to try this out on an incognito browser window. @@ -150,10 +155,5 @@ You might want to try this out on an incognito browser window. >**Note:** Make sure the groups exist and are assigned to the Okta app. -You can take a look of the [SAML documentation][saml] on external groups since +You can take a look of the [SAML documentation](../../integration/saml.md#marking-users-as-external-based-on-saml-groups) on external groups since it works the same. - -[okta-sso]: https://www.okta.com/products/single-sign-on/ -[saml]: ../../integration/saml.md#external-groups -[reconf]: ../restart_gitlab.md#omnibus-gitlab-reconfigure -[restart]: ../restart_gitlab.md#installations-from-source diff --git a/doc/administration/auth/smartcard.md b/doc/administration/auth/smartcard.md new file mode 100644 index 00000000000..b33c5359b44 --- /dev/null +++ b/doc/administration/auth/smartcard.md @@ -0,0 +1,186 @@ +# Smartcard authentication **[PREMIUM ONLY]** + +GitLab supports authentication using smartcards. + +## Authentication methods + +GitLab supports two authentication methods: + +- X.509 certificates with local databases. +- LDAP servers. + +### Authentication against a local database with X.509 certificates + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/issues/726) in +[GitLab Premium](https://about.gitlab.com/pricing/) 11.6 as an experimental +feature. Smartcard authentication against local databases may change or be +removed completely in future releases. + +Smartcards with X.509 certificates can be used to authenticate with GitLab. + +To use a smartcard with an X.509 certificate to authenticate against a local +database with GitLab, `CN` and `emailAddress` must be defined in the +certificate. For example: + +``` +Certificate: + Data: + Version: 1 (0x0) + Serial Number: 12856475246677808609 (0xb26b601ecdd555e1) + Signature Algorithm: sha256WithRSAEncryption + Issuer: O=Random Corp Ltd, CN=Random Corp + Validity + Not Before: Oct 30 12:00:00 2018 GMT + Not After : Oct 30 12:00:00 2019 GMT + Subject: CN=Gitlab User, emailAddress=gitlab-user@example.com +``` + +### Authentication against an LDAP server + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/issues/7693) in +[GitLab Premium](https://about.gitlab.com/pricing/) 11.8 as an experimental +feature. Smartcard authentication against an LDAP server may change or be +removed completely in future releases. + +GitLab implements a standard way of certificate matching following +[RFC4523](https://tools.ietf.org/html/rfc4523). It uses the +`certificateExactMatch` certificate matching rule against the `userCertificate` +attribute. As a prerequisite, you must use an LDAP server that: + +- Supports the `certificateExactMatch` matching rule. +- Has the certificate stored in the `userCertificate` attribute. + +## Configure GitLab for smartcard authentication + +**For Omnibus installations** + +1. Edit `/etc/gitlab/gitlab.rb`: + + ```ruby + gitlab_rails['smartcard_enabled'] = true + gitlab_rails['smartcard_ca_file'] = "/etc/ssl/certs/CA.pem" + gitlab_rails['smartcard_client_certificate_required_port'] = 3444 + ``` + +1. Save the file and [reconfigure](../restart_gitlab.md#omnibus-gitlab-reconfigure) + GitLab for the changes to take effect. + +--- + +**For installations from source** + +1. Configure NGINX to request a client side certificate + + In NGINX configuration, an **additional** server context must be defined with + the same configuration except: + + - The additional NGINX server context must be configured to run on a different + port: + + ``` + listen *:3444 ssl; + ``` + + - The additional NGINX server context must be configured to require the client + side certificate: + + ``` + ssl_verify_depth 2; + ssl_client_certificate /etc/ssl/certs/CA.pem; + ssl_verify_client on; + ``` + + - The additional NGINX server context must be configured to forward the client + side certificate: + + ``` + proxy_set_header X-SSL-Client-Certificate $ssl_client_escaped_cert; + ``` + + For example, the following is an example server context in an NGINX + configuration file (eg. in `/etc/nginx/sites-available/gitlab-ssl`): + + ``` + server { + listen *:3444 ssl; + + # certificate for configuring SSL + ssl_certificate /path/to/example.com.crt; + ssl_certificate_key /path/to/example.com.key; + + ssl_verify_depth 2; + # CA certificate for client side certificate verification + ssl_client_certificate /etc/ssl/certs/CA.pem; + ssl_verify_client on; + + location / { + proxy_set_header Host $http_host; + proxy_set_header X-Real-IP $remote_addr; + proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; + proxy_set_header X-Forwarded-Proto $scheme; + proxy_set_header Upgrade $http_upgrade; + proxy_set_header Connection $connection_upgrade; + + proxy_set_header X-SSL-Client-Certificate $ssl_client_escaped_cert; + + proxy_read_timeout 300; + + proxy_pass http://gitlab-workhorse; + } + } + ``` + +1. Edit `config/gitlab.yml`: + + ```yaml + ## Smartcard authentication settings + smartcard: + # Allow smartcard authentication + enabled: true + + # Path to a file containing a CA certificate + ca_file: '/etc/ssl/certs/CA.pem' + + # Port where the client side certificate is requested by NGINX + client_certificate_required_port: 3444 + ``` + +1. Save the file and [restart](../restart_gitlab.md#installations-from-source) + GitLab for the changes to take effect. + +### Additional steps when authenticating against an LDAP server + +**For Omnibus installations** + +1. Edit `/etc/gitlab/gitlab.rb`: + + ```ruby + gitlab_rails['ldap_servers'] = YAML.load <<-EOS + main: + # snip... + # Enable smartcard authentication against the LDAP server. Valid values + # are "false", "optional", and "required". + smartcard_auth: optional + EOS + ``` + +1. Save the file and [reconfigure](../restart_gitlab.md#omnibus-gitlab-reconfigure) + GitLab for the changes to take effect. + +**For installations from source** + +1. Edit `config/gitlab.yml`: + + ```yaml + production: + ldap: + servers: + main: + # snip... + # Enable smartcard authentication against the LDAP server. Valid values + # are "false", "optional", and "required". + smartcard_auth: optional + ``` + +1. Save the file and [restart](../restart_gitlab.md#installations-from-source) + GitLab for the changes to take effect. diff --git a/doc/administration/build_artifacts.md b/doc/administration/build_artifacts.md index 623a5321f32..85ae2946bc8 100644 --- a/doc/administration/build_artifacts.md +++ b/doc/administration/build_artifacts.md @@ -1 +1,5 @@ +--- +redirect_to: 'job_artifacts.md' +--- + This document was moved to [job_artifacts](job_artifacts.md). diff --git a/doc/administration/compliance.md b/doc/administration/compliance.md index 0414b3ec12e..246addb6dc9 100644 --- a/doc/administration/compliance.md +++ b/doc/administration/compliance.md @@ -6,13 +6,13 @@ GitLab’s [security features](../security/README.md) may also help you meet rel |Feature |GitLab tier |GitLab.com | | ---------| :--------: | :-------: | -|**[Restrict SSH Keys](../README.html#administrator-documentation)**<br>Control the technology and key length of SSH keys used to access GitLab|Core+|| -|**[Granular user roles and flexible permissions](../user/permissions.html)**<br>Manage access and permissions with five different user roles and settings for external users. Set permissions according to people's role, rather than either read or write access to a repository. Don't share the source code with people that only need access to the issue tracker.|Core+|✓| -|**[Enforce TOS acceptance](../user/admin_area/settings/terms.html)**<br>Enforce your users accepting new terms of service by blocking GitLab traffic.|Core+|| -|**[Email all users of a project, group, or entire server](../user/admin_area/settings/terms.html)**<br>An admin can email groups of users based on project or group membership, or email everyone using the GitLab instance. This is great for scheduled maintenance or upgrades.|Starter+|| +|**[Restrict SSH Keys](../security/ssh_keys_restrictions.md)**<br>Control the technology and key length of SSH keys used to access GitLab|Core+|| +|**[Granular user roles and flexible permissions](../user/permissions.md)**<br>Manage access and permissions with five different user roles and settings for external users. Set permissions according to people's role, rather than either read or write access to a repository. Don't share the source code with people that only need access to the issue tracker.|Core+|✓| +|**[Enforce TOS acceptance](../user/admin_area/settings/terms.md)**<br>Enforce your users accepting new terms of service by blocking GitLab traffic.|Core+|| +|**[Email all users of a project, group, or entire server](../user/admin_area/settings/terms.md)**<br>An admin can email groups of users based on project or group membership, or email everyone using the GitLab instance. This is great for scheduled maintenance or upgrades.|Starter+|| |**[Omnibus package supports log forwarding](https://docs.gitlab.com/omnibus/settings/logs.html#udp-log-forwarding)**<br>Forward your logs to a central system.|Starter+|| -|**[Lock project membership to group](../workflow/groups.html#lock-project-membership-to-members-of-this-group)**<br>Group owners can prevent new members from being added to projects within a group.|Starter+|✓| -|**[LDAP group sync](https://docs.gitlab.com/ee/administration/auth/ldap-ee.html#group-sync)**<br>GitLab Enterprise Edition gives admins the ability to automatically sync groups and manage SSH keys, permissions, and authentication, so you can focus on building your product, not configuring your tools.|Starter+|| -|**[LDAP group sync filters](https://docs.gitlab.com/ee/administration/auth/ldap-ee.html#group-sync)**<br>GitLab Enterprise Edition Premium gives more flexibility to synchronize with LDAP based on filters, meaning you can leverage LDAP attributes to map GitLab permissions.|Premium+|| -|**[Audit logs](https://docs.gitlab.com/ee/administration/audit_events.html)**<br>To maintain the integrity of your code, GitLab Enterprise Edition Premium gives admins the ability to view any modifications made within the GitLab server in an advanced audit log system, so you can control, analyze and track every change.|Premium+|| -|**[Auditor users](https://docs.gitlab.com/ee/administration/auditor_users.html)**<br>Auditor users are users who are given read-only access to all projects, groups, and other resources on the GitLab instance.|Premium+||
\ No newline at end of file +|**[Lock project membership to group](../user/group/index.md#member-lock-starter)**<br>Group owners can prevent new members from being added to projects within a group.|Starter+|✓| +|**[LDAP group sync](auth/ldap-ee.md#group-sync)**<br>GitLab Enterprise Edition gives admins the ability to automatically sync groups and manage SSH keys, permissions, and authentication, so you can focus on building your product, not configuring your tools.|Starter+|| +|**[LDAP group sync filters](auth/ldap-ee.md#group-sync)**<br>GitLab Enterprise Edition Premium gives more flexibility to synchronize with LDAP based on filters, meaning you can leverage LDAP attributes to map GitLab permissions.|Premium+|| +|**[Audit logs](audit_events.md)**<br>To maintain the integrity of your code, GitLab Enterprise Edition Premium gives admins the ability to view any modifications made within the GitLab server in an advanced audit log system, so you can control, analyze and track every change.|Premium+|| +|**[Auditor users](auditor_users.md)**<br>Auditor users are users who are given read-only access to all projects, groups, and other resources on the GitLab instance.|Premium+|| diff --git a/doc/administration/container_registry.md b/doc/administration/container_registry.md index a1ac4a2a57c..4d55f2357c1 100644 --- a/doc/administration/container_registry.md +++ b/doc/administration/container_registry.md @@ -1,6 +1,7 @@ # GitLab Container Registry administration > **Notes:** +> > - [Introduced][ce-4040] in GitLab 8.8. > - Container Registry manifest `v1` support was added in GitLab 8.9 to support > Docker versions earlier than 1.10. @@ -361,6 +362,10 @@ configuring a different storage driver. By default the GitLab Container Registry is configured to use the filesystem driver, which makes use of [storage path](#container-registry-storage-path) configuration. +NOTE: **Note:** Enabling a storage driver other than `filesystem` would mean +that your Docker client needs to be able to access the storage backend directly. +In that case, you must use an address that resolves and is accessible outside GitLab server. + The different supported drivers are: | Driver | Description | @@ -368,26 +373,23 @@ The different supported drivers are: | filesystem | Uses a path on the local filesystem | | azure | Microsoft Azure Blob Storage | | gcs | Google Cloud Storage | -| s3 | Amazon Simple Storage Service | +| s3 | Amazon Simple Storage Service. Be sure to configure your storage bucket with the correct [S3 Permission Scopes](https://docs.docker.com/registry/storage-drivers/s3/#s3-permission-scopes). | | swift | OpenStack Swift Object Storage | | oss | Aliyun OSS | Read more about the individual driver's config options in the [Docker Registry docs][storage-config]. -> **Warning** GitLab will not backup Docker images that are not stored on the +CAUTION: **Warning:** GitLab will not backup Docker images that are not stored on the filesystem. Remember to enable backups with your object storage provider if desired. -> -> **Important** Enabling storage driver other than `filesystem` would mean -that your Docker client needs to be able to access the storage backend directly. -So you must use an address that resolves and is accessible outside GitLab server. ---- +NOTE: **Note:** +`regionendpoint` is only required when configuring an S3 compatible service such as Minio. It takes a URL such as `http://127.0.0.1:9000`. **Omnibus GitLab installations** -To configure the storage driver in Omnibus: +To configure the `s3` storage driver in Omnibus: 1. Edit `/etc/gitlab/gitlab.rb`: @@ -397,29 +399,29 @@ To configure the storage driver in Omnibus: 'accesskey' => 's3-access-key', 'secretkey' => 's3-secret-key-for-access-key', 'bucket' => 'your-s3-bucket', - 'region' => 'your-s3-region' + 'region' => 'your-s3-region', + 'regionendpoint' => 'your-s3-regionendpoint' } } ``` 1. Save the file and [reconfigure GitLab][] for the changes to take effect. ---- - **Installations from source** Configuring the storage driver is done in your registry config YML file created when you [deployed your docker registry][registry-deploy]. -Example: +`s3` storage driver example: -``` +```yml storage: s3: accesskey: 'AKIAKIAKI' secretkey: 'secret123' bucket: 'gitlab-registry-bucket-AKIAKIAKI' region: 'your-s3-region' + regionendpoint: 'your-s3-regionendpoint' cache: blobdescriptor: inmemory delete: @@ -657,6 +659,37 @@ Start with a value between `25000000` (25MB) and `50000000` (50MB). 1. Save the file and [restart GitLab][] for the changes to take effect. +### Supporting older Docker clients + +As of GitLab 11.9, we began shipping version 2.7.1 of the Docker container registry, which disables the schema1 manifest by default. If you are still using older Docker clients (1.9 or older), you may experience an error pushing images. See [omnibus-4145](https://gitlab.com/gitlab-org/omnibus-gitlab/issues/4145) for more details. + +You can add a configuration option for backwards compatibility. + +**For Omnibus installations** + +1. Edit `/etc/gitlab/gitlab.rb`: + + ```ruby + registry['compatibility_schema1_enabled'] = true + ``` + +1. Save the file and [reconfigure GitLab][] for the changes to take effect. + +--- + +**For installations from source** + +1. Edit the YML configuration file you created when you [deployed the registry][registry-deploy]. Add the following snippet: + + ```yaml + compatibility: + schema1: + enabled: true + ``` + +1. Restart the registry for the changes to take affect. + + [ce-18239]: https://gitlab.com/gitlab-org/gitlab-ce/issues/18239 [docker-insecure-self-signed]: https://docs.docker.com/registry/insecure/#use-self-signed-certificates [reconfigure gitlab]: restart_gitlab.md#omnibus-gitlab-reconfigure diff --git a/doc/administration/custom_hooks.md b/doc/administration/custom_hooks.md index 60ad4bf4e8f..113514e1ee8 100644 --- a/doc/administration/custom_hooks.md +++ b/doc/administration/custom_hooks.md @@ -1,12 +1,16 @@ -# Custom Git Hooks +# Custom server-side Git hooks -> **Note:** Custom Git hooks must be configured on the filesystem of the GitLab +NOTE: **Note:** +Custom Git hooks must be configured on the filesystem of the GitLab server. Only GitLab server administrators will be able to complete these tasks. Please explore [webhooks] and [CI] as an option if you do not have filesystem access. For a user configurable Git hook interface, see -[Push Rules](https://docs.gitlab.com/ee/push_rules/push_rules.html), +[Push Rules](../push_rules/push_rules.md), available in GitLab Enterprise Edition. +NOTE: **Note:** +Custom Git hooks won't be replicated to secondary nodes if you use [GitLab Geo](geo/replication/index.md) + Git natively supports hooks that are executed on different actions. Examples of server-side git hooks include pre-receive, post-receive, and update. See [Git SCM Server-Side Hooks][hooks] for more information about each hook type. @@ -14,15 +18,14 @@ See [Git SCM Server-Side Hooks][hooks] for more information about each hook type As of gitlab-shell version 2.2.0 (which requires GitLab 7.5+), GitLab administrators can add custom git hooks to any GitLab project. -## Setup - -Normally, Git hooks are placed in the repository or project's `hooks` directory. -GitLab creates a symlink from each project's `hooks` directory to the -gitlab-shell `hooks` directory for ease of maintenance between gitlab-shell -upgrades. As such, custom hooks are implemented a little differently. Behavior -is exactly the same once the hook is created, though. +## Create a custom Git hook for a repository -Follow the steps below to set up a custom hook: +Server-side Git hooks are typically placed in the repository's `hooks` +subdirectory. In GitLab, hook directories are are symlinked to the gitlab-shell +`hooks` directory for ease of maintenance between gitlab-shell upgrades. +Custom hooks are implemented differently, but the behavior is exactly the same +once the hook is created. Follow the steps below to set up a custom hook for a +repository: 1. Pick a project that needs a custom Git hook. 1. On the GitLab server, navigate to the project's repository directory. @@ -42,33 +45,56 @@ Follow the steps below to set up a custom hook: That's it! Assuming the hook code is properly implemented the hook will fire as appropriate. +## Set a global Git hook for all repositories + +To create a Git hook that applies to all of your repositories in +your instance, set a global Git hook. Since all the repositories' `hooks` +directories are symlinked to gitlab-shell's `hooks` directory, adding any hook +to the gitlab-shell `hooks` directory will also apply it to all repositories. Follow +the steps below to properly set up a custom hook for all repositories: + +1. On the GitLab server, navigate to the configured custom hook directory. The + default is in the gitlab-shell directory. The gitlab-shell `hook` directory + for an installation from source the path is usually + `/home/git/gitlab-shell/hooks`. For Omnibus installs the path is usually + `/opt/gitlab/embedded/service/gitlab-shell/hooks`. + To look in a different directory for the global custom hooks, + set `custom_hooks_dir` in the gitlab-shell config. For + Omnibus installations, this can be set in `gitlab.rb`; and in source + installations, this can be set in `gitlab-shell/config.yml`. +1. Create a new directory in this location. Depending on your hook, it will be + either a `pre-receive.d`, `post-receive.d`, or `update.d` directory. +1. Inside this new directory, add your hook. Hooks can be + in any language. Ensure the 'shebang' at the top properly reflects the language + type. For example, if the script is in Ruby the shebang will probably be + `#!/usr/bin/env ruby`. +1. Make the hook file executable and make sure it's owned by Git. + +Now test the hook to see that it's functioning properly. + ## Chained hooks support > [Introduced][93] in GitLab Shell 4.1.0 and GitLab 8.15. -Hooks can be also placed in `hooks/<hook_name>.d` (global) or -`custom_hooks/<hook_name>.d` (per project) directories supporting chained +Hooks can be also global or be set per project directories and support a chained execution of the hooks. -NOTE: **Note:** `<hook_name>.d` would need to be either `pre-receive.d`, -`post-receive.d`, or `update.d` to work properly. Any other names will be ignored. +NOTE: **Note:** +`<hook_name>.d` would need to be either `pre-receive.d`, +`post-receive.d`, or `update.d` to work properly. Any other names will be ignored. -To look in a different directory for the global custom hooks (those in -`hooks/<hook_name.d>`), set `custom_hooks_dir` in gitlab-shell config. For -Omnibus installations, this can be set in `gitlab.rb`; and in source -installations, this can be set in `gitlab-shell/config.yml`. +NOTE: **Note:** +Files in `.d` directories need to be executable and not match the backup file +pattern (`*~`). The hooks are searched and executed in this order: 1. `gitlab-shell/hooks` directory as known to Gitaly -1. `<project>.git/hooks/<hook_name>` - executed by `git` itself, this is `gitlab-shell/hooks/<hook_name>` +1. `<project>.git/hooks/<hook_name>` - executed by `git` itself, this is symlinked to `gitlab-shell/hooks/<hook_name>` 1. `<project>.git/custom_hooks/<hook_name>` - per project hook (this is already existing behavior) 1. `<project>.git/custom_hooks/<hook_name>.d/*` - per project hooks 1. `<project>.git/hooks/<hook_name>.d/*` OR `<custom_hooks_dir>/<hook_name.d>/*` - global hooks: all executable files (minus editor backup files) -Files in `.d` directories need to be executable and not match the backup file -pattern (`*~`). - The hooks of the same type are executed in order and execution stops on the first script exiting with a non-zero value. @@ -76,9 +102,21 @@ first script exiting with a non-zero value. > [Introduced][5073] in GitLab 8.10. -If the commit is declined or an error occurs during the Git hook check, -the STDERR or STDOUT message of the hook will be present in GitLab's UI. -STDERR takes precedence over STDOUT. +To have custom error messages appear in GitLab's UI when the commit is +declined or an error occurs during the Git hook, your script should: + +- Send the custom error messages to either the script's `stdout` or `stderr`. +- Prefix each message with `GL-HOOK-ERR:` with no characters appearing before the prefix. + +### Example custom error message + +This hook script written in bash will generate the following message in GitLab's UI: + +```bash +#!/bin/sh +echo "GL-HOOK-ERR: My custom error message."; +exit 1 +```  diff --git a/doc/administration/database_load_balancing.md b/doc/administration/database_load_balancing.md new file mode 100644 index 00000000000..7f3be402b84 --- /dev/null +++ b/doc/administration/database_load_balancing.md @@ -0,0 +1,277 @@ +# Database Load Balancing **[PREMIUM ONLY]** + +> [Introduced][ee-1283] in [GitLab Premium][eep] 9.0. + +Distribute read-only queries among multiple database servers. + +## Overview + +Database load balancing improves the distribution of database workloads across +multiple computing resources. Load balancing aims to optimize resource use, +maximize throughput, minimize response time, and avoid overload of any single +resource. Using multiple components with load balancing instead of a single +component may increase reliability and availability through redundancy. +[_Wikipedia article_][wikipedia] + +When database load balancing is enabled in GitLab, the load is balanced using +a simple round-robin algorithm, without any external dependencies such as Redis. +Load balancing is not enabled for Sidekiq as this would lead to consistency +problems, and Sidekiq mostly performs writes anyway. + +In the following image, you can see the load is balanced rather evenly among +all the secondaries (`db4`, `db5`, `db6`). Because `SELECT` queries are not +sent to the primary (unless necessary), the primary (`db3`) hardly has any load. + + + +## Requirements + +For load balancing to work you will need at least PostgreSQL 9.2 or newer, +[**MySQL is not supported**][db-req]. You also need to make sure that you have +at least 1 secondary in [hot standby][hot-standby] mode. + +Load balancing also requires that the configured hosts **always** point to the +primary, even after a database failover. Furthermore, the additional hosts to +balance load among must **always** point to secondary databases. This means that +you should put a load balance in front of every database, and have GitLab connect +to those load balancers. + +For example, say you have a primary (`db1.gitlab.com`) and two secondaries, +`db2.gitlab.com` and `db3.gitlab.com`. For this setup you will need to have 3 +load balancers, one for every host. For example: + +* `primary.gitlab.com` forwards to `db1.gitlab.com` +* `secondary1.gitlab.com` forwards to `db2.gitlab.com` +* `secondary2.gitlab.com` forwards to `db3.gitlab.com` + +Now let's say that a failover happens and db2 becomes the new primary. This +means forwarding should now happen as follows: + +* `primary.gitlab.com` forwards to `db2.gitlab.com` +* `secondary1.gitlab.com` forwards to `db1.gitlab.com` +* `secondary2.gitlab.com` forwards to `db3.gitlab.com` + +GitLab does not take care of this for you, so you will need to do so yourself. + +Finally, load balancing requires that GitLab can connect to all hosts using the +same credentials and port as configured in the +[Enabling load balancing](#enabling-load-balancing) section. Using +different ports or credentials for different hosts is not supported. + +## Use cases + +- For GitLab instances with thousands of users and high traffic, you can use + database load balancing to reduce the load on the primary database and + increase responsiveness, thus resulting in faster page load inside GitLab. + +## Enabling load balancing + +For the environment in which you want to use load balancing, you'll need to add +the following. This will balance the load between `host1.example.com` and +`host2.example.com`. + +**In Omnibus installations:** + +1. Edit `/etc/gitlab/gitlab.rb` and add the following line: + + ```ruby + gitlab_rails['db_load_balancing'] = { 'hosts' => ['host1.example.com', 'host2.example.com'] } + ``` + +1. Save the file and [reconfigure GitLab][] for the changes to take effect. + +--- + +**In installations from source:** + +1. Edit `/home/git/gitlab/config/database.yml` and add or amend the following lines: + + ```yaml + production: + username: gitlab + database: gitlab + encoding: unicode + load_balancing: + hosts: + - host1.example.com + - host2.example.com + ``` + +1. Save the file and [restart GitLab][] for the changes to take effect. + +## Service Discovery + +> [Introduced][ee-5883] in [GitLab Premium][eep] 11.0. + +Service discovery allows GitLab to automatically retrieve a list of secondary +databases to use, instead of having to manually specify these in the +`database.yml` configuration file. Service discovery works by periodically +checking a DNS A record, using the IPs returned by this record as the addresses +for the secondaries. For service discovery to work, all you need is a DNS server +and an A record containing the IP addresses of your secondaries. + +To use service discovery you need to change your `database.yml` configuration +file so it looks like the following: + +```yaml +production: + username: gitlab + database: gitlab + encoding: unicode + load_balancing: + discover: + nameserver: localhost + record: secondary.postgresql.service.consul + port: 8600 + interval: 60 + disconnect_timeout: 120 +``` + +Here the `discover:` section specifies the configuration details to use for +service discovery. + +### Configuration + +The following options can be set: + +| Option | Description | Default | +|----------------------|---------------------------------------------------------------------------------------------------|-----------| +| `nameserver` | The nameserver to use for looking up the DNS record. | localhost | +| `record` | The A record to look up. This option is required for service discovery to work. | | +| `port` | The port of the nameserver. | 8600 | +| `interval` | The minimum time in seconds between checking the DNS record. | 60 | +| `disconnect_timeout` | The time in seconds after which an old connection is closed, after the list of hosts was updated. | 120 | +| `use_tcp` | Lookup DNS resources using TCP instead of UDP | false | + +The `interval` value specifies the _minimum_ time between checks. If the A +record has a TTL greater than this value, then service discovery will honor said +TTL. For example, if the TTL of the A record is 90 seconds, then service +discovery will wait at least 90 seconds before checking the A record again. + +When the list of hosts is updated, it might take a while for the old connections +to be terminated. The `disconnect_timeout` setting can be used to enforce an +upper limit on the time it will take to terminate all old database connections. + +Some nameservers (like [Consul][consul-udp]) can return a truncated list of hosts when +queried over UDP. To overcome this issue, you can use TCP for querying by setting +`use_tcp` to `true`. + +### Forking + +If you use an application server that forks, such as Unicorn, you _have to_ +update your Unicorn configuration to start service discovery _after_ a fork. +Failure to do so will lead to service discovery only running in the parent +process. If you are using Unicorn, then you can add the following to your +Unicorn configuration file: + +```ruby +after_fork do |server, worker| + defined?(Gitlab::Database::LoadBalancing) && + Gitlab::Database::LoadBalancing.start_service_discovery +end +``` + +This will ensure that service discovery is started in both the parent and all +child processes. + +## Balancing queries + +Read-only `SELECT` queries will be balanced among all the secondary hosts. +Everything else (including transactions) will be executed on the primary. +Queries such as `SELECT ... FOR UPDATE` are also executed on the primary. + +## Prepared statements + +Prepared statements don't work well with load balancing and are disabled +automatically when load balancing is enabled. This should have no impact on +response timings. + +## Primary sticking + +After a write has been performed, GitLab will stick to using the primary for a +certain period of time, scoped to the user that performed the write. GitLab will +revert back to using secondaries when they have either caught up, or after 30 +seconds. + +## Failover handling + +In the event of a failover or an unresponsive database, the load balancer will +try to use the next available host. If no secondaries are available the +operation is performed on the primary instead. + +In the event of a connection error being produced when writing data, the +operation will be retried up to 3 times using an exponential back-off. + +When using load balancing, you should be able to safely restart a database server +without it immediately leading to errors being presented to the users. + +## Logging + +The load balancer logs various messages, such as: + +* When a host is marked as offline +* When a host comes back online +* When all secondaries are offline + +Each log message contains the tag `[DB-LB]` to make searching/filtering of such +log entries easier. For example: + +``` +[DB-LB] Host 10.123.2.5 came back online +[DB-LB] Marking host 10.123.2.7 as offline +[DB-LB] Marking host 10.123.2.7 as offline +[DB-LB] Marking host 10.123.2.7 as offline +[DB-LB] Marking host 10.123.2.7 as offline +[DB-LB] Marking host 10.123.2.7 as offline +[DB-LB] Host 10.123.2.6 came back online +[DB-LB] Marking host 10.123.2.7 as offline +[DB-LB] Marking host 10.123.2.7 as offline +[DB-LB] Marking host 10.123.2.7 as offline +[DB-LB] Host 10.123.2.7 came back online +[DB-LB] Host 10.123.2.7 came back online +``` + +## Handling Stale Reads + +> [Introduced][ee-3526] in [GitLab Premium][eep] 10.3. + +To prevent reading from an outdated secondary the load balancer will check if it +is in sync with the primary. If the data is determined to be recent enough the +secondary can be used, otherwise it will be ignored. To reduce the overhead of +these checks we only perform these checks at certain intervals. + +There are three configuration options that influence this behaviour: + +| Option | Description | Default | +|------------------------------|----------------------------------------------------------------------------------------------------------------|------------| +| `max_replication_difference` | The amount of data (in bytes) a secondary is allowed to lag behind when it hasn't replicated data for a while. | 8 MB | +| `max_replication_lag_time` | The maximum number of seconds a secondary is allowed to lag behind before we stop using it. | 60 seconds | +| `replica_check_interval` | The minimum number of seconds we have to wait before checking the status of a secondary. | 60 seconds | + +The defaults should be sufficient for most users. Should you want to change them +you can specify them in `config/database.yml` like so: + +```yaml +production: + username: gitlab + database: gitlab + encoding: unicode + load_balancing: + hosts: + - host1.example.com + - host2.example.com + max_replication_difference: 16777216 # 16 MB + max_replication_lag_time: 30 + replica_check_interval: 30 +``` + +[hot-standby]: https://www.postgresql.org/docs/9.6/static/hot-standby.html +[ee-1283]: https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/1283 +[eep]: https://about.gitlab.com/pricing/ +[reconfigure gitlab]: restart_gitlab.md#omnibus-gitlab-reconfigure "How to reconfigure Omnibus GitLab" +[restart gitlab]: restart_gitlab.md#installations-from-source "How to restart GitLab" +[wikipedia]: https://en.wikipedia.org/wiki/Load_balancing_(computing) +[db-req]: ../install/requirements.md#database +[ee-3526]: https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/3526 +[ee-5883]: https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/5883 +[consul-udp]: https://www.consul.io/docs/agent/dns.html#udp-based-dns-queries diff --git a/doc/administration/dependency_proxy.md b/doc/administration/dependency_proxy.md new file mode 100644 index 00000000000..4dc1f4dcba4 --- /dev/null +++ b/doc/administration/dependency_proxy.md @@ -0,0 +1,150 @@ +# GitLab Dependency Proxy administration **[PREMIUM ONLY]** + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/issues/7934) in [GitLab Premium](https://about.gitlab.com/pricing) 11.11. + +GitLab can be utilized as a dependency proxy for a variety of common package managers. + +This is the administration documentation. If you want to learn how to use the +dependency proxies, see the [user guide](../user/group/dependency_proxy/index.md). + +## Enabling the Dependency Proxy feature + +NOTE: **Note:** +Dependency proxy requires the Puma web server to be enabled. +Puma support is EXPERIMENTAL at this time. + +To enable the Dependency proxy feature: + +**Omnibus GitLab installations** + +1. Edit `/etc/gitlab/gitlab.rb` and add the following line: + + ```ruby + gitlab_rails['dependency_proxy_enabled'] = true + ``` + +1. Save the file and [reconfigure GitLab][] for the changes to take effect. +1. Enable the [Puma web server](https://docs.gitlab.com/omnibus/settings/puma.html). + +**Installations from source** + +1. After the installation is complete, you will have to configure the `dependency_proxy` + section in `config/gitlab.yml`. Set to `true` to enable it: + + ```yaml + dependency_proxy: + enabled: true + ``` + +1. [Restart GitLab] for the changes to take effect. +1. Enable the [Puma web server](../install/installation.md#using-puma). + +## Changing the storage path + +By default, the dependency proxy files are stored locally, but you can change the default +local location or even use object storage. + +### Changing the local storage path + +The dependency proxy files for Omnibus GitLab installations are stored under +`/var/opt/gitlab/gitlab-rails/shared/dependency_proxy/` and for source +installations under `shared/dependency_proxy/` (relative to the git home directory). +To change the local storage path: + +**Omnibus GitLab installations** + +1. Edit `/etc/gitlab/gitlab.rb` and add the following line: + + ```ruby + gitlab_rails['dependency_proxy_storage_path'] = "/mnt/dependency_proxy" + ``` + +1. Save the file and [reconfigure GitLab][] for the changes to take effect. + +**Installations from source** + +1. Edit the `dependency_proxy` section in `config/gitlab.yml`: + + ```yaml + dependency_proxy: + enabled: true + storage_path: shared/dependency_proxy + ``` +1. [Restart GitLab] for the changes to take effect. + +### Using object storage + +Instead of relying on the local storage, you can use an object storage to +upload the blobs of the dependency proxy: + +**Omnibus GitLab installations** + +1. Edit `/etc/gitlab/gitlab.rb` and add the following lines (uncomment where + necessary): + + ```ruby + gitlab_rails['dependency_proxy_enabled'] = true + gitlab_rails['dependency_proxy_storage_path'] = "/var/opt/gitlab/gitlab-rails/shared/dependency_proxy" + gitlab_rails['dependency_proxy_object_store_enabled'] = true + gitlab_rails['dependency_proxy_object_store_remote_directory'] = "dependency_proxy" # The bucket name. + gitlab_rails['dependency_proxy_object_store_direct_upload'] = false # Use Object Storage directly for uploads instead of background uploads if enabled (Default: false). + gitlab_rails['dependency_proxy_object_store_background_upload'] = true # Temporary option to limit automatic upload (Default: true). + gitlab_rails['dependency_proxy_object_store_proxy_download'] = false # Passthrough all downloads via GitLab instead of using Redirects to Object Storage. + gitlab_rails['dependency_proxy_object_store_connection'] = { + ## + ## If the provider is AWS S3, uncomment the following + ## + #'provider' => 'AWS', + #'region' => 'eu-west-1', + #'aws_access_key_id' => 'AWS_ACCESS_KEY_ID', + #'aws_secret_access_key' => 'AWS_SECRET_ACCESS_KEY', + ## + ## If the provider is other than AWS (an S3-compatible one), uncomment the following + ## + #'host' => 's3.amazonaws.com', + #'aws_signature_version' => 4 # For creation of signed URLs. Set to 2 if provider does not support v4. + #'endpoint' => 'https://s3.amazonaws.com' # Useful for S3-compliant services such as DigitalOcean Spaces. + #'path_style' => false # If true, use 'host/bucket_name/object' instead of 'bucket_name.host/object'. + } + ``` + +1. Save the file and [reconfigure GitLab][] for the changes to take effect. + +**Installations from source** + +1. Edit the `dependency_proxy` section in `config/gitlab.yml` (uncomment where necessary): + + ```yaml + dependency_proxy: + enabled: true + ## + ## The location where build dependency_proxy are stored (default: shared/dependency_proxy). + ## + #storage_path: shared/dependency_proxy + object_store: + enabled: false + remote_directory: dependency_proxy # The bucket name. + #direct_upload: false # Use Object Storage directly for uploads instead of background uploads if enabled (Default: false). + #background_upload: true # Temporary option to limit automatic upload (Default: true). + #proxy_download: false # Passthrough all downloads via GitLab instead of using Redirects to Object Storage. + connection: + ## + ## If the provider is AWS S3, uncomment the following + ## + #provider: AWS + #region: us-east-1 + #aws_access_key_id: AWS_ACCESS_KEY_ID + #aws_secret_access_key: AWS_SECRET_ACCESS_KEY + ## + ## If the provider is other than AWS (an S3-compatible one), uncomment the following + ## + #host: 's3.amazonaws.com' # default: s3.amazonaws.com. + #aws_signature_version: 4 # For creation of signed URLs. Set to 2 if provider does not support v4. + #endpoint: 'https://s3.amazonaws.com' # Useful for S3-compliant services such as DigitalOcean Spaces. + #path_style: false # If true, use 'host/bucket_name/object' instead of 'bucket_name.host/object'. + ``` + +1. [Restart GitLab] for the changes to take effect. + +[reconfigure gitlab]: restart_gitlab.md#omnibus-gitlab-reconfigure "How to reconfigure Omnibus GitLab" +[restart gitlab]: restart_gitlab.md#omnibus-gitlab-reconfigure "How to reconfigure Omnibus GitLab" diff --git a/doc/administration/geo/disaster_recovery/background_verification.md b/doc/administration/geo/disaster_recovery/background_verification.md new file mode 100644 index 00000000000..d4c8c2d3624 --- /dev/null +++ b/doc/administration/geo/disaster_recovery/background_verification.md @@ -0,0 +1,182 @@ +# Automatic background verification **[PREMIUM ONLY]** + +NOTE: **Note:** +Automatic background verification of repositories and wikis was added in +GitLab EE 10.6 but is enabled by default only on GitLab EE 11.1. You can +disable or enable this feature manually by following +[these instructions](#disabling-or-enabling-the-automatic-background-verification). + +Automatic background verification ensures that the transferred data matches a +calculated checksum. If the checksum of the data on the **primary** node matches checksum of the +data on the **secondary** node, the data transferred successfully. Following a planned failover, +any corrupted data may be **lost**, depending on the extent of the corruption. + +If verification fails on the **primary** node, this indicates that Geo is +successfully replicating a corrupted object; restore it from backup or remove it +it from the **primary** node to resolve the issue. + +If verification succeeds on the **primary** node but fails on the **secondary** node, +this indicates that the object was corrupted during the replication process. +Geo actively try to correct verification failures marking the repository to +be resynced with a backoff period. If you want to reset the verification for +these failures, so you should follow [these instructions][reset-verification]. + +If verification is lagging significantly behind replication, consider giving +the node more time before scheduling a planned failover. + +## Disabling or enabling the automatic background verification + +Run the following commands in a Rails console on the **primary** node: + +```sh +gitlab-rails console +``` + +To check if automatic background verification is enabled: + +```ruby +Gitlab::Geo.repository_verification_enabled? +``` + +To disable automatic background verification: + +```ruby +Feature.disable('geo_repository_verification') +``` + +To enable automatic background verification: + +```ruby +Feature.enable('geo_repository_verification') +``` + +## Repository verification + +Navigate to the **Admin Area > Geo** dashboard on the **primary** node and expand +the **Verification information** tab for that node to view automatic checksumming +status for repositories and wikis. Successes are shown in green, pending work +in grey, and failures in red. + + + +Navigate to the **Admin Area > Geo** dashboard on the **secondary** node and expand +the **Verification information** tab for that node to view automatic verification +status for repositories and wikis. As with checksumming, successes are shown in +green, pending work in grey, and failures in red. + + + +## Using checksums to compare Geo nodes + +To check the health of Geo **secondary** nodes, we use a checksum over the list of +Git references and their values. The checksum includes `HEAD`, `heads`, `tags`, +`notes`, and GitLab-specific references to ensure true consistency. If two nodes +have the same checksum, then they definitely hold the same references. We compute +the checksum for every node after every update to make sure that they are all +in sync. + +## Repository re-verification + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/8550) in GitLab Enterprise Edition 11.6. Available in [GitLab Premium](https://about.gitlab.com/pricing/). + +Due to bugs or transient infrastructure failures, it is possible for Git +repositories to change unexpectedly without being marked for verification. +Geo constantly reverifies the repositories to ensure the integrity of the +data. The default and recommended re-verification interval is 7 days, though +an interval as short as 1 day can be set. Shorter intervals reduce risk but +increase load and vice versa. + +Navigate to the **Admin Area > Geo** dashboard on the **primary** node, and +click the **Edit** button for the **primary** node to customize the minimum +re-verification interval: + + + +The automatic background re-verification is enabled by default, but you can +disable if you need. Run the following commands in a Rails console on the +**primary** node: + +```sh +gitlab-rails console +``` + +To disable automatic background re-verification: + +```ruby +Feature.disable('geo_repository_reverification') +``` + +To enable automatic background re-verification: + +```ruby +Feature.enable('geo_repository_reverification') +``` + +## Reset verification for projects where verification has failed + +Geo actively try to correct verification failures marking the repository to +be resynced with a backoff period. If you want to reset them manually, this +rake task marks projects where verification has failed or the checksum mismatch +to be resynced without the backoff period: + +For repositories: + +```sh +sudo gitlab-rake geo:verification:repository:reset +``` + +For wikis: + +```sh +sudo gitlab-rake geo:verification:wiki:reset +``` + +## Reconcile differences with checksum mismatches + +If the **primary** and **secondary** nodes have a checksum verification mismatch, the cause may not be apparent. To find the cause of a checksum mismatch: + +1. Navigate to the **Admin Area > Projects** dashboard on the **primary** node, find the + project that you want to check the checksum differences and click on the + **Edit** button: +  + +1. On the project admin page get the **Gitaly storage name**, and **Gitaly relative path**: +  + +1. Navigate to the project's repository directory on both **primary** and **secondary** nodes (the path is usually `/var/opt/gitlab/git-data/repositories`). Note that if `git_data_dirs` is customized, check the directory layout on your server to be sure. + + ```sh + cd /var/opt/gitlab/git-data/repositories + ``` + +1. Run the following command on the **primary** node, redirecting the output to a file: + + ```sh + git show-ref --head | grep -E "HEAD|(refs/(heads|tags|keep-around|merge-requests|environments|notes)/)" > primary-node-refs + ``` + +1. Run the following command on the **secondary** node, redirecting the output to a file: + + ```sh + git show-ref --head | grep -E "HEAD|(refs/(heads|tags|keep-around|merge-requests|environments|notes)/)" > secondary-node-refs + ``` + +1. Copy the files from the previous steps on the same system, and do a diff between the contents: + + ```sh + diff primary-node-refs secondary-node-refs + ``` + +## Current limitations + +Until [issue #5064][ee-5064] is completed, background verification doesn't cover +CI job artifacts and traces, LFS objects, or user uploads in file storage. +Verify their integrity manually by following [these instructions][foreground-verification] +on both nodes, and comparing the output between them. + +Data in object storage is **not verified**, as the object store is responsible +for ensuring the integrity of the data. + +[reset-verification]: background_verification.md#reset-verification-for-projects-where-verification-has-failed +[foreground-verification]: ../../raketasks/check.md +[ee-5064]: https://gitlab.com/gitlab-org/gitlab-ee/issues/5064 diff --git a/doc/administration/geo/disaster_recovery/bring_primary_back.md b/doc/administration/geo/disaster_recovery/bring_primary_back.md new file mode 100644 index 00000000000..f4d31a98080 --- /dev/null +++ b/doc/administration/geo/disaster_recovery/bring_primary_back.md @@ -0,0 +1,61 @@ +# Bring a demoted primary node back online **[PREMIUM ONLY]** + +After a failover, it is possible to fail back to the demoted **primary** node to +restore your original configuration. This process consists of two steps: + +1. Making the old **primary** node a **secondary** node. +1. Promoting a **secondary** node to a **primary** node. + +CAUTION: **Caution:** +If you have any doubts about the consistency of the data on this node, we recommend setting it up from scratch. + +## Configure the former **primary** node to be a **secondary** node + +Since the former **primary** node will be out of sync with the current **primary** node, the first step is to bring the former **primary** node up to date. Note, deletion of data stored on disk like +repositories and uploads will not be replayed when bringing the former **primary** node back +into sync, which may result in increased disk usage. +Alternatively, you can [set up a new **secondary** GitLab instance][setup-geo] to avoid this. + +To bring the former **primary** node up to date: + +1. SSH into the former **primary** node that has fallen behind. +1. Make sure all the services are up: + + ```sh + sudo gitlab-ctl start + ``` + + > **Note 1:** If you [disabled the **primary** node permanently][disaster-recovery-disable-primary], + > you need to undo those steps now. For Debian/Ubuntu you just need to run + > `sudo systemctl enable gitlab-runsvdir`. For CentOS 6, you need to install + > the GitLab instance from scratch and set it up as a **secondary** node by + > following [Setup instructions][setup-geo]. In this case, you don't need to follow the next step. + > + > **Note 2:** If you [changed the DNS records](index.md#step-4-optional-updating-the-primary-domain-dns-record) + > for this node during disaster recovery procedure you may need to [block + > all the writes to this node](https://gitlab.com/gitlab-org/gitlab-ee/blob/master/doc/gitlab-geo/planned-failover.md#block-primary-traffic) + > during this procedure. + +1. [Setup database replication][database-replication]. Note that in this + case, **primary** node refers to the current **primary** node, and **secondary** node refers to the + former **primary** node. + +If you have lost your original **primary** node, follow the +[setup instructions][setup-geo] to set up a new **secondary** node. + +## Promote the **secondary** node to **primary** node + +When the initial replication is complete and the **primary** node and **secondary** node are +closely in sync, you can do a [planned failover]. + +## Restore the **secondary** node + +If your objective is to have two nodes again, you need to bring your **secondary** +node back online as well by repeating the first step +([configure the former **primary** node to be a **secondary** node](#configure-the-former-primary-node-to-be-a-secondary-node)) +for the **secondary** node. + +[setup-geo]: ../replication/index.md#setup-instructions +[database-replication]: ../replication/database.md +[disaster-recovery-disable-primary]: index.md#step-2-permanently-disable-the-primary-node +[planned failover]: planned_failover.md diff --git a/doc/administration/geo/disaster_recovery/img/checksum-differences-admin-project-page.png b/doc/administration/geo/disaster_recovery/img/checksum-differences-admin-project-page.png Binary files differnew file mode 100644 index 00000000000..fd51523104b --- /dev/null +++ b/doc/administration/geo/disaster_recovery/img/checksum-differences-admin-project-page.png diff --git a/doc/administration/geo/disaster_recovery/img/checksum-differences-admin-projects.png b/doc/administration/geo/disaster_recovery/img/checksum-differences-admin-projects.png Binary files differnew file mode 100644 index 00000000000..b2a6da69d3d --- /dev/null +++ b/doc/administration/geo/disaster_recovery/img/checksum-differences-admin-projects.png diff --git a/doc/administration/geo/disaster_recovery/img/replication-status.png b/doc/administration/geo/disaster_recovery/img/replication-status.png Binary files differnew file mode 100644 index 00000000000..d7085927c75 --- /dev/null +++ b/doc/administration/geo/disaster_recovery/img/replication-status.png diff --git a/doc/administration/geo/disaster_recovery/img/reverification-interval.png b/doc/administration/geo/disaster_recovery/img/reverification-interval.png Binary files differnew file mode 100644 index 00000000000..ad4597a4f49 --- /dev/null +++ b/doc/administration/geo/disaster_recovery/img/reverification-interval.png diff --git a/doc/administration/geo/disaster_recovery/img/verification-status-primary.png b/doc/administration/geo/disaster_recovery/img/verification-status-primary.png Binary files differnew file mode 100644 index 00000000000..2503408ec5d --- /dev/null +++ b/doc/administration/geo/disaster_recovery/img/verification-status-primary.png diff --git a/doc/administration/geo/disaster_recovery/img/verification-status-secondary.png b/doc/administration/geo/disaster_recovery/img/verification-status-secondary.png Binary files differnew file mode 100644 index 00000000000..462274d8b14 --- /dev/null +++ b/doc/administration/geo/disaster_recovery/img/verification-status-secondary.png diff --git a/doc/administration/geo/disaster_recovery/index.md b/doc/administration/geo/disaster_recovery/index.md new file mode 100644 index 00000000000..71dc797f281 --- /dev/null +++ b/doc/administration/geo/disaster_recovery/index.md @@ -0,0 +1,322 @@ +# Disaster Recovery **[PREMIUM ONLY]** + +Geo replicates your database, your Git repositories, and few other assets. +We will support and replicate more data in the future, that will enable you to +failover with minimal effort, in a disaster situation. + +See [Geo current limitations][geo-limitations] for more information. + +CAUTION: **Warning:** +Disaster recovery for multi-secondary configurations is in **Alpha**. +For the latest updates, check the multi-secondary [Disaster Recovery epic][gitlab-org&65]. + +## Promoting a **secondary** Geo node in single-secondary configurations + +We don't currently provide an automated way to promote a Geo replica and do a +failover, but you can do it manually if you have `root` access to the machine. + +This process promotes a **secondary** Geo node to a **primary** node. To regain +geographic redundancy as quickly as possible, you should add a new **secondary** node +immediately after following these instructions. + +### Step 1. Allow replication to finish if possible + +If the **secondary** node is still replicating data from the **primary** node, follow +[the planned failover docs][planned-failover] as closely as possible in +order to avoid unnecessary data loss. + +### Step 2. Permanently disable the **primary** node + +CAUTION: **Warning:** +If the **primary** node goes offline, there may be data saved on the **primary** node +that has not been replicated to the **secondary** node. This data should be treated +as lost if you proceed. + +If an outage on the **primary** node happens, you should do everything possible to +avoid a split-brain situation where writes can occur in two different GitLab +instances, complicating recovery efforts. So to prepare for the failover, we +must disable the **primary** node. + +1. SSH into the **primary** node to stop and disable GitLab, if possible: + + ```sh + sudo gitlab-ctl stop + ``` + + Prevent GitLab from starting up again if the server unexpectedly reboots: + + ```sh + sudo systemctl disable gitlab-runsvdir + ``` + + > **CentOS only**: In CentOS 6 or older, there is no easy way to prevent GitLab from being + > started if the machine reboots isn't available (see [gitlab-org/omnibus-gitlab#3058]). + > It may be safest to uninstall the GitLab package completely: + + ```sh + yum remove gitlab-ee + ``` + + > **Ubuntu 14.04 LTS**: If you are using an older version of Ubuntu + > or any other distro based on the Upstart init system, you can prevent GitLab + > from starting if the machine reboots by doing the following: + + ```sh + initctl stop gitlab-runsvvdir + echo 'manual' > /etc/init/gitlab-runsvdir.override + initctl reload-configuration + ``` + +1. If you do not have SSH access to the **primary** node, take the machine offline and + prevent it from rebooting by any means at your disposal. + Since there are many ways you may prefer to accomplish this, we will avoid a + single recommendation. You may need to: + - Reconfigure the load balancers. + - Change DNS records (e.g., point the primary DNS record to the **secondary** + node in order to stop usage of the **primary** node). + - Stop the virtual servers. + - Block traffic through a firewall. + - Revoke object storage permissions from the **primary** node. + - Physically disconnect a machine. + +1. If you plan to + [update the primary domain DNS record](#step-4-optional-updating-the-primary-domain-dns-record), + you may wish to lower the TTL now to speed up propagation. + +### Step 3. Promoting a **secondary** node + +NOTE: **Note:** +A new **secondary** should not be added at this time. If you want to add a new +**secondary**, do this after you have completed the entire process of promoting +the **secondary** to the **primary**. + +#### Promoting a **secondary** node running on a single machine + +1. SSH in to your **secondary** node and login as root: + + ```sh + sudo -i + ``` + +1. Edit `/etc/gitlab/gitlab.rb` to reflect its new status as **primary** by + removing any lines that enabled the `geo_secondary_role`: + + ```ruby + ## In pre-11.5 documentation, the role was enabled as follows. Remove this line. + geo_secondary_role['enable'] = true + + ## In 11.5+ documentation, the role was enabled as follows. Remove this line. + roles ['geo_secondary_role'] + ``` + +1. Promote the **secondary** node to the **primary** node. Execute: + + ```sh + gitlab-ctl promote-to-primary-node + ``` + +1. Verify you can connect to the newly promoted **primary** node using the URL used + previously for the **secondary** node. +1. If successful, the **secondary** node has now been promoted to the **primary** node. + +#### Promoting a **secondary** node with HA + +The `gitlab-ctl promote-to-primary-node` command cannot be used yet in +conjunction with High Availability or with multiple machines, as it can only +perform changes on a **secondary** with only a single machine. Instead, you must +do this manually. + +1. SSH in to the database node in the **secondary** and trigger PostgreSQL to + promote to read-write: + + ```bash + sudo gitlab-pg-ctl promote + ``` + +1. Edit `/etc/gitlab/gitlab.rb` on every machine in the **secondary** to + reflect its new status as **primary** by removing any lines that enabled the + `geo_secondary_role`: + + ```ruby + ## In pre-11.5 documentation, the role was enabled as follows. Remove this line. + geo_secondary_role['enable'] = true + + ## In 11.5+ documentation, the role was enabled as follows. Remove this line. + roles ['geo_secondary_role'] + ``` + + After making these changes [Reconfigure GitLab](../../restart_gitlab.md#omnibus-gitlab-reconfigure) each + machine so the changes take effect. + +1. Promote the **secondary** to **primary**. SSH into a single application + server and execute: + + ```bash + sudo gitlab-rake geo:set_secondary_as_primary + ``` + +1. Verify you can connect to the newly promoted **primary** using the URL used + previously for the **secondary**. +1. Success! The **secondary** has now been promoted to **primary**. + +### Step 4. (Optional) Updating the primary domain DNS record + +Updating the DNS records for the primary domain to point to the **secondary** node +will prevent the need to update all references to the primary domain to the +secondary domain, like changing Git remotes and API URLs. + +1. SSH into the **secondary** node and login as root: + + ```sh + sudo -i + ``` + +1. Update the primary domain's DNS record. After updating the primary domain's + DNS records to point to the **secondary** node, edit `/etc/gitlab/gitlab.rb` on the + **secondary** node to reflect the new URL: + + ```ruby + # Change the existing external_url configuration + external_url 'https://<new_external_url>' + ``` + + NOTE: **Note** + Changing `external_url` won't prevent access via the old secondary URL, as + long as the secondary DNS records are still intact. + +1. Reconfigure the **secondary** node for the change to take effect: + + ```sh + gitlab-ctl reconfigure + ``` + +1. Execute the command below to update the newly promoted **primary** node URL: + + ```sh + gitlab-rake geo:update_primary_node_url + ``` + + This command will use the changed `external_url` configuration defined + in `/etc/gitlab/gitlab.rb`. + +1. Verify you can connect to the newly promoted **primary** using its URL. + If you updated the DNS records for the primary domain, these changes may + not have yet propagated depending on the previous DNS records TTL. + +### Step 5. (Optional) Add **secondary** Geo node to a promoted **primary** node + +Promoting a **secondary** node to **primary** node using the process above does not enable +Geo on the new **primary** node. + +To bring a new **secondary** node online, follow the [Geo setup instructions][setup-geo]. + +### Step 6. (Optional) Removing the secondary's tracking database + +Every **secondary** has a special tracking database that is used to save the status of the synchronization of all the items from the **primary**. +Because the **secondary** is already promoted, that data in the tracking database is no longer required. + +The data can be removed with the following command: + +```sh +sudo rm -rf /var/opt/gitlab/geo-postgresql +``` + +## Promoting secondary Geo replica in multi-secondary configurations + +If you have more than one **secondary** node and you need to promote one of them, we suggest you follow +[Promoting a **secondary** Geo node in single-secondary configurations](#promoting-a-secondary-geo-node-in-single-secondary-configurations) +and after that you also need two extra steps. + +### Step 1. Prepare the new **primary** node to serve one or more **secondary** nodes + +1. SSH into the new **primary** node and login as root: + + ```sh + sudo -i + ``` + +1. Edit `/etc/gitlab/gitlab.rb` + + ```ruby + ## Enable a Geo Primary role (if you haven't yet) + roles ['geo_primary_role'] + + ## + # Allow PostgreSQL client authentication from the primary and secondary IPs. These IPs may be + # public or VPC addresses in CIDR format, for example ['198.51.100.1/32', '198.51.100.2/32'] + ## + postgresql['md5_auth_cidr_addresses'] = ['<primary_node_ip>/32', '<secondary_node_ip>/32'] + + # Every secondary server needs to have its own slot so specify the number of secondary nodes you're going to have + postgresql['max_replication_slots'] = 1 + + ## + ## Disable automatic database migrations temporarily + ## (until PostgreSQL is restarted and listening on the private address). + ## + gitlab_rails['auto_migrate'] = false + + ``` + + (For more details about these settings you can read [Configure the primary server][configure-the-primary-server]) + +1. Save the file and reconfigure GitLab for the database listen changes and + the replication slot changes to be applied. + + ```sh + gitlab-ctl reconfigure + ``` + + Restart PostgreSQL for its changes to take effect: + + ```sh + gitlab-ctl restart postgresql + ``` + +1. Re-enable migrations now that PostgreSQL is restarted and listening on the + private address. + + Edit `/etc/gitlab/gitlab.rb` and **change** the configuration to `true`: + + ```ruby + gitlab_rails['auto_migrate'] = true + ``` + + Save the file and reconfigure GitLab: + + ```sh + gitlab-ctl reconfigure + ``` + +### Step 2. Initiate the replication process + +Now we need to make each **secondary** node listen to changes on the new **primary** node. To do that you need +to [initiate the replication process][initiate-the-replication-process] again but this time +for another **primary** node. All the old replication settings will be overwritten. + +## Troubleshooting + +### I followed the disaster recovery instructions and now two-factor auth is broken! + +The setup instructions for Geo prior to 10.5 failed to replicate the +`otp_key_base` secret, which is used to encrypt the two-factor authentication +secrets stored in the database. If it differs between **primary** and **secondary** +nodes, users with two-factor authentication enabled won't be able to log in +after a failover. + +If you still have access to the old **primary** node, you can follow the +instructions in the +[Upgrading to GitLab 10.5][updating-geo] +section to resolve the error. Otherwise, the secret is lost and you'll need to +[reset two-factor authentication for all users][sec-tfa]. + +[gitlab-org&65]: https://gitlab.com/groups/gitlab-org/-/epics/65 +[geo-limitations]: ../replication/index.md#current-limitations +[planned-failover]: planned_failover.md +[setup-geo]: ../replication/index.md#setup-instructions +[updating-geo]: ../replication/updating_the_geo_nodes.md#upgrading-to-gitlab-105 +[sec-tfa]: ../../../security/two_factor_authentication.md#disabling-2fa-for-everyone +[gitlab-org/omnibus-gitlab#3058]: https://gitlab.com/gitlab-org/omnibus-gitlab/issues/3058 +[gitlab-org/gitlab-ee#4284]: https://gitlab.com/gitlab-org/gitlab-ee/issues/4284 +[initiate-the-replication-process]: ../replication/database.html#step-3-initiate-the-replication-process +[configure-the-primary-server]: ../replication/database.html#step-1-configure-the-primary-server diff --git a/doc/administration/geo/disaster_recovery/planned_failover.md b/doc/administration/geo/disaster_recovery/planned_failover.md new file mode 100644 index 00000000000..b8071b5993f --- /dev/null +++ b/doc/administration/geo/disaster_recovery/planned_failover.md @@ -0,0 +1,227 @@ +# Disaster recovery for planned failover **[PREMIUM ONLY]** + +The primary use-case of Disaster Recovery is to ensure business continuity in +the event of unplanned outage, but it can be used in conjunction with a planned +failover to migrate your GitLab instance between regions without extended +downtime. + +As replication between Geo nodes is asynchronous, a planned failover requires +a maintenance window in which updates to the **primary** node are blocked. The +length of this window is determined by your replication capacity - once the +**secondary** node is completely synchronized with the **primary** node, the failover can occur without +data loss. + +This document assumes you already have a fully configured, working Geo setup. +Please read it and the [Disaster Recovery][disaster-recovery] failover +documentation in full before proceeding. Planned failover is a major operation, +and if performed incorrectly, there is a high risk of data loss. Consider +rehearsing the procedure until you are comfortable with the necessary steps and +have a high degree of confidence in being able to perform them accurately. + +## Not all data is automatically replicated + +If you are using any GitLab features that Geo [doesn't support][limitations], +you must make separate provisions to ensure that the **secondary** node has an +up-to-date copy of any data associated with that feature. This may extend the +required scheduled maintenance period significantly. + +A common strategy for keeping this period as short as possible for data stored +in files is to use `rsync` to transfer the data. An initial `rsync` can be +performed ahead of the maintenance window; subsequent `rsync`s (including a +final transfer inside the maintenance window) will then transfer only the +*changes* between the **primary** node and the **secondary** nodes. + +Repository-centric strategies for using `rsync` effectively can be found in the +[moving repositories][moving-repositories] documentation; these strategies can +be adapted for use with any other file-based data, such as GitLab Pages (to +be found in `/var/opt/gitlab/gitlab-rails/shared/pages` if using Omnibus). + +## Pre-flight checks + +Follow these steps before scheduling a planned failover to ensure the process +will go smoothly. + +### Object storage + +Some classes of non-repository data can use object storage in preference to +file storage. Geo [does not replicate data in object storage](../replication/object_storage.md), +leaving that task up to the object store itself. For a planned failover, this +means you can decouple the replication of this data from the failover of the +GitLab service. + +If you're already using object storage, simply verify that your **secondary** +node has access to the same data as the **primary** node - they must either they share the +same object storage configuration, or the **secondary** node should be configured to +access a [geographically-replicated][os-repl] copy provided by the object store +itself. + +If you have a large GitLab installation or cannot tolerate downtime, consider +[migrating to Object Storage][os-conf] **before** scheduling a planned failover. +Doing so reduces both the length of the maintenance window, and the risk of data +loss as a result of a poorly executed planned failover. + +### Review the configuration of each **secondary** node + +Database settings are automatically replicated to the **secondary** node, but the +`/etc/gitlab/gitlab.rb` file must be set up manually, and differs between +nodes. If features such as Mattermost, OAuth or LDAP integration are enabled +on the **primary** node but not the **secondary** node, they will be lost during failover. + +Review the `/etc/gitlab/gitlab.rb` file for both nodes and ensure the **secondary** node +supports everything the **primary** node does **before** scheduling a planned failover. + +### Run system checks + +Run the following on both **primary** and **secondary** nodes: + +```sh +gitlab-rake gitlab:check +gitlab-rake gitlab:geo:check +``` + +If any failures are reported on either node, they should be resolved **before** +scheduling a planned failover. + +### Check that secrets match between nodes + +The SSH host keys and `/etc/gitlab/gitlab-secrets.json` files should be +identical on all nodes. Check this by running the following on all nodes and +comparing the output: + +```sh +sudo sha256sum /etc/ssh/ssh_host* /etc/gitlab/gitlab-secrets.json +``` + +If any files differ, replace the content on the **secondary** node with the +content from the **primary** node. + +### Ensure Geo replication is up-to-date + +The maintenance window won't end until Geo replication and verification is +completely finished. To keep the window as short as possible, you should +ensure these processes are close to 100% as possible during active use. + +Navigate to the **Admin Area > Geo** dashboard on the **secondary** node to +review status. Replicated objects (shown in green) should be close to 100%, +and there should be no failures (shown in red). If a large proportion of +objects aren't yet replicated (shown in grey), consider giving the node more +time to complete + + + +If any objects are failing to replicate, this should be investigated before +scheduling the maintenance window. Following a planned failover, anything that +failed to replicate will be **lost**. + +You can use the [Geo status API](../../../api/geo_nodes.md#retrieve-project-sync-or-verification-failures-that-occurred-on-the-current-node) to review failed objects and +the reasons for failure. + +A common cause of replication failures is the data being missing on the +**primary** node - you can resolve these failures by restoring the data from backup, +or removing references to the missing data. + +### Verify the integrity of replicated data + +This [content was moved to another location][background-verification]. + +### Notify users of scheduled maintenance + +On the **primary** node, navigate to **Admin Area > Messages**, add a broadcast +message. You can check under **Admin Area > Geo** to estimate how long it +will take to finish syncing. An example message would be: + +> A scheduled maintenance will take place at XX:XX UTC. We expect it to take +> less than 1 hour. + +## Prevent updates to the **primary** node + +Until a [read-only mode][ce-19739] is implemented, updates must be prevented +from happening manually. Note that your **secondary** node still needs read-only +access to the **primary** node during the maintenance window. + +1. At the scheduled time, using your cloud provider or your node's firewall, block + all HTTP, HTTPS and SSH traffic to/from the **primary** node, **except** for your IP and + the **secondary** node's IP. + + For instance, you might run the following commands on the server(s) making up your **primary** node: + + ```sh + sudo iptables -A INPUT -p tcp -s <secondary_node_ip> --destination-port 22 -j ACCEPT + sudo iptables -A INPUT -p tcp -s <your_ip> --destination-port 22 -j ACCEPT + sudo iptables -A INPUT --destination-port 22 -j REJECT + + sudo iptables -A INPUT -p tcp -s <secondary_node_ip> --destination-port 80 -j ACCEPT + sudo iptables -A INPUT -p tcp -s <your_ip> --destination-port 80 -j ACCEPT + sudo iptables -A INPUT --tcp-dport 80 -j REJECT + + sudo iptables -A INPUT -p tcp -s <secondary_node_ip> --destination-port 443 -j ACCEPT + sudo iptables -A INPUT -p tcp -s <your_ip> --destination-port 443 -j ACCEPT + sudo iptables -A INPUT --tcp-dport 443 -j REJECT + ``` + + From this point, users will be unable to view their data or make changes on the + **primary** node. They will also be unable to log in to the **secondary** node. + However, existing sessions will work for the remainder of the maintenance period, and + public data will be accessible throughout. + +1. Verify the **primary** node is blocked to HTTP traffic by visiting it in browser via + another IP. The server should refuse connection. + +1. Verify the **primary** node is blocked to Git over SSH traffic by attempting to pull an + existing Git repository with an SSH remote URL. The server should refuse + connection. + +1. Disable non-Geo periodic background jobs on the primary node by navigating + to **Admin Area > Monitoring > Background Jobs > Cron** , pressing `Disable All`, + and then pressing `Enable` for the `geo_sidekiq_cron_config_worker` cron job. + This job will re-enable several other cron jobs that are essential for planned + failover to complete successfully. + +## Finish replicating and verifying all data + +1. If you are manually replicating any data not managed by Geo, trigger the + final replication process now. +1. On the **primary** node, navigate to **Admin Area > Monitoring > Background Jobs > Queues** + and wait for all queues except those with `geo` in the name to drop to 0. + These queues contain work that has been submitted by your users; failing over + before it is completed will cause the work to be lost. +1. On the **primary** node, navigate to **Admin Area > Geo** and wait for the + following conditions to be true of the **secondary** node you are failing over to: + - All replication meters to each 100% replicated, 0% failures. + - All verification meters reach 100% verified, 0% failures. + - Database replication lag is 0ms. + - The Geo log cursor is up to date (0 events behind). + +1. On the **secondary** node, navigate to **Admin Area > Monitoring > Background Jobs > Queues** + and wait for all the `geo` queues to drop to 0 queued and 0 running jobs. +1. On the **secondary** node, use [these instructions][foreground-verification] + to verify the integrity of CI artifacts, LFS objects and uploads in file + storage. + +At this point, your **secondary** node will contain an up-to-date copy of everything the +**primary** node has, meaning nothing will be lost when you fail over. + +## Promote the **secondary** node + +Finally, follow the [Disaster Recovery docs][disaster-recovery] to promote the +**secondary** node to a **primary** node. This process will cause a brief outage on the **secondary** node, and users may need to log in again. + +Once it is completed, the maintenance window is over! Your new **primary** node will now +begin to diverge from the old one. If problems do arise at this point, failing +back to the old **primary** node [is possible][bring-primary-back], but likely to result +in the loss of any data uploaded to the new primary in the meantime. + +Don't forget to remove the broadcast message after failover is complete. + +[bring-primary-back]: bring_primary_back.md +[ce-19739]: https://gitlab.com/gitlab-org/gitlab-ce/issues/19739 +[container-registry]: ../replication/container_registry.md +[disaster-recovery]: index.md +[ee-4930]: https://gitlab.com/gitlab-org/gitlab-ee/issues/4930 +[ee-5064]: https://gitlab.com/gitlab-org/gitlab-ee/issues/5064 +[foreground-verification]: ../../raketasks/check.md +[background-verification]: background_verification.md +[limitations]: ../replication/index.md#current-limitations +[moving-repositories]: ../../operations/moving_repositories.md +[os-conf]: ../replication/object_storage.md#configuration +[os-repl]: ../replication/object_storage.md#replication diff --git a/doc/administration/geo/replication/configuration.md b/doc/administration/geo/replication/configuration.md new file mode 100644 index 00000000000..3d4f69d3abe --- /dev/null +++ b/doc/administration/geo/replication/configuration.md @@ -0,0 +1,308 @@ +# Geo configuration **[PREMIUM ONLY]** + +## Configuring a new **secondary** node + +NOTE: **Note:** +This is the final step in setting up a **secondary** Geo node. Stages of the +setup process must be completed in the documented order. +Before attempting the steps in this stage, [complete all prior stages][setup-geo-omnibus]. + +The basic steps of configuring a **secondary** node are to: + +- Replicate required configurations between the **primary** node and the **secondary** nodes. +- Configure a tracking database on each **secondary** node. +- Start GitLab on each **secondary** node. + +You are encouraged to first read through all the steps before executing them +in your testing/production environment. + +> **Notes:** +> - **Do not** setup any custom authentication for the **secondary** nodes. This will be + handled by the **primary** node. +> - Any change that requires access to the **Admin Area** needs to be done in the + **primary** node because the **secondary** node is a read-only replica. + +### Step 1. Manually replicate secret GitLab values + +GitLab stores a number of secret values in the `/etc/gitlab/gitlab-secrets.json` +file which *must* be the same on all nodes. Until there is +a means of automatically replicating these between nodes (see issue [gitlab-org/gitlab-ee#3789]), +they must be manually replicated to the **secondary** node. + +1. SSH into the **primary** node, and execute the command below: + + ```sh + sudo cat /etc/gitlab/gitlab-secrets.json + ``` + + This will display the secrets that need to be replicated, in JSON format. + +1. SSH into the **secondary** node and login as the `root` user: + + ```sh + sudo -i + ``` + +1. Make a backup of any existing secrets: + + ```sh + mv /etc/gitlab/gitlab-secrets.json /etc/gitlab/gitlab-secrets.json.`date +%F` + ``` + +1. Copy `/etc/gitlab/gitlab-secrets.json` from the **primary** node to the **secondary** node, or + copy-and-paste the file contents between nodes: + + ```sh + sudo editor /etc/gitlab/gitlab-secrets.json + + # paste the output of the `cat` command you ran on the primary + # save and exit + ``` + +1. Ensure the file permissions are correct: + + ```sh + chown root:root /etc/gitlab/gitlab-secrets.json + chmod 0600 /etc/gitlab/gitlab-secrets.json + ``` + +1. Reconfigure the **secondary** node for the change to take effect: + + ```sh + gitlab-ctl reconfigure + gitlab-ctl restart + ``` + +### Step 2. Manually replicate the **primary** node's SSH host keys + +GitLab integrates with the system-installed SSH daemon, designating a user +(typically named git) through which all access requests are handled. + +In a [Disaster Recovery] situation, GitLab system +administrators will promote a **secondary** node to the **primary** node. DNS records for the +**primary** domain should also be updated to point to the new **primary** node +(previously a **secondary** node). Doing so will avoid the need to update Git remotes and API URLs. + +This will cause all SSH requests to the newly promoted **primary** node to +fail due to SSH host key mismatch. To prevent this, the primary SSH host +keys must be manually replicated to the **secondary** node. + +1. SSH into the **secondary** node and login as the `root` user: + + ```sh + sudo -i + ``` + +1. Make a backup of any existing SSH host keys: + + ```sh + find /etc/ssh -iname ssh_host_* -exec cp {} {}.backup.`date +%F` \; + ``` + +1. Copy OpenSSH host keys from the **primary** node: + + If you can access your **primary** node using the **root** user: + + ```sh + # Run this from the secondary node, change `<primary_node_fqdn>` for the IP or FQDN of the server + scp root@<primary_node_fqdn>:/etc/ssh/ssh_host_*_key* /etc/ssh + ``` + + If you only have access through a user with **sudo** privileges: + + ```sh + # Run this from your primary node: + sudo tar --transform 's/.*\///g' -zcvf ~/geo-host-key.tar.gz /etc/ssh/ssh_host_*_key* + + # Run this from your secondary node: + scp <user_with_sudo>@<primary_node_fqdn>:geo-host-key.tar.gz . + tar zxvf ~/geo-host-key.tar.gz -C /etc/ssh + ``` + +1. On your **secondary** node, ensure the file permissions are correct: + + ```sh + chown root:root /etc/ssh/ssh_host_*_key* + chmod 0600 /etc/ssh/ssh_host_*_key* + ``` + +1. To verify key fingerprint matches, execute the following command on both nodes: + + ```sh + for file in /etc/ssh/ssh_host_*_key; do ssh-keygen -lf $file; done + ``` + + You should get an output similar to this one and they should be identical on both nodes: + + ```sh + 1024 SHA256:FEZX2jQa2bcsd/fn/uxBzxhKdx4Imc4raXrHwsbtP0M root@serverhostname (DSA) + 256 SHA256:uw98R35Uf+fYEQ/UnJD9Br4NXUFPv7JAUln5uHlgSeY root@serverhostname (ECDSA) + 256 SHA256:sqOUWcraZQKd89y/QQv/iynPTOGQxcOTIXU/LsoPmnM root@serverhostname (ED25519) + 2048 SHA256:qwa+rgir2Oy86QI+PZi/QVR+MSmrdrpsuH7YyKknC+s root@serverhostname (RSA) + ``` + +1. Verify that you have the correct public keys for the existing private keys: + + ```sh + # This will print the fingerprint for private keys: + for file in /etc/ssh/ssh_host_*_key; do ssh-keygen -lf $file; done + + # This will print the fingerprint for public keys: + for file in /etc/ssh/ssh_host_*_key.pub; do ssh-keygen -lf $file; done + ``` + + NOTE: **Note**: + The output for private keys and public keys command should generate the same fingerprint. + +1. Restart sshd on your **secondary** node: + + ```sh + # Debian or Ubuntu installations + sudo service ssh reload + + # CentOS installations + sudo service sshd reload + ``` + +### Step 3. Add the **secondary** node + +1. Visit the **primary** node's **Admin Area > Geo** + (`/admin/geo/nodes`) in your browser. +1. Add the **secondary** node by providing its full URL. **Do NOT** check the + **This is a primary node** checkbox. +1. Optionally, choose which groups or storage shards should be replicated by the + **secondary** node. Leave blank to replicate all. Read more in + [selective synchronization](#selective-synchronization). +1. Click the **Add node** button. +1. SSH into your GitLab **secondary** server and restart the services: + + ```sh + gitlab-ctl restart + ``` + + Check if there are any common issue with your Geo setup by running: + + ```sh + gitlab-rake gitlab:geo:check + ``` + +1. SSH into your **primary** server and login as root to verify the + **secondary** node is reachable or there are any common issue with your Geo setup: + + ```sh + gitlab-rake gitlab:geo:check + ``` + +Once added to the admin panel and restarted, the **secondary** node will automatically start +replicating missing data from the **primary** node in a process known as **backfill**. +Meanwhile, the **primary** node will start to notify each **secondary** node of any changes, so +that the **secondary** node can act on those notifications immediately. + +Make sure the **secondary** node is running and accessible. +You can login to the **secondary** node with the same credentials as used for the **primary** node. + +### Step 4. Enabling Hashed Storage + +Using Hashed Storage significantly improves Geo replication. Project and group +renames no longer require synchronization between nodes. + +1. Visit the **primary** node's **Admin Area > Settings > Repository** + (`/admin/application_settings/repository`) in your browser. +1. In the **Repository storage** section, check **Use hashed storage paths for newly created and renamed projects**. + +### Step 5. (Optional) Configuring the **secondary** node to trust the **primary** node + +You can safely skip this step if your **primary** node uses a CA-issued HTTPS certificate. + +If your **primary** node is using a self-signed certificate for *HTTPS* support, you will +need to add that certificate to the **secondary** node's trust store. Retrieve the +certificate from the **primary** node and follow +[these instructions][omnibus-ssl] +on the **secondary** node. + +### Step 6. Enable Git access over HTTP/HTTPS + +Geo synchronizes repositories over HTTP/HTTPS, and therefore requires this clone +method to be enabled. Navigate to **Admin Area > Settings** +(`/admin/application_settings`) on the **primary** node, and set +`Enabled Git access protocols` to `Both SSH and HTTP(S)` or `Only HTTP(S)`. + +### Step 7. Verify proper functioning of the **secondary** node + +Your **secondary** node is now configured! + +You can login to the **secondary** node with the same credentials you used for the +**primary** node. Visit the **secondary** node's **Admin Area > Geo** +(`/admin/geo/nodes`) in your browser to check if it's correctly identified as a +**secondary** Geo node and if Geo is enabled. + +The initial replication, or 'backfill', will probably still be in progress. You +can monitor the synchronization process on each geo node from the **primary** +node's Geo Nodes dashboard in your browser. + + + +If your installation isn't working properly, check the +[troubleshooting document]. + +The two most obvious issues that can become apparent in the dashboard are: + +1. Database replication not working well. +1. Instance to instance notification not working. In that case, it can be + something of the following: + - You are using a custom certificate or custom CA (see the + [troubleshooting document]). + - The instance is firewalled (check your firewall rules). + +Please note that disabling a **secondary** node will stop the synchronization process. + +Please note that if `git_data_dirs` is customized on the **primary** node for multiple +repository shards you must duplicate the same configuration on each **secondary** node. + +Point your users to the ["Using a Geo Server" guide][using-geo]. + +Currently, this is what is synced: + +- Git repositories. +- Wikis. +- LFS objects. +- Issues, merge requests, snippets, and comment attachments. +- Users, groups, and project avatars. + +## Selective synchronization + +Geo supports selective synchronization, which allows admins to choose +which projects should be synchronized by **secondary** nodes. +A subset of projects can be chosen, either by group or by storage shard. The +former is ideal for replicating data belonging to a subset of users, while the +latter is more suited to progressively rolling out Geo to a large GitLab +instance. + +It is important to note that selective synchronization: + +1. Does not restrict permissions from **secondary** nodes. +1. Does not hide project metadata from **secondary** nodes. + - Since Geo currently relies on PostgreSQL replication, all project metadata + gets replicated to **secondary** nodes, but repositories that have not been + selected will be empty. +1. Does not reduce the number of events generated for the Geo event log. + - The **primary** node generates events as long as any **secondary** nodes are present. + Selective synchronization restrictions are implemented on the **secondary** nodes, + not the **primary** node. + +## Upgrading Geo + +See the [updating the Geo nodes document](updating_the_geo_nodes.md). + +## Troubleshooting + +See the [troubleshooting document](troubleshooting.md). + +[setup-geo-omnibus]: index.md#using-omnibus-gitlab +[Hashed Storage]: ../../repository_storage_types.md +[Disaster Recovery]: ../disaster_recovery/index.md +[gitlab-org/gitlab-ee#3789]: https://gitlab.com/gitlab-org/gitlab-ee/issues/3789 +[gitlab-com/infrastructure#2821]: https://gitlab.com/gitlab-com/infrastructure/issues/2821 +[omnibus-ssl]: https://docs.gitlab.com/omnibus/settings/ssl.html +[troubleshooting document]: troubleshooting.md +[using-geo]: using_a_geo_server.md diff --git a/doc/administration/geo/replication/database.md b/doc/administration/geo/replication/database.md new file mode 100644 index 00000000000..1e5a56c3f4e --- /dev/null +++ b/doc/administration/geo/replication/database.md @@ -0,0 +1,508 @@ +# Geo database replication **[PREMIUM ONLY]** + +NOTE: **Note:** +The following steps are for Omnibus installs only. Using Geo with source-based installs was **deprecated** in GitLab 11.5. + +NOTE: **Note:** +If your GitLab installation uses external (not managed by Omnibus) PostgreSQL +instances, the Omnibus roles will not be able to perform all necessary +configuration steps. In this case, +[follow the Geo with external PostgreSQL instances document instead](external_database.md). + +NOTE: **Note:** +The stages of the setup process must be completed in the documented order. +Before attempting the steps in this stage, [complete all prior stages][toc]. + +This document describes the minimal steps you have to take in order to +replicate your **primary** GitLab database to a **secondary** node's database. You may +have to change some values according to your database setup, how big it is, etc. + +You are encouraged to first read through all the steps before executing them +in your testing/production environment. + +## PostgreSQL replication + +The GitLab **primary** node where the write operations happen will connect to +the **primary** database server, and **secondary** nodes will +connect to their own database servers (which are also read-only). + +NOTE: **Note:** +In database documentation, you may see "**primary**" being referenced as "master" +and "**secondary**" as either "slave" or "standby" server (read-only). + +We recommend using [PostgreSQL replication slots][replication-slots-article] +to ensure that the **primary** node retains all the data necessary for the **secondary** nodes to +recover. See below for more details. + +The following guide assumes that: + +- You are using Omnibus and therefore you are using PostgreSQL 9.6 or later + which includes the [`pg_basebackup` tool][pgback] and improved + [Foreign Data Wrapper][FDW] support. +- You have a **primary** node already set up (the GitLab server you are + replicating from), running Omnibus' PostgreSQL (or equivalent version), and + you have a new **secondary** server set up with the same versions of the OS, + PostgreSQL, and GitLab on all nodes. + +CAUTION: **Warning:** +Geo works with streaming replication. Logical replication is not supported at this time. +There is an [issue where support is being discussed](https://gitlab.com/gitlab-org/gitlab-ee/issues/7420). + +### Step 1. Configure the **primary** server + +1. SSH into your GitLab **primary** server and login as root: + + ```sh + sudo -i + ``` + +1. Execute the command below to define the node as **primary** node: + + ```sh + gitlab-ctl set-geo-primary-node + ``` + + This command will use your defined `external_url` in `/etc/gitlab/gitlab.rb`. + +1. GitLab 10.4 and up only: Do the following to make sure the `gitlab` database user has a password defined: + + Generate a MD5 hash of the desired password: + + ```sh + gitlab-ctl pg-password-md5 gitlab + # Enter password: <your_password_here> + # Confirm password: <your_password_here> + # fca0b89a972d69f00eb3ec98a5838484 + ``` + + Edit `/etc/gitlab/gitlab.rb`: + + ```ruby + # Fill with the hash generated by `gitlab-ctl pg-password-md5 gitlab` + postgresql['sql_user_password'] = '<md5_hash_of_your_password>' + + # Every node that runs Unicorn or Sidekiq needs to have the database + # password specified as below. If you have a high-availability setup, this + # must be present in all application nodes. + gitlab_rails['db_password'] = '<your_password_here>' + ``` + +1. Omnibus GitLab already has a [replication user] + called `gitlab_replicator`. You must set the password for this user manually. + You will be prompted to enter a password: + + ```sh + gitlab-ctl set-replication-password + ``` + + This command will also read the `postgresql['sql_replication_user']` Omnibus + setting in case you have changed `gitlab_replicator` username to something + else. + + If you are using an external database not managed by Omnibus GitLab, you need + to create the replicator user and define a password to it manually: + + ```sql + --- Create a new user 'replicator' + CREATE USER gitlab_replicator; + + --- Set/change a password and grants replication privilege + ALTER USER gitlab_replicator WITH REPLICATION ENCRYPTED PASSWORD '<replication_password>'; + ``` + +1. Configure PostgreSQL to listen on network interfaces: + + For security reasons, PostgreSQL does not listen on any network interfaces + by default. However, Geo requires the **secondary** node to be able to + connect to the **primary** node's database. For this reason, we need the address of + each node. Note: For external PostgreSQL instances, see [additional instructions](external_database.md). + + If you are using a cloud provider, you can lookup the addresses for each + Geo node through your cloud provider's management console. + + To lookup the address of a Geo node, SSH in to the Geo node and execute: + + ```sh + ## + ## Private address + ## + ip route get 255.255.255.255 | awk '{print "Private address:", $NF; exit}' + + ## + ## Public address + ## + echo "External address: $(curl --silent ipinfo.io/ip)" + ``` + + In most cases, the following addresses will be used to configure GitLab + Geo: + + | Configuration | Address | + |:----------------------------------------|:------------------------------------------------------| + | `postgresql['listen_address']` | **Primary** node's public or VPC private address. | + | `postgresql['md5_auth_cidr_addresses']` | **Secondary** node's public or VPC private addresses. | + + If you are using Google Cloud Platform, SoftLayer, or any other vendor that + provides a virtual private cloud (VPC) you can use the **secondary** node's private + address (corresponds to "internal address" for Google Cloud Platform) for + `postgresql['md5_auth_cidr_addresses']` and `postgresql['listen_address']`. + + The `listen_address` option opens PostgreSQL up to network connections + with the interface corresponding to the given address. See [the PostgreSQL + documentation][pg-docs-runtime-conn] for more details. + + Depending on your network configuration, the suggested addresses may not + be correct. If your **primary** node and **secondary** nodes connect over a local + area network, or a virtual network connecting availability zones like + [Amazon's VPC](https://aws.amazon.com/vpc/) or [Google's VPC](https://cloud.google.com/vpc/) + you should use the **secondary** node's private address for `postgresql['md5_auth_cidr_addresses']`. + + Edit `/etc/gitlab/gitlab.rb` and add the following, replacing the IP + addresses with addresses appropriate to your network configuration: + + ```ruby + ## + ## Geo Primary role + ## - configure dependent flags automatically to enable Geo + ## + roles ['geo_primary_role'] + + ## + ## Primary address + ## - replace '<primary_node_ip>' with the public or VPC address of your Geo primary node + ## + postgresql['listen_address'] = '<primary_node_ip>' + + ## + # Allow PostgreSQL client authentication from the primary and secondary IPs. These IPs may be + # public or VPC addresses in CIDR format, for example ['198.51.100.1/32', '198.51.100.2/32'] + ## + postgresql['md5_auth_cidr_addresses'] = ['<primary_node_ip>/32', '<secondary_node_ip>/32'] + + ## + ## Replication settings + ## - set this to be the number of Geo secondary nodes you have + ## + postgresql['max_replication_slots'] = 1 + # postgresql['max_wal_senders'] = 10 + # postgresql['wal_keep_segments'] = 10 + + ## + ## Disable automatic database migrations temporarily + ## (until PostgreSQL is restarted and listening on the private address). + ## + gitlab_rails['auto_migrate'] = false + ``` + +1. Optional: If you want to add another **secondary** node, the relevant setting would look like: + + ```ruby + postgresql['md5_auth_cidr_addresses'] = ['<primary_node_ip>/32', '<secondary_node_ip>/32', '<another_secondary_node_ip>/32'] + ``` + + You may also want to edit the `wal_keep_segments` and `max_wal_senders` to + match your database replication requirements. Consult the [PostgreSQL - + Replication documentation][pg-docs-runtime-replication] + for more information. + +1. Save the file and reconfigure GitLab for the database listen changes and + the replication slot changes to be applied: + + ```sh + gitlab-ctl reconfigure + ``` + + Restart PostgreSQL for its changes to take effect: + + ```sh + gitlab-ctl restart postgresql + ``` + +1. Re-enable migrations now that PostgreSQL is restarted and listening on the + private address. + + Edit `/etc/gitlab/gitlab.rb` and **change** the configuration to `true`: + + ```ruby + gitlab_rails['auto_migrate'] = true + ``` + + Save the file and reconfigure GitLab: + + ```sh + gitlab-ctl reconfigure + ``` + +1. Now that the PostgreSQL server is set up to accept remote connections, run + `netstat -plnt | grep 5432` to make sure that PostgreSQL is listening on port + `5432` to the **primary** server's private address. + +1. A certificate was automatically generated when GitLab was reconfigured. This + will be used automatically to protect your PostgreSQL traffic from + eavesdroppers, but to protect against active ("man-in-the-middle") attackers, + the **secondary** node needs a copy of the certificate. Make a copy of the PostgreSQL + `server.crt` file on the **primary** node by running this command: + + ```sh + cat ~gitlab-psql/data/server.crt + ``` + + Copy the output into a clipboard or into a local file. You + will need it when setting up the **secondary** node! The certificate is not sensitive + data. + +### Step 2. Configure the **secondary** server + +1. SSH into your GitLab **secondary** server and login as root: + + ``` + sudo -i + ``` + +1. Stop application server and Sidekiq + + ``` + gitlab-ctl stop unicorn + gitlab-ctl stop sidekiq + ``` + + NOTE: **Note**: + This step is important so we don't try to execute anything before the node is fully configured. + +1. [Check TCP connectivity][rake-maintenance] to the **primary** node's PostgreSQL server: + + ```sh + gitlab-rake gitlab:tcp_check[<primary_node_ip>,5432] + ``` + + NOTE: **Note**: + If this step fails, you may be using the wrong IP address, or a firewall may + be preventing access to the server. Check the IP address, paying close + attention to the difference between public and private addresses and ensure + that, if a firewall is present, the **secondary** node is permitted to connect to the + **primary** node on port 5432. + +1. Create a file `server.crt` in the **secondary** server, with the content you got on the last step of the **primary** node's setup: + + ``` + editor server.crt + ``` + +1. Set up PostgreSQL TLS verification on the **secondary** node: + + Install the `server.crt` file: + + ```sh + install \ + -D \ + -o gitlab-psql \ + -g gitlab-psql \ + -m 0400 \ + -T server.crt ~gitlab-psql/.postgresql/root.crt + ``` + + PostgreSQL will now only recognize that exact certificate when verifying TLS + connections. The certificate can only be replicated by someone with access + to the private key, which is **only** present on the **primary** node. + +1. Test that the `gitlab-psql` user can connect to the **primary** node's database + (the default Omnibus database name is gitlabhq_production): + + ```sh + sudo \ + -u gitlab-psql /opt/gitlab/embedded/bin/psql \ + --list \ + -U gitlab_replicator \ + -d "dbname=gitlabhq_production sslmode=verify-ca" \ + -W \ + -h <primary_node_ip> + ``` + + When prompted enter the password you set in the first step for the + `gitlab_replicator` user. If all worked correctly, you should see + the list of **primary** node's databases. + + A failure to connect here indicates that the TLS configuration is incorrect. + Ensure that the contents of `~gitlab-psql/data/server.crt` on the **primary** node + match the contents of `~gitlab-psql/.postgresql/root.crt` on the **secondary** node. + +1. Configure PostgreSQL to enable FDW support: + + This step is similar to how we configured the **primary** instance. + We need to enable this, to enable FDW support, even if using a single node. + + Edit `/etc/gitlab/gitlab.rb` and add the following, replacing the IP + addresses with addresses appropriate to your network configuration: + + ```ruby + ## + ## Geo Secondary role + ## - configure dependent flags automatically to enable Geo + ## + roles ['geo_secondary_role'] + + ## + ## Secondary address + ## - replace '<secondary_node_ip>' with the public or VPC address of your Geo secondary node + ## + postgresql['listen_address'] = '<secondary_node_ip>' + postgresql['md5_auth_cidr_addresses'] = ['<secondary_node_ip>/32'] + + ## + ## Database credentials password (defined previously in primary node) + ## - replicate same values here as defined in primary node + ## + postgresql['sql_user_password'] = '<md5_hash_of_your_password>' + gitlab_rails['db_password'] = '<your_password_here>' + + ## + ## Enable FDW support for the Geo Tracking Database (improves performance) + ## + geo_secondary['db_fdw'] = true + ``` + + For external PostgreSQL instances, see [additional instructions](external_database.md). + If you bring a former **primary** node back online to serve as a **secondary** node, then you also need to remove `roles ['geo_primary_role']` or `geo_primary_role['enable'] = true`. + +1. Reconfigure GitLab for the changes to take effect: + + ```sh + gitlab-ctl reconfigure + ``` + +1. Restart PostgreSQL for the IP change to take effect and reconfigure again: + + ```sh + gitlab-ctl restart postgresql + gitlab-ctl reconfigure + ``` + + This last reconfigure will provision the FDW configuration and enable it. + +### Step 3. Initiate the replication process + +Below we provide a script that connects the database on the **secondary** node to +the database on the **primary** node, replicates the database, and creates the +needed files for streaming replication. + +The directories used are the defaults that are set up in Omnibus. If you have +changed any defaults, configure it as you see fit replacing the directories and paths. + +CAUTION: **Warning:** +Make sure to run this on the **secondary** server as it removes all PostgreSQL's +data before running `pg_basebackup`. + +1. SSH into your GitLab **secondary** server and login as root: + + ```sh + sudo -i + ``` + +1. Choose a database-friendly name to use for your **secondary** node to + use as the replication slot name. For example, if your domain is + `secondary.geo.example.com`, you may use `secondary_example` as the slot + name as shown in the commands below. + +1. Execute the command below to start a backup/restore and begin the replication + CAUTION: **Warning:** Each Geo **secondary** node must have its own unique replication slot name. + Using the same slot name between two secondaries will break PostgreSQL replication. + + ```sh + gitlab-ctl replicate-geo-database \ + --slot-name=<secondary_node_name> \ + --host=<primary_node_ip> + ``` + + When prompted, enter the _plaintext_ password you set up for the `gitlab_replicator` + user in the first step. + + This command also takes a number of additional options. You can use `--help` + to list them all, but here are a couple of tips: + - If PostgreSQL is listening on a non-standard port, add `--port=` as well. + - If your database is too large to be transferred in 30 minutes, you will need + to increase the timeout, e.g., `--backup-timeout=3600` if you expect the + initial replication to take under an hour. + - Pass `--sslmode=disable` to skip PostgreSQL TLS authentication altogether + (e.g., you know the network path is secure, or you are using a site-to-site + VPN). This is **not** safe over the public Internet! + - You can read more details about each `sslmode` in the + [PostgreSQL documentation][pg-docs-ssl]; + the instructions above are carefully written to ensure protection against + both passive eavesdroppers and active "man-in-the-middle" attackers. + - Change the `--slot-name` to the name of the replication slot + to be used on the **primary** database. The script will attempt to create the + replication slot automatically if it does not exist. + - If you're repurposing an old server into a Geo **secondary** node, you'll need to + add `--force` to the command line. + - When not in a production machine you can disable backup step if you + really sure this is what you want by adding `--skip-backup` + +The replication process is now complete. + +## PGBouncer support (optional) + +[PGBouncer](http://pgbouncer.github.io/) may be used with GitLab Geo to pool +PostgreSQL connections. We recommend using PGBouncer if you use GitLab in a +high-availability configuration with a cluster of nodes supporting a Geo +**primary** node and another cluster of nodes supporting a Geo **secondary** node. For more +information, see [High Availability with GitLab Omnibus](../../high_availability/database.md#high-availability-with-gitlab-omnibus-premium-only). + +For a Geo **secondary** node to work properly with PGBouncer in front of the database, +it will need a separate read-only user to make [PostgreSQL FDW queries][FDW] +work: + +1. On the **primary** Geo database, enter the PostgreSQL on the console as an + admin user. If you are using an Omnibus-managed database, log onto the **primary** + node that is running the PostgreSQL database (the default Omnibus database name is gitlabhq_production): + + ```sh + sudo \ + -u gitlab-psql /opt/gitlab/embedded/bin/psql \ + -h /var/opt/gitlab/postgresql gitlabhq_production + ``` + +1. Then create the read-only user: + + ```sql + -- NOTE: Use the password defined earlier + CREATE USER gitlab_geo_fdw WITH password 'mypassword'; + GRANT CONNECT ON DATABASE gitlabhq_production to gitlab_geo_fdw; + GRANT USAGE ON SCHEMA public TO gitlab_geo_fdw; + GRANT SELECT ON ALL TABLES IN SCHEMA public TO gitlab_geo_fdw; + GRANT SELECT ON ALL SEQUENCES IN SCHEMA public TO gitlab_geo_fdw; + + -- Tables created by "gitlab" should be made read-only for "gitlab_geo_fdw" + -- automatically. + ALTER DEFAULT PRIVILEGES FOR USER gitlab IN SCHEMA public GRANT SELECT ON TABLES TO gitlab_geo_fdw; + ALTER DEFAULT PRIVILEGES FOR USER gitlab IN SCHEMA public GRANT SELECT ON SEQUENCES TO gitlab_geo_fdw; + ``` + +1. On the **secondary** nodes, change `/etc/gitlab/gitlab.rb`: + + ``` + geo_postgresql['fdw_external_user'] = 'gitlab_geo_fdw' + ``` + +1. Save the file and reconfigure GitLab for the changes to be applied: + + ```sh + gitlab-ctl reconfigure + ``` + +## MySQL replication + +MySQL replication is not supported for Geo. + +## Troubleshooting + +Read the [troubleshooting document](troubleshooting.md). + +[replication-slots-article]: https://medium.com/@tk512/replication-slots-in-postgresql-b4b03d277c75 +[pgback]: http://www.postgresql.org/docs/9.2/static/app-pgbasebackup.html +[replication user]:https://wiki.postgresql.org/wiki/Streaming_Replication +[FDW]: https://www.postgresql.org/docs/9.6/static/postgres-fdw.html +[toc]: index.md#using-omnibus-gitlab +[rake-maintenance]: ../../raketasks/maintenance.md +[pg-docs-ssl]: https://www.postgresql.org/docs/9.6/static/libpq-ssl.html#LIBPQ-SSL-PROTECTION +[pg-docs-runtime-conn]: https://www.postgresql.org/docs/9.6/static/runtime-config-connection.html +[pg-docs-runtime-replication]: https://www.postgresql.org/docs/9.6/static/runtime-config-replication.html diff --git a/doc/administration/geo/replication/docker_registry.md b/doc/administration/geo/replication/docker_registry.md new file mode 100644 index 00000000000..5b02b861c61 --- /dev/null +++ b/doc/administration/geo/replication/docker_registry.md @@ -0,0 +1,23 @@ +# Docker Registry for a secondary node **[PREMIUM ONLY]** + +You can set up a [Docker Registry] on your +**secondary** Geo node that mirrors the one on the **primary** Geo node. + +## Storage support + +CAUTION: **Warning:** +If you use [local storage][registry-storage] +for the Container Registry you **cannot** replicate it to a **secondary** node. + +Docker Registry currently supports a few types of storages. If you choose a +distributed storage (`azure`, `gcs`, `s3`, `swift`, or `oss`) for your Docker +Registry on the **primary** node, you can use the same storage for a **secondary** +Docker Registry as well. For more information, read the +[Load balancing considerations][registry-load-balancing] +when deploying the Registry, and how to set up the storage driver for GitLab's +integrated [Container Registry][registry-storage]. + +[ee]: https://about.gitlab.com/pricing/ +[Docker Registry]: https://docs.docker.com/registry/ +[registry-storage]: ../../container_registry.md#container-registry-storage-driver +[registry-load-balancing]: https://docs.docker.com/registry/deploying/#load-balancing-considerations diff --git a/doc/administration/geo/replication/external_database.md b/doc/administration/geo/replication/external_database.md new file mode 100644 index 00000000000..177ca68613e --- /dev/null +++ b/doc/administration/geo/replication/external_database.md @@ -0,0 +1,219 @@ +# Geo with external PostgreSQL instances **[PREMIUM ONLY]** + +This document is relevant if you are using a PostgreSQL instance that is *not +managed by Omnibus*. This includes cloud-managed instances like AWS RDS, or +manually installed and configured PostgreSQL instances. + +NOTE: **Note**: +We strongly recommend running Omnibus-managed instances as they are actively +developed and tested. We aim to be compatible with most external +(not managed by Omnibus) databases but we do not guarantee compatibility. + +## **Primary** node + +1. SSH into a GitLab **primary** application server and login as root: + + ```sh + sudo -i + ``` + +1. Execute the command below to define the node as **primary** node: + + ```sh + gitlab-ctl set-geo-primary-node + ``` + + This command will use your defined `external_url` in `/etc/gitlab/gitlab.rb`. + + +### Configure the external database to be replicated + +To set up an external database, you can either: + +- Set up streaming replication yourself (for example, in AWS RDS). +- Perform the Omnibus configuration manually as follows. + +#### Leverage your cloud provider's tools to replicate the primary database + +Given you have a primary node set up on AWS EC2 that uses RDS. +You can now just create a read-only replica in a different region and the +replication process will be managed by AWS. Make sure you've set Network ACL, Subnet, and +Security Group according to your needs, so the secondary application node can access the database. +Skip to the [Configure secondary application node](#configure-secondary-application-nodes-to-use-the-external-read-replica) section below. + +#### Manually configure the primary database for replication + +The [geo_primary_role](https://docs.gitlab.com/omnibus/roles/#gitlab-geo-roles) +configures the **primary** node's database to be replicated by making changes to +`pg_hba.conf` and `postgresql.conf`. Make the following configuration changes +manually to your external database configuration: + +``` +## +## Geo Primary Role +## - pg_hba.conf +## +host replication gitlab_replicator <trusted secondary IP>/32 md5 +``` + +``` +## +## Geo Primary Role +## - postgresql.conf +## +sql_replication_user = gitlab_replicator +wal_level = hot_standby +max_wal_senders = 10 +wal_keep_segments = 50 +max_replication_slots = 1 # number of secondary instances +hot_standby = on +``` + +## **Secondary** nodes + +### Manually configure the replica database + +Make the following configuration changes manually to your `postgresql.conf` +of external replica database: + +``` +## +## Geo Secondary Role +## - postgresql.conf +## +wal_level = hot_standby +max_wal_senders = 10 +wal_keep_segments = 10 +hot_standby = on +``` + +### Configure **secondary** application nodes to use the external read-replica + +With Omnibus, the +[geo_secondary_role](https://docs.gitlab.com/omnibus/roles/#gitlab-geo-roles) +has three main functions: + +1. Configure the replica database. +1. Configure the tracking database. +1. Enable the [Geo Log Cursor](index.md#geo-log-cursor) (not covered in this section). + +To configure the connection to the external read-replica database and enable Log Cursor: + +1. SSH into a GitLab **secondary** application server and login as root: + + ```bash + sudo -i + ``` + +1. Edit `/etc/gitlab/gitlab.rb` and add the following + + ```ruby + ## + ## Geo Secondary role + ## - configure dependent flags automatically to enable Geo + ## + roles ['geo_secondary_role'] + + # note this is shared between both databases, + # make sure you define the same password in both + gitlab_rails['db_password'] = '<your_password_here>' + + gitlab_rails['db_username'] = 'gitlab' + gitlab_rails['db_host'] = '<database_read_replica_host>' + ``` +1. Save the file and [reconfigure GitLab](../../restart_gitlab.md#omnibus-gitlab-reconfigure) + +### Configure the tracking database + +**Secondary** nodes use a separate PostgreSQL installation as a tracking +database to keep track of replication status and automatically recover from +potential replication issues. Omnibus automatically configures a tracking database +when `roles ['geo_secondary_role']` is set. For high availability, +refer to [Geo High Availability](../../high_availability/README.md). +If you want to run this database external to Omnibus, please follow the instructions below. + +The tracking database requires an [FDW](https://www.postgresql.org/docs/9.6/static/postgres-fdw.html) +connection with the **secondary** replica database for improved performance. + +If you have an external database ready to be used as the tracking database, +follow the instructions below to use it: + +NOTE: **Note:** +If you want to use AWS RDS as a tracking database, make sure it has access to +the secondary database. Unfortunately, just assigning the same security group is not enough as +outbound rules do not apply to RDS PostgreSQL databases. Therefore, you need to explicitly add an inbound +rule to the read-replica's security group allowing any TCP traffic from +the tracking database on port 5432. + +1. SSH into a GitLab **secondary** server and login as root: + + ```bash + sudo -i + ``` + +1. Edit `/etc/gitlab/gitlab.rb` with the connection params and credentials for + the machine with the PostgreSQL instance: + + ```ruby + geo_secondary['db_username'] = 'gitlab_geo' + geo_secondary['db_password'] = '<your_password_here>' + + geo_secondary['db_host'] = '<tracking_database_host>' + geo_secondary['db_port'] = <tracking_database_port> # change to the correct port + geo_secondary['db_fdw'] = true # enable FDW + geo_postgresql['enable'] = false # don't use internal managed instance + ``` + +1. Save the file and [reconfigure GitLab](../../restart_gitlab.md#omnibus-gitlab-reconfigure) + +1. Run the tracking database migrations: + + ```bash + gitlab-rake geo:db:create + gitlab-rake geo:db:migrate + ``` + +1. Configure the + [PostgreSQL FDW](https://www.postgresql.org/docs/9.6/static/postgres-fdw.html) + connection and credentials: + + Save the script below in a file, ex. `/tmp/geo_fdw.sh` and modify the connection + params to match your environment. Execute it to set up the FDW connection. + + ```bash + #!/bin/bash + + # Secondary Database connection params: + DB_HOST="<public_ip_or_vpc_private_ip>" + DB_NAME="gitlabhq_production" + DB_USER="gitlab" + DB_PASS="<your_password_here>" + DB_PORT="5432" + + # Tracking Database connection params: + GEO_DB_HOST="<public_ip_or_vpc_private_ip>" + GEO_DB_NAME="gitlabhq_geo_production" + GEO_DB_USER="gitlab_geo" + GEO_DB_PORT="5432" + + query_exec () { + gitlab-psql -h $GEO_DB_HOST -d $GEO_DB_NAME -p $GEO_DB_PORT -c "${1}" + } + + query_exec "CREATE EXTENSION postgres_fdw;" + query_exec "CREATE SERVER gitlab_secondary FOREIGN DATA WRAPPER postgres_fdw OPTIONS (host '${DB_HOST}', dbname '${DB_NAME}', port '${DB_PORT}');" + query_exec "CREATE USER MAPPING FOR ${GEO_DB_USER} SERVER gitlab_secondary OPTIONS (user '${DB_USER}', password '${DB_PASS}');" + query_exec "CREATE SCHEMA gitlab_secondary;" + query_exec "GRANT USAGE ON FOREIGN SERVER gitlab_secondary TO ${GEO_DB_USER};" + ``` + + NOTE: **Note:** The script template above uses `gitlab-psql` as it's intended to be executed from the Geo machine, + but you can change it to `psql` and run it from any machine that has access to the database. We also recommend using + `psql` for AWS RDS. + +1. Save the file and [restart GitLab](../../restart_gitlab.md#omnibus-gitlab-restart) +1. Populate the FDW tables: + + ```bash + gitlab-rake geo:db:refresh_foreign_tables + ``` diff --git a/doc/administration/geo/replication/faq.md b/doc/administration/geo/replication/faq.md new file mode 100644 index 00000000000..dd1af0dbf9c --- /dev/null +++ b/doc/administration/geo/replication/faq.md @@ -0,0 +1,62 @@ +# Geo Frequently Asked Questions **[PREMIUM ONLY]** + +## What are the minimum requirements to run Geo? + +The requirements are listed [on the index page](index.md#requirements-for-running-geo) + +## How does Geo know which projects to sync? + +On each **secondary** node, there is a read-only replicated copy of the GitLab database. +A **secondary** node also has a tracking database where it stores which projects have been synced. +Geo compares the two databases to find projects that are not yet tracked. + +At the start, this tracking database is empty, so Geo will start trying to update from every project that it can see in the GitLab database. + +For each project to sync: + +1. Geo will issue a `git fetch geo --mirror` to get the latest information from the **primary** node. +If there are no changes, the sync will be fast and end quickly. Otherwise, it will pull the latest commits. +1. The **secondary** node will update the tracking database to store the fact that it has synced projects A, B, C, etc. +1. Repeat until all projects are synced. + +When someone pushes a commit to the **primary** node, it generates an event in the GitLab database that the repository has changed. +The **secondary** node sees this event, marks the project in question as dirty, and schedules the project to be resynced. + +To ensure that problems with pipelines (for example, syncs failing too many times or jobs being lost) don't permanently stop projects syncing, Geo also periodically checks the tracking database for projects that are marked as dirty. This check happens when +the number of concurrent syncs falls below `repos_max_capacity` and there are no new projects waiting to be synced. + +Geo also has a checksum feature which runs a SHA256 sum across all the Git references to the SHA values. +If the refs don't match between the **primary** node and the **secondary** node, then the **secondary** node will mark that project as dirty and try to resync it. +So even if we have an outdated tracking database, the validation should activate and find discrepancies in the repository state and resync. + +## Can I use Geo in a disaster recovery situation? + +Yes, but there are limitations to what we replicate (see +[What data is replicated to a **secondary** node?](#what-data-is-replicated-to-a-secondary-node)). + +Read the documentation for [Disaster Recovery](../disaster_recovery/index.md). + +## What data is replicated to a **secondary** node? + +We currently replicate project repositories, LFS objects, generated +attachments / avatars and the whole database. This means user accounts, +issues, merge requests, groups, project data, etc., will be available for +query. + +## Can I git push to a **secondary** node? + +Yes! Pushing directly to a **secondary** node (for both HTTP and SSH, including git-lfs) was [introduced](https://about.gitlab.com/2018/09/22/gitlab-11-3-released/) in [GitLab Premium](https://about.gitlab.com/pricing/#self-managed) 11.3. + +## How long does it take to have a commit replicated to a **secondary** node? + +All replication operations are asynchronous and are queued to be dispatched. Therefore, it depends on a lot of +factors including the amount of traffic, how big your commit is, the +connectivity between your nodes, your hardware, etc. + +## What if the SSH server runs at a different port? + +That's totally fine. We use HTTP(s) to fetch repository changes from the **primary** node to all **secondary** nodes. + +## Is this possible to set up a Docker Registry for a **secondary** node that mirrors the one on the **primary** node? + +Yes. See [Docker Registry for a **secondary** node](docker_registry.md). diff --git a/doc/administration/geo/replication/high_availability.md b/doc/administration/geo/replication/high_availability.md new file mode 100644 index 00000000000..921a3ef1c7a --- /dev/null +++ b/doc/administration/geo/replication/high_availability.md @@ -0,0 +1,295 @@ +# Geo High Availability **[PREMIUM ONLY]** + +This document describes a minimal reference architecture for running Geo +in a high availability configuration. If your HA setup differs from the one +described, it is possible to adapt these instructions to your needs. + +## Architecture overview + + + +_[diagram source - gitlab employees only][diagram-source]_ + +The topology above assumes that the **primary** and **secondary** Geo clusters +are located in two separate locations, on their own virtual network +with private IP addresses. The network is configured such that all machines within +one geographic location can communicate with each other using their private IP addresses. +The IP addresses given are examples and may be different depending on the +network topology of your deployment. + +The only external way to access the two Geo deployments is by HTTPS at +`gitlab.us.example.com` and `gitlab.eu.example.com` in the example above. + +NOTE: **Note:** +The **primary** and **secondary** Geo deployments must be able to communicate to each other over HTTPS. + +## Redis and PostgreSQL High Availability + +The **primary** and **secondary** Redis and PostgreSQL should be configured +for high availability. Because of the additional complexity involved +in setting up this configuration for PostgreSQL and Redis, +it is not covered by this Geo HA documentation. + +For more information about setting up a highly available PostgreSQL cluster and Redis cluster using the omnibus package see the high availability documentation for +[PostgreSQL](../../high_availability/database.md) and +[Redis](../../high_availability/redis.md), respectively. + +NOTE: **Note:** +It is possible to use cloud hosted services for PostgreSQL and Redis, but this is beyond the scope of this document. + +## Prerequisites: A working GitLab HA cluster + +This cluster will serve as the **primary** node. Use the +[GitLab HA documentation](../../high_availability/README.md) to set this up. + +## Configure the GitLab cluster to be the **primary** node + +The following steps enable a GitLab cluster to serve as the **primary** node. + +### Step 1: Configure the **primary** frontend servers + +1. Edit `/etc/gitlab/gitlab.rb` and add the following: + + ```ruby + ## + ## Enable the Geo primary role + ## + roles ['geo_primary_role'] + + ## + ## Disable automatic migrations + ## + gitlab_rails['auto_migrate'] = false + ``` + +After making these changes, [reconfigure GitLab][gitlab-reconfigure] so the changes take effect. + +NOTE: **Note:** PostgreSQL and Redis should have already been disabled on the +application servers, and connections from the application servers to those +services on the backend servers configured, during normal GitLab HA set up. See +high availability configuration documentation for +[PostgreSQL](../../high_availability/database.md#configuring-the-application-nodes) +and [Redis](../../high_availability/redis.md#example-configuration-for-the-gitlab-application). + +The **primary** database will require modification later, as part of +[step 2](#step-2-configure-the-main-read-only-replica-postgresql-database-on-the-secondary-node). + +## Configure a **secondary** node + +A **secondary** cluster is similar to any other GitLab HA cluster, with two +major differences: + +* The main PostgreSQL database is a read-only replica of the **primary** node's + PostgreSQL database. +* There is also a single PostgreSQL database for the **secondary** cluster, + called the "tracking database", which tracks the synchronization state of + various resources. + +Therefore, we will set up the HA components one-by-one, and include deviations +from the normal HA setup. + +### Step 1: Configure the Redis and NFS services on the **secondary** node + +Configure the following services, again using the non-Geo high availability +documentation: + +* [Configuring Redis for GitLab HA](../../high_availability/redis.md) for high + availability. +* [NFS](../../high_availability/nfs.md) which will store data that is + synchronized from the **primary** node. + +### Step 2: Configure the main read-only replica PostgreSQL database on the **secondary** node + +NOTE: **Note:** The following documentation assumes the database will be run on +a single node only, rather than as a PostgreSQL cluster. + +Configure the [**secondary** database](database.md) as a read-only replica of +the **primary** database. Use the following as a guide. + +1. Edit `/etc/gitlab/gitlab.rb` in the replica database machine, and add the + following: + + ```ruby + ## + ## Configure the PostgreSQL role + ## + roles ['postgres_role'] + + ## + ## Secondary address + ## - replace '<secondary_node_ip>' with the public or VPC address of your Geo secondary node + ## - replace '<tracking_database_ip>' with the public or VPC address of your Geo tracking database node + ## + postgresql['listen_address'] = '<secondary_node_ip>' + postgresql['md5_auth_cidr_addresses'] = ['<secondary_node_ip>/32', '<tracking_database_ip>/32'] + + ## + ## Database credentials password (defined previously in primary node) + ## - replicate same values here as defined in primary node + ## + postgresql['sql_user_password'] = '<md5_hash_of_your_password>' + gitlab_rails['db_password'] = '<your_password_here>' + + ## + ## When running the Geo tracking database on a separate machine, disable it + ## here and allow connections from the tracking database host. And ensure + ## the tracking database IP is in postgresql['md5_auth_cidr_addresses'] above. + ## + geo_postgresql['enable'] = false + ``` + +After making these changes, [reconfigure GitLab][gitlab-reconfigure] so the changes take effect. + +If using an external PostgreSQL instance, refer also to +[Geo with external PostgreSQL instances](external_database.md). + +### Step 3: Configure the tracking database on the **secondary** node + +NOTE: **Note:** This documentation assumes the tracking database will be run on +only a single machine, rather than as a PostgreSQL cluster. + +Configure the tracking database. + +1. Edit `/etc/gitlab/gitlab.rb` in the tracking database machine, and add the + following: + + ```ruby + ## + ## Enable the Geo secondary tracking database + ## + geo_postgresql['enable'] = true + geo_postgresql['listen_address'] = '<ip_address_of_this_host>' + geo_postgresql['sql_user_password'] = '<tracking_database_password_md5_hash>' + + ## + ## Configure FDW connection to the replica database + ## + geo_secondary['db_fdw'] = true + geo_postgresql['fdw_external_password'] = '<replica_database_password_plaintext>' + geo_postgresql['md5_auth_cidr_addresses'] = ['<replica_database_ip>/32'] + gitlab_rails['db_host'] = '<replica_database_ip>' + + # Prevent reconfigure from attempting to run migrations on the replica DB + gitlab_rails['auto_migrate'] = false + + ## + ## Disable all other services that aren't needed, since we don't have a role + ## that does this. + ## + alertmanager['enable'] = false + consul['enable'] = false + gitaly['enable'] = false + gitlab_monitor['enable'] = false + gitlab_workhorse['enable'] = false + nginx['enable'] = false + node_exporter['enable'] = false + pgbouncer_exporter['enable'] = false + postgresql['enable'] = false + prometheus['enable'] = false + redis['enable'] = false + redis_exporter['enable'] = false + repmgr['enable'] = false + sidekiq['enable'] = false + unicorn['enable'] = false + ``` + +After making these changes, [reconfigure GitLab][gitlab-reconfigure] so the changes take effect. + +If using an external PostgreSQL instance, refer also to +[Geo with external PostgreSQL instances](external_database.md). + +### Step 4: Configure the frontend application servers on the **secondary** node + +In the architecture overview, there are two machines running the GitLab +application services. These services are enabled selectively in the +configuration. + +Configure the application servers following +[Configuring GitLab for HA](../../high_availability/gitlab.md), then make the +following modifications: + +1. Edit `/etc/gitlab/gitlab.rb` on each application server in the **secondary** + cluster, and add the following: + + ```ruby + ## + ## Enable the Geo secondary role + ## + roles ['geo_secondary_role', 'application_role'] + + ## + ## Disable automatic migrations + ## + gitlab_rails['auto_migrate'] = false + + ## + ## Configure the connection to the tracking DB. And disable application + ## servers from running tracking databases. + ## + geo_secondary['db_host'] = '<geo_tracking_db_host>' + geo_secondary['db_password'] = '<geo_tracking_db_password>' + geo_postgresql['enable'] = false + + ## + ## Configure connection to the streaming replica database, if you haven't + ## already + ## + gitlab_rails['db_host'] = '<replica_database_host>' + gitlab_rails['db_password'] = '<replica_database_password>' + + ## + ## Configure connection to Redis, if you haven't already + ## + gitlab_rails['redis_host'] = '<redis_host>' + gitlab_rails['redis_password'] = '<redis_password>' + + ## + ## If you are using custom users not managed by Omnibus, you need to specify + ## UIDs and GIDs like below, and ensure they match between servers in a + ## cluster to avoid permissions issues + ## + user['uid'] = 9000 + user['gid'] = 9000 + web_server['uid'] = 9001 + web_server['gid'] = 9001 + registry['uid'] = 9002 + registry['gid'] = 9002 + ``` + +NOTE: **Note:** +If you had set up PostgreSQL cluster using the omnibus package and you had set +up `postgresql['sql_user_password'] = 'md5 digest of secret'` setting, keep in +mind that `gitlab_rails['db_password']` and `geo_secondary['db_password']` +mentioned above contains the plaintext passwords. This is used to let the Rails +servers connect to the databases. + +NOTE: **Note:** +Make sure that current node IP is listed in `postgresql['md5_auth_cidr_addresses']` setting of your remote database. + +After making these changes [Reconfigure GitLab][gitlab-reconfigure] so the changes take effect. + +On the secondary the following GitLab frontend services will be enabled: + +* geo-logcursor +* gitlab-pages +* gitlab-workhorse +* logrotate +* nginx +* registry +* remote-syslog +* sidekiq +* unicorn + +Verify these services by running `sudo gitlab-ctl status` on the frontend +application servers. + +### Step 5: Set up the LoadBalancer for the **secondary** node + +In this topology, a load balancer is required at each geographic location to +route traffic to the application servers. + +See [Load Balancer for GitLab HA](../../high_availability/load_balancer.md) for +more information. + +[diagram-source]: https://docs.google.com/drawings/d/1z0VlizKiLNXVVVaERFwgsIOuEgjcUqDTWPdQYsE7Z4c/edit +[gitlab-reconfigure]: ../../restart_gitlab.md#omnibus-gitlab-reconfigure diff --git a/doc/administration/geo/replication/img/geo_architecture.png b/doc/administration/geo/replication/img/geo_architecture.png Binary files differnew file mode 100644 index 00000000000..d318cd5d0f4 --- /dev/null +++ b/doc/administration/geo/replication/img/geo_architecture.png diff --git a/doc/administration/geo/replication/img/geo_node_dashboard.png b/doc/administration/geo/replication/img/geo_node_dashboard.png Binary files differnew file mode 100644 index 00000000000..99792d0770d --- /dev/null +++ b/doc/administration/geo/replication/img/geo_node_dashboard.png diff --git a/doc/administration/geo/replication/img/geo_node_healthcheck.png b/doc/administration/geo/replication/img/geo_node_healthcheck.png Binary files differnew file mode 100644 index 00000000000..33a31f7ab49 --- /dev/null +++ b/doc/administration/geo/replication/img/geo_node_healthcheck.png diff --git a/doc/administration/geo/replication/img/geo_overview.png b/doc/administration/geo/replication/img/geo_overview.png Binary files differnew file mode 100644 index 00000000000..01c1615212c --- /dev/null +++ b/doc/administration/geo/replication/img/geo_overview.png diff --git a/doc/administration/geo/replication/index.md b/doc/administration/geo/replication/index.md new file mode 100644 index 00000000000..54377f7ae4e --- /dev/null +++ b/doc/administration/geo/replication/index.md @@ -0,0 +1,292 @@ +# Geo Replication **[PREMIUM ONLY]** + +Geo is the solution for widely distributed development teams. + +## Overview + +Fetching large repositories can take a long time for teams located far from a single GitLab instance. + +Geo provides local, read-only instances of your GitLab instances, reducing the time it takes to clone and fetch large repositories and speeding up development. + +> - Geo is part of [GitLab Premium](https://about.gitlab.com/pricing/#self-managed). +> - Introduced in GitLab Enterprise Edition 8.9. +> - We recommend you use: +> - At least GitLab Enterprise Edition 10.0 for basic Geo features. +> - The latest version for a better experience. +> - Make sure that all nodes run the same GitLab version. +> - Geo requires PostgreSQL 9.6 and Git 2.9, in addition to GitLab's usual [minimum requirements](../../../install/requirements.md). +> - Using Geo in combination with [High Availability](../../high_availability/README.md) is considered **Generally Available** (GA) in GitLab [GitLab Premium](https://about.gitlab.com/pricing/) 10.4. + +For a video introduction to Geo, see [Introduction to GitLab Geo - GitLab Features](https://www.youtube.com/watch?v=-HDLxSjEh6w). + +CAUTION: **Caution:** +Geo undergoes significant changes from release to release. Upgrades **are** supported and [documented](#updating-geo), but you should ensure that you're using the right version of the documentation for your installation. + +To make sure you're using the right version of the documentation, navigate to [the source version of this page on GitLab.com](https://gitlab.com/gitlab-org/gitlab-ee/blob/master/doc/administration/geo/replication/index.md) and choose the appropriate release from the **Switch branch/tag** dropdown. For example, [`v11.2.3-ee`](https://gitlab.com/gitlab-org/gitlab-ee/blob/v11.2.3-ee/doc/administration/geo/replication/index.md). + +## Use cases + +Implementing Geo provides the following benefits: + +- Reduce from minutes to seconds the time taken for your distributed developers to clone and fetch large repositories and projects. +- Enable all of your developers to contribute ideas and work in parallel, no matter where they are. +- Balance the load between your **primary** and **secondary** nodes, or offload your automated tests to a **secondary** node. + +In addition, it: + +- Can be used for cloning and fetching projects, in addition to reading any data available in the GitLab web interface (see [current limitations](#current-limitations)). +- Overcomes slow connections between distant offices, saving time by improving speed for distributed teams. +- Helps reducing the loading time for automated tasks, custom integrations, and internal workflows. +- Can quickly fail over to a **secondary** node in a [disaster recovery](../disaster_recovery/index.md) scenario. +- Allows [planned failover](../disaster_recovery/planned_failover.md) to a **secondary** node. + +Geo provides: + +- Read-only **secondary** nodes: Maintain one **primary** GitLab node while still enabling read-only **secondary** nodes for each of your distributed teams. +- Authentication system hooks: **Secondary** nodes receives all authentication data (like user accounts and logins) from the **primary** instance. +- An intuitive UI: **Secondary** nodes utilize the same web interface your team has grown accustomed to. In addition, there are visual notifications that block write operations and make it clear that a user is on a **secondary** node. + +## How it works + +Your Geo instance can be used for cloning and fetching projects, in addition to reading any data. This will make working with large repositories over large distances much faster. + + + +When Geo is enabled, the: + +- Original instance is known as the **primary** node. +- Replicated read-only nodes are known as **secondary** nodes. + +Keep in mind that: + +- **Secondary** nodes talk to the **primary** node to: + - Get user data for logins (API). + - Replicate repositories, LFS Objects, and Attachments (HTTPS + JWT). +- Since GitLab Premium 10.0, the **primary** node no longer talks to **secondary** nodes to notify for changes (API). +- Pushing directly to a **secondary** node (for both HTTP and SSH, including git-lfs) was [introduced](https://about.gitlab.com/2018/09/22/gitlab-11-3-released/) in [GitLab Premium](https://about.gitlab.com/pricing/#self-managed) 11.3. +- There are [limitations](#current-limitations) in the current implementation. + +### Architecture + +The following diagram illustrates the underlying architecture of Geo. + + + +In this diagram: + +- There is the **primary** node and the details of one **secondary** node. +- Writes to the database can only be performed on the **primary** node. A **secondary** node receives database + updates via PostgreSQL streaming replication. +- If present, the [LDAP server](#ldap) should be configured to replicate for [Disaster Recovery](../disaster_recovery/index.md) scenarios. +- A **secondary** node performs different type of synchronizations against the **primary** node, using a special + authorization protected by JWT: + - Repositories are cloned/updated via Git over HTTPS. + - Attachments, LFS objects, and other files are downloaded via HTTPS using a private API endpoint. + +From the perspective of a user performing Git operations: + +- The **primary** node behaves as a full read-write GitLab instance. +- **Secondary** nodes are read-only but proxy Git push operations to the **primary** node. This makes **secondary** nodes appear to support push operations themselves. + +To simplify the diagram, some necessary components are omitted. Note that: + +- Git over SSH requires [`gitlab-shell`](https://gitlab.com/gitlab-org/gitlab-shell) and OpenSSH. +- Git over HTTPS required [`gitlab-workhorse`](https://gitlab.com/gitlab-org/gitlab-workhorse). + +Note that a **secondary** node needs two different PostgreSQL databases: + +- A read-only database instance that streams data from the main GitLab database. +- [Another database instance](#geo-tracking-database) used internally by the **secondary** node to record what data has been replicated. + +In **secondary** nodes, there is an additional daemon: [Geo Log Cursor](#geo-log-cursor). + +## Requirements for running Geo + +The following are required to run Geo: + +- An operating system that supports OpenSSH 6.9+ (needed for + [fast lookup of authorized SSH keys in the database](../../operations/fast_ssh_key_lookup.md)) + The following operating systems are known to ship with a current version of OpenSSH: + - [CentOS](https://www.centos.org) 7.4+ + - [Ubuntu](https://www.ubuntu.com) 16.04+ +- PostgreSQL 9.6+ with [FDW](https://www.postgresql.org/docs/9.6/postgres-fdw.html) support and [Streaming Replication](https://wiki.postgresql.org/wiki/Streaming_Replication) +- Git 2.9+ + +### Firewall rules + +The following table lists basic ports that must be open between the **primary** and **secondary** nodes for Geo. + +| **Primary** node | **Secondary** node | Protocol | +|:-----------------|:-------------------|:-------------| +| 80 | 80 | HTTP | +| 443 | 443 | TCP or HTTPS | +| 22 | 22 | TCP | +| 5432 | | PostgreSQL | + +See the full list of ports used by GitLab in [Package defaults](https://docs.gitlab.com/omnibus/package-information/defaults.html) + +NOTE: **Note:** +[Web terminal](../../../ci/environments.md#web-terminals) support requires your load balancer to correctly handle WebSocket connections. +When using HTTP or HTTPS proxying, your load balancer must be configured to pass through the `Connection` and `Upgrade` hop-by-hop headers. See the [web terminal](../../integration/terminal.md) integration guide for more details. + +NOTE: **Note:** +When using HTTPS protocol for port 443, you will need to add an SSL certificate to the load balancers. +If you wish to terminate SSL at the GitLab application server instead, use TCP protocol. + +### LDAP + +We recommend that if you use LDAP on your **primary** node, you also set up secondary LDAP servers on each **secondary** node. Otherwise, users will not be able to perform Git operations over HTTP(s) on the **secondary** node using HTTP Basic Authentication. However, Git via SSH and personal access tokens will still work. + +NOTE: **Note:** +It is possible for all **secondary** nodes to share an LDAP server, but additional latency can be an issue. Also, consider what LDAP server will be available in a [disaster recovery](../disaster_recovery/index.md) scenario if a **secondary** node is promoted to be a **primary** node. + +Check for instructions on how to set up replication in your LDAP service. Instructions will be different depending on the software or service used. For example, OpenLDAP provides [these instructions](https://www.openldap.org/doc/admin24/replication.html). + +### Geo Tracking Database + +The tracking database instance is used as metadata to control what needs to be updated on the disk of the local instance. For example: + +- Download new assets. +- Fetch new LFS Objects. +- Fetch changes from a repository that has recently been updated. + +Because the replicated database instance is read-only, we need this additional database instance for each **secondary** node. +The tracking database requires the `postgres_fdw` extension. + +### Geo Log Cursor + +This daemon: + +- Reads a log of events replicated by the **primary** node to the **secondary** database instance. +- Updates the Geo Tracking Database instance with changes that need to be executed. + +When something is marked to be updated in the tracking database instance, asynchronous jobs running on the **secondary** node will execute the required operations and update the state. + +This new architecture allows GitLab to be resilient to connectivity issues between the nodes. It doesn't matter how long the **secondary** node is disconnected from the **primary** node as it will be able to replay all the events in the correct order and become synchronized with the **primary** node again. + +## Setup instructions + +These instructions assume you have a working instance of GitLab. They guide you through: + +1. Making your existing instance the **primary** node. +1. Adding **secondary** nodes. + +CAUTION: **Caution:** +The steps below should be followed in the order they appear. **Make sure the GitLab version is the same on all nodes.** + +### Using Omnibus GitLab + +If you installed GitLab using the Omnibus packages (highly recommended): + +1. [Install GitLab Enterprise Edition](https://about.gitlab.com/installation/) on the server that will serve as the **secondary** node. Do not create an account or log in to the new **secondary** node. +1. [Upload the GitLab License](../../../user/admin_area/license.md) on the **primary** node to unlock Geo. The license must be for [GitLab Premium](https://about.gitlab.com/pricing/) or higher. +1. [Set up the database replication](database.md) (`primary (read-write) <-> secondary (read-only)` topology). +1. [Configure fast lookup of authorized SSH keys in the database](../../operations/fast_ssh_key_lookup.md). This step is required and needs to be done on **both** the **primary** and **secondary** nodes. +1. [Configure GitLab](configuration.md) to set the **primary** and **secondary** nodes. +1. Optional: [Configure a secondary LDAP server](../../auth/ldap.md) for the **secondary** node. See [notes on LDAP](#ldap). +1. [Follow the "Using a Geo Server" guide](using_a_geo_server.md). + +## Post-installation documentation + +After installing GitLab on the **secondary** nodes and performing the initial configuration, see the following documentation for post-installation information. + +### Configuring Geo + +For information on configuring Geo, see [Geo configuration](configuration.md). + +### Updating Geo + +For information on how to update your Geo nodes to the latest GitLab version, see [Updating the Geo nodes](updating_the_geo_nodes.md). + +### Configuring Geo high availability + +For information on configuring Geo for high availability, see [Geo High Availability](high_availability.md). + +### Configuring Geo with Object Storage + +For information on configuring Geo with object storage, see [Geo with Object storage](object_storage.md). + +### Disaster Recovery + +For information on using Geo in disaster recovery situations to mitigate data-loss and restore services, see [Disaster Recovery](../disaster_recovery/index.md). + +### Replicating the Container Registry + +For more information on how to replicate the Container Registry, see [Docker Registry for a **secondary** node](docker_registry.md). + +### Security Review + +For more information on Geo security, see [Geo security review](security_review.md). + +### Tuning Geo + +For more information on tuning Geo, see [Tuning Geo](tuning.md). + +## Remove Geo node + +For more information on removing a Geo node, see [Removing **secondary** Geo nodes](remove_geo_node.md). + +## Current limitations + +CAUTION: **Caution:** +This list of limitations only reflects the latest version of GitLab. If you are using an older version, extra limitations may be in place. + +- Pushing directly to a **secondary** node redirects (for HTTP) or proxies (for SSH) the request to the **primary** node instead of [handling it directly](https://gitlab.com/gitlab-org/gitlab-ee/issues/1381), except when using Git over HTTP with credentials embedded within the URI. For example, `https://user:password@secondary.tld`. +- The **primary** node has to be online for OAuth login to happen. Existing sessions and Git are not affected. +- The installation takes multiple manual steps that together can take about an hour depending on circumstances. We are working on improving this experience. See [gitlab-org/omnibus-gitlab#2978](https://gitlab.com/gitlab-org/omnibus-gitlab/issues/2978) for details. +- Real-time updates of issues/merge requests (for example, via long polling) doesn't work on the **secondary** node. +- [Selective synchronization](configuration.md#selective-synchronization) applies only to files and repositories. Other datasets are replicated to the **secondary** node in full, making it inappropriate for use as an access control mechanism. +- Object pools for forked project deduplication work only on the **primary** node, and are duplicated on the **secondary** node. +- [External merge request diffs](../../merge_request_diffs.md) will not be replicated if they are on-disk, and viewing merge requests will fail. However, external MR diffs in object storage **are** supported. The default configuration (in-database) does work. + +### Limitations on replication + +Only the following items are replicated to the **secondary** node: + +- All database content. For example, snippets, epics, issues, merge requests, groups, and project metadata. +- Project repositories. +- Project wiki repositories. +- User uploads. For example, attachments to issues, merge requests, epics, and avatars. +- CI job artifacts and traces. + +DANGER: **DANGER** +Data not on this list is unavailable on the **secondary** node. Failing over without manually replicating data not on this list will cause the data to be **lost**. + +### Examples of data not replicated + +Take special note that these examples of GitLab features are both: + +- Commonly used. +- **Not** replicated by Geo at present. + +Examples include: + +- [Elasticsearch integration](../../../integration/elasticsearch.md). +- [Container Registry](../../container_registry.md). [Object Storage](object_storage.md) can mitigate this. +- [GitLab Pages](../../pages/index.md). +- [Mattermost integration](https://docs.gitlab.com/omnibus/gitlab-mattermost/). + +CAUTION: **Caution:** +If you wish to use them on a **secondary** node, or to execute a failover successfully, you will need to replicate their data using some other means. + +## Frequently Asked Questions + +For answers to common questions, see the [Geo FAQ](faq.md). + +## Log files + +Since GitLab 9.5, Geo stores structured log messages in a `geo.log` file. For Omnibus installations, this file is at `/var/log/gitlab/gitlab-rails/geo.log`. + +This file contains information about when Geo attempts to sync repositories and files. Each line in the file contains a separate JSON entry that can be ingested into Elasticsearch, Splunk, etc. + +For example: + +```json +{"severity":"INFO","time":"2017-08-06T05:40:16.104Z","message":"Repository update","project_id":1,"source":"repository","resync_repository":true,"resync_wiki":true,"class":"Gitlab::Geo::LogCursor::Daemon","cursor_delay_s":0.038} +``` + +This message shows that Geo detected that a repository update was needed for project `1`. + +## Troubleshooting + +For troubleshooting steps, see [Geo Troubleshooting](troubleshooting.md). diff --git a/doc/administration/geo/replication/object_storage.md b/doc/administration/geo/replication/object_storage.md new file mode 100644 index 00000000000..c3c11dbaf1e --- /dev/null +++ b/doc/administration/geo/replication/object_storage.md @@ -0,0 +1,43 @@ +# Geo with Object storage **[PREMIUM ONLY]** + +Geo can be used in combination with Object Storage (AWS S3, or +other compatible object storage). + +## Configuration + +At this time it is required that if object storage is enabled on the +**primary** node, it must also be enabled on each **secondary** node. + +**Secondary** nodes can use the same storage bucket as the **primary** node, or +they can use a replicated storage bucket. At this time GitLab does not +take care of content replication in object storage. + +For LFS, follow the documentation to +[set up LFS object storage](../../../workflow/lfs/lfs_administration.md#storing-lfs-objects-in-remote-object-storage). + +For CI job artifacts, there is similar documentation to configure +[jobs artifact object storage](../../job_artifacts.md#using-object-storage) + +For user uploads, there is similar documentation to configure [upload object storage](../../uploads.md#using-object-storage-core-only) + +You should enable and configure object storage on both **primary** and **secondary** +nodes. Migrating existing data to object storage should be performed on the +**primary** node only. **Secondary** nodes will automatically notice that the migrated +files are now in object storage. + +## Replication + +When using Amazon S3, you can use +[CRR](https://docs.aws.amazon.com/AmazonS3/latest/dev/crr.html) to +have automatic replication between the bucket used by the **primary** node and +the bucket used by **secondary** nodes. + +If you are using Google Cloud Storage, consider using +[Multi-Regional Storage](https://cloud.google.com/storage/docs/storage-classes#multi-regional). +Or you can use the [Storage Transfer Service](https://cloud.google.com/storage/transfer/), +although this only supports daily synchronization. + +For manual synchronization, or scheduled by `cron`, please have a look at: + +- [`s3cmd sync`](http://s3tools.org/s3cmd-sync) +- [`gsutil rsync`](https://cloud.google.com/storage/docs/gsutil/commands/rsync) diff --git a/doc/administration/geo/replication/remove_geo_node.md b/doc/administration/geo/replication/remove_geo_node.md new file mode 100644 index 00000000000..b190fe7d42d --- /dev/null +++ b/doc/administration/geo/replication/remove_geo_node.md @@ -0,0 +1,50 @@ +# Removing secondary Geo nodes **[PREMIUM ONLY]** + +**Secondary** nodes can be removed from the Geo cluster using the Geo admin page of the **primary** node. To remove a **secondary** node: + +1. Navigate to **Admin Area > Geo** (`/admin/geo/nodes`). +1. Click the **Remove** button for the **secondary** node you want to remove. +1. Confirm by clicking **Remove** when the prompt appears. + +Once removed from the Geo admin page, you must stop and uninstall the **secondary** node: + +1. On the **secondary** node, stop GitLab: + + ```bash + sudo gitlab-ctl stop + ``` +1. On the **secondary** node, uninstall GitLab: + + ```bash + # Stop gitlab and remove its supervision process + sudo gitlab-ctl uninstall + + # Debian/Ubuntu + sudo dpkg --remove gitlab-ee + + # Redhat/Centos + sudo rpm --erase gitlab-ee + ``` + +Once GitLab has been uninstalled from the **secondary** node, the replication slot must be dropped from the **primary** node's database as follows: + +1. On the **primary** node, start a PostgreSQL console session: + + ```bash + sudo gitlab-psql + ``` + + NOTE: **Note:** + Using `gitlab-rails dbconsole` will not work, because managing replication slots requires superuser permissions. + +1. Find the name of the relevant replication slot. This is the slot that is specified with `--slot-name` when running the replicate command: `gitlab-ctl replicate-geo-database`. + + ```sql + SELECT * FROM pg_replication_slots; + ``` + +1. Remove the replication slot for the **secondary** node: + + ```sql + SELECT pg_drop_replication_slot('<name_of_slot>'); + ``` diff --git a/doc/administration/geo/replication/security_review.md b/doc/administration/geo/replication/security_review.md new file mode 100644 index 00000000000..cd54e2dc8c4 --- /dev/null +++ b/doc/administration/geo/replication/security_review.md @@ -0,0 +1,284 @@ +# Geo security review (Q&A) **[PREMIUM ONLY]** + +The following security review of the Geo feature set focuses on security +aspects of the feature as they apply to customers running their own GitLab +instances. The review questions are based in part on the [application security architecture](https://www.owasp.org/index.php/Application_Security_Architecture_Cheat_Sheet) +questions from [owasp.org](https://www.owasp.org). + +## Business Model + +### What geographic areas does the application service? + +- This varies by customer. Geo allows customers to deploy to multiple areas, + and they get to choose where they are. +- Region and node selection is entirely manual. + +## Data Essentials + +### What data does the application receive, produce, and process? + +- Geo streams almost all data held by a GitLab instance between sites. This + includes full database replication, most files (user-uploaded attachments, + etc) and repository + wiki data. In a typical configuration, this will + happen across the public Internet, and be TLS-encrypted. +- PostgreSQL replication is TLS-encrypted. +- See also: [only TLSv1.2 should be supported](https://gitlab.com/gitlab-org/omnibus-gitlab/issues/2948) + +### How can the data be classified into categories according to its sensitivity? + +- GitLab’s model of sensitivity is centered around public vs. internal vs. + private projects. Geo replicates them all indiscriminately. “Selective sync” + exists for files and repositories (but not database content), which would permit + only less-sensitive projects to be replicated to a **secondary** node if desired. +- See also: [developing a data classification policy](https://gitlab.com/gitlab-com/security/issues/4). + +### What data backup and retention requirements have been defined for the application? + +- Geo is designed to provide replication of a certain subset of the application + data. It is part of the solution, rather than part of the problem. + +## End-Users + +### Who are the application's end‐users? + +- **Secondary** nodes are created in regions that are distant (in terms of + Internet latency) from the main GitLab installation (the **primary** node). They are + intended to be used by anyone who would ordinarily use the **primary** node, who finds + that the **secondary** node is closer to them (in terms of Internet latency). + +### How do the end‐users interact with the application? + +- **Secondary** nodes provide all the interfaces a **primary** node does + (notably a HTTP/HTTPS web application, and HTTP/HTTPS or SSH git repository + access), but is constrained to read-only activities. The principal use case is + envisioned to be cloning git repositories from the **secondary** node in favor of the + **primary** node, but end-users may use the GitLab web interface to view projects, + issues, merge requests, snippets, etc. + +### What security expectations do the end‐users have? + +- The replication process must be secure. It would typically be unacceptable to + transmit the entire database contents or all files and repositories across the + public Internet in plaintext, for instance. +- **Secondary** nodes must have the same access controls over its content as the + **primary** node - unauthenticated users must not be able to gain access to privileged + information on the **primary** node by querying the **secondary** node. +- Attackers must not be able to impersonate the **secondary** node to the **primary** node, and + thus gain access to privileged information. + +## Administrators + +### Who has administrative capabilities in the application? + +- Nothing Geo-specific. Any user where `admin: true` is set in the database is + considered an admin with super-user privileges. +- See also: [more granular access control](https://gitlab.com/gitlab-org/gitlab-ce/issues/32730) + (not geo-specific) +- Much of Geo’s integration (database replication, for instance) must be + configured with the application, typically by system administrators. + +### What administrative capabilities does the application offer? + +- **Secondary** nodes may be added, modified, or removed by users with + administrative access. +- The replication process may be controlled (start/stop) via the Sidekiq + administrative controls. + +## Network + +### What details regarding routing, switching, firewalling, and load‐balancing have been defined? + +- Geo requires the **primary** node and **secondary** node to be able to communicate with each + other across a TCP/IP network. In particular, the **secondary** nodes must be able to + access HTTP/HTTPS and PostgreSQL services on the **primary** node. + +### What core network devices support the application? + +- Varies from customer to customer. + +### What network performance requirements exist? + +- Maximum replication speeds between **primary** node and **secondary** node is limited by the + available bandwidth between sites. No hard requirements exist - time to complete + replication (and ability to keep up with changes on the **primary** node) is a function + of the size of the data set, tolerance for latency, and available network + capacity. + +### What private and public network links support the application? + +- Customers choose their own networks. As sites are intended to be + geographically separated, it is envisioned that replication traffic will pass + over the public Internet in a typical deployment, but this is not a requirement. + +## Systems + +### What operating systems support the application? + +- Geo imposes no additional restrictions on operating system (see the + [GitLab installation](https://about.gitlab.com/installation/) page for more + details), however we recommend using the operating systems listed in the [Geo documentation](index.md#requirements-for-running-geo). + +### What details regarding required OS components and lock‐down needs have been defined? + +- The supported installation method (Omnibus) packages most components itself. +- There are significant dependencies on the system-installed OpenSSH daemon (Geo + requires users to set up custom authentication methods) and the omnibus or + system-provided PostgreSQL daemon (it must be configured to listen on TCP, + additional users and replication slots must be added, etc). +- The process for dealing with security updates (for example, if there is a + significant vulnerability in OpenSSH or other services, and the customer + wants to patch those services on the OS) is identical to the non-Geo + situation: security updates to OpenSSH would be provided to the user via the + usual distribution channels. Geo introduces no delay there. + +## Infrastructure Monitoring + +### What network and system performance monitoring requirements have been defined? + +- None specific to Geo. + +### What mechanisms exist to detect malicious code or compromised application components? + +- None specific to Geo. + +### What network and system security monitoring requirements have been defined? + +- None specific to Geo. + +## Virtualization and Externalization + +### What aspects of the application lend themselves to virtualization? + +- All. + +## What virtualization requirements have been defined for the application? + +- Nothing Geo-specific, but everything in GitLab needs to have full + functionality in such an environment. + +### What aspects of the product may or may not be hosted via the cloud computing model? + +- GitLab is “cloud native” and this applies to Geo as much as to the rest of the + product. Deployment in clouds is a common and supported scenario. + +## If applicable, what approach(es) to cloud computing will be taken (Managed Hosting versus "Pure" Cloud, a "full machine" approach such as AWS-EC2 versus a "hosted database" approach such as AWS-RDS and Azure, etc)? + +- To be decided by our customers, according to their operational needs. + +## Environment + +### What frameworks and programming languages have been used to create the application? + +- Ruby on Rails, Ruby. + +### What process, code, or infrastructure dependencies have been defined for the application? + +- Nothing specific to Geo. + +### What databases and application servers support the application? + +- PostgreSQL >= 9.6, Redis, Sidekiq, Unicorn. + +### How will database connection strings, encryption keys, and other sensitive components be stored, accessed, and protected from unauthorized detection? + +- There are some Geo-specific values. Some are shared secrets which must be + securely transmitted from the **primary** node to the **secondary** node at setup time. Our + documentation recommends transmitting them from the **primary** node to the system + administrator via SSH, and then back out to the **secondary** node in the same manner. + In particular, this includes the PostgreSQL replication credentials and a secret + key (`db_key_base`) which is used to decrypt certain columns in the database. + The `db_key_base` secret is stored unencrypted on the filesystem, in + `/etc/gitlab/gitlab-secrets.json`, along with a number of other secrets. There is + no at-rest protection for them. + +## Data Processing + +### What data entry paths does the application support? + +- Data is entered via the web application exposed by GitLab itself. Some data is + also entered using system administration commands on the GitLab servers (e.g., + `gitlab-ctl set-primary-node`). +- **Secondary** nodes also receive inputs via PostgreSQL streaming replication from the **primary** node. + +### What data output paths does the application support? + +- **Primary** nodes output via PostgreSQL streaming replication to the **secondary** node. + Otherwise, principally via the web application exposed by GitLab itself, and via + SSH `git clone` operations initiated by the end-user. + +### How does data flow across the application's internal components? + +- **Secondary** nodes and **primary** nodes interact via HTTP/HTTPS (secured with JSON web + tokens) and via PostgreSQL streaming replication. +- Within a **primary** node or **secondary** node, the SSOT is the filesystem and the database + (including Geo tracking database on **secondary** node). The various internal components + are orchestrated to make alterations to these stores. + +### What data input validation requirements have been defined? + +- **Secondary** nodes must have a faithful replication of the **primary** node’s data. + +### What data does the application store and how? + +- Git repositories and files, tracking information related to the them, and the GitLab database contents. + +### What data is or may need to be encrypted and what key management requirements have been defined? + +- Neither **primary** nodes or **secondary** nodes encrypt Git repository or filesystem data at + rest. A subset of database columns are encrypted at rest using the `db_otp_key`. +- A static secret shared across all hosts in a GitLab deployment. +- In transit, data should be encrypted, although the application does permit + communication to proceed unencrypted. The two main transits are the **secondary** node’s + replication process for PostgreSQL, and for git repositories/files. Both should + be protected using TLS, with the keys for that managed via Omnibus per existing + configuration for end-user access to GitLab. + +### What capabilities exist to detect the leakage of sensitive data? + +- Comprehensive system logs exist, tracking every connection to GitLab and PostgreSQL. + +### What encryption requirements have been defined for data in transit - including transmission over WAN, LAN, SecureFTP, or publicly accessible protocols such as http: and https:? + +- Data must have the option to be encrypted in transit, and be secure against + both passive and active attack (e.g., MITM attacks should not be possible). + +## Access + +### What user privilege levels does the application support? + +- Geo adds one type of privilege: **secondary** nodes can access a special Geo API to + download files over HTTP/HTTPS, and to clone repositories using HTTP/HTTPS. + +### What user identification and authentication requirements have been defined? + +- **Secondary** nodes identify to Geo **primary** nodes via OAuth or JWT authentication + based on the shared database (HTTP access) or a PostgreSQL replication user (for + database replication). The database replication also requires IP-based access + controls to be defined. + +### What user authorization requirements have been defined? + +- **Secondary** nodes must only be able to *read* data. They are not currently able to mutate data on the **primary** node. + +### What session management requirements have been defined? + +- Geo JWTs are defined to last for only two minutes before needing to be regenerated. +- Geo JWTs are generated for one of the following specific scopes: + - Geo API access. + - Git access. + - LFS and File ID. + - Upload and File ID. + - Job Artifact and File ID. + +### What access requirements have been defined for URI and Service calls? + +- **Secondary** nodes make many calls to the **primary** node's API. This is how file + replication proceeds, for instance. This endpoint is only accessible with a JWT token. +- The **primary** node also makes calls to the **secondary** node to get status information. + +## Application Monitoring + +### What application auditing requirements have been defined? How are audit and debug logs accessed, stored, and secured? + +- Structured JSON log is written to the filesystem, and can also be ingested + into a Kibana installation for further analysis. diff --git a/doc/administration/geo/replication/troubleshooting.md b/doc/administration/geo/replication/troubleshooting.md new file mode 100644 index 00000000000..c5bdd36ba70 --- /dev/null +++ b/doc/administration/geo/replication/troubleshooting.md @@ -0,0 +1,514 @@ +# Geo Troubleshooting **[PREMIUM ONLY]** + +NOTE: **Note:** +This list is an attempt to document all the moving parts that can go wrong. +We are working into getting all this steps verified automatically in a +rake task in the future. + +Setting up Geo requires careful attention to details and sometimes it's easy to +miss a step. Here is a list of questions you should ask to try to detect +what you need to fix (all commands and path locations are for Omnibus installs): + +## First check the health of the **secondary** node + +Visit the **primary** node's **Admin Area > Geo** (`/admin/geo/nodes`) in +your browser. We perform the following health checks on each **secondary** node +to help identify if something is wrong: + +- Is the node running? +- Is the node's secondary database configured for streaming replication? +- Is the node's secondary tracking database configured? +- Is the node's secondary tracking database connected? +- Is the node's secondary tracking database up-to-date? + + + +For information on how to resolve common errors reported from the UI, see [common errors](#common-errors). + +If the UI is not working, or you are unable to log in, you can run the Geo +health check manually to get this information as well as a few more details. +This rake task can be run on an app node in the **primary** or **secondary** +Geo nodes: + +```sh +sudo gitlab-rake gitlab:geo:check +``` + +Example output: + +``` +Checking Geo ... + +GitLab Geo is available ... yes +GitLab Geo is enabled ... yes +GitLab Geo secondary database is correctly configured ... yes +Database replication enabled? ... yes +Database replication working? ... yes +GitLab Geo tracking database is configured to use Foreign Data Wrapper? ... yes +GitLab Geo tracking database Foreign Data Wrapper schema is up-to-date? ... yes +GitLab Geo HTTP(S) connectivity ... +* Can connect to the primary node ... yes +HTTP/HTTPS repository cloning is enabled ... yes +Machine clock is synchronized ... yes +Git user has default SSH configuration? ... yes +OpenSSH configured to use AuthorizedKeysCommand ... yes +GitLab configured to disable writing to authorized_keys file ... yes +GitLab configured to store new projects in hashed storage? ... yes +All projects are in hashed storage? ... yes + +Checking Geo ... Finished +``` + +Current sync information can be found manually by running this rake task on any +**secondary** app node: + +```sh +sudo gitlab-rake geo:status +``` + +Example output: + +``` +http://secondary.example.com/ +----------------------------------------------------- + GitLab Version: 11.10.4-ee + Geo Role: Secondary + Health Status: Healthy + Repositories: 289/289 (100%) + Verified Repositories: 289/289 (100%) + Wikis: 289/289 (100%) + Verified Wikis: 289/289 (100%) + LFS Objects: 8/8 (100%) + Attachments: 5/5 (100%) + CI job artifacts: 0/0 (0%) + Repositories Checked: 0/289 (0%) + Sync Settings: Full + Database replication lag: 0 seconds + Last event ID seen from primary: 10215 (about 2 minutes ago) + Last event ID processed by cursor: 10215 (about 2 minutes ago) + Last status report was: 2 minutes ago +``` + +## Is Postgres replication working? + +### Are my nodes pointing to the correct database instance? + +You should make sure your **primary** Geo node points to the instance with +writing permissions. + +Any **secondary** nodes should point only to read-only instances. + +### Can Geo detect my current node correctly? + +Geo uses the defined node from the **Admin Area > Geo** screen, and tries to match +it with the value defined in the `/etc/gitlab/gitlab.rb` configuration file. +The relevant line looks like: `external_url "http://gitlab.example.com"`. + +To check if the node on the current machine is correctly detected type: + +```sh +sudo gitlab-rails runner "puts Gitlab::Geo.current_node.inspect" +``` + +and expect something like: + +``` +#<GeoNode id: 2, schema: "https", host: "gitlab.example.com", port: 443, relative_url_root: "", primary: false, ...> +``` + +By running the command above, `primary` should be `true` when executed in +the **primary** node, and `false` on any **secondary** node. + +## How do I fix the message, "ERROR: replication slots can only be used if max_replication_slots > 0"? + +This means that the `max_replication_slots` PostgreSQL variable needs to +be set on the **primary** database. In GitLab 9.4, we have made this setting +default to 1. You may need to increase this value if you have more +**secondary** nodes. Be sure to restart PostgreSQL for this to take +effect. See the [PostgreSQL replication +setup][database-pg-replication] guide for more details. + +## How do I fix the message, "FATAL: could not start WAL streaming: ERROR: replication slot "geo_secondary_my_domain_com" does not exist"? + +This occurs when PostgreSQL does not have a replication slot for the +**secondary** node by that name. You may want to rerun the [replication +process](database.md) on the **secondary** node . + +## How do I fix the message, "Command exceeded allowed execution time" when setting up replication? + +This may happen while [initiating the replication process][database-start-replication] on the **secondary** node, +and indicates that your initial dataset is too large to be replicated in the default timeout (30 minutes). + +Re-run `gitlab-ctl replicate-geo-database`, but include a larger value for +`--backup-timeout`: + +```sh +sudo gitlab-ctl \ + replicate-geo-database \ + --host=<primary_node_hostname> \ + --slot-name=<secondary_slot_name> \ + --backup-timeout=21600 +``` + +This will give the initial replication up to six hours to complete, rather than +the default thirty minutes. Adjust as required for your installation. + +## How do I fix the message, "PANIC: could not write to file 'pg_xlog/xlogtemp.123': No space left on device" + +Determine if you have any unused replication slots in the **primary** database. This can cause large amounts of +log data to build up in `pg_xlog`. Removing the unused slots can reduce the amount of space used in the `pg_xlog`. + +1. Start a PostgreSQL console session: + + ```sh + sudo gitlab-psql gitlabhq_production + ``` + + > Note that using `gitlab-rails dbconsole` will not work, because managing replication slots requires superuser permissions. + +1. View your replication slots with: + + ```sql + SELECT * FROM pg_replication_slots; + ``` + +Slots where `active` is `f` are not active. + +- When this slot should be active, because you have a **secondary** node configured using that slot, + log in to that **secondary** node and check the PostgreSQL logs why the replication is not running. + +- If you are no longer using the slot (e.g. you no longer have Geo enabled), you can remove it with in the + PostgreSQL console session: + + ```sql + SELECT pg_drop_replication_slot('<name_of_extra_slot>'); + ``` + +## Very large repositories never successfully synchronize on the **secondary** node + +GitLab places a timeout on all repository clones, including project imports +and Geo synchronization operations. If a fresh `git clone` of a repository +on the primary takes more than a few minutes, you may be affected by this. +To increase the timeout, add the following line to `/etc/gitlab/gitlab.rb` +on the **secondary** node: + +```ruby +gitlab_rails['gitlab_shell_git_timeout'] = 10800 +``` + +Then reconfigure GitLab: + +```sh +sudo gitlab-ctl reconfigure +``` + +This will increase the timeout to three hours (10800 seconds). Choose a time +long enough to accommodate a full clone of your largest repositories. + +## How to reset Geo **secondary** node replication + +If you get a **secondary** node in a broken state and want to reset the replication state, +to start again from scratch, there are a few steps that can help you: + +1. Stop Sidekiq and the Geo LogCursor + + It's possible to make Sidekiq stop gracefully, but making it stop getting new jobs and + wait until the current jobs to finish processing. + + You need to send a **SIGTSTP** kill signal for the first phase and them a **SIGTERM** + when all jobs have finished. Otherwise just use the `gitlab-ctl stop` commands. + + ```sh + gitlab-ctl status sidekiq + # run: sidekiq: (pid 10180) <- this is the PID you will use + kill -TSTP 10180 # change to the correct PID + + gitlab-ctl stop sidekiq + gitlab-ctl stop geo-logcursor + ``` + + You can watch sidekiq logs to know when sidekiq jobs processing have finished: + + ```sh + gitlab-ctl tail sidekiq + ``` + +1. Rename repository storage folders and create new ones + + ```sh + mv /var/opt/gitlab/git-data/repositories /var/opt/gitlab/git-data/repositories.old + mkdir -p /var/opt/gitlab/git-data/repositories + chown git:git /var/opt/gitlab/git-data/repositories + ``` + + TIP: **Tip** + You may want to remove the `/var/opt/gitlab/git-data/repositories.old` in the future + as soon as you confirmed that you don't need it anymore, to save disk space. + +1. _(Optional)_ Rename other data folders and create new ones + + CAUTION: **Caution**: + You may still have files on the **secondary** node that have been removed from **primary** node but + removal have not been reflected. If you skip this step, they will never be removed + from this Geo node. + + Any uploaded content like file attachments, avatars or LFS objects are stored in a + subfolder in one of the two paths below: + + 1. /var/opt/gitlab/gitlab-rails/shared + 1. /var/opt/gitlab/gitlab-rails/uploads + + To rename all of them: + + ```sh + gitlab-ctl stop + + mv /var/opt/gitlab/gitlab-rails/shared /var/opt/gitlab/gitlab-rails/shared.old + mkdir -p /var/opt/gitlab/gitlab-rails/shared + + mv /var/opt/gitlab/gitlab-rails/uploads /var/opt/gitlab/gitlab-rails/uploads.old + mkdir -p /var/opt/gitlab/gitlab-rails/uploads + ``` + + Reconfigure in order to recreate the folders and make sure permissions and ownership + are correctly + + ```sh + gitlab-ctl reconfigure + ``` + +1. Reset the Tracking Database + + ```sh + gitlab-rake geo:db:reset + ``` + +1. Restart previously stopped services + + ```sh + gitlab-ctl start + ``` + +## How do I fix a "Foreign Data Wrapper (FDW) is not configured" error? + +When setting up Geo, you might see this warning in the `gitlab-rake +gitlab:geo:check` output: + +``` +GitLab Geo tracking database Foreign Data Wrapper schema is up-to-date? ... foreign data wrapper is not configured +``` + +There are a few key points to remember: + +1. The FDW settings are configured on the Geo **tracking** database. +1. The configured foreign server enables a login to the Geo +**secondary**, read-only database. + +By default, the Geo secondary and tracking database are running on the +same host on different ports. That is, 5432 and 5431 respectively. + +### Checking configuration + +NOTE: **Note:** +The following steps are for Omnibus installs only. Using Geo with source-based installs was **deprecated** in GitLab 11.5. + +To check the configuration: + +1. Enter the database console: + + ```sh + gitlab-geo-psql + ``` + +1. Check whether any tables are present. If everything is working, you +should see something like this: + + ```sql + gitlabhq_geo_production=# SELECT * from information_schema.foreign_tables; + foreign_table_catalog | foreign_table_schema | foreign_table_name | foreign_server_catalog | foreign_server_n + ame + -------------------------+----------------------+-------------------------------------------------+-------------------------+----------------- + ---- + gitlabhq_geo_production | gitlab_secondary | abuse_reports | gitlabhq_geo_production | gitlab_secondary + gitlabhq_geo_production | gitlab_secondary | appearances | gitlabhq_geo_production | gitlab_secondary + gitlabhq_geo_production | gitlab_secondary | application_setting_terms | gitlabhq_geo_production | gitlab_secondary + gitlabhq_geo_production | gitlab_secondary | application_settings | gitlabhq_geo_production | gitlab_secondary + <snip> + ``` + + However, if the query returns with `0 rows`, then continue onto the next steps. + +1. Check that the foreign server mapping is correct via `\des+`. The + results should look something like this: + + ```sql + gitlabhq_geo_production=# \des+ + List of foreign servers + -[ RECORD 1 ]--------+------------------------------------------------------------ + Name | gitlab_secondary + Owner | gitlab-psql + Foreign-data wrapper | postgres_fdw + Access privileges | "gitlab-psql"=U/"gitlab-psql" + + | gitlab_geo=U/"gitlab-psql" + Type | + Version | + FDW Options | (host '0.0.0.0', port '5432', dbname 'gitlabhq_production') + Description | + ``` + + NOTE: **Note:** Pay particular attention to the host and port under + FDW options. That configuration should point to the Geo secondary + database. + + If you need to experiment with changing the host or password, the + following queries demonstrate how: + + ```sql + ALTER SERVER gitlab_secondary OPTIONS (SET host '<my_new_host>'); + ALTER SERVER gitlab_secondary OPTIONS (SET port 5432); + ``` + + If you change the host and/or port, you will also have to adjust the + following settings in `/etc/gitlab/gitlab.rb` and run `gitlab-ctl + reconfigure`: + + - `gitlab_rails['db_host']` + - `gitlab_rails['db_port']` + +1. Check that the user mapping is configured properly via `\deu+`: + + ```sql + gitlabhq_geo_production=# \deu+ + List of user mappings + Server | User name | FDW Options + ------------------+------------+-------------------------------------------------------------------------------- + gitlab_secondary | gitlab_geo | ("user" 'gitlab', password 'YOUR-PASSWORD-HERE') + (1 row) + ``` + + Make sure the password is correct. You can test that logins work by running `psql`: + + ```sh + # Connect to the tracking database as the `gitlab_geo` user + sudo \ + -u git /opt/gitlab/embedded/bin/psql \ + -h /var/opt/gitlab/geo-postgresql \ + -p 5431 \ + -U gitlab_geo \ + -W \ + -d gitlabhq_geo_production + ``` + + If you need to correct the password, the following query shows how: + + ```sql + ALTER USER MAPPING FOR gitlab_geo SERVER gitlab_secondary OPTIONS (SET password '<my_new_password>'); + ``` + + If you change the user or password, you will also have to adjust the + following settings in `/etc/gitlab/gitlab.rb` and run `gitlab-ctl + reconfigure`: + + - `gitlab_rails['db_username']` + - `gitlab_rails['db_password']` + + If you are using [PgBouncer in front of the secondary + database](database.md#pgbouncer-support-optional), be sure to update + the following settings: + + - `geo_postgresql['fdw_external_user']` + - `geo_postgresql['fdw_external_password']` + +### Manual reload of FDW schema + +If you're still unable to get FDW working, you may want to try a manual +reload of the FDW schema. To manually reload the FDW schema: + +1. On the node running the Geo tracking database, enter the PostgreSQL console via + the `gitlab_geo` user: + + ```sh + sudo \ + -u git /opt/gitlab/embedded/bin/psql \ + -h /var/opt/gitlab/geo-postgresql \ + -p 5431 \ + -U gitlab_geo \ + -W \ + -d gitlabhq_geo_production + ``` + + Be sure to adjust the port and hostname for your configuration. You + may be asked to enter a password. + +1. Reload the schema via: + + ```sql + DROP SCHEMA IF EXISTS gitlab_secondary CASCADE; + CREATE SCHEMA gitlab_secondary; + GRANT USAGE ON FOREIGN SERVER gitlab_secondary TO gitlab_geo; + IMPORT FOREIGN SCHEMA public FROM SERVER gitlab_secondary INTO gitlab_secondary; + ``` + +1. Test that queries work: + + ```sql + SELECT * from information_schema.foreign_tables; + SELECT * FROM gitlab_secondary.projects limit 1; + ``` + +[database-start-replication]: database.md#step-3-initiate-the-replication-process +[database-pg-replication]: database.md#postgresql-replication + +## Common errors + +This section documents common errors reported in the admin UI and how to fix them. + +### Geo database configuration file is missing + +GitLab cannot find or doesn't have permission to access the `database_geo.yml` configuration file. + +In an Omnibus GitLab installation, the file should be in `/var/opt/gitlab/gitlab-rails/etc`. +If it doesn't exist or inadvertent changes have been made to it, run `sudo gitlab-ctl reconfigure` to restore it to its correct state. + + +If this path is mounted on a remote volume, please check your volume configuration and that it has correct permissions. + +### Geo node has a database that is writable which is an indication it is not configured for replication with the primary node. + +This error refers to a problem with the database replica on a **secondary** node, +which Geo expects to have access to. It usually means, either: + +- An unsupported replication method was used (for example, logical replication). +- The instructions to setup a [Geo database replication](database.md) were not followed correctly. + +A common source of confusion with **secondary** nodes is that it requires two separate +PostgreSQL instances: + +- A read-only replica of the **primary** node. +- A regular, writable instance that holds replication metadata. That is, the Geo tracking database. + +### Geo node does not appear to be replicating the database from the primary node. + +The most common problems that prevent the database from replicating correctly are: + +- **Secondary** nodes cannot reach the **primary** node. Check credentials, firewall rules, etc. +- SSL certificate problems. Make sure you copied `/etc/gitlab/gitlab-secrets.json` from the **primary** node. +- Database storage disk is full. +- Database replication slot is misconfigured. +- Database is not using a replication slot or another alternative and cannot catch-up because WAL files were purged. + +Make sure you follow the [Geo database replication](database.md) instructions for supported configuration. + +### Geo database version (...) does not match latest migration (...) + +If you are using GitLab Omnibus installation, something might have failed during upgrade. You can: + +- Run `sudo gitlab-ctl reconfigure`. +- Manually trigger the database migration by running: `sudo gitlab-rake geo:db:migrate` as root on the **secondary** node. + +### Geo database is not configured to use Foreign Data Wrapper + +This error means the Geo Tracking Database doesn't have the FDW server and credentials +configured. + +See [How do I fix a "Foreign Data Wrapper (FDW) is not configured" error?](#how-do-i-fix-a-foreign-data-wrapper-fdw-is-not-configured-error). diff --git a/doc/administration/geo/replication/tuning.md b/doc/administration/geo/replication/tuning.md new file mode 100644 index 00000000000..1943f2230df --- /dev/null +++ b/doc/administration/geo/replication/tuning.md @@ -0,0 +1,17 @@ +# Tuning Geo **[PREMIUM ONLY]** + +## Changing the sync capacity values + +In the Geo admin page (`/admin/geo/nodes`), there are several variables that +can be tuned to improve performance of Geo: + +- Repository sync capacity. +- File sync capacity. + +Increasing these values will increase the number of jobs that are scheduled. +However, this may not lead to more downloads in parallel unless the number of +available Sidekiq threads is also increased. For example, if repository sync +capacity is increased from 25 to 50, you may also want to increase the number +of Sidekiq threads from 25 to 50. See the +[Sidekiq concurrency documentation](../../operations/extra_sidekiq_processes.md#number-of-threads) +for more details. diff --git a/doc/administration/geo/replication/updating_the_geo_nodes.md b/doc/administration/geo/replication/updating_the_geo_nodes.md new file mode 100644 index 00000000000..933a75c47d8 --- /dev/null +++ b/doc/administration/geo/replication/updating_the_geo_nodes.md @@ -0,0 +1,448 @@ +# Updating the Geo nodes **[PREMIUM ONLY]** + +Depending on which version of Geo you are updating to/from, there may be +different steps. + +## General update steps + +In order to update the Geo nodes when a new GitLab version is released, +all you need to do is update GitLab itself: + +1. Log into each node (**primary** and **secondary** nodes). +1. [Update GitLab][update]. +1. [Update tracking database on **secondary** node](#update-tracking-database-on-secondary-node) when + the tracking database is enabled. +1. [Test](#check-status-after-updating) **primary** and **secondary** nodes, and check version in each. + +## Upgrading to GitLab 10.8 + +Before 10.8, broadcast messages would not propagate without flushing the cache on the **secondary** nodes. This has been fixed in 10.8, but requires one last cache flush on each **secondary** node: + +```sh +sudo gitlab-rake cache:clear +``` + +## Upgrading to GitLab 10.6 + +In 10.4, we started to recommend that you define a password for database user (`gitlab`). + +We now require this change as we use this password to enable the Foreign Data Wrapper, as a way to optimize +the Geo Tracking Database. We are also improving security by disabling the use of **trust** +authentication method. + +1. **[primary]** Login to your **primary** node and run: + + ```sh + gitlab-ctl pg-password-md5 gitlab + # Enter password: <your_password_here> + # Confirm password: <your_password_here> + # fca0b89a972d69f00eb3ec98a5838484 + ``` + + Copy the generated hash and edit `/etc/gitlab/gitlab.rb`: + + ```ruby + # Fill with the hash generated by `gitlab-ctl pg-password-md5 gitlab` + postgresql['sql_user_password'] = '<md5_hash_of_your_password>' + + # Every node that runs Unicorn or Sidekiq needs to have the database + # password specified as below. If you have a high-availability setup, this + # must be present in all application nodes. + gitlab_rails['db_password'] = '<your_password_here>' + ``` + + Still in the configuration file, locate and remove the `trust_auth_cidr_address`: + + ```ruby + postgresql['trust_auth_cidr_addresses'] = ['127.0.0.1/32','1.2.3.4/32'] # <- Remove this + ``` + +1. **[primary]** Reconfigure and restart: + + ```sh + sudo gitlab-ctl reconfigure + sudo gitlab-ctl restart + ``` + +1. **[secondary]** Login to all **secondary** nodes and edit `/etc/gitlab/gitlab.rb`: + + ```ruby + # Fill with the hash generated by `gitlab-ctl pg-password-md5 gitlab` + postgresql['sql_user_password'] = '<md5_hash_of_your_password>' + + # Every node that runs Unicorn or Sidekiq needs to have the database + # password specified as below. If you have a high-availability setup, this + # must be present in all application nodes. + gitlab_rails['db_password'] = '<your_password_here>' + + # Enable Foreign Data Wrapper + geo_secondary['db_fdw'] = true + + # Secondary address in CIDR format, for example '5.6.7.8/32' + postgresql['md5_auth_cidr_addresses'] = ['<secondary_node_ip>/32'] + ``` + + Still in the configuration file, locate and remove the `trust_auth_cidr_address`: + + ```ruby + postgresql['trust_auth_cidr_addresses'] = ['127.0.0.1/32','5.6.7.8/32'] # <- Remove this + ``` + +1. **[secondary]** Reconfigure and restart: + + ```sh + sudo gitlab-ctl reconfigure + sudo gitlab-ctl restart + ``` + +## Upgrading to GitLab 10.5 + +For Geo Disaster Recovery to work with minimum downtime, your **secondary** node +should use the same set of secrets as the **primary** node. However, setup instructions +prior to the 10.5 release only synchronized the `db_key_base` secret. + +To rectify this error on existing installations, you should **overwrite** the +contents of `/etc/gitlab/gitlab-secrets.json` on each **secondary** node with the +contents of `/etc/gitlab/gitlab-secrets.json` on the **primary** node, then run the +following command on each **secondary** node: + +```sh +sudo gitlab-ctl reconfigure +``` + +If you do not perform this step, you may find that two-factor authentication +[is broken following DR](../disaster_recovery/index.html#i-followed-the-disaster-recovery-instructions-and-now-two-factor-auth-is-broken). + +To prevent SSH requests to the newly promoted **primary** node from failing +due to SSH host key mismatch when updating the **primary** node domain's DNS record +you should perform the step to [Manually replicate **primary** SSH host keys](configuration.md#step-2-manually-replicate-the-primary-nodes-ssh-host-keys) in each +**secondary** node. + +## Upgrading to GitLab 10.4 + +There are no Geo-specific steps to take! + +## Upgrading to GitLab 10.3 + +### Support for SSH repository synchronization removed + +In GitLab 10.2, synchronizing secondaries over SSH was deprecated. In 10.3, +support is removed entirely. All installations will switch to the HTTP/HTTPS +cloning method instead. Before upgrading, ensure that all your Geo nodes are +configured to use this method and that it works for your installation. In +particular, ensure that [Git access over HTTP/HTTPS is enabled](configuration.md#step-6-enable-git-access-over-httphttps). + +Synchronizing repositories over the public Internet using HTTP is insecure, so +you should ensure that you have HTTPS configured before upgrading. Note that +file synchronization is **also** insecure in these cases! + +## Upgrading to GitLab 10.2 + +### Secure PostgreSQL replication + +Support for TLS-secured PostgreSQL replication has been added. If you are +currently using PostgreSQL replication across the open internet without an +external means of securing the connection (e.g., a site-to-site VPN), then you +should immediately reconfigure your **primary** and **secondary** PostgreSQL instances +according to the [updated instructions][database]. + +If you *are* securing the connections externally and wish to continue doing so, +ensure you include the new option `--sslmode=prefer` in future invocations of +`gitlab-ctl replicate-geo-database`. + +### HTTPS repository sync + +Support for replicating repositories and wikis over HTTP/HTTPS has been added. +Replicating over SSH has been deprecated, and support for this option will be +removed in a future release. + +To switch to HTTP/HTTPS replication, log into the **primary** node as an admin and visit +**Admin Area > Geo** (`/admin/geo/nodes`). For each **secondary** node listed, +press the "Edit" button, change the "Repository cloning" setting from +"SSH (deprecated)" to "HTTP/HTTPS", and press "Save changes". This should take +effect immediately. + +Any new secondaries should be created using HTTP/HTTPS replication - this is the +default setting. + +After you've verified that HTTP/HTTPS replication is working, you should remove +the now-unused SSH keys from your secondaries, as they may cause problems if the +**secondary** node if ever promoted to a **primary** node: + +1. **[secondary]** Login to **all** your **secondary** nodes and run: + + ```ruby + sudo -u git -H rm ~git/.ssh/id_rsa ~git/.ssh/id_rsa.pub + ``` + +### Hashed Storage + +CAUTION: **Warning:** +Hashed storage is in **Alpha**. It is considered experimental and not +production-ready. See [Hashed Storage] for more detail. + +If you previously enabled Hashed Storage and migrated all your existing +projects to Hashed Storage, disabling hashed storage will not migrate projects +to their previous project based storage path. As such, once enabled and +migrated we recommend leaving Hashed Storage enabled. + +## Upgrading to GitLab 10.1 + +CAUTION: **Warning:** +Hashed storage is in **Alpha**. It is considered experimental and not +production-ready. See [Hashed Storage] for more detail. + +[Hashed storage] was introduced in GitLab 10.0, and a [migration path][hashed-migration] +for existing repositories was added in GitLab 10.1. + +## Upgrading to GitLab 10.0 + +Since GitLab 10.0, we require all **Geo** systems to [use SSH key lookups via +the database][ssh-fast-lookup] to avoid having to maintain consistency of the +`authorized_keys` file for SSH access. Failing to do this will prevent users +from being able to clone via SSH. + +Note that in older versions of Geo, attachments downloaded on the **secondary** +nodes would be saved to the wrong directory. We recommend that you do the +following to clean this up. + +On the **secondary** Geo nodes, run as root: + +```sh +mv /var/opt/gitlab/gitlab-rails/working /var/opt/gitlab/gitlab-rails/working.old +mkdir /var/opt/gitlab/gitlab-rails/working +chmod 700 /var/opt/gitlab/gitlab-rails/working +chown git:git /var/opt/gitlab/gitlab-rails/working +``` + +You may delete `/var/opt/gitlab/gitlab-rails/working.old` any time. + +Once this is done, we advise restarting GitLab on the **secondary** nodes for the +new working directory to be used: + +```sh +sudo gitlab-ctl restart +``` + +## Upgrading from GitLab 9.3 or older + +If you started running Geo on GitLab 9.3 or older, we recommend that you +resync your **secondary** PostgreSQL databases to use replication slots. If you +started using Geo with GitLab 9.4 or 10.x, no further action should be +required because replication slots are used by default. However, if you +started with GitLab 9.3 and upgraded later, you should still follow the +instructions below. + +When in doubt, it does not hurt to do a resync. The easiest way to do this in +Omnibus is the following: + + 1. Make sure you have Omnibus GitLab on the **primary** server. + 1. Run `gitlab-ctl reconfigure` and `gitlab-ctl restart postgresql`. This will enable replication slots on the **primary** database. + 1. Check the steps about defining `postgresql['sql_user_password']`, `gitlab_rails['db_password']`. + 1. Make sure `postgresql['max_replication_slots']` matches the number of **secondary** Geo nodes locations. + 1. Install GitLab on the **secondary** server. + 1. Re-run the [database replication process][database-replication]. + +## Special update notes for 9.0.x + +> **IMPORTANT**: +With GitLab 9.0, the PostgreSQL version is upgraded to 9.6 and manual steps are +required in order to update the **secondary** nodes and keep the Streaming +Replication working. Downtime is required, so plan ahead. + +The following steps apply only if you upgrade from a 8.17 GitLab version to +9.0+. For previous versions, update to GitLab 8.17 first before attempting to +upgrade to 9.0+. + +--- + +Make sure to follow the steps in the exact order as they appear below and pay +extra attention in what node (either **primary** or **secondary**) you execute them! Each step +is prepended with the relevant node for better clarity: + +1. **[secondary]** Login to **all** your **secondary** nodes and stop all services: + + ```ruby + sudo gitlab-ctl stop + ``` + +1. **[secondary]** Make a backup of the `recovery.conf` file on **all** + **secondary** nodes to preserve PostgreSQL's credentials: + + ```sh + sudo cp /var/opt/gitlab/postgresql/data/recovery.conf /var/opt/gitlab/ + ``` + +1. **[primary]** Update the **primary** node to GitLab 9.0 following the + [regular update docs][update]. At the end of the update, the **primary** node + will be running with PostgreSQL 9.6. + +1. **[primary]** To prevent a de-synchronization of the repository replication, + stop all services except `postgresql` as we will use it to re-initialize the + **secondary** node's database: + + ```sh + sudo gitlab-ctl stop + sudo gitlab-ctl start postgresql + ``` + +1. **[secondary]** Run the following steps on each of the **secondary** nodes: + + 1. **[secondary]** Stop all services: + + ```sh + sudo gitlab-ctl stop + ``` + + 1. **[secondary]** Prevent running database migrations: + + ```sh + sudo touch /etc/gitlab/skip-auto-migrations + ``` + + 1. **[secondary]** Move the old database to another directory: + + ```sh + sudo mv /var/opt/gitlab/postgresql{,.bak} + ``` + + 1. **[secondary]** Update to GitLab 9.0 following the [regular update docs][update]. + At the end of the update, the node will be running with PostgreSQL 9.6. + + 1. **[secondary]** Make sure all services are up: + + ```sh + sudo gitlab-ctl start + ``` + + 1. **[secondary]** Reconfigure GitLab: + + ```sh + sudo gitlab-ctl reconfigure + ``` + + 1. **[secondary]** Run the PostgreSQL upgrade command: + + ```sh + sudo gitlab-ctl pg-upgrade + ``` + + 1. **[secondary]** See the stored credentials for the database that you will + need to re-initialize the replication: + + ```sh + sudo grep -s primary_conninfo /var/opt/gitlab/recovery.conf + ``` + + 1. **[secondary]** Create the `replica.sh` script as described in the + [database configuration document][database-source-replication]. + + 1. 1. **[secondary]** Save the snippet below in a file, let's say `/tmp/replica.sh`. Modify the + embedded paths if necessary: + + ``` + #!/bin/bash + + PORT="5432" + USER="gitlab_replicator" + echo --------------------------------------------------------------- + echo WARNING: Make sure this script is run from the secondary server + echo --------------------------------------------------------------- + echo + echo Enter the IP or FQDN of the primary PostgreSQL server + read HOST + echo Enter the password for $USER@$HOST + read -s PASSWORD + echo Enter the required sslmode + read SSLMODE + + echo Stopping PostgreSQL and all GitLab services + sudo service gitlab stop + sudo service postgresql stop + + echo Backing up postgresql.conf + sudo -u postgres mv /var/opt/gitlab/postgresql/data/postgresql.conf /var/opt/gitlab/postgresql/ + + echo Cleaning up old cluster directory + sudo -u postgres rm -rf /var/opt/gitlab/postgresql/data + + echo Starting base backup as the replicator user + echo Enter the password for $USER@$HOST + sudo -u postgres /opt/gitlab/embedded/bin/pg_basebackup -h $HOST -D /var/opt/gitlab/postgresql/data -U gitlab_replicator -v -x -P + + echo Writing recovery.conf file + sudo -u postgres bash -c "cat > /var/opt/gitlab/postgresql/data/recovery.conf <<- _EOF1_ + standby_mode = 'on' + primary_conninfo = 'host=$HOST port=$PORT user=$USER password=$PASSWORD sslmode=$SSLMODE' + _EOF1_ + " + + echo Restoring postgresql.conf + sudo -u postgres mv /var/opt/gitlab/postgresql/postgresql.conf /var/opt/gitlab/postgresql/data/ + + echo Starting PostgreSQL + sudo service postgresql start + ``` + + 1. **[secondary]** Run the recovery script using the credentials from the + previous step: + + ```sh + sudo bash /tmp/replica.sh + ``` + + 1. **[secondary]** Reconfigure GitLab: + + ```sh + sudo gitlab-ctl reconfigure + ``` + + 1. **[secondary]** Start all services: + + ```sh + sudo gitlab-ctl start + ``` + + 1. **[secondary]** Repeat the steps for the remaining **secondary** nodes. + +1. **[primary]** After all **secondary** nodes are updated, start all services in + **primary** node: + + ```sh + sudo gitlab-ctl start + ``` + +## Check status after updating + +Now that the update process is complete, you may want to check whether +everything is working correctly: + +1. Run the Geo raketask on all nodes, everything should be green: + + ```sh + sudo gitlab-rake gitlab:geo:check + ``` + +1. Check the **primary** node's Geo dashboard for any errors. +1. Test the data replication by pushing code to the **primary** node and see if it + is received by **secondary** nodes. + +## Update tracking database on **secondary** node + +After updating a **secondary** node, you might need to run migrations on +the tracking database. The tracking database was added in GitLab 9.1, +and it is required since 10.0. + +1. Run database migrations on tracking database: + + ```sh + sudo gitlab-rake geo:db:migrate + ``` + +1. Repeat this step for each **secondary** node. + +[update]: ../../../update/README.md +[database]: database.md +[Hashed Storage]: ../../repository_storage_types.md +[hashed-migration]: ../../raketasks/storage.md +[ssh-fast-lookup]: ../../operations/fast_ssh_key_lookup.md diff --git a/doc/administration/geo/replication/using_a_geo_server.md b/doc/administration/geo/replication/using_a_geo_server.md new file mode 100644 index 00000000000..f1f1fe48748 --- /dev/null +++ b/doc/administration/geo/replication/using_a_geo_server.md @@ -0,0 +1,18 @@ +[//]: # (Please update EE::GitLab::GeoGitAccess::GEO_SERVER_DOCS_URL if this file is moved) + +# Using a Geo Server **[PREMIUM ONLY]** + +After you set up the [database replication and configure the Geo nodes][req], use your closest GitLab node as you would a normal standalone GitLab instance. + +Pushing directly to a **secondary** node (for both HTTP, SSH including git-lfs) was [introduced](https://about.gitlab.com/2018/09/22/gitlab-11-3-released/) in [GitLab Premium](https://about.gitlab.com/pricing/#self-managed) 11.3. + +Example of the output you will see when pushing to a **secondary** node: + +```bash +$ git push +> GitLab: You're pushing to a Geo secondary. +> GitLab: We'll help you by proxying this request to the primary: ssh://git@primary.geo/user/repo.git +Everything up-to-date +``` + +[req]: index.md#setup-instructions diff --git a/doc/administration/git_protocol.md b/doc/administration/git_protocol.md index 11b2adeeeb8..1780e1babe9 100644 --- a/doc/administration/git_protocol.md +++ b/doc/administration/git_protocol.md @@ -5,13 +5,16 @@ description: "Set and configure Git protocol v2" # Configuring Git Protocol v2 > [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/issues/46555) in GitLab 11.4. -> [Temporarily disabled](https://gitlab.com/gitlab-org/gitlab-ce/issues/55769) in GitLab 11.5.8, 11.6.6, 11.7.1, and 11.8+ +> Temporarily disabled (see [confidential issue](../user/project/issues/confidential_issues.md) +> `https://gitlab.com/gitlab-org/gitlab-ce/issues/55769`) in GitLab 11.5.8, 11.6.6, 11.7.1, and 11.8+. NOTE: **Note:** -Git protocol v2 support has been [temporarily disabled](https://gitlab.com/gitlab-org/gitlab-ce/issues/55769), -as a feature used to hide certain internal references does not function when it +Git protocol v2 support has been temporarily disabled +because a feature used to hide certain internal references does not function when it is enabled, and this has a security impact. Once this problem has been resolved, -protocol v2 support will be re-enabled. +protocol v2 support will be re-enabled. For more information, see the +[confidential issue](../user/project/issues/confidential_issues.md) +`https://gitlab.com/gitlab-org/gitlab-ce/issues/55769`. Git protocol v2 improves the v1 wire protocol in several ways and is enabled by default in GitLab for HTTP requests. In order to enable SSH, @@ -28,7 +31,7 @@ From the client side, `git` `v2.18.0` or newer must be installed. From the server side, if we want to configure SSH we need to set the `sshd` server to accept the `GIT_PROTOCOL` environment. -In installations using [GitLab Helm Charts](../install/kubernetes/gitlab_chart.md) +In installations using [GitLab Helm Charts](https://docs.gitlab.com/charts/) and [All-in-one docker image](https://docs.gitlab.com/omnibus/docker/), the SSH service is already configured to accept the `GIT_PROTOCOL` environment and users need not do anything more. diff --git a/doc/administration/gitaly/index.md b/doc/administration/gitaly/index.md index 0795d3dad40..53a85dfad6c 100644 --- a/doc/administration/gitaly/index.md +++ b/doc/administration/gitaly/index.md @@ -45,9 +45,17 @@ installations that are larger than a single machine. Most installations will be better served with the default configuration used by Omnibus and the GitLab source installation guide. -Starting with GitLab 11.4, Gitaly is a replacement for NFS except -when the [Elastic Search indexer](https://gitlab.com/gitlab-org/gitlab-elasticsearch-indexer) -is used. +Starting with GitLab 11.4, Gitaly is able to serve all Git requests without +needed a shared NFS mount for Git repository data. +Between 11.4 and 11.8 the exception was the +[Elastic Search indexer](https://gitlab.com/gitlab-org/gitlab-elasticsearch-indexer). +But since 11.8 the indexer uses Gitaly for data access as well. NFS can still +be leveraged for redudancy on block level of the Git data. But only has to +be mounted on the Gitaly server. + +NOTE: **Note:** While Gitaly can be used as a replacement for NFS, we do not recommend +using EFS as it may impact GitLab's performance. Please review the [relevant documentation](../high_availability/nfs.md#avoid-using-awss-elastic-file-system-efs) +for more details. ### Network architecture @@ -62,13 +70,16 @@ is used. - Gitaly addresses must be specified in such a way that they resolve correctly for ALL Gitaly clients - Gitaly clients are: unicorn, sidekiq, gitlab-workhorse, - gitlab-shell, and Gitaly itself + gitlab-shell, Elasticsearch Indexer, and Gitaly itself - special case: a Gitaly server must be able to make RPC calls **to itself** via its own (Gitaly address, Gitaly token) pair as specified in `gitlab-rails/config/gitlab.yml` - Gitaly servers must not be exposed to the public internet -Gitaly network traffic is unencrypted so you should use a firewall to +Gitaly network traffic is unencrypted by default, but supports +[TLS](#tls-support). Authentication is done through a static token. + +NOTE: **Note:** Gitaly network traffic is unencrypted so we recommend a firewall to restrict access to your Gitaly server. Below we describe how to configure a Gitaly server at address @@ -76,9 +87,19 @@ Below we describe how to configure a Gitaly server at address your GitLab installation has two repository storages, `default` and `storage1`. +### Installation + +First install Gitaly using either Omnibus or from source. + +Omnibus: [Download/install](https://about.gitlab.com/installation) the Omnibus GitLab +package you want using **steps 1 and 2** from the GitLab downloads page but +**_do not_** provide the `EXTERNAL_URL=` value. + +Source: [Install Gitaly](../../install/installation.md#install-gitaly) + ### Client side token configuration -Start by configuring a token on the client side. +Configure a token on the client side. Omnibus installations: @@ -104,7 +125,7 @@ changes to be picked up. Next, on the Gitaly server, we need to configure storage paths, enable the network listener and configure the token. -Note: if you want to reduce the risk of downtime when you enable +NOTE: **Note:** if you want to reduce the risk of downtime when you enable authentication you can temporarily disable enforcement, see [the documentation on configuring Gitaly authentication](https://gitlab.com/gitlab-org/gitaly/blob/master/doc/configuration/README.md#authentication) @@ -116,12 +137,17 @@ the Gitaly server. The easiest way to accomplish this is to copy `/etc/gitlab/gi from an existing GitLab server to the Gitaly server. Without this shared secret, Git operations in GitLab will result in an API error. -> **NOTE:** In most or all cases the storage paths below end in `/repositories` which is +NOTE: **Note:** In most or all cases the storage paths below end in `/repositories` which is different than `path` in `git_data_dirs` of Omnibus installations. Check the directory layout on your Gitaly server to be sure. Omnibus installations: +<!-- +updates to following example must also be made at +https://gitlab.com/charts/gitlab/blob/master/doc/advanced/external-gitaly/external-omnibus-gitaly.md#configure-omnibus-gitlab +--> + ```ruby # /etc/gitlab/gitlab.rb @@ -141,6 +167,7 @@ gitlab_rails['auto_migrate'] = false # Configure the gitlab-shell API callback URL. Without this, `git push` will # fail. This can be your 'front door' GitLab URL or an internal load # balancer. +# Don't forget to copy `/etc/gitlab/gitlab-secrets.json` from web server to Gitaly server. gitlab_rails['internal_api_url'] = 'https://gitlab.example.com' # Make Gitaly accept connections on all network interfaces. You must use @@ -193,18 +220,20 @@ network, firewall, or name resolution problem preventing your GitLab server from reaching the Gitaly server then all Gitaly requests will fail. +Additionally, you need to +[disable Rugged if previously manually enabled](../high_availability/nfs.md#improving-nfs-performance-with-gitlab). + We assume that your Gitaly server can be reached at -`gitaly.internal:8075` from your GitLab server, and that your GitLab -NFS shares are mounted at `/mnt/gitlab/default` and -`/mnt/gitlab/storage1` respectively. +`gitaly.internal:8075` from your GitLab server, and that Gitaly can read and +write to `/mnt/gitlab/default` and `/mnt/gitlab/storage1` respectively. Omnibus installations: ```ruby # /etc/gitlab/gitlab.rb git_data_dirs({ - 'default' => { 'path' => '/mnt/gitlab/default', 'gitaly_address' => 'tcp://gitaly.internal:8075' }, - 'storage1' => { 'path' => '/mnt/gitlab/storage1', 'gitaly_address' => 'tcp://gitaly.internal:8075' }, + 'default' => { 'gitaly_address' => 'tcp://gitaly.internal:8075' }, + 'storage1' => { 'gitaly_address' => 'tcp://gitaly.internal:8075' }, }) gitlab_rails['gitaly_token'] = 'abc123secret' @@ -218,10 +247,8 @@ gitlab: repositories: storages: default: - path: /mnt/gitlab/default/repositories gitaly_address: tcp://gitaly.internal:8075 storage1: - path: /mnt/gitlab/storage1/repositories gitaly_address: tcp://gitaly.internal:8075 gitaly: @@ -236,14 +263,26 @@ repository from your GitLab server over HTTP. ## TLS support -> [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/22602) in GitLab 11.7. +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/22602) in GitLab 11.8. -Gitaly supports TLS credentials for GRPC authentication. To be able to communicate +Gitaly supports TLS encryption. To be able to communicate with a Gitaly instance that listens for secure connections you will need to use `tls://` url scheme in the `gitaly_address` of the corresponding storage entry in the gitlab configuration. The admin needs to bring their own certificate as we do not provide that automatically. -The certificate to be used needs to be installed on all Gitaly nodes and on all client nodes that communicate with it following procedures described in [GitLab custom certificate configuration](https://docs.gitlab.com/omnibus/settings/ssl.html#install-custom-public-certificates) +The certificate to be used needs to be installed on all Gitaly nodes and on all client nodes that communicate with it following procedures described in [GitLab custom certificate configuration](https://docs.gitlab.com/omnibus/settings/ssl.html#install-custom-public-certificates). + +Note that it is possible to configure Gitaly servers with both an +unencrypted listening address `listen_addr` and an encrypted listening +address `tls_listen_addr` at the same time. This allows you to do a +gradual transition from unencrypted to encrypted traffic, if necessary. + +To observe what type of connections are actually being used in a +production environment you can use the following Prometheus query: + +``` +sum(rate(gitaly_connections_total[5m])) by (type) +``` ### Example TLS configuration @@ -300,6 +339,66 @@ certificate_path = '/path/to/cert.pem' key_path = '/path/to/key.pem' ``` +## Gitaly-ruby + +Gitaly was developed to replace Ruby application code in gitlab-ce/ee. +In order to save time and/or avoid the risk of rewriting existing +application logic, in some cases we chose to copy some application code +from gitlab-ce into Gitaly almost as-is. To be able to run that code, we +made gitaly-ruby, which is a sidecar process for the main Gitaly Go +process. Some examples of things that are implemented in gitaly-ruby are +RPC's that deal with wiki's, and RPC's that create commits on behalf of +a user, such as merge commits. + +### Number of gitaly-ruby workers + +Gitaly-ruby has much less capacity than Gitaly itself. If your Gitaly +server has to handle a lot of request, the default setting of having +just 1 active gitaly-ruby sidecar might not be enough. If you see +ResourceExhausted errors from Gitaly it's very likely that you have not +enough gitaly-ruby capacity. + +You can increase the number of gitaly-ruby processes on your Gitaly +server with the following settings. + +Omnibus: + +```ruby +# /etc/gitlab/gitlab.rb +# Default is 2 workers. The minimum is 2; 1 worker is always reserved as +# a passive stand-by. +gitaly['ruby_num_workers'] = 4 +``` + +Source: + +```toml +# /home/git/gitaly/config.toml +[gitaly-ruby] +num_workers = 4 +``` + +### Observing gitaly-ruby traffic + +Gitaly-ruby is a somewhat hidden, internal implementation detail of +Gitaly. There is not that much visibility into what goes on inside +gitaly-ruby processes. + +If you have Prometheus set up to scrape your Gitaly process, you can see +request rates and error codes for individual RPC's in gitaly-ruby by +querying `grpc_client_handled_total`. Strictly speaking this metric does +not differentiate between gitaly-ruby and other RPC's, but in practice +(as of GitLab 11.9), all gRPC calls made by Gitaly itself are internal +calls from the main Gitaly process to one of its gitaly-ruby sidecars. + +Assuming your `grpc_client_handled_total` counter only observes Gitaly, +the following query shows you RPC's are (most likely) internally +implemented as calls to gitaly-ruby. + +``` +sum(rate(grpc_client_handled_total[5m])) by (grpc_method) > 0 +``` + ## Disabling or enabling the Gitaly service in a cluster environment If you are running Gitaly [as a remote diff --git a/doc/administration/high_availability/README.md b/doc/administration/high_availability/README.md index 49fe80fb2a6..002deeaf945 100644 --- a/doc/administration/high_availability/README.md +++ b/doc/administration/high_availability/README.md @@ -1,4 +1,4 @@ -# High Availability +# Scaling and High Availability GitLab supports several different types of clustering and high-availability. The solution you choose will be based on the level of scalability and @@ -13,51 +13,197 @@ of Git, developers can still commit code locally even when GitLab is not available. However, some GitLab features such as the issue tracker and Continuous Integration are not available when GitLab is down. -**Keep in mind that all Highly Available solutions come with a trade-off between +**Keep in mind that all highly-available solutions come with a trade-off between cost/complexity and uptime**. The more uptime you want, the more complex the solution. And the more complex the solution, the more work is involved in setting up and maintaining it. High availability is not free and every HA solution should balance the costs against the benefits. -## Architecture - -There are two kinds of setups: - -- active/active -- active/passive - -### Active/Active - -This architecture scales easily because all application servers handle -user requests simultaneously. The database, Redis, and GitLab application are -all deployed on separate servers. The configuration is **only** highly-available -if the database, Redis and storage are also configured as such. - -Follow the steps below to configure an active/active setup: +There are many options when choosing a highly-available GitLab architecture. We +recommend engaging with GitLab Support to choose the best architecture for your +use-case. This page contains some various options and guidelines based on +experience with GitLab.com and Enterprise Edition on-premises customers. + +For a detailed insight into how GitLab scales and configures GitLab.com, you can +watch [this 1 hour Q&A](https://www.youtube.com/watch?v=uCU8jdYzpac) +with [John Northrup](https://gitlab.com/northrup), one of our infrastructure +engineers, and live questions coming in from some of our customers. + +## GitLab Components + +The following components need to be considered for a scaled or highly-available +environment. In many cases components can be combined on the same nodes to reduce +complexity. + +- Unicorn/Workhorse - Web-requests (UI, API, Git over HTTP) +- Sidekiq - Asynchronous/Background jobs +- PostgreSQL - Database + - Consul - Database service discovery and health checks/failover + - PGBouncer - Database pool manager +- Redis - Key/Value store (User sessions, cache, queue for Sidekiq) + - Sentinel - Redis health check/failover manager +- Gitaly - Provides high-level RPC access to Git repositories + +## Scalable Architecture Examples + +When an organization reaches a certain threshold it will be necessary to scale +the GitLab instance. Still, true high availability may not be necessary. There +are options for scaling GitLab instances relatively easily without incurring the +infrastructure and maintenance costs of full high availability. + +### Basic Scaling + +This is the simplest form of scaling and will work for the majority of +cases. Backend components such as PostgreSQL, Redis and storage are offloaded +to their own nodes while the remaining GitLab components all run on 2 or more +application nodes. + +This form of scaling also works well in a cloud environment when it is more +cost-effective to deploy several small nodes rather than a single +larger one. + +- 1 PostgreSQL node +- 1 Redis node +- 1 NFS/Gitaly storage server +- 2 or more GitLab application nodes (Unicorn, Workhorse, Sidekiq) + +#### Installation Instructions + +Complete the following installation steps in order. A link at the end of each +section will bring you back to the Scalable Architecture Examples section so +you can continue with the next step. + +1. [PostgreSQL](database.md#postgresql-in-a-scaled-environment) +1. [Redis](redis.md#redis-in-a-scaled-environment) +1. [Gitaly](gitaly.md) (recommended) or [NFS](nfs.md) +1. [GitLab application nodes](gitlab.md) + +### Full Scaling + +For very large installations it may be necessary to further split components +for maximum scalability. In a fully-scaled architecture the application node +is split into separate Sidekiq and Unicorn/Workhorse nodes. One indication that +this architecture is required is if Sidekiq queues begin to periodically increase +in size, indicating that there is contention or not enough resources. + +- 1 PostgreSQL node +- 1 Redis node +- 2 or more NFS/Gitaly storage servers +- 2 or more Sidekiq nodes +- 2 or more GitLab application nodes (Unicorn, Workhorse) + +## High Availability Architecture Examples + +When organizations require scaling *and* high availability the following +architectures can be utilized. As the introduction section at the top of this +page mentions, there is a tradeoff between cost/complexity and uptime. Be sure +this complexity is absolutely required before taking the step into full +high availability. + +For all examples below, we recommend running Consul and Redis Sentinel on +dedicated nodes. If Consul is running on PostgreSQL nodes or Sentinel on +Redis nodes there is a potential that high resource usage by PostgreSQL or +Redis could prevent communication between the other Consul and Sentinel nodes. +This may lead to the other nodes believing a failure has occurred and automated +failover is necessary. Isolating them from the services they monitor reduces +the chances of split-brain. + +The examples below do not really address high availability of NFS. Some enterprises +have access to NFS appliances that manage availability. This is the best case +scenario. In the future, GitLab may offer a more user-friendly solution to +[GitLab HA Storage](https://gitlab.com/gitlab-org/omnibus-gitlab/issues/2472). + +There are many options in between each of these examples. Work with GitLab Support +to understand the best starting point for your workload and adapt from there. + +### Horizontal + +This is the simplest form of high availability and scaling. It requires the +fewest number of individual servers (virtual or physical) but does have some +trade-offs and limits. + +This architecture will work well for many GitLab customers. Larger customers +may begin to notice certain events cause contention/high load - for example, +cloning many large repositories with binary files, high API usage, a large +number of enqueued Sidekiq jobs, etc. If this happens you should consider +moving to a hybrid or fully distributed architecture depending on what is causing +the contention. + +- 3 PostgreSQL nodes +- 2 Redis nodes +- 3 Consul/Sentinel nodes +- 2 or more GitLab application nodes (Unicorn, Workhorse, Sidekiq, PGBouncer) +- 1 NFS/Gitaly server + + + +### Hybrid + +In this architecture, certain components are split on dedicated nodes so high +resource usage of one component does not interfere with others. In larger +environments this is a good architecture to consider if you foresee or do have +contention due to certain workloads. + +- 3 PostgreSQL nodes +- 1 PgBouncer node +- 2 Redis nodes +- 3 Consul/Sentinel nodes +- 2 or more Sidekiq nodes +- 2 or more GitLab application nodes (Unicorn, Workhorse) +- 1 or more NFS/Gitaly servers +- 1 Monitoring node (Prometheus, Grafana) + + + +#### Reference Architecture + +- **Status:** Work-in-progress +- **Supported Users (approximate):** 10,000 +- **Related Issues:** [gitlab-com/support/support-team-meta#1513](https://gitlab.com/gitlab-com/support/support-team-meta/issues/1513), + [gitlab-org/quality/team-tasks#110](https://gitlab.com/gitlab-org/quality/team-tasks/issues/110) + +The Support and Quality teams are in the process of building and performance testing +an environment that will support about 10,000 users. The specifications below +are a work-in-progress representation of the work so far. Quality will be +certifying this environment in FY20-Q2. The specifications may be adjusted +prior to certification based on performance testing. + +- 3 PostgreSQL - 4 CPU, 8GB RAM +- 1 PgBouncer - 2 CPU, 4GB RAM +- 2 Redis - 2 CPU, 8GB RAM +- 3 Consul/Sentinel - 2 CPU, 2GB RAM +- 4 Sidekiq - 4 CPU, 8GB RAM +- 5 GitLab application nodes - 20 CPU, 64GB RAM +- 1 Gitaly - 20 CPU, 64GB RAM +- 1 Monitoring node - 4 CPU, 8GB RAM + +### Fully Distributed + +This architecture scales to hundreds of thousands of users and projects and is +the basis of the GitLab.com architecture. While this scales well it also comes +with the added complexity of many more nodes to configure, manage and monitor. + +- 3 PostgreSQL nodes +- 4 or more Redis nodes (2 separate clusters for persistent and cache data) +- 3 Consul nodes +- 3 Sentinel nodes +- Multiple dedicated Sidekiq nodes (Split into real-time, best effort, ASAP, + CI Pipeline and Pull Mirror sets) +- 2 or more Git nodes (Git over SSH/Git over HTTP) +- 2 or more API nodes (All requests to `/api`) +- 2 or more Web nodes (All other web requests) +- 2 or more NFS/Gitaly servers + + + +The following pages outline the steps necessary to configure each component +separately: 1. [Configure the database](database.md) 1. [Configure Redis](redis.md) 1. [Configure Redis for GitLab source installations](redis_source.md) 1. [Configure NFS](nfs.md) + 1. [NFS Client and Host setup](nfs_host_client_setup.md) 1. [Configure the GitLab application servers](gitlab.md) 1. [Configure the load balancers](load_balancer.md) - - -### Active/Passive - -For pure high-availability/failover with no scaling you can use an -active/passive configuration. This utilizes DRBD (Distributed Replicated -Block Device) to keep all data in sync. DRBD requires a low latency link to -remain in sync. It is not advisable to attempt to run DRBD between data centers -or in different cloud availability zones. - -> **Note:** GitLab recommends against choosing this HA method because of the - complexity of managing DRBD and crafting automatic failover. This is - *compatible* with GitLab, but not officially *supported*. If you are - an EE customer, support will help you with GitLab related problems, but if the - root cause is identified as DRBD, we will not troubleshoot further. - -Components/Servers Required: 2 servers/virtual machines (one active/one passive) - - diff --git a/doc/administration/high_availability/alpha_database.md b/doc/administration/high_availability/alpha_database.md new file mode 100644 index 00000000000..7bf20be60e6 --- /dev/null +++ b/doc/administration/high_availability/alpha_database.md @@ -0,0 +1,6 @@ +--- +redirect_to: 'database.md' +--- + +This documentation has been moved to the main +[database documentation](database.md#configure_using_omnibus_for_high_availability). diff --git a/doc/administration/high_availability/consul.md b/doc/administration/high_availability/consul.md new file mode 100644 index 00000000000..056b7fc15d9 --- /dev/null +++ b/doc/administration/high_availability/consul.md @@ -0,0 +1,105 @@ +# Working with the bundled Consul service **[PREMIUM ONLY]** + +## Overview + +As part of its High Availability stack, GitLab Premium includes a bundled version of [Consul](http://consul.io) that can be managed through `/etc/gitlab/gitlab.rb`. + +A Consul cluster consists of multiple server agents, as well as client agents that run on other nodes which need to talk to the consul cluster. + +## Operations + +### Checking cluster membership + +To see which nodes are part of the cluster, run the following on any member in the cluster +``` +# /opt/gitlab/embedded/bin/consul members +Node Address Status Type Build Protocol DC +consul-b XX.XX.X.Y:8301 alive server 0.9.0 2 gitlab_consul +consul-c XX.XX.X.Y:8301 alive server 0.9.0 2 gitlab_consul +consul-c XX.XX.X.Y:8301 alive server 0.9.0 2 gitlab_consul +db-a XX.XX.X.Y:8301 alive client 0.9.0 2 gitlab_consul +db-b XX.XX.X.Y:8301 alive client 0.9.0 2 gitlab_consul +``` + +Ideally all nodes will have a `Status` of `alive`. + +### Restarting the server cluster + +**Note**: This section only applies to server agents. It is safe to restart client agents whenever needed. + +If it is necessary to restart the server cluster, it is important to do this in a controlled fashion in order to maintain quorum. If quorum is lost, you will need to follow the consul [outage recovery](#outage-recovery) process to recover the cluster. + +To be safe, we recommend you only restart one server agent at a time to ensure the cluster remains intact. + +For larger clusters, it is possible to restart multiple agents at a time. See the [Consul consensus document](https://www.consul.io/docs/internals/consensus.html#deployment-table) for how many failures it can tolerate. This will be the number of simulateneous restarts it can sustain. + +## Troubleshooting + +### Consul server agents unable to communicate + +By default, the server agents will attempt to [bind](https://www.consul.io/docs/agent/options.html#_bind) to '0.0.0.0', but they will advertise the first private IP address on the node for other agents to communicate with them. If the other nodes cannot communicate with a node on this address, then the cluster will have a failed status. + +You will see messages like the following in `gitlab-ctl tail consul` output if you are running into this issue: + +``` +2017-09-25_19:53:39.90821 2017/09/25 19:53:39 [WARN] raft: no known peers, aborting election +2017-09-25_19:53:41.74356 2017/09/25 19:53:41 [ERR] agent: failed to sync remote state: No cluster leader +``` + + +To fix this: + +1. Pick an address on each node that all of the other nodes can reach this node through. +1. Update your `/etc/gitlab/gitlab.rb` + + ```ruby + consul['configuration'] = { + ... + bind_addr: 'IP ADDRESS' + } + ``` +1. Run `gitlab-ctl reconfigure` + +If you still see the errors, you may have to [erase the consul database and reinitialize](#recreate-from-scratch) on the affected node. + +### Consul agents do not start - Multiple private IPs + +In the case that a node has multiple private IPs the agent be confused as to which of the private addresses to advertise, and then immediately exit on start. + +You will see messages like the following in `gitlab-ctl tail consul` output if you are running into this issue: + +``` +2017-11-09_17:41:45.52876 ==> Starting Consul agent... +2017-11-09_17:41:45.53057 ==> Error creating agent: Failed to get advertise address: Multiple private IPs found. Please configure one. +``` + +To fix this: + +1. Pick an address on the node that all of the other nodes can reach this node through. +1. Update your `/etc/gitlab/gitlab.rb` + + ```ruby + consul['configuration'] = { + ... + bind_addr: 'IP ADDRESS' + } + ``` +1. Run `gitlab-ctl reconfigure` + +### Outage recovery + +If you lost enough server agents in the cluster to break quorum, then the cluster is considered failed, and it will not function without manual intervenetion. + +#### Recreate from scratch +By default, GitLab does not store anything in the consul cluster that cannot be recreated. To erase the consul database and reinitialize + +``` +# gitlab-ctl stop consul +# rm -rf /var/opt/gitlab/consul/data +# gitlab-ctl start consul +``` + +After this, the cluster should start back up, and the server agents rejoin. Shortly after that, the client agents should rejoin as well. + +#### Recover a failed cluster +If you have taken advantage of consul to store other data, and want to restore the failed cluster, please follow the [Consul guide](https://www.consul.io/docs/guides/outage.html) to recover a failed cluster. diff --git a/doc/administration/high_availability/database.md b/doc/administration/high_availability/database.md index c1eeb40b98f..3b874e5d312 100644 --- a/doc/administration/high_availability/database.md +++ b/doc/administration/high_availability/database.md @@ -1,11 +1,6 @@ -# Configuring a Database for GitLab HA +# Configuring PostgreSQL for Scaling and High Availability -You can choose to install and manage a database server (PostgreSQL/MySQL) -yourself, or you can use GitLab Omnibus packages to help. GitLab recommends -PostgreSQL. This is the database that will be installed if you use the -Omnibus package to manage your database. - -## Configure your own database server +## Provide your own PostgreSQL instance **[CORE ONLY]** If you're hosting GitLab on a cloud provider, you can optionally use a managed service for PostgreSQL. For example, AWS offers a managed Relational @@ -20,93 +15,1144 @@ If you use a cloud-managed service, or provide your own PostgreSQL: 1. Configure the GitLab application servers with the appropriate details. This step is covered in [Configuring GitLab for HA](gitlab.md). -## Configure using Omnibus +## PostgreSQL in a Scaled Environment -1. Download/install GitLab Omnibus using **steps 1 and 2** from - [GitLab downloads](https://about.gitlab.com/downloads). Do not complete other - steps on the download page. -1. Create/edit `/etc/gitlab/gitlab.rb` and use the following configuration. - Be sure to change the `external_url` to match your eventual GitLab front-end - URL. If there is a directive listed below that you do not see in the configuration, be sure to add it. +This section is relevant for [Scaled Architecture](README.md#scalable-architecture-examples) +environments including [Basic Scaling](README.md#basic-scaling) and +[Full Scaling](README.md#full-scaling). - ```ruby - external_url 'https://gitlab.example.com' +### Provide your own PostgreSQL instance **[CORE ONLY]** + +If you want to use your own deployed PostgreSQL instance(s), +see [Provide your own PostgreSQL instance](#provide-your-own-postgresql-instance-core-only) +for more details. However, you can use the GitLab Omnibus package to easily +deploy the bundled PostgreSQL. + +### Standalone PostgreSQL using GitLab Omnibus **[CORE ONLY]** + +1. SSH into the PostgreSQL server. +1. [Download/install](https://about.gitlab.com/installation) the Omnibus GitLab + package you want using **steps 1 and 2** from the GitLab downloads page. + - Do not complete any other steps on the download page. +1. Generate a password hash for PostgreSQL. This assumes you will use the default + username of `gitlab` (recommended). The command will request a password + and confirmation. Use the value that is output by this command in the next + step as the value of `POSTGRESQL_PASSWORD_HASH`. + ```sh + sudo gitlab-ctl pg-password-md5 gitlab + ``` + +1. Edit `/etc/gitlab/gitlab.rb` and add the contents below, updating placeholder + values appropriately. + + - `POSTGRESQL_PASSWORD_HASH` - The value output from the previous step + - `APPLICATION_SERVER_IP_BLOCKS` - A space delimited list of IP subnets or IP + addresses of the GitLab application servers that will connect to the + database. Example: `%w(123.123.123.123/32 123.123.123.234/32)` + + ```ruby # Disable all components except PostgreSQL roles ['postgres_role'] + repmgr['enable'] = false + consul['enable'] = false + prometheus['enable'] = false + alertmanager['enable'] = false + pgbouncer_exporter['enable'] = false + redis_exporter['enable'] = false + gitlab_monitor['enable'] = false + + postgresql['listen_address'] = '0.0.0.0' + postgresql['port'] = 5432 + + # Replace POSTGRESQL_PASSWORD_HASH with a generated md5 value + postgresql['sql_user_password'] = 'POSTGRESQL_PASSWORD_HASH' + + # Replace XXX.XXX.XXX.XXX/YY with Network Address + # ???? + postgresql['trust_auth_cidr_addresses'] = %w(APPLICATION_SERVER_IP_BLOCKS) + + # Disable automatic database migrations + gitlab_rails['auto_migrate'] = false + ``` + + NOTE: **Note:** The role `postgres_role` was introduced with GitLab 10.3 + +1. [Reconfigure GitLab] for the changes to take effect. +1. Note the PostgreSQL node's IP address or hostname, port, and + plain text password. These will be necessary when configuring the GitLab + application servers later. + +Advanced configuration options are supported and can be added if +needed. + +Continue configuration of other components by going +[back to Scaled Architectures](README.md#scalable-architecture-examples) + +## PostgreSQL with High Availability + +This section is relevant for [High Availability Architecture](README.md#high-availability-architecture-examples) +environments including [Horizontal](README.md#horizontal), +[Hybrid](README.md#hybrid), and +[Fully Distributed](README.md#fully-distributed). + +### Provide your own PostgreSQL instance **[CORE ONLY]** + +If you want to use your own deployed PostgreSQL instance(s), +see [Provide your own PostgreSQL instance](#provide-your-own-postgresql-instance-core-only) +for more details. However, you can use the GitLab Omnibus package to easily +deploy the bundled PostgreSQL. + +### High Availability with GitLab Omnibus **[PREMIUM ONLY]** + +> Important notes: +> +> - This document will focus only on configuration supported with [GitLab Premium](https://about.gitlab.com/pricing/), using the Omnibus GitLab package. +> - If you are a Community Edition or Starter user, consider using a cloud hosted solution. +> - This document will not cover installations from source. +> +> - If HA setup is not what you were looking for, see the [database configuration document](http://docs.gitlab.com/omnibus/settings/database.html) +> for the Omnibus GitLab packages. +> +> Please read this document fully before attempting to configure PostgreSQL HA +> for GitLab. +> +> This configuration is GA in EE 10.2. + +The recommended configuration for a PostgreSQL HA requires: + +- A minimum of three database nodes + - Each node will run the following services: + - `PostgreSQL` - The database itself + - `repmgrd` - A service to monitor, and handle failover in case of a failure + - `Consul` agent - Used for service discovery, to alert other nodes when failover occurs +- A minimum of three `Consul` server nodes +- A minimum of one `pgbouncer` service node + +You also need to take into consideration the underlying network topology, +making sure you have redundant connectivity between all Database and GitLab instances, +otherwise the networks will become a single point of failure. + +#### Architecture + + + +Database nodes run two services with PostgreSQL: + +- Repmgrd. Monitors the cluster and handles failover when issues with the master occur. The failover consists of: + - Selecting a new master for the cluster. + - Promoting the new node to master. + - Instructing remaining servers to follow the new master node. + + On failure, the old master node is automatically evicted from the cluster, and should be rejoined manually once recovered. +- Consul. Monitors the status of each node in the database cluster and tracks its health in a service definition on the consul cluster. + +Alongside pgbouncer, there is a consul agent that watches the status of the PostgreSQL service. If that status changes, consul runs a script which updates the configuration and reloads pgbouncer + +##### Connection flow + +Each service in the package comes with a set of [default ports](https://docs.gitlab.com/omnibus/package-information/defaults.html#ports). You may need to make specific firewall rules for the connections listed below: + +- Application servers connect to [PgBouncer default port](https://docs.gitlab.com/omnibus/package-information/defaults.html#pgbouncer) +- PgBouncer connects to the primary database servers [PostgreSQL default port](https://docs.gitlab.com/omnibus/package-information/defaults.html#postgresql) +- Repmgr connects to the database servers [PostgreSQL default port](https://docs.gitlab.com/omnibus/package-information/defaults.html#postgresql) +- Postgres secondaries connect to the primary database servers [PostgreSQL default port](https://docs.gitlab.com/omnibus/package-information/defaults.html#postgresql) +- Consul servers and agents connect to each others [Consul default ports](https://docs.gitlab.com/omnibus/package-information/defaults.html#consul) + +#### Required information + +Before proceeding with configuration, you will need to collect all the necessary +information. + +##### Network information + +PostgreSQL does not listen on any network interface by default. It needs to know +which IP address to listen on in order to be accessible to other services. +Similarly, PostgreSQL access is controlled based on the network source. + +This is why you will need: + +- IP address of each nodes network interface. This can be set to `0.0.0.0` to + listen on all interfaces. It cannot be set to the loopack address `127.0.0.1`. +- Network Address. This can be in subnet (i.e. `192.168.0.0/255.255.255.0`) + or CIDR (i.e. `192.168.0.0/24`) form. + +##### User information + +Various services require different configuration to secure +the communication as well as information required for running the service. +Bellow you will find details on each service and the minimum required +information you need to provide. + +##### Consul information + +When using default setup, minimum configuration requires: + +- `CONSUL_USERNAME`. Defaults to `gitlab-consul` +- `CONSUL_DATABASE_PASSWORD`. Password for the database user. +- `CONSUL_PASSWORD_HASH`. This is a hash generated out of consul username/password pair. + Can be generated with: + + ```sh + sudo gitlab-ctl pg-password-md5 CONSUL_USERNAME + ``` + +- `CONSUL_SERVER_NODES`. The IP addresses or DNS records of the Consul server nodes. + +Few notes on the service itself: + +- The service runs under a system account, by default `gitlab-consul`. + - If you are using a different username, you will have to specify it. We +will refer to it with `CONSUL_USERNAME`, +- There will be a database user created with read only access to the repmgr +database +- Passwords will be stored in the following locations: + - `/etc/gitlab/gitlab.rb`: hashed + - `/var/opt/gitlab/pgbouncer/pg_auth`: hashed + - `/var/opt/gitlab/gitlab-consul/.pgpass`: plaintext + +##### PostgreSQL information + +When configuring PostgreSQL, we will set `max_wal_senders` to one more than +the number of database nodes in the cluster. +This is used to prevent replication from using up all of the +available database connections. + +In this document we are assuming 3 database nodes, which makes this configuration: + +``` +postgresql['max_wal_senders'] = 4 +``` + +As previously mentioned, you'll have to prepare the network subnets that will +be allowed to authenticate with the database. +You'll also need to supply the IP addresses or DNS records of Consul +server nodes. + +We will need the following password information for the application's database user: + +- `POSTGRESQL_USERNAME`. Defaults to `gitlab` +- `POSTGRESQL_USER_PASSWORD`. The password for the database user +- `POSTGRESQL_PASSWORD_HASH`. This is a hash generated out of the username/password pair. + Can be generated with: + + ```sh + sudo gitlab-ctl pg-password-md5 POSTGRESQL_USERNAME + ``` + +##### Pgbouncer information + +When using default setup, minimum configuration requires: + +- `PGBOUNCER_USERNAME`. Defaults to `pgbouncer` +- `PGBOUNCER_PASSWORD`. This is a password for pgbouncer service. +- `PGBOUNCER_PASSWORD_HASH`. This is a hash generated out of pgbouncer username/password pair. + Can be generated with: + + ```sh + sudo gitlab-ctl pg-password-md5 PGBOUNCER_USERNAME + ``` + +- `PGBOUNCER_NODE`, is the IP address or a FQDN of the node running Pgbouncer. + +Few notes on the service itself: + +- The service runs as the same system account as the database + - In the package, this is by default `gitlab-psql` +- If you use a non-default user account for Pgbouncer service (by default `pgbouncer`), you will have to specify this username. We will refer to this requirement with `PGBOUNCER_USERNAME`. +- The service will have a regular database user account generated for it + - This defaults to `repmgr` +- Passwords will be stored in the following locations: + - `/etc/gitlab/gitlab.rb`: hashed, and in plain text + - `/var/opt/gitlab/pgbouncer/pg_auth`: hashed + +##### Repmgr information + +When using default setup, you will only have to prepare the network subnets that will +be allowed to authenticate with the service. + +Few notes on the service itself: + +- The service runs under the same system account as the database + - In the package, this is by default `gitlab-psql` +- The service will have a superuser database user account generated for it + - This defaults to `gitlab_repmgr` + +#### Installing Omnibus GitLab + +First, make sure to [download/install](https://about.gitlab.com/installation) +GitLab Omnibus **on each node**. + +Make sure you install the necessary dependencies from step 1, +add GitLab package repository from step 2. +When installing the GitLab package, do not supply `EXTERNAL_URL` value. + +#### Configuring the Consul nodes + +On each Consul node perform the following: + +1. Make sure you collect [`CONSUL_SERVER_NODES`](#consul-information) before executing the next step. + +1. Edit `/etc/gitlab/gitlab.rb` replacing values noted in the `# START user configuration` section: + + ```ruby + # Disable all components except Consul + roles ['consul_role'] + + # START user configuration + # Replace placeholders: + # + # Y.Y.Y.Y consul1.gitlab.example.com Z.Z.Z.Z + # with the addresses gathered for CONSUL_SERVER_NODES + consul['configuration'] = { + server: true, + retry_join: %w(Y.Y.Y.Y consul1.gitlab.example.com Z.Z.Z.Z) + } + + # Disable auto migrations + gitlab_rails['auto_migrate'] = false + # + # END user configuration + ``` + + > `consul_role` was introduced with GitLab 10.3 + +1. [Reconfigure GitLab] for the changes to take effect. + +##### Consul Checkpoint + +Before moving on, make sure Consul is configured correctly. Run the following +command to verify all server nodes are communicating: + +```sh +/opt/gitlab/embedded/bin/consul members +``` + +The output should be similar to: + +``` +Node Address Status Type Build Protocol DC +CONSUL_NODE_ONE XXX.XXX.XXX.YYY:8301 alive server 0.9.2 2 gitlab_consul +CONSUL_NODE_TWO XXX.XXX.XXX.YYY:8301 alive server 0.9.2 2 gitlab_consul +CONSUL_NODE_THREE XXX.XXX.XXX.YYY:8301 alive server 0.9.2 2 gitlab_consul +``` + +If any of the nodes isn't `alive` or if any of the three nodes are missing, +check the [Troubleshooting section](#troubleshooting) before proceeding. + +#### Configuring the Database nodes + +1. Make sure you collect [`CONSUL_SERVER_NODES`](#consul-information), [`PGBOUNCER_PASSWORD_HASH`](#pgbouncer-information), [`POSTGRESQL_PASSWORD_HASH`](#postgresql-information), the [number of db nodes](#postgresql-information), and the [network address](#network-information) before executing the next step. + +1. On the master database node, edit `/etc/gitlab/gitlab.rb` replacing values noted in the `# START user configuration` section: + + ```ruby + # Disable all components except PostgreSQL and Repmgr and Consul + roles ['postgres_role'] # PostgreSQL configuration - gitlab_rails['db_password'] = 'DB password' - postgresql['md5_auth_cidr_addresses'] = ['0.0.0.0/0'] postgresql['listen_address'] = '0.0.0.0' + postgresql['hot_standby'] = 'on' + postgresql['wal_level'] = 'replica' + postgresql['shared_preload_libraries'] = 'repmgr_funcs' # Disable automatic database migrations gitlab_rails['auto_migrate'] = false + + # Configure the consul agent + consul['services'] = %w(postgresql) + + # START user configuration + # Please set the real values as explained in Required Information section + # + # Replace PGBOUNCER_PASSWORD_HASH with a generated md5 value + postgresql['pgbouncer_user_password'] = 'PGBOUNCER_PASSWORD_HASH' + # Replace POSTGRESQL_PASSWORD_HASH with a generated md5 value + postgresql['sql_user_password'] = 'POSTGRESQL_PASSWORD_HASH' + # Replace X with value of number of db nodes + 1 + postgresql['max_wal_senders'] = X + + # Replace XXX.XXX.XXX.XXX/YY with Network Address + postgresql['trust_auth_cidr_addresses'] = %w(XXX.XXX.XXX.XXX/YY) + repmgr['trust_auth_cidr_addresses'] = %w(127.0.0.1/32 XXX.XXX.XXX.XXX/YY) + + # Replace placeholders: + # + # Y.Y.Y.Y consul1.gitlab.example.com Z.Z.Z.Z + # with the addresses gathered for CONSUL_SERVER_NODES + consul['configuration'] = { + retry_join: %w(Y.Y.Y.Y consul1.gitlab.example.com Z.Z.Z.Z) + } + # + # END user configuration ``` -1. Run `sudo gitlab-ctl reconfigure` to install and configure PostgreSQL. + > `postgres_role` was introduced with GitLab 10.3 + +1. On secondary nodes, add all the configuration specified above for primary node + to `/etc/gitlab/gitlab.rb`. In addition, append the following configuration + to inform gitlab-ctl that they are standby nodes initially and it need not + attempt to register them as primary node + ``` + # HA setting to specify if a node should attempt to be master on initialization + repmgr['master_on_initialization'] = false + ``` + +1. [Reconfigure GitLab] for the changes to take effect. + +> Please note: +> +> - If you want your database to listen on a specific interface, change the config: +> `postgresql['listen_address'] = '0.0.0.0'`. +> - If your Pgbouncer service runs under a different user account, +> you also need to specify: `postgresql['pgbouncer_user'] = PGBOUNCER_USERNAME` in +> your configuration. + +##### Database nodes post-configuration - > **Note**: This `reconfigure` step will result in some errors. - That's OK - don't be alarmed. +###### Primary node + +Select one node as a primary node. 1. Open a database prompt: + ```sh + gitlab-psql -d gitlabhq_production ``` - su - gitlab-psql - /bin/bash - psql -h /var/opt/gitlab/postgresql -d template1 - # Output: +1. Enable the `pg_trgm` extension: + + ```sh + CREATE EXTENSION pg_trgm; + ``` - psql (9.2.15) - Type "help" for help. +1. Exit the database prompt by typing `\q` and Enter. - template1=# +1. Verify the cluster is initialized with one node: + + ```sh + gitlab-ctl repmgr cluster show + ``` + + The output should be similar to the following: + + ``` + Role | Name | Upstream | Connection String + ----------+----------|----------|---------------------------------------- + * master | HOSTNAME | | host=HOSTNAME user=gitlab_repmgr dbname=gitlab_repmgr ``` -1. Run the following command at the database prompt and you will be asked to - enter the new password for the PostgreSQL superuser. +1. Note down the hostname/ip in the connection string: `host=HOSTNAME`. We will + refer to the hostname in the next section as `MASTER_NODE_NAME`. If the value + is not an IP address, it will need to be a resolvable name (via DNS or + `/etc/hosts`) +###### Secondary nodes + +1. Set up the repmgr standby: + + ```sh + gitlab-ctl repmgr standby setup MASTER_NODE_NAME ``` - \password - # Output: + Do note that this will remove the existing data on the node. The command + has a wait time. + + The output should be similar to the following: - Enter new password: - Enter it again: + ```console + # gitlab-ctl repmgr standby setup MASTER_NODE_NAME + Doing this will delete the entire contents of /var/opt/gitlab/postgresql/data + If this is not what you want, hit Ctrl-C now to exit + To skip waiting, rerun with the -w option + Sleeping for 30 seconds + Stopping the database + Removing the data + Cloning the data + Starting the database + Registering the node with the cluster + ok: run: repmgrd: (pid 19068) 0s ``` -1. Similarly, set the password for the `gitlab` database user. Use the same - password that you specified in the `/etc/gitlab/gitlab.rb` file for - `gitlab_rails['db_password']`. +1. Verify the node now appears in the cluster: + ```sh + gitlab-ctl repmgr cluster show ``` - \password gitlab - # Output: + The output should be similar to the following: - Enter new password: - Enter it again: ``` -1. Exit from editing `template1` prompt by typing `\q` and Enter. -1. Enable the `pg_trgm` extension within the `gitlabhq_production` database: - + Role | Name | Upstream | Connection String + ----------+---------|-----------|------------------------------------------------ + * master | MASTER | | host=MASTER_NODE_NAME user=gitlab_repmgr dbname=gitlab_repmgr + standby | STANDBY | MASTER | host=STANDBY_HOSTNAME user=gitlab_repmgr dbname=gitlab_repmgr ``` + +Repeat the above steps on all secondary nodes. + +##### Database checkpoint + +Before moving on, make sure the databases are configured correctly. Run the +following command on the **primary** node to verify that replication is working +properly: + +```sh +gitlab-ctl repmgr cluster show +``` + +The output should be similar to: + +``` +Role | Name | Upstream | Connection String +----------+--------------|--------------|-------------------------------------------------------------------- +* master | MASTER | | host=MASTER port=5432 user=gitlab_repmgr dbname=gitlab_repmgr + standby | STANDBY | MASTER | host=STANDBY port=5432 user=gitlab_repmgr dbname=gitlab_repmgr +``` + +If the 'Role' column for any node says "FAILED", check the +[Troubleshooting section](#troubleshooting) before proceeding. + +Also, check that the check master command works successfully on each node: + +```sh +su - gitlab-consul +gitlab-ctl repmgr-check-master || echo 'This node is a standby repmgr node' +``` + +This command relies on exit codes to tell Consul whether a particular node is a master +or secondary. The most important thing here is that this command does not produce errors. +If there are errors it's most likely due to incorrect `gitlab-consul` database user permissions. +Check the [Troubleshooting section](#troubleshooting) before proceeding. + +#### Configuring the Pgbouncer node + +1. Make sure you collect [`CONSUL_SERVER_NODES`](#consul-information), [`CONSUL_PASSWORD_HASH`](#consul-information), and [`PGBOUNCER_PASSWORD_HASH`](#pgbouncer-information) before executing the next step. + +1. Edit `/etc/gitlab/gitlab.rb` replacing values noted in the `# START user configuration` section: + + ```ruby + # Disable all components except Pgbouncer and Consul agent + roles ['pgbouncer_role'] + + # Configure Pgbouncer + pgbouncer['admin_users'] = %w(pgbouncer gitlab-consul) + + # Configure Consul agent + consul['watchers'] = %w(postgresql) + + # START user configuration + # Please set the real values as explained in Required Information section + # Replace CONSUL_PASSWORD_HASH with with a generated md5 value + # Replace PGBOUNCER_PASSWORD_HASH with with a generated md5 value + pgbouncer['users'] = { + 'gitlab-consul': { + password: 'CONSUL_PASSWORD_HASH' + }, + 'pgbouncer': { + password: 'PGBOUNCER_PASSWORD_HASH' + } + } + # Replace placeholders: + # + # Y.Y.Y.Y consul1.gitlab.example.com Z.Z.Z.Z + # with the addresses gathered for CONSUL_SERVER_NODES + consul['configuration'] = { + retry_join: %w(Y.Y.Y.Y consul1.gitlab.example.com Z.Z.Z.Z) + } + # + # END user configuration + ``` + + > `pgbouncer_role` was introduced with GitLab 10.3 + +1. [Reconfigure GitLab] for the changes to take effect. + +1. Create a `.pgpass` file so Consule is able to + reload pgbouncer. Enter the `PGBOUNCER_PASSWORD` twice when asked: + + ```sh + gitlab-ctl write-pgpass --host 127.0.0.1 --database pgbouncer --user pgbouncer --hostuser gitlab-consul + ``` + +##### PGBouncer Checkpoint + +1. Ensure the node is talking to the current master: + + ```sh + gitlab-ctl pgb-console # You will be prompted for PGBOUNCER_PASSWORD + ``` + + If there is an error `psql: ERROR: Auth failed` after typing in the + password, ensure you previously generated the MD5 password hashes with the correct + format. The correct format is to concatenate the password and the username: + `PASSWORDUSERNAME`. For example, `Sup3rS3cr3tpgbouncer` would be the text + needed to generate an MD5 password hash for the `pgbouncer` user. + +1. Once the console prompt is available, run the following queries: + + ```sh + show databases ; show clients ; + ``` + + The output should be similar to the following: + + ``` + name | host | port | database | force_user | pool_size | reserve_pool | pool_mode | max_connections | current_connections + ---------------------+-------------+------+---------------------+------------+-----------+--------------+-----------+-----------------+--------------------- + gitlabhq_production | MASTER_HOST | 5432 | gitlabhq_production | | 20 | 0 | | 0 | 0 + pgbouncer | | 6432 | pgbouncer | pgbouncer | 2 | 0 | statement | 0 | 0 + (2 rows) + + type | user | database | state | addr | port | local_addr | local_port | connect_time | request_time | ptr | link | remote_pid | tls + ------+-----------+---------------------+---------+----------------+-------+------------+------------+---------------------+---------------------+-----------+------+------------+----- + C | pgbouncer | pgbouncer | active | 127.0.0.1 | 56846 | 127.0.0.1 | 6432 | 2017-08-21 18:09:59 | 2017-08-21 18:10:48 | 0x22b3880 | | 0 | + (2 rows) + ``` + +#### Configuring the Application nodes + +These will be the nodes running the `gitlab-rails` service. You may have other +attributes set, but the following need to be set. + +1. Edit `/etc/gitlab/gitlab.rb`: + + ```ruby + # Disable PostgreSQL on the application node + postgresql['enable'] = false + + gitlab_rails['db_host'] = 'PGBOUNCER_NODE' + gitlab_rails['db_port'] = 6432 + gitlab_rails['db_password'] = 'POSTGRESQL_USER_PASSWORD' + gitlab_rails['auto_migrate'] = false + ``` + +1. [Reconfigure GitLab] for the changes to take effect. + +##### Application node post-configuration + +Ensure that all migrations ran: + +```sh +gitlab-rake gitlab:db:configure +``` + +> **Note**: If you encounter a `rake aborted!` error stating that PGBouncer is failing to connect to +PostgreSQL it may be that your PGBouncer node's IP address is missing from +PostgreSQL's `trust_auth_cidr_addresses` in `gitlab.rb` on your database nodes. See +[PGBouncer error `ERROR: pgbouncer cannot connect to server`](#pgbouncer-error-error-pgbouncer-cannot-connect-to-server) +in the Troubleshooting section before proceeding. + +##### Ensure GitLab is running + +At this point, your GitLab instance should be up and running. Verify you are +able to login, and create issues and merge requests. If you have troubles check +the [Troubleshooting section](#troubleshooting). + +#### Example configuration + +Here we'll show you some fully expanded example configurations. + +##### Example recommended setup + +This example uses 3 consul servers, 3 postgresql servers, and 1 application node. + +We start with all servers on the same 10.6.0.0/16 private network range, they +can connect to each freely other on those addresses. + +Here is a list and description of each machine and the assigned IP: + +- `10.6.0.11`: Consul 1 +- `10.6.0.12`: Consul 2 +- `10.6.0.13`: Consul 3 +- `10.6.0.21`: PostgreSQL master +- `10.6.0.22`: PostgreSQL secondary +- `10.6.0.23`: PostgreSQL secondary +- `10.6.0.31`: GitLab application + +All passwords are set to `toomanysecrets`, please do not use this password or derived hashes. + +The external_url for GitLab is `http://gitlab.example.com` + +Please note that after the initial configuration, if a failover occurs, the PostgresSQL master will change to one of the available secondaries until it is failed back. + +##### Example recommended setup for Consul servers + +On each server edit `/etc/gitlab/gitlab.rb`: + +```ruby +# Disable all components except Consul +roles ['consul_role'] + +consul['configuration'] = { + server: true, + retry_join: %w(10.6.0.11 10.6.0.12 10.6.0.13) +} +``` + +[Reconfigure Omnibus GitLab][reconfigure GitLab] for the changes to take effect. + +##### Example recommended setup for PostgreSQL servers + +###### Primary node + +On primary node edit `/etc/gitlab/gitlab.rb`: + +```ruby +# Disable all components except PostgreSQL and Repmgr and Consul +roles ['postgres_role'] + +# PostgreSQL configuration +postgresql['listen_address'] = '0.0.0.0' +postgresql['hot_standby'] = 'on' +postgresql['wal_level'] = 'replica' +postgresql['shared_preload_libraries'] = 'repmgr_funcs' + +# Disable automatic database migrations +gitlab_rails['auto_migrate'] = false + +# Configure the consul agent +consul['services'] = %w(postgresql) + +postgresql['pgbouncer_user_password'] = '771a8625958a529132abe6f1a4acb19c' +postgresql['sql_user_password'] = '450409b85a0223a214b5fb1484f34d0f' +postgresql['max_wal_senders'] = 4 + +postgresql['trust_auth_cidr_addresses'] = %w(10.6.0.0/16) +repmgr['trust_auth_cidr_addresses'] = %w(10.6.0.0/16) + +consul['configuration'] = { + retry_join: %w(10.6.0.11 10.6.0.12 10.6.0.13) +} +``` + +[Reconfigure Omnibus GitLab][reconfigure GitLab] for the changes to take effect. + +###### Secondary nodes + +On secondary nodes, edit `/etc/gitlab/gitlab.rb` and add all the configuration +added to primary node, noted above. In addition, append the following +configuration: + +``` +# HA setting to specify if a node should attempt to be master on initialization +repmgr['master_on_initialization'] = false +``` + +[Reconfigure Omnibus GitLab][reconfigure GitLab] for the changes to take effect. + +##### Example recommended setup for application server + +On the server edit `/etc/gitlab/gitlab.rb`: + +```ruby +external_url 'http://gitlab.example.com' + +gitlab_rails['db_host'] = '127.0.0.1' +gitlab_rails['db_port'] = 6432 +gitlab_rails['db_password'] = 'toomanysecrets' +gitlab_rails['auto_migrate'] = false + +postgresql['enable'] = false +pgbouncer['enable'] = true +consul['enable'] = true + +# Configure Pgbouncer +pgbouncer['admin_users'] = %w(pgbouncer gitlab-consul) + +# Configure Consul agent +consul['watchers'] = %w(postgresql) + +pgbouncer['users'] = { + 'gitlab-consul': { + password: '5e0e3263571e3704ad655076301d6ebe' + }, + 'pgbouncer': { + password: '771a8625958a529132abe6f1a4acb19c' + } +} + +consul['configuration'] = { + retry_join: %w(10.6.0.11 10.6.0.12 10.6.0.13) +} +``` + +[Reconfigure Omnibus GitLab][reconfigure GitLab] for the changes to take effect. + +##### Example recommended setup manual steps + +After deploying the configuration follow these steps: + +1. On `10.6.0.21`, our primary database + + Enable the `pg_trgm` extension + + ```sh gitlab-psql -d gitlabhq_production - + ``` + + ``` CREATE EXTENSION pg_trgm; + ``` + +1. On `10.6.0.22`, our first standby database - # Output: + Make this node a standby of the primary - CREATE EXTENSION + ```sh + gitlab-ctl repmgr standby setup 10.6.0.21 ``` -1. Exit the database prompt by typing `\q` and Enter. -1. Exit the `gitlab-psql` user by running `exit` twice. -1. Run `sudo gitlab-ctl reconfigure` a final time. -1. Configure the GitLab application servers with the appropriate details. - This step is covered in [Configuring GitLab for HA](gitlab.md). ---- +1. On `10.6.0.23`, our second standby database + + Make this node a standby of the primary + + ```sh + gitlab-ctl repmgr standby setup 10.6.0.21 + ``` + +1. On `10.6.0.31`, our application server + + Set gitlab-consul's pgbouncer password to `toomanysecrets` + + ```sh + gitlab-ctl write-pgpass --host 127.0.0.1 --database pgbouncer --user pgbouncer --hostuser gitlab-consul + ``` + + Run database migrations + + ```sh + gitlab-rake gitlab:db:configure + ``` + +#### Example minimal setup + +This example uses 3 postgresql servers, and 1 application node. + +It differs from the [recommended setup](#example-recommended-setup) by moving the consul servers into the same servers we use for PostgreSQL. +The trade-off is between reducing server counts, against the increased operational complexity of needing to deal with postgres [failover](#failover-procedure) and [restore](#restore-procedure) procedures in addition to [consul outage recovery](consul.md#outage-recovery) on the same set of machines. + +In this example we start with all servers on the same 10.6.0.0/16 private network range, they can connect to each freely other on those addresses. + +Here is a list and description of each machine and the assigned IP: + +- `10.6.0.21`: PostgreSQL master +- `10.6.0.22`: PostgreSQL secondary +- `10.6.0.23`: PostgreSQL secondary +- `10.6.0.31`: GitLab application + +All passwords are set to `toomanysecrets`, please do not use this password or derived hashes. + +The external_url for GitLab is `http://gitlab.example.com` + +Please note that after the initial configuration, if a failover occurs, the PostgresSQL master will change to one of the available secondaries until it is failed back. + +##### Example minimal configuration for database servers + +##### Primary node + +On primary database node edit `/etc/gitlab/gitlab.rb`: + +```ruby +# Disable all components except PostgreSQL, Repmgr, and Consul +roles ['postgres_role'] + +# PostgreSQL configuration +postgresql['listen_address'] = '0.0.0.0' +postgresql['hot_standby'] = 'on' +postgresql['wal_level'] = 'replica' +postgresql['shared_preload_libraries'] = 'repmgr_funcs' + +# Disable automatic database migrations +gitlab_rails['auto_migrate'] = false + +# Configure the consul agent +consul['services'] = %w(postgresql) + +postgresql['pgbouncer_user_password'] = '771a8625958a529132abe6f1a4acb19c' +postgresql['sql_user_password'] = '450409b85a0223a214b5fb1484f34d0f' +postgresql['max_wal_senders'] = 4 + +postgresql['trust_auth_cidr_addresses'] = %w(10.6.0.0/16) +repmgr['trust_auth_cidr_addresses'] = %w(10.6.0.0/16) + +consul['configuration'] = { + server: true, + retry_join: %w(10.6.0.21 10.6.0.22 10.6.0.23) +} +``` + +[Reconfigure Omnibus GitLab][reconfigure GitLab] for the changes to take effect. + +###### Secondary nodes + +On secondary nodes, edit `/etc/gitlab/gitlab.rb` and add all the information added +to primary node, noted above. In addition, append the following configuration + +``` +# HA setting to specify if a node should attempt to be master on initialization +repmgr['master_on_initialization'] = false +``` + +##### Example minimal configuration for application server + +On the server edit `/etc/gitlab/gitlab.rb`: + +```ruby +external_url 'http://gitlab.example.com' + +gitlab_rails['db_host'] = '127.0.0.1' +gitlab_rails['db_port'] = 6432 +gitlab_rails['db_password'] = 'toomanysecrets' +gitlab_rails['auto_migrate'] = false + +postgresql['enable'] = false +pgbouncer['enable'] = true +consul['enable'] = true + +# Configure Pgbouncer +pgbouncer['admin_users'] = %w(pgbouncer gitlab-consul) + +# Configure Consul agent +consul['watchers'] = %w(postgresql) + +pgbouncer['users'] = { + 'gitlab-consul': { + password: '5e0e3263571e3704ad655076301d6ebe' + }, + 'pgbouncer': { + password: '771a8625958a529132abe6f1a4acb19c' + } +} + +consul['configuration'] = { + retry_join: %w(10.6.0.21 10.6.0.22 10.6.0.23) +} +``` + +[Reconfigure Omnibus GitLab][reconfigure GitLab] for the changes to take effect. + +##### Example minimal setup manual steps + +The manual steps for this configuration are the same as for the [example recommended setup](#example-recommended-setup-manual-steps). + +#### Failover procedure + +By default, if the master database fails, `repmgrd` should promote one of the +standby nodes to master automatically, and consul will update pgbouncer with +the new master. + +If you need to failover manually, you have two options: + +**Shutdown the current master database** + +Run: + +```sh +gitlab-ctl stop postgresql +``` + +The automated failover process will see this and failover to one of the +standby nodes. + +**Or perform a manual failover** + +1. Ensure the old master node is not still active. +1. Login to the server that should become the new master and run: + + ```sh + gitlab-ctl repmgr standby promote + ``` + +1. If there are any other standby servers in the cluster, have them follow + the new master server: + + ```sh + gitlab-ctl repmgr standby follow NEW_MASTER + ``` + +#### Restore procedure + +If a node fails, it can be removed from the cluster, or added back as a standby +after it has been restored to service. + +- If you want to remove the node from the cluster, on any other node in the + cluster, run: + + ```sh + gitlab-ctl repmgr standby unregister --node=X + ``` + + where X is the value of node in `repmgr.conf` on the old server. + + To find this, you can use: + + ```sh + awk -F = '$1 == "node" { print $2 }' /var/opt/gitlab/postgresql/repmgr.conf + ``` + + It will output something like: + + ``` + 959789412 + ``` + + Then you will use this id to unregister the node: + + ```sh + gitlab-ctl repmgr standby unregister --node=959789412 + ``` + +- To add the node as a standby server: + + ```sh + gitlab-ctl repmgr standby follow NEW_MASTER + gitlab-ctl restart repmgrd + ``` + + CAUTION: **Warning:** When the server is brought back online, and before + you switch it to a standby node, repmgr will report that there are two masters. + If there are any clients that are still attempting to write to the old master, + this will cause a split, and the old master will need to be resynced from + scratch by performing a `gitlab-ctl repmgr standby setup NEW_MASTER`. + +#### Alternate configurations + +##### Database authorization + +By default, we give any host on the database network the permission to perform +repmgr operations using PostgreSQL's `trust` method. If you do not want this +level of trust, there are alternatives. + +You can trust only the specific nodes that will be database clusters, or you +can require md5 authentication. + +##### Trust specific addresses + +If you know the IP address, or FQDN of all database and pgbouncer nodes in the +cluster, you can trust only those nodes. + +In `/etc/gitlab/gitlab.rb` on all of the database nodes, set +`repmgr['trust_auth_cidr_addresses']` to an array of strings containing all of +the addresses. + +If setting to a node's FQDN, they must have a corresponding PTR record in DNS. +If setting to a node's IP address, specify it as `XXX.XXX.XXX.XXX/32`. + +For example: + +```ruby +repmgr['trust_auth_cidr_addresses'] = %w(192.168.1.44/32 db2.example.com) +``` + +##### MD5 Authentication + +If you are running on an untrusted network, repmgr can use md5 authentication +with a [.pgpass file](https://www.postgresql.org/docs/9.6/static/libpq-pgpass.html) +to authenticate. + +You can specify by IP address, FQDN, or by subnet, using the same format as in +the previous section: + +1. On the current master node, create a password for the `gitlab` and + `gitlab_repmgr` user: + + ```sh + gitlab-psql -d template1 + template1=# \password gitlab_repmgr + Enter password: **** + Confirm password: **** + template1=# \password gitlab + ``` + +1. On each database node: + + 1. Edit `/etc/gitlab/gitlab.rb`: + 1. Ensure `repmgr['trust_auth_cidr_addresses']` is **not** set + 1. Set `postgresql['md5_auth_cidr_addresses']` to the desired value + 1. Set `postgresql['sql_replication_user'] = 'gitlab_repmgr'` + 1. Reconfigure with `gitlab-ctl reconfigure` + 1. Restart postgresql with `gitlab-ctl restart postgresql` + + 1. Create a `.pgpass` file. Enter the `gitlab_repmgr` password twice to + when asked: + + ```sh + gitlab-ctl write-pgpass --user gitlab_repmgr --hostuser gitlab-psql --database '*' + ``` + +1. On each pgbouncer node, edit `/etc/gitlab/gitlab.rb`: + 1. Ensure `gitlab_rails['db_password']` is set to the plaintext password for + the `gitlab` database user + 1. [Reconfigure GitLab] for the changes to take effect + +## Troubleshooting + +#### Consul and PostgreSQL changes not taking effect. + +Due to the potential impacts, `gitlab-ctl reconfigure` only reloads Consul and PostgreSQL, it will not restart the services. However, not all changes can be activated by reloading. + +To restart either service, run `gitlab-ctl restart SERVICE` + +For PostgreSQL, it is usually safe to restart the master node by default. Automatic failover defaults to a 1 minute timeout. Provided the database returns before then, nothing else needs to be done. To be safe, you can stop `repmgrd` on the standby nodes first with `gitlab-ctl stop repmgrd`, then start afterwards with `gitlab-ctl start repmgrd`. + +On the consul server nodes, it is important to restart the consul service in a controlled fashion. Read our [consul documentation](consul.md#restarting-the-server-cluster) for instructions on how to restart the service. + +#### `gitlab-ctl repmgr-check-master` command produces errors + +If this command displays errors about database permissions it is likely that something failed during +install, resulting in the `gitlab-consul` database user getting incorrect permissions. Follow these +steps to fix the problem: + +1. On the master database node, connect to the database prompt - `gitlab-psql -d template1` +1. Delete the `gitlab-consul` user - `DROP USER "gitlab-consul";` +1. Exit the database prompt - `\q` +1. [Reconfigure GitLab] and the user will be re-added with the proper permissions. +1. Change to the `gitlab-consul` user - `su - gitlab-consul` +1. Try the check command again - `gitlab-ctl repmgr-check-master`. + +Now there should not be errors. If errors still occur then there is another problem. + +#### PGBouncer error `ERROR: pgbouncer cannot connect to server` + +You may get this error when running `gitlab-rake gitlab:db:configure` or you +may see the error in the PGBouncer log file. + +``` +PG::ConnectionBad: ERROR: pgbouncer cannot connect to server +``` + +The problem may be that your PGBouncer node's IP address is not included in the +`trust_auth_cidr_addresses` setting in `/etc/gitlab/gitlab.rb` on the database nodes. + +You can confirm that this is the issue by checking the PostgreSQL log on the master +database node. If you see the following error then `trust_auth_cidr_addresses` +is the problem. + +``` +2018-03-29_13:59:12.11776 FATAL: no pg_hba.conf entry for host "123.123.123.123", user "pgbouncer", database "gitlabhq_production", SSL off +``` + +To fix the problem, add the IP address to `/etc/gitlab/gitlab.rb`. + +``` +postgresql['trust_auth_cidr_addresses'] = %w(123.123.123.123/32 <other_cidrs>) +``` + +[Reconfigure GitLab] for the changes to take effect. + +#### Issues with other components + +If you're running into an issue with a component not outlined here, be sure to check the troubleshooting section of their specific documentation page. + +- [Consul](consul.md#troubleshooting) +- [PostgreSQL](http://docs.gitlab.com/omnibus/settings/database.html#troubleshooting) +- [GitLab application](gitlab.md#troubleshooting) + +## Configure using Omnibus + +**Note**: We recommend that you follow the instructions here for a full [PostgreSQL cluster](#high-availability-with-gitlab-omnibus-premium-only). +If you are reading this section due to an old bookmark, you can find that old documentation [in the repository](https://gitlab.com/gitlab-org/gitlab-ce/blob/v10.1.4/doc/administration/high_availability/database.md#configure-using-omnibus). Read more on high-availability configuration: @@ -114,3 +1160,6 @@ Read more on high-availability configuration: 1. [Configure NFS](nfs.md) 1. [Configure the GitLab application servers](gitlab.md) 1. [Configure the load balancers](load_balancer.md) +1. [Manage the bundled Consul cluster](consul.md) + +[reconfigure GitLab]: ../restart_gitlab.md#omnibus-gitlab-reconfigure diff --git a/doc/administration/high_availability/gitaly.md b/doc/administration/high_availability/gitaly.md new file mode 100644 index 00000000000..1d8e6c999cb --- /dev/null +++ b/doc/administration/high_availability/gitaly.md @@ -0,0 +1,21 @@ +# Configuring Gitaly for Scaled and High Availability + +Gitaly does not yet support full high availability. However, Gitaly is quite +stable and is in use on GitLab.com. Scaled and highly available GitLab environments +should consider using Gitaly on a separate node. + +See the [Gitaly HA Epic](https://gitlab.com/groups/gitlab-org/-/epics/289) to +track plans and progress toward high availability support. + +This document is relevant for [Scaled Architecture](README.md#scalable-architecture-examples) +environments and [High Availability Architecture](README.md#high-availability-architecture-examples). + +## Running Gitaly on its own server + +See [Running Gitaly on its own server](../gitaly/index.md#running-gitaly-on-its-own-server) +in Gitaly documentation. + +Continue configuration of other components by going back to: + +- [Scaled Architectures](README.md#scalable-architecture-examples) +- [High Availability Architectures](README.md#high-availability-architecture-examples) diff --git a/doc/administration/high_availability/gitlab.md b/doc/administration/high_availability/gitlab.md index e554c06532e..888426ece5c 100644 --- a/doc/administration/high_availability/gitlab.md +++ b/doc/administration/high_availability/gitlab.md @@ -1,8 +1,4 @@ -# Configuring GitLab for HA - -Assuming you have already configured a [database](database.md), [Redis](redis.md), and [NFS](nfs.md), you can -configure the GitLab application server(s) now. Complete the steps below -for each GitLab application server in your environment. +# Configuring GitLab Scaling and High Availability > **Note:** There is some additional configuration near the bottom for additional GitLab application servers. It's important to read and understand @@ -146,6 +142,14 @@ the share is exported and exists on the NFS server and try to remount. --- +## Upgrading GitLab HA + +GitLab HA installations can be upgraded with no downtime, but the +upgrade process must be carefully coordinated to avoid failures. See the +[Omnibus GitLab multi-node upgrade +document](https://docs.gitlab.com/omnibus/update/#multi-node--ha-deployment) +for more details. + Read more on high-availability configuration: 1. [Configure the database](database.md) diff --git a/doc/administration/high_availability/img/fully-distributed.png b/doc/administration/high_availability/img/fully-distributed.png Binary files differnew file mode 100644 index 00000000000..ad23207134e --- /dev/null +++ b/doc/administration/high_availability/img/fully-distributed.png diff --git a/doc/administration/high_availability/img/geo-ha-diagram.png b/doc/administration/high_availability/img/geo-ha-diagram.png Binary files differnew file mode 100644 index 00000000000..da5d612827c --- /dev/null +++ b/doc/administration/high_availability/img/geo-ha-diagram.png diff --git a/doc/administration/high_availability/img/horizontal.png b/doc/administration/high_availability/img/horizontal.png Binary files differnew file mode 100644 index 00000000000..c3bd489d96f --- /dev/null +++ b/doc/administration/high_availability/img/horizontal.png diff --git a/doc/administration/high_availability/img/hybrid.png b/doc/administration/high_availability/img/hybrid.png Binary files differnew file mode 100644 index 00000000000..7d4a56bf0ea --- /dev/null +++ b/doc/administration/high_availability/img/hybrid.png diff --git a/doc/administration/high_availability/img/pg_ha_architecture.png b/doc/administration/high_availability/img/pg_ha_architecture.png Binary files differnew file mode 100644 index 00000000000..ef870f652ae --- /dev/null +++ b/doc/administration/high_availability/img/pg_ha_architecture.png diff --git a/doc/administration/high_availability/nfs.md b/doc/administration/high_availability/nfs.md index 74b0e2c8184..490c2699458 100644 --- a/doc/administration/high_availability/nfs.md +++ b/doc/administration/high_availability/nfs.md @@ -37,6 +37,32 @@ options: circumstances it could lead to data loss if a failure occurs before data has synced. +### Improving NFS performance with GitLab + +NOTE: **Note:** This is only available starting in certain versions of GitLab: 11.5.11, +11.6.11, 11.7.12, 11.8.8, 11.9.0 and up (e.g. 11.10, 11.11, etc.) + +If you are using NFS to share Git data, we recommend that you enable a +number of feature flags that will allow GitLab application processes to +access Git data directly instead of going through the [Gitaly +service](../gitaly/index.md). Depending on your workload and disk +performance, these flags may help improve performance. See [the +issue](https://gitlab.com/gitlab-org/gitlab-ce/issues/57317) for more +details. + +To do this, run the Rake task: + +```sh +sudo gitlab-rake gitlab:features:enable_rugged +``` + +If you need to undo this setting for some reason such as switching to [Gitaly without NFS](gitaly.md) +(recommended), run: + +```sh +sudo gitlab-rake gitlab:features:disable_rugged +``` + ### Known issues On some customer systems, we have seen NFS clients slow precipitously due to @@ -47,9 +73,8 @@ bug](https://bugzilla.redhat.com/show_bug.cgi?id=1552203) that may be fixed in [more recent kernels with this commit](https://github.com/torvalds/linux/commit/95da1b3a5aded124dd1bda1e3cdb876184813140). -Users encountering a similar issue may be advised to disable the NFS server -delegation feature, which is an optimization to reduce the number of network -round-trips needed to read or write files. To disable NFS server delegations +GitLab recommends all NFS users disable the NFS server +delegation feature. To disable NFS server delegations on an Linux NFS server, do the following: 1. On the NFS server, run: @@ -78,6 +103,11 @@ stored on a local volume. For more details on another person's experience with EFS, see [Amazon's Elastic File System: Burst Credits](https://rawkode.com/2017/04/16/amazons-elastic-file-system-burst-credits/) +## Avoid using CephFS and GlusterFS + +GitLab strongly recommends against using CephFS and GlusterFS. +These distributed file systems are not well-suited for GitLab's input/output access patterns because git uses many small files and access times and file locking times to propagate will make git activity very slow. + ## Avoid using PostgreSQL with NFS GitLab strongly recommends against running your PostgreSQL database @@ -93,7 +123,7 @@ Additionally, this configuration is specifically warned against in the >to the NFS server can cause data corruption problems. For supported database architecture, please see our documentation on -[Configuring a Database for GitLab HA](https://docs.gitlab.com/ee/administration/high_availability/database.html). +[Configuring a Database for GitLab HA](database.md). ## NFS Client mount options @@ -122,7 +152,6 @@ mountpoint └── gitlab-data ├── builds ├── git-data - ├── home-git ├── shared └── uploads ``` @@ -135,16 +164,11 @@ configuration to move each data location to a subdirectory: ```ruby git_data_dirs({"default" => { "path" => "/gitlab-nfs/gitlab-data/git-data"} }) -user['home'] = '/gitlab-nfs/gitlab-data/home' gitlab_rails['uploads_directory'] = '/gitlab-nfs/gitlab-data/uploads' gitlab_rails['shared_path'] = '/gitlab-nfs/gitlab-data/shared' gitlab_ci['builds_directory'] = '/gitlab-nfs/gitlab-data/builds' ``` -To move the `git` home directory, all GitLab services must be stopped. Run -`gitlab-ctl stop && initctl stop gitlab-runsvdir`. Then continue with the -reconfigure. - Run `sudo gitlab-ctl reconfigure` to start using the central location. Please be aware that if you had existing data you will need to manually copy/rsync it to these new locations and then restart GitLab. @@ -174,14 +198,13 @@ are empty before attempting a restore. Read more about the ## Multiple NFS mounts -When using default Omnibus configuration you will need to share 5 data locations +When using default Omnibus configuration you will need to share 4 data locations between all GitLab cluster nodes. No other locations should be shared. The -following are the 5 locations need to be shared: +following are the 4 locations need to be shared: | Location | Description | Default configuration | | -------- | ----------- | --------------------- | | `/var/opt/gitlab/git-data` | Git repository data. This will account for a large portion of your data | `git_data_dirs({"default" => { "path" => "/var/opt/gitlab/git-data"} })` -| `/var/opt/gitlab/.ssh` | SSH `authorized_keys` file and keys used to import repositories from some other Git services | `user['home'] = '/var/opt/gitlab/'` | `/var/opt/gitlab/gitlab-rails/uploads` | User uploaded attachments | `gitlab_rails['uploads_directory'] = '/var/opt/gitlab/gitlab-rails/uploads'` | `/var/opt/gitlab/gitlab-rails/shared` | Build artifacts, GitLab Pages, LFS objects, temp files, etc. If you're using LFS this may also account for a large portion of your data | `gitlab_rails['shared_path'] = '/var/opt/gitlab/gitlab-rails/shared'` | `/var/opt/gitlab/gitlab-ci/builds` | GitLab CI build traces | `gitlab_ci['builds_directory'] = '/var/opt/gitlab/gitlab-ci/builds'` diff --git a/doc/administration/high_availability/nfs_host_client_setup.md b/doc/administration/high_availability/nfs_host_client_setup.md new file mode 100644 index 00000000000..a8d69b9ab0a --- /dev/null +++ b/doc/administration/high_availability/nfs_host_client_setup.md @@ -0,0 +1,135 @@ +# Configuring NFS for GitLab HA + +Setting up NFS for a GitLab HA setup allows all applications nodes in a cluster +to share the same files and maintain data consistency. Application nodes in an HA +setup act as clients while the NFS server plays host. + +> Note: The instructions provided in this documentation allow for setting a quick +proof of concept but will leave NFS as potential single point of failure and +therefore not recommended for use in production. Explore options such as [Pacemaker +and Corosync](http://clusterlabs.org/) for highly available NFS in production. + +Below are instructions for setting up an application node(client) in an HA cluster +to read from and write to a central NFS server(host). + +NOTE: **Note:** +Using EFS may negatively impact performance. Please review the [relevant documentation](nfs.md#avoid-using-awss-elastic-file-system-efs) for additional details. + +## NFS Server Setup + +> Follow the instructions below to set up and configure your NFS server. + +### Step 1 - Install NFS Server on Host + +Installing the nfs-kernel-server package allows you to share directories with the clients running the GitLab application. + +```sh +apt-get update +apt-get install nfs-kernel-server +``` + +### Step 2 - Export Host's Home Directory to Client + +In this setup we will share the home directory on the host with the client. Edit the exports file as below to share the host's home directory with the client. If you have multiple clients running GitLab you must enter the client IP addresses in line in the `/etc/exports` file. + +```text +#/etc/exports for one client +/home <client-ip-address>(rw,sync,no_root_squash,no_subtree_check) + +#/etc/exports for three clients +/home <client-ip-address>(rw,sync,no_root_squash,no_subtree_check) <client-2-ip-address>(rw,sync,no_root_squash,no_subtree_check) <client-3-ip-address>(rw,sync,no_root_squash,no_subtree_check) +``` + +Restart the NFS server after making changes to the `exports` file for the changes +to take effect. + +```sh +systemctl restart nfs-kernel-server +``` + +NOTE: **Note:** +You may need to update your server's firewall. See the [firewall section](#nfs-in-a-firewalled-environment) at the end of this guide. + +## Client/ GitLab application node Setup + +> Follow the instructions below to connect any GitLab rails application node running +inside your HA environment to the NFS server configured above. + +### Step 1 - Install NFS Common on Client + +The nfs-common provides NFS functionality without installing server components which +we don't need running on the application nodes. + +```sh +apt-get update +apt-get install nfs-common +``` + +### Step 2 - Create Mount Points on Client + +Create a directory on the client that we can mount the shared directory from the host. +Please note that if your mount point directory contains any files they will be hidden +once the remote shares are mounted. An empty/new directory on the client is recommended +for this purpose. + +```sh +mkdir -p /nfs/home +``` + +Confirm that the mount point works by mounting it on the client and checking that +it is mounted with the command below: + +```sh +mount <host_ip_address>:/home +df -h +``` + +### Step 3 - Set up Automatic Mounts on Boot + +Edit `/etc/fstab` on client as below to mount the remote shares automatically at boot. +Note that GitLab requires advisory file locking, which is only supported natively in +NFS version 4. NFSv3 also supports locking as long as Linux Kernel 2.6.5+ is used. +We recommend using version 4 and do not specifically test NFSv3. + +```text +#/etc/fstab +165.227.159.85:/home /nfs/home nfs4 defaults,soft,rsize=1048576,wsize=1048576,noatime,nofail,lookupcache=positive 0 2 +``` + +Reboot the client and confirm that the mount point is mounted automatically. + +### Step 4 - Set up GitLab to Use NFS mounts + +When using the default Omnibus configuration you will need to share 5 data locations +between all GitLab cluster nodes. No other locations should be shared. Changing the +default file locations in `gitlab.rb` on the client allows you to have one main mount +point and have all the required locations as subdirectories to use the NFS mount for +git-data. + +```text +git_data_dirs({"default" => {"path" => "/nfs/home/var/opt/gitlab-data/git-data"}}) +gitlab_rails['uploads_directory'] = '/nfs/home/var/opt/gitlab-data/uploads' +gitlab_rails['shared_path'] = '/nfs/home/var/opt/gitlab-data/shared' +gitlab_ci['builds_directory'] = '/nfs/home/var/opt/gitlab-data/builds' +``` + +Save the changes in `gitlab.rb` and run `gitlab-ctl reconfigure`. + +## NFS in a Firewalled Environment + +If the traffic between your NFS server and NFS client(s) is subject to port filtering +by a firewall, then you will need to reconfigure that firewall to allow NFS communication. + +[This guide from TDLP](http://tldp.org/HOWTO/NFS-HOWTO/security.html#FIREWALLS) +covers the basics of using NFS in a firewalled environment. Additionally, we encourage you to +search for and review the specific documentation for your OS/distro and your firewall software. + +Example for Ubuntu: + +Check that NFS traffic from the client is allowed by the firewall on the host by running +the command: `sudo ufw status`. If it's being blocked, then you can allow traffic from a specific +client with the command below. + +```sh +sudo ufw allow from <client-ip-address> to any port nfs +``` diff --git a/doc/administration/high_availability/pgbouncer.md b/doc/administration/high_availability/pgbouncer.md new file mode 100644 index 00000000000..762179cf756 --- /dev/null +++ b/doc/administration/high_availability/pgbouncer.md @@ -0,0 +1,132 @@ +# Working with the bundle Pgbouncer service + +## Overview + +As part of its High Availability stack, GitLab Premium includes a bundled version of [Pgbouncer](https://pgbouncer.github.io/) that can be managed through `/etc/gitlab/gitlab.rb`. + +In a High Availability setup, Pgbouncer is used to seamlessly migrate database connections between servers in a failover scenario. + +Additionally, it can be used in a non-HA setup to pool connections, speeding up response time while reducing resource usage. + +It is recommended to run pgbouncer alongside the `gitlab-rails` service, or on its own dedicated node in a cluster. + +## Operations + +### Running Pgbouncer as part of an HA GitLab installation + +See our [HA documentation for PostgreSQL](database.md) for information on running pgbouncer as part of a HA setup + +### Running Pgbouncer as part of a non-HA GitLab installation + +1. Generate PGBOUNCER_USER_PASSWORD_HASH with the command `gitlab-ctl pg-password-md5 pgbouncer` + +1. Generate SQL_USER_PASSWORD_HASH with the command `gitlab-ctl pg-password-md5 gitlab`. We'll also need to enter the plaintext SQL_USER_PASSWORD later + +1. On your database node, ensure the following is set in your `/etc/gitlab/gitlab.rb` + + ```ruby + postgresql['pgbouncer_user_password'] = 'PGBOUNCER_USER_PASSWORD_HASH' + postgresql['sql_user_password'] = 'SQL_USER_PASSWORD_HASH' + postgresql['listen_address'] = 'XX.XX.XX.Y' # Where XX.XX.XX.Y is the ip address on the node postgresql should listen on + postgresql['md5_auth_cidr_addresses'] = %w(AA.AA.AA.B/32) # Where AA.AA.AA.B is the IP address of the pgbouncer node + ``` + +1. Run `gitlab-ctl reconfigure` + + **Note:** If the database was already running, it will need to be restarted after reconfigure by running `gitlab-ctl restart postgresql`. + +1. On the node you are running pgbouncer on, make sure the following is set in `/etc/gitlab/gitlab.rb` + + ```ruby + pgbouncer['enable'] = true + pgbouncer['databases'] = { + gitlabhq_production: { + host: 'DATABASE_HOST', + user: 'pgbouncer', + password: 'PGBOUNCER_USER_PASSWORD_HASH' + } + } + ``` + +1. Run `gitlab-ctl reconfigure` + +1. On the node running unicorn, make sure the following is set in `/etc/gitlab/gitlab.rb` + + ```ruby + gitlab_rails['db_host'] = 'PGBOUNCER_HOST' + gitlab_rails['db_port'] = '6432' + gitlab_rails['db_password'] = 'SQL_USER_PASSWORD' + ``` + +1. Run `gitlab-ctl reconfigure` + +1. At this point, your instance should connect to the database through pgbouncer. If you are having issues, see the [Troubleshooting](#troubleshooting) section + +### Interacting with pgbouncer + +#### Administrative console + +As part of omnibus-gitlab, we provide a command `gitlab-ctl pgb-console` to automatically connect to the pgbouncer administrative console. Please see the [pgbouncer documentation](https://pgbouncer.github.io/usage.html#admin-console) for detailed instructions on how to interact with the console. + +To start a session, run + +```shell +# gitlab-ctl pgb-console +Password for user pgbouncer: +psql (9.6.8, server 1.7.2/bouncer) +Type "help" for help. + +pgbouncer=# +``` + +The password you will be prompted for is the PGBOUNCER_USER_PASSWORD + +To get some basic information about the instance, run +```shell +pgbouncer=# show databases; show clients; show servers; + name | host | port | database | force_user | pool_size | reserve_pool | pool_mode | max_connections | current_connections +---------------------+-----------+------+---------------------+------------+-----------+--------------+-----------+-----------------+--------------------- + gitlabhq_production | 127.0.0.1 | 5432 | gitlabhq_production | | 100 | 5 | | 0 | 1 + pgbouncer | | 6432 | pgbouncer | pgbouncer | 2 | 0 | statement | 0 | 0 +(2 rows) + + type | user | database | state | addr | port | local_addr | local_port | connect_time | request_time | ptr | link +| remote_pid | tls +------+-----------+---------------------+--------+-----------+-------+------------+------------+---------------------+---------------------+-----------+------ ++------------+----- + C | gitlab | gitlabhq_production | active | 127.0.0.1 | 44590 | 127.0.0.1 | 6432 | 2018-04-24 22:13:10 | 2018-04-24 22:17:10 | 0x12444c0 | +| 0 | + C | gitlab | gitlabhq_production | active | 127.0.0.1 | 44592 | 127.0.0.1 | 6432 | 2018-04-24 22:13:10 | 2018-04-24 22:17:10 | 0x12447c0 | +| 0 | + C | gitlab | gitlabhq_production | active | 127.0.0.1 | 44594 | 127.0.0.1 | 6432 | 2018-04-24 22:13:10 | 2018-04-24 22:17:10 | 0x1244940 | +| 0 | + C | gitlab | gitlabhq_production | active | 127.0.0.1 | 44706 | 127.0.0.1 | 6432 | 2018-04-24 22:14:22 | 2018-04-24 22:16:31 | 0x1244ac0 | +| 0 | + C | gitlab | gitlabhq_production | active | 127.0.0.1 | 44708 | 127.0.0.1 | 6432 | 2018-04-24 22:14:22 | 2018-04-24 22:15:15 | 0x1244c40 | +| 0 | + C | gitlab | gitlabhq_production | active | 127.0.0.1 | 44794 | 127.0.0.1 | 6432 | 2018-04-24 22:15:15 | 2018-04-24 22:15:15 | 0x1244dc0 | +| 0 | + C | gitlab | gitlabhq_production | active | 127.0.0.1 | 44798 | 127.0.0.1 | 6432 | 2018-04-24 22:15:15 | 2018-04-24 22:16:31 | 0x1244f40 | +| 0 | + C | pgbouncer | pgbouncer | active | 127.0.0.1 | 44660 | 127.0.0.1 | 6432 | 2018-04-24 22:13:51 | 2018-04-24 22:17:12 | 0x1244640 | +| 0 | +(8 rows) + + type | user | database | state | addr | port | local_addr | local_port | connect_time | request_time | ptr | link | rem +ote_pid | tls +------+--------+---------------------+-------+-----------+------+------------+------------+---------------------+---------------------+-----------+------+---- +--------+----- + S | gitlab | gitlabhq_production | idle | 127.0.0.1 | 5432 | 127.0.0.1 | 35646 | 2018-04-24 22:15:15 | 2018-04-24 22:17:10 | 0x124dca0 | | + 19980 | +(1 row) +``` + +## Troubleshooting + +In case you are experiencing any issues connecting through pgbouncer, the first place to check is always the logs: + +```shell +# gitlab-ctl tail pgbouncer +``` + +Additionally, you can check the output from `show databases` in the [Administrative console](#administrative-console). In the output, you would expect to see values in the `host` field for the `gitlabhq_production` database. Additionally, `current_connections` should be greater than 1. diff --git a/doc/administration/high_availability/redis.md b/doc/administration/high_availability/redis.md index 55d85ad86e6..1aaa709fc8f 100644 --- a/doc/administration/high_availability/redis.md +++ b/doc/administration/high_availability/redis.md @@ -1,6 +1,103 @@ -# Configuring Redis for GitLab HA +# Configuring Redis for Scaling and High Availability -> Experimental Redis Sentinel support was [Introduced][ce-1877] in GitLab 8.11. +## Provide your own Redis instance **[CORE ONLY]** + +The following are the requirements for providing your own Redis instance: + +- Redis version 2.8 or higher. Version 3.2 or higher is recommend as this is + what ships with the GitLab Omnibus package. +- Standalone Redis or Redis high availability with Sentinel are supported. Redis + Cluster is not supported. +- Managed Redis from cloud providers such as AWS Elasticache will work. If these + services support high availability, be sure it is not the Redis Cluster type. + +Note the Redis node's IP address or hostname, port, and password (if required). +These will be necessary when configuring the GitLab application servers later. + +## Redis in a Scaled Environment + +This section is relevant for [Scaled Architecture](README.md#scalable-architecture-examples) +environments including [Basic Scaling](README.md#basic-scaling) and +[Full Scaling](README.md#full-scaling). + +### Provide your own Redis instance **[CORE ONLY]** + +If you want to use your own deployed Redis instance(s), +see [Provide your own Redis instance](#provide-your-own-redis-instance-core-only) +for more details. However, you can use the GitLab Omnibus package to easily +deploy the bundled Redis. + +### Standalone Redis using GitLab Omnibus **[CORE ONLY]** + +The GitLab Omnibus package can be used to configure a standalone Redis server. +In this configuration Redis is not highly available, and represents a single +point of failure. However, in a scaled environment the objective is to allow +the environment to handle more users or to increase throughput. Redis itself +is generally stable and can handle many requests so it is an acceptable +trade off to have only a single instance. See [Scaling and High Availability](README.md) +for an overview of GitLab scaling and high availability options. + +The steps below are the minimum necessary to configure a Redis server with +Omnibus: + +1. SSH into the Redis server. +1. [Download/install](https://about.gitlab.com/installation) the Omnibus GitLab + package you want using **steps 1 and 2** from the GitLab downloads page. + - Do not complete any other steps on the download page. + +1. Edit `/etc/gitlab/gitlab.rb` and add the contents: + + ```ruby + ## Enable Redis + redis['enable'] = true + + ## Disable all other services + sidekiq['enable'] = false + gitlab_workhorse['enable'] = false + unicorn['enable'] = false + postgresql['enable'] = false + nginx['enable'] = false + prometheus['enable'] = false + alertmanager['enable'] = false + pgbouncer_exporter['enable'] = false + gitlab_monitor['enable'] = false + gitaly['enable'] = false + + redis['bind'] = '0.0.0.0' + redis['port'] = '6379' + redis['password'] = 'SECRET_PASSWORD_HERE' + + gitlab_rails['auto_migrate'] = false + ``` + +1. [Reconfigure Omnibus GitLab][reconfigure] for the changes to take effect. +1. Note the Redis node's IP address or hostname, port, and + Redis password. These will be necessary when configuring the GitLab + application servers later. + +Advanced configuration options are supported and can be added if +needed. + +Continue configuration of other components by going +[back to Scaled Architectures](README.md#scalable-architecture-examples) + +## Redis with High Availability + +This section is relevant for [High Availability Architecture](README.md#high-availability-architecture-examples) +environments including [Horizontal](README.md#horizontal), +[Hybrid](README.md#hybrid), and +[Fully Distributed](README.md#fully-distributed). + +### Provide your own Redis instance **[CORE ONLY]** + +If you want to use your own deployed Redis instance(s), +see [Provide your own Redis instance](#provide-your-own-redis-instance-core-only) +for more details. However, you can use the GitLab Omnibus package to easily +deploy the bundled Redis. + +### High Availability with GitLab Omnibus **[PREMIUM ONLY]** + +> Experimental Redis Sentinel support was [introduced in GitLab 8.11][ce-1877]. Starting with 8.14, Redis Sentinel is no longer experimental. If you've used it with versions `< 8.14` before, please check the updated documentation here. @@ -14,6 +111,7 @@ a hosted cloud solution or you can use the one that comes bundled with Omnibus GitLab packages. > **Notes:** +> > - Redis requires authentication for High Availability. See > [Redis Security](https://redis.io/topics/security) documentation for more > information. We recommend using a combination of a Redis password and tight @@ -52,9 +150,8 @@ failure. Make sure that you read this document once as a whole before configuring the components below. -### High Availability with Sentinel - > **Notes:** +> > - Starting with GitLab `8.11`, you can configure a list of Redis Sentinel > servers that will monitor a group of Redis servers to provide failover support. > - Starting with GitLab `8.14`, the Omnibus GitLab Enterprise Edition package @@ -231,6 +328,7 @@ Pick the one that suits your needs. This is the section where we install and set up the new Redis instances. > **Notes:** +> > - We assume that you have installed GitLab and all HA components from scratch. If you > already have it installed and running, read how to > [switch from a single-machine installation to Redis HA](#switching-from-an-existing-single-machine-installation-to-redis-ha). @@ -267,10 +365,9 @@ The prerequisites for a HA Redis setup are the following: 1. Edit `/etc/gitlab/gitlab.rb` and add the contents: ```ruby - # Enable the master role and disable all other services in the machine - # (you can still enable Sentinel). - redis_master_role['enable'] = true - + # Specify server role as 'redis_master_role' + roles ['redis_master_role'] + # IP address pointing to a local IP that the other machines can reach to. # You can also set bind to '0.0.0.0' which listen in all interfaces. # If you really need to bind to an external accessible IP, make @@ -284,6 +381,7 @@ The prerequisites for a HA Redis setup are the following: # Set up password authentication for Redis (use the same password in all nodes). redis['password'] = 'redis-password-goes-here' ``` + 1. Only the primary GitLab application server should handle migrations. To prevent database migrations from running on upgrade, add the following @@ -295,6 +393,10 @@ The prerequisites for a HA Redis setup are the following: 1. [Reconfigure Omnibus GitLab][reconfigure] for the changes to take effect. +> Note: You can specify multiple roles like sentinel and redis as: +> roles ['redis_sentinel_role', 'redis_master_role']. Read more about high +> availability roles at https://docs.gitlab.com/omnibus/roles/ + ### Step 2. Configuring the slave Redis instances 1. SSH into the **slave** Redis server. @@ -307,11 +409,9 @@ The prerequisites for a HA Redis setup are the following: 1. Edit `/etc/gitlab/gitlab.rb` and add the contents: ```ruby - # Enable the slave role and disable all other services in the machine - # (you can still enable Sentinel). This will also set automatically - # `redis['master'] = false`. - redis_slave_role['enable'] = true - + # Specify server role as 'redis_slave_role' + roles ['redis_slave_role'] + # IP address pointing to a local IP that the other machines can reach to. # You can also set bind to '0.0.0.0' which listen in all interfaces. # If you really need to bind to an external accessible IP, make @@ -333,17 +433,19 @@ The prerequisites for a HA Redis setup are the following: #redis['master_port'] = 6379 ``` -1. To prevent database migrations from running on upgrade, run: +1. To prevent reconfigure from running automatically on upgrade, run: ``` sudo touch /etc/gitlab/skip-auto-reconfigure ``` - Only the primary GitLab application server should handle migrations. - 1. [Reconfigure Omnibus GitLab][reconfigure] for the changes to take effect. 1. Go through the steps again for all the other slave nodes. +> Note: You can specify multiple roles like sentinel and redis as: +> roles ['redis_sentinel_role', 'redis_slave_role']. Read more about high +> availability roles at https://docs.gitlab.com/omnibus/roles/ + --- These values don't have to be changed again in `/etc/gitlab/gitlab.rb` after @@ -369,7 +471,7 @@ Now that the Redis servers are all set up, let's configure the Sentinel servers. If you are not sure if your Redis servers are working and replicating -correctly, please read the [Troubleshooting Replication](#troubleshooting-replication) +correctly, please read the [Troubleshooting Replication](#troubleshooting-redis-replication) and fix it before proceeding with Sentinel setup. You must have at least `3` Redis Sentinel servers, and they need to @@ -397,13 +499,13 @@ multiple machines with the Sentinel daemon. be duplicate below): ```ruby - redis_sentinel_role['enable'] = true + roles ['redis_sentinel_role'] # Must be the same in every sentinel node redis['master_name'] = 'gitlab-redis' # The same password for Redis authentication you set up for the master node. - redis['password'] = 'redis-password-goes-here' + redis['master_password'] = 'redis-password-goes-here' # The IP of the master Redis node. redis['master_ip'] = '10.0.0.1' @@ -570,8 +672,7 @@ or a failover promotes a different **Master** node. In `/etc/gitlab/gitlab.rb`: ```ruby -redis_master_role['enable'] = true -redis_sentinel_role['enable'] = true +roles ['redis_sentinel_role', 'redis_master_role'] redis['bind'] = '10.0.0.1' redis['port'] = 6379 redis['password'] = 'redis-password-goes-here' @@ -593,8 +694,7 @@ sentinel['quorum'] = 2 In `/etc/gitlab/gitlab.rb`: ```ruby -redis_slave_role['enable'] = true -redis_sentinel_role['enable'] = true +roles ['redis_sentinel_role', 'redis_slave_role'] redis['bind'] = '10.0.0.2' redis['port'] = 6379 redis['password'] = 'redis-password-goes-here' @@ -616,8 +716,7 @@ sentinel['quorum'] = 2 In `/etc/gitlab/gitlab.rb`: ```ruby -redis_slave_role['enable'] = true -redis_sentinel_role['enable'] = true +roles ['redis_sentinel_role', 'redis_slave_role'] redis['bind'] = '10.0.0.3' redis['port'] = 6379 redis['password'] = 'redis-password-goes-here' @@ -640,7 +739,7 @@ In `/etc/gitlab/gitlab.rb`: ```ruby redis['master_name'] = 'gitlab-redis' -redis['password'] = 'redis-password-goes-here' +redis['master_password'] = 'redis-password-goes-here' gitlab_rails['redis_sentinels'] = [ {'host' => '10.0.0.1', 'port' => 26379}, {'host' => '10.0.0.2', 'port' => 26379}, @@ -761,15 +860,11 @@ Before proceeding with the troubleshooting below, check your firewall rules: ### Troubleshooting Redis replication You can check if everything is correct by connecting to each server using -`redis-cli` application, and sending the `INFO` command. +`redis-cli` application, and sending the `info replication` command as below. -If authentication was correctly defined, it should fail with: -`NOAUTH Authentication required` error. Try to authenticate with the -previous defined password with `AUTH redis-password-goes-here` and -try the `INFO` command again. - -Look for the `# Replication` section where you should see some important -information like the `role` of the server. +``` +/opt/gitlab/embedded/bin/redis-cli -h <redis-host-or-ip> -a '<redis-password>' info replication +``` When connected to a `master` redis, you will see the number of connected `slaves`, and a list of each with connection details: @@ -839,7 +934,7 @@ To make sure your configuration is correct: 1. Run in the console: ```ruby - redis = Redis.new(Gitlab::Redis.params) + redis = Redis.new(Gitlab::Redis::SharedState.params) redis.info ``` diff --git a/doc/administration/high_availability/redis_source.md b/doc/administration/high_availability/redis_source.md index 14e2784c419..be6b547372a 100644 --- a/doc/administration/high_availability/redis_source.md +++ b/doc/administration/high_availability/redis_source.md @@ -46,7 +46,7 @@ valuable information for the general setup. Assuming that the Redis master instance IP is `10.0.0.1`: -1. [Install Redis](../../install/installation.md#6-redis) +1. [Install Redis](../../install/installation.md#7-redis). 1. Edit `/etc/redis/redis.conf`: ```conf @@ -72,7 +72,7 @@ Assuming that the Redis master instance IP is `10.0.0.1`: Assuming that the Redis slave instance IP is `10.0.0.2`: -1. [Install Redis](../../install/installation.md#6-redis) +1. [Install Redis](../../install/installation.md#7-redis). 1. Edit `/etc/redis/redis.conf`: ```conf diff --git a/doc/administration/housekeeping.md b/doc/administration/housekeeping.md index 058346df56d..1b01419e062 100644 --- a/doc/administration/housekeeping.md +++ b/doc/administration/housekeeping.md @@ -30,7 +30,7 @@ the `pushes_since_gc` value is 200 a `git gc` will be run. `git add`. - `git repack` ([man page][man-repack]) re-organize existing packs into a single, more efficient pack. -You can find this option under your **[Project] > Edit Project**. +You can find this option under your project's **Settings > General > Advanced**. --- diff --git a/doc/administration/img/custom_hooks_error_msg.png b/doc/administration/img/custom_hooks_error_msg.png Binary files differindex 845f0de19ce..4f25c471908 100644 --- a/doc/administration/img/custom_hooks_error_msg.png +++ b/doc/administration/img/custom_hooks_error_msg.png diff --git a/doc/administration/img/db_load_balancing_postgres_stats.png b/doc/administration/img/db_load_balancing_postgres_stats.png Binary files differnew file mode 100644 index 00000000000..8b311616e7b --- /dev/null +++ b/doc/administration/img/db_load_balancing_postgres_stats.png diff --git a/doc/administration/img/high_availability/active-active-diagram.png b/doc/administration/img/high_availability/active-active-diagram.png Binary files differdeleted file mode 100644 index 4f5984b88fe..00000000000 --- a/doc/administration/img/high_availability/active-active-diagram.png +++ /dev/null diff --git a/doc/administration/img/high_availability/active-passive-diagram.png b/doc/administration/img/high_availability/active-passive-diagram.png Binary files differdeleted file mode 100644 index 3b42ce5911c..00000000000 --- a/doc/administration/img/high_availability/active-passive-diagram.png +++ /dev/null diff --git a/doc/administration/img/housekeeping_settings.png b/doc/administration/img/housekeeping_settings.png Binary files differindex acc4506993a..356de51f0cc 100644 --- a/doc/administration/img/housekeeping_settings.png +++ b/doc/administration/img/housekeeping_settings.png diff --git a/doc/administration/img/instance_review_button.png b/doc/administration/img/instance_review_button.png Binary files differnew file mode 100644 index 00000000000..b7604d7c7e5 --- /dev/null +++ b/doc/administration/img/instance_review_button.png diff --git a/doc/administration/incoming_email.md b/doc/administration/incoming_email.md index 658b2f55d30..8271c579f5b 100644 --- a/doc/administration/incoming_email.md +++ b/doc/administration/incoming_email.md @@ -10,6 +10,8 @@ GitLab has several features based on receiving incoming emails: - [New merge request by email](../user/project/merge_requests/index.md#create-new-merge-requests-by-email): allow GitLab users to create a new merge request by sending an email to a user-specific email address. +- [Service Desk](../user/project/service_desk.md): provide e-mail support to + your customers through GitLab. **[PREMIUM]** ## Requirements diff --git a/doc/administration/index.md b/doc/administration/index.md index 6e08d4633cd..95a0e84deb6 100644 --- a/doc/administration/index.md +++ b/doc/administration/index.md @@ -20,7 +20,7 @@ GitLab Community Edition installations only have access to Core features. GitLab.com is administered by GitLab, Inc., therefore, only GitLab team members have access to its admin configurations. If you're a GitLab.com user, please check the -[user documentation](../user/index.html). +[user documentation](../user/index.md). NOTE: **Note:** Non-administrator users don’t have access to GitLab administration tools and settings. @@ -32,8 +32,15 @@ Learn how to install, configure, update, and maintain your GitLab instance. ### Installing GitLab - [Install](../install/README.md): Requirements, directory structures, and installation methods. + - [Database load balancing](database_load_balancing.md): Distribute database queries among multiple database servers. **[STARTER ONLY]** + - [Omnibus support for external MySQL DB](https://docs.gitlab.com/omnibus/settings/database.html#using-a-mysql-database-management-server-enterprise-edition-only): Omnibus package supports configuring an external MySQL database. **[STARTER ONLY]** + - [Omnibus support for log forwarding](https://docs.gitlab.com/omnibus/settings/logs.html#udp-log-shipping-gitlab-enterprise-edition-only) **[STARTER ONLY]** - [High Availability](high_availability/README.md): Configure multiple servers for scaling or high availability. - [High Availability on AWS](../university/high-availability/aws/README.md): Set up GitLab HA on Amazon AWS. +- [Geo](geo/replication/index.md): Replicate your GitLab instance to other geographic locations as a read-only fully operational version. **[PREMIUM ONLY]** +- [Disaster Recovery](geo/disaster_recovery/index.md): Quickly fail-over to a different site with minimal effort in a disaster situation. **[PREMIUM ONLY]** +- [Pivotal Tile](../install/pivotal/index.md): Deploy GitLab as a pre-configured appliance using Ops Manager (BOSH) for Pivotal Cloud Foundry. **[PREMIUM ONLY]** +- [Add License](../user/admin_area/license.md): Upload a license at install time to unlock features that are in paid tiers of GitLab. **[STARTER ONLY]** ### Configuring GitLab @@ -41,10 +48,11 @@ Learn how to install, configure, update, and maintain your GitLab instance. - [System hooks](../system_hooks/system_hooks.md): Notifications when users, projects and keys are changed. - [Security](../security/README.md): Learn what you can do to further secure your GitLab instance. - [Usage statistics, version check, and usage ping](../user/admin_area/settings/usage_statistics.md): Enable or disable information about your instance to be sent to GitLab, Inc. +- [Global user settings](user_settings.md): Configure instance-wide user permissions. - [Polling](polling.md): Configure how often the GitLab UI polls for updates. - [GitLab Pages configuration](pages/index.md): Enable and configure GitLab Pages. -- [GitLab Pages configuration for GitLab source installations](pages/source.md): Enable and configure GitLab Pages on - [source installations](../install/installation.md#installation-from-source). +- [GitLab Pages configuration for GitLab source installations](pages/source.md): Enable and configure GitLab Pages on [source installations](../install/installation.md#installation-from-source). +- [Uploads configuration](uploads.md): Configure GitLab uploads storage. - [Environment variables](environment_variables.md): Supported environment variables that can be used to override their defaults values in order to configure GitLab. - [Plugins](plugins.md): With custom plugins, GitLab administrators can introduce custom integrations without modifying GitLab's source code. - [Enforcing Terms of Service](../user/admin_area/settings/terms.md) @@ -53,6 +61,9 @@ Learn how to install, configure, update, and maintain your GitLab instance. - [Diff limits](../user/admin_area/diff_limits.md): Configure the diff rendering size limits of branch comparison pages. - [Merge request diffs storage](merge_request_diffs.md): Configure merge requests diffs external storage. - [Broadcast Messages](../user/admin_area/broadcast_messages.md): Send messages to GitLab users through the UI. +- [Elasticsearch](../integration/elasticsearch.md): Enable Elasticsearch to empower GitLab's Advanced Global Search. Useful when you deal with a huge amount of data. **[STARTER ONLY]** +- [External Classification Policy Authorization](../user/admin_area/settings/external_authorization.md) **[PREMIUM ONLY]** +- [Upload a license](../user/admin_area/license.md): Upload a license to unlock features that are in paid tiers of GitLab. **[STARTER ONLY]** - [Admin Area](../user/admin_area/index.md): for self-managed instance-wide configuration and maintenance. #### Customizing GitLab's appearance @@ -62,6 +73,7 @@ Learn how to install, configure, update, and maintain your GitLab instance. - [Branded login page](../customization/branded_login_page.md): Customize the login page with your own logo, title, and description. - [Welcome message](../customization/welcome_message.md): Add a custom welcome message to the sign-in page. - ["New Project" page](../customization/new_project_page.md): Customize the text to be displayed on the page that opens whenever your users create a new project. +- [Additional custom email text](../user/admin_area/settings/email.md#custom-additional-text-premium-only): Add additional custom text to emails sent from GitLab. **[PREMIUM ONLY]** ### Maintaining GitLab @@ -95,33 +107,40 @@ Learn how to install, configure, update, and maintain your GitLab instance. - [Libravatar](../customization/libravatar.md): Use Libravatar instead of Gravatar for user avatars. - [Sign-up restrictions](../user/admin_area/settings/sign_up_restrictions.md): block email addresses of specific domains, or whitelist only specific domains. - [Access restrictions](../user/admin_area/settings/visibility_and_access_controls.md#enabled-git-access-protocols): Define which Git access protocols can be used to talk to GitLab (SSH, HTTP, HTTPS). -- [Authentication and Authorization](auth/README.md): Configure external authentication with LDAP, SAML, CAS and additional providers. See also other [authentication](../topics/authentication/index.md#gitlab-administrators) topics (for example, enforcing 2FA). +- [Authentication and Authorization](auth/README.md): Configure external authentication with LDAP, SAML, CAS and additional providers. + - [Sync LDAP](auth/ldap-ee.md) **[STARTER ONLY]** + - [Kerberos authentication](../integration/kerberos.md) **[STARTER ONLY]** + - See also other [authentication](../topics/authentication/index.md#gitlab-administrators) topics (for example, enforcing 2FA). +- [Email users](../tools/email.md): Email GitLab users from within GitLab. **[STARTER ONLY]** +- [User Cohorts](../user/admin_area/user_cohorts.md): Display the monthly cohorts of new users and their activities over time. +- [Audit logs and events](audit_events.md): View the changes made within the GitLab server for: + - Groups and projects. **[STARTER]** + - Instances. **[PREMIUM ONLY]** +- [Auditor users](auditor_users.md): Users with read-only access to all projects, groups, and other resources on the GitLab instance. **[PREMIUM ONLY]** - [Incoming email](incoming_email.md): Configure incoming emails to allow - users to [reply by email], create [issues by email] and - [merge requests by email], and to enable [Service Desk]. + users to [reply by email](reply_by_email.md), create [issues by email](../user/project/issues/create_new_issue.md#new-issue-via-email) and + [merge requests by email](../user/project/merge_requests/index.md#create-new-merge-requests-by-email), and to enable [Service Desk](../user/project/service_desk.md). - [Postfix for incoming email](reply_by_email_postfix_setup.md): Set up a basic Postfix mail server with IMAP authentication on Ubuntu for incoming emails. - [Abuse reports](../user/admin_area/abuse_reports.md): View and resolve abuse reports from your users. -[reply by email]: reply_by_email.md -[issues by email]: ../user/project/issues/create_new_issue.md#new-issue-via-email -[merge requests by email]: ../user/project/merge_requests/index.md#create-new-merge-requests-by-email - ## Project settings - [Container Registry](container_registry.md): Configure Container Registry with GitLab. - [Issue closing pattern](issue_closing_pattern.md): Customize how to close an issue from commit messages. - [Gitaly](gitaly/index.md): Configuring Gitaly, GitLab's Git repository storage service. -- [Default labels](../user/admin_area/labels.html): Create labels that will be automatically added to every new project. +- [Default labels](../user/admin_area/labels.md): Create labels that will be automatically added to every new project. - [Restrict the use of public or internal projects](../public_access/public_access.md#restricting-the-use-of-public-or-internal-projects): Restrict the use of visibility levels for users when they create a project or a snippet. -- [Custom project templates](https://docs.gitlab.com/ee/user/admin_area/custom_project_templates.html): Configure a set of projects to be used as custom templates when creating a new project. **[PREMIUM ONLY]** +- [Custom project templates](../user/admin_area/custom_project_templates.md): Configure a set of projects to be used as custom templates when creating a new project. **[PREMIUM ONLY]** +- [Packages](packages.md): Enable GitLab to act as a Maven repository or NPM registry. **[PREMIUM ONLY]** ### Repository settings - [Repository checks](repository_checks.md): Periodic Git repository checks. - [Repository storage paths](repository_storage_paths.md): Manage the paths used to store repositories. - [Repository storage rake tasks](raketasks/storage.md): A collection of rake tasks to list and migrate existing projects and attachments associated with it from Legacy storage to Hashed storage. +- [Limit repository size](../user/admin_area/settings/account_and_limit_settings.md): Set a hard limit for your repositories' size. **[STARTER ONLY]** ## Continuous Integration settings @@ -130,7 +149,7 @@ Learn how to install, configure, update, and maintain your GitLab instance. - [Job artifacts](job_artifacts.md): Enable, disable, and configure job artifacts (a set of files and directories which are outputted by a job when it completes successfully). - [Job traces](job_traces.md): Information about the job traces (logs). - [Register Shared and specific Runners](../ci/runners/README.md#registering-a-shared-runner): Learn how to register and configure Shared and specific Runners to your own instance. -- [Shared Runners pipelines quota](../user/admin_area/settings/continuous_integration.md#shared-runners-pipeline-minutes-quota): Limit the usage of pipeline minutes for Shared Runners. +- [Shared Runners pipelines quota](../user/admin_area/settings/continuous_integration.md#shared-runners-pipeline-minutes-quota-starter-only): Limit the usage of pipeline minutes for Shared Runners. **[STARTER ONLY]** - [Enable/disable Auto DevOps](../topics/autodevops/index.md#enablingdisabling-auto-devops): Enable or disable Auto DevOps for your instance. ## Git configuration options @@ -158,6 +177,10 @@ Learn how to install, configure, update, and maintain your GitLab instance. - [Request Profiling](monitoring/performance/request_profiling.md): Get a detailed profile on slow requests. - [Performance Bar](monitoring/performance/performance_bar.md): Get performance information for the current page. +## Analytics + +- [Pseudonymizer](pseudonymizer.md): Export data from GitLab's database to CSV files in a secure way. **[ULTIMATE]** + ## Troubleshooting - [Debugging tips](troubleshooting/debug.md): Tips to debug problems when things go wrong diff --git a/doc/administration/instance_review.md b/doc/administration/instance_review.md new file mode 100644 index 00000000000..b1244f44e95 --- /dev/null +++ b/doc/administration/instance_review.md @@ -0,0 +1,17 @@ +# Instance Review **[CORE ONLY]** + +> [Introduced][6995] in [GitLab Core][ee] 11.3. + +If you are running a medium size instance of GitLab Core edition you are qualified for a free Instance Review. You can find the button in the User menu. + + + +When you click the button you will be redirected to a form with prefilled data obtained from your instance. + +Once you submit the data to GitLab Inc. you can see the initial report. + +Additionally you will be contacted by our team for further review which should help you to improve your usage of GitLab. + +[6995]: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/6995 +[ee]: https://about.gitlab.com/pricing/ + diff --git a/doc/administration/integration/plantuml.md b/doc/administration/integration/plantuml.md index d383d1efe70..82e0c14ffc2 100644 --- a/doc/administration/integration/plantuml.md +++ b/doc/administration/integration/plantuml.md @@ -27,7 +27,7 @@ own PlantUML server is easy in Debian/Ubuntu distributions using Tomcat. First you need to create a `plantuml.war` file from the source code: ``` -sudo apt-get install graphviz openjdk-7-jdk git-core maven +sudo apt-get install graphviz openjdk-8-jdk git-core maven git clone https://github.com/plantuml/plantuml-server.git cd plantuml-server mvn package @@ -57,8 +57,7 @@ you can change these defaults by editing the `/etc/tomcat7/server.xml` file. You need to enable PlantUML integration from Settings under Admin Area. To do that, login with an Admin account and do following: - - in GitLab go to **Admin Area** and then **Settings** - - scroll to bottom of the page until PlantUML section + - in GitLab go to **Admin Area**->**Settings**->**Integrations**->**PlantUML** - check **Enable PlantUML** checkbox - set the PlantUML instance as **PlantUML URL** diff --git a/doc/administration/integration/terminal.md b/doc/administration/integration/terminal.md index 25d85d1687b..c34858cd0db 100644 --- a/doc/administration/integration/terminal.md +++ b/doc/administration/integration/terminal.md @@ -11,7 +11,7 @@ One of the things it uses these credentials for is providing access to ## How it works A detailed overview of the architecture of web terminals and how they work -can be found in [this document](https://gitlab.com/gitlab-org/gitlab-workhorse/blob/master/doc/terminal.md). +can be found in [this document](https://gitlab.com/gitlab-org/gitlab-workhorse/blob/master/doc/channel.md). In brief: - GitLab relies on the user to provide their own Kubernetes credentials, and to @@ -43,6 +43,11 @@ detail below. ## Enabling and disabling terminal support +NOTE: **Note:** AWS Elastic Load Balancers (ELBs) do not support web sockets. +AWS Application Load Balancers (ALBs) must be used if you want web terminals +to work. See [AWS Elastic Load Balancing Product Comparison](https://aws.amazon.com/elasticloadbalancing/features/#compare) +for more information. + As web terminals use WebSockets, every HTTP/HTTPS reverse proxy in front of Workhorse needs to be configured to pass the `Connection` and `Upgrade` headers through to the next one in the chain. If you installed GitLab using Omnibus, or diff --git a/doc/administration/job_artifacts.md b/doc/administration/job_artifacts.md index 25ae535d1ec..05e15fc303b 100644 --- a/doc/administration/job_artifacts.md +++ b/doc/administration/job_artifacts.md @@ -1,14 +1,15 @@ # Jobs artifacts administration > **Notes:** +> > - Introduced in GitLab 8.2 and GitLab Runner 0.7.0. > - Starting with GitLab 8.4 and GitLab Runner 1.0, the artifacts archive format changed to `ZIP`. > - Starting with GitLab 8.17, builds are renamed to jobs. > - This is the administration documentation. For the user guide see [pipelines/job_artifacts](../user/project/pipelines/job_artifacts.md). -Artifacts is a list of files and directories which are attached to a job -after it completes successfully. This feature is enabled by default in all -GitLab installations. Keep reading if you want to know how to disable it. +Artifacts is a list of files and directories which are attached to a job after it +finishes. This feature is enabled by default in all GitLab installations. Keep reading +if you want to know how to disable it. ## Disabling job artifacts @@ -41,8 +42,9 @@ To disable artifacts site-wide, follow the steps below. ## Storing job artifacts -After a successful job, GitLab Runner uploads an archive containing the job -artifacts to GitLab. +GitLab Runner can upload an archive containing the job artifacts to GitLab. By default, +this is done when the job succeeds, but can also be done on failure, or always, via the +[`artifacts:when`](../ci/yaml/README.md#artifactswhen) parameter. ### Using local storage @@ -86,6 +88,7 @@ _The artifacts are stored by default in ### Using object storage > **Notes:** +> > - [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/1762) in > [GitLab Premium](https://about.gitlab.com/pricing/) 9.4. > - Since version 9.5, artifacts are [browsable](../user/project/pipelines/job_artifacts.md#browsing-artifacts), @@ -98,6 +101,9 @@ artifacts, you can use an object storage like AWS S3 instead. This configuration relies on valid AWS credentials to be configured already. Use an object storage option like AWS S3 to store job artifacts. +DANGER: **Danger:** +If you're enabling S3 in [GitLab HA](high_availability/README.md), you will need to have an [NFS mount set up for CI traces and artifacts](high_availability/nfs.md#a-single-nfs-mount) or enable [live tracing](job_traces.md#new-live-trace-architecture). If these settings are not set, you will risk job traces disappearing or not being saved. + ### Object Storage Settings For source installations the following settings are nested under `artifacts:` and then `object_store:`. On omnibus installs they are prefixed by `artifacts_object_store_`. @@ -274,7 +280,7 @@ you can flip the feature flag from a Rails console. ## Set the maximum file size of the artifacts Provided the artifacts are enabled, you can change the maximum file size of the -artifacts through the [Admin area settings](../user/admin_area/settings/continuous_integration.md#maximum-artifacts-size). +artifacts through the [Admin area settings](../user/admin_area/settings/continuous_integration.md#maximum-artifacts-size-core-only). ## Storage statistics diff --git a/doc/administration/job_traces.md b/doc/administration/job_traces.md index 63945506f3c..aa9d87562a3 100644 --- a/doc/administration/job_traces.md +++ b/doc/administration/job_traces.md @@ -10,13 +10,13 @@ In the following table you can see the phases a trace goes through. | Phase | State | Condition | Data flow | Stored path | | ----- | ----- | --------- | --------- | ----------- | -| 1: patching | Live trace | When a job is running | GitLab Runner => Unicorn => file storage |`#{ROOT_PATH}/builds/#{YYYY_mm}/#{project_id}/#{job_id}.log`| -| 2: overwriting | Live trace | When a job is finished | GitLab Runner => Unicorn => file storage |`#{ROOT_PATH}/builds/#{YYYY_mm}/#{project_id}/#{job_id}.log`| -| 3: archiving | Archived trace | After a job is finished | Sidekiq moves live trace to artifacts folder |`#{ROOT_PATH}/shared/artifacts/#{disk_hash}/#{YYYY_mm_dd}/#{job_id}/#{job_artifact_id}/job.log`| +| 1: patching | Live trace | When a job is running | GitLab Runner => Unicorn => file storage |`#{ROOT_PATH}/gitlab-ci/builds/#{YYYY_mm}/#{project_id}/#{job_id}.log`| +| 2: overwriting | Live trace | When a job is finished | GitLab Runner => Unicorn => file storage |`#{ROOT_PATH}/gitlab-ci/builds/#{YYYY_mm}/#{project_id}/#{job_id}.log`| +| 3: archiving | Archived trace | After a job is finished | Sidekiq moves live trace to artifacts folder |`#{ROOT_PATH}/gitlab-rails/shared/artifacts/#{disk_hash}/#{YYYY_mm_dd}/#{job_id}/#{job_artifact_id}/job.log`| | 4: uploading | Archived trace | After a trace is archived | Sidekiq moves archived trace to [object storage](#uploading-traces-to-object-storage) (if configured) |`#{bucket_name}/#{disk_hash}/#{YYYY_mm_dd}/#{job_id}/#{job_artifact_id}/job.log`| The `ROOT_PATH` varies per your environment. For Omnibus GitLab it -would be `/var/opt/gitlab/gitlab-ci`, whereas for installations from source +would be `/var/opt/gitlab`, whereas for installations from source it would be `/home/git/gitlab`. ## Changing the job traces local location diff --git a/doc/administration/logs.md b/doc/administration/logs.md index a836bda7778..c5cfb8d5016 100644 --- a/doc/administration/logs.md +++ b/doc/administration/logs.md @@ -23,16 +23,19 @@ requests from the API are logged to a separate file in `api_json.log`. Each line contains a JSON line that can be ingested by Elasticsearch, Splunk, etc. For example: ```json -{"method":"GET","path":"/gitlab/gitlab-ce/issues/1234","format":"html","controller":"Projects::IssuesController","action":"show","status":200,"duration":229.03,"view":174.07,"db":13.24,"time":"2017-08-08T20:15:54.821Z","params":[{"key":"param_key","value":"param_value"}],"remote_ip":"18.245.0.1","user_id":1,"username":"admin","gitaly_calls":76,"queue_duration": 112.47} +{"method":"GET","path":"/gitlab/gitlab-ce/issues/1234","format":"html","controller":"Projects::IssuesController","action":"show","status":200,"duration":229.03,"view":174.07,"db":13.24,"time":"2017-08-08T20:15:54.821Z","params":[{"key":"param_key","value":"param_value"}],"remote_ip":"18.245.0.1","user_id":1,"username":"admin","gitaly_calls":76,"gitaly_duration":7.41,"queue_duration": 112.47} ``` -In this example, you can see this was a GET request for a specific issue. Notice each line also contains performance data: +In this example, you can see this was a GET request for a specific +issue. Notice each line also contains performance data. All times are in +milliseconds: -1. `duration`: total time in milliseconds taken to retrieve the request -1. `queue_duration`: total time in milliseconds that the request was queued inside GitLab Workhorse +1. `duration`: total time taken to retrieve the request +1. `queue_duration`: total time that the request was queued inside GitLab Workhorse 1. `view`: total time taken inside the Rails views 1. `db`: total time to retrieve data from the database 1. `gitaly_calls`: total number of calls made to Gitaly +1. `gitaly_duration`: total time taken by Gitaly calls User clone/fetch activity using http transport appears in this log as `action: git_upload_pack`. @@ -85,7 +88,7 @@ Introduced in GitLab 10.0, this file lives in It helps you see requests made directly to the API. For example: ```json -{"time":"2018-10-29T12:49:42.123Z","severity":"INFO","duration":709.08,"db":14.59,"view":694.49,"status":200,"method":"GET","path":"/api/v4/projects","params":[{"key":"action","value":"git-upload-pack"},{"key":"changes","value":"_any"},{"key":"key_id","value":"secret"},{"key":"secret_token","value":"[FILTERED]"}],"host":"localhost","ip":"::1","ua":"Ruby","route":"/api/:version/projects","user_id":1,"username":"root","queue_duration":100.31,"gitaly_calls":30} +{"time":"2018-10-29T12:49:42.123Z","severity":"INFO","duration":709.08,"db":14.59,"view":694.49,"status":200,"method":"GET","path":"/api/v4/projects","params":[{"key":"action","value":"git-upload-pack"},{"key":"changes","value":"_any"},{"key":"key_id","value":"secret"},{"key":"secret_token","value":"[FILTERED]"}],"host":"localhost","ip":"::1","ua":"Ruby","route":"/api/:version/projects","user_id":1,"username":"root","queue_duration":100.31,"gitaly_calls":30,"gitaly_duration":5.36} ``` This entry above shows an access to an internal endpoint to check whether an @@ -277,6 +280,28 @@ installations from source. Currently it logs the progress of project imports from the Bitbucket Server importer. Future importers may use this file. +## `auth.log` + +Introduced in GitLab 12.0. This file lives in `/var/log/gitlab/gitlab-rails/auth.log` for +Omnibus GitLab packages or in `/home/git/gitlab/log/auth.log` for +installations from source. + +It logs information whenever [Rack Attack] registers an abusive request. + +## `graphql_json.log` + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/issues/59587) in GitLab 12.0. + +This file lives in `/var/log/gitlab/gitlab-rails/graphql_json.log` for +Omnibus GitLab packages or in `/home/git/gitlab/log/graphql_json.log` for +installations from source. + +GraphQL queries are recorded in that file. For example: + +```json +{"query_string":"query IntrospectionQuery{__schema {queryType { name },mutationType { name }}}...(etc)","variables":{"a":1,"b":2},"complexity":181,"depth":1,"duration":7} +``` + ## Reconfigure Logs Reconfigure log files live in `/var/log/gitlab/reconfigure` for Omnibus GitLab @@ -295,3 +320,4 @@ Omnibus GitLab packages or in `/home/git/gitlab/log/sidekiq_exporter.log` for installations from source. [repocheck]: repository_checks.md +[Rack Attack]: ../security/rack_attack.md diff --git a/doc/administration/maven_packages.md b/doc/administration/maven_packages.md new file mode 100644 index 00000000000..d8551f64ece --- /dev/null +++ b/doc/administration/maven_packages.md @@ -0,0 +1,5 @@ +--- +redirect_to: 'packages.md' +--- + +This document was moved to [another location](packages.md). diff --git a/doc/administration/maven_repository.md b/doc/administration/maven_repository.md new file mode 100644 index 00000000000..d8551f64ece --- /dev/null +++ b/doc/administration/maven_repository.md @@ -0,0 +1,5 @@ +--- +redirect_to: 'packages.md' +--- + +This document was moved to [another location](packages.md). diff --git a/doc/administration/merge_request_diffs.md b/doc/administration/merge_request_diffs.md index c34a9519ace..5e9ba4f640f 100644 --- a/doc/administration/merge_request_diffs.md +++ b/doc/administration/merge_request_diffs.md @@ -145,3 +145,47 @@ The connection settings match those provided by [Fog](https://github.com/fog), a ``` 1. Save the file and [restart GitLab](restart_gitlab.md#installations-from-source) for the changes to take effect. + +### Alternative in-database storage + +Enabling external diffs may reduce the performance of merge requests, as they +must be retrieved in a separate operation to other data. A compromise may be +reached by only storing outdated diffs externally, while keeping current diffs +in the database. + +To enable this feature, perform the following steps: + +**In Omnibus installations:** + +1. Edit `/etc/gitlab/gitlab.rb` and add the following line: + + ```ruby + gitlab_rails['external_diffs_when'] = 'outdated' + ``` + +1. Save the file and [reconfigure GitLab](restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. + +**In installations from source:** + +1. Edit `/home/git/gitlab/config/gitlab.yml` and add or amend the following + lines: + + ```yaml + external_diffs: + enabled: true + when: outdated + ``` + +1. Save the file and [restart GitLab](restart_gitlab.md#installations-from-source) for the changes to take effect. + +With this feature enabled, diffs will initially stored in the database, rather +than externally. They will be moved to external storage once any of these +conditions become true: + +- A newer version of the merge request diff exists +- The merge request was merged more than seven days ago +- The merge request was closed more than seven day ago + +These rules strike a balance between space and performance by only storing +frequently-accessed diffs in the database. Diffs that are less likely to be +accessed are moved to external storage instead. diff --git a/doc/administration/monitoring/index.md b/doc/administration/monitoring/index.md index d18dddf09c0..fa0459b24ff 100644 --- a/doc/administration/monitoring/index.md +++ b/doc/administration/monitoring/index.md @@ -7,4 +7,4 @@ Explore our features to monitor your GitLab instance: - [GitHub imports](github_imports.md): Monitor the health and progress of GitLab's GitHub importer with various Prometheus metrics. - [Monitoring uptime](../../user/admin_area/monitoring/health_check.md): Check the server status using the health check endpoint. - [IP whitelists](ip_whitelist.md): Configure GitLab for monitoring endpoints that provide health check information when probed. -- [nginx_status](https://docs.gitlab.com/omnibus/settings/nginx.html#enabling-disabling-nginx_status): Monitor your Nginx server status +- [nginx_status](https://docs.gitlab.com/omnibus/settings/nginx.html#enablingdisabling-nginx_status): Monitor your Nginx server status diff --git a/doc/administration/monitoring/performance/grafana_configuration.md b/doc/administration/monitoring/performance/grafana_configuration.md index ab43ec2cc4f..187fb2f73a1 100644 --- a/doc/administration/monitoring/performance/grafana_configuration.md +++ b/doc/administration/monitoring/performance/grafana_configuration.md @@ -3,7 +3,7 @@ [Grafana](http://grafana.org/) is a tool that allows you to visualize time series metrics through graphs and dashboards. It supports several backend data stores, including InfluxDB. GitLab writes performance data to InfluxDB -and Grafana will allow you to query InfluxDB to display useful graphs. +and Grafana will allow you to query to display useful graphs. For the easiest installation and configuration, install Grafana on the same server as InfluxDB. For larger installations, you may want to split out these @@ -11,11 +11,13 @@ services. ## Installation -Grafana supplies package repositories (Yum/Apt) for easy installation. +[GitLab Omnibus can help you install Grafana (recommended)](https://docs.gitlab.com/omnibus/settings/grafana.html) +or Grafana supplies package repositories (Yum/Apt) for easy installation. See [Grafana installation documentation](http://docs.grafana.org/installation/) for detailed steps. -> **Note**: Before starting Grafana for the first time, set the admin user +NOTE: **Note:** +Before starting Grafana for the first time, set the admin user and password in `/etc/grafana/grafana.ini`. Otherwise, the default password will be `admin`. diff --git a/doc/administration/monitoring/performance/img/request_profiling_token.png b/doc/administration/monitoring/performance/img/request_profiling_token.png Binary files differindex 9f3dd7f08ca..ee819fcb437 100644 --- a/doc/administration/monitoring/performance/img/request_profiling_token.png +++ b/doc/administration/monitoring/performance/img/request_profiling_token.png diff --git a/doc/administration/monitoring/performance/introduction.md b/doc/administration/monitoring/performance/introduction.md index 37a5388d2fc..7ace0ec5a93 100644 --- a/doc/administration/monitoring/performance/introduction.md +++ b/doc/administration/monitoring/performance/introduction.md @@ -1 +1,5 @@ +--- +redirect_to: 'index.md' +--- + This document was moved to [another location](index.md). diff --git a/doc/administration/monitoring/performance/prometheus.md b/doc/administration/monitoring/performance/prometheus.md index d73ef5d1789..2c5bab46dd9 100644 --- a/doc/administration/monitoring/performance/prometheus.md +++ b/doc/administration/monitoring/performance/prometheus.md @@ -1 +1,5 @@ +--- +redirect_to: '../prometheus/index.md' +--- + This document was moved to [monitoring/prometheus](../prometheus/index.md). diff --git a/doc/administration/monitoring/prometheus/gitlab_metrics.md b/doc/administration/monitoring/prometheus/gitlab_metrics.md index 3bfcc9a289e..3dcd1593099 100644 --- a/doc/administration/monitoring/prometheus/gitlab_metrics.md +++ b/doc/administration/monitoring/prometheus/gitlab_metrics.md @@ -43,10 +43,52 @@ The following metrics are available: | redis_ping_latency_seconds | Gauge | 9.4 | Round trip time of the redis ping | | user_session_logins_total | Counter | 9.4 | Counter of how many users have logged in | | upload_file_does_not_exist | Counter | 10.7 in EE, 11.5 in CE | Number of times an upload record could not find its file | -| failed_login_captcha_total | Gauge | 11.0 | Counter of failed CAPTCHA attempts during login | -| successful_login_captcha_total | Gauge | 11.0 | Counter of successful CAPTCHA attempts during login | -| unicorn_active_connections | Gauge | 11.0 | The number of active Unicorn connections (workers) | -| unicorn_queued_connections | Gauge | 11.0 | The number of queued Unicorn connections | +| failed_login_captcha_total | Gauge | 11.0 | Counter of failed CAPTCHA attempts during login | +| successful_login_captcha_total | Gauge | 11.0 | Counter of successful CAPTCHA attempts during login | +| unicorn_active_connections | Gauge | 11.0 | The number of active Unicorn connections (workers) | +| unicorn_queued_connections | Gauge | 11.0 | The number of queued Unicorn connections | +| unicorn_workers | Gauge | 12.0 | The number of Unicorn workers | + +## Sidekiq Metrics available for Geo **[PREMIUM]** + +Sidekiq jobs may also gather metrics, and these metrics can be accessed if the Sidekiq exporter is enabled (e.g. via +the `monitoring.sidekiq_exporter` configuration option in `gitlab.yml`. + +| Metric | Type | Since | Description | Labels | +|:-------------------------------------------- |:------- |:----- |:----------- |:------ | +| geo_db_replication_lag_seconds | Gauge | 10.2 | Database replication lag (seconds) | url +| geo_repositories | Gauge | 10.2 | Total number of repositories available on primary | url +| geo_repositories_synced | Gauge | 10.2 | Number of repositories synced on secondary | url +| geo_repositories_failed | Gauge | 10.2 | Number of repositories failed to sync on secondary | url +| geo_lfs_objects | Gauge | 10.2 | Total number of LFS objects available on primary | url +| geo_lfs_objects_synced | Gauge | 10.2 | Number of LFS objects synced on secondary | url +| geo_lfs_objects_failed | Gauge | 10.2 | Number of LFS objects failed to sync on secondary | url +| geo_attachments | Gauge | 10.2 | Total number of file attachments available on primary | url +| geo_attachments_synced | Gauge | 10.2 | Number of attachments synced on secondary | url +| geo_attachments_failed | Gauge | 10.2 | Number of attachments failed to sync on secondary | url +| geo_last_event_id | Gauge | 10.2 | Database ID of the latest event log entry on the primary | url +| geo_last_event_timestamp | Gauge | 10.2 | UNIX timestamp of the latest event log entry on the primary | url +| geo_cursor_last_event_id | Gauge | 10.2 | Last database ID of the event log processed by the secondary | url +| geo_cursor_last_event_timestamp | Gauge | 10.2 | Last UNIX timestamp of the event log processed by the secondary | url +| geo_status_failed_total | Counter | 10.2 | Number of times retrieving the status from the Geo Node failed | url +| geo_last_successful_status_check_timestamp | Gauge | 10.2 | Last timestamp when the status was successfully updated | url +| geo_lfs_objects_synced_missing_on_primary | Gauge | 10.7 | Number of LFS objects marked as synced due to the file missing on the primary | url +| geo_job_artifacts_synced_missing_on_primary | Gauge | 10.7 | Number of job artifacts marked as synced due to the file missing on the primary | url +| geo_attachments_synced_missing_on_primary | Gauge | 10.7 | Number of attachments marked as synced due to the file missing on the primary | url +| geo_repositories_checksummed_count | Gauge | 10.7 | Number of repositories checksummed on primary | url +| geo_repositories_checksum_failed_count | Gauge | 10.7 | Number of repositories failed to calculate the checksum on primary | url +| geo_wikis_checksummed_count | Gauge | 10.7 | Number of wikis checksummed on primary | url +| geo_wikis_checksum_failed_count | Gauge | 10.7 | Number of wikis failed to calculate the checksum on primary | url +| geo_repositories_verified_count | Gauge | 10.7 | Number of repositories verified on secondary | url +| geo_repositories_verification_failed_count | Gauge | 10.7 | Number of repositories failed to verify on secondary | url +| geo_repositories_checksum_mismatch_count | Gauge | 10.7 | Number of repositories that checksum mismatch on secondary | url +| geo_wikis_verified_count | Gauge | 10.7 | Number of wikis verified on secondary | url +| geo_wikis_verification_failed_count | Gauge | 10.7 | Number of wikis failed to verify on secondary | url +| geo_wikis_checksum_mismatch_count | Gauge | 10.7 | Number of wikis that checksum mismatch on secondary | url +| geo_repositories_checked_count | Gauge | 11.1 | Number of repositories that have been checked via `git fsck` | url +| geo_repositories_checked_failed_count | Gauge | 11.1 | Number of repositories that have a failure from `git fsck` | url +| geo_repositories_retrying_verification_count | Gauge | 11.2 | Number of repositories verification failures that Geo is actively trying to correct on secondary | url +| geo_wikis_retrying_verification_count | Gauge | 11.2 | Number of wikis verification failures that Geo is actively trying to correct on secondary | url ### Ruby metrics @@ -59,9 +101,31 @@ Some basic Ruby runtime metrics are available: | ruby_file_descriptors | Gauge | 11.1 | File descriptors per process | | ruby_memory_bytes | Gauge | 11.1 | Memory usage by process | | ruby_sampler_duration_seconds_total | Counter | 11.1 | Time spent collecting stats | +| ruby_process_cpu_seconds_total | Gauge | 12.0 | Total amount of CPU time per process | +| ruby_process_max_fds | Gauge | 12.0 | Maximum number of open file descriptors per process | +| ruby_process_resident_memory_bytes | Gauge | 12.0 | Memory usage by process, measured in bytes | +| ruby_process_start_time_seconds | Gauge | 12.0 | The elapsed time between system boot and the process started, measured in seconds | [GC.stat]: https://ruby-doc.org/core-2.3.0/GC.html#method-c-stat +## Puma Metrics **[EXPERIMENTAL]** + +When Puma is used instead of Unicorn, following metrics are available: + +| Metric | Type | Since | Description | +|:-------------------------------------------- |:------- |:----- |:----------- | +| puma_workers | Gauge | 12.0 | Total number of workers | +| puma_running_workers | Gauge | 12.0 | Number of booted workers | +| puma_stale_workers | Gauge | 12.0 | Number of old workers | +| puma_phase | Gauge | 12.0 | Phase number (increased during phased restarts) | +| puma_running | Gauge | 12.0 | Number of running threads | +| puma_queued_connections | Gauge | 12.0 | Number of connections in that worker's "todo" set waiting for a worker thread | +| puma_active_connections | Gauge | 12.0 | Number of threads processing a request | +| puma_pool_capacity | Gauge | 12.0 | Number of requests the worker is capable of taking right now | +| puma_max_threads | Gauge | 12.0 | Maximum number of worker threads | +| puma_idle_threads | Gauge | 12.0 | Number of spawned threads which are not processing a request | +| rack_state_total | Gauge | 12.0 | Number of requests in a given rack state | + ## Metrics shared directory GitLab's Prometheus client requires a directory to store metrics data shared between multi-process services. diff --git a/doc/administration/monitoring/prometheus/index.md b/doc/administration/monitoring/prometheus/index.md index 8f65cd6418c..ce65d77274b 100644 --- a/doc/administration/monitoring/prometheus/index.md +++ b/doc/administration/monitoring/prometheus/index.md @@ -1,6 +1,7 @@ # Monitoring GitLab with Prometheus > **Notes:** +> > - Prometheus and the various exporters listed in this page are bundled in the > Omnibus GitLab package. Check each exporter's documentation for the timeline > they got added. For installations from source you will have to install them @@ -17,7 +18,7 @@ access to high quality time-series monitoring of GitLab services. ## Overview Prometheus works by periodically connecting to data sources and collecting their -performance metrics via the [various exporters](#prometheus-exporters). To view +performance metrics via the [various exporters](#bundled-software-metrics). To view and work with the monitoring data, you can either [connect directly to Prometheus](#viewing-performance-metrics) or utilize a dashboard tool like [Grafana]. @@ -95,13 +96,15 @@ To use an external Prometheus server: 1. Set each bundled service's [exporter](#bundled-software-metrics) to listen on a network address, for example: - ```ruby + ```ruby gitlab_monitor['listen_address'] = '0.0.0.0' + sidekiq['listen_address'] = '0.0.0.0' gitlab_monitor['listen_port'] = '9168' - gitaly['prometheus_listen_addr'] = "0.0.0.0:9236" node_exporter['listen_address'] = '0.0.0.0:9100' redis_exporter['listen_address'] = '0.0.0.0:9121' postgres_exporter['listen_address'] = '0.0.0.0:9187' + gitaly['prometheus_listen_addr'] = "0.0.0.0:9236" + gitlab_workhorse['prometheus_listen_addr'] = "0.0.0.0:9229" ``` 1. Install and set up a dedicated Prometheus instance, if necessary, using the [official installation instructions](https://prometheus.io/docs/prometheus/latest/installation/). @@ -111,6 +114,18 @@ To use an external Prometheus server: gitlab_rails['monitoring_whitelist'] = ['127.0.0.0/8', '192.168.0.1'] ``` +1. To scrape nginx metrics, you'll also need to configure nginx to allow the Prometheus server + IP. For example: + + ```ruby + nginx['status']['options'] = { + "server_tokens" => "off", + "access_log" => "off", + "allow" => "192.168.0.1", + "deny" => "all", + } + ``` + 1. [Reconfigure GitLab][reconfigure] to apply the changes 1. Edit the Prometheus server's configuration file. 1. Add each node's exporters to the Prometheus server's @@ -202,6 +217,12 @@ The Postgres exporter allows you to measure various PostgreSQL metrics. [➔ Read more about the Postgres exporter.](postgres_exporter.md) +### PgBouncer exporter + +The PgBouncer exporter allows you to measure various PgBouncer metrics. + +[➔ Read more about the PgBouncer exporter.](pgbouncer_exporter.md) + ### GitLab monitor exporter The GitLab monitor exporter allows you to measure various GitLab metrics, pulled from Redis and the database. diff --git a/doc/administration/monitoring/prometheus/pgbouncer_exporter.md b/doc/administration/monitoring/prometheus/pgbouncer_exporter.md new file mode 100644 index 00000000000..d76834fdbea --- /dev/null +++ b/doc/administration/monitoring/prometheus/pgbouncer_exporter.md @@ -0,0 +1,34 @@ +# PgBouncer exporter + +>**Note:** +Available since [Omnibus GitLab 11.0][2493]. For installations from source +you'll have to install and configure it yourself. + +The [PgBouncer exporter] allows you to measure various PgBouncer metrics. + +To enable the PgBouncer exporter: + +1. [Enable Prometheus](index.md#configuring-prometheus) +1. Edit `/etc/gitlab/gitlab.rb` +1. Add or find and uncomment the following line, making sure it's set to `true`: + + ```ruby + pgbouncer_exporter['enable'] = true + ``` + +1. Save the file and [reconfigure GitLab][reconfigure] for the changes to + take effect. + +Prometheus will now automatically begin collecting performance data from +the PgBouncer exporter exposed under `localhost:9188`. + +The PgBouncer exporter will also be enabled by default if the [pgbouncer_role][postgres roles] +is enabled. + +[← Back to the main Prometheus page](index.md) + +[2493]: https://gitlab.com/gitlab-org/omnibus-gitlab/merge_requests/2493 +[PgBouncer exporter]: https://github.com/stanhu/pgbouncer_exporter +[postgres roles]: https://docs.gitlab.com/omnibus/roles/#postgres-roles +[prometheus]: https://prometheus.io +[reconfigure]: ../../restart_gitlab.md#omnibus-gitlab-reconfigure diff --git a/doc/administration/npm_registry.md b/doc/administration/npm_registry.md new file mode 100644 index 00000000000..d8551f64ece --- /dev/null +++ b/doc/administration/npm_registry.md @@ -0,0 +1,5 @@ +--- +redirect_to: 'packages.md' +--- + +This document was moved to [another location](packages.md). diff --git a/doc/administration/operations.md b/doc/administration/operations.md index 4797d2a3206..9cd78105bbb 100644 --- a/doc/administration/operations.md +++ b/doc/administration/operations.md @@ -1 +1,5 @@ +--- +redirect_to: 'operations/index.md' +--- + This document was moved to [another location](operations/index.md). diff --git a/doc/administration/operations/cleaning_up_redis_sessions.md b/doc/administration/operations/cleaning_up_redis_sessions.md index b45ca99fd80..20c19445404 100644 --- a/doc/administration/operations/cleaning_up_redis_sessions.md +++ b/doc/administration/operations/cleaning_up_redis_sessions.md @@ -27,7 +27,7 @@ rcli() { # This example works for Omnibus installations of GitLab 7.3 or newer. For an # installation from source you will have to change the socket path and the # path to redis-cli. - sudo /opt/gitlab/embedded/bin/redis-cli -s /var/opt/gitlab/redis/redis.socket "$@" + sudo /opt/gitlab/embedded/bin/redis-cli -s /var/opt/gitlab/redis/redis.shared_state.socket "$@" } # test the new shell function; the response should be PONG diff --git a/doc/administration/operations/extra_sidekiq_processes.md b/doc/administration/operations/extra_sidekiq_processes.md new file mode 100644 index 00000000000..286b99aceb5 --- /dev/null +++ b/doc/administration/operations/extra_sidekiq_processes.md @@ -0,0 +1,218 @@ +# Extra Sidekiq processes **[STARTER ONLY]** + +GitLab Enterprise Edition allows one to start an extra set of Sidekiq processes +besides the default one. These processes can be used to consume a dedicated set +of queues. This can be used to ensure certain queues always have dedicated +workers, no matter the number of jobs that need to be processed. + +## Starting extra processes via Omnibus GitLab + +To enable `sidekiq-cluster`, you must apply the `sidekiq_cluster['enable'] = true` +setting `/etc/gitlab/gitlab.rb`: + +```ruby +sidekiq_cluster['enable'] = true +``` + +You will then specify how many additional processes to create via `sidekiq-cluster` +as well as which queues for them to handle. This is done via the +`sidekiq_cluster['queue_groups']` setting. This is an array whose items contain +which queues to process. Each item in the array will equate to one additional +sidekiq process. + +As an example, to make additional sidekiq processes that process the +`elastic_indexer` and `mailers` queues, you would apply the following: + +```ruby +sidekiq_cluster['queue_groups'] = [ + "elastic_indexer", + "mailers" +] +``` + +To have an additional sidekiq process handle multiple queues, you simply put a +comma after the first queue name and then put the next queue name: + +```ruby +sidekiq_cluster['queue_groups'] = [ + "elastic_indexer,elastic_commit_indexer", + "mailers" +] +``` + +Keep in mind, all changes must be followed by reconfiguring your GitLab +application via `sudo gitlab-ctl reconfigure`. + +### Monitoring + +Once the Sidekiq processes are added, you can visit the "Background Jobs" +section under the admin area in GitLab (`/admin/background_jobs`). + + + +### All queues with exceptions + +To have the additional sidekiq processes work on every queue EXCEPT the ones +you list: + +1. Edit `/etc/gitlab/gitlab.rb` and add: + + ```ruby + sidekiq_cluster['negate'] = true + ``` + +1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. + + +### Limiting concurrency + +1. Edit `/etc/gitlab/gitlab.rb` and add: + + ```ruby + sidekiq_cluster['concurrency'] = 25 + ``` + +1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. + +Keep in mind, this normally would not exceed the number of CPU cores available. + +### Modifying the check interval + +To modify the check interval for the additional Sidekiq processes: + +1. Edit `/etc/gitlab/gitlab.rb` and add: + + ```ruby + sidekiq_cluster['interval'] = 5 + ``` + +1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. + +This tells the additional processes how often to check for enqueued jobs. + +## Starting extra processes via command line + +Starting extra Sidekiq processes can be done using the command +`/opt/gitlab/embedded/service/gitlab-rails/ee/bin/sidekiq-cluster`. This command +takes arguments using the following syntax: + +```bash +/opt/gitlab/embedded/service/gitlab-rails/ee/bin/sidekiq-cluster [QUEUE,QUEUE,...] [QUEUE, ...] +``` + +Each separate argument denotes a group of queues that have to be processed by a +Sidekiq process. Multiple queues can be processed by the same process by +separating them with a comma instead of a space. + +Instead of a queue, a queue namespace can also be provided, to have the process +automatically listen on all queues in that namespace without needing to +explicitly list all the queue names. For more information about queue namespaces, +see the relevant section in the +[Sidekiq style guide](../../development/sidekiq_style_guide.md#queue-namespaces). + +For example, say you want to start 2 extra processes: one to process the +"process_commit" queue, and one to process the "post_receive" queue. This can be +done as follows: + +```bash +/opt/gitlab/embedded/service/gitlab-rails/ee/bin/sidekiq-cluster process_commit post_receive +``` + +If you instead want to start one process processing both queues you'd use the +following syntax: + +```bash +/opt/gitlab/embedded/service/gitlab-rails/ee/bin/sidekiq-cluster process_commit,post_receive +``` + +If you want to have one Sidekiq process process the "process_commit" and +"post_receive" queues, and one process to process the "gitlab_shell" queue, +you'd use the following: + +```bash +/opt/gitlab/embedded/service/gitlab-rails/ee/bin/sidekiq-cluster process_commit,post_receive gitlab_shell +``` + +### Monitoring + +The `sidekiq-cluster` command will not terminate once it has started the desired +amount of Sidekiq processes. Instead, the process will continue running and +forward any signals to the child processes. This makes it easy to stop all +Sidekiq processes as you simply send a signal to the `sidekiq-cluster` process, +instead of having to send it to the individual processes. + +If the `sidekiq-cluster` process crashes or receives a `SIGKILL`, the child +processes will terminate themselves after a few seconds. This ensures you don't +end up with zombie Sidekiq processes. + +All of this makes monitoring the processes fairly easy. Simply hook up +`sidekiq-cluster` to your supervisor of choice (e.g. runit) and you're good to +go. + +If a child process died the `sidekiq-cluster` command will signal all remaining +process to terminate, then terminate itself. This removes the need for +`sidekiq-cluster` to re-implement complex process monitoring/restarting code. +Instead you should make sure your supervisor restarts the `sidekiq-cluster` +process whenever necessary. + +### PID files + +The `sidekiq-cluster` command can store its PID in a file. By default no PID +file is written, but this can be changed by passing the `--pidfile` option to +`sidekiq-cluster`. For example: + +```bash +/opt/gitlab/embedded/service/gitlab-rails/ee/bin/sidekiq-cluster --pidfile /var/run/gitlab/sidekiq_cluster.pid process_commit +``` + +Keep in mind that the PID file will contain the PID of the `sidekiq-cluster` +command and not the PID(s) of the started Sidekiq processes. + +### Environment + +The Rails environment can be set by passing the `--environment` flag to the +`sidekiq-cluster` command, or by setting `RAILS_ENV` to a non-empty value. The +default value is "development". + +### All queues with exceptions + +You're able to run all queues in `sidekiq_queues.yml` file on a single or +multiple processes with exceptions using the `--negate` flag. + +For example, say you want to run a single process for all queues, +except "process_commit" and "post_receive". You can do so by executing: + +```bash +sidekiq-cluster process_commit,post_receive --negate +``` + +For multiple processes of all queues (except "process_commit" and "post_receive"): + +```bash +sidekiq-cluster process_commit,post_receive process_commit,post_receive --negate +``` + +### Limiting concurrency + +By default, `sidekiq-cluster` will spin up extra Sidekiq processes that use +one thread per queue up to a maximum of 50. If you wish to change the cap, use +the `-m N` option. For example, this would cap the maximum number of threads to 1: + +```bash +/opt/gitlab/embedded/service/gitlab-rails/ee/bin/sidekiq-cluster process_commit,post_receive -m 1 +``` + +For each queue group, the concurrency factor will be set to min(number of +queues, N). Setting the value to 0 will disable the limit. + +Note that each thread requires a Redis connection, so adding threads may +increase Redis latency and potentially cause client timeouts. See the [Sidekiq +documentation about Redis](https://github.com/mperham/sidekiq/wiki/Using-Redis) +for more details. + +## Number of threads + +Each process started using `sidekiq-cluster` (whether it be via command line or +via the gitlab.rb file) starts with a number of threads that equals the number +of queues, plus one spare thread. For example, a process that handles the +"process_commit" and "post_receive" queues will use 3 threads in total. diff --git a/doc/administration/operations/fast_ssh_key_lookup.md b/doc/administration/operations/fast_ssh_key_lookup.md index c293df3fc57..3631ea0822f 100644 --- a/doc/administration/operations/fast_ssh_key_lookup.md +++ b/doc/administration/operations/fast_ssh_key_lookup.md @@ -30,6 +30,19 @@ instructions will break installations using older versions of OpenSSH, such as those included with CentOS 6 as of September 2017. If you want to use this feature for CentOS 6, follow [the instructions on how to build and install a custom OpenSSH package](#compiling-a-custom-version-of-openssh-for-centos-6) before continuing. +## Fast lookup is required for Geo **[PREMIUM]** + +By default, GitLab manages an `authorized_keys` file, which contains all the +public SSH keys for users allowed to access GitLab. However, to maintain a +single source of truth, [Geo](../geo/replication/index.md) needs to be configured to perform SSH fingerprint +lookups via database lookup. + +As part of [setting up Geo](../geo/replication/index.md#setup-instructions), +you will be required to follow the steps outlined below for both the primary and +secondary nodes, but note that the `Write to "authorized keys" file` checkbox +only needs to be unchecked on the primary node since it will be reflected +automatically on the secondary if database replication is working. + ## Setting up fast lookup via GitLab Shell GitLab Shell provides a way to authorize SSH users via a fast, indexed lookup @@ -58,6 +71,9 @@ sudo service sshd reload Confirm that SSH is working by removing your user's SSH key in the UI, adding a new one, and attempting to pull a repo. +> **Note:** For Omnibus Docker, `AuthorizedKeysCommand` is setup by default in +GitLab 11.11 and later. + > **Warning:** Do not disable writes until SSH is confirmed to be working perfectly, because the file will quickly become out-of-date. @@ -66,7 +82,7 @@ file will still be scanned. So git SSH performance will still be slow for many users as long as a large file exists. You can disable any more writes to the `authorized_keys` file by unchecking -`Write to "authorized_keys" file` in the Application Settings of your GitLab +`Write to "authorized_keys" file` in the **Admin Area > Settings > Network > Performance optimization** of your GitLab installation.  diff --git a/doc/administration/operations/img/sidekiq-cluster.png b/doc/administration/operations/img/sidekiq-cluster.png Binary files differnew file mode 100644 index 00000000000..4eb1849010e --- /dev/null +++ b/doc/administration/operations/img/sidekiq-cluster.png diff --git a/doc/administration/operations/index.md b/doc/administration/operations/index.md index 32f36d68c50..df795a48169 100644 --- a/doc/administration/operations/index.md +++ b/doc/administration/operations/index.md @@ -11,6 +11,7 @@ Keep your GitLab instance up and running smoothly. by GitLab to another file system or another server. - [Sidekiq MemoryKiller](sidekiq_memory_killer.md): Configure Sidekiq MemoryKiller to restart Sidekiq. +- [Extra Sidekiq operations](extra_sidekiq_processes.md): Configure an extra set of Sidekiq processes to ensure certain queues always have dedicated workers, no matter the amount of jobs that need to be processed. **[STARTER ONLY]** - [Unicorn](unicorn.md): Understand Unicorn and unicorn-worker-killer. - Speed up SSH operations by [Authorizing SSH users via a fast, indexed lookup to the GitLab database](fast_ssh_key_lookup.md), and/or diff --git a/doc/administration/operations/sidekiq_memory_killer.md b/doc/administration/operations/sidekiq_memory_killer.md index cbffd883774..8eac42f2fe2 100644 --- a/doc/administration/operations/sidekiq_memory_killer.md +++ b/doc/administration/operations/sidekiq_memory_killer.md @@ -17,6 +17,11 @@ With the default settings, the MemoryKiller will cause a Sidekiq restart no more often than once every 15 minutes, with the restart causing about one minute of delay for incoming background jobs. +Some background jobs rely on long-running external processes. To ensure these +are cleanly terminated when Sidekiq is restarted, each Sidekiq process should be +run as a process group leader (e.g., using `chpst -P`). If using Omnibus or the +`bin/background_jobs` script with `runit` installed, this is handled for you. + ## Configuring the MemoryKiller The MemoryKiller is controlled using environment variables. diff --git a/doc/administration/operations/speed_up_ssh.md b/doc/administration/operations/speed_up_ssh.md index 89265b3018b..6dc83c42f53 100644 --- a/doc/administration/operations/speed_up_ssh.md +++ b/doc/administration/operations/speed_up_ssh.md @@ -1 +1,5 @@ +--- +redirect_to: 'fast_ssh_key_lookup.md' +--- + This document was moved to [another location](fast_ssh_key_lookup.md). diff --git a/doc/administration/packages.md b/doc/administration/packages.md new file mode 100644 index 00000000000..0d5f784b71e --- /dev/null +++ b/doc/administration/packages.md @@ -0,0 +1,174 @@ +# GitLab Packages administration **[PREMIUM ONLY]** + +GitLab Packages allows organizations to utilize GitLab as a private repository +for a variety of common package managers. Users are able to build and publish +packages, which can be easily consumed as a dependency in downstream projects. + +The Packages feature allows GitLab to act as a repository for the following: + +| Software repository | Description | Available in GitLab version | +| ------------------- | ----------- | --------------------------- | +| [Maven Repository](../user/project/packages/maven_repository.md) | The GitLab Maven Repository enables every project in GitLab to have its own space to store [Maven](https://maven.apache.org/) packages. | 11.3+ | +| [NPM Registry](../user/project/packages/npm_registry.md) | The GitLab NPM Registry enables every project in GitLab to have its own space to store [NPM](https://www.npmjs.com/) packages. | 11.7+ | + +Don't you see your package management system supported yet? +Please consider contributing +to GitLab. This [development documentation](../development/packages.md) will guide you through the process. + +## Enabling the Packages feature + +NOTE: **Note:** +After the Packages feature is enabled, the repositories are available +for all new projects by default. To enable it for existing projects, users will +have to explicitly do so in the project's settings. + +To enable the Packages feature: + +**Omnibus GitLab installations** + +1. Edit `/etc/gitlab/gitlab.rb` and add the following line: + + ```ruby + gitlab_rails['packages_enabled'] = true + ``` + +1. Save the file and [reconfigure GitLab][] for the changes to take effect. + +**Installations from source** + +1. After the installation is complete, you will have to configure the `packages` + section in `config/gitlab.yml`. Set to `true` to enable it: + + ```yaml + packages: + enabled: true + ``` +1. [Restart GitLab] for the changes to take effect. + +## Changing the storage path + +By default, the packages are stored locally, but you can change the default +local location or even use object storage. + +### Changing the local storage path + +The packages for Omnibus GitLab installations are stored under +`/var/opt/gitlab/gitlab-rails/shared/packages/` and for source +installations under `shared/packages/` (relative to the git homedir). +To change the local storage path: + +**Omnibus GitLab installations** + +1. Edit `/etc/gitlab/gitlab.rb` and add the following line: + + ```ruby + gitlab_rails['packages_storage_path'] = "/mnt/packages" + ``` + +1. Save the file and [reconfigure GitLab][] for the changes to take effect. + +**Installations from source** + +1. Edit the `packages` section in `config/gitlab.yml`: + + ```yaml + packages: + enabled: true + storage_path: shared/packages + ``` +1. [Restart GitLab] for the changes to take effect. + +### Using object storage + +Instead of relying on the local storage, you can use an object storage to +upload packages: + +**Omnibus GitLab installations** + +1. Edit `/etc/gitlab/gitlab.rb` and add the following lines (uncomment where + necessary): + + ```ruby + gitlab_rails['packages_enabled'] = true + gitlab_rails['packages_storage_path'] = "/var/opt/gitlab/gitlab-rails/shared/packages" + gitlab_rails['packages_object_store_enabled'] = true + gitlab_rails['packages_object_store_remote_directory'] = "packages" # The bucket name. + gitlab_rails['packages_object_store_direct_upload'] = false # Use Object Storage directly for uploads instead of background uploads if enabled (Default: false). + gitlab_rails['packages_object_store_background_upload'] = true # Temporary option to limit automatic upload (Default: true). + gitlab_rails['packages_object_store_proxy_download'] = false # Passthrough all downloads via GitLab instead of using Redirects to Object Storage. + gitlab_rails['packages_object_store_connection'] = { + ## + ## If the provider is AWS S3, uncomment the following + ## + #'provider' => 'AWS', + #'region' => 'eu-west-1', + #'aws_access_key_id' => 'AWS_ACCESS_KEY_ID', + #'aws_secret_access_key' => 'AWS_SECRET_ACCESS_KEY', + ## + ## If the provider is other than AWS (an S3-compatible one), uncomment the following + ## + #'host' => 's3.amazonaws.com', + #'aws_signature_version' => 4 # For creation of signed URLs. Set to 2 if provider does not support v4. + #'endpoint' => 'https://s3.amazonaws.com' # Useful for S3-compliant services such as DigitalOcean Spaces. + #'path_style' => false # If true, use 'host/bucket_name/object' instead of 'bucket_name.host/object'. + } + ``` + +1. Save the file and [reconfigure GitLab][] for the changes to take effect. + +**Installations from source** + +1. Edit the `packages` section in `config/gitlab.yml` (uncomment where necessary): + + ```yaml + packages: + enabled: true + ## + ## The location where build packages are stored (default: shared/packages). + ## + #storage_path: shared/packages + object_store: + enabled: false + remote_directory: packages # The bucket name. + #direct_upload: false # Use Object Storage directly for uploads instead of background uploads if enabled (Default: false). + #background_upload: true # Temporary option to limit automatic upload (Default: true). + #proxy_download: false # Passthrough all downloads via GitLab instead of using Redirects to Object Storage. + connection: + ## + ## If the provider is AWS S3, uncomment the following + ## + #provider: AWS + #region: us-east-1 + #aws_access_key_id: AWS_ACCESS_KEY_ID + #aws_secret_access_key: AWS_SECRET_ACCESS_KEY + ## + ## If the provider is other than AWS (an S3-compatible one), uncomment the following + ## + #host: 's3.amazonaws.com' # default: s3.amazonaws.com. + #aws_signature_version: 4 # For creation of signed URLs. Set to 2 if provider does not support v4. + #endpoint: 'https://s3.amazonaws.com' # Useful for S3-compliant services such as DigitalOcean Spaces. + #path_style: false # If true, use 'host/bucket_name/object' instead of 'bucket_name.host/object'. + ``` + +1. [Restart GitLab] for the changes to take effect. + +### Migrating local packages to object storage + +After [configuring the object storage](#using-object-storage), you may use the +following task to migrate existing packages from the local storage to the remote one. +The processing will be done in a background worker and requires **no downtime**. + +For Omnibus GitLab: + +```sh +sudo gitlab-rake "gitlab:packages:migrate" +``` + +For installations from source: + +```bash +RAILS_ENV=production sudo -u git -H bundle exec rake gitlab:packages:migrate +``` + +[reconfigure gitlab]: restart_gitlab.md#omnibus-gitlab-reconfigure "How to reconfigure Omnibus GitLab" +[restart gitlab]: restart_gitlab.md#omnibus-gitlab-reconfigure "How to reconfigure Omnibus GitLab" diff --git a/doc/administration/pages/index.md b/doc/administration/pages/index.md index 279ad018aed..3a7ca517d56 100644 --- a/doc/administration/pages/index.md +++ b/doc/administration/pages/index.md @@ -5,6 +5,7 @@ description: 'Learn how to administer GitLab Pages.' # GitLab Pages administration > **Notes:** +> > - [Introduced][ee-80] in GitLab EE 8.3. > - Custom CNAMEs with TLS support were [introduced][ee-173] in GitLab EE 8.5. > - GitLab Pages [were ported][ce-14605] to Community Edition in GitLab 8.17. @@ -124,7 +125,7 @@ The Pages daemon doesn't listen to the outside world. pages_external_url 'http://example.io' ``` -1. [Reconfigure GitLab][reconfigure] +1. [Reconfigure GitLab][reconfigure]. Watch the [video tutorial][video-admin] for this configuration. @@ -156,7 +157,26 @@ outside world. where `pages-nginx.crt` and `pages-nginx.key` are the SSL cert and key, respectively. -1. [Reconfigure GitLab][reconfigure] +1. [Reconfigure GitLab][reconfigure]. + +### Additional configuration for Docker container + +The GitLab Pages daemon will not have permissions to bind mounts when it runs +in a Docker container. To overcome this issue you'll need to change the chroot +behavior: + +1. Edit `/etc/gitlab/gitlab.rb`. +1. Set the `inplace_chroot` to `true` for GitLab Pages: + + ```shell + gitlab_pages['inplace_chroot'] = true + ``` + +1. [Reconfigure GitLab][reconfigure]. + +NOTE: **Note:** +`inplace_chroot` option might not work with the other features, such as [Pages Access Control](#access-control). +The [GitLab Pages README](https://gitlab.com/gitlab-org/gitlab-pages#caveats) has more information about caveats and workarounds. ## Advanced configuration @@ -194,7 +214,7 @@ world. Custom domains are supported, but no TLS. `192.0.2.2` and `2001::2` are the secondary IPs the GitLab Pages daemon listens on. If you don't have IPv6, you can omit the IPv6 address. -1. [Reconfigure GitLab][reconfigure] +1. [Reconfigure GitLab][reconfigure]. ### Custom domains with TLS support @@ -228,7 +248,7 @@ world. Custom domains and TLS are supported. `192.0.2.2` and `2001::2` are the secondary IPs where the GitLab Pages daemon listens on. If you don't have IPv6, you can omit the IPv6 address. -1. [Reconfigure GitLab][reconfigure] +1. [Reconfigure GitLab][reconfigure]. ### Custom domain verification @@ -271,6 +291,20 @@ Pages access control is disabled by default. To enable it: 1. [Reconfigure GitLab][reconfigure]. 1. Users can now configure it in their [projects' settings](../../user/project/pages/introduction.md#gitlab-pages-access-control-core-only). +### Running behind a proxy + +Like the rest of GitLab, Pages can be used in those environments where external +internet connectivity is gated by a proxy. In order to use a proxy for GitLab +pages: + +1. Configure in `/etc/gitlab/gitlab.rb`: + + ```ruby + gitlab_pages['http_proxy'] = 'http://example:8080' + ``` + +1. [Reconfigure Gitlab][reconfigure] for the changes to take effect. + ## Activate verbose logging for daemon Verbose logging was [introduced](https://gitlab.com/gitlab-org/omnibus-gitlab/merge_requests/2533) in @@ -286,7 +320,7 @@ Follow the steps below to configure verbose logging of GitLab Pages daemon. gitlab_pages['log_verbose'] = true ``` -1. [Reconfigure GitLab][reconfigure] +1. [Reconfigure GitLab][reconfigure]. ## Change storage path @@ -301,7 +335,7 @@ are stored. gitlab_rails['pages_path'] = "/mnt/storage/pages" ``` -1. [Reconfigure GitLab][reconfigure] +1. [Reconfigure GitLab][reconfigure]. ## Configure listener for reverse proxy requests @@ -324,7 +358,7 @@ Omnibus GitLab 11.1. gitlab_pages['listen_proxy'] = "localhost:10080" ``` -1. [Reconfigure GitLab][reconfigure] +1. [Reconfigure GitLab][reconfigure]. ## Set maximum pages size @@ -356,6 +390,7 @@ Follow the steps below to configure GitLab Pages in a separate server. gitaly['enable'] = false alertmanager['enable'] = false node_exporter['enable'] = false + gitlab_rails['auto_migrate'] = false ``` 1. Run `sudo gitlab-ctl reconfigure`. 1. On `app1` apply the following changes to `/etc/gitlab/gitlab.rb`: diff --git a/doc/administration/pages/source.md b/doc/administration/pages/source.md index 60800d445b8..2100f7cd707 100644 --- a/doc/administration/pages/source.md +++ b/doc/administration/pages/source.md @@ -137,7 +137,7 @@ The Pages daemon doesn't listen to the outside world. ``` gitlab_pages_enabled=true - gitlab_pages_options="-pages-domain example.io -pages-root $app_root/shared/pages -listen-proxy 127.0.0.1:8090 + gitlab_pages_options="-pages-domain example.io -pages-root $app_root/shared/pages -listen-proxy 127.0.0.1:8090" ``` 1. Copy the `gitlab-pages` Nginx configuration file: diff --git a/doc/administration/pseudonymizer.md b/doc/administration/pseudonymizer.md new file mode 100644 index 00000000000..036e1d3fe61 --- /dev/null +++ b/doc/administration/pseudonymizer.md @@ -0,0 +1,103 @@ +# Pseudonymizer **[ULTIMATE]** + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/5532) in [GitLab Ultimate][ee] 11.1. + +As GitLab's database hosts sensitive information, using it unfiltered for analytics +implies high security requirements. To help alleviate this constraint, the Pseudonymizer +service is used to export GitLab's data in a pseudonymized way. + +CAUTION: **Warning:** +This process is not impervious. If the source data is available, it's possible for +a user to correlate data to the pseudonymized version. + +The Pseudonymizer currently uses `HMAC(SHA256)` to mutate fields that shouldn't +be textually exported. This ensures that: + +- the end-user of the data source cannot infer/revert the pseudonymized fields +- the referential integrity is maintained + +## Configuration + +To configure the pseudonymizer, you need to: + +- Provide a manifest file that describes which fields should be included or + pseudonymized ([example `manifest.yml` file](https://gitlab.com/gitlab-org/gitlab-ee/tree/master/config/pseudonymizer.yml)). + A default manifest is provided with the GitLab installation. Using a relative file path will be resolved from the Rails root. + Alternatively, you can use an absolute file path. +- Use an object storage and specify the connection parameters in the `pseudonymizer.upload.connection` configuration option. + +**For Omnibus installations:** + +1. Edit `/etc/gitlab/gitlab.rb` and add the following lines by replacing with + the values you want: + + ```ruby + gitlab_rails['pseudonymizer_manifest'] = 'config/pseudonymizer.yml' + gitlab_rails['pseudonymizer_upload_remote_directory'] = 'gitlab-elt' # bucket name + gitlab_rails['pseudonymizer_upload_connection'] = { + 'provider' => 'AWS', + 'region' => 'eu-central-1', + 'aws_access_key_id' => 'AWS_ACCESS_KEY_ID', + 'aws_secret_access_key' => 'AWS_SECRET_ACCESS_KEY' + } + ``` + + NOTE: **Note:** + If you are using AWS IAM profiles, be sure to omit the AWS access key and secret access key/value pairs. + + ```ruby + gitlab_rails['pseudonymizer_upload_connection'] = { + 'provider' => 'AWS', + 'region' => 'eu-central-1', + 'use_iam_profile' => true + } + ``` + +1. Save the file and [reconfigure GitLab](restart_gitlab.md#omnibus-gitlab-reconfigure) + for the changes to take effect. + +--- + +**For installations from source:** + +1. Edit `/home/git/gitlab/config/gitlab.yml` and add or amend the following + lines: + + ```yaml + pseudonymizer: + manifest: config/pseudonymizer.yml + upload: + remote_directory: 'gitlab-elt' # bucket name + connection: + provider: AWS + aws_access_key_id: AWS_ACCESS_KEY_ID + aws_secret_access_key: AWS_SECRET_ACCESS_KEY + region: eu-central-1 + ``` + +1. Save the file and [restart GitLab](restart_gitlab.md#installations-from-source) + for the changes to take effect. + +## Usage + +You can optionally run the pseudonymizer using the following environment variables: + +- `PSEUDONYMIZER_OUTPUT_DIR` - where to store the output CSV files (defaults to `/tmp`) +- `PSEUDONYMIZER_BATCH` - the batch size when querying the DB (defaults to `100000`) + +```bash +## Omnibus +sudo gitlab-rake gitlab:db:pseudonymizer + +## Source +sudo -u git -H bundle exec rake gitlab:db:pseudonymizer RAILS_ENV=production +``` + +This will produce some CSV files that might be very large, so make sure the +`PSEUDONYMIZER_OUTPUT_DIR` has sufficient space. As a rule of thumb, at least +10% of the database size is recommended. + +After the pseudonymizer has run, the output CSV files should be uploaded to the +configured object storage and deleted from the local disk. + +[ee]: https://about.gitlab.com/pricing/ diff --git a/doc/administration/raketasks/geo.md b/doc/administration/raketasks/geo.md new file mode 100644 index 00000000000..9f3b31442f3 --- /dev/null +++ b/doc/administration/raketasks/geo.md @@ -0,0 +1,57 @@ +# Geo Rake Tasks **[PREMIUM ONLY]** + +## Git housekeeping + +There are few tasks you can run to schedule a git housekeeping to start at the +next repository sync in a **Secondary node**: + +### Incremental Repack + +This is equivalent of running `git repack -d` on a _bare_ repository. + +**Omnibus Installation** + +``` +sudo gitlab-rake geo:git:housekeeping:incremental_repack +``` + +**Source Installation** + +```bash +sudo -u git -H bundle exec rake geo:git:housekeeping:incremental_repack RAILS_ENV=production +``` + +### Full Repack + +This is equivalent of running `git repack -d -A --pack-kept-objects` on a +_bare_ repository which will optionally, write a reachability bitmap index +when this is enabled in GitLab. + +**Omnibus Installation** + +``` +sudo gitlab-rake geo:git:housekeeping:full_repack +``` + +**Source Installation** + +```bash +sudo -u git -H bundle exec rake geo:git:housekeeping:full_repack RAILS_ENV=production +``` + +### GC + +This is equivalent of running `git gc` on a _bare_ repository, optionally writing +a reachability bitmap index when this is enabled in GitLab. + +**Omnibus Installation** + +``` +sudo gitlab-rake geo:git:housekeeping:gc +``` + +**Source Installation** + +```bash +sudo -u git -H bundle exec rake geo:git:housekeeping:gc RAILS_ENV=production +``` diff --git a/doc/administration/raketasks/maintenance.md b/doc/administration/raketasks/maintenance.md index b295b7d5dc4..0b4c1ae15d6 100644 --- a/doc/administration/raketasks/maintenance.md +++ b/doc/administration/raketasks/maintenance.md @@ -205,25 +205,6 @@ cd /home/git/gitlab sudo -u git -H bundle exec rake gitlab:track_deployment RAILS_ENV=production ``` -## Create or repair repository hooks symlink - -If the GitLab shell hooks directory location changes or another circumstance -leads to the hooks symlink becoming missing or invalid, run this Rake task -to create or repair the symlinks. - -**Omnibus Installation** - -``` -sudo gitlab-rake gitlab:shell:create_hooks -``` - -**Source Installation** - -``` -cd /home/git/gitlab -sudo -u git -H bundle exec rake gitlab:shell:create_hooks RAILS_ENV=production -``` - ## Check TCP connectivity to a remote site Sometimes you need to know if your GitLab installation can connect to a TCP diff --git a/doc/administration/raketasks/project_import_export.md b/doc/administration/raketasks/project_import_export.md index f43bba0a7a7..6ca23aabdec 100644 --- a/doc/administration/raketasks/project_import_export.md +++ b/doc/administration/raketasks/project_import_export.md @@ -32,5 +32,4 @@ bundle exec rake gitlab:import_export:data RAILS_ENV=production ``` [ce-3050]: https://gitlab.com/gitlab-org/gitlab-ce/issues/3050 -[feature-flags]: https://docs.gitlab.com/ee/api/features.html [tmp]: ../../development/shared_files.md diff --git a/doc/administration/raketasks/storage.md b/doc/administration/raketasks/storage.md index 7ad38abe4f5..42a1a1c2e60 100644 --- a/doc/administration/raketasks/storage.md +++ b/doc/administration/raketasks/storage.md @@ -34,17 +34,62 @@ export ID_FROM=20 export ID_TO=50 ``` -You can monitor the progress in the _Admin > Monitoring > Background jobs_ screen. -There is a specific Queue you can watch to see how long it will take to finish: **project_migrate_hashed_storage** +You can monitor the progress in the **Admin Area > Monitoring > Background Jobs** page. +There is a specific Queue you can watch to see how long it will take to finish: +`hashed_storage:hashed_storage_project_migrate` After it reaches zero, you can confirm every project has been migrated by running the commands bellow. If you find it necessary, you can run this migration script again to schedule missing projects. -Any error or warning will be logged in the sidekiq's log file. +Any error or warning will be logged in Sidekiq's log file. + +NOTE: **Note:** +If Geo is enabled, each project that is successfully migrated generates an event to replicate the changes on any **secondary** nodes. You only need the `gitlab:storage:migrate_to_hashed` rake task to migrate your repositories, but we have additional commands below that helps you inspect projects and attachments in both legacy and hashed storage. +## Rollback from Hashed storage to Legacy storage + +If you need to rollback the storage migration for any reason, you can follow the steps described here. + +NOTE: **Note:** Hashed Storage will be required in future version of GitLab. + +To prevent new projects from being created in the Hashed storage, +you need to undo the [enable hashed storage][storage-migration] changes. + +This task will schedule all your existing projects and associated attachments to be rolled back to the +Legacy storage type. + +For Omnibus installations, run the following: + +```bash +sudo gitlab-rake gitlab:storage:rollback_to_legacy +``` + +For source installations, run the following: + +```bash +sudo -u git -H bundle exec rake gitlab:storage:rollback_to_legacy RAILS_ENV=production +``` + +Both commands accept a range as environment variable: + +```bash +# to rollback any migrated project from ID 20 to 50. +export ID_FROM=20 +export ID_TO=50 +``` + +You can monitor the progress in the **Admin Area > Monitoring > Background Jobs** page. +On the **Queues** tab, you can watch the `hashed_storage:hashed_storage_project_rollback` queue to see how long the process will take to finish. + + +After it reaches zero, you can confirm every project has been rolled back by running the commands bellow. +If some projects weren't rolled back, you can run this rollback script again to schedule further rollbacks. + +Any error or warning will be logged in Sidekiq's log file. + ## List projects on Legacy storage To have a simple summary of projects using **Legacy** storage: diff --git a/doc/administration/raketasks/uploads/migrate.md b/doc/administration/raketasks/uploads/migrate.md index b5c40478ea5..fd8ea8d3162 100644 --- a/doc/administration/raketasks/uploads/migrate.md +++ b/doc/administration/raketasks/uploads/migrate.md @@ -2,7 +2,7 @@ ## Migrate to Object Storage -After [configuring the object storage](../../uploads.md#using-object-storage) for GitLab's uploads, you may use this task to migrate existing uploads from the local storage to the remote storage. +After [configuring the object storage](../../uploads.md#using-object-storage-core-only) for GitLab's uploads, you may use this task to migrate existing uploads from the local storage to the remote storage. >**Note:** All of the processing will be done in a background worker and requires **no downtime**. diff --git a/doc/administration/raketasks/uploads/sanitize.md b/doc/administration/raketasks/uploads/sanitize.md new file mode 100644 index 00000000000..54a423b9571 --- /dev/null +++ b/doc/administration/raketasks/uploads/sanitize.md @@ -0,0 +1,62 @@ +# Uploads Sanitize tasks + +## Requirements + +You need `exiftool` installed on your system. If you installed GitLab: + +- Using the Omnibus package, you're all set. +- From source, make sure `exiftool` is installed: + + ```sh + # Debian/Ubuntu + sudo apt-get install libimage-exiftool-perl + + # RHEL/CentOS + sudo yum install perl-Image-ExifTool + ``` + +## Remove EXIF data from existing uploads + +Since 11.9 EXIF data are automatically stripped from JPG or TIFF image uploads. +Because EXIF data may contain sensitive information (e.g. GPS location), you +can remove EXIF data also from existing images which were uploaded before +with the following command: + +```bash +sudo RAILS_ENV=production -u git -H bundle exec rake gitlab:uploads:sanitize:remove_exif +``` + +This command by default runs in dry mode and it doesn't remove EXIF data. It can be used for +checking if (and how many) images should be sanitized. + +The rake task accepts following parameters. + +Parameter | Type | Description +--------- | ---- | ----------- +`start_id` | integer | Only uploads with equal or greater ID will be processed +`stop_id` | integer | Only uploads with equal or smaller ID will be processed +`dry_run` | boolean | Do not remove EXIF data, only check if EXIF data are present or not, default: true +`sleep_time` | float | Pause for number of seconds after processing each image, default: 0.3 seconds + +If you have too many uploads, you can speed up sanitization by setting +`sleep_time` to a lower value or by running multiple rake tasks in parallel, +each with a separate range of upload IDs (by setting `start_id` and `stop_id`). + +To run the command without dry mode and remove EXIF data from all uploads, you can use: + +```bash +sudo RAILS_ENV=production -u git -H bundle exec rake gitlab:uploads:sanitize:remove_exif[,,false,] 2>&1 | tee exif.log +``` + +To run the command without dry mode on uploads with ID between 100 and 5000 and pause for 0.1 second, you can use: + +```bash +sudo RAILS_ENV=production -u git -H bundle exec rake gitlab:uploads:sanitize:remove_exif[100,5000,false,0.1] 2>&1 | tee exif.log +``` + +Because the output of commands will be probably long, the output is written also into exif.log file. + +If sanitization fails for an upload, an error message should be in the output of the rake task (typical reasons may +be that the file is missing in the storage or it's not a valid image). Please +[report](https://gitlab.com/gitlab-org/gitlab-ce/issues/new) any issues at `gitlab.com` and use +prefix 'EXIF' in issue title with the error output and (if possible) the image. diff --git a/doc/administration/repository_checks.md b/doc/administration/repository_checks.md index 8b725e50f58..7cf8f20a9dc 100644 --- a/doc/administration/repository_checks.md +++ b/doc/administration/repository_checks.md @@ -31,7 +31,7 @@ panel. If the repository check fails for some repository you should look up the error in `repocheck.log`: -- in the [admin panel](logs.md#repocheck-log) +- in the [admin panel](logs.md#repochecklog) - or on disk, see: - `/var/log/gitlab/gitlab-rails` for Omnibus installations - `/home/git/gitlab/log` for installations from source diff --git a/doc/administration/repository_storage_paths.md b/doc/administration/repository_storage_paths.md index 7f25423171f..4aafc06cfdc 100644 --- a/doc/administration/repository_storage_paths.md +++ b/doc/administration/repository_storage_paths.md @@ -62,6 +62,8 @@ files and add the full paths of the alternative repository storage paths. In the example below, we add two more mountpoints that are named `nfs` and `cephfs` respectively. +NOTE: **Note:** This example uses NFS and CephFS. We do not recommend using EFS for storage as it may impact GitLab's performance. See the [relevant documentation](high_availability/nfs.md#avoid-using-awss-elastic-file-system-efs) for more details. + **For installations from source** 1. Edit `gitlab.yml` and add the storage paths: diff --git a/doc/administration/repository_storage_types.md b/doc/administration/repository_storage_types.md index 4934aaf39f7..4db3cbb9958 100644 --- a/doc/administration/repository_storage_types.md +++ b/doc/administration/repository_storage_types.md @@ -2,6 +2,24 @@ > [Introduced][ce-28283] in GitLab 10.0. +Two different storage layouts can be used +to store the repositories on disk and their characteristics. + +GitLab can be configured to use one or multiple repository shard locations +that can be: + +- Mounted to the local disk +- Exposed as an NFS shared volume +- Acessed via [gitaly] on its own machine. + +In GitLab, this is configured in `/etc/gitlab/gitlab.rb` by the `git_data_dirs({})` +configuration hash. The storage layouts discussed here will apply to any shard +defined in it. + +The `default` repository shard that is available in any installations +that haven't customized it, points to the local folder: `/var/opt/gitlab/git-data`. +Anything discussed below is expected to be part of that folder. + ## Legacy Storage Legacy Storage is the storage behavior prior to version 10.0. For historical @@ -29,15 +47,15 @@ Any change in the URL will need to be reflected on disk (when groups / users or projects are renamed). This can add a lot of load in big installations, especially if using any type of network based filesystem. -For GitLab Geo in particular: Geo does work with legacy storage, but in some +CAUTION: **Caution:** +For Geo in particular: Geo does work with legacy storage, but in some edge cases due to race conditions it can lead to errors when a project is renamed multiple times in short succession, or a project is deleted and recreated under the same name very quickly. We expect these race events to be rare, and we have not observed a race condition side-effect happening yet. - This pattern also exists in other objects stored in GitLab, like issue Attachments, GitLab Pages artifacts, Docker Containers for the integrated -Registry, etc. +Registry, etc. Hashed storage is a requirement for Geo. ## Hashed Storage @@ -66,34 +84,12 @@ by another folder with the next 2 characters. They are both stored in a special "@hashed/#{hash[0..1]}/#{hash[2..3]}/#{hash}.wiki.git" ``` -### How to migrate to Hashed Storage - -In GitLab, go to **Admin > Settings**, find the **Repository Storage** section -and select "_Use hashed storage paths for newly created and renamed projects_". - -To migrate your existing projects to the new storage type, check the specific -[rake tasks]. - -[ce-28283]: https://gitlab.com/gitlab-org/gitlab-ce/issues/28283 -[rake tasks]: raketasks/storage.md#migrate-existing-projects-to-hashed-storage -[storage-paths]: repository_storage_types.md - -#### Rollback - -There is no automated rollback implemented. Below are the steps required to rollback -from each storage migration. +### Hashed object pools -The rollback has to be performed in the reverse order. To get into "Legacy" state, -you need to rollback Attachments first, then Project. - -Also note that if Geo is enabled, after the migration was triggered, an event is generated -to replicate the operation on any Secondary node. That means the on disk changes will also -need to be performed on these nodes as well. Database changes will propagate without issues. - -You must make sure the migration event was already processed or otherwise it may migrate -the files back to Hashed state again. - -#### Hashed object pools +CAUTION: **Beta:** +Hashed objects pools are considered beta, and are not ready for production use. +Follow [gitaly#1548](https://gitlab.com/gitlab-org/gitaly/issues/1548) for +updates. For deduplication of public forks and their parent repository, objects are pooled in an object pool. These object pools are a third repository where shared objects @@ -110,36 +106,60 @@ enabled for individual projects by executing be on hashed storage, should not be a fork itself, and hashed storage should be enabled for all new projects. -##### Attachments +### How to migrate to Hashed Storage -To rollback single Attachment migration, rename `aa/bb/abcdef1234567890...` folder back to `namespace/project`. +To start a migration, enable Hashed Storage for new projects: + +1. Go to **Admin > Settings > Repository** and expand the **Repository Storage** section. +2. Select the **Use hashed storage paths for newly created and renamed projects** checkbox. -Both folder names can be generated by the `FileUploader.absolute_base_dir(project)`, you -just need to switch the version from the `project` back to the previous one. +Check if the change breaks any existing integration you may have that +either runs on the same machine as your repositories are located, or may login to that machine +to access data (for example, a remote backup solution). -```ruby -project.storage_version -# => 2 +To schedule a complete rollout, see the +[rake task documentation for storage migration][rake/migrate-to-hashed] for instructions. -FileUploader.absolute_base_dir(project) -# => "/opt/gitlab/embedded/service/gitlab-rails/public/uploads/@hashed/d4/73/d4735e3a265e16eee03f59718b9b5d03019c07d8b6c51f90da3a666eec13ab35" +If you do have any existing integration, you may want to do a small rollout first, +to validate. You can do so by specifying a range with the operation. -project.storage_version = 1 +This is an example of how to limit the rollout to Project IDs 50 to 100, running in +an Omnibus Gitlab installation: -FileUploader.absolute_base_dir(project) -# => "/opt/gitlab/embedded/service/gitlab-rails/public/uploads/gitlab/gitlab-shell-renamed" +```bash +sudo gitlab-rake gitlab:storage:migrate_to_hashed ID_FROM=50 ID_TO=100 ``` -##### Project +Check the [documentation][rake/migrate-to-hashed] for additional information and instructions for +source-based installation. + +#### Rollback + +Similar to the migration, to disable Hashed Storage for new +projects: -To rollback single Project migration, move `@hashed/aa/bb/aabbcdef1234567890abcdef.git` and `@hashed/aa/bb/aabbcdef1234567890abcdef.wiki.git` -back to `namespace/project.git` and `namespace/project.wiki.git` respectively and switch the version from the `project` back to `null`. +1. Go to **Admin > Settings > Repository** and expand the **Repository Storage** section. +2. Uncheck the **Use hashed storage paths for newly created and renamed projects** checkbox. + +To schedule a complete rollback, see the +[rake task documentation for storage rollback](raketasks/storage.md#rollback-from-hashed-storage-to-legacy-storage) for instructions. + +The rollback task also supports specifying a range of Project IDs. Here is an example +of limiting the rollout to Project IDs 50 to 100, in an Omnibus Gitlab installation: + +```bash +sudo gitlab-rake gitlab:storage:rollback_to_legacy ID_FROM=50 ID_TO=100 +``` + +If you have a Geo setup, please note that the rollback will not be reflected automatically +on the **secondary** node. You may need to wait for a backfill operation to kick-in and remove +the remaining repositories from the special `@hashed/` folder manually. ### Hashed Storage coverage We are incrementally moving every storable object in GitLab to the Hashed Storage pattern. You can check the current coverage status below (and also see -the [issue](https://gitlab.com/gitlab-com/infrastructure/issues/2821)). +the [issue][ce-2821]). Note that things stored in an S3 compatible endpoint will not have the downsides mentioned earlier, if they are not prefixed with `#{namespace}/#{project_name}`, @@ -156,6 +176,7 @@ which is true for CI Cache and LFS Objects. | CI Artifacts | No | No | Yes | 9.4 / 10.6 | | CI Cache | No | No | Yes | - | | LFS Objects | Yes | Similar | Yes | 10.0 / 10.7 | +| Repository pools| No | Yes | - | 11.6 | #### Implementation Details @@ -180,3 +201,9 @@ LFS Objects implements a similar storage pattern using 2 chars, 2 level folders, ``` They are also S3 compatible since **10.0** (GitLab Premium), and available in GitLab Core since **10.7**. + +[ce-2821]: https://gitlab.com/gitlab-com/infrastructure/issues/2821 +[ce-28283]: https://gitlab.com/gitlab-org/gitlab-ce/issues/28283 +[rake/migrate-to-hashed]: raketasks/storage.md#migrate-existing-projects-to-hashed-storage +[storage-paths]: repository_storage_types.md +[gitaly]: gitaly/index.md diff --git a/doc/administration/repository_storages.md b/doc/administration/repository_storages.md index cf6de15743f..af7a385e5a0 100644 --- a/doc/administration/repository_storages.md +++ b/doc/administration/repository_storages.md @@ -1 +1,5 @@ +--- +redirect_to: 'repository_storage_paths.md' +--- + This document was moved to [another location](repository_storage_paths.md). diff --git a/doc/administration/troubleshooting/migration.md b/doc/administration/troubleshooting/migration.md new file mode 100644 index 00000000000..4d2d268b9df --- /dev/null +++ b/doc/administration/troubleshooting/migration.md @@ -0,0 +1,82 @@ +# Migrations problems + +## Legacy upload migration + +> Introduced in GitLab 12.0. + + The migration takes all attachments uploaded by legacy `AttachmentUploader` and + migrate them to the path that current uploaders expect. + +Although it should not usually happen there could possibly be some attachments belonging to +LegacyDiffNotes. These attachments can't be seen before running the migration by users and +they should not be present in your instance. + +However, if you have some of them, you will need to handle them manually. +You can find the ids of failed notes in logs as "MigrateLegacyUploads: LegacyDiffNote" + +1. Run a Rails console: + + ```sh + sudo gitlab-rails console production + ``` + + or for source installs: + + ```sh + bundle exec rails console production + ``` + + 1. Check the failed upload and find the note (you can see their ids in the logs) + + ```ruby + upload = Upload.find(upload_id) + note = Note.find(note_id) + ``` + + + 1. Check the path - it should contain `system/note/attachment` + + ```ruby + upload.absolut_path + ``` + + 1. Check the path in the uploader - it should differ from the upload path and should contain `system/legacy_diff_note` + + ```ruby + uploader = upload.build_uploader + uploader.file + ``` + + 1. First, you need to move the file to the path that is expected from the uploader + + ```ruby + old_path = upload.absolute_path + new_path = upload.absolute_path.sub('-/system/note/attachment', '-/system/legacy_diff_note') + new_dir = File.dirname(new_path) + FileUtils.mkdir_p(new_dir) + + FileUtils.mv(old_path, new_path) + ``` + + 1. You then need to move the file to the `FileUploader` and create a new `Upload` object + + ```ruby + file_uploader = UploadService.new(note.project, File.read(new_path)).execute + ``` + + 1. And update the legacy note to contain the file. + + ```ruby + new_text = "#{note.note} \n #{file_uploader.markdown_link}" + note.update!( + note: new_text + ) + ``` + + 1. And finally, you can remove the old upload + + ```ruby + upload.destroy + ``` + +If you have any problems feel free to contact [GitLab Support](https://about.gitlab.com/support/). diff --git a/doc/administration/uploads.md b/doc/administration/uploads.md index 9dfe085425f..708b59a273b 100644 --- a/doc/administration/uploads.md +++ b/doc/administration/uploads.md @@ -18,7 +18,7 @@ below. >**Notes:** For historical reasons, uploads are stored into a base directory, which by default is `uploads/-/system`. It is strongly discouraged to change this configuration option on an existing GitLab installation. -_The uploads are stored by default in `/var/opt/gitlab/gitlab-rails/uploads/-/system`._ +_The uploads are stored by default in `/var/opt/gitlab/gitlab-rails/uploads`._ 1. To change the storage path for example to `/mnt/storage/uploads`, edit `/etc/gitlab/gitlab.rb` and add the following line: @@ -53,7 +53,7 @@ _The uploads are stored by default in > **Notes:** > > - [Introduced][ee-3867] in [GitLab Premium][eep] 10.5. -> - [Introduced][ce17358] in [GitLab Core][ce] 10.7. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/17358) in [GitLab Core][ce] 10.7. > - Since version 11.1, we support direct_upload to S3. If you don't want to use the local disk where GitLab is installed to store the @@ -152,4 +152,3 @@ _The uploads are stored by default in [eep]: https://about.gitlab.com/gitlab-ee/ "GitLab Premium" [ce]: https://about.gitlab.com/gitlab-ce/ "GitLab Community Edition" [ee-3867]: https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/3867 -[ce-17358]: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/17358 diff --git a/doc/administration/user_settings.md b/doc/administration/user_settings.md new file mode 100644 index 00000000000..f9654655949 --- /dev/null +++ b/doc/administration/user_settings.md @@ -0,0 +1,35 @@ +# Modifying global user settings + +GitLab administrators can modify user settings for the entire GitLab instance. + +## Disallow users creating top-level groups + +By default, new users can create top-level groups. To disable this, modify the appropriate configuration file. + +For Omnibus installations, add the following to `/etc/gitlab/gitlab.rb`: + +```ruby +gitlab_rails['gitlab_default_can_create_group'] = false +``` + +For source installations, uncomment the following line in `config/gitlab.yml`: + +```yaml +# default_can_create_group: false # default: true +``` + +## Disallow users changing usernames + +By default, new users can change their usernames. To disable this, modify the appropriate configuration file. + +For Omnibus installations, add the following to `/etc/gitlab/gitlab.rb`: + +```ruby +gitlab_rails['gitlab_username_changing_enabled'] = false +``` + +For source installations, uncomment the following line in `config/gitlab.yml`: + +```yaml +# username_changing_enabled: false # default: true - User can change her username/namespace +``` diff --git a/doc/analytics/README.md b/doc/analytics/README.md new file mode 100644 index 00000000000..bfb15f6c4f3 --- /dev/null +++ b/doc/analytics/README.md @@ -0,0 +1,5 @@ +--- +redirect_to: '../user/group/index.md#user-contribution-analysis-starter' +--- + +This document was moved to [another location](../user/group/index.md#user-contribution-analysis-starter) diff --git a/doc/analytics/contribution_analytics.md b/doc/analytics/contribution_analytics.md new file mode 100644 index 00000000000..e36f55071a4 --- /dev/null +++ b/doc/analytics/contribution_analytics.md @@ -0,0 +1,5 @@ +--- +redirect_to: '../user/group/contribution_analytics/index.md' +--- + +This document was moved to [another location](../user/group/contribution_analytics/index.md). diff --git a/doc/api/README.md b/doc/api/README.md index 48d9ad8f30d..3a1064b787e 100644 --- a/doc/api/README.md +++ b/doc/api/README.md @@ -21,72 +21,83 @@ See also: The following API resources are available in the project context: -| Resource | Available endpoints | -|:------------------------------------------------------|:--------------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| [Access requests](access_requests.md) | `/projects/:id/access_requests` (also available for groups) | -| [Award emoji](award_emoji.md) | `/projects/:id/issues/.../award_emoji`, `/projects/:id/merge_requests/.../award_emoji`, `/projects/:id/snippets/.../award_emoji` | -| [Branches](branches.md) | `/projects/:id/repository/branches/`, `/projects/:id/repository/merged_branches` | -| [Commits](commits.md) | `/projects/:id/repository/commits`, `/projects/:id/statuses` | -| [Container Registry](container_registry.md) | `/projects/:id/registry/repositories` | -| [Custom attributes](custom_attributes.md) | `/projects/:id/custom_attributes` (also available for groups and users) | -| [Deploy keys](deploy_keys.md) | `/projects/:id/deploy_keys` (also available standalone) | -| [Deployments](deployments.md) | `/projects/:id/deployments` | -| [Discussions](discussions.md) (threaded comments) | `/projects/:id/issues/.../discussions`, `/projects/:id/snippets/.../discussions`, `/projects/:id/merge_requests/.../discussions`, `/projects/:id/commits/.../discussions` | -| [Environments](environments.md) | `/projects/:id/environments` | -| [Events](events.md) | `/projects/:id/events` (also available for users and standalone) | -| [Issues](issues.md) | `/projects/:id/issues` (also available for groups and standalone) | -| [Issue boards](boards.md) | `/projects/:id/boards` | -| [Jobs](jobs.md) | `/projects/:id/jobs`, `/projects/:id/pipelines/.../jobs` | -| [Labels](labels.md) | `/projects/:id/labels` | -| [Members](members.md) | `/projects/:id/members` (also available for groups) | -| [Merge requests](merge_requests.md) | `/projects/:id/merge_requests` (also available for groups and standalone) | -| [Notes](notes.md) (comments) | `/projects/:id/issues/.../notes`, `/projects/:id/snippets/.../notes`, `/projects/:id/merge_requests/.../notes` | -| [Notification settings](notification_settings.md) | `/projects/:id/notification_settings` (also available for groups and standalone) | -| [Pages domains](pages_domains.md) | `/projects/:id/pages` (also available standalone) | -| [Pipelines](pipelines.md) | `/projects/:id/pipelines` | -| [Pipeline schedules](pipeline_schedules.md) | `/projects/:id/pipeline_schedules` | -| [Pipeline triggers](pipeline_triggers.md) | `/projects/:id/triggers` | -| [Projects](projects.md) including setting Webhooks | `/projects`, `/projects/:id/hooks` (also available for users) | -| [Project badges](project_badges.md) | `/projects/:id/badges` | -| [Project clusters](project_clusters.md) | `/projects/:id/clusters` | -| [Project-level variables](project_level_variables.md) | `/projects/:id/variables` | -| [Project import/export](project_import_export.md) | `/projects/:id/export`, `/projects/import`, `/projects/:id/import` | -| [Project milestones](milestones.md) | `/projects/:id/milestones` | -| [Project snippets](project_snippets.md) | `/projects/:id/snippets` | -| [Project templates](project_templates.md) | `/projects/:id/templates` | -| [Protected branches](protected_branches.md) | `/projects/:id/protected_branches` | -| [Protected tags](protected_tags.md) | `/projects/:id/protected_tags` | -| [Releases](releases/index.md) | `/projects/:id/releases` | -| [Release links](releases/links.md) | `/projects/:id/releases/.../assets/links` | -| [Repositories](repositories.md) | `/projects/:id/repository` | -| [Repository files](repository_files.md) | `/projects/:id/repository/files` | -| [Repository submodules](repository_submodules.md) | `/projects/:id/repository/submodules` | -| [Resource label events](resource_label_events.md) | `/projects/:id/issues/.../resource_label_events`, `/projects/:id/merge_requests/.../resource_label_events` | -| [Runners](runners.md) | `/projects/:id/runners` (also available standalone) | -| [Search](search.md) | `/projects/:id/search` (also available for groups and standalone) | -| [Services](services.md) | `/projects/:id/services` | -| [Tags](tags.md) | `/projects/:id/repository/tags` | -| [Wikis](wikis.md) | `/projects/:id/wikis` | +| Resource | Available endpoints | +|:--------------------------------------------------------------------|:------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| [Access requests](access_requests.md) | `/projects/:id/access_requests` (also available for groups) | +| [Award emoji](award_emoji.md) | `/projects/:id/issues/.../award_emoji`, `/projects/:id/merge_requests/.../award_emoji`, `/projects/:id/snippets/.../award_emoji` | +| [Branches](branches.md) | `/projects/:id/repository/branches/`, `/projects/:id/repository/merged_branches` | +| [Commits](commits.md) | `/projects/:id/repository/commits`, `/projects/:id/statuses` | +| [Container Registry](container_registry.md) | `/projects/:id/registry/repositories` | +| [Custom attributes](custom_attributes.md) | `/projects/:id/custom_attributes` (also available for groups and users) | +| [Deploy keys](deploy_keys.md) | `/projects/:id/deploy_keys` (also available standalone) | +| [Deployments](deployments.md) | `/projects/:id/deployments` | +| [Discussions](discussions.md) (threaded comments) | `/projects/:id/issues/.../discussions`, `/projects/:id/snippets/.../discussions`, `/projects/:id/merge_requests/.../discussions`, `/projects/:id/commits/.../discussions` (also available for groups) | +| [Environments](environments.md) | `/projects/:id/environments` | +| [Events](events.md) | `/projects/:id/events` (also available for users and standalone) | +| [Issues](issues.md) | `/projects/:id/issues` (also available for groups and standalone) | +| [Issue boards](boards.md) | `/projects/:id/boards` | +| [Issue links](issue_links.md) **[STARTER]** | `/projects/:id/issues/.../links` | +| [Jobs](jobs.md) | `/projects/:id/jobs`, `/projects/:id/pipelines/.../jobs` | +| [Labels](labels.md) | `/projects/:id/labels` | +| [Managed licenses](managed_licenses.md) **[ULTIMATE]** | `/projects/:id/managed_licenses` | +| [Members](members.md) | `/projects/:id/members` (also available for groups) | +| [Merge request approvals](merge_request_approvals.md) **[STARTER]** | `/projects/:id/approvals`, `/projects/:id/merge_requests/.../approvals` | +| [Merge requests](merge_requests.md) | `/projects/:id/merge_requests` (also available for groups and standalone) | +| [Notes](notes.md) (comments) | `/projects/:id/issues/.../notes`, `/projects/:id/snippets/.../notes`, `/projects/:id/merge_requests/.../notes` (also available for groups) | +| [Notification settings](notification_settings.md) | `/projects/:id/notification_settings` (also available for groups and standalone) | +| [Packages](packages.md) **[PREMIUM]** | `/projects/:id/packages` | +| [Pages domains](pages_domains.md) | `/projects/:id/pages` (also available standalone) | +| [Pipelines](pipelines.md) | `/projects/:id/pipelines` | +| [Pipeline schedules](pipeline_schedules.md) | `/projects/:id/pipeline_schedules` | +| [Pipeline triggers](pipeline_triggers.md) | `/projects/:id/triggers` | +| [Projects](projects.md) including setting Webhooks | `/projects`, `/projects/:id/hooks` (also available for users) | +| [Project badges](project_badges.md) | `/projects/:id/badges` | +| [Project clusters](project_clusters.md) | `/projects/:id/clusters` | +| [Project-level variables](project_level_variables.md) | `/projects/:id/variables` | +| [Project import/export](project_import_export.md) | `/projects/:id/export`, `/projects/import`, `/projects/:id/import` | +| [Project milestones](milestones.md) | `/projects/:id/milestones` | +| [Project snippets](project_snippets.md) | `/projects/:id/snippets` | +| [Project templates](project_templates.md) | `/projects/:id/templates` | +| [Protected branches](protected_branches.md) | `/projects/:id/protected_branches` | +| [Protected tags](protected_tags.md) | `/projects/:id/protected_tags` | +| [Releases](releases/index.md) | `/projects/:id/releases` | +| [Release links](releases/links.md) | `/projects/:id/releases/.../assets/links` | +| [Repositories](repositories.md) | `/projects/:id/repository` | +| [Repository files](repository_files.md) | `/projects/:id/repository/files` | +| [Repository submodules](repository_submodules.md) | `/projects/:id/repository/submodules` | +| [Resource label events](resource_label_events.md) | `/projects/:id/issues/.../resource_label_events`, `/projects/:id/merge_requests/.../resource_label_events` (also available for groups) | +| [Runners](runners.md) | `/projects/:id/runners` (also available standalone) | +| [Search](search.md) | `/projects/:id/search` (also available for groups and standalone) | +| [Services](services.md) | `/projects/:id/services` | +| [Tags](tags.md) | `/projects/:id/repository/tags` | +| [Vulnerabilities](vulnerabilities.md) **[ULTIMATE]** | `/projects/:id/vulnerabilities` (also available for groups) | +| [Wikis](wikis.md) | `/projects/:id/wikis` | ### Group resources The following API resources are available in the group context: -| Resource | Available endpoints | -|:--------------------------------------------------|:---------------------------------------------------------------------------------| -| [Access requests](access_requests.md) | `/groups/:id/access_requests/` (also available for projects) | -| [Custom attributes](custom_attributes.md) | `/groups/:id/custom_attributes` (also available for projects and users) | -| [Groups](groups.md) | `/groups`, `/groups/.../subgroups` | -| [Group badges](group_badges.md) | `/groups/:id/badges` | -| [Group issue boards](group_boards.md) | `/groups/:id/boards` | -| [Group labels](group_labels.md) | `/groups/:id/labels` | -| [Group-level variables](group_level_variables.md) | `/groups/:id/variables` | -| [Group milestones](group_milestones.md) | `/groups/:id/milestones` | -| [Issues](issues.md) | `/groups/:id/issues` (also available for projects and standalone) | -| [Members](members.md) | `/groups/:id/members` (also available for projects) | -| [Merge requests](merge_requests.md) | `/groups/:id/merge_requests` (also available for projects and standalone) | -| [Notification settings](notification_settings.md) | `/groups/:id/notification_settings` (also available for projects and standalone) | -| [Search](search.md) | `/groups/:id/search` (also available for projects and standalone) | +| Resource | Available endpoints | +|:-----------------------------------------------------------------|:---------------------------------------------------------------------------------| +| [Access requests](access_requests.md) | `/groups/:id/access_requests/` (also available for projects) | +| [Custom attributes](custom_attributes.md) | `/groups/:id/custom_attributes` (also available for projects and users) | +| [Discussions](discussions.md) (threaded comments) **[ULTIMATE]** | `/groups/:id/epics/.../discussions` (also available for projects) | +| [Epic issues](epic_issues.md) **[ULTIMATE]** | `/groups/:id/epics/.../issues` | +| [Epic links](epic_links.md) **[ULTIMATE]** | `/groups/:id/epics/.../epics` | +| [Epics](epics.md) **[ULTIMATE]** | `/groups/:id/epics` | +| [Groups](groups.md) | `/groups`, `/groups/.../subgroups` | +| [Group badges](group_badges.md) | `/groups/:id/badges` | +| [Group issue boards](group_boards.md) | `/groups/:id/boards` | +| [Group labels](group_labels.md) | `/groups/:id/labels` | +| [Group-level variables](group_level_variables.md) | `/groups/:id/variables` | +| [Group milestones](group_milestones.md) | `/groups/:id/milestones` | +| [Issues](issues.md) | `/groups/:id/issues` (also available for projects and standalone) | +| [Members](members.md) | `/groups/:id/members` (also available for projects) | +| [Merge requests](merge_requests.md) | `/groups/:id/merge_requests` (also available for projects and standalone) | +| [Notes](notes.md) (comments) | `/groups/:id/epics/.../notes` (also available for projects) | +| [Notification settings](notification_settings.md) | `/groups/:id/notification_settings` (also available for projects and standalone) | +| [Resource label events](resource_label_events.md) | `/groups/:id/epics/.../resource_label_events` (also available for projects) | +| [Search](search.md) | `/groups/:id/search` (also available for projects and standalone) | ### Standalone resources @@ -102,9 +113,11 @@ The following API resources are available outside of project and group contexts | [Deploy keys](deploy_keys.md) | `/deploy_keys` (also available for projects) | | [Events](events.md) | `/events`, `/users/:id/events` (also available for projects) | | [Feature flags](features.md) | `/features` | +| [Geo Nodes](geo_nodes.md) **[PREMIUM ONLY]** | `/geo_nodes` | | [Import repository from GitHub](import.md) | `/import/github` | | [Issues](issues.md) | `/issues` (also available for groups and projects) | | [Keys](keys.md) | `/keys` | +| [License](license.md) **[CORE ONLY]** | `/license` | | [Markdown](markdown.md) | `/markdown` | | [Merge requests](merge_requests.md) | `/merge_requests` (also available for groups and projects) | | [Namespaces](namespaces.md) | `/namespaces` | @@ -131,6 +144,11 @@ Endpoints are available for: - [GitLab CI YAML templates](templates/gitlab_ci_ymls.md). - [Open source license templates](templates/licenses.md). +## SCIM **[SILVER ONLY]** + +[GitLab.com Silver and above](https://about.gitlab.com/pricing/) provides an [SCIM API](scim.md) that implements [the RFC7644 protocol](https://tools.ietf.org/html/rfc7644) and provides +the `/Users` endpoint. The base URL is: `/api/scim/v2/groups/:group_path/Users/`. + ## Road to GraphQL Going forward, we will start on moving to @@ -274,7 +292,7 @@ personal access tokens, and to using the [Sudo](#sudo) feature, since the user's password/token may not be known or may change over time. For more information, refer to the -[users API](users.md#retrieve-user-impersonation-tokens) docs. +[users API](users.md#create-an-impersonation-token) docs. Impersonation tokens are used exactly like regular personal access tokens, and can be passed in either the `private_token` parameter or the `Private-Token` header. diff --git a/doc/api/boards.md b/doc/api/boards.md index 28c73db6b98..a96206f5df3 100644 --- a/doc/api/boards.md +++ b/doc/api/boards.md @@ -141,6 +141,173 @@ Example response: } ``` +## Create a board **[STARTER]** + +Creates a board. + +``` +POST /projects/:id/boards +``` + +| Attribute | Type | Required | Description | +| --------- | ---- | -------- | ----------- | +| `id` | integer/string | yes | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding) owned by the authenticated user | +| `name` | string | yes | The name of the new board | + +```bash +curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" https://gitlab.example.com/api/v4/projects/5/boards?name=newboard +``` + +Example response: + +```json + { + "id": 1, + "project": { + "id": 5, + "name": "Diaspora Project Site", + "name_with_namespace": "Diaspora / Diaspora Project Site", + "path": "diaspora-project-site", + "path_with_namespace": "diaspora/diaspora-project-site", + "http_url_to_repo": "http://example.com/diaspora/diaspora-project-site.git", + "web_url": "http://example.com/diaspora/diaspora-project-site" + }, + "name": "newboard", + "milestone": { + "id": 12 + "title": "10.0" + }, + "lists" : [ + { + "id" : 1, + "label" : { + "name" : "Testing", + "color" : "#F0AD4E", + "description" : null + }, + "position" : 1 + }, + { + "id" : 2, + "label" : { + "name" : "Ready", + "color" : "#FF0000", + "description" : null + }, + "position" : 2 + }, + { + "id" : 3, + "label" : { + "name" : "Production", + "color" : "#FF5F00", + "description" : null + }, + "position" : 3 + } + ] + } +``` + +## Update a board **[STARTER]** + +> [Introduced][ee-5954] in [GitLab Starter](https://about.gitlab.com/pricing/) 11.1. + +Updates a board. + +``` +PUT /projects/:id/boards/:board_id +``` + +| Attribute | Type | Required | Description | +| ------------------- | -------------- | -------- | ----------- | +| `id` | integer/string | yes | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding) owned by the authenticated user | +| `board_id` | integer | yes | The ID of a board | +| `name` | string | no | The new name of the board | +| `assignee_id` | integer | no | The assignee the board should be scoped to | +| `milestone_id` | integer | no | The milestone the board should be scoped to | +| `labels` | string | no | Comma-separated list of label names which the board should be scoped to | +| `weight` | integer | no | The weight range from 0 to 9, to which the board should be scoped to | + + +```bash +curl --request PUT --header "PRIVATE-TOKEN: <your_access_token>" https://gitlab.example.com/api/v4/projects/5/boards/1?name=new_name&milestone_id=43&assignee_id=1&labels=Doing&weight=4 +``` + +Example response: + +```json + { + "id": 1, + "project": { + "id": 5, + "name": "Diaspora Project Site", + "name_with_namespace": "Diaspora / Diaspora Project Site", + "path": "diaspora-project-site", + "path_with_namespace": "diaspora/diaspora-project-site", + "created_at": "2018-07-03T05:48:49.982Z", + "default_branch": null, + "tag_list": [], + "ssh_url_to_repo": "ssh://user@example.com/diaspora/diaspora-project-site.git", + "http_url_to_repo": "http://example.com/diaspora/diaspora-project-site.git", + "web_url": "http://example.com/diaspora/diaspora-project-site", + "readme_url": null, + "avatar_url": null, + "star_count": 0, + "forks_count": 0, + "last_activity_at": "2018-07-03T05:48:49.982Z" + }, + "lists": [], + "name": "new_name", + "group": null, + "milestone": { + "id": 43, + "iid": 1, + "project_id": 15, + "title": "Milestone 1", + "description": "Milestone 1 desc", + "state": "active", + "created_at": "2018-07-03T06:36:42.618Z", + "updated_at": "2018-07-03T06:36:42.618Z", + "due_date": null, + "start_date": null, + "web_url": "http://example.com/root/board1/milestones/1" + }, + "assignee": { + "id": 1, + "name": "Administrator", + "username": "root", + "state": "active", + "avatar_url": "https://www.gravatar.com/avatar/e64c7d89f26bd1972efa854d13d7dd61?s=80&d=identicon", + "web_url": "http://example.com/root" + }, + "labels": [{ + "id": 10, + "name": "Doing", + "color": "#5CB85C", + "description": null + }], + "weight": 4 + } +``` + +## Delete a board **[STARTER]** + +Deletes a board. + +``` +DELETE /projects/:id/boards/:board_id +``` + +| Attribute | Type | Required | Description | +| --------- | ---- | -------- | ----------- | +| `id` | integer/string | yes | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding) owned by the authenticated user | +| `board_id` | integer | yes | The ID of a board | + +```bash +curl --request DELETE --header "PRIVATE-TOKEN: <your_access_token>" https://gitlab.example.com/api/v4/projects/5/boards/1 +``` + ## List board lists Get a list of the board's lists. @@ -237,7 +404,15 @@ POST /projects/:id/boards/:board_id/lists | --------- | ---- | -------- | ----------- | | `id` | integer/string | yes | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding) owned by the authenticated user | | `board_id` | integer | yes | The ID of a board | -| `label_id` | integer | yes | The ID of a label | +| `label_id` | integer | no | The ID of a label | +| `assignee_id` **[PREMIUM]** | integer | no | The ID of a user | +| `milestone_id` **[PREMIUM]** | integer | no | The ID of a milestone | + +NOTE: **Note**: +Label, assignee and milestone arguments are mutually exclusive, +that is, only one of them are accepted in a request. +Check the [Issue Board docs](../user/project/issue_board.md#summary-of-features-per-tier) +for more information regarding the required license for each list type. ```bash curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" https://gitlab.example.com/api/v4/projects/5/boards/1/lists?label_id=5 @@ -307,3 +482,5 @@ DELETE /projects/:id/boards/:board_id/lists/:list_id ```bash curl --request DELETE --header "PRIVATE-TOKEN: <your_access_token>" https://gitlab.example.com/api/v4/projects/5/boards/1/lists/1 ``` + +[ee-5954]: https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/5954 diff --git a/doc/api/build_triggers.md b/doc/api/build_triggers.md index 20d924ab35e..bad7a655d08 100644 --- a/doc/api/build_triggers.md +++ b/doc/api/build_triggers.md @@ -1 +1,5 @@ -This document was moved to [Pipeline Triggers](pipeline_triggers.md). +--- +redirect_to: 'pipeline_triggers.md' +--- + +This document was moved to [another location](pipeline_triggers.md). diff --git a/doc/api/builds.md b/doc/api/builds.md index a6edda68bc4..0154d35cab6 100644 --- a/doc/api/builds.md +++ b/doc/api/builds.md @@ -1 +1,5 @@ +--- +redirect_to: 'jobs.md' +--- + This document was moved to [another location](jobs.md). diff --git a/doc/api/commits.md b/doc/api/commits.md index 8d36ae7d559..25015fad9e3 100644 --- a/doc/api/commits.md +++ b/doc/api/commits.md @@ -75,10 +75,12 @@ POST /projects/:id/repository/commits | `branch` | string | yes | Name of the branch to commit into. To create a new branch, also provide `start_branch`. | | `commit_message` | string | yes | Commit message | | `start_branch` | string | no | Name of the branch to start the new commit from | +| `start_project` | integer/string | no | The project ID or [URL-encoded path of the project](README.md#namespaced-path-encoding) to start the commit from. Defaults to the value of `id`. | | `actions[]` | array | yes | An array of action hashes to commit as a batch. See the next table for what attributes it can take. | | `author_email` | string | no | Specify the commit author's email address | | `author_name` | string | no | Specify the commit author's name | | `stats` | boolean | no | Include commit stats. Default is true | +| `force` | boolean | no | When `true` overwrites the target branch with a new commit based on the `start_branch` | | `actions[]` Attribute | Type | Required | Description | | --------------------- | ---- | -------- | ----------- | @@ -129,6 +131,7 @@ curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" --header "Cont ``` Example response: + ```json { "id": "ed899a2f4b50b4370feeea94676502b42383c746", @@ -154,6 +157,32 @@ Example response: } ``` +GitLab supports [form encoding](README.md#encoding-api-parameters-of-array-and-hash-types). The following is an example using Commit API with form encoding: + +```bash +curl --request POST \ + --form "branch=master" \ + --form "commit_message=some commit message" \ + --form "start_branch=master" \ + --form "actions[][action]=create" \ + --form "actions[][file_path]=foo/bar" \ + --form "actions[][content]=</path/to/local.file" \ + --form "actions[][action]=delete" \ + --form "actions[][file_path]=foo/bar2" \ + --form "actions[][action]=move" \ + --form "actions[][file_path]=foo/bar3" \ + --form "actions[][previous_path]=foo/bar4" \ + --form "actions[][content]=</path/to/local1.file" \ + --form "actions[][action]=update" \ + --form "actions[][file_path]=foo/bar5" \ + --form "actions[][content]=</path/to/local2.file" \ + --form "actions[][action]=chmod" \ + --form "actions[][file_path]=foo/bar5" \ + --form "actions[][execute_filemode]=true" \ + --header "PRIVATE-TOKEN: <your_access_token>" \ + "https://gitlab.example.com/api/v4/projects/1/repository/commits" +``` + ## Get a single commit Get a specific commit identified by the commit hash or name of a branch or tag. @@ -195,9 +224,9 @@ Example response: "last_pipeline" : { "id": 8, "ref": "master", - "sha": "2dc6aa325a317eda67812f05600bdf0fcdc70ab0" + "sha": "2dc6aa325a317eda67812f05600bdf0fcdc70ab0", "status": "created" - } + }, "stats": { "additions": 15, "deletions": 10, @@ -474,7 +503,7 @@ GET /projects/:id/repository/commits/:sha/statuses | `sha` | string | yes | The commit SHA | `ref` | string | no | The name of a repository branch or tag or, if not given, the default branch | `stage` | string | no | Filter by [build stage](../ci/yaml/README.md#stages), e.g., `test` -| `name` | string | no | Filter by [job name](../ci/yaml/README.md#jobs), e.g., `bundler:audit` +| `name` | string | no | Filter by [job name](../ci/yaml/README.md#introduction), e.g., `bundler:audit` | `all` | boolean | no | Return all statuses, not only the latest ones ```bash diff --git a/doc/api/discussions.md b/doc/api/discussions.md index 7d68d0ae744..9defef4fd53 100644 --- a/doc/api/discussions.md +++ b/doc/api/discussions.md @@ -1,6 +1,21 @@ # Discussions API -Discussions are set of related notes on snippets, issues, merge requests or commits. +Discussions are a set of related notes on: + +- Snippets +- Issues +- Epics **[ULTIMATE]** +- Merge requests +- Commits + +This includes system notes, which are notes about changes to the object (for example, when a milestone changes, there will be a corresponding system note). Label notes are not part of this API, but recorded as separate events in [resource label events](resource_label_events.md). + +## Discussions pagination + +By default, `GET` requests return 20 results at a time because the API results +are paginated. + +Read more on [pagination](README.md#pagination). ## Issues @@ -144,7 +159,8 @@ curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" https://gitlab ### Add note to existing issue discussion -Adds a new note to the discussion. +Adds a new note to the discussion. This can also +[create a discussion from a single comment](../user/discussions/#start-a-discussion-by-replying-to-a-standard-comment). ``` POST /projects/:id/issues/:issue_iid/discussions/:discussion_id/notes @@ -414,6 +430,214 @@ Parameters: curl --request DELETE --header "PRIVATE-TOKEN: <your_access_token>" https://gitlab.example.com/api/v4/projects/5/snippets/11/discussions/636 ``` +## Epics **[ULTIMATE]** + +### List group epic discussions + +Gets a list of all discussions for a single epic. + +``` +GET /groups/:id/epics/:epic_id/discussions +``` + +| Attribute | Type | Required | Description | +| ------------------- | ---------------- | ---------- | ------------ | +| `id` | integer/string | yes | The ID or [URL-encoded path of the group](README.md#namespaced-path-encoding) | +| `epic_id` | integer | yes | The ID of an epic | + +```json +[ + { + "id": "6a9c1750b37d513a43987b574953fceb50b03ce7", + "individual_note": false, + "notes": [ + { + "id": 1126, + "type": "DiscussionNote", + "body": "discussion text", + "attachment": null, + "author": { + "id": 1, + "name": "root", + "username": "root", + "state": "active", + "avatar_url": "https://www.gravatar.com/avatar/00afb8fb6ab07c3ee3e9c1f38777e2f4?s=80&d=identicon", + "web_url": "http://localhost:3000/root" + }, + "created_at": "2018-03-03T21:54:39.668Z", + "updated_at": "2018-03-03T21:54:39.668Z", + "system": false, + "noteable_id": 3, + "noteable_type": "Epic", + "noteable_id": null, + "resolvable": false + }, + { + "id": 1129, + "type": "DiscussionNote", + "body": "reply to the discussion", + "attachment": null, + "author": { + "id": 1, + "name": "root", + "username": "root", + "state": "active", + "avatar_url": "https://www.gravatar.com/avatar/00afb8fb6ab07c3ee3e9c1f38777e2f4?s=80&d=identicon", + "web_url": "http://localhost:3000/root" + }, + "created_at": "2018-03-04T13:38:02.127Z", + "updated_at": "2018-03-04T13:38:02.127Z", + "system": false, + "noteable_id": 3, + "noteable_type": "Epic", + "noteable_id": null, + "resolvable": false + } + ] + }, + { + "id": "87805b7c09016a7058e91bdbe7b29d1f284a39e6", + "individual_note": true, + "notes": [ + { + "id": 1128, + "type": null, + "body": "a single comment", + "attachment": null, + "author": { + "id": 1, + "name": "root", + "username": "root", + "state": "active", + "avatar_url": "https://www.gravatar.com/avatar/00afb8fb6ab07c3ee3e9c1f38777e2f4?s=80&d=identicon", + "web_url": "http://localhost:3000/root" + }, + "created_at": "2018-03-04T09:17:22.520Z", + "updated_at": "2018-03-04T09:17:22.520Z", + "system": false, + "noteable_id": 3, + "noteable_type": "Epic", + "noteable_id": null, + "resolvable": false + } + ] + } +] +``` + +```bash +curl --request GET --header "PRIVATE-TOKEN: <your_access_token>" https://gitlab.example.com/api/v4/groups/5/epics/11/discussions +``` + +### Get single epic discussion + +Returns a single discussion for a specific group epic + +``` +GET /groups/:id/epics/:epic_id/discussions/:discussion_id +``` + +Parameters: + +| Attribute | Type | Required | Description | +| --------------- | -------------- | -------- | ----------- | +| `id` | integer/string | yes | The ID or [URL-encoded path of the group](README.md#namespaced-path-encoding) | +| `epic_id` | integer | yes | The ID of an epic | +| `discussion_id` | integer | yes | The ID of a discussion | + +```bash +curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" https://gitlab.example.com/api/v4/groups/5/epics/11/discussions/6a9c1750b37d513a43987b574953fceb50b03ce7 +``` + +### Create new epic discussion + +Creates a new discussion to a single group epic. This is similar to creating +a note but but another comments (replies) can be added to it later. + +``` +POST /groups/:id/epics/:epic_id/discussions +``` + +Parameters: + +| Attribute | Type | Required | Description | +| --------------- | -------------- | -------- | ----------- | +| `id` | integer/string | yes | The ID or [URL-encoded path of the group](README.md#namespaced-path-encoding) | +| `epic_id` | integer | yes | The ID of an epic | +| `body` | string | yes | The content of a discussion | +| `created_at` | string | no | Date time string, ISO 8601 formatted, e.g. 2016-03-11T03:45:40Z (requires admin or project/group owner rights) | + +```bash +curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" https://gitlab.example.com/api/v4/groups/5/epics/11/discussions?body=comment +``` + +### Add note to existing epic discussion + +Adds a new note to the discussion. This can also +[create a discussion from a single comment](../user/discussions/#start-a-discussion-by-replying-to-a-standard-comment). + +``` +POST /groups/:id/epics/:epic_id/discussions/:discussion_id/notes +``` + +Parameters: + +| Attribute | Type | Required | Description | +| --------------- | -------------- | -------- | ----------- | +| `id` | integer/string | yes | The ID or [URL-encoded path of the group](README.md#namespaced-path-encoding) | +| `epic_id` | integer | yes | The ID of an epic | +| `discussion_id` | integer | yes | The ID of a discussion | +| `note_id` | integer | yes | The ID of a discussion note | +| `body` | string | yes | The content of a discussion | +| `created_at` | string | no | Date time string, ISO 8601 formatted, e.g. 2016-03-11T03:45:40Z (requires admin or project/group owner rights) | + +```bash +curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" https://gitlab.example.com/api/v4/groups/5/epics/11/discussions/6a9c1750b37d513a43987b574953fceb50b03ce7/notes?body=comment +``` + +### Modify existing epic discussion note + +Modify existing discussion note of an epic. + +``` +PUT /groups/:id/epics/:epic_id/discussions/:discussion_id/notes/:note_id +``` + +Parameters: + +| Attribute | Type | Required | Description | +| --------------- | -------------- | -------- | ----------- | +| `id` | integer/string | yes | The ID or [URL-encoded path of the group](README.md#namespaced-path-encoding) | +| `epic_id` | integer | yes | The ID of an epic | +| `discussion_id` | integer | yes | The ID of a discussion | +| `note_id` | integer | yes | The ID of a discussion note | +| `body` | string | yes | The content of a discussion | + +```bash +curl --request PUT --header "PRIVATE-TOKEN: <your_access_token>" https://gitlab.example.com/api/v4/groups/5/epics/11/discussions/6a9c1750b37d513a43987b574953fceb50b03ce7/notes/1108?body=comment +``` + +### Delete an epic discussion note + +Deletes an existing discussion note of an epic. + +``` +DELETE /groups/:id/epics/:epic_id/discussions/:discussion_id/notes/:note_id +``` + +Parameters: + +| Attribute | Type | Required | Description | +| --------------- | -------------- | -------- | ----------- | +| `id` | integer/string | yes | The ID or [URL-encoded path of the group](README.md#namespaced-path-encoding) | +| `epic_id` | integer | yes | The ID of an epic | +| `discussion_id` | integer | yes | The ID of a discussion | +| `note_id` | integer | yes | The ID of a discussion note | + +```bash +curl --request DELETE --header "PRIVATE-TOKEN: <your_access_token>" https://gitlab.example.com/api/v4/groups/5/epics/11/discussions/636 +``` + ## Merge requests ### List project merge request discussions @@ -643,7 +867,8 @@ curl --request PUT --header "PRIVATE-TOKEN: <your_access_token>" https://gitlab. ### Add note to existing merge request discussion -Adds a new note to the discussion. +Adds a new note to the discussion. This can also +[create a discussion from a single comment](../user/discussions/#start-a-discussion-by-replying-to-a-standard-comment). ``` POST /projects/:id/merge_requests/:merge_request_iid/discussions/:discussion_id/notes diff --git a/doc/api/environments.md b/doc/api/environments.md index 4a38dd73747..ebcdc546d08 100644 --- a/doc/api/environments.md +++ b/doc/api/environments.md @@ -29,6 +29,111 @@ Example response: ] ``` +## Get a specific environment + +``` +GET /projects/:id/environments/:environment_id +``` + +| Attribute | Type | Required | Description | +|-----------|---------|----------|---------------------| +| `id` | integer/string | yes | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding) owned by the authenticated user | +| `environment_id` | integer | yes | The ID of the environment | + +```bash +curl --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/1/environments/1" +``` + +Example of response + +```json +{ + "id": 1, + "name": "review/fix-foo", + "slug": "review-fix-foo-dfjre3", + "external_url": "https://review-fix-foo-dfjre3.example.gitlab.com" + "last_deployment": { + "id": 100, + "iid": 34, + "ref": "fdroid", + "sha": "416d8ea11849050d3d1f5104cf8cf51053e790ab", + "created_at": "2019-03-25T18:55:13.252Z", + "user": { + "id": 1, + "name": "Administrator", + "state": "active", + "username": "root", + "avatar_url": "http://www.gravatar.com/avatar/e64c7d89f26bd1972efa854d13d7dd61?s=80&d=identicon", + "web_url": "http://localhost:3000/root" + } + "deployable": { + "id": 710, + "status": "success", + "stage": "deploy", + "name": "staging", + "ref": "fdroid", + "tag": false, + "coverage": null, + "created_at": "2019-03-25T18:55:13.215Z", + "started_at": "2019-03-25T12:54:50.082Z", + "finished_at": "2019-03-25T18:55:13.216Z", + "duration": 21623.13423, + "user": { + "id": 1, + "name": "Administrator", + "username": "root", + "state": "active", + "avatar_url": "http://www.gravatar.com/avatar/e64c7d89f26bd1972efa854d13d7dd61?s=80&d=identicon", + "web_url": "http://gitlab.dev/root", + "created_at": "2015-12-21T13:14:24.077Z", + "bio": null, + "location": null, + "public_email": "", + "skype": "", + "linkedin": "", + "twitter": "", + "website_url": "", + "organization": null + } + "commit": { + "id": "416d8ea11849050d3d1f5104cf8cf51053e790ab", + "short_id": "416d8ea1", + "created_at": "2016-01-02T15:39:18.000Z", + "parent_ids": [ + "e9a4449c95c64358840902508fc827f1a2eab7df" + ], + "title": "Removed fabric to fix #40", + "message": "Removed fabric to fix #40\n", + "author_name": "Administrator", + "author_email": "admin@example.com", + "authored_date": "2016-01-02T15:39:18.000Z", + "committer_name": "Administrator", + "committer_email": "admin@example.com", + "committed_date": "2016-01-02T15:39:18.000Z" + }, + "pipeline": { + "id": 34, + "sha": "416d8ea11849050d3d1f5104cf8cf51053e790ab", + "ref": "fdroid", + "status": "success", + "web_url": "http://localhost:3000/Commit451/lab-coat/pipelines/34" + }, + "web_url": "http://localhost:3000/Commit451/lab-coat/-/jobs/710", + "artifacts": [ + { + "file_type": "trace", + "size": 1305, + "filename": "job.log", + "file_format": null + } + ], + "runner": null, + "artifacts_expire_at": null + } + } +} +``` + ## Create a new environment Creates a new environment with the given name and external_url. diff --git a/doc/api/epic_issues.md b/doc/api/epic_issues.md new file mode 100644 index 00000000000..ec59ea7068e --- /dev/null +++ b/doc/api/epic_issues.md @@ -0,0 +1,413 @@ +# Epic Issues API **[ULTIMATE]** + +Every API call to epic_issues must be authenticated. + +If a user is not a member of a group and the group is private, a `GET` request on that group will result to a `404` status code. + +Epics are available only in Ultimate. If epics feature is not available a `403` status code will be returned. + +## List issues for an epic + +Gets all issues that are assigned to an epic and the authenticated user has access to. + +``` +GET /groups/:id/epics/:epic_iid/issues +``` + +| Attribute | Type | Required | Description | +| ------------------- | ---------------- | ---------- | ---------------------------------------------------------------------------------------| +| `id` | integer/string | yes | The ID or [URL-encoded path of the group](README.md#namespaced-path-encoding) owned by the authenticated user | +| `epic_iid` | integer/string | yes | The internal ID of the epic. | + +```bash +curl --header "PRIVATE-TOKEN: <your_access_token>" https://gitlab.example.com/api/v4/groups/1/epics/5/issues/ +``` + +Example response: + +```json +[ + { + "id": 76, + "iid": 6, + "project_id": 8, + "title" : "Consequatur vero maxime deserunt laboriosam est voluptas dolorem.", + "description" : "Ratione dolores corrupti mollitia soluta quia.", + "state": "opened", + "created_at": "2017-11-15T13:39:24.670Z", + "updated_at": "2018-01-04T10:49:19.506Z", + "closed_at": null, + "labels": [], + "milestone": { + "id": 38, + "iid": 3, + "project_id": 8, + "title": "v2.0", + "description": "In tempore culpa inventore quo accusantium.", + "state": "closed", + "created_at": "2017-11-15T13:39:13.825Z", + "updated_at": "2017-11-15T13:39:13.825Z", + "due_date": null, + "start_date": null + }, + "assignees": [{ + "id": 7, + "name": "Pamella Huel", + "username": "arnita", + "state": "active", + "avatar_url": "http://www.gravatar.com/avatar/a2f5c6fcef64c9c69cb8779cb292be1b?s=80&d=identicon", + "web_url": "http://localhost:3001/arnita" + }], + "assignee": { + "id": 7, + "name": "Pamella Huel", + "username": "arnita", + "state": "active", + "avatar_url": "http://www.gravatar.com/avatar/a2f5c6fcef64c9c69cb8779cb292be1b?s=80&d=identicon", + "web_url": "http://localhost:3001/arnita" + }, + "author": { + "id": 13, + "name": "Michell Johns", + "username": "chris_hahn", + "state": "active", + "avatar_url": "http://www.gravatar.com/avatar/30e3b2122ccd6b8e45e8e14a3ffb58fc?s=80&d=identicon", + "web_url": "http://localhost:3001/chris_hahn" + }, + "user_notes_count": 8, + "upvotes": 0, + "downvotes": 0, + "due_date": null, + "confidential": false, + "weight": null, + "discussion_locked": null, + "web_url": "http://localhost:3001/h5bp/html5-boilerplate/issues/6", + "time_stats": { + "time_estimate": 0, + "total_time_spent": 0, + "human_time_estimate": null, + "human_total_time_spent": null + }, + "_links":{ + "self": "http://localhost:3001/api/v4/projects/8/issues/6", + "notes": "http://localhost:3001/api/v4/projects/8/issues/6/notes", + "award_emoji": "http://localhost:3001/api/v4/projects/8/issues/6/award_emoji", + "project": "http://localhost:3001/api/v4/projects/8" + }, + "subscribed": true, + "epic_issue_id": 2 + } +] +``` + +**Note**: `assignee` column is deprecated, now we show it as a single-sized array `assignees` to conform to the GitLab EE API. + +## Assign an issue to the epic + +Creates an epic - issue association. If the issue in question belongs to another epic it is unassigned from that epic. + +``` +POST /groups/:id/epics/:epic_iid/issues/:issue_id +``` + +| Attribute | Type | Required | Description | +| ------------------- | ---------------- | ---------- | ---------------------------------------------------------------------------------------| +| `id` | integer/string | yes | The ID or [URL-encoded path of the group](README.md#namespaced-path-encoding) owned by the authenticated user | +| `epic_iid` | integer/string | yes | The internal ID of the epic. | +| `issue_id` | integer/string | yes | The ID of the issue. | + +```bash +curl --header POST "PRIVATE-TOKEN: <your_access_token>" https://gitlab.example.com/api/v4/groups/1/epics/5/issues/55 +``` + +Example response: + +```json +{ + "id": 11, + "epic": { + "id": 30, + "iid": 5, + "title": "Ea cupiditate dolores ut vero consequatur quasi veniam voluptatem et non.", + "description": "Molestias dolorem eos vitae expedita impedit necessitatibus quo voluptatum.", + "author": { + "id": 7, + "name": "Pamella Huel", + "username": "arnita", + "state": "active", + "avatar_url": "http://www.gravatar.com/avatar/a2f5c6fcef64c9c69cb8779cb292be1b?s=80&d=identicon", + "web_url": "http://localhost:3001/arnita" + }, + "start_date": null, + "end_date": null + }, + "issue": { + "id": 55, + "iid": 13, + "project_id": 8, + "title": "Beatae laborum voluptatem voluptate eligendi ex accusamus.", + "description": "Quam veritatis debitis omnis aliquam sit.", + "state": "opened", + "created_at": "2017-11-05T13:59:12.782Z", + "updated_at": "2018-01-05T10:33:03.900Z", + "closed_at": null, + "labels": [], + "milestone": { + "id": 48, + "iid": 6, + "project_id": 8, + "title": "Sprint - Sed sed maxime temporibus ipsa ullam qui sit.", + "description": "Quos veritatis qui expedita sunt deleniti accusamus.", + "state": "active", + "created_at": "2017-11-05T13:59:12.445Z", + "updated_at": "2017-11-05T13:59:12.445Z", + "due_date": "2017-11-13", + "start_date": "2017-11-05" + }, + "assignees": [{ + "id": 10, + "name": "Lu Mayer", + "username": "kam", + "state": "active", + "avatar_url": "http://www.gravatar.com/avatar/018729e129a6f31c80a6327a30196823?s=80&d=identicon", + "web_url": "http://localhost:3001/kam" + }], + "assignee": { + "id": 10, + "name": "Lu Mayer", + "username": "kam", + "state": "active", + "avatar_url": "http://www.gravatar.com/avatar/018729e129a6f31c80a6327a30196823?s=80&d=identicon", + "web_url": "http://localhost:3001/kam" + }, + "author": { + "id": 25, + "name": "User 3", + "username": "user3", + "state": "active", + "avatar_url": "http://www.gravatar.com/avatar/97d6d9441ff85fdc730e02a6068d267b?s=80&d=identicon", + "web_url": "http://localhost:3001/user3" + }, + "user_notes_count": 0, + "upvotes": 0, + "downvotes": 0, + "due_date": null, + "confidential": false, + "weight": null, + "discussion_locked": null, + "web_url": "http://localhost:3001/h5bp/html5-boilerplate/issues/13", + "time_stats": { + "time_estimate": 0, + "total_time_spent": 0, + "human_time_estimate": null, + "human_total_time_spent": null + } + } +} +``` + +**Note**: `assignee` column is deprecated, now we show it as a single-sized array `assignees` to conform to the GitLab EE API. + +## Remove an issue from the epic + +Removes an epic - issue association. + +``` +DELETE /groups/:id/epics/:epic_iid/issues/:epic_issue_id +``` + +| Attribute | Type | Required | Description | +| ------------------- | ---------------- | ---------- | -----------------------------------------------------------------------------------------------------| +| `id` | integer/string | yes | The ID or [URL-encoded path of the group](README.md#namespaced-path-encoding) owned by the authenticated user | +| `epic_iid` | integer/string | yes | The internal ID of the epic. | +| `epic_issue_id` | integer/string | yes | The ID of the issue - epic association. | + +```bash +curl --header DELETE "PRIVATE-TOKEN: <your_access_token>" https://gitlab.example.com/api/v4/groups/1/epics/5/issues/11 +``` + +Example response: + +```json +{ + "id": 11, + "epic": { + "id": 30, + "iid": 5, + "title": "Ea cupiditate dolores ut vero consequatur quasi veniam voluptatem et non.", + "description": "Molestias dolorem eos vitae expedita impedit necessitatibus quo voluptatum.", + "author": { + "id": 7, + "name": "Pamella Huel", + "username": "arnita", + "state": "active", + "avatar_url": "http://www.gravatar.com/avatar/a2f5c6fcef64c9c69cb8779cb292be1b?s=80&d=identicon", + "web_url": "http://localhost:3001/arnita" + }, + "start_date": null, + "end_date": null + }, + "issue": { + "id": 223, + "iid": 13, + "project_id": 8, + "title": "Beatae laborum voluptatem voluptate eligendi ex accusamus.", + "description": "Quam veritatis debitis omnis aliquam sit.", + "state": "opened", + "created_at": "2017-11-05T13:59:12.782Z", + "updated_at": "2018-01-05T10:33:03.900Z", + "closed_at": null, + "labels": [], + "milestone": { + "id": 48, + "iid": 6, + "project_id": 8, + "title": "Sprint - Sed sed maxime temporibus ipsa ullam qui sit.", + "description": "Quos veritatis qui expedita sunt deleniti accusamus.", + "state": "active", + "created_at": "2017-11-05T13:59:12.445Z", + "updated_at": "2017-11-05T13:59:12.445Z", + "due_date": "2017-11-13", + "start_date": "2017-11-05" + }, + "assignees": [{ + "id": 10, + "name": "Lu Mayer", + "username": "kam", + "state": "active", + "avatar_url": "http://www.gravatar.com/avatar/018729e129a6f31c80a6327a30196823?s=80&d=identicon", + "web_url": "http://localhost:3001/kam" + }], + "assignee": { + "id": 10, + "name": "Lu Mayer", + "username": "kam", + "state": "active", + "avatar_url": "http://www.gravatar.com/avatar/018729e129a6f31c80a6327a30196823?s=80&d=identicon", + "web_url": "http://localhost:3001/kam" + }, + "author": { + "id": 25, + "name": "User 3", + "username": "user3", + "state": "active", + "avatar_url": "http://www.gravatar.com/avatar/97d6d9441ff85fdc730e02a6068d267b?s=80&d=identicon", + "web_url": "http://localhost:3001/user3" + }, + "user_notes_count": 0, + "upvotes": 0, + "downvotes": 0, + "due_date": null, + "confidential": false, + "weight": null, + "discussion_locked": null, + "web_url": "http://localhost:3001/h5bp/html5-boilerplate/issues/13", + "time_stats": { + "time_estimate": 0, + "total_time_spent": 0, + "human_time_estimate": null, + "human_total_time_spent": null + } + } +} +``` + +**Note**: `assignee` column is deprecated, now we show it as a single-sized array `assignees` to conform to the GitLab EE API. + +## Update epic - issue association + +Updates an epic - issue association. + +``` +PUT /groups/:id/epics/:epic_iid/issues/:epic_issue_id +``` + +| Attribute | Type | Required | Description | +| ------------------- | ---------------- | ---------- | -----------------------------------------------------------------------------------------------------| +| `id` | integer/string | yes | The ID or [URL-encoded path of the group](README.md#namespaced-path-encoding) owned by the authenticated user | +| `epic_iid` | integer/string | yes | The internal ID of the epic. | +| `epic_issue_id` | integer/string | yes | The ID of the issue - epic association. | +| `move_before_id` | integer/string | no | The ID of the issue - epic association that should be placed before the link in the question. | +| `move_after_id` | integer/string | no | The ID of the issue - epic association that should be placed after the link in the question. | + +```bash +curl --header PUT "PRIVATE-TOKEN: <your_access_token>" https://gitlab.example.com/api/v4/groups/1/epics/5/issues/11?move_before_id=20 +``` + +Example response: + +```json +[ + { + "id": 30, + "iid": 6, + "project_id": 8, + "title" : "Consequatur vero maxime deserunt laboriosam est voluptas dolorem.", + "description" : "Ratione dolores corrupti mollitia soluta quia.", + "state": "opened", + "created_at": "2017-11-15T13:39:24.670Z", + "updated_at": "2018-01-04T10:49:19.506Z", + "closed_at": null, + "labels": [], + "milestone": { + "id": 38, + "iid": 3, + "project_id": 8, + "title": "v2.0", + "description": "In tempore culpa inventore quo accusantium.", + "state": "closed", + "created_at": "2017-11-15T13:39:13.825Z", + "updated_at": "2017-11-15T13:39:13.825Z", + "due_date": null, + "start_date": null + }, + "assignees": [{ + "id": 7, + "name": "Pamella Huel", + "username": "arnita", + "state": "active", + "avatar_url": "http://www.gravatar.com/avatar/a2f5c6fcef64c9c69cb8779cb292be1b?s=80&d=identicon", + "web_url": "http://localhost:3001/arnita" + }], + "assignee": { + "id": 7, + "name": "Pamella Huel", + "username": "arnita", + "state": "active", + "avatar_url": "http://www.gravatar.com/avatar/a2f5c6fcef64c9c69cb8779cb292be1b?s=80&d=identicon", + "web_url": "http://localhost:3001/arnita" + }, + "author": { + "id": 13, + "name": "Michell Johns", + "username": "chris_hahn", + "state": "active", + "avatar_url": "http://www.gravatar.com/avatar/30e3b2122ccd6b8e45e8e14a3ffb58fc?s=80&d=identicon", + "web_url": "http://localhost:3001/chris_hahn" + }, + "user_notes_count": 8, + "upvotes": 0, + "downvotes": 0, + "due_date": null, + "confidential": false, + "weight": null, + "discussion_locked": null, + "web_url": "http://localhost:3001/h5bp/html5-boilerplate/issues/6", + "time_stats": { + "time_estimate": 0, + "total_time_spent": 0, + "human_time_estimate": null, + "human_total_time_spent": null + }, + "_links":{ + "self": "http://localhost:3001/api/v4/projects/8/issues/6", + "notes": "http://localhost:3001/api/v4/projects/8/issues/6/notes", + "award_emoji": "http://localhost:3001/api/v4/projects/8/issues/6/award_emoji", + "project": "http://localhost:3001/api/v4/projects/8" + }, + "subscribed": true, + "epic_issue_id": 11, + "relative_position": 55 + } +] +``` diff --git a/doc/api/epic_links.md b/doc/api/epic_links.md new file mode 100644 index 00000000000..9ad90a6d0f1 --- /dev/null +++ b/doc/api/epic_links.md @@ -0,0 +1,254 @@ +# Epic Links API **[ULTIMATE]** + +>**Note:** +> This endpoint was [introduced](https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/9188) in GitLab 11.8. + +Manages parent-child [epic relationships](../user/group/epics/index.md#multi-level-child-epics). + +Every API call to `epic_links` must be authenticated. + +If a user is not a member of a group and the group is private, a `GET` request on that group will result to a `404` status code. + +Epics are available only in the [Ultimate/Gold tier](https://about.gitlab.com/pricing/). If the epics feature is not available, a `403` status code will be returned. + +## List epics related to a given epic + +Gets all child epics of an epic. + +``` +GET /groups/:id/epics/:epic_iid/epics +``` + +| Attribute | Type | Required | Description | +| ---------- | -------------- | -------- | ------------------------------------------------------------------------------------------------------------- | +| `id` | integer/string | yes | The ID or [URL-encoded path of the group](README.md#namespaced-path-encoding) owned by the authenticated user | +| `epic_iid` | integer | yes | The internal ID of the epic. | + +```bash +curl --header "PRIVATE-TOKEN: <your_access_token>" https://gitlab.example.com/api/v4/groups/1/epics/5/epics/ +``` + +Example response: + +```json +[ + { + "id": 29, + "iid": 6, + "group_id": 1, + "parent_id": 5, + "title": "Accusamus iste et ullam ratione voluptatem omnis debitis dolor est.", + "description": "Molestias dolorem eos vitae expedita impedit necessitatibus quo voluptatum.", + "author": { + "id": 10, + "name": "Lu Mayer", + "username": "kam", + "state": "active", + "avatar_url": "http://www.gravatar.com/avatar/018729e129a6f31c80a6327a30196823?s=80&d=identicon", + "web_url": "http://localhost:3001/kam" + }, + "start_date": null, + "start_date_is_fixed": false, + "start_date_fixed": null, + "start_date_from_milestones": null, + "end_date": "2018-07-31", + "due_date": "2018-07-31", + "due_date_is_fixed": false, + "due_date_fixed": null, + "due_date_from_milestones": "2018-07-31", + "created_at": "2018-07-17T13:36:22.770Z", + "updated_at": "2018-07-18T12:22:05.239Z", + "labels": [] + } +] +``` + +## Assign a child epic + +Creates an association between two epics, designating one as the parent epic and the other as the child epic. A parent epic can have multiple child epics. If the new child epic already belonged to another epic, it is unassigned from that previous parent. + +``` +POST /groups/:id/epics/:epic_iid/epics +``` + +| Attribute | Type | Required | Description | +| --------------- | -------------- | -------- | ------------------------------------------------------------------------------------------------------------------ | +| `id` | integer/string | yes | The ID or [URL-encoded path of the group](README.md#namespaced-path-encoding) owned by the authenticated user | +| `epic_iid` | integer | yes | The internal ID of the epic. | +| `child_epic_id` | integer | yes | The global ID of the child epic. Internal ID can't be used because they can conflict with epics from other groups. | + +```bash +curl --header POST "PRIVATE-TOKEN: <your_access_token>" https://gitlab.example.com/api/v4/groups/1/epics/5/epics/6 +``` + +Example response: + +```json +{ + "id": 6, + "iid": 38, + "group_id": 1, + "parent_id": 5 + "title": "Accusamus iste et ullam ratione voluptatem omnis debitis dolor est.", + "description": "Molestias dolorem eos vitae expedita impedit necessitatibus quo voluptatum.", + "author": { + "id": 10, + "name": "Lu Mayer", + "username": "kam", + "state": "active", + "avatar_url": "http://www.gravatar.com/avatar/018729e129a6f31c80a6327a30196823?s=80&d=identicon", + "web_url": "http://localhost:3001/kam" + }, + "start_date": null, + "start_date_is_fixed": false, + "start_date_fixed": null, + "start_date_from_milestones": null, + "end_date": "2018-07-31", + "due_date": "2018-07-31", + "due_date_is_fixed": false, + "due_date_fixed": null, + "due_date_from_milestones": "2018-07-31", + "created_at": "2018-07-17T13:36:22.770Z", + "updated_at": "2018-07-18T12:22:05.239Z", + "labels": [] +} +``` + +## Create and assign a child epic + +Creates a a new epic and associates it with provided parent epic. The response is LinkedEpic object. + +``` +POST /groups/:id/epics/:epic_iid/epics +``` + +| Attribute | Type | Required | Description | +| --------------- | -------------- | -------- | ------------------------------------------------------------------------------------------------------------------ | +| `id` | integer/string | yes | The ID or [URL-encoded path of the group](README.md#namespaced-path-encoding) owned by the authenticated user | +| `epic_iid` | integer | yes | The internal ID of the (future parent) epic. | +| `title` | string | yes | The title of a newly created epic. | + +```bash +curl --header POST "PRIVATE-TOKEN: <your_access_token>" https://gitlab.example.com/api/v4/groups/1/epics/5/epics?title=Newpic +``` + +Example response: + +```json +{ + "id": 24, + "iid": 2, + "title": "child epic", + "group_id": 49, + "parent_id": 23, + "has_children": false, + "has_issues": false, + "reference": "&2", + "url": "http://localhost/groups/group16/-/epics/2", + "relation_url": "http://localhost/groups/group16/-/epics/1/links/24" +} +``` + +## Re-order a child epic + +``` +PUT /groups/:id/epics/:epic_iid/epics/:child_epic_id +``` + +| Attribute | Type | Required | Description | +| ---------------- | -------------- | -------- | ------------------------------------------------------------------------------------------------------------------ | +| `id` | integer/string | yes | The ID or [URL-encoded path of the group](README.md#namespaced-path-encoding) owned by the authenticated user. | +| `epic_iid` | integer | yes | The internal ID of the epic. | +| `child_epic_id` | integer | yes | The global ID of the child epic. Internal ID can't be used because they can conflict with epics from other groups. | +| `move_before_id` | integer | no | The global ID of a sibling epic that should be placed before the child epic. | +| `move_after_id` | integer | no | The global ID of a sibling epic that should be placed after the child epic. | + +```bash +curl --header PUT "PRIVATE-TOKEN: <your_access_token>" https://gitlab.example.com/api/v4/groups/1/epics/4/epics/5 +``` + +Example response: + +```json +[ + { + "id": 29, + "iid": 6, + "group_id": 1, + "parent_id": 5, + "title": "Accusamus iste et ullam ratione voluptatem omnis debitis dolor est.", + "description": "Molestias dolorem eos vitae expedita impedit necessitatibus quo voluptatum.", + "author": { + "id": 10, + "name": "Lu Mayer", + "username": "kam", + "state": "active", + "avatar_url": "http://www.gravatar.com/avatar/018729e129a6f31c80a6327a30196823?s=80&d=identicon", + "web_url": "http://localhost:3001/kam" + }, + "start_date": null, + "start_date_is_fixed": false, + "start_date_fixed": null, + "start_date_from_milestones": null, + "end_date": "2018-07-31", + "due_date": "2018-07-31", + "due_date_is_fixed": false, + "due_date_fixed": null, + "due_date_from_milestones": "2018-07-31", + "created_at": "2018-07-17T13:36:22.770Z", + "updated_at": "2018-07-18T12:22:05.239Z", + "labels": [] + } +] +``` + +## Unassign a child epic + +Unassigns a child epic from a parent epic. + +``` +DELETE /groups/:id/epics/:epic_iid/epics/:child_epic_id +``` + +| Attribute | Type | Required | Description | +| --------------- | -------------- | -------- | ------------------------------------------------------------------------------------------------------------------ | +| `id` | integer/string | yes | The ID or [URL-encoded path of the group](README.md#namespaced-path-encoding) owned by the authenticated user. | +| `epic_iid` | integer | yes | The internal ID of the epic. | +| `child_epic_id` | integer | yes | The global ID of the child epic. Internal ID can't be used because they can conflict with epics from other groups. | + +```bash +curl --header DELETE "PRIVATE-TOKEN: <your_access_token>" https://gitlab.example.com/api/v4/groups/1/epics/4/epics/5 +``` + +Example response: + +```json +{ + "id": 5, + "iid": 38, + "group_id": 1, + "parent_id": null, + "title": "Accusamus iste et ullam ratione voluptatem omnis debitis dolor est.", + "description": "Molestias dolorem eos vitae expedita impedit necessitatibus quo voluptatum.", + "author": { + "id": 10, + "name": "Lu Mayer", + "username": "kam", + "state": "active", + "avatar_url": "http://www.gravatar.com/avatar/018729e129a6f31c80a6327a30196823?s=80&d=identicon", + "web_url": "http://localhost:3001/kam" + }, + "start_date": null, + "start_date_is_fixed": false, + "start_date_fixed": null, + "start_date_from_milestones": null, + "end_date": "2018-07-31", + "due_date": "2018-07-31", + "due_date_is_fixed": false, + "due_date_fixed": null, + "due_date_from_milestones": "2018-07-31", + "created_at": "2018-07-17T13:36:22.770Z", + "updated_at": "2018-07-18T12:22:05.239Z", + "labels": [] +} +``` diff --git a/doc/api/epics.md b/doc/api/epics.md new file mode 100644 index 00000000000..0541cfaa715 --- /dev/null +++ b/doc/api/epics.md @@ -0,0 +1,360 @@ +# Epics API **[ULTIMATE]** + +Every API call to epic must be authenticated. + +If a user is not a member of a group and the group is private, a `GET` request on that group will result to a `404` status code. + +If epics feature is not available a `403` status code will be returned. + +## Epic issues API + +The [epic issues API](epic_issues.md) allows you to interact with issues associated with an epic. + +# Milestone dates integration + +> [Introduced][ee-6448] in GitLab 11.3. + +Since start date and due date can be dynamically sourced from related issue milestones, when user has edit permission, additional fields will be shown. These include two boolean fields `start_date_is_fixed` and `due_date_is_fixed`, and four date fields `start_date_fixed`, `start_date_from_milestones`, `due_date_fixed` and `due_date_from_milestones`. + +`end_date` has been deprecated in favor of `due_date`. + +## Epics pagination + +By default, `GET` requests return 20 results at a time because the API results +are paginated. + +Read more on [pagination](README.md#pagination). + +## List epics for a group + +Gets all epics of the requested group and its subgroups. + +``` +GET /groups/:id/epics +GET /groups/:id/epics?author_id=5 +GET /groups/:id/epics?labels=bug,reproduced +GET /groups/:id/epics?state=opened +``` + +| Attribute | Type | Required | Description | +| ------------------- | ---------------- | ---------- | --------------------------------------------------------------------------------------------------------------------------- | +| `id` | integer/string | yes | The ID or [URL-encoded path of the group](README.md#namespaced-path-encoding) owned by the authenticated user | +| `author_id` | integer | no | Return epics created by the given user `id` | +| `labels` | string | no | Return epics matching a comma separated list of labels names. Label names from the epic group or a parent group can be used | +| `order_by` | string | no | Return epics ordered by `created_at` or `updated_at` fields. Default is `created_at` | +| `sort` | string | no | Return epics sorted in `asc` or `desc` order. Default is `desc` | +| `search` | string | no | Search epics against their `title` and `description` | +| `state` | string | no | Search epics against their `state`, possible filters: `opened`, `closed` and `all`, default: `all` | +| `created_after` | datetime | no | Return epics created on or after the given time | +| `created_before` | datetime | no | Return epics created on or before the given time | +| `updated_after` | datetime | no | Return epics updated on or after the given time | +| `updated_before` | datetime | no | Return epics updated on or before the given time | + +```bash +curl --header "PRIVATE-TOKEN: <your_access_token>" https://gitlab.example.com/api/v4/groups/1/epics +``` + +Example response: + +```json +[ + { + "id": 29, + "iid": 4, + "group_id": 7, + "title": "Accusamus iste et ullam ratione voluptatem omnis debitis dolor est.", + "description": "Molestias dolorem eos vitae expedita impedit necessitatibus quo voluptatum.", + "state": "opened", + "author": { + "id": 10, + "name": "Lu Mayer", + "username": "kam", + "state": "active", + "avatar_url": "http://www.gravatar.com/avatar/018729e129a6f31c80a6327a30196823?s=80&d=identicon", + "web_url": "http://localhost:3001/kam" + }, + "start_date": null, + "start_date_is_fixed": false, + "start_date_fixed": null, + "start_date_from_milestones": null, + "end_date": "2018-07-31", + "due_date": "2018-07-31", + "due_date_is_fixed": false, + "due_date_fixed": null, + "due_date_from_milestones": "2018-07-31", + "created_at": "2018-07-17T13:36:22.770Z", + "updated_at": "2018-07-18T12:22:05.239Z", + "labels": [], + "upvotes": 4, + "downvotes": 0 + } +] +``` + +## Single epic + +Gets a single epic + +``` +GET /groups/:id/epics/:epic_iid +``` + +| Attribute | Type | Required | Description | +| ------------------- | ---------------- | ---------- | ---------------------------------------------------------------------------------------| +| `id` | integer/string | yes | The ID or [URL-encoded path of the group](README.md#namespaced-path-encoding) owned by the authenticated user | +| `epic_iid` | integer/string | yes | The internal ID of the epic. | + +```bash +curl --header "PRIVATE-TOKEN: <your_access_token>" https://gitlab.example.com/api/v4/groups/1/epics/5 +``` + +Example response: + +```json +{ + "id": 30, + "iid": 5, + "group_id": 7, + "title": "Ea cupiditate dolores ut vero consequatur quasi veniam voluptatem et non.", + "description": "Molestias dolorem eos vitae expedita impedit necessitatibus quo voluptatum.", + "state": "opened", + "author":{ + "id": 7, + "name": "Pamella Huel", + "username": "arnita", + "state": "active", + "avatar_url": "http://www.gravatar.com/avatar/a2f5c6fcef64c9c69cb8779cb292be1b?s=80&d=identicon", + "web_url": "http://localhost:3001/arnita" + }, + "start_date": null, + "start_date_is_fixed": false, + "start_date_fixed": null, + "start_date_from_milestones": null, + "end_date": "2018-07-31", + "due_date": "2018-07-31", + "due_date_is_fixed": false, + "due_date_fixed": null, + "due_date_from_milestones": "2018-07-31", + "created_at": "2018-07-17T13:36:22.770Z", + "updated_at": "2018-07-18T12:22:05.239Z", + "labels": [], + "upvotes": 4, + "downvotes": 0 +} +``` + +## New epic + +Creates a new epic. + +NOTE: **Note:** +Starting with GitLab [11.3][ee-6448], `start_date` and `end_date` should no longer be assigned +directly, as they now represent composite values. You can configure it via the `*_is_fixed` and +`*_fixed` fields instead. + +``` +POST /groups/:id/epics +``` + +| Attribute | Type | Required | Description | +| ------------------- | ---------------- | ---------- | ---------------------------------------------------------------------------------------| +| `id` | integer/string | yes | The ID or [URL-encoded path of the group](README.md#namespaced-path-encoding) owned by the authenticated user | +| `title` | string | yes | The title of the epic | +| `labels` | string | no | The comma separated list of labels | +| `description` | string | no | The description of the epic | +| `start_date_is_fixed` | boolean | no | Whether start date should be sourced from `start_date_fixed` or from milestones (since 11.3) | +| `start_date_fixed` | string | no | The fixed start date of an epic (since 11.3) | +| `due_date_is_fixed` | boolean | no | Whether due date should be sourced from `due_date_fixed` or from milestones (since 11.3) | +| `due_date_fixed` | string | no | The fixed due date of an epic (since 11.3) | +| `parent_id` | integer/string | no | The id of a parent epic (since 11.11) | + +```bash +curl --header POST "PRIVATE-TOKEN: <your_access_token>" https://gitlab.example.com/api/v4/groups/1/epics?title=Epic&description=Epic%20description +``` + +Example response: + +```json +{ + "id": 33, + "iid": 6, + "group_id": 7, + "title": "Epic", + "description": "Epic description", + "state": "opened", + "author": { + "name" : "Alexandra Bashirian", + "avatar_url" : null, + "state" : "active", + "web_url" : "https://gitlab.example.com/eileen.lowe", + "id" : 18, + "username" : "eileen.lowe" + }, + "start_date": null, + "start_date_is_fixed": false, + "start_date_fixed": null, + "start_date_from_milestones": null, + "end_date": "2018-07-31", + "due_date": "2018-07-31", + "due_date_is_fixed": false, + "due_date_fixed": null, + "due_date_from_milestones": "2018-07-31", + "created_at": "2018-07-17T13:36:22.770Z", + "updated_at": "2018-07-18T12:22:05.239Z", + "labels": [], + "upvotes": 4, + "downvotes": 0 +} +``` + +## Update epic + +Updates an epic. + +NOTE: **Note:** +Starting with GitLab [11.3][ee-6448], `start_date` and `end_date` should no longer be assigned +directly, as they now represent composite values. You can configure it via the `*_is_fixed` and +`*_fixed` fields instead. + +``` +PUT /groups/:id/epics/:epic_iid +``` + +| Attribute | Type | Required | Description | +| ------------------- | ---------------- | ---------- | ---------------------------------------------------------------------------------------| +| `id` | integer/string | yes | The ID or [URL-encoded path of the group](README.md#namespaced-path-encoding) owned by the authenticated user | +| `epic_iid` | integer/string | yes | The internal ID of the epic | +| `title` | string | no | The title of an epic | +| `description` | string | no | The description of an epic | +| `labels` | string | no | The comma separated list of labels | +| `start_date_is_fixed` | boolean | no | Whether start date should be sourced from `start_date_fixed` or from milestones (since 11.3) | +| `start_date_fixed` | string | no | The fixed start date of an epic (since 11.3) | +| `due_date_is_fixed` | boolean | no | Whether due date should be sourced from `due_date_fixed` or from milestones (since 11.3) | +| `due_date_fixed` | string | no | The fixed due date of an epic (since 11.3) | +| `state_event` | string | no | State event for an epic. Set `close` to close the epic and `reopen` to reopen it (since 11.4) | + +```bash +curl --header PUT "PRIVATE-TOKEN: <your_access_token>" https://gitlab.example.com/api/v4/groups/1/epics/5?title=New%20Title +``` + +Example response: + +```json +{ + "id": 33, + "iid": 6, + "group_id": 7, + "title": "New Title", + "description": "Epic description", + "state": "opened", + "author": { + "name" : "Alexandra Bashirian", + "avatar_url" : null, + "state" : "active", + "web_url" : "https://gitlab.example.com/eileen.lowe", + "id" : 18, + "username" : "eileen.lowe" + }, + "start_date": null, + "start_date_is_fixed": false, + "start_date_fixed": null, + "start_date_from_milestones": null, + "end_date": "2018-07-31", + "due_date": "2018-07-31", + "due_date_is_fixed": false, + "due_date_fixed": null, + "due_date_from_milestones": "2018-07-31", + "created_at": "2018-07-17T13:36:22.770Z", + "updated_at": "2018-07-18T12:22:05.239Z", + "labels": [], + "upvotes": 4, + "downvotes": 0 +} +``` + +## Delete epic + +Deletes an epic + +``` +DELETE /groups/:id/epics/:epic_iid +``` + +| Attribute | Type | Required | Description | +| ------------------- | ---------------- | ---------- | ---------------------------------------------------------------------------------------| +| `id` | integer/string | yes | The ID or [URL-encoded path of the group](README.md#namespaced-path-encoding) owned by the authenticated user | +| `epic_iid` | integer/string | yes | The internal ID of the epic. | + +```bash +curl --header DELETE "PRIVATE-TOKEN: <your_access_token>" https://gitlab.example.com/api/v4/groups/1/epics/5 +``` + +## Create a todo + +Manually creates a todo for the current user on an epic. If +there already exists a todo for the user on that epic, status code `304` is +returned. + +``` +POST /groups/:id/epics/:epic_iid/todo +``` + +| Attribute | Type | Required | Description | +|-------------|---------|----------|--------------------------------------| +| `id` | integer/string | yes | The ID or [URL-encoded path of the group](README.md#namespaced-path-encoding) owned by the authenticated user | +| `epic_iid ` | integer | yes | The internal ID of a group's epic | + +```bash +curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" https://gitlab.example.com/api/v4/groups/1/epics/5/todo +``` + +Example response: + +```json +{ + "id": 112, + "group": { + "id": 1, + "name": "Gitlab", + "path": "gitlab", + "kind": "group", + "full_path": "base/gitlab", + "parent_id": null + }, + "author": { + "name": "Administrator", + "username": "root", + "id": 1, + "state": "active", + "avatar_url": "http://www.gravatar.com/avatar/e64c7d89f26bd1972efa854d13d7dd61?s=80&d=identicon", + "web_url": "https://gitlab.example.com/root" + }, + "action_name": "marked", + "target_type": "epic", + "target": { + "id": 30, + "iid": 5, + "group_id": 1, + "title": "Ea cupiditate dolores ut vero consequatur quasi veniam voluptatem et non.", + "description": "Molestias dolorem eos vitae expedita impedit necessitatibus quo voluptatum.", + "author":{ + "id": 7, + "name": "Pamella Huel", + "username": "arnita", + "state": "active", + "avatar_url": "http://www.gravatar.com/avatar/a2f5c6fcef64c9c69cb8779cb292be1b?s=80&d=identicon", + "web_url": "http://localhost:3001/arnita" + }, + "start_date": null, + "end_date": null, + "created_at": "2018-01-21T06:21:13.165Z", + "updated_at": "2018-01-22T12:41:41.166Z" + }, + "target_url": "https://gitlab.example.com/groups/epics/5", + "body": "Vel voluptas atque dicta mollitia adipisci qui at.", + "state": "pending", + "created_at": "2016-07-01T11:09:13.992Z" +} +``` + +[ee-6448]: https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/6448 diff --git a/doc/api/geo_nodes.md b/doc/api/geo_nodes.md new file mode 100644 index 00000000000..ea31abdd87e --- /dev/null +++ b/doc/api/geo_nodes.md @@ -0,0 +1,404 @@ +# Geo Nodes API **[PREMIUM ONLY]** + +In order to interact with Geo node endpoints, you need to authenticate yourself +as an admin. + +## Retrieve configuration about all Geo nodes + +``` +GET /geo_nodes +``` + +```bash +curl --header "PRIVATE-TOKEN: <your_access_token>" https://gitlab.example.com/api/v4/geo_nodes +``` + +Example response: + +```json +[ + { + "id": 1, + "name": "us-node", + "url": "https://primary.example.com/", + "internal_url": "https://internal.example.com/", + "primary": true, + "enabled": true, + "current": true, + "files_max_capacity": 10, + "repos_max_capacity": 25, + "verification_max_capacity": 100, + "clone_protocol": "http" + }, + { + "id": 2, + "name": "cn-node", + "url": "https://secondary.example.com/", + "internal_url": "https://secondary.example.com/", + "primary": false, + "enabled": true, + "current": false, + "files_max_capacity": 10, + "repos_max_capacity": 25, + "verification_max_capacity": 100, + "clone_protocol": "http" + } +] +``` + +## Retrieve configuration about a specific Geo node + +``` +GET /geo_nodes/:id +``` + +```bash +curl --header "PRIVATE-TOKEN: <your_access_token>" https://gitlab.example.com/api/v4/geo_nodes/1 +``` + +Example response: + +```json +{ + "id": 1, + "name": "us-node", + "url": "https://primary.example.com/", + "internal_url": "https://primary.example.com/", + "primary": true, + "enabled": true, + "current": true, + "files_max_capacity": 10, + "repos_max_capacity": 25, + "verification_max_capacity": 100, + "clone_protocol": "http" +} +``` + +## Edit a Geo node + +Updates settings of an existing Geo node. + +_This can only be run against a primary Geo node._ + +``` +PUT /geo_nodes/:id +``` + +| Attribute | Type | Required | Description | +|----------------------|---------|-----------|---------------------------------------------------------------------------| +| `id` | integer | yes | The ID of the Geo node. | +| `enabled` | boolean | no | Flag indicating if the Geo node is enabled. | +| `name` | string | yes | The unique identifier for the Geo node. Must match `geo_node_name` if it is set in gitlab.rb, otherwise it must match `external_url`. | +| `url` | string | yes | The user-facing URL of the Geo node. | +| `internal_url` | string | no | The URL defined on the primary node that secondary nodes should use to contact it. Returns `url` if not set.| +| `files_max_capacity` | integer | no | Control the maximum concurrency of LFS/attachment backfill for this secondary node. | +| `repos_max_capacity` | integer | no | Control the maximum concurrency of repository backfill for this secondary node. | +| `verification_max_capacity` | integer | no | Control the maximum concurrency of verification for this node. | + +Example response: + +```json +{ + "id": 1, + "name": "cn-node", + "url": "https://secondary.example.com/", + "internal_url": "https://secondary.example.com/", + "primary": false, + "enabled": true, + "current": true, + "files_max_capacity": 10, + "repos_max_capacity": 25, + "verification_max_capacity": 100, + "clone_protocol": "http" +} +``` + +## Delete a Geo node + +Removes the Geo node. + +NOTE: **Note:** +Only a Geo primary node will accept this request. + +``` +DELETE /geo_nodes/:id +``` + +| Attribute | Type | Required | Description | +|-----------|---------|----------|-------------------------| +| `id` | integer | yes | The ID of the Geo node. | + +## Repair a Geo node + +To repair the OAuth authentication of a Geo node. + +_This can only be run against a primary Geo node._ + +``` +POST /geo_nodes/:id/repair +``` + +Example response: + +```json +{ + "id": 1, + "name": "us-node", + "url": "https://primary.example.com/", + "internal_url": "https://primary.example.com/", + "primary": true, + "enabled": true, + "current": true, + "files_max_capacity": 10, + "repos_max_capacity": 25, + "verification_max_capacity": 100, + "clone_protocol": "http" +} +``` + +## Retrieve status about all Geo nodes + +``` +GET /geo_nodes/status +``` + +```bash +curl --header "PRIVATE-TOKEN: <your_access_token>" https://gitlab.example.com/api/v4/geo_nodes/status +``` + +Example response: + +```json +[ + { + "geo_node_id": 1, + "healthy": true, + "health": "Healthy", + "health_status": "Healthy", + "missing_oauth_application": false, + "attachments_count": 1, + "attachments_synced_count": nil, + "attachments_failed_count": nil, + "attachments_synced_missing_on_primary_count": 0, + "attachments_synced_in_percentage": "0.00%", + "db_replication_lag_seconds": nil, + "lfs_objects_count": 0, + "lfs_objects_synced_count": nil, + "lfs_objects_failed_count": nil, + "lfs_objects_synced_missing_on_primary_count": 0, + "lfs_objects_synced_in_percentage": "0.00%", + "job_artifacts_count": 2, + "job_artifacts_synced_count": nil, + "job_artifacts_failed_count": nil, + "job_artifacts_synced_missing_on_primary_count": 0, + "job_artifacts_synced_in_percentage": "0.00%", + "projects_count": 41, + "repositories_failed_count": nil, + "repositories_synced_count": nil, + "repositories_synced_in_percentage": "0.00%", + "wikis_failed_count": nil, + "wikis_synced_count": nil, + "wikis_synced_in_percentage": "0.00%", + "replication_slots_count": 1, + "replication_slots_used_count": 1, + "replication_slots_used_in_percentage": "100.00%", + "replication_slots_max_retained_wal_bytes": 0, + "repositories_checked_count": 20, + "repositories_checked_failed_count": 20, + "repositories_checked_in_percentage": "100.00%", + "repositories_checksummed_count": 20, + "repositories_checksum_failed_count": 5, + "repositories_checksummed_in_percentage": "48.78%", + "wikis_checksummed_count": 10, + "wikis_checksum_failed_count": 3, + "wikis_checksummed_in_percentage": "24.39%", + "repositories_verified_count": 20, + "repositories_verification_failed_count": 5, + "repositories_verified_in_percentage": "48.78%", + "repositories_checksum_mismatch_count": 3, + "wikis_verified_count": 10, + "wikis_verification_failed_count": 3, + "wikis_verified_in_percentage": "24.39%", + "wikis_checksum_mismatch_count": 1, + "repositories_retrying_verification_count": 1, + "wikis_retrying_verification_count": 3, + "repositories_checked_count": 7, + "repositories_checked_failed_count": 2, + "repositories_checked_in_percentage": "17.07%", + "last_event_id": 23, + "last_event_timestamp": 1509681166, + "cursor_last_event_id": nil, + "cursor_last_event_timestamp": 0, + "last_successful_status_check_timestamp": 1510125024, + "version": "10.3.0", + "revision": "33d33a096a", + }, + { + "geo_node_id": 2, + "healthy": true, + "health": "Healthy", + "health_status": "Healthy", + "missing_oauth_application": false, + "attachments_count": 1, + "attachments_synced_count": 1, + "attachments_failed_count": 0, + "attachments_synced_missing_on_primary_count": 0, + "attachments_synced_in_percentage": "100.00%", + "db_replication_lag_seconds": 0, + "lfs_objects_count": 0, + "lfs_objects_synced_count": 0, + "lfs_objects_failed_count": 0, + "lfs_objects_synced_missing_on_primary_count": 0, + "lfs_objects_synced_in_percentage": "0.00%", + "job_artifacts_count": 2, + "job_artifacts_synced_count": 1, + "job_artifacts_failed_count": 1, + "job_artifacts_synced_missing_on_primary_count": 0, + "job_artifacts_synced_in_percentage": "50.00%", + "projects_count": 41, + "repositories_failed_count": 1, + "repositories_synced_count": 40, + "repositories_synced_in_percentage": "97.56%", + "wikis_failed_count": 0, + "wikis_synced_count": 41, + "wikis_synced_in_percentage": "100.00%", + "replication_slots_count": nil, + "replication_slots_used_count": nil, + "replication_slots_used_in_percentage": "0.00%", + "replication_slots_max_retained_wal_bytes": nil, + "repositories_checksummed_count": 20, + "repositories_checksum_failed_count": 5, + "repositories_checksummed_in_percentage": "48.78%", + "wikis_checksummed_count": 10, + "wikis_checksum_failed_count": 3, + "wikis_checksummed_in_percentage": "24.39%", + "repositories_verified_count": 20, + "repositories_verification_failed_count": 5, + "repositories_verified_in_percentage": "48.78%", + "repositories_checksum_mismatch_count": 3, + "wikis_verified_count": 10, + "wikis_verification_failed_count": 3, + "wikis_verified_in_percentage": "24.39%", + "wikis_checksum_mismatch_count": 1, + "repositories_retrying_verification_count": 4, + "wikis_retrying_verification_count": 2, + "repositories_checked_count": 5, + "repositories_checked_failed_count": 1, + "repositories_checked_in_percentage": "12.20%", + "last_event_id": 23, + "last_event_timestamp": 1509681166, + "cursor_last_event_id": 23, + "cursor_last_event_timestamp": 1509681166, + "last_successful_status_check_timestamp": 1510125024, + "version": "10.3.0", + "revision": "33d33a096a" + } +] +``` + +NOTE: **Note:** +In GitLab 12.0, deprecated fields `wikis_count` and `repositories_count` were removed. Use `projects_count` instead. + +## Retrieve status about a specific Geo node + +``` +GET /geo_nodes/:id/status +``` + +```bash +curl --header "PRIVATE-TOKEN: <your_access_token>" https://gitlab.example.com/api/v4/geo_nodes/2/status +``` + +Example response: + +```json +{ + "geo_node_id": 2, + "healthy": true, + "health": "Healthy", + "health_status": "Healthy", + "missing_oauth_application": false, + "attachments_count": 1, + "attachments_synced_count": 1, + "attachments_failed_count": 0, + "attachments_synced_missing_on_primary_count": 0, + "attachments_synced_in_percentage": "100.00%", + "db_replication_lag_seconds": 0, + "lfs_objects_count": 0, + "lfs_objects_synced_count": 0, + "lfs_objects_failed_count": 0, + "lfs_objects_synced_missing_on_primary_count": 0, + "lfs_objects_synced_in_percentage": "0.00%", + "job_artifacts_count": 2, + "job_artifacts_synced_count": 1, + "job_artifacts_failed_count": 1, + "job_artifacts_synced_missing_on_primary_count": 0, + "job_artifacts_synced_in_percentage": "50.00%", + "projects_count": 41, + "repositories_failed_count": 1, + "repositories_synced_count": 40, + "repositories_synced_in_percentage": "97.56%", + "wikis_failed_count": 0, + "wikis_synced_count": 41, + "wikis_synced_in_percentage": "100.00%", + "replication_slots_count": nil, + "replication_slots_used_count": nil, + "replication_slots_used_in_percentage": "0.00%", + "replication_slots_max_retained_wal_bytes": nil, + "last_event_id": 23, + "last_event_timestamp": 1509681166, + "cursor_last_event_id": 23, + "cursor_last_event_timestamp": 1509681166, + "last_successful_status_check_timestamp": 1510125268, + "version": "10.3.0", + "revision": "33d33a096a" +} +``` + +Note: The `health_status` parameter can only be in an "Healthy" or "Unhealthy" state, while the `health` parameter can be empty, "Healthy", or contain the actual error message. + +NOTE: **Note:** +In GitLab 12.0, deprecated fields `wikis_count` and `repositories_count` were removed. Use `projects_count` instead. + +## Retrieve project sync or verification failures that occurred on the current node + +This only works on a secondary node. + +``` +GET /geo_nodes/current/failures +``` + +| Attribute | Type | Required | Description | +| --------- | ---- | -------- | ----------- | +| `type` | string | no | Type of failed objects (`repository`/`wiki`) | +| `failure_type` | string | no | Type of failures (`sync`/`checksum_mismatch`/`verification`) | + +This endpoint uses [Pagination](README.md#pagination). + +```bash +curl --header "PRIVATE-TOKEN: <your_access_token>" https://gitlab.example.com/api/v4/geo_nodes/current/failures +``` + +Example response: + +```json +[ + { + "project_id": 3, + "last_repository_synced_at": "2017-10-31 14:25:55 UTC", + "last_repository_successful_sync_at": "2017-10-31 14:26:04 UTC", + "last_wiki_synced_at": "2017-10-31 14:26:04 UTC", + "last_wiki_successful_sync_at": "2017-10-31 14:26:11 UTC", + "repository_retry_count": null, + "wiki_retry_count": 1, + "last_repository_sync_failure": null, + "last_wiki_sync_failure": "Error syncing Wiki repository", + "last_repository_verification_failure": "", + "last_wiki_verification_failure": "", + "repository_verification_checksum_sha": "da39a3ee5e6b4b0d32e5bfef9a601890afd80709", + "wiki_verification_checksum_sha": "da39a3ee5e6b4b0d3255bfef9ef0189aafd80709", + "repository_checksum_mismatch": false, + "wiki_checksum_mismatch": false + } +] +``` diff --git a/doc/api/graphql/index.md b/doc/api/graphql/index.md index ec48bf4940b..88e657a5d2f 100644 --- a/doc/api/graphql/index.md +++ b/doc/api/graphql/index.md @@ -16,6 +16,20 @@ added to the API without creating breaking changes. This allows us to have a versionless API as described in [the GraphQL documentation](https://graphql.org/learn/best-practices/#versioning). +## Vision + +We want the GraphQL API to be the **primary** means of interacting +programmatically with GitLab. To achieve this, it needs full coverage - anything +possible in the REST API should also be possible in the GraphQL API. + +To help us meet this vision, the frontend should use GraphQL in preference to +the REST API for new features, although the alpha status of GraphQL may prevent +this from being a possibility at times. + +There are no plans to deprecate the REST API. To reduce the technical burden of +supporting two APIs in parallel, they should share implementations as much as +possible. + ## Enabling the GraphQL feature The GraphQL API itself is currently in Alpha, and therefore hidden behind a @@ -29,7 +43,19 @@ curl --data "value=100" --header "PRIVATE-TOKEN: <your_access_token>" https://gi ## Available queries -A first iteration of a GraphQL API includes a query for: `project`. Within a project it is also possible to fetch a `mergeRequest` by IID. +A first iteration of a GraphQL API includes the following queries + +1. `project` : Within a project it is also possible to fetch a `mergeRequest` by IID. +1. `group` : Only basic group information is currently supported. +1. `namespace` : Within a namespace it is also possible to fetch `projects`. + +### Multiplex queries + +GitLab supports batching queries into a single request using +[apollo-link-batch-http](https://www.apollographql.com/docs/link/links/batch-http). More +info about multiplexed queries is also available for +[graphql-ruby](https://graphql-ruby.org/queries/multiplex.html) the +library GitLab uses on the backend. ## GraphiQL diff --git a/doc/api/group_level_variables.md b/doc/api/group_level_variables.md index 3551bfa3f8b..7b00df6d775 100644 --- a/doc/api/group_level_variables.md +++ b/doc/api/group_level_variables.md @@ -22,10 +22,12 @@ curl --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/a [ { "key": "TEST_VARIABLE_1", + "variable_type": "env_var", "value": "TEST_1" }, { "key": "TEST_VARIABLE_2", + "variable_type": "env_var", "value": "TEST_2" } ] @@ -51,6 +53,7 @@ curl --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/a ```json { "key": "TEST_VARIABLE_1", + "variable_type": "env_var", "value": "TEST_1" } ``` @@ -63,12 +66,13 @@ Create a new variable. POST /groups/:id/variables ``` -| Attribute | Type | required | Description | -|-------------|---------|----------|-----------------------| -| `id` | integer/string | yes | The ID of a group or [URL-encoded path of the group](README.md#namespaced-path-encoding) owned by the authenticated user | -| `key` | string | yes | The `key` of a variable; must have no more than 255 characters; only `A-Z`, `a-z`, `0-9`, and `_` are allowed | -| `value` | string | yes | The `value` of a variable | -| `protected` | boolean | no | Whether the variable is protected | +| Attribute | Type | required | Description | +|-----------------|---------|----------|-----------------------| +| `id` | integer/string | yes | The ID of a group or [URL-encoded path of the group](README.md#namespaced-path-encoding) owned by the authenticated user | +| `key` | string | yes | The `key` of a variable; must have no more than 255 characters; only `A-Z`, `a-z`, `0-9`, and `_` are allowed | +| `value` | string | yes | The `value` of a variable | +| `variable_type` | string | no | The type of a variable. Available types are: `env_var` (default) and `file` | +| `protected` | boolean | no | Whether the variable is protected | ``` curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/groups/1/variables" --form "key=NEW_VARIABLE" --form "value=new value" @@ -78,6 +82,7 @@ curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" "https://gitla { "key": "NEW_VARIABLE", "value": "new value", + "variable_type": "env_var", "protected": false } ``` @@ -90,12 +95,13 @@ Update a group's variable. PUT /groups/:id/variables/:key ``` -| Attribute | Type | required | Description | -|-------------|---------|----------|-------------------------| -| `id` | integer/string | yes | The ID of a group or [URL-encoded path of the group](README.md#namespaced-path-encoding) owned by the authenticated user | -| `key` | string | yes | The `key` of a variable | -| `value` | string | yes | The `value` of a variable | -| `protected` | boolean | no | Whether the variable is protected | +| Attribute | Type | required | Description | +|-----------------|---------|----------|-------------------------| +| `id` | integer/string | yes | The ID of a group or [URL-encoded path of the group](README.md#namespaced-path-encoding) owned by the authenticated user | +| `key` | string | yes | The `key` of a variable | +| `value` | string | yes | The `value` of a variable | +| `variable_type` | string | no | The type of a variable. Available types are: `env_var` (default) and `file` | +| `protected` | boolean | no | Whether the variable is protected | ``` curl --request PUT --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/groups/1/variables/NEW_VARIABLE" --form "value=updated value" @@ -105,6 +111,7 @@ curl --request PUT --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab { "key": "NEW_VARIABLE", "value": "updated value", + "variable_type": "env_var", "protected": true } ``` diff --git a/doc/api/group_milestones.md b/doc/api/group_milestones.md index eb974267084..260eb09cc38 100644 --- a/doc/api/group_milestones.md +++ b/doc/api/group_milestones.md @@ -1,6 +1,5 @@ # Group milestones API -> **Notes:** > [Introduced][ce-12819] in GitLab 9.5. ## List group milestones @@ -13,6 +12,7 @@ GET /groups/:id/milestones?iids[]=42 GET /groups/:id/milestones?iids[]=42&iids[]=43 GET /groups/:id/milestones?state=active GET /groups/:id/milestones?state=closed +GET /groups/:id/milestones?title=1.0 GET /groups/:id/milestones?search=version ``` @@ -23,6 +23,7 @@ Parameters: | `id` | integer/string | yes | The ID or [URL-encoded path of the group](README.md#namespaced-path-encoding) owned by the authenticated user | | `iids[]` | Array[integer] | optional | Return only the milestones having the given `iid` | | `state` | string | optional | Return only `active` or `closed` milestones | +| `title` | string | optional | Return only the milestones having the given `title` | | `search` | string | optional | Return only milestones with a title or description matching the provided string | ```bash diff --git a/doc/api/groups.md b/doc/api/groups.md index 907b443d355..20789a1d4a4 100644 --- a/doc/api/groups.md +++ b/doc/api/groups.md @@ -68,6 +68,7 @@ GET /groups?statistics=true "statistics": { "storage_size" : 212, "repository_size" : 33, + "wiki_size" : 100, "lfs_objects_size" : 123, "job_artifacts_size" : 57 diff --git a/doc/api/issue_links.md b/doc/api/issue_links.md new file mode 100644 index 00000000000..1c7db6a8e4c --- /dev/null +++ b/doc/api/issue_links.md @@ -0,0 +1,215 @@ +# Issue links API **[STARTER]** + +## List issue relations + +Get a list of related issues of a given issue, sorted by the relationship creation datetime (ascending). +Issues will be filtered according to the user authorizations. + +``` +GET /projects/:id/issues/:issue_iid/links +``` + +Parameters: + +| Attribute | Type | Required | Description | +|-------------|---------|----------|--------------------------------------| +| `id` | integer/string | yes | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding) owned by the authenticated user | +| `issue_iid` | integer | yes | The internal ID of a project's issue | + +```json +[ + { + "id" : 84, + "iid" : 14, + "issue_link_id": 1 + "project_id" : 4, + "created_at" : "2016-01-07T12:44:33.959Z", + "title" : "Issues with auth", + "state" : "opened", + "assignees" : [], + "assignee" : null, + "labels" : [ + "bug" + ], + "author" : { + "name" : "Alexandra Bashirian", + "avatar_url" : null, + "state" : "active", + "web_url" : "https://gitlab.example.com/eileen.lowe", + "id" : 18, + "username" : "eileen.lowe" + }, + "description" : null, + "updated_at" : "2016-01-07T12:44:33.959Z", + "milestone" : null, + "subscribed" : true, + "user_notes_count": 0, + "due_date": null, + "web_url": "http://example.com/example/example/issues/14", + "confidential": false, + "weight": null, + } +] +``` + +## Create an issue link + +Creates a two-way relation between two issues. User must be allowed to update both issues in order to succeed. + +``` +POST /projects/:id/issues/:issue_iid/links +``` + +| Attribute | Type | Required | Description | +|-------------|---------|----------|--------------------------------------| +| `id` | integer/string | yes | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding) owned by the authenticated user | +| `issue_iid` | integer | yes | The internal ID of a project's issue | +| `target_project_id` | integer/string | yes | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding) of a target project | +| `target_issue_iid` | integer/string | yes | The internal ID of a target project's issue | + + +```json +{ + "source_issue" : { + "id" : 83, + "iid" : 11, + "project_id" : 4, + "created_at" : "2016-01-07T12:44:33.959Z", + "title" : "Issues with auth", + "state" : "opened", + "assignees" : [], + "assignee" : null, + "labels" : [ + "bug" + ], + "author" : { + "name" : "Alexandra Bashirian", + "avatar_url" : null, + "state" : "active", + "web_url" : "https://gitlab.example.com/eileen.lowe", + "id" : 18, + "username" : "eileen.lowe" + }, + "description" : null, + "updated_at" : "2016-01-07T12:44:33.959Z", + "milestone" : null, + "subscribed" : true, + "user_notes_count": 0, + "due_date": null, + "web_url": "http://example.com/example/example/issues/11", + "confidential": false, + "weight": null, + }, + "target_issue" : { + "id" : 84, + "iid" : 14, + "project_id" : 4, + "created_at" : "2016-01-07T12:44:33.959Z", + "title" : "Issues with auth", + "state" : "opened", + "assignees" : [], + "assignee" : null, + "labels" : [ + "bug" + ], + "author" : { + "name" : "Alexandra Bashirian", + "avatar_url" : null, + "state" : "active", + "web_url" : "https://gitlab.example.com/eileen.lowe", + "id" : 18, + "username" : "eileen.lowe" + }, + "description" : null, + "updated_at" : "2016-01-07T12:44:33.959Z", + "milestone" : null, + "subscribed" : true, + "user_notes_count": 0, + "due_date": null, + "web_url": "http://example.com/example/example/issues/14", + "confidential": false, + "weight": null, + } +} +``` + +## Delete an issue link + +Deletes an issue link, thus removes the two-way relationship. + +``` +DELETE /projects/:id/issues/:issue_iid/links/:issue_link_id +``` + + +| Attribute | Type | Required | Description | +|-------------|---------|----------|--------------------------------------| +| `id` | integer/string | yes | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding) owned by the authenticated user | +| `issue_iid` | integer | yes | The internal ID of a project's issue | +| `issue_link_id` | integer/string | yes | The ID of an issue relationship | + + +```json +{ + "source_issue" : { + "id" : 83, + "iid" : 11, + "project_id" : 4, + "created_at" : "2016-01-07T12:44:33.959Z", + "title" : "Issues with auth", + "state" : "opened", + "assignees" : [], + "assignee" : null, + "labels" : [ + "bug" + ], + "author" : { + "name" : "Alexandra Bashirian", + "avatar_url" : null, + "state" : "active", + "web_url" : "https://gitlab.example.com/eileen.lowe", + "id" : 18, + "username" : "eileen.lowe" + }, + "description" : null, + "updated_at" : "2016-01-07T12:44:33.959Z", + "milestone" : null, + "subscribed" : true, + "user_notes_count": 0, + "due_date": null, + "web_url": "http://example.com/example/example/issues/11", + "confidential": false, + "weight": null, + }, + "target_issue" : { + "id" : 84, + "iid" : 14, + "project_id" : 4, + "created_at" : "2016-01-07T12:44:33.959Z", + "title" : "Issues with auth", + "state" : "opened", + "assignees" : [], + "assignee" : null, + "labels" : [ + "bug" + ], + "author" : { + "name" : "Alexandra Bashirian", + "avatar_url" : null, + "state" : "active", + "web_url" : "https://gitlab.example.com/eileen.lowe", + "id" : 18, + "username" : "eileen.lowe" + }, + "description" : null, + "updated_at" : "2016-01-07T12:44:33.959Z", + "milestone" : null, + "subscribed" : true, + "user_notes_count": 0, + "due_date": null, + "web_url": "http://example.com/example/example/issues/14", + "confidential": false, + "weight": null, + } +} +``` diff --git a/doc/api/issues.md b/doc/api/issues.md index d8b2ff07e30..0d96cfa1b21 100644 --- a/doc/api/issues.md +++ b/doc/api/issues.md @@ -37,23 +37,26 @@ GET /issues?confidential=true | Attribute | Type | Required | Description | | ------------------- | ---------------- | ---------- | --------------------------------------------------------------------------------------------------------------------------------------------------- | -| `state` | string | no | Return all issues or just those that are `opened` or `closed` | +| `state` | string | no | Return `all` issues or just those that are `opened` or `closed` | | `labels` | string | no | Comma-separated list of label names, issues must have all labels to be returned. `None` lists all issues with no labels. `Any` lists all issues with at least one label. `No+Label` (Deprecated) lists all issues with no labels. Predefined names are case-insensitive. | +| `with_labels_details`| Boolean | no | If `true`, response will return more details for each label in labels field: `:name`, `:color`, `:description`, `:text_color`. Default is `false`. | | `milestone` | string | no | The milestone title. `None` lists all issues with no milestone. `Any` lists all issues that have an assigned milestone. | | `scope` | string | no | Return issues for the given scope: `created_by_me`, `assigned_to_me` or `all`. Defaults to `created_by_me`<br> For versions before 11.0, use the now deprecated `created-by-me` or `assigned-to-me` scopes instead.<br> _([Introduced][ce-13004] in GitLab 9.5. [Changed to snake_case][ce-18935] in GitLab 11.0)_ | -| `author_id` | integer | no | Return issues created by the given user `id`. Combine with `scope=all` or `scope=assigned_to_me`. _([Introduced][ce-13004] in GitLab 9.5)_ | -| `assignee_id` | integer | no | Return issues assigned to the given user `id`. `None` returns unassigned issues. `Any` returns issues with an assignee. _([Introduced][ce-13004] in GitLab 9.5)_ | +| `author_id` | integer | no | Return issues created by the given user `id`. Mutually exclusive with `author_username`. Combine with `scope=all` or `scope=assigned_to_me`. _([Introduced][ce-13004] in GitLab 9.5)_ | +| `author_username` | string | no | Return issues created by the given `username`. Simillar to `author_id` and mutually exclusive with `author_id`. | +| `assignee_id` | integer | no | Return issues assigned to the given user `id`. Mutually exclusive with `assignee_username`. `None` returns unassigned issues. `Any` returns issues with an assignee. _([Introduced][ce-13004] in GitLab 9.5)_ | +| `assignee_username` | Array[String] | no | Return issues assigned to the given `username`. Simillar to `assignee_id` and mutually exclusive with `assignee_id`. In CE version `assignee_username` array should only contain a single value or an invalid param error will be returned otherwise. | | `my_reaction_emoji` | string | no | Return issues reacted by the authenticated user by the given `emoji`. `None` returns issues not given a reaction. `Any` returns issues given at least one reaction. _([Introduced][ce-14016] in GitLab 10.0)_ | | `iids[]` | Array[integer] | no | Return only the issues having the given `iid` | | `order_by` | string | no | Return issues ordered by `created_at` or `updated_at` fields. Default is `created_at` | | `sort` | string | no | Return issues sorted in `asc` or `desc` order. Default is `desc` | | `search` | string | no | Search issues against their `title` and `description` | -| `in` | string | no | Modify the scope of the `search` attribute. `title`, `description`, or a string joining them with comma. Default is `title,description` | +| `in` | string | no | Modify the scope of the `search` attribute. `title`, `description`, or a string joining them with comma. Default is `title,description` | | `created_after` | datetime | no | Return issues created on or after the given time | | `created_before` | datetime | no | Return issues created on or before the given time | | `updated_after` | datetime | no | Return issues updated on or after the given time | | `updated_before` | datetime | no | Return issues updated on or before the given time | -| `confidential ` | Boolean | no | Filter confidential or public issues. | +| `confidential ` | Boolean | no | Filter confidential or public issues. | ```bash curl --header "PRIVATE-TOKEN: <your_access_token>" https://gitlab.example.com/api/v4/issues @@ -109,7 +112,7 @@ Example response: "title" : "Consequatur vero maxime deserunt laboriosam est voluptas dolorem.", "created_at" : "2016-01-04T15:31:51.081Z", "iid" : 6, - "labels" : [], + "labels" : ["foo", "bar"], "upvotes": 4, "downvotes": 0, "merge_requests_count": 0, @@ -122,8 +125,21 @@ Example response: "human_time_estimate": null, "human_total_time_spent": null }, + "has_tasks": true, + "task_status": "10 of 15 tasks completed", "confidential": false, - "discussion_locked": false + "discussion_locked": false, + "_links":{ + "self":"http://example.com/api/v4/projects/1/issues/76", + "notes":"`http://example.com/`api/v4/projects/1/issues/76/notes", + "award_emoji":"http://example.com/api/v4/projects/1/issues/76/award_emoji", + "project":"http://example.com/api/v4/projects/1" + }, + "subscribed": false, + "task_completion_status":{ + "count":0, + "completed_count":0 + } } ] ``` @@ -158,11 +174,14 @@ GET /groups/:id/issues?confidential=true | `id` | integer/string | yes | The ID or [URL-encoded path of the group](README.md#namespaced-path-encoding) owned by the authenticated user | | `state` | string | no | Return all issues or just those that are `opened` or `closed` | | `labels` | string | no | Comma-separated list of label names, issues must have all labels to be returned. `None` lists all issues with no labels. `Any` lists all issues with at least one label. `No+Label` (Deprecated) lists all issues with no labels. Predefined names are case-insensitive. | +| `with_labels_details`| Boolean | no | If `true`, response will return more details for each label in labels field: `:name`, `:color`, `:description`, `:text_color`. Default is `false`. | | `iids[]` | Array[integer] | no | Return only the issues having the given `iid` | | `milestone` | string | no | The milestone title. `None` lists all issues with no milestone. `Any` lists all issues that have an assigned milestone. | | `scope` | string | no | Return issues for the given scope: `created_by_me`, `assigned_to_me` or `all`.<br> For versions before 11.0, use the now deprecated `created-by-me` or `assigned-to-me` scopes instead.<br> _([Introduced][ce-13004] in GitLab 9.5. [Changed to snake_case][ce-18935] in GitLab 11.0)_ | -| `author_id` | integer | no | Return issues created by the given user `id` _([Introduced][ce-13004] in GitLab 9.5)_ | -| `assignee_id` | integer | no | Return issues assigned to the given user `id`. `None` returns unassigned issues. `Any` returns issues with an assignee. _([Introduced][ce-13004] in GitLab 9.5)_ | +| `author_id` | integer | no | Return issues created by the given user `id`. Mutually exclusive with `author_username`. Combine with `scope=all` or `scope=assigned_to_me`. _([Introduced][ce-13004] in GitLab 9.5)_ | +| `author_username` | string | no | Return issues created by the given `username`. Simillar to `author_id` and mutually exclusive with `author_id`. | +| `assignee_id` | integer | no | Return issues assigned to the given user `id`. Mutually exclusive with `assignee_username`. `None` returns unassigned issues. `Any` returns issues with an assignee. _([Introduced][ce-13004] in GitLab 9.5)_ | +| `assignee_username` | Array[String] | no | Return issues assigned to the given `username`. Simillar to `assignee_id` and mutually exclusive with `assignee_id`. In CE version `assignee_username` array should only contain a single value or an invalid param error will be returned otherwise. | | `my_reaction_emoji` | string | no | Return issues reacted by the authenticated user by the given `emoji`. `None` returns issues not given a reaction. `Any` returns issues given at least one reaction. _([Introduced][ce-14016] in GitLab 10.0)_ | | `order_by` | string | no | Return issues ordered by `created_at` or `updated_at` fields. Default is `created_at` | | `sort` | string | no | Return issues sorted in `asc` or `desc` order. Default is `desc` | @@ -221,7 +240,7 @@ Example response: "id" : 9, "name" : "Dr. Luella Kovacek" }, - "labels" : [], + "labels" : ["foo", "bar"], "upvotes": 4, "downvotes": 0, "merge_requests_count": 0, @@ -240,8 +259,21 @@ Example response: "human_time_estimate": null, "human_total_time_spent": null }, + "has_tasks": true, + "task_status": "10 of 15 tasks completed", "confidential": false, - "discussion_locked": false + "discussion_locked": false, + "_links":{ + "self":"http://example.com/api/v4/projects/4/issues/41", + "notes":"`http://example.com/`api/v4/projects/4/issues/41/notes", + "award_emoji":"http://example.com/api/v4/projects/4/issues/41/award_emoji", + "project":"http://example.com/api/v4/projects/4" + }, + "subscribed": false, + "task_completion_status":{ + "count":0, + "completed_count":0 + } } ] ``` @@ -277,10 +309,13 @@ GET /projects/:id/issues?confidential=true | `iids[]` | Array[integer] | no | Return only the milestone having the given `iid` | | `state` | string | no | Return all issues or just those that are `opened` or `closed` | | `labels` | string | no | Comma-separated list of label names, issues must have all labels to be returned. `None` lists all issues with no labels. `Any` lists all issues with at least one label. `No+Label` (Deprecated) lists all issues with no labels. Predefined names are case-insensitive. | +| `with_labels_details`| Boolean | no | If `true`, response will return more details for each label in labels field: `:name`, `:color`, `:description`, `:text_color`. Default is `false`. | | `milestone` | string | no | The milestone title. `None` lists all issues with no milestone. `Any` lists all issues that have an assigned milestone. | | `scope` | string | no | Return issues for the given scope: `created_by_me`, `assigned_to_me` or `all`.<br> For versions before 11.0, use the now deprecated `created-by-me` or `assigned-to-me` scopes instead.<br> _([Introduced][ce-13004] in GitLab 9.5. [Changed to snake_case][ce-18935] in GitLab 11.0)_ | -| `author_id` | integer | no | Return issues created by the given user `id` _([Introduced][ce-13004] in GitLab 9.5)_ | -| `assignee_id` | integer | no | Return issues assigned to the given user `id`. `None` returns unassigned issues. `Any` returns issues with an assignee. _([Introduced][ce-13004] in GitLab 9.5)_ | +| `author_id` | integer | no | Return issues created by the given user `id`. Mutually exclusive with `author_username`. Combine with `scope=all` or `scope=assigned_to_me`. _([Introduced][ce-13004] in GitLab 9.5)_ | +| `author_username` | string | no | Return issues created by the given `username`. Simillar to `author_id` and mutually exclusive with `author_id`. | +| `assignee_id` | integer | no | Return issues assigned to the given user `id`. Mutually exclusive with `assignee_username`. `None` returns unassigned issues. `Any` returns issues with an assignee. _([Introduced][ce-13004] in GitLab 9.5)_ | +| `assignee_username` | Array[String] | no | Return issues assigned to the given `username`. Simillar to `assignee_id` and mutually exclusive with `assignee_id`. In CE version `assignee_username` array should only contain a single value or an invalid param error will be returned otherwise. | | `my_reaction_emoji` | string | no | Return issues reacted by the authenticated user by the given `emoji`. `None` returns issues not given a reaction. `Any` returns issues given at least one reaction. _([Introduced][ce-14016] in GitLab 10.0)_ | | `order_by` | string | no | Return issues ordered by `created_at` or `updated_at` fields. Default is `created_at` | | `sort` | string | no | Return issues sorted in `asc` or `desc` order. Default is `desc` | @@ -340,7 +375,7 @@ Example response: "id" : 9, "name" : "Dr. Luella Kovacek" }, - "labels" : [], + "labels" : ["foo", "bar"], "upvotes": 4, "downvotes": 0, "merge_requests_count": 0, @@ -366,8 +401,21 @@ Example response: "human_time_estimate": null, "human_total_time_spent": null }, + "has_tasks": true, + "task_status": "10 of 15 tasks completed", "confidential": false, - "discussion_locked": false + "discussion_locked": false, + "_links":{ + "self":"http://example.com/api/v4/projects/4/issues/41", + "notes":"`http://example.com/`api/v4/projects/4/issues/41/notes", + "award_emoji":"http://example.com/api/v4/projects/4/issues/41/award_emoji", + "project":"http://example.com/api/v4/projects/4" + }, + "subscribed": false, + "task_completion_status":{ + "count":0, + "completed_count":0 + } } ] ``` @@ -464,6 +512,10 @@ Example response: "notes": "http://example.com/api/v4/projects/1/issues/2/notes", "award_emoji": "http://example.com/api/v4/projects/1/issues/2/award_emoji", "project": "http://example.com/api/v4/projects/1" + }, + "task_completion_status":{ + "count":0, + "completed_count":0 } } ``` @@ -547,6 +599,10 @@ Example response: "notes": "http://example.com/api/v4/projects/1/issues/2/notes", "award_emoji": "http://example.com/api/v4/projects/1/issues/2/award_emoji", "project": "http://example.com/api/v4/projects/1" + }, + "task_completion_status":{ + "count":0, + "completed_count":0 } } ``` @@ -638,6 +694,10 @@ Example response: "notes": "http://example.com/api/v4/projects/1/issues/2/notes", "award_emoji": "http://example.com/api/v4/projects/1/issues/2/award_emoji", "project": "http://example.com/api/v4/projects/1" + }, + "task_completion_status":{ + "count":0, + "completed_count":0 } } ``` @@ -646,37 +706,6 @@ Example response: **Note**: The `closed_by` attribute was [introduced in GitLab 10.6][ce-17042]. This value will only be present for issues which were closed after GitLab 10.6 and when the user account that closed the issue still exists. -## Bulk update issues - -> [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/21368) in GitLab 11.9. - -Update multiple issues using a single API call. Returns the number of successfully updated issues. - -``` -PUT /projects/:id/issues/bulk_update -``` - -| Attribute | Type | Required | Description **** | -|----------------|---------|----------|------------------------------------------------------------------------------------------------------------| -| `id` | integer/string | yes | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding) owned by the authenticated user. | -| `issuable_ids` | Array[integer] | yes | The IDs of issues to be updated. | -| `assignee_ids` | Array[integer] | no | The ID of the user(s) to assign the issue to. Set to `0` or provide an empty value to unassign all assignees. | -| `milestone_id` | integer | no | The global ID of a milestone to assign the issue to. Set to `0` or provide an empty value to unassign a milestone.| -| `add_label_ids` | Array[integer] | no | Comma-separated label IDs to be added. | -| `remove_label_ids` | Array[integer] | no | Comma-separated label IDs to be added. | -| `state_event` | string | no | The state event of an issue. Set `close` to close the issue and `reopen` to reopen it. | -| `subscription_event` | string | no | The subscription_event event of an issue. Set `subscribe` to subscribe to the issue and `unsubscribe` to unsubscribe from it. | - -```bash -curl --request PUT --header "PRIVATE-TOKEN: <your_access_token>" https://gitlab.example.com/api/v4/projects/4/issues/bulk_update?issuable_ids[]=1&issuable_ids[]=2&state_event=close -``` - -Example response: - -```json -{ "message": "2 issues updated" } -``` - ## Delete an issue Only for admins and project owners. Soft deletes the issue in question. @@ -775,6 +804,10 @@ Example response: "notes": "http://example.com/api/v4/projects/1/issues/2/notes", "award_emoji": "http://example.com/api/v4/projects/1/issues/2/award_emoji", "project": "http://example.com/api/v4/projects/1" + }, + "task_completion_status":{ + "count":0, + "completed_count":0 } } ``` @@ -860,6 +893,10 @@ Example response: "notes": "http://example.com/api/v4/projects/1/issues/2/notes", "award_emoji": "http://example.com/api/v4/projects/1/issues/2/award_emoji", "project": "http://example.com/api/v4/projects/1" + }, + "task_completion_status":{ + "count":0, + "completed_count":0 } } ``` @@ -926,7 +963,11 @@ Example response: "due_date": null, "web_url": "http://example.com/example/example/issues/12", "confidential": false, - "discussion_locked": false + "discussion_locked": false, + "task_completion_status":{ + "count":0, + "completed_count":0 + } } ``` @@ -1024,7 +1065,11 @@ Example response: "due_date": null, "web_url": "http://example.com/example/example/issues/110", "confidential": false, - "discussion_locked": false + "discussion_locked": false, + "task_completion_status":{ + "count":0, + "completed_count":0 + } }, "target_url": "https://gitlab.example.com/gitlab-org/gitlab-ci/issues/10", "body": "Vel voluptas atque dicta mollitia adipisci qui at.", diff --git a/doc/api/issues_statistics.md b/doc/api/issues_statistics.md new file mode 100644 index 00000000000..82bc9c142cc --- /dev/null +++ b/doc/api/issues_statistics.md @@ -0,0 +1,177 @@ +# Issues Statistics API + +Every API call to issues_statistics must be authenticated. + +If a user is not a member of a project and the project is private, a `GET` +request on that project will result to a `404` status code. + +## Get issues statistics + +Gets issues count statistics on all issues the authenticated user has access to. By default it +returns only issues created by the current user. To get all issues, +use parameter `scope=all`. + +``` +GET /issues_statistics +GET /issues_statistics?labels=foo +GET /issues_statistics?labels=foo,bar +GET /issues_statistics?labels=foo,bar&state=opened +GET /issues_statistics?milestone=1.0.0 +GET /issues_statistics?milestone=1.0.0&state=opened +GET /issues_statistics?iids[]=42&iids[]=43 +GET /issues_statistics?author_id=5 +GET /issues_statistics?assignee_id=5 +GET /issues_statistics?my_reaction_emoji=star +GET /issues_statistics?search=foo&in=title +GET /issues_statistics?confidential=true +``` + +| Attribute | Type | Required | Description | +| ------------------- | ---------------- | ---------- | --------------------------------------------------------------------------------------------------------------------------------------------------- | +| `labels` | string | no | Comma-separated list of label names, issues must have all labels to be returned. `None` lists all issues with no labels. `Any` lists all issues with at least one label. | +| `milestone` | string | no | The milestone title. `None` lists all issues with no milestone. `Any` lists all issues that have an assigned milestone. | +| `scope` | string | no | Return issues for the given scope: `created_by_me`, `assigned_to_me` or `all`. Defaults to `created_by_me` | +| `author_id` | integer | no | Return issues created by the given user `id`. Mutually exclusive with `author_username`. Combine with `scope=all` or `scope=assigned_to_me`. | +| `author_username` | string | no | Return issues created by the given `username`. Similar to `author_id` and mutually exclusive with `author_id`. | +| `assignee_id` | integer | no | Return issues assigned to the given user `id`. Mutually exclusive with `assignee_username`. `None` returns unassigned issues. `Any` returns issues with an assignee. | +| `assignee_username` | Array[String] | no | Return issues assigned to the given `username`. Similar to `assignee_id` and mutually exclusive with `assignee_id`. In CE version `assignee_username` array should only contain a single value or an invalid param error will be returned otherwise. | +| `my_reaction_emoji` | string | no | Return issues reacted by the authenticated user by the given `emoji`. `None` returns issues not given a reaction. `Any` returns issues given at least one reaction. | +| `iids[]` | Array[integer] | no | Return only the issues having the given `iid` | +| `search` | string | no | Search issues against their `title` and `description` | +| `in` | string | no | Modify the scope of the `search` attribute. `title`, `description`, or a string joining them with comma. Default is `title,description` | +| `created_after` | datetime | no | Return issues created on or after the given time | +| `created_before` | datetime | no | Return issues created on or before the given time | +| `updated_after` | datetime | no | Return issues updated on or after the given time | +| `updated_before` | datetime | no | Return issues updated on or before the given time | +| `confidential ` | Boolean | no | Filter confidential or public issues. | + +```bash +curl --header "PRIVATE-TOKEN: <your_access_token>" https://gitlab.example.com/api/v4/issues_statistics +``` + +Example response: + +```json +{ + "statistics": { + "counts": { + "all": 20, + "closed": 5, + "opened": 15 + } + } +} +``` + +## Get group issues statistics + +Gets issues count statistics for given group. + +``` +GET /groups/:id/issues_statistics +GET /groups/:id/issues_statistics?labels=foo +GET /groups/:id/issues_statistics?labels=foo,bar +GET /groups/:id/issues_statistics?labels=foo,bar&state=opened +GET /groups/:id/issues_statistics?milestone=1.0.0 +GET /groups/:id/issues_statistics?milestone=1.0.0&state=opened +GET /groups/:id/issues_statistics?iids[]=42&iids[]=43 +GET /groups/:id/issues_statistics?search=issue+title+or+description +GET /groups/:id/issues_statistics?author_id=5 +GET /groups/:id/issues_statistics?assignee_id=5 +GET /groups/:id/issues_statistics?my_reaction_emoji=star +GET /groups/:id/issues_statistics?confidential=true +``` + +| Attribute | Type | Required | Description | +| ------------------- | ---------------- | ---------- | ----------------------------------------------------------------------------------------------------------------------------- | +| `id` | integer/string | yes | The ID or [URL-encoded path of the group](README.md#namespaced-path-encoding) owned by the authenticated user | +| `labels` | string | no | Comma-separated list of label names, issues must have all labels to be returned. `None` lists all issues with no labels. `Any` lists all issues with at least one label. | +| `iids[]` | Array[integer] | no | Return only the issues having the given `iid` | +| `milestone` | string | no | The milestone title. `None` lists all issues with no milestone. `Any` lists all issues that have an assigned milestone. | +| `scope` | string | no | Return issues for the given scope: `created_by_me`, `assigned_to_me` or `all`. | +| `author_id` | integer | no | Return issues created by the given user `id`. Mutually exclusive with `author_username`. Combine with `scope=all` or `scope=assigned_to_me`. | +| `author_username` | string | no | Return issues created by the given `username`. Similar to `author_id` and mutually exclusive with `author_id`. | +| `assignee_id` | integer | no | Return issues assigned to the given user `id`. Mutually exclusive with `assignee_username`. `None` returns unassigned issues. `Any` returns issues with an assignee. | +| `assignee_username` | Array[String] | no | Return issues assigned to the given `username`. Similar to `assignee_id` and mutually exclusive with `assignee_id`. In CE version `assignee_username` array should only contain a single value or an invalid param error will be returned otherwise. | +| `my_reaction_emoji` | string | no | Return issues reacted by the authenticated user by the given `emoji`. `None` returns issues not given a reaction. `Any` returns issues given at least one reaction. | +| `search` | string | no | Search group issues against their `title` and `description` | +| `created_after` | datetime | no | Return issues created on or after the given time | +| `created_before` | datetime | no | Return issues created on or before the given time | +| `updated_after` | datetime | no | Return issues updated on or after the given time | +| `updated_before` | datetime | no | Return issues updated on or before the given time | +| `confidential ` | Boolean | no | Filter confidential or public issues. | + +```bash +curl --header "PRIVATE-TOKEN: <your_access_token>" https://gitlab.example.com/api/v4/groups/4/issues_statistics +``` + +Example response: + +```json +{ + "statistics": { + "counts": { + "all": 20, + "closed": 5, + "opened": 15 + } + } +} +``` + +## Get project issues statistics + +Gets issues count statistics for given project. + +``` +GET /projects/:id/issues_statistics +GET /projects/:id/issues_statistics?labels=foo +GET /projects/:id/issues_statistics?labels=foo,bar +GET /projects/:id/issues_statistics?labels=foo,bar&state=opened +GET /projects/:id/issues_statistics?milestone=1.0.0 +GET /projects/:id/issues_statistics?milestone=1.0.0&state=opened +GET /projects/:id/issues_statistics?iids[]=42&iids[]=43 +GET /projects/:id/issues_statistics?search=issue+title+or+description +GET /projects/:id/issues_statistics?author_id=5 +GET /projects/:id/issues_statistics?assignee_id=5 +GET /projects/:id/issues_statistics?my_reaction_emoji=star +GET /projects/:id/issues_statistics?confidential=true +``` + +| Attribute | Type | Required | Description | +| ------------------- | ---------------- | ---------- | ----------------------------------------------------------------------------------------------------------------------------- | +| `id` | integer/string | yes | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding) owned by the authenticated user | +| `iids[]` | Array[integer] | no | Return only the milestone having the given `iid` | +| `labels` | string | no | Comma-separated list of label names, issues must have all labels to be returned. `None` lists all issues with no labels. `Any` lists all issues with at least one label. | +| `milestone` | string | no | The milestone title. `None` lists all issues with no milestone. `Any` lists all issues that have an assigned milestone. | +| `scope` | string | no | Return issues for the given scope: `created_by_me`, `assigned_to_me` or `all`. | +| `author_id` | integer | no | Return issues created by the given user `id`. Mutually exclusive with `author_username`. Combine with `scope=all` or `scope=assigned_to_me`. | +| `author_username` | string | no | Return issues created by the given `username`. Similar to `author_id` and mutually exclusive with `author_id`. | +| `assignee_id` | integer | no | Return issues assigned to the given user `id`. Mutually exclusive with `assignee_username`. `None` returns unassigned issues. `Any` returns issues with an assignee. | +| `assignee_username` | Array[String] | no | Return issues assigned to the given `username`. Similar to `assignee_id` and mutually exclusive with `assignee_id`. In CE version `assignee_username` array should only contain a single value or an invalid param error will be returned otherwise. | +| `my_reaction_emoji` | string | no | Return issues reacted by the authenticated user by the given `emoji`. `None` returns issues not given a reaction. `Any` returns issues given at least one reaction. | +| `search` | string | no | Search project issues against their `title` and `description` | +| `created_after` | datetime | no | Return issues created on or after the given time | +| `created_before` | datetime | no | Return issues created on or before the given time | +| `updated_after` | datetime | no | Return issues updated on or after the given time | +| `updated_before` | datetime | no | Return issues updated on or before the given time | +| `confidential ` | Boolean | no | Filter confidential or public issues. | + + +```bash +curl --header "PRIVATE-TOKEN: <your_access_token>" https://gitlab.example.com/api/v4/projects/4/issues_statistics +``` + +Example response: + +```json +{ + "statistics": { + "counts": { + "all": 20, + "closed": 5, + "opened": 15 + } + } +} +``` diff --git a/doc/api/jobs.md b/doc/api/jobs.md index 085e321b35f..72973b69117 100644 --- a/doc/api/jobs.md +++ b/doc/api/jobs.md @@ -10,7 +10,7 @@ GET /projects/:id/jobs | Attribute | Type | Required | Description | |-----------|--------------------------------|----------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| `id` | integer/string | yes | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding) owned by the authenticated user. | +| `id` | integer/string | yes | ID or [URL-encoded path of the project](README.md#namespaced-path-encoding) owned by the authenticated user. | | `scope` | string **or** array of strings | no | Scope of jobs to show. Either one of or an array of the following: `created`, `pending`, `running`, `failed`, `success`, `canceled`, `skipped`, or `manual`. All jobs are returned if `scope` is not provided. | ```sh @@ -32,6 +32,7 @@ Example of response "title": "Test the CI integration." }, "coverage": null, + "allow_failure": false, "created_at": "2015-12-24T15:51:21.727Z", "started_at": "2015-12-24T17:54:24.729Z", "finished_at": "2015-12-24T17:54:24.921Z", @@ -81,6 +82,7 @@ Example of response "title": "Test the CI integration." }, "coverage": null, + "allow_failure": false, "created_at": "2015-12-24T15:51:21.802Z", "started_at": "2015-12-24T17:54:27.722Z", "finished_at": "2015-12-24T17:54:27.895Z", @@ -142,8 +144,8 @@ GET /projects/:id/pipelines/:pipeline_id/jobs | Attribute | Type | Required | Description | |---------------|--------------------------------|----------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| `id` | integer/string | yes | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding) owned by the authenticated user. | -| `pipeline_id` | integer | yes | The ID of a pipeline. | +| `id` | integer/string | yes | ID or [URL-encoded path of the project](README.md#namespaced-path-encoding) owned by the authenticated user. | +| `pipeline_id` | integer | yes | ID of a pipeline. | | `scope` | string **or** array of strings | no | Scope of jobs to show. Either one of or an array of the following: `created`, `pending`, `running`, `failed`, `success`, `canceled`, `skipped`, or `manual`. All jobs are returned if `scope` is not provided. | ```sh @@ -165,6 +167,7 @@ Example of response "title": "Test the CI integration." }, "coverage": null, + "allow_failure": false, "created_at": "2015-12-24T15:51:21.727Z", "started_at": "2015-12-24T17:54:24.729Z", "finished_at": "2015-12-24T17:54:24.921Z", @@ -214,6 +217,7 @@ Example of response "title": "Test the CI integration." }, "coverage": null, + "allow_failure": false, "created_at": "2015-12-24T15:51:21.802Z", "started_at": "2015-12-24T17:54:27.722Z", "finished_at": "2015-12-24T17:54:27.895Z", @@ -275,8 +279,8 @@ GET /projects/:id/jobs/:job_id | Attribute | Type | Required | Description | |-----------|----------------|----------|------------------------------------------------------------------------------------------------------------------| -| `id` | integer/string | yes | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding) owned by the authenticated user. | -| `job_id` | integer | yes | The ID of a job. | +| `id` | integer/string | yes | ID or [URL-encoded path of the project](README.md#namespaced-path-encoding) owned by the authenticated user. | +| `job_id` | integer | yes | ID of a job. | ```sh curl --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/1/jobs/8" @@ -296,6 +300,7 @@ Example of response "title": "Test the CI integration." }, "coverage": null, + "allow_failure": false, "created_at": "2015-12-24T15:51:21.880Z", "started_at": "2015-12-24T17:54:30.733Z", "finished_at": "2015-12-24T17:54:31.198Z", @@ -341,30 +346,58 @@ Example of response > **Notes**: > > - [Introduced][ce-2893] in GitLab 8.5. +> - The use of `CI_JOB_TOKEN` in the artifacts download API was [introduced][ee-2346] +> in [GitLab Premium][ee] 9.5. -Get job artifacts of a project. +Get the job's artifacts zipped archive of a project. ``` GET /projects/:id/jobs/:job_id/artifacts ``` -| Attribute | Type | Required | Description | -|-----------|----------------|----------|------------------------------------------------------------------------------------------------------------------| -| `id` | integer/string | yes | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding) owned by the authenticated user. | -| `job_id` | integer | yes | The ID of a job. | +| Attribute | Type | Required | Description | +|-------------|----------------|----------|-------------------------------------------------------------------------------------------------------------------------------------------------| +| `id` | integer/string | yes | ID or [URL-encoded path of the project](README.md#namespaced-path-encoding) owned by the authenticated user. | +| `job_id` | integer | yes | ID of a job. | +| `job_token` **[PREMIUM]** | string | no | To be used with [triggers] for multi-project pipelines. It should be invoked only inside `.gitlab-ci.yml`. Its value is always `$CI_JOB_TOKEN`. | -Example requests: +Example request using the `PRIVATE-TOKEN` header: ```sh -curl --location --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/1/jobs/8/artifacts" +curl --output artifacts.zip --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/1/jobs/42/artifacts" ``` +To use this in a [`script` definition](../ci/yaml/README.md#script) inside +`.gitlab-ci.yml` **[PREMIUM]**, you can use either: + +- The `JOB-TOKEN` header with the GitLab-provided `CI_JOB_TOKEN` variable. + For example, the following job will download the artifacts of the job with ID + `42`. Note that the command is wrapped into single quotes since it contains a + colon (`:`): + + ```yaml + artifact_download: + stage: test + script: + - 'curl --location --output artifacts.zip --header "JOB-TOKEN: $CI_JOB_TOKEN" "https://gitlab.example.com/api/v4/projects/1/jobs/42/artifacts"' + ``` + +- Or the `job_token` attribute with the GitLab-provided `CI_JOB_TOKEN` variable. + For example, the following job will download the artifacts of the job with ID `42`: + + ```yaml + artifact_download: + stage: test + script: + - 'curl --location --output artifacts.zip "https://gitlab.example.com/api/v4/projects/1/jobs/42/artifacts?job_token=$CI_JOB_TOKEN"' + ``` + Possible response status codes: | Status | Description | |-----------|---------------------------------| -| 200 | Serves the artifacts file | -| 404 | Build not found or no artifacts | +| 200 | Serves the artifacts file. | +| 404 | Build not found or no artifacts.| [ce-2893]: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/2893 @@ -373,9 +406,13 @@ Possible response status codes: > **Notes**: > > - [Introduced][ce-5347] in GitLab 8.10. +> - The use of `CI_JOB_TOKEN` in the artifacts download API was [introduced][ee-2346] +> in [GitLab Premium][ee] 9.5. -Download the artifacts archive from the given reference name and job provided the -job finished successfully. +Download the artifacts zipped archive from the given reference name and job, +provided the job finished successfully. This is the same as +[getting the job's artifacts](#get-job-artifacts), but by defining the job's +name instead of its ID. ``` GET /projects/:id/jobs/artifacts/:ref_name/download?job=name @@ -383,24 +420,51 @@ GET /projects/:id/jobs/artifacts/:ref_name/download?job=name Parameters -| Attribute | Type | Required | Description | -|------------|----------------|----------|------------------------------------------------------------------------------------------------------------------| -| `id` | integer/string | yes | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding) owned by the authenticated user. | -| `ref_name` | string | yes | Branch or tag name in repository. HEAD or SHA references are not supported. | -| `job` | string | yes | The name of the job. | +| Attribute | Type | Required | Description | +|-------------|----------------|----------|-------------------------------------------------------------------------------------------------------------------------------------------------| +| `id` | integer/string | yes | ID or [URL-encoded path of the project](README.md#namespaced-path-encoding) owned by the authenticated user. | +| `ref_name` | string | yes | Branch or tag name in repository. HEAD or SHA references are not supported. | +| `job` | string | yes | The name of the job. | +| `job_token` **[PREMIUM]** | string | no | To be used with [triggers] for multi-project pipelines. It should be invoked only inside `.gitlab-ci.yml`. Its value is always `$CI_JOB_TOKEN`. | -Example requests: +Example request using the `PRIVATE-TOKEN` header: ```sh curl --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/1/jobs/artifacts/master/download?job=test" ``` +To use this in a [`script` definition](../ci/yaml/README.md#script) inside +`.gitlab-ci.yml` **[PREMIUM]**, you can use either: + +- The `JOB-TOKEN` header with the GitLab-provided `CI_JOB_TOKEN` variable. + For example, the following job will download the artifacts of the `test` job + of the `master` branch. Note that the command is wrapped into single quotes + since it contains a colon (`:`): + + ```yaml + artifact_download: + stage: test + script: + - 'curl --location --output artifacts.zip --header "JOB-TOKEN: $CI_JOB_TOKEN" "https://gitlab.example.com/api/v4/projects/$CI_PROJECT_ID/jobs/artifacts/master/download?job=test"' + ``` + +- Or the `job_token` attribute with the GitLab-provided `CI_JOB_TOKEN` variable. + For example, the following job will download the artifacts of the `test` job + of the `master` branch: + + ```yaml + artifact_download: + stage: test + script: + - 'curl --location --output artifacts.zip "https://gitlab.example.com/api/v4/projects/$CI_PROJECT_ID/jobs/artifacts/master/download?job=test&job_token=$CI_JOB_TOKEN"' + ``` + Possible response status codes: | Status | Description | |-----------|---------------------------------| -| 200 | Serves the artifacts file | -| 404 | Build not found or no artifacts | +| 200 | Serves the artifacts file. | +| 404 | Build not found or no artifacts.| [ce-5347]: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/5347 @@ -409,7 +473,7 @@ Possible response status codes: > Introduced in GitLab 10.0 Download a single artifact file from a job with a specified ID from within -the job's artifacts archive. The file is extracted from the archive and +the job's artifacts zipped archive. The file is extracted from the archive and streamed to the client. ``` @@ -420,7 +484,7 @@ Parameters | Attribute | Type | Required | Description | |-----------------|----------------|----------|------------------------------------------------------------------------------------------------------------------| -| `id` | integer/string | yes | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding) owned by the authenticated user. | +| `id` | integer/string | yes | ID or [URL-encoded path of the project](README.md#namespaced-path-encoding) owned by the authenticated user. | | `job_id ` | integer | yes | The unique job identifier. | | `artifact_path` | string | yes | Path to a file inside the artifacts archive. | @@ -454,7 +518,7 @@ Parameters: | Attribute | Type | Required | Description | |-----------------|----------------|----------|------------------------------------------------------------------------------------------------------------------| -| `id` | integer/string | yes | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding) owned by the authenticated user. | +| `id` | integer/string | yes | ID or [URL-encoded path of the project](README.md#namespaced-path-encoding) owned by the authenticated user. | | `ref_name` | string | yes | Branch or tag name in repository. HEAD or SHA references are not supported. | | `artifact_path` | string | yes | Path to a file inside the artifacts archive. | | `job` | string | yes | The name of the job. | @@ -483,8 +547,8 @@ GET /projects/:id/jobs/:job_id/trace | Attribute | Type | Required | Description | |-----------|----------------|----------|------------------------------------------------------------------------------------------------------------------| -| id | integer/string | yes | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding) owned by the authenticated user. | -| job_id | integer | yes | The ID of a job. | +| id | integer/string | yes | ID or [URL-encoded path of the project](README.md#namespaced-path-encoding) owned by the authenticated user. | +| job_id | integer | yes | ID of a job. | ```sh curl --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/1/jobs/8/trace" @@ -507,8 +571,8 @@ POST /projects/:id/jobs/:job_id/cancel | Attribute | Type | Required | Description | |-----------|----------------|----------|------------------------------------------------------------------------------------------------------------------| -| `id` | integer/string | yes | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding) owned by the authenticated user. | -| `job_id` | integer | yes | The ID of a job. | +| `id` | integer/string | yes | ID or [URL-encoded path of the project](README.md#namespaced-path-encoding) owned by the authenticated user. | +| `job_id` | integer | yes | ID of a job. | ```sh curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/1/jobs/1/cancel" @@ -528,6 +592,7 @@ Example of response "title": "Test the CI integration." }, "coverage": null, + "allow_failure": false, "created_at": "2016-01-11T10:13:33.506Z", "started_at": "2016-01-11T10:14:09.526Z", "finished_at": null, @@ -555,8 +620,8 @@ POST /projects/:id/jobs/:job_id/retry | Attribute | Type | Required | Description | |-----------|----------------|----------|------------------------------------------------------------------------------------------------------------------| -| `id` | integer/string | yes | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding) owned by the authenticated user. | -| `job_id` | integer | yes | The ID of a job. | +| `id` | integer/string | yes | ID or [URL-encoded path of the project](README.md#namespaced-path-encoding) owned by the authenticated user. | +| `job_id` | integer | yes | ID of a job. | ```sh curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/1/jobs/1/retry" @@ -576,6 +641,7 @@ Example of response "title": "Test the CI integration." }, "coverage": null, + "allow_failure": false, "created_at": "2016-01-11T10:13:33.506Z", "started_at": null, "finished_at": null, @@ -605,8 +671,8 @@ Parameters | Attribute | Type | Required | Description | |-----------|----------------|----------|------------------------------------------------------------------------------------------------------------------| -| `id` | integer/string | yes | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding) owned by the authenticated user. | -| `job_id` | integer | yes | The ID of a job. | +| `id` | integer/string | yes | ID or [URL-encoded path of the project](README.md#namespaced-path-encoding) owned by the authenticated user. | +| `job_id` | integer | yes | ID of a job. | Example of request @@ -628,6 +694,7 @@ Example of response "title": "Test the CI integration." }, "coverage": null, + "allow_failure": false, "download_url": null, "id": 42, "name": "rubocop", @@ -658,8 +725,8 @@ Parameters | Attribute | Type | Required | Description | |-----------|----------------|----------|------------------------------------------------------------------------------------------------------------------| -| `id` | integer/string | yes | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding) owned by the authenticated user. | -| `job_id` | integer | yes | The ID of a job. | +| `id` | integer/string | yes | ID or [URL-encoded path of the project](README.md#namespaced-path-encoding) owned by the authenticated user. | +| `job_id` | integer | yes | ID of a job. | Example request: @@ -681,6 +748,7 @@ Example response: "title": "Test the CI integration." }, "coverage": null, + "allow_failure": false, "download_url": null, "id": 42, "name": "rubocop", @@ -699,6 +767,33 @@ Example response: } ``` +## Delete artifacts + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/25522) in GitLab 11.9. + +Delete artifacts of a job. + +``` +DELETE /projects/:id/jobs/:job_id/artifacts +``` + +| Attribute | Type | Required | Description | +|-----------|----------------|----------|------------------------------------------------------------------------------------------------------------------| +| `id` | integer/string | yes | ID or [URL-encoded path of the project](README.md#namespaced-path-encoding) | +| `job_id` | integer | yes | ID of a job. | + + +Example request: + +```sh +curl --request DELETE --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/1/jobs/1/artifacts" +``` + +NOTE: **Note:** +At least Maintainer role is required to delete artifacts. + +If the artifacts were deleted successfully, a response with status `204 No Content` is returned. + ## Play a job Triggers a manual action to start a job. @@ -709,8 +804,8 @@ POST /projects/:id/jobs/:job_id/play | Attribute | Type | Required | Description | |-----------|----------------|----------|------------------------------------------------------------------------------------------------------------------| -| `id` | integer/string | yes | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding) owned by the authenticated user. | -| `job_id` | integer | yes | The ID of a job. | +| `id` | integer/string | yes | ID or [URL-encoded path of the project](README.md#namespaced-path-encoding) owned by the authenticated user. | +| `job_id` | integer | yes | ID of a job. | ```sh curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/1/jobs/1/play" @@ -730,6 +825,7 @@ Example of response "title": "Test the CI integration." }, "coverage": null, + "allow_failure": false, "created_at": "2016-01-11T10:13:33.506Z", "started_at": null, "finished_at": null, @@ -746,3 +842,7 @@ Example of response "user": null } ``` + +[ee]: https://about.gitlab.com/pricing/ +[ee-2346]: https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/2346 +[triggers]: ../ci/triggers/README.md#when-a-pipeline-depends-on-the-artifacts-of-another-pipeline-premium diff --git a/doc/api/license.md b/doc/api/license.md new file mode 100644 index 00000000000..2a8de64bdbf --- /dev/null +++ b/doc/api/license.md @@ -0,0 +1,176 @@ +# License **[CORE ONLY]** + +In order to interact with license endpoints, you need to authenticate yourself +as an admin. + +## Retrieve information about the current license + +``` +GET /license +``` + +```json +{ + "id": 2, + "plan": "gold", + "created_at": "2018-02-27T23:21:58.674Z", + "starts_at": "2018-01-27", + "expires_at": "2022-01-27", + "historical_max": 300, + "expired": false, + "overage": 200, + "user_limit": 100, + "active_users": 300, + "licensee": { + "Name": "John Doe1" + }, + "add_ons": { + "GitLab_FileLocks": 1, + "GitLab_Auditor_User": 1 + } +} +``` + +## Retrieve information about all licenses + +``` +GET /licenses +``` + +```json +[ + { + "id": 1, + "plan": "silver", + "created_at": "2018-02-27T23:21:58.674Z", + "starts_at": "2018-01-27", + "expires_at": "2022-01-27", + "historical_max": 300, + "expired": false, + "overage": 200, + "user_limit": 100, + "licensee": { + "Name": "John Doe1" + }, + "add_ons": { + "GitLab_FileLocks": 1, + "GitLab_Auditor_User": 1 + } + }, + { + "id": 2, + "plan": "gold", + "created_at": "2018-02-27T23:21:58.674Z", + "starts_at": "2018-01-27", + "expires_at": "2022-01-27", + "historical_max": 300, + "expired": false, + "overage": 200, + "user_limit": 100, + "licensee": { + "Name": "Doe John" + }, + "add_ons": { + "GitLab_FileLocks": 1, + } + } +] +``` + +Overage is the difference between the number of active users and the licensed number of users. +This is calculated differently depending on whether the license has expired or not. + +- If the license has expired, it uses the historical maximum active user count (`historical_max`). +- If the license has not expired, it uses the current active users count. + +Returns: + +- `200 OK` with response containing the licenses in JSON format. This will be an empty JSON array if there are no licenses. +- `403 Forbidden` if the current user in not permitted to read the licenses. + +## Add a new license + +``` +POST /license +``` + +| Attribute | Type | Required | Description | +| --------- | ---- | -------- | ----------- | +| `license` | string | yes | The license string | + +```bash +curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/license?license=eyJkYXRhIjoiMHM5Q...S01Udz09XG4ifQ==" +``` + +Example response: + +```json +{ + "id": 1, + "plan": "gold", + "created_at": "2018-02-27T23:21:58.674Z", + "starts_at": "2018-01-27", + "expires_at": "2022-01-27", + "historical_max": 300, + "expired": false, + "overage": 200, + "user_limit": 100, + "active_users": 300, + "licensee": { + "Name": "John Doe1" + }, + "add_ons": { + "GitLab_FileLocks": 1, + "GitLab_Auditor_User": 1 + } +} +``` + +Returns: + +- `201 Created` if the license is successfully added. +- `400 Bad Request` if the license couldn't be added, with an error message explaining the reason. + + +## Delete a license + +``` +DELETE /license/:id +``` + +| Attribute | Type | Required | Description | +| --------- | ---- | -------- | ----------- | +| `id` | integer | yes | ID of the GitLab license. | + +```bash +curl --request DELETE --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/license/:id" +``` + +Example response: + +```json +{ + "id": 2, + "plan": "gold", + "created_at": "2018-02-27T23:21:58.674Z", + "starts_at": "2018-01-27", + "expires_at": "2022-01-27", + "historical_max": 300, + "expired": false, + "overage": 200, + "user_limit": 100, + "licensee": { + "Name": "John Doe" + }, + "add_ons": { + "GitLab_FileLocks": 1, + "GitLab_Auditor_User": 1 + } +} +``` + +Returns: + +- `204 No Content` if the license is successfully deleted. +- `403 Forbidden` if the current user in not permitted to delete the license. +- `404 Not Found` if the license to delete could not be found. diff --git a/doc/api/license_templates.md b/doc/api/license_templates.md new file mode 100644 index 00000000000..1b68af9ce31 --- /dev/null +++ b/doc/api/license_templates.md @@ -0,0 +1,5 @@ +--- +redirect_to: 'templates/licenses.md' +--- + +This document was moved to [another location](templates/licenses.md). diff --git a/doc/api/managed_licenses.md b/doc/api/managed_licenses.md new file mode 100644 index 00000000000..47b193111b6 --- /dev/null +++ b/doc/api/managed_licenses.md @@ -0,0 +1,136 @@ +# Managed Licenses API **[ULTIMATE]** + +## List managed licenses + +Get all managed licenses for a given project. + +``` +GET /projects/:id/managed_licenses +``` + +| Attribute | Type | Required | Description | +| --------- | ------- | -------- | --------------------- | +| `id` | integer/string | yes | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding) | + +```bash +curl --header "PRIVATE-TOKEN: <your_access_token>" https://gitlab.example.com/api/v4/projects/1/managed_licenses +``` + +Example response: + +```json +[ + { + "id": 1, + "name": "MIT", + "approval_status": "approved" + }, + { + "id": 3, + "name": "ISC", + "approval_status": "blacklisted" + } +] +``` + +## Show an existing managed license + +Shows an existing managed license. + +``` +GET /projects/:id/managed_licenses/:managed_license_id +``` + +| Attribute | Type | Required | Description | +| --------------- | ------- | --------------------------------- | ------------------------------- | +| `id` | integer/string | yes | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding) owned by the authenticated user | +| `managed_license_id` | integer/string | yes | The ID or URL-encoded name of the license belonging to the project | + +```bash +curl --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/1/managed_licenses/6" +``` + +Example response: + +```json +{ + "id": 1, + "name": "MIT", + "approval_status": "blacklisted" +} +``` + +## Create a new managed license + +Creates a new managed license for the given project with the given name and approval status. + +``` +POST /projects/:id/managed_licenses +``` + +| Attribute | Type | Required | Description | +| ------------- | ------- | -------- | ---------------------------- | +| `id` | integer/string | yes | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding) owned by the authenticated user | +| `name` | string | yes | The name of the managed license | +| `approval_status` | string | yes | The approval status. "approved" or "blacklisted" | + +```bash +curl --data "name=MIT&approval_status=blacklisted" --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/1/managed_licenses" +``` + +Example response: + +```json +{ + "id": 1, + "name": "MIT", + "approval_status": "approved" +} +``` + +## Delete a managed license + +Deletes a managed license with a given id. + +``` +DELETE /projects/:id/managed_licenses/:managed_license_id +``` + +| Attribute | Type | Required | Description | +| --------- | ------- | -------- | --------------------- | +| `id` | integer/string | yes | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding) owned by the authenticated user | +| `managed_license_id` | integer/string | yes | The ID or URL-encoded name of the license belonging to the project | + +```bash +curl --request DELETE --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/1/managed_licenses/4" +``` + +When successful, it replies with an HTTP 204 response. + +## Edit an existing managed license + +Updates an existing managed license with a new approval status. + +``` +PATCH /projects/:id/managed_licenses/:managed_license_id +``` + +| Attribute | Type | Required | Description | +| --------------- | ------- | --------------------------------- | ------------------------------- | +| `id` | integer/string | yes | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding) owned by the authenticated user | +| `managed_license_id` | integer/string | yes | The ID or URL-encoded name of the license belonging to the project | +| `approval_status` | string | yes | The approval status. "approved" or "blacklisted" | + +```bash +curl --request PATCH --data "approval_status=blacklisted" --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/1/managed_licenses/6" +``` + +Example response: + +```json +{ + "id": 1, + "name": "MIT", + "approval_status": "blacklisted" +} +``` diff --git a/doc/api/members.md b/doc/api/members.md index 0593d2c20ea..8784d577f99 100644 --- a/doc/api/members.md +++ b/doc/api/members.md @@ -62,7 +62,9 @@ Example response: ## List all members of a group or project including inherited members Gets a list of group or project members viewable by the authenticated user, including inherited members through ancestor groups. -Returns multiple times the same user (with different member attributes) when the user is a member of the project/group and of one or more ancestor group. +When a user is a member of the project/group and of one or more ancestor groups the user is returned only once with the project access_level (if exists) +or the access_level for the user in the first group which he belongs to in the project groups ancestors chain. +**Note:** We plan to [change](https://gitlab.com/gitlab-org/gitlab-ce/issues/62284) this behavior to return highest access_level instead. ``` GET /groups/:id/members/all diff --git a/doc/api/merge_request_approvals.md b/doc/api/merge_request_approvals.md new file mode 100644 index 00000000000..ddac81328b9 --- /dev/null +++ b/doc/api/merge_request_approvals.md @@ -0,0 +1,425 @@ +# Merge request approvals API **[STARTER]** + +Configuration for approvals on all Merge Requests (MR) in the project. Must be authenticated for all endpoints. + +## Project-level MR approvals + +### Get Configuration + +>**Note:** This API endpoint is only available on 10.6 Starter and above. + +You can request information about a project's approval configuration using the +following endpoint: + +``` +GET /projects/:id/approvals +``` + +**Parameters:** + +| Attribute | Type | Required | Description | +| --------- | ------- | -------- | ------------------- | +| `id` | integer | yes | The ID of a project | + +```json +{ + "approvers": [ + { + "user": { + "id": 5, + "name": "John Doe6", + "username": "user5", + "state":"active","avatar_url":"https://www.gravatar.com/avatar/4aea8cf834ed91844a2da4ff7ae6b491?s=80\u0026d=identicon","web_url":"http://localhost/user5" + } + } + ], + "approver_groups": [ + { + "group": { + "id": 1, + "name": "group1", + "path": "group1", + "description": "", + "visibility": "public", + "lfs_enabled": false, + "avatar_url": null, + "web_url": "http://localhost/groups/group1", + "request_access_enabled": false, + "full_name": "group1", + "full_path": "group1", + "parent_id": null, + "ldap_cn": null, + "ldap_access": null + } + } + ], + "approvals_before_merge": 2, + "reset_approvals_on_push": true, + "disable_overriding_approvers_per_merge_request": false +} +``` + +### Change configuration + +>**Note:** This API endpoint is only available on 10.6 Starter and above. + +If you are allowed to, you can change approval configuration using the following +endpoint: + +``` +POST /projects/:id/approvals +``` + +**Parameters:** + +| Attribute | Type | Required | Description | +| ------------------------------------------------ | ------- | -------- | ---------------------------------------------------------- | +| `id` | integer | yes | The ID of a project | +| `approvals_before_merge` | integer | no | How many approvals are required before an MR can be merged | +| `reset_approvals_on_push` | boolean | no | Reset approvals on a new push | +| `disable_overriding_approvers_per_merge_request` | boolean | no | Allow/Disallow overriding approvers per MR | +| `merge_requests_author_approval` | boolean | no | Allow/Disallow authors be able to self approve merge requests | + +```json +{ + "approvers": [ + { + "user": { + "id": 5, + "name": "John Doe6", + "username": "user5", + "state":"active","avatar_url":"https://www.gravatar.com/avatar/4aea8cf834ed91844a2da4ff7ae6b491?s=80\u0026d=identicon","web_url":"http://localhost/user5" + } + } + ], + "approver_groups": [ + { + "group": { + "id": 1, + "name": "group1", + "path": "group1", + "description": "", + "visibility": "public", + "lfs_enabled": false, + "avatar_url": null, + "web_url": "http://localhost/groups/group1", + "request_access_enabled": false, + "full_name": "group1", + "full_path": "group1", + "parent_id": null, + "ldap_cn": null, + "ldap_access": null + } + } + ], + "approvals_before_merge": 2, + "reset_approvals_on_push": true, + "disable_overriding_approvers_per_merge_request": false, + "merge_requests_author_approval": false +} +``` + +### Change allowed approvers + +>**Note:** This API endpoint is only available on 10.6 Starter and above. + +If you are allowed to, you can change approvers and approver groups using +the following endpoint: + +``` +PUT /projects/:id/approvers +``` + +**Important:** Approvers and groups not in the request will be **removed** + +**Parameters:** + +| Attribute | Type | Required | Description | +| -------------------- | ------- | -------- | --------------------------------------------------- | +| `id` | integer | yes | The ID of a project | +| `approver_ids` | Array | yes | An array of User IDs that can approve MRs | +| `approver_group_ids` | Array | yes | An array of Group IDs whose members can approve MRs | + +```json +{ + "approvers": [ + { + "user": { + "id": 5, + "name": "John Doe6", + "username": "user5", + "state":"active","avatar_url":"https://www.gravatar.com/avatar/4aea8cf834ed91844a2da4ff7ae6b491?s=80\u0026d=identicon","web_url":"http://localhost/user5" + } + } + ], + "approver_groups": [ + { + "group": { + "id": 1, + "name": "group1", + "path": "group1", + "description": "", + "visibility": "public", + "lfs_enabled": false, + "avatar_url": null, + "web_url": "http://localhost/groups/group1", + "request_access_enabled": false, + "full_name": "group1", + "full_path": "group1", + "parent_id": null, + "ldap_cn": null, + "ldap_access": null + } + } + ], + "approvals_before_merge": 2, + "reset_approvals_on_push": true, + "disable_overriding_approvers_per_merge_request": false +} +``` + + +## Merge Request-level MR approvals + +Configuration for approvals on a specific Merge Request. Must be authenticated for all endpoints. + +### Get Configuration + +>**Note:** This API endpoint is only available on 8.9 Starter and above. + +You can request information about a merge request's approval status using the +following endpoint: + +``` +GET /projects/:id/merge_requests/:merge_request_iid/approvals +``` + +**Parameters:** + +| Attribute | Type | Required | Description | +|---------------------|---------|----------|---------------------| +| `id` | integer | yes | The ID of a project | +| `merge_request_iid` | integer | yes | The IID of MR | + +```json +{ + "id": 5, + "iid": 5, + "project_id": 1, + "title": "Approvals API", + "description": "Test", + "state": "opened", + "created_at": "2016-06-08T00:19:52.638Z", + "updated_at": "2016-06-08T21:20:42.470Z", + "merge_status": "cannot_be_merged", + "approvals_required": 2, + "approvals_left": 1, + "approved_by": [ + { + "user": { + "name": "Administrator", + "username": "root", + "id": 1, + "state": "active", + "avatar_url": "http://www.gravatar.com/avatar/e64c7d89f26bd1972efa854d13d7dd61?s=80\u0026d=identicon", + "web_url": "http://localhost:3000/u/root" + } + } + ], + "approvers": [], + "approver_groups": [] +} +``` + +### Change approval configuration + +>**Note:** This API endpoint is only available on 10.6 Starter and above. + +If you are allowed to, you can change `approvals_required` using the following +endpoint: + +``` +POST /projects/:id/merge_requests/:merge_request_iid/approvals +``` + +**Parameters:** + +| Attribute | Type | Required | Description | +|----------------------|---------|----------|--------------------------------------------| +| `id` | integer | yes | The ID of a project | +| `merge_request_iid` | integer | yes | The IID of MR | +| `approvals_required` | integer | yes | Approvals required before MR can be merged | + + +```json +{ + "id": 5, + "iid": 5, + "project_id": 1, + "title": "Approvals API", + "description": "Test", + "state": "opened", + "created_at": "2016-06-08T00:19:52.638Z", + "updated_at": "2016-06-08T21:20:42.470Z", + "merge_status": "cannot_be_merged", + "approvals_required": 2, + "approvals_left": 2, + "approved_by": [], + "approvers": [], + "approver_groups": [] +} +``` + +### Change allowed approvers for Merge Request + +>**Note:** This API endpoint is only available on 10.6 Starter and above. + +If you are allowed to, you can change approvers and approver groups using +the following endpoint: + +``` +PUT /projects/:id/merge_requests/:merge_request_iid/approvers +``` + +**Important:** Approvers and groups not in the request will be **removed** + +**Parameters:** + +| Attribute | Type | Required | Description | +|----------------------|---------|----------|-----------------------------------------------------------| +| `id` | integer | yes | The ID of a project | +| `merge_request_iid` | integer | yes | The IID of MR | +| `approver_ids` | Array | yes | An array of User IDs that can approve the MR | +| `approver_group_ids` | Array | yes | An array of Group IDs whose members can approve the MR | + +```json +{ + "id": 5, + "iid": 5, + "project_id": 1, + "title": "Approvals API", + "description": "Test", + "state": "opened", + "created_at": "2016-06-08T00:19:52.638Z", + "updated_at": "2016-06-08T21:20:42.470Z", + "merge_status": "cannot_be_merged", + "approvals_required": 2, + "approvals_left": 2, + "approved_by": [], + "approvers": [ + { + "user": { + "name": "Administrator", + "username": "root", + "id": 1, + "state": "active", + "avatar_url": "http://www.gravatar.com/avatar/e64c7d89f26bd1972efa854d13d7dd61?s=80\u0026d=identicon", + "web_url": "http://localhost:3000/u/root" + } + } + ], + "approver_groups": [ + { + "group": { + "id": 5, + "name": "group1", + "path": "group1", + "description": "", + "visibility": "public", + "lfs_enabled": false, + "avatar_url": null, + "web_url": "http://localhost/groups/group1", + "request_access_enabled": false, + "full_name": "group1", + "full_path": "group1", + "parent_id": null, + "ldap_cn": null, + "ldap_access": null + } + } + ] +} +``` + +## Approve Merge Request + +>**Note:** This API endpoint is only available on 8.9 Starter and above. + +If you are allowed to, you can approve a merge request using the following +endpoint: + +``` +POST /projects/:id/merge_requests/:merge_request_iid/approve +``` + +**Parameters:** + +| Attribute | Type | Required | Description | +|---------------------|---------|----------|-------------------------| +| `id` | integer | yes | The ID of a project | +| `merge_request_iid` | integer | yes | The IID of MR | +| `sha` | string | no | The HEAD of the MR | +| `approval_password` **[STARTER]** | string | no | Current user's password. Required if [**Require user password to approve**](../user/project/merge_requests/merge_request_approvals.md#require-authentication-when-approving-a-merge-request-starter) is enabled in the project settings. | + +The `sha` parameter works in the same way as +when [accepting a merge request](merge_requests.md#accept-mr): if it is passed, then it must +match the current HEAD of the merge request for the approval to be added. If it +does not match, the response code will be `409`. + +```json +{ + "id": 5, + "iid": 5, + "project_id": 1, + "title": "Approvals API", + "description": "Test", + "state": "opened", + "created_at": "2016-06-08T00:19:52.638Z", + "updated_at": "2016-06-09T21:32:14.105Z", + "merge_status": "can_be_merged", + "approvals_required": 2, + "approvals_left": 0, + "approved_by": [ + { + "user": { + "name": "Administrator", + "username": "root", + "id": 1, + "state": "active", + "avatar_url": "http://www.gravatar.com/avatar/e64c7d89f26bd1972efa854d13d7dd61?s=80\u0026d=identicon", + "web_url": "http://localhost:3000/u/root" + } + }, + { + "user": { + "name": "Nico Cartwright", + "username": "ryley", + "id": 2, + "state": "active", + "avatar_url": "http://www.gravatar.com/avatar/cf7ad14b34162a76d593e3affca2adca?s=80\u0026d=identicon", + "web_url": "http://localhost:3000/u/ryley" + } + } + ], + "approvers": [], + "approver_groups": [] +} +``` + +## Unapprove Merge Request + +>**Note:** This API endpoint is only available on 9.0 Starter and above. + +If you did approve a merge request, you can unapprove it using the following +endpoint: + +``` +POST /projects/:id/merge_requests/:merge_request_iid/unapprove +``` + +**Parameters:** + +| Attribute | Type | Required | Description | +|---------------------|---------|----------|---------------------| +| `id` | integer | yes | The ID of a project | +| `merge_request_iid` | integer | yes | The IID of MR | diff --git a/doc/api/merge_requests.md b/doc/api/merge_requests.md index ba47e507b79..96a956ad03a 100644 --- a/doc/api/merge_requests.md +++ b/doc/api/merge_requests.md @@ -93,6 +93,14 @@ Parameters: "avatar_url": null, "web_url" : "https://gitlab.example.com/admin" }, + "assignees": [{ + "name": "Miss Monserrate Beier", + "username": "axel.block", + "id": 12, + "state": "active", + "avatar_url": "http://www.gravatar.com/avatar/46f6f7dc858ada7be1853f7fb96e81da?s=80&d=identicon", + "web_url": "https://gitlab.example.com/axel.block" + }], "source_project_id": 2, "target_project_id": 3, "labels": [ @@ -130,7 +138,11 @@ Parameters: "human_time_estimate": null, "human_total_time_spent": null }, - "squash": false + "squash": false, + "task_completion_status":{ + "count":0, + "completed_count":0 + } } ] ``` @@ -227,6 +239,14 @@ Parameters: "avatar_url": null, "web_url" : "https://gitlab.example.com/admin" }, + "assignees": [{ + "name": "Miss Monserrate Beier", + "username": "axel.block", + "id": 12, + "state": "active", + "avatar_url": "http://www.gravatar.com/avatar/46f6f7dc858ada7be1853f7fb96e81da?s=80&d=identicon", + "web_url": "https://gitlab.example.com/axel.block" + }], "source_project_id": 2, "target_project_id": 3, "labels": [ @@ -264,7 +284,11 @@ Parameters: "human_time_estimate": null, "human_total_time_spent": null }, - "squash": false + "squash": false, + "task_completion_status":{ + "count":0, + "completed_count":0 + } } ] ``` @@ -351,6 +375,14 @@ Parameters: "avatar_url": null, "web_url" : "https://gitlab.example.com/admin" }, + "assignees": [{ + "name": "Miss Monserrate Beier", + "username": "axel.block", + "id": 12, + "state": "active", + "avatar_url": "http://www.gravatar.com/avatar/46f6f7dc858ada7be1853f7fb96e81da?s=80&d=identicon", + "web_url": "https://gitlab.example.com/axel.block" + }], "source_project_id": 2, "target_project_id": 3, "labels": [ @@ -386,7 +418,11 @@ Parameters: "human_time_estimate": null, "human_total_time_spent": null }, - "squash": false + "squash": false, + "task_completion_status":{ + "count":0, + "completed_count":0 + } } ] ``` @@ -445,6 +481,14 @@ Parameters: "avatar_url": null, "web_url" : "https://gitlab.example.com/admin" }, + "assignees": [{ + "name": "Miss Monserrate Beier", + "username": "axel.block", + "id": 12, + "state": "active", + "avatar_url": "http://www.gravatar.com/avatar/46f6f7dc858ada7be1853f7fb96e81da?s=80&d=identicon", + "web_url": "https://gitlab.example.com/axel.block" + }], "source_project_id": 2, "target_project_id": 3, "labels": [ @@ -513,7 +557,11 @@ Parameters: "start_sha": "c380d3acebd181f13629a25d2e2acca46ffe1e00" }, "diverged_commits_count": 2, - "rebase_in_progress": false + "rebase_in_progress": false, + "task_completion_status":{ + "count":0, + "completed_count":0 + } } ``` @@ -547,7 +595,7 @@ Parameters: "state": "active", "avatar_url": "http://www.gravatar.com/avatar/10fc7f102be8de7657fb4d80898bbfe3?s=80&d=identicon", "web_url": "http://localhost/user2" - }, + } ] ``` @@ -629,6 +677,14 @@ Parameters: "avatar_url": "http://www.gravatar.com/avatar/e64c7d89f26bd1972efa854d13d7dd61?s=40&d=identicon", "web_url" : "https://gitlab.example.com/root" }, + "assignees": [{ + "name": "Miss Monserrate Beier", + "username": "axel.block", + "id": 12, + "state": "active", + "avatar_url": "http://www.gravatar.com/avatar/46f6f7dc858ada7be1853f7fb96e81da?s=80&d=identicon", + "web_url": "https://gitlab.example.com/axel.block" + }], "source_project_id": 4, "target_project_id": 4, "labels": [ ], @@ -662,7 +718,11 @@ Parameters: "total_time_spent": 0, "human_time_estimate": null, "human_total_time_spent": null - } + }, + "task_completion_status":{ + "count":0, + "completed_count":0 + }, "changes": [ { "old_path": "VERSION", @@ -718,6 +778,7 @@ POST /projects/:id/merge_requests | `target_branch` | string | yes | The target branch | | `title` | string | yes | Title of MR | | `assignee_id` | integer | no | Assignee user ID | +| `assignee_ids` | Array[integer] | no | The ID of the user(s) to assign the MR to. Set to `0` or provide an empty value to unassign all assignees. | | `description` | string | no | Description of MR | | `target_project_id` | integer | no | The target project (numeric id) | | `labels` | string | no | Labels for MR as a comma-separated list | @@ -824,7 +885,11 @@ POST /projects/:id/merge_requests "head_sha": "2be7ddb704c7b6b83732fdd5b9f09d5a397b5f8f", "start_sha": "c380d3acebd181f13629a25d2e2acca46ffe1e00" }, - "diverged_commits_count": 2 + "diverged_commits_count": 2, + "task_completion_status":{ + "count":0, + "completed_count":0 + } } ``` @@ -843,6 +908,7 @@ PUT /projects/:id/merge_requests/:merge_request_iid | `target_branch` | string | no | The target branch | | `title` | string | no | Title of MR | | `assignee_id` | integer | no | The ID of the user to assign the merge request to. Set to `0` or provide an empty value to unassign all assignees. | +| `assignee_ids` | Array[integer] | no | The ID of the user(s) to assign the MR to. Set to `0` or provide an empty value to unassign all assignees. | | `milestone_id` | integer | no | The global ID of a milestone to assign the merge request to. Set to `0` or provide an empty value to unassign a milestone.| | `labels` | string | no | Comma-separated label names for a merge request. Set to an empty string to unassign all labels. | | `description` | string | no | Description of MR | @@ -885,6 +951,14 @@ Must include at least one non-required attribute from above. "avatar_url": null, "web_url" : "https://gitlab.example.com/admin" }, + "assignees": [{ + "name": "Miss Monserrate Beier", + "username": "axel.block", + "id": 12, + "state": "active", + "avatar_url": "http://www.gravatar.com/avatar/46f6f7dc858ada7be1853f7fb96e81da?s=80&d=identicon", + "web_url": "https://gitlab.example.com/axel.block" + }], "source_project_id": 2, "target_project_id": 3, "labels": [ @@ -952,41 +1026,14 @@ Must include at least one non-required attribute from above. "head_sha": "2be7ddb704c7b6b83732fdd5b9f09d5a397b5f8f", "start_sha": "c380d3acebd181f13629a25d2e2acca46ffe1e00" }, - "diverged_commits_count": 2 + "diverged_commits_count": 2, + "task_completion_status":{ + "count":0, + "completed_count":0 + } } ``` -## Bulk update merge requests - -> [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/21368) in GitLab 11.9. - -Update multiple merge requests using a single API call. Returns the number of successfully updated merge requests. - -``` -PUT /projects/:id/merge_requests/bulk_update -``` - -| Attribute | Type | Required | Description **** | -|----------------|---------|----------|------------------------------------------------------------------------------------------------------------| -| `id` | integer/string | yes | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding) owned by the authenticated user. | -| `issuable_ids` | Array[integer] | yes | The IDs of merge requests to be updated. | -| `assignee_ids` | Array[integer] | no | The ID of the user(s) to assign the issue to. Set to `0` or provide an empty value to unassign all assignees. | -| `milestone_id` | integer | no | The global ID of a milestone to assign the issue to. Set to `0` or provide an empty value to unassign a milestone.| -| `add_label_ids` | Array[integer] | no | Comma-separated label IDs to be added. | -| `remove_label_ids` | Array[integer] | no | Comma-separated label IDs to be added. | -| `state_event` | string | no | The state event of an issue. Set `close` to close the issue and `reopen` to reopen it. | -| `subscription_event` | string | no | The subscription_event event of an issue. Set `subscribe` to subscribe to the issue and `unsubscribe` to unsubscribe from it. | - -```bash -curl --request PUT --header "PRIVATE-TOKEN: <your_access_token>" https://gitlab.example.com/api/v4/projects/4/merge_requests/bulk_update?issuable_ids[]=1&issuable_ids[]=2&state_event=close -``` - -Example response: - -```json -{ "message": "2 merge_requests updated" } -``` - ## Delete a merge request Only for admins and project owners. Soft deletes the merge request in question. @@ -1061,6 +1108,14 @@ Parameters: "avatar_url": null, "web_url" : "https://gitlab.example.com/admin" }, + "assignees": [{ + "name": "Miss Monserrate Beier", + "username": "axel.block", + "id": 12, + "state": "active", + "avatar_url": "http://www.gravatar.com/avatar/46f6f7dc858ada7be1853f7fb96e81da?s=80&d=identicon", + "web_url": "https://gitlab.example.com/axel.block" + }], "source_project_id": 2, "target_project_id": 3, "labels": [ @@ -1128,7 +1183,41 @@ Parameters: "head_sha": "2be7ddb704c7b6b83732fdd5b9f09d5a397b5f8f", "start_sha": "c380d3acebd181f13629a25d2e2acca46ffe1e00" }, - "diverged_commits_count": 2 + "diverged_commits_count": 2, + "task_completion_status":{ + "count":0, + "completed_count":0 + } +} +``` + +## Returns the up to date merge-ref HEAD commit + +Merge the changes between the merge request source and target branches into `refs/merge-requests/:iid/merge` +ref, of the target project repository, if possible. This ref will have the state the target branch would have if +a regular merge action was taken. + +This is not a regular merge action given it doesn't change the merge request target branch state in any manner. + +This ref (`refs/merge-requests/:iid/merge`) isn't necessarily overwritten when submitting +requests to this API, though it'll make sure the ref has the latest possible state. + +If the merge request has conflicts, is empty or already merged, you'll get a `400` and a descriptive error message. + +It returns the HEAD commit of `refs/merge-requests/:iid/merge` in the response body in case of `200`. + +``` +GET /projects/:id/merge_requests/:merge_request_iid/merge_ref +``` + +Parameters: + +- `id` (required) - The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding) owned by the authenticated user +- `merge_request_iid` (required) - Internal ID of MR + +```json +{ + "commit_id": "854a3a7a17acbcc0bbbea170986df1eb60435f34" } ``` @@ -1177,6 +1266,14 @@ Parameters: "avatar_url": null, "web_url" : "https://gitlab.example.com/admin" }, + "assignees": [{ + "name": "Miss Monserrate Beier", + "username": "axel.block", + "id": 12, + "state": "active", + "avatar_url": "http://www.gravatar.com/avatar/46f6f7dc858ada7be1853f7fb96e81da?s=80&d=identicon", + "web_url": "https://gitlab.example.com/axel.block" + }], "source_project_id": 2, "target_project_id": 3, "labels": [ @@ -1244,7 +1341,11 @@ Parameters: "head_sha": "2be7ddb704c7b6b83732fdd5b9f09d5a397b5f8f", "start_sha": "c380d3acebd181f13629a25d2e2acca46ffe1e00" }, - "diverged_commits_count": 2 + "diverged_commits_count": 2, + "task_completion_status":{ + "count":0, + "completed_count":0 + } } ``` @@ -1280,7 +1381,7 @@ If the rebase operation is ongoing, the response will include the following: ```json { - "rebase_in_progress": true + "rebase_in_progress": true, "merge_error": null } ``` @@ -1291,7 +1392,7 @@ the following: ```json { "rebase_in_progress": false, - "merge_error": null, + "merge_error": null } ``` @@ -1300,7 +1401,7 @@ If the rebase operation fails, the response will include the following: ```json { "rebase_in_progress": false, - "merge_error": "Rebase failed. Please rebase locally", + "merge_error": "Rebase failed. Please rebase locally" } ``` @@ -1433,6 +1534,14 @@ Example response: "avatar_url": null, "web_url" : "https://gitlab.example.com/admin" }, + "assignees": [{ + "name": "Miss Monserrate Beier", + "username": "axel.block", + "id": 12, + "state": "active", + "avatar_url": "http://www.gravatar.com/avatar/46f6f7dc858ada7be1853f7fb96e81da?s=80&d=identicon", + "web_url": "https://gitlab.example.com/axel.block" + }], "source_project_id": 2, "target_project_id": 3, "labels": [ @@ -1499,7 +1608,11 @@ Example response: "head_sha": "2be7ddb704c7b6b83732fdd5b9f09d5a397b5f8f", "start_sha": "c380d3acebd181f13629a25d2e2acca46ffe1e00" }, - "diverged_commits_count": 2 + "diverged_commits_count": 2, + "task_completion_status":{ + "count":0, + "completed_count":0 + } } ``` @@ -1554,6 +1667,14 @@ Example response: "avatar_url": null, "web_url" : "https://gitlab.example.com/admin" }, + "assignees": [{ + "name": "Miss Monserrate Beier", + "username": "axel.block", + "id": 12, + "state": "active", + "avatar_url": "http://www.gravatar.com/avatar/46f6f7dc858ada7be1853f7fb96e81da?s=80&d=identicon", + "web_url": "https://gitlab.example.com/axel.block" + }], "source_project_id": 2, "target_project_id": 3, "labels": [ @@ -1620,7 +1741,11 @@ Example response: "head_sha": "2be7ddb704c7b6b83732fdd5b9f09d5a397b5f8f", "start_sha": "c380d3acebd181f13629a25d2e2acca46ffe1e00" }, - "diverged_commits_count": 2 + "diverged_commits_count": 2, + "task_completion_status":{ + "count":0, + "completed_count":0 + } } ``` @@ -1695,6 +1820,14 @@ Example response: "avatar_url": "http://www.gravatar.com/avatar/733005fcd7e6df12d2d8580171ccb966?s=80&d=identicon", "web_url": "https://gitlab.example.com/barrett.krajcik" }, + "assignees": [{ + "name": "Miss Monserrate Beier", + "username": "axel.block", + "id": 12, + "state": "active", + "avatar_url": "http://www.gravatar.com/avatar/46f6f7dc858ada7be1853f7fb96e81da?s=80&d=identicon", + "web_url": "https://gitlab.example.com/axel.block" + }], "source_project_id": 3, "target_project_id": 3, "labels": [], diff --git a/doc/api/milestones.md b/doc/api/milestones.md index fa8f8a0bcf0..90d8c033cd6 100644 --- a/doc/api/milestones.md +++ b/doc/api/milestones.md @@ -10,6 +10,7 @@ GET /projects/:id/milestones?iids[]=42 GET /projects/:id/milestones?iids[]=42&iids[]=43 GET /projects/:id/milestones?state=active GET /projects/:id/milestones?state=closed +GET /projects/:id/milestones?title=1.0 GET /projects/:id/milestones?search=version ``` @@ -20,6 +21,7 @@ Parameters: | `id` | integer/string | yes | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding) owned by the authenticated user | | `iids[]` | Array[integer] | optional | Return only the milestones having the given `iid` | | `state` | string | optional | Return only `active` or `closed` milestones | +| `title` | string | optional | Return only the milestones having the given `title` | | `search` | string | optional | Return only milestones with a title or description matching the provided string | ```bash @@ -90,7 +92,7 @@ Parameters: - `description` (optional) - The description of a milestone - `due_date` (optional) - The due date of the milestone - `start_date` (optional) - The start date of the milestone -- `state_event` (optional) - The state event of the milestone (close|activate) +- `state_event` (optional) - The state event of the milestone (close or activate) ## Delete project milestone @@ -130,3 +132,18 @@ Parameters: - `id` (required) - The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding) owned by the authenticated user - `milestone_id` (required) - The ID of a project milestone + +## Promote project milestone to a group milestone + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/issues/53861) in GitLab 11.9 + +Only for users with developer access to the group. + +``` +POST /projects/:id/milestones/:milestone_id/promote +``` + +Parameters: + +- `id` (required) - The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding) owned by the authenticated user +- `milestone_id` (required) - The ID of a project milestone diff --git a/doc/api/namespaces.md b/doc/api/namespaces.md index b8bc4c40124..b66a3198ffb 100644 --- a/doc/api/namespaces.md +++ b/doc/api/namespaces.md @@ -64,9 +64,9 @@ Get all namespaces that match a string in their name or path. GET /namespaces?search=foobar ``` -| Attribute | Type | Required | Description | -| --------- | ---- | -------- | ----------- | -| `search` | string | no | Returns a list of namespaces the user is authorized to see based on the search criteria | +| Attribute | Type | Required | Description | +| --------- | ------ | -------- | ----------- | +| `search` | string | no | Returns a list of namespaces the user is authorized to see based on the search criteria | Example request: @@ -98,9 +98,9 @@ Get a namespace by ID. GET /namespaces/:id ``` -| Attribute | Type | Required | Description | -| --------- | ---- | -------- | ----------- | -| `id` | integer/string | yes | ID or path of the namespace | +| Attribute | Type | Required | Description | +| --------- | -------------- | -------- | ----------- | +| `id` | integer/string | yes | ID or [URL-encoded path of the namespace](README.md#namespaced-path-encoding) | Example request: diff --git a/doc/api/notes.md b/doc/api/notes.md index dd4e18b14e6..c09129c22d4 100644 --- a/doc/api/notes.md +++ b/doc/api/notes.md @@ -1,6 +1,20 @@ # Notes API -Notes are comments on snippets, issues or merge requests. +Notes are comments on: + +- Snippets +- Issues +- Merge requests +- Epics **[ULTIMATE]** + +This includes system notes, which are notes about changes to the object (for example, when a milestone changes, there will be a corresponding system note). Label notes are not part of this API, but recorded as separate events in [resource label events](resource_label_events.md). + +## Notes pagination + +By default, `GET` requests return 20 results at a time because the API results +are paginated. + +Read more on [pagination](README.md#pagination). ## Issues @@ -381,3 +395,126 @@ Parameters: ```bash curl --request DELETE --header "PRIVATE-TOKEN: <your_access_token>" https://gitlab.example.com/api/v4/projects/5/merge_requests/7/notes/1602 ``` + +## Epics **[ULTIMATE]** + +### List all epic notes + +Gets a list of all notes for a single epic. Epic notes are comments users can post to an epic. + +``` +GET /groups/:id/epics/:epic_id/notes +GET /groups/:id/epics/:epic_id/notes?sort=asc&order_by=updated_at +``` + +| Attribute | Type | Required | Description | +| ------------------- | ---------------- | ---------- | ----------- | +| `id` | integer/string | yes | The ID or [URL-encoded path of the group](README.md#namespaced-path-encoding) | +| `epic_id` | integer | yes | The ID of a group epic | +| `sort` | string | no | Return epic notes sorted in `asc` or `desc` order. Default is `desc` | +| `order_by` | string | no | Return epic notes ordered by `created_at` or `updated_at` fields. Default is `created_at` | + +```bash +curl --request GET --header "PRIVATE-TOKEN: <your_access_token>" https://gitlab.example.com/api/v4/groups/5/epics/11/notes +``` + +### Get single epic note + +Returns a single note for a given epic. + +``` +GET /groups/:id/epics/:epic_id/notes/:note_id +``` + +Parameters: + +| Attribute | Type | Required | Description | +| --------- | -------------- | -------- | ----------- | +| `id` | integer/string | yes | The ID or [URL-encoded path of the group](README.md#namespaced-path-encoding) | +| `epic_id` | integer | yes | The ID of an epic | +| `note_id` | integer | yes | The ID of a note | + +```json +{ + "id": 52, + "title": "Epic", + "file_name": "epic.rb", + "author": { + "id": 1, + "username": "pipin", + "email": "admin@example.com", + "name": "Pip", + "state": "active", + "created_at": "2013-09-30T13:46:01Z" + }, + "expires_at": null, + "updated_at": "2013-10-02T07:34:20Z", + "created_at": "2013-10-02T07:34:20Z" +} +``` + +```bash +curl --request GET --header "PRIVATE-TOKEN: <your_access_token>" https://gitlab.example.com/api/v4/groups/5/epics/11/notes/1 +``` + +### Create new epic note + +Creates a new note for a single epic. Epic notes are comments users can post to an epic. +If you create a note where the body only contains an Award Emoji, you'll receive this object back. + +``` +POST /groups/:id/epics/:epic_id/notes +``` + +Parameters: + +| Attribute | Type | Required | Description | +| --------- | -------------- | -------- | ----------- | +| `id` | integer/string | yes | The ID or [URL-encoded path of the group](README.md#namespaced-path-encoding) | +| `epic_id` | integer | yes | The ID of an epic | +| `body` | string | yes | The content of a note | + +```bash +curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" https://gitlab.example.com/api/v4/projects/5/snippet/11/notes?body=note +``` + +### Modify existing epic note + +Modify existing note of an epic. + +``` +PUT /groups/:id/epics/:epic_id/notes/:note_id +``` + +Parameters: + +| Attribute | Type | Required | Description | +| --------- | -------------- | -------- | ----------- | +| `id` | integer/string | yes | The ID or [URL-encoded path of the group](README.md#namespaced-path-encoding) | +| `epic_id` | integer | yes | The ID of an epic | +| `note_id` | integer | yes | The ID of a note | +| `body` | string | yes | The content of a note | + +```bash +curl --request PUT --header "PRIVATE-TOKEN: <your_access_token>" https://gitlab.example.com/api/v4/projects/5/snippet/11/notes?body=note +``` + +### Delete an epic note + +Deletes an existing note of an epic. + +``` +DELETE /groups/:id/epics/:epic_id/notes/:note_id +``` + +Parameters: + +| Attribute | Type | Required | Description | +| --------- | -------------- | -------- | ----------- | +| `id` | integer/string | yes | The ID or [URL-encoded path of the group](README.md#namespaced-path-encoding) | +| `epic_id` | integer | yes | The ID of an epic | +| `note_id` | integer | yes | The ID of a note | + +```bash +curl --request DELETE --header "PRIVATE-TOKEN: <your_access_token>" https://gitlab.example.com/api/v4/groups/5/epics/52/notes/1659 +``` diff --git a/doc/api/packages.md b/doc/api/packages.md new file mode 100644 index 00000000000..618e5c3056a --- /dev/null +++ b/doc/api/packages.md @@ -0,0 +1,152 @@ +# Packages API **[PREMIUM]** + +This is the API docs of [GitLab Packages](../administration/packages.md). + +## List project packages + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/9259) in GitLab 11.8. + +Get a list of project packages. Both Maven and NPM packages are included in results. +When accessed without authentication, only packages of public projects are returned. + +``` +GET /projects/:id/packages +``` + +| Attribute | Type | Required | Description | +| --------- | ---- | -------- | ----------- | +| `id` | integer/string | yes | ID or [URL-encoded path of the project](README.md#namespaced-path-encoding) | + +```bash +curl --header "PRIVATE-TOKEN: <your_access_token>" https://gitlab.example.com/api/v4/projects/:id/packages +``` + +Example response: + +```json +[ + { + "id": 1, + "name": "com/mycompany/my-app", + "version": "1.0-SNAPSHOT", + "package_type": "maven" + }, + { + "id": 2, + "name": "@foo/bar", + "version": "1.0.3", + "package_type": "npm" + } +] +``` + +By default, the `GET` request will return 20 results, since the API is [paginated](README.md#pagination). + +## Get a project package + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/9667) in GitLab 11.9. + +Get a single project package. + +``` +GET /projects/:id/packages/:package_id +``` + +| Attribute | Type | Required | Description | +| --------- | ---- | -------- | ----------- | +| `id` | integer/string | yes | ID or [URL-encoded path of the project](README.md#namespaced-path-encoding). | +| `package_id` | integer | yes | ID of a package. | + +```bash +curl --header "PRIVATE-TOKEN: <your_access_token>" https://gitlab.example.com/api/v4/projects/:id/packages/:package_id +``` + +Example response: + +```json +{ + "id": 1, + "name": "com/mycompany/my-app", + "version": "1.0-SNAPSHOT", + "package_type": "maven" +} +``` + +## List package files + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/9305) in GitLab 11.8. + +Get a list of package files of a single package. + +``` +GET /projects/:id/packages/:package_id/package_files +``` + +| Attribute | Type | Required | Description | +| --------- | ---- | -------- | ----------- | +| `id` | integer/string | yes | ID or [URL-encoded path of the project](README.md#namespaced-path-encoding) | +| `package_id` | integer | yes | ID of a package. | + +```bash +curl --header "PRIVATE-TOKEN: <your_access_token>" https://gitlab.example.com/api/v4/projects/1/packages/4/package_files +``` + +Example response: + +```json +[ + { + "id": 25, + "package_id": 4, + "created_at": "2018-11-07T15:25:52.199Z", + "file_name": "my-app-1.5-20181107.152550-1.jar", + "size": 2421, + "file_md5": "58e6a45a629910c6ff99145a688971ac", + "file_sha1": "ebd193463d3915d7e22219f52740056dfd26cbfe" + }, + { + "id": 26, + "package_id": 4, + "created_at": "2018-11-07T15:25:56.776Z", + "file_name": "my-app-1.5-20181107.152550-1.pom", + "size": 1122, + "file_md5": "d90f11d851e17c5513586b4a7e98f1b2", + "file_sha1": "9608d068fe88aff85781811a42f32d97feb440b5" + }, + { + "id": 27, + "package_id": 4, + "created_at": "2018-11-07T15:26:00.556Z", + "file_name": "maven-metadata.xml", + "size": 767, + "file_md5": "6dfd0cce1203145a927fef5e3a1c650c", + "file_sha1": "d25932de56052d320a8ac156f745ece73f6a8cd2" + } +] +``` + +By default, the `GET` request will return 20 results, since the API is [paginated](README.md#pagination). + +## Delete a project package + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/9623) in GitLab 11.9. + +Deletes a project package. + +``` +DELETE /projects/:id/packages/:package_id +``` + +| Attribute | Type | Required | Description | +| --------- | ---- | -------- | ----------- | +| `id` | integer/string | yes | ID or [URL-encoded path of the project](README.md#namespaced-path-encoding) | +| `package_id` | integer | yes | ID of a package. | + +```bash +curl --request DELETE --header "PRIVATE-TOKEN: <your_access_token>" https://gitlab.example.com/api/v4/projects/:id/packages/:package_id +``` + +Can return the following status codes: + +- `204 No Content`, if the package was deleted successfully. +- `404 Not Found`, if the package was not found. diff --git a/doc/api/pipeline_schedules.md b/doc/api/pipeline_schedules.md index 137f1fdddec..470e55425f8 100644 --- a/doc/api/pipeline_schedules.md +++ b/doc/api/pipeline_schedules.md @@ -1,4 +1,4 @@ -# Pipeline schedules +# Pipeline schedules API You can read more about [pipeline schedules](../user/project/pipelines/schedules.md). @@ -88,6 +88,7 @@ curl --header "PRIVATE-TOKEN: k5ESFgWY2Qf5xEvDcFxZ" "https://gitlab.example.com/ "variables": [ { "key": "TEST_VARIABLE_1", + "variable_type": "env_var", "value": "TEST_1" } ] @@ -278,9 +279,9 @@ curl --request DELETE --header "PRIVATE-TOKEN: k5ESFgWY2Qf5xEvDcFxZ" "https://gi } ``` -## Pipeline schedule variable +## Pipeline schedule variables -> [Introduced][ce-34518] in GitLab 10.0. +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/issues/34518) in GitLab 10.0. ## Create a new pipeline schedule variable @@ -296,6 +297,7 @@ POST /projects/:id/pipeline_schedules/:pipeline_schedule_id/variables | `pipeline_schedule_id` | integer | yes | The pipeline schedule id | | `key` | string | yes | The `key` of a variable; must have no more than 255 characters; only `A-Z`, `a-z`, `0-9`, and `_` are allowed | | `value` | string | yes | The `value` of a variable | +| `variable_type` | string | no | The type of a variable. Available types are: `env_var` (default) and `file` | ```sh curl --request POST --header "PRIVATE-TOKEN: k5ESFgWY2Qf5xEvDcFxZ" --form "key=NEW_VARIABLE" --form "value=new value" "https://gitlab.example.com/api/v4/projects/29/pipeline_schedules/13/variables" @@ -304,6 +306,7 @@ curl --request POST --header "PRIVATE-TOKEN: k5ESFgWY2Qf5xEvDcFxZ" --form "key=N ```json { "key": "NEW_VARIABLE", + "variable_type": "env_var", "value": "new value" } ``` @@ -322,6 +325,7 @@ PUT /projects/:id/pipeline_schedules/:pipeline_schedule_id/variables/:key | `pipeline_schedule_id` | integer | yes | The pipeline schedule id | | `key` | string | yes | The `key` of a variable | | `value` | string | yes | The `value` of a variable | +| `variable_type` | string | no | The type of a variable. Available types are: `env_var` (default) and `file` | ```sh curl --request PUT --header "PRIVATE-TOKEN: k5ESFgWY2Qf5xEvDcFxZ" --form "value=updated value" "https://gitlab.example.com/api/v4/projects/29/pipeline_schedules/13/variables/NEW_VARIABLE" @@ -331,6 +335,7 @@ curl --request PUT --header "PRIVATE-TOKEN: k5ESFgWY2Qf5xEvDcFxZ" --form "value= { "key": "NEW_VARIABLE", "value": "updated value" + "variable_type": "env_var", } ``` @@ -358,5 +363,3 @@ curl --request DELETE --header "PRIVATE-TOKEN: k5ESFgWY2Qf5xEvDcFxZ" "https://gi "value": "updated value" } ``` - -[ce-34518]: https://gitlab.com/gitlab-org/gitlab-ce/issues/34518
\ No newline at end of file diff --git a/doc/api/pipelines.md b/doc/api/pipelines.md index 43bbf463c8d..753faec3cc8 100644 --- a/doc/api/pipelines.md +++ b/doc/api/pipelines.md @@ -93,6 +93,37 @@ Example of response } ``` +### Get variables of a pipeline + +``` +GET /projects/:id/pipelines/:pipeline_id/variables +``` + +| Attribute | Type | Required | Description | +|------------|---------|----------|---------------------| +| `id` | integer/string | yes | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding) owned by the authenticated user | +| `pipeline_id` | integer | yes | The ID of a pipeline | + +``` +curl --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/1/pipelines/46/variables" +``` + +Example of response + +```json +[ + { + "key": "RUN_NIGHTLY_BUILD", + "variable_type": "env_var", + "value": "true" + }, + { + "key": "foo", + "value": "bar" + } +] +``` + ## Create a new pipeline > [Introduced][ce-7209] in GitLab 8.14 @@ -105,7 +136,7 @@ POST /projects/:id/pipeline |------------|---------|----------|---------------------| | `id` | integer/string | yes | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding) owned by the authenticated user | | `ref` | string | yes | Reference to commit | -| `variables` | array | no | An array containing the variables available in the pipeline, matching the structure [{ 'key' => 'UPLOAD_TO_S3', 'value' => 'true' }] | +| `variables` | array | no | An array containing the variables available in the pipeline, matching the structure [{ 'key' => 'UPLOAD_TO_S3', 'variable_type' => 'file', 'value' => 'true' }] | ``` curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/1/pipeline?ref=master" diff --git a/doc/api/project_clusters.md b/doc/api/project_clusters.md index 8efb98fe1fc..327781f6c93 100644 --- a/doc/api/project_clusters.md +++ b/doc/api/project_clusters.md @@ -33,6 +33,7 @@ Example response: { "id":18, "name":"cluster-1", + "domain":"example.com", "created_at":"2019-01-02T20:18:12.563Z", "provider_type":"user", "platform_type":"kubernetes", @@ -90,6 +91,7 @@ Example response: { "id":18, "name":"cluster-1", + "domain":"example.com", "created_at":"2019-01-02T20:18:12.563Z", "provider_type":"user", "platform_type":"kubernetes", @@ -157,12 +159,15 @@ Parameters: | --------- | ---- | -------- | ----------- | | `id` | integer | yes | The ID of the project owned by the authenticated user | | `name` | String | yes | The name of the cluster | +| `domain` | String | no | The [base domain](../user/project/clusters/index.md#base-domain) of the cluster | | `enabled` | Boolean | no | Determines if cluster is active or not, defaults to true | +| `managed` | Boolean | no | Determines if GitLab will manage namespaces and service accounts for this cluster, defaults to true | | `platform_kubernetes_attributes[api_url]` | String | yes | The URL to access the Kubernetes API | | `platform_kubernetes_attributes[token]` | String | yes | The token to authenticate against Kubernetes | | `platform_kubernetes_attributes[ca_cert]` | String | no | TLS certificate (needed if API is using a self-signed TLS certificate | | `platform_kubernetes_attributes[namespace]` | String | no | The unique namespace related to the project | | `platform_kubernetes_attributes[authorization_type]` | String | no | The cluster authorization type: `rbac`, `abac` or `unknown_authorization`. Defaults to `rbac`. | +| `environment_scope` | String | no | The associated environment to the cluster. Defaults to `*` **[PREMIUM]** | Example request: @@ -247,10 +252,12 @@ Parameters: | `id` | integer | yes | The ID of the project owned by the authenticated user | | `cluster_id` | integer | yes | The ID of the cluster | | `name` | String | no | The name of the cluster | +| `domain` | String | no | The [base domain](../user/project/clusters/index.md#base-domain) of the cluster | | `platform_kubernetes_attributes[api_url]` | String | no | The URL to access the Kubernetes API | | `platform_kubernetes_attributes[token]` | String | no | The token to authenticate against Kubernetes | | `platform_kubernetes_attributes[ca_cert]` | String | no | TLS certificate (needed if API is using a self-signed TLS certificate | | `platform_kubernetes_attributes[namespace]` | String | no | The unique namespace related to the project | +| `environment_scope` | String | no | The associated environment to the cluster **[PREMIUM]** | NOTE: **Note:** `name`, `api_url`, `ca_cert` and `token` can only be updated if the cluster was added @@ -262,7 +269,7 @@ Example request: ```bash curl --header 'Private-Token: <your_access_token>' https://gitlab.example.com/api/v4/projects/26/clusters/24 \ -H "Content-Type:application/json" \ --X PUT --data '{"name":"new-cluster-name","api_url":"https://new-api-url.com"}' +-X PUT --data '{"name":"new-cluster-name","domain":"new-domain.com","api_url":"https://new-api-url.com"}' ``` Example response: @@ -271,6 +278,7 @@ Example response: { "id":24, "name":"new-cluster-name", + "domain":"new-domain.com", "created_at":"2019-01-03T21:53:40.610Z", "provider_type":"user", "platform_type":"kubernetes", diff --git a/doc/api/project_level_variables.md b/doc/api/project_level_variables.md index 438bebe62f5..66a749e4811 100644 --- a/doc/api/project_level_variables.md +++ b/doc/api/project_level_variables.md @@ -1,4 +1,4 @@ -# Project-level Variables API +# Project-level Variables API ## List project variables @@ -20,10 +20,12 @@ curl --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/a [ { "key": "TEST_VARIABLE_1", + "variable_type": "env_var", "value": "TEST_1" }, { "key": "TEST_VARIABLE_2", + "variable_type": "env_var", "value": "TEST_2" } ] @@ -49,7 +51,10 @@ curl --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/a ```json { "key": "TEST_VARIABLE_1", - "value": "TEST_1" + "variable_type": "env_var", + "value": "TEST_1", + "protected": false, + "masked": true } ``` @@ -61,12 +66,15 @@ Create a new variable. POST /projects/:id/variables ``` -| Attribute | Type | required | Description | -|-------------|---------|----------|-----------------------| -| `id` | integer/string | yes | The ID of a project or [urlencoded NAMESPACE/PROJECT_NAME of the project](README.md#namespaced-path-encoding) owned by the authenticated user | -| `key` | string | yes | The `key` of a variable; must have no more than 255 characters; only `A-Z`, `a-z`, `0-9`, and `_` are allowed | -| `value` | string | yes | The `value` of a variable | -| `protected` | boolean | no | Whether the variable is protected | +| Attribute | Type | required | Description | +|---------------------|---------|----------|-----------------------| +| `id` | integer/string | yes | The ID of a project or [urlencoded NAMESPACE/PROJECT_NAME of the project](README.md#namespaced-path-encoding) owned by the authenticated user | +| `key` | string | yes | The `key` of a variable; must have no more than 255 characters; only `A-Z`, `a-z`, `0-9`, and `_` are allowed | +| `value` | string | yes | The `value` of a variable | +| `variable_type` | string | no | The type of a variable. Available types are: `env_var` (default) and `file` | +| `protected` | boolean | no | Whether the variable is protected | +| `masked` | boolean | no | Whether the variable is masked | +| `environment_scope` | string | no | The `environment_scope` of the variable **[PREMIUM]** | ``` curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/1/variables" --form "key=NEW_VARIABLE" --form "value=new value" @@ -76,7 +84,11 @@ curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" "https://gitla { "key": "NEW_VARIABLE", "value": "new value", - "protected": false + "protected": false, + "variable_type": "env_var", + "protected": false, + "masked": false, + "environment_scope": "*" } ``` @@ -88,12 +100,15 @@ Update a project's variable. PUT /projects/:id/variables/:key ``` -| Attribute | Type | required | Description | -|-------------|---------|----------|-------------------------| -| `id` | integer/string | yes | The ID of a project or [urlencoded NAMESPACE/PROJECT_NAME of the project](README.md#namespaced-path-encoding) owned by the authenticated user | -| `key` | string | yes | The `key` of a variable | -| `value` | string | yes | The `value` of a variable | -| `protected` | boolean | no | Whether the variable is protected | +| Attribute | Type | required | Description | +|---------------------|---------|----------|-------------------------| +| `id` | integer/string | yes | The ID of a project or [urlencoded NAMESPACE/PROJECT_NAME of the project](README.md#namespaced-path-encoding) owned by the authenticated user | +| `key` | string | yes | The `key` of a variable | +| `value` | string | yes | The `value` of a variable | +| `variable_type` | string | no | The type of a variable. Available types are: `env_var` (default) and `file` | +| `protected` | boolean | no | Whether the variable is protected | +| `masked` | boolean | no | Whether the variable is masked | +| `environment_scope` | string | no | The `environment_scope` of the variable **[PREMIUM]** | ``` curl --request PUT --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/1/variables/NEW_VARIABLE" --form "value=updated value" @@ -103,7 +118,10 @@ curl --request PUT --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab { "key": "NEW_VARIABLE", "value": "updated value", - "protected": true + "variable_type": "env_var", + "protected": true, + "masked": false, + "environment_scope": "*" } ``` diff --git a/doc/api/project_snippets.md b/doc/api/project_snippets.md index f02674adfe2..0ccb0517e08 100644 --- a/doc/api/project_snippets.md +++ b/doc/api/project_snippets.md @@ -121,7 +121,6 @@ Parameters: ## Get user agent details -> **Notes:** > [Introduced][ce-29508] in GitLab 9.4. Available only for admins. diff --git a/doc/api/project_statistics.md b/doc/api/project_statistics.md new file mode 100644 index 00000000000..34d73abfcbf --- /dev/null +++ b/doc/api/project_statistics.md @@ -0,0 +1,49 @@ +# Project statistics API + +Every API call to [project](../user/project/index.md) statistics must be authenticated. + +## Get the statistics of the last 30 days + +Retrieving the statistics requires write access to the repository. +Currently only HTTP fetches statistics are returned. +Fetches statistics includes both clones and pulls count and are HTTP only, SSH fetches are not included. + +``` +GET /projects/:id/statistics +``` + +| Attribute | Type | Required | Description | +| ---------- | ------ | -------- | ----------- | +| `id ` | integer / string | yes | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding) | + +Example response: + +```json +{ + "fetches": { + "total": 50, + "days": [ + { + "count": 10, + "date": "2018-01-10" + }, + { + "count": 10, + "date": "2018-01-09" + }, + { + "count": 10, + "date": "2018-01-08" + }, + { + "count": 10, + "date": "2018-01-07" + }, + { + "count": 10, + "date": "2018-01-06" + } + ] + } +} +``` diff --git a/doc/api/project_templates.md b/doc/api/project_templates.md index 3b5b12c8da3..0a94a8d47ae 100644 --- a/doc/api/project_templates.md +++ b/doc/api/project_templates.md @@ -15,7 +15,7 @@ templates are also available from this API endpoint. Support will be added for [Issue and Merge Request templates](../user/project/description_templates.md) in a future release. -Support for [Group-level file templates](../user/group/index.md#group-level-file-templates-premium) +Support for [Group-level file templates](../user/group/index.md#group-file-templates-premium) **[PREMIUM]** was [added](https://gitlab.com/gitlab-org/gitlab-ee/issues/5987) in GitLab 11.5 diff --git a/doc/api/projects.md b/doc/api/projects.md index 0a950352ecf..75669d85803 100644 --- a/doc/api/projects.md +++ b/doc/api/projects.md @@ -153,6 +153,7 @@ When the user is authenticated and `simple` is not set this returns something li "commit_count": 37, "storage_size": 1038090, "repository_size": 1038090, + "wiki_size" : 0, "lfs_objects_size": 0, "job_artifacts_size": 0 }, @@ -234,6 +235,7 @@ When the user is authenticated and `simple` is not set this returns something li "commit_count": 12, "storage_size": 2066080, "repository_size": 2066080, + "wiki_size" : 0, "lfs_objects_size": 0, "job_artifacts_size": 0 }, @@ -342,6 +344,7 @@ GET /users/:user_id/projects "commit_count": 37, "storage_size": 1038090, "repository_size": 1038090, + "wiki_size" : 0, "lfs_objects_size": 0, "job_artifacts_size": 0 }, @@ -423,6 +426,7 @@ GET /users/:user_id/projects "commit_count": 12, "storage_size": 2066080, "repository_size": 2066080, + "wiki_size" : 0, "lfs_objects_size": 0, "job_artifacts_size": 0 }, @@ -494,7 +498,9 @@ GET /projects/:id "name": "Diaspora", "path": "diaspora", "kind": "group", - "full_path": "diaspora" + "full_path": "diaspora", + "avatar_url": "http://localhost:3000/uploads/group/avatar/3/foo.jpg", + "web_url": "http://localhost:3000/groups/diaspora" }, "import_status": "none", "import_error": null, @@ -546,6 +552,7 @@ GET /projects/:id "commit_count": 37, "storage_size": 1038090, "repository_size": 1038090, + "wiki_size" : 0, "lfs_objects_size": 0, "job_artifacts_size": 0 }, @@ -561,6 +568,8 @@ GET /projects/:id } ``` +**Note**: The `web_url` and `avatar_url` attributes on `namespace` were [introduced][ce-27427] in GitLab 11.11. + If the project is a fork, and you provide a valid token to authenticate, the `forked_from_project` field will appear in the response. @@ -1587,3 +1596,4 @@ GET /projects/:id/snapshot [eep]: https://about.gitlab.com/pricing/ "Available only in GitLab Premium" [ee-6137]: https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/6137 +[ce-27427]: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/27427 diff --git a/doc/api/releases/links.md b/doc/api/releases/links.md index fd7b9d6e6e2..9c91264ed65 100644 --- a/doc/api/releases/links.md +++ b/doc/api/releases/links.md @@ -15,7 +15,7 @@ GET /projects/:id/releases/:tag_name/assets/links | Attribute | Type | Required | Description | | ------------- | -------------- | -------- | --------------------------------------- | -| `id` | integer/string | yes | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding). | +| `id` | integer/string | yes | The ID or [URL-encoded path of the project](../README.md#namespaced-path-encoding). | | `tag_name` | string | yes | The tag associated with the Release. | Example request: @@ -53,7 +53,7 @@ GET /projects/:id/releases/:tag_name/assets/links/:link_id | Attribute | Type | Required | Description | | ------------- | -------------- | -------- | --------------------------------------- | -| `id` | integer/string | yes | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding). | +| `id` | integer/string | yes | The ID or [URL-encoded path of the project](../README.md#namespaced-path-encoding). | | `tag_name` | string | yes | The tag associated with the Release. | | `link_id` | integer | yes | The id of the link. | @@ -84,7 +84,7 @@ POST /projects/:id/releases/:tag_name/assets/links | Attribute | Type | Required | Description | | ------------- | -------------- | -------- | --------------------------------------- | -| `id` | integer/string | yes | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding). | +| `id` | integer/string | yes | The ID or [URL-encoded path of the project](../README.md#namespaced-path-encoding). | | `tag_name` | string | yes | The tag associated with the Release. | | `name` | string | yes | The name of the link. | | `url` | string | yes | The URL of the link. | @@ -120,7 +120,7 @@ PUT /projects/:id/releases/:tag_name/assets/links/:link_id | Attribute | Type | Required | Description | | ------------- | -------------- | -------- | --------------------------------------- | -| `id` | integer/string | yes | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding). | +| `id` | integer/string | yes | The ID or [URL-encoded path of the project](../README.md#namespaced-path-encoding). | | `tag_name` | string | yes | The tag associated with the Release. | | `link_id` | integer | yes | The id of the link. | | `name` | string | no | The name of the link. | @@ -156,7 +156,7 @@ DELETE /projects/:id/releases/:tag_name/assets/links/:link_id | Attribute | Type | Required | Description | | ------------- | -------------- | -------- | --------------------------------------- | -| `id` | integer/string | yes | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding). | +| `id` | integer/string | yes | The ID or [URL-encoded path of the project](../README.md#namespaced-path-encoding). | | `tag_name` | string | yes | The tag associated with the Release. | | `link_id` | integer | yes | The id of the link. | diff --git a/doc/api/repository_files.md b/doc/api/repository_files.md index 6fcc06ea8cd..87c7f371de1 100644 --- a/doc/api/repository_files.md +++ b/doc/api/repository_files.md @@ -181,7 +181,7 @@ Currently gitlab-shell has a boolean return code, preventing GitLab from specify ## Delete existing file in repository -This allows you to delete a single file. For deleting multiple files with a singleh request see the [commits API](commits.html#create-a-commit-with-multiple-files-and-actions). +This allows you to delete a single file. For deleting multiple files with a single request, see the [commits API](commits.html#create-a-commit-with-multiple-files-and-actions). ``` DELETE /projects/:id/repository/files/:file_path diff --git a/doc/api/repository_storage_health.md b/doc/api/repository_storage_health.md deleted file mode 100644 index edf4b04acea..00000000000 --- a/doc/api/repository_storage_health.md +++ /dev/null @@ -1,5 +0,0 @@ -# Circuitbreaker API - -NOTE: **Deprecated:** -Support of the circuit breaker is removed, as Gitaly can be configured to -to work without NFS and [communicate solely over HTTP](../administration/gitaly/index.md). diff --git a/doc/api/resource_label_events.md b/doc/api/resource_label_events.md index e1f9ffa9472..f0a7ac4e41d 100644 --- a/doc/api/resource_label_events.md +++ b/doc/api/resource_label_events.md @@ -88,6 +88,92 @@ Parameters: curl --request GET --header "PRIVATE-TOKEN: <your_access_token>" https://gitlab.example.com/api/v4/projects/5/issues/11/resource_label_events/1 ``` +## Epics **[ULTIMATE]** + +### List group epic label events + +Gets a list of all label events for a single epic. + +``` +GET /groups/:id/epics/:epic_id/resource_label_events +``` + +| Attribute | Type | Required | Description | +| ------------------- | ---------------- | ---------- | ------------ | +| `id` | integer/string | yes | The ID or [URL-encoded path of the group](README.md#namespaced-path-encoding) | +| `epic_id` | integer | yes | The ID of an epic | + +```json +[ + { + "id": 106, + "user": { + "id": 1, + "name": "Administrator", + "username": "root", + "state": "active", + "avatar_url": "https://www.gravatar.com/avatar/e64c7d89f26bd1972efa854d13d7dd61?s=80&d=identicon", + "web_url": "http://gitlab.example.com/root" + }, + "created_at": "2018-08-19T11:43:01.746Z", + "resource_type": "Epic", + "resource_id": 33, + "label": { + "id": 73, + "name": "a1", + "color": "#34495E", + "description": "" + }, + "action": "add" + }, + { + "id": 107, + "user": { + "id": 1, + "name": "Administrator", + "username": "root", + "state": "active", + "avatar_url": "https://www.gravatar.com/avatar/e64c7d89f26bd1972efa854d13d7dd61?s=80&d=identicon", + "web_url": "http://gitlab.example.com/root" + }, + "created_at": "2018-08-19T11:43:01.746Z", + "resource_type": "Epic", + "resource_id": 33, + "label": { + "id": 37, + "name": "glabel2", + "color": "#A8D695", + "description": "" + }, + "action": "add" + } +] +``` + +```bash +curl --request GET --header "PRIVATE-TOKEN: <your_access_token>" https://gitlab.example.com/api/v4/groups/1/epics/11/resource_label_events +``` + +### Get single epic label event + +Returns a single label event for a specific group epic + +``` +GET /groups/:id/epics/:epic_id/resource_label_events/:resource_label_event_id +``` + +Parameters: + +| Attribute | Type | Required | Description | +| --------------- | -------------- | -------- | ----------- | +| `id` | integer/string | yes | The ID or [URL-encoded path of the group](README.md#namespaced-path-encoding) | +| `epic_id` | integer | yes | The ID of an epic | +| `resource_label_event_id` | integer | yes | The ID of a label event | + +```bash +curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" https://gitlab.example.com/api/v4/groups/1/epics/11/resource_label_events/107 +``` + ## Merge requests ### List project merge request label events diff --git a/doc/api/runners.md b/doc/api/runners.md index 35c18649fec..2d91428d1c1 100644 --- a/doc/api/runners.md +++ b/doc/api/runners.md @@ -4,6 +4,29 @@ [ce-2640]: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/2640 +## Registration and authentication tokens + +There are two tokens to take into account when connecting a Runner with GitLab. + +| Token | Description | +| ----- | ----------- | +| Registration token | Token used to [register the Runner](https://docs.gitlab.com/runner/register/). It can be [obtained through GitLab](../ci/runners/README.md). | +| Authentication token | Token used to authenticate the Runner with the GitLab instance. It is obtained either automatically when [registering a Runner](https://docs.gitlab.com/runner/register/), or manually when [registering the Runner via the Runners API](#register-a-new-runner). | + +Here's an example of how the two tokens are used in Runner registration: + +1. You register the Runner via the GitLab API using a registration token, and an + authentication token is returned. +1. You use that authentication token and add it to the + [Runner's configuration file](https://docs.gitlab.com/runner/commands/#configuration-file): + + ```toml + [[runners]] + token = "<authentication_token>" + ``` + +GitLab and Runner are then connected. + ## List owned runners Get a list of specific runners available to the user. @@ -13,13 +36,15 @@ GET /runners GET /runners?scope=active GET /runners?type=project_type GET /runners?status=active +GET /runners?tag_list=tag1,tag2 ``` -| Attribute | Type | Required | Description | -|-----------|---------|----------|---------------------| -| `scope` | string | no | Deprecated: Use `type` or `status` instead. The scope of specific runners to show, one of: `active`, `paused`, `online`, `offline`; showing all runners if none provided | -| `type` | string | no | The type of runners to show, one of: `instance_type`, `group_type`, `project_type` | -| `status` | string | no | The status of runners to show, one of: `active`, `paused`, `online`, `offline` | +| Attribute | Type | Required | Description | +|-------------|----------------|----------|---------------------| +| `scope` | string | no | Deprecated: Use `type` or `status` instead. The scope of specific runners to show, one of: `active`, `paused`, `online`, `offline`; showing all runners if none provided | +| `type` | string | no | The type of runners to show, one of: `instance_type`, `group_type`, `project_type` | +| `status` | string | no | The status of runners to show, one of: `active`, `paused`, `online`, `offline` | +| `tag_list` | Array[String] | no | List of of the runner's tags | ``` curl --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/runners" @@ -62,13 +87,15 @@ GET /runners/all GET /runners/all?scope=online GET /runners/all?type=project_type GET /runners/all?status=active +GET /runners/all?tag_list=tag1,tag2 ``` -| Attribute | Type | Required | Description | -|-----------|---------|----------|---------------------| -| `scope` | string | no | Deprecated: Use `type` or `status` instead. The scope of runners to show, one of: `specific`, `shared`, `active`, `paused`, `online`, `offline`; showing all runners if none provided | -| `type` | string | no | The type of runners to show, one of: `instance_type`, `group_type`, `project_type` | -| `status` | string | no | The status of runners to show, one of: `active`, `paused`, `online`, `offline` | +| Attribute | Type | Required | Description | +|-------------|----------------|----------|---------------------| +| `scope` | string | no | Deprecated: Use `type` or `status` instead. The scope of runners to show, one of: `specific`, `shared`, `active`, `paused`, `online`, `offline`; showing all runners if none provided | +| `type` | string | no | The type of runners to show, one of: `instance_type`, `group_type`, `project_type` | +| `status` | string | no | The status of runners to show, one of: `active`, `paused`, `online`, `offline` | +| `tag_list` | Array[String] | no | List of of the runner's tags | ``` curl --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/runners/all" @@ -252,6 +279,8 @@ curl --request DELETE --header "PRIVATE-TOKEN: <your_access_token>" "https://git ## List runner's jobs +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/15432) in GitLab 10.3. + List jobs that are being processed or were processed by specified Runner. ``` @@ -347,14 +376,16 @@ GET /projects/:id/runners GET /projects/:id/runners?scope=active GET /projects/:id/runners?type=project_type GET /projects/:id/runners?status=active +GET /projects/:id/runners?tag_list=tag1,tag2 ``` -| Attribute | Type | Required | Description | -|-----------|----------------|----------|---------------------| -| `id` | integer/string | yes | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding) owned by the authenticated user | -| `scope` | string | no | Deprecated: Use `type` or `status` instead. The scope of specific runners to show, one of: `active`, `paused`, `online`, `offline`; showing all runners if none provided | -| `type` | string | no | The type of runners to show, one of: `instance_type`, `group_type`, `project_type` | -| `status` | string | no | The status of runners to show, one of: `active`, `paused`, `online`, `offline` | +| Attribute | Type | Required | Description | +|------------|----------------|----------|---------------------| +| `id` | integer/string | yes | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding) owned by the authenticated user | +| `scope` | string | no | Deprecated: Use `type` or `status` instead. The scope of specific runners to show, one of: `active`, `paused`, `online`, `offline`; showing all runners if none provided | +| `type` | string | no | The type of runners to show, one of: `instance_type`, `group_type`, `project_type` | +| `status` | string | no | The status of runners to show, one of: `active`, `paused`, `online`, `offline` | +| `tag_list` | Array[String] | no | List of of the runner's tags | ``` curl --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/9/runners" @@ -448,17 +479,18 @@ POST /runners | Attribute | Type | Required | Description | |-------------|---------|----------|---------------------| -| `token` | string | yes | Registration token ([Read how to obtain a token](../ci/runners/README.md)) | +| `token` | string | yes | [Registration token](#registration-and-authentication-tokens). | | `description`| string | no | Runner's description| | `info` | hash | no | Runner's metadata | | `active` | boolean| no | Whether the Runner is active | | `locked` | boolean| no | Whether the Runner should be locked for current project | | `run_untagged` | boolean | no | Whether the Runner should handle untagged jobs | | `tag_list` | Array[String] | no | List of Runner's tags | +| `access_level` | string | no | The access_level of the runner; `not_protected` or `ref_protected` | | `maximum_timeout` | integer | no | Maximum timeout set when this Runner will handle the job | ``` -curl --request POST "https://gitlab.example.com/api/v4/runners" --form "token=ipzXrMhuyyJPifUt6ANz" --form "description=test-1-20150125-test" --form "tag_list=ruby,mysql,tag1,tag2" +curl --request POST "https://gitlab.example.com/api/v4/runners" --form "token=<registration_token>" --form "description=test-1-20150125-test" --form "tag_list=ruby,mysql,tag1,tag2" ``` Response: @@ -486,10 +518,10 @@ DELETE /runners | Attribute | Type | Required | Description | |-------------|---------|----------|---------------------| -| `token` | string | yes | Runner's authentication token | +| `token` | string | yes | Runner's [authentication token](#registration-and-authentication-tokens). | ``` -curl --request DELETE "https://gitlab.example.com/api/v4/runners" --form "token=ebb6fc00521627750c8bb750f2490e" +curl --request DELETE "https://gitlab.example.com/api/v4/runners" --form "token=<authentication_token>" ``` Response: @@ -508,10 +540,10 @@ POST /runners/verify | Attribute | Type | Required | Description | |-------------|---------|----------|---------------------| -| `token` | string | yes | Runner's authentication token | +| `token` | string | yes | Runner's [authentication token](#registration-and-authentication-tokens). | ``` -curl --request POST "https://gitlab.example.com/api/v4/runners/verify" --form "token=ebb6fc00521627750c8bb750f2490e" +curl --request POST "https://gitlab.example.com/api/v4/runners/verify" --form "token=<authentication_token>" ``` Response: diff --git a/doc/api/scim.md b/doc/api/scim.md new file mode 100644 index 00000000000..3870ea788e7 --- /dev/null +++ b/doc/api/scim.md @@ -0,0 +1,235 @@ +# SCIM API **[SILVER ONLY]** + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/9388) in [GitLab Silver](https://about.gitlab.com/pricing/) 11.10. + +The SCIM API implements the [the RFC7644 protocol](https://tools.ietf.org/html/rfc7644). + +NOTE: **Note:** +[Group SSO](../user/group/saml_sso/index.md) and the feature +flag `:group_scim` must be enabled for the group. For more information, see [SCIM setup documentation](../user/group/saml_sso/scim_setup.md#requirements). + +## Get a list of SAML users + +NOTE: **Note:** +This endpoint is used as part of the SCIM syncing mechanism and it only returns +a single user based on a unique ID which should match the `extern_uid` of the user. + +```text +GET /api/scim/v2/groups/:group_path/Users +``` + +Parameters: + +| Attribute | Type | Required | Description | +|:----------|:--------|:---------|:----------------------------------------------------------------------------------------------------------------------------------------| +| `filter` | string | yes | A [filter](#available-filters) expression. | +| `group_path` | string | yes | Full path to the group. | + +Example request: + +```sh +curl 'https://example.gitlab.com/api/scim/v2/groups/test_group/Users?filter=id%20eq%20"0b1d561c-21ff-4092-beab-8154b17f82f2"' --header "Authorization: Bearer <your_scim_token>" --header "Content-Type: application/scim+json" +``` + +Example response: + +```json +{ + "schemas": [ + "urn:ietf:params:scim:api:messages:2.0:ListResponse" + ], + "totalResults": 1, + "itemsPerPage": 20, + "startIndex": 1, + "Resources": [ + { + "schemas": [ + "urn:ietf:params:scim:schemas:core:2.0:User" + ], + "id": "0b1d561c-21ff-4092-beab-8154b17f82f2", + "active": true, + "name.formatted": "Test User", + "userName": "username", + "meta": { "resourceType":"User" }, + "emails": [ + { + "type": "work", + "value": "name@example.com", + "primary": true + } + ] + } + ] +} +``` + +## Get a single SAML user + +```text +GET /api/scim/v2/groups/:group_path/Users/:id +``` + +Parameters: + +| Attribute | Type | Required | Description | +|:----------|:--------|:---------|:----------------------------------------------------------------------------------------------------------------------------------------| +| `id` | string | yes | External UID of the user. | +| `group_path` | string | yes | Full path to the group. | + +Example request: + +```sh +curl 'https://example.gitlab.com/api/scim/v2/groups/test_group/Users/f0b1d561c-21ff-4092-beab-8154b17f82f2' --header "Authorization: Bearer <your_scim_token>" --header "Content-Type: application/scim+json" +``` + +Example response: + +```json +{ + "schemas": [ + "urn:ietf:params:scim:schemas:core:2.0:User" + ], + "id": "0b1d561c-21ff-4092-beab-8154b17f82f2", + "active": true, + "name.formatted": "Test User", + "userName": "username", + "meta": { "resourceType":"User" }, + "emails": [ + { + "type": "work", + "value": "name@example.com", + "primary": true + } + ] +} +``` + +## Create a SAML user + +```text +POST /api/scim/v2/groups/:group_path/Users/ +``` + +Parameters: + +| Attribute | Type | Required | Description | +|:---------------|:----------|:----|:--------------------------| +| `externalId` | string | yes | External UID of the user. | +| `userName` | string | yes | Username of the user. | +| `emails` | JSON string | yes | Work email. | +| `name` | JSON string | yes | Name of the user. | +| `meta` | string | no | Resource type (`User'). | + +Example request: + +```sh +curl --verbose --request POST 'https://example.gitlab.com/api/scim/v2/groups/test_group/Users' --data '{"externalId":"test_uid","active":null,"userName":"username","emails":[{"primary":true,"type":"work","value":"name@example.com"}],"name":{"formatted":"Test User","familyName":"User","givenName":"Test"},"schemas":["urn:ietf:params:scim:schemas:core:2.0:User"],"meta":{"resourceType":"User"}}' --header "Authorization: Bearer <your_scim_token>" --header "Content-Type: application/scim+json" +``` + +Example response: + +```json +{ + "schemas": [ + "urn:ietf:params:scim:schemas:core:2.0:User" + ], + "id": "0b1d561c-21ff-4092-beab-8154b17f82f2", + "active": true, + "name.formatted": "Test User", + "userName": "username", + "meta": { "resourceType":"User" }, + "emails": [ + { + "type": "work", + "value": "name@example.com", + "primary": true + } + ] +} +``` + +Returns a `201` status code if successful. + +## Update a single SAML user + +Fields that can be updated are: + +| SCIM/IdP field | GitLab field | +|:----------|:--------| +| id/externalId | extern_uid | +| name.formatted | name | +| emails\[type eq "work"\].value | email | +| active | Identity removal if `active = false` | +| userName | username | + +```text +PATCH /api/scim/v2/groups/:group_path/Users/:id +``` + +Parameters: + +| Attribute | Type | Required | Description | +|:----------|:--------|:---------|:----------------------------------------------------------------------------------------------------------------------------------------| +| `id` | string | yes | External UID of the user. | +| `group_path` | string | yes | Full path to the group. | +| `Operations` | JSON string | yes | An [operations](#available-operations) expression. | + +Example request: + +```sh +curl --verbose --request PATCH 'https://example.gitlab.com/api/scim/v2/groups/test_group/Users/f0b1d561c-21ff-4092-beab-8154b17f82f2' --data '{ "Operations": [{"op":"Add","path":"name.formatted","value":"New Name"}] }' --header "Authorization: Bearer <your_scim_token>" --header "Content-Type: application/scim+json" +``` + +Returns an empty response with a `204` status code if successful. + +## Remove a single SAML user + +Removes the user's SSO identity and group membership. + +```text +DELETE /api/scim/v2/groups/:group_path/Users/:id +``` + +Parameters: + +| Attribute | Type | Required | Description | +|:----------|:--------|:---------|:----------------------------------------------------------------------------------------------------------------------------------------| +| `id` | string | yes | External UID of the user. | +| `group_path` | string | yes | Full path to the group. | + +Example request: + +```sh +curl --verbose --request DELETE 'https://example.gitlab.com/api/scim/v2/groups/test_group/Users/f0b1d561c-21ff-4092-beab-8154b17f82f2' --header "Authorization: Bearer <your_scim_token>" --header "Content-Type: application/scim+json" +``` + +Returns an empty response with a `204` status code if successful. + +## Available filters + +They match an expression as specified in [the RFC7644 filtering section](https://tools.ietf.org/html/rfc7644#section-3.4.2.2). + +| Filter | Description | +| ----- | ----------- | +| `eq` | The attribute matches exactly the specified value. | + +Example: + +``` +id eq a-b-c-d +``` + +## Available operations + +They perform an operation as specified in [the RFC7644 update section](https://tools.ietf.org/html/rfc7644#section-3.5.2). + +| Operator | Description | +| ----- | ----------- | +| `Replace` | The attribute's value is updated. | +| `Add` | The attribute has a new value. | + +Example: + +```json +{ "op": "Add", "path": "name.formatted", "value": "New Name" } +``` diff --git a/doc/api/search.md b/doc/api/search.md index aa601648b2c..da81c8321c9 100644 --- a/doc/api/search.md +++ b/doc/api/search.md @@ -17,7 +17,9 @@ GET /search | `scope` | string | yes | The scope to search in | | `search` | string | yes | The search query | -Search the expression within the specified scope. Currently these scopes are supported: projects, issues, merge_requests, milestones, snippet_titles, snippet_blobs. +Search the expression within the specified scope. Currently these scopes are supported: projects, issues, merge_requests, milestones, snippet_titles, snippet_blobs, users. + +If Elasticsearch is enabled additional scopes available are blobs, wiki_blobs and commits. Find more about [the feature](../integration/elasticsearch.md). **[STARTER]** The response depends on the requested scope. @@ -253,7 +255,7 @@ Example response: ### Scope: snippet_blobs ```bash -curl --request GET --header "PRIVATE-TOKEN: <your_access_token>" https://gitlab.example.com/api/v4/search?scope=snippet_blos&search=test +curl --request GET --header "PRIVATE-TOKEN: <your_access_token>" https://gitlab.example.com/api/v4/search?scope=snippet_blobs&search=test ``` Example response: @@ -281,6 +283,119 @@ Example response: ] ``` +### Scope: wiki_blobs **[STARTER]** + +This scope is available only if [Elasticsearch](../integration/elasticsearch.md) is enabled. + +```bash +curl --request GET --header "PRIVATE-TOKEN: <your_access_token>" https://gitlab.example.com/api/v4/search?scope=wiki_blobs&search=bye +``` + +Example response: + +```json + +[ + { + "basename": "home", + "data": "hello\n\nand bye\n\nend", + "filename": "home.md", + "id": null, + "ref": "master", + "startline": 5, + "project_id": 6 + } +] +``` + +### Scope: commits **[STARTER]** + +This scope is available only if [Elasticsearch](../integration/elasticsearch.md) is enabled. + +```bash +curl --request GET --header "PRIVATE-TOKEN: <your_access_token>" https://gitlab.example.com/api/v4/search?scope=commits&search=bye +``` + +Example response: + +```json + +[ + { + "id": "4109c2d872d5fdb1ed057400d103766aaea97f98", + "short_id": "4109c2d8", + "title": "goodbye $.browser", + "created_at": "2013-02-18T22:02:54.000Z", + "parent_ids": [ + "59d05353ab575bcc2aa958fe1782e93297de64c9" + ], + "message": "goodbye $.browser\n", + "author_name": "angus croll", + "author_email": "anguscroll@gmail.com", + "authored_date": "2013-02-18T22:02:54.000Z", + "committer_name": "angus croll", + "committer_email": "anguscroll@gmail.com", + "committed_date": "2013-02-18T22:02:54.000Z", + "project_id": 6 + } +] +``` + +### Scope: blobs **[STARTER]** + +This scope is available only if [Elasticsearch](../integration/elasticsearch.md) is enabled. + +Filters are available for this scope: +- filename +- path +- extension + +to use a filter simply include it in your query like so: `a query filename:some_name*`. + +You may use wildcards (`*`) to use glob matching. + +```bash +curl --request GET --header "PRIVATE-TOKEN: <your_access_token>" https://gitlab.example.com/api/v4/search?scope=blobs&search=installation +``` + +Example response: + +```json + +[ + { + "basename": "README", + "data": "```\n\n## Installation\n\nQuick start using the [pre-built", + "filename": "README.md", + "id": null, + "ref": "master", + "startline": 46, + "project_id": 6 + } +] +``` + +### Scope: users + +```bash +curl --request GET --header "PRIVATE-TOKEN: <your_access_token>" https://gitlab.example.com/api/v4/search?scope=users&search=doe +``` + +Example response: + +```json +[ + { + "id": 1, + "name": "John Doe1", + "username": "user1", + "state": "active", + "avatar_url": "http://www.gravatar.com/avatar/c922747a93b40d1ea88262bf1aebee62?s=80&d=identicon", + "web_url": "http://localhost/user1" + } +] +``` + ## Group Search API Search within the specified group. @@ -297,7 +412,9 @@ GET /groups/:id/search | `scope` | string | yes | The scope to search in | | `search` | string | yes | The search query | -Search the expression within the specified scope. Currently these scopes are supported: projects, issues, merge_requests, milestones. +Search the expression within the specified scope. Currently these scopes are supported: projects, issues, merge_requests, milestones, users. + +If Elasticsearch is enabled additional scopes available are blobs, wiki_blobs and commits. Find more about [the feature](../integration/elasticsearch.md). **[STARTER]** The response depends on the requested scope. @@ -499,6 +616,119 @@ Example response: ] ``` +### Scope: wiki_blobs **[STARTER]** + +This scope is available only if [Elasticsearch](../integration/elasticsearch.md) is enabled. + +```bash +curl --request GET --header "PRIVATE-TOKEN: <your_access_token>" https://gitlab.example.com/api/v4/groups/6/search?scope=wiki_blobs&search=bye +``` + +Example response: + +```json + +[ + { + "basename": "home", + "data": "hello\n\nand bye\n\nend", + "filename": "home.md", + "id": null, + "ref": "master", + "startline": 5, + "project_id": 6 + } +] +``` + +### Scope: commits **[STARTER]** + +This scope is available only if [Elasticsearch](../integration/elasticsearch.md) is enabled. + +```bash +curl --request GET --header "PRIVATE-TOKEN: <your_access_token>" https://gitlab.example.com/api/v4/groups/6/search?scope=commits&search=bye +``` + +Example response: + +```json + +[ + { + "id": "4109c2d872d5fdb1ed057400d103766aaea97f98", + "short_id": "4109c2d8", + "title": "goodbye $.browser", + "created_at": "2013-02-18T22:02:54.000Z", + "parent_ids": [ + "59d05353ab575bcc2aa958fe1782e93297de64c9" + ], + "message": "goodbye $.browser\n", + "author_name": "angus croll", + "author_email": "anguscroll@gmail.com", + "authored_date": "2013-02-18T22:02:54.000Z", + "committer_name": "angus croll", + "committer_email": "anguscroll@gmail.com", + "committed_date": "2013-02-18T22:02:54.000Z", + "project_id": 6 + } +] +``` + +### Scope: blobs **[STARTER]** + +This scope is available only if [Elasticsearch](../integration/elasticsearch.md) is enabled. + +Filters are available for this scope: +- filename +- path +- extension + +to use a filter simply include it in your query like so: `a query filename:some_name*`. + +You may use wildcards (`*`) to use glob matching. + +```bash +curl --request GET --header "PRIVATE-TOKEN: <your_access_token>" https://gitlab.example.com/api/v4/groups/6/search?scope=blobs&search=installation +``` + +Example response: + +```json + +[ + { + "basename": "README", + "data": "```\n\n## Installation\n\nQuick start using the [pre-built", + "filename": "README.md", + "id": null, + "ref": "master", + "startline": 46, + "project_id": 6 + } +] +``` + +### Scope: users + +```bash +curl --request GET --header "PRIVATE-TOKEN: <your_access_token>" https://gitlab.example.com/api/v4/groups/3/search?scope=users&search=doe +``` + +Example response: + +```json +[ + { + "id": 1, + "name": "John Doe1", + "username": "user1", + "state": "active", + "avatar_url": "http://www.gravatar.com/avatar/c922747a93b40d1ea88262bf1aebee62?s=80&d=identicon", + "web_url": "http://localhost/user1" + } +] +``` + ## Project Search API Search within the specified project. @@ -515,7 +745,7 @@ GET /projects/:id/search | `scope` | string | yes | The scope to search in | | `search` | string | yes | The search query | -Search the expression within the specified scope. Currently these scopes are supported: issues, merge_requests, milestones, notes, wiki_blobs, commits, blobs. +Search the expression within the specified scope. Currently these scopes are supported: issues, merge_requests, milestones, notes, wiki_blobs, commits, blobs, users. The response depends on the requested scope. @@ -828,4 +1058,25 @@ Example response: ] ``` +### Scope: users + +```bash +curl --request GET --header "PRIVATE-TOKEN: <your_access_token>" https://gitlab.example.com/api/v4/projects/6/search?scope=users&search=doe +``` + +Example response: + +```json +[ + { + "id": 1, + "name": "John Doe1", + "username": "user1", + "state": "active", + "avatar_url": "http://www.gravatar.com/avatar/c922747a93b40d1ea88262bf1aebee62?s=80&d=identicon", + "web_url": "http://localhost/user1" + } +] +``` + [ce-41763]: https://gitlab.com/gitlab-org/gitlab-ce/issues/41763 diff --git a/doc/api/services.md b/doc/api/services.md index 9275a1ccda8..898cfad7254 100644 --- a/doc/api/services.md +++ b/doc/api/services.md @@ -449,6 +449,45 @@ Get Hangouts Chat service settings for a project. GET /projects/:id/services/hangouts-chat ``` +## HipChat + +Private group chat and IM + +### Create/Edit HipChat service + +Set HipChat service for a project. + +``` +PUT /projects/:id/services/hipchat +``` + +Parameters: + +| Parameter | Type | Required | Description | +| --------- | ---- | -------- | ----------- | +| `token` | string | true | Room token | +| `color` | string | false | The room color | +| `notify` | boolean | false | Enable notifications | +| `room` | string | false |Room name or ID | +| `api_version` | string | false | Leave blank for default (v2) | +| `server` | string | false | Leave blank for default. For example, `https://hipchat.example.com`. | + +### Delete HipChat service + +Delete HipChat service for a project. + +``` +DELETE /projects/:id/services/hipchat +``` + +### Get HipChat service settings + +Get HipChat service settings for a project. + +``` +GET /projects/:id/services/hipchat +``` + ## Irker (IRC gateway) Send IRC messages, on update, to a list of recipients through an Irker gateway. @@ -505,10 +544,9 @@ GET /projects/:id/services/jira Set JIRA service for a project. -> **Notes:** -> - Starting with GitLab 8.14, `api_url`, `issues_url`, `new_issue_url` and -> `project_url` are replaced by `project_key`, `url`. If you are using an -> older version, [follow this documentation][old-jira-api]. +> Starting with GitLab 8.14, `api_url`, `issues_url`, `new_issue_url` and +> `project_url` are replaced by `url`. If you are using an +> older version, [follow this documentation][old-jira-api]. ``` PUT /projects/:id/services/jira @@ -519,11 +557,11 @@ Parameters: | Parameter | Type | Required | Description | | --------- | ---- | -------- | ----------- | | `url` | string | yes | The URL to the JIRA project which is being linked to this GitLab project. For example, `https://jira.example.com`. | -| `project_key` | string | yes | The short identifier for your JIRA project, all uppercase, e.g., `PROJ`. | +| `api_url` | string | no | The base URL to the JIRA instance API. Web URL value will be used if not set. For example, `https://jira-api.example.com`. | | `username` | string | yes | The username of the user created to be used with GitLab/JIRA. | | `password` | string | yes | The password of the user created to be used with GitLab/JIRA. | | `active` | boolean | no | Activates or deactivates the service. Defaults to false (deactivated). | -| `jira_issue_transition_id` | integer | no | The ID of a transition that moves issues to a closed state. You can find this number under the JIRA workflow administration (**Administration > Issues > Workflows**) by selecting **View** under **Operations** of the desired workflow of your project. The ID of each state can be found inside the parenthesis of each transition name under the **Transitions (id)** column ([see screenshot][trans]). By default, this ID is set to `2`. | +| `jira_issue_transition_id` | string | no | The ID of a transition that moves issues to a closed state. You can find this number under the JIRA workflow administration (**Administration > Issues > Workflows**) by selecting **View** under **Operations** of the desired workflow of your project. The ID of each state can be found inside the parenthesis of each transition name under the **Transitions (id)** column ([see screenshot][trans]). By default, this ID is set to `2`. | ### Delete JIRA service @@ -716,6 +754,7 @@ Parameters: | `recipients` | string | yes | Comma-separated list of recipient email addresses | | `add_pusher` | boolean | no | Add pusher to recipients list | | `notify_only_broken_pipelines` | boolean | no | Notify only broken pipelines | +| `notify_only_default_branch` | boolean | no | Send notifications only for the default branch ([introduced in GitLab 12.0](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/28271)) | ### Delete Pipeline-Emails service @@ -1062,6 +1101,75 @@ Get JetBrains TeamCity CI service settings for a project. GET /projects/:id/services/teamcity ``` +## Jenkins CI **[STARTER]** + +A continuous integration and build server + +### Create/Edit Jenkins CI service + +Set Jenkins CI service for a project. + +``` +PUT /projects/:id/services/jenkins +``` + +Parameters: + +- `jenkins_url` (**required**) - Jenkins URL like http://jenkins.example.com +- `project_name` (**required**) - The URL-friendly project name. Example: my_project_name +- `username` (optional) - A user with access to the Jenkins server, if applicable +- `password` (optional) - The password of the user + +### Delete Jenkins CI service + +Delete Jenkins CI service for a project. + +``` +DELETE /projects/:id/services/jenkins +``` + +### Get Jenkins CI service settings + +Get Jenkins CI service settings for a project. + +``` +GET /projects/:id/services/jenkins +``` + +## Jenkins CI (Deprecated) Service + +A continuous integration and build server + +### Create/Edit Jenkins CI (Deprecated) service + +Set Jenkins CI (Deprecated) service for a project. + +``` +PUT /projects/:id/services/jenkins-deprecated +``` + +Parameters: + +- `project_url` (**required**) - Jenkins project URL like http://jenkins.example.com/job/my-project/ +- `multiproject_enabled` (optional) - Multi-project mode is configured in Jenkins GitLab Hook plugin +- `pass_unstable` (optional) - Unstable builds will be treated as passing + +### Delete Jenkins CI (Deprecated) service + +Delete Jenkins CI (Deprecated) service for a project. + +``` +DELETE /projects/:id/services/jenkins-deprecated +``` + +### Get Jenkins CI (Deprecated) service settings + +Get Jenkins CI (Deprecated) service settings for a project. + +``` +GET /projects/:id/services/jenkins-deprecated +``` + [jira-doc]: ../user/project/integrations/jira.md [old-jira-api]: https://gitlab.com/gitlab-org/gitlab-ce/blob/8-13-stable/doc/api/services.md#jira @@ -1102,3 +1210,39 @@ GET /projects/:id/services/mock-ci ``` [11435]: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/11435 + +## YouTrack + +YouTrack issue tracker + +### Create/Edit YouTrack service + +Set YouTrack service for a project. + +``` +PUT /projects/:id/services/youtrack +``` + +Parameters: + +| Parameter | Type | Required | Description | +| --------- | ---- | -------- | ----------- | +| `issues_url` | string | true | Issue url | +| `project_url` | string | true | Project url | +| `description` | string | false | Description | + +### Delete YouTrack Service + +Delete YouTrack service for a project. + +``` +DELETE /projects/:id/services/youtrack +``` + +### Get YouTrack Service Settings + +Get YouTrack service settings for a project. + +``` +GET /projects/:id/services/youtrack +``` diff --git a/doc/api/settings.md b/doc/api/settings.md index 2e0a2a09133..c2a1f7feefd 100644 --- a/doc/api/settings.md +++ b/doc/api/settings.md @@ -161,7 +161,7 @@ are listed in the descriptions of the relevant settings. | `email_author_in_body` | boolean | no | Some email servers do not support overriding the email sender name. Enable this option to include the name of the author of the issue, merge request or comment in the email body instead. | | `enabled_git_access_protocol` | string | no | Enabled protocols for Git access. Allowed values are: `ssh`, `http`, and `nil` to allow both protocols. | | `enforce_terms` | boolean | no | (**If enabled, requires:** `terms`) Enforce application ToS to all users. | -| `first_day_of_week` | integer | no | Start day of the week for calendar views and date pickers. Valid values are `0` (default) for Sunday and `1` for Monday. | +| `first_day_of_week` | integer | no | Start day of the week for calendar views and date pickers. Valid values are `0` (default) for Sunday, `1` for Monday, and `6` for Saturday. | | `gitaly_timeout_default` | integer | no | Default Gitaly timeout, in seconds. This timeout is not enforced for git fetch/push operations or Sidekiq jobs. Set to `0` to disable timeouts. | | `gitaly_timeout_fast` | integer | no | Gitaly fast operation timeout, in seconds. Some Gitaly operations are expected to be fast. If they exceed this threshold, there may be a problem with a storage shard and 'failing fast' can help maintain the stability of the GitLab instance. Set to `0` to disable timeouts. | | `gitaly_timeout_medium` | integer | no | Medium Gitaly timeout, in seconds. This should be a value between the Fast and the Default timeout. Set to `0` to disable timeouts. | diff --git a/doc/api/snippets.md b/doc/api/snippets.md index f90447e124e..1ce0b1e7a62 100644 --- a/doc/api/snippets.md +++ b/doc/api/snippets.md @@ -165,15 +165,15 @@ Parameters: |:--------------|:-------|:---------|:---------------------------------------------------| | `title` | string | yes | Title of a snippet. | | `file_name` | string | yes | Name of a snippet file. | -| `content` | string | yes | Content of a snippet. | +| `code` | string | yes | Content of a snippet. | | `description` | string | no | Description of a snippet. | -| `visibility` | string | no | Snippet's [visibility](#snippet-visibility-level). | +| `visibility` | string | yes | Snippet's [visibility](#snippet-visibility-level). | Example request: ```sh curl --request POST \ - --data '{"title": "This is a snippet", "content": "Hello world", "description": "Hello World snippet", "file_name": "test.txt", "visibility": "internal" }' \ + --data '{"title": "This is a snippet", "code": "Hello world", "description": "Hello World snippet", "file_name": "test.txt", "visibility": "internal" }' \ --header 'Content-Type: application/json' \ --header "PRIVATE-TOKEN: valid_api_token" \ https://gitlab.example.com/api/v4/snippets @@ -222,14 +222,14 @@ Parameters: | `title` | string | no | Title of a snippet. | | `file_name` | string | no | Name of a snippet file. | | `description` | string | no | Description of a snippet. | -| `content` | string | no | Content of a snippet. | +| `code` | string | no | Content of a snippet. | | `visibility` | string | no | Snippet's [visibility](#snippet-visibility-level). | Example request: ```sh curl --request PUT \ - --data '{"title": "foo", "content": "bar"}' \ + --data '{"title": "foo", "code": "bar"}' \ --header 'Content-Type: application/json' \ --header "PRIVATE-TOKEN: valid_api_token" \ https://gitlab.example.com/api/v4/snippets/1 diff --git a/doc/api/suggestions.md b/doc/api/suggestions.md index e88d536282a..188989bc94e 100644 --- a/doc/api/suggestions.md +++ b/doc/api/suggestions.md @@ -24,8 +24,6 @@ Example response: ```json { "id": 36, - "from_original_line": 10, - "to_original_line": 10, "from_line": 10, "to_line": 10, "appliable": false, diff --git a/doc/api/users.md b/doc/api/users.md index b0977810120..47028c679b8 100644 --- a/doc/api/users.md +++ b/doc/api/users.md @@ -33,7 +33,7 @@ GET /users ] ``` -You can also search for users by email or username with: `/users?search=John` +You can also search for users by name or primary email using `?search=`. For example. `/users?search=John`. In addition, you can lookup users by username: @@ -256,7 +256,8 @@ Parameters: "can_create_project": true, "two_factor_enabled": true, "external": false, - "private_profile": false + "private_profile": false, + "highest_role":10 } ``` diff --git a/doc/api/vulnerabilities.md b/doc/api/vulnerabilities.md new file mode 100644 index 00000000000..390d0966244 --- /dev/null +++ b/doc/api/vulnerabilities.md @@ -0,0 +1,113 @@ +# Vulnerabilities API **[ULTIMATE]** + +Every API call to vulnerabilities must be authenticated. + +If a user is not a member of a project and the project is private, a `GET` +request on that project will result in a `404` status code. + +CAUTION: **Caution:** +This API is in an alpha stage and considered unstable. +The response payload may be subject to change or breakage +across GitLab releases. + +## Vulnerabilities pagination + +By default, `GET` requests return 20 results at a time because the API results +are paginated. + +Read more on [pagination](README.md#pagination). + +## List project vulnerabilities + +List all of a project's vulnerabilities. + +``` +GET /projects/:id/vulnerabilities +GET /projects/:id/vulnerabilities?report_type=sast +GET /projects/:id/vulnerabilities?report_type=container_scanning +GET /projects/:id/vulnerabilities?report_type=sast,dast +GET /projects/:id/vulnerabilities?scope=all +GET /projects/:id/vulnerabilities?scope=dismissed +GET /projects/:id/vulnerabilities?severity=high +GET /projects/:id/vulnerabilities?confidence=unknown,experimental +``` + +| Attribute | Type | Required | Description | +| ------------------- | ---------------- | ---------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| `id` | integer/string | yes | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding) owned by the authenticated user. | +| `report_type` | Array[string] | no | Returns vulnerabilities belonging to specified report type. Valid values: `sast`, `dast`, `dependency_scanning`, or `container_scanning`. | +| `scope` | string | no | Returns vulnerabilities for the given scope: `all` or `dismissed`. Defaults to `dismissed` | +| `severity` | Array[string] | no | Returns vulnerabilities belonging to specified severity level: `undefined`, `info`, `unknown`, `low`, `medium`, `high`, or `critical`. Defaults to all' | +| `confidence` | Array[string] | no | Returns vulnerabilities belonging to specified confidence level: `undefined`, `ignore`, `unknown`, `experimental`, `low`, `medium`, `high`, or `confirmed`. Defaults to all | + +```bash +curl --header "PRIVATE-TOKEN: <your_access_token>" https://gitlab.example.com/api/v4/projects/4/vulnerabilities +``` + +Example response: + +```json +[ + { + "id": null, + "report_type": "dependency_scanning", + "name": "Authentication bypass via incorrect DOM traversal and canonicalization in saml2-js", + "severity": "unknown", + "confidence": "undefined", + "scanner": { + "external_id": "gemnasium", + "name": "Gemnasium" + }, + "identifiers": [ + { + "external_type": "gemnasium", + "external_id": "9952e574-7b5b-46fa-a270-aeb694198a98", + "name": "Gemnasium-9952e574-7b5b-46fa-a270-aeb694198a98", + "url": "https://deps.sec.gitlab.com/packages/npm/saml2-js/versions/1.5.0/advisories" + }, + { + "external_type": "cve", + "external_id": "CVE-2017-11429", + "name": "CVE-2017-11429", + "url": "https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2017-11429" + } + ], + "project_fingerprint": "fa6f5b6c5d240b834ac5e901dc69f9484cef89ec", + "create_vulnerability_feedback_issue_path": "/tests/yarn-remediation-test/vulnerability_feedback", + "create_vulnerability_feedback_merge_request_path": "/tests/yarn-remediation-test/vulnerability_feedback", + "create_vulnerability_feedback_dismissal_path": "/tests/yarn-remediation-test/vulnerability_feedback", + "project": { + "id": 31, + "name": "yarn-remediation-test", + "full_path": "/tests/yarn-remediation-test", + "full_name": "tests / yarn-remediation-test" + }, + "dismissal_feedback": null, + "issue_feedback": null, + "merge_request_feedback": null, + "description": "Some XML DOM traversal and canonicalization APIs may be inconsistent in handling of comments within XML nodes. Incorrect use of these APIs by some SAML libraries results in incorrect parsing of the inner text of XML nodes such that any inner text after the comment is lost prior to cryptographically signing the SAML message. Text after the comment therefore has no impact on the signature on the SAML message.\r\n\r\nA remote attacker can modify SAML content for a SAML service provider without invalidating the cryptographic signature, which may allow attackers to bypass primary authentication for the affected SAML service provider.", + "links": [ + { + "url": "https://github.com/Clever/saml2/commit/3546cb61fd541f219abda364c5b919633609ef3d#diff-af730f9f738de1c9ad87596df3f6de84R279" + }, + { + "url": "https://www.kb.cert.org/vuls/id/475445" + }, + { + "url": "https://github.com/Clever/saml2/issues/127" + } + ], + "location": { + "file": "yarn.lock", + "dependency": { + "package": { + "name": "saml2-js" + }, + "version": "1.5.0" + } + }, + "solution": "Upgrade to fixed version.\r\n", + "blob_path": "/tests/yarn-remediation-test/blob/cc6c4a0778460455ae5d16ca7025ca9ca1ca75ac/yarn.lock" + } +] +``` diff --git a/doc/articles/artifactory_and_gitlab/index.md b/doc/articles/artifactory_and_gitlab/index.md index 6a590b53727..ed9fd135e7c 100644 --- a/doc/articles/artifactory_and_gitlab/index.md +++ b/doc/articles/artifactory_and_gitlab/index.md @@ -1 +1,5 @@ +--- +redirect_to: '../../ci/examples/artifactory_and_gitlab/index.md' +--- + This document was moved to [another location](../../ci/examples/artifactory_and_gitlab/index.md) diff --git a/doc/articles/how_to_configure_ldap_gitlab_ce/index.md b/doc/articles/how_to_configure_ldap_gitlab_ce/index.md index a8320c12e6b..8e2e54711e7 100644 --- a/doc/articles/how_to_configure_ldap_gitlab_ce/index.md +++ b/doc/articles/how_to_configure_ldap_gitlab_ce/index.md @@ -1 +1,5 @@ +--- +redirect_to: '../../administration/auth/how_to_configure_ldap_gitlab_ce/index.md' +--- + This document was moved to [another location](../../administration/auth/how_to_configure_ldap_gitlab_ce/index.md). diff --git a/doc/articles/how_to_configure_ldap_gitlab_ee/index.md b/doc/articles/how_to_configure_ldap_gitlab_ee/index.md new file mode 100644 index 00000000000..3e6f3130437 --- /dev/null +++ b/doc/articles/how_to_configure_ldap_gitlab_ee/index.md @@ -0,0 +1,5 @@ +--- +redirect_to: '../../administration/auth/how_to_configure_ldap_gitlab_ee/index.md' +--- + +This document was moved to [another location](../../administration/auth/how_to_configure_ldap_gitlab_ee/index.md). diff --git a/doc/articles/how_to_install_git/index.md b/doc/articles/how_to_install_git/index.md index 3e6003a33b7..62598101895 100644 --- a/doc/articles/how_to_install_git/index.md +++ b/doc/articles/how_to_install_git/index.md @@ -1 +1,5 @@ +--- +redirect_to: '../../topics/git/how_to_install_git/index.md' +--- + This document was moved to [another location](../../topics/git/how_to_install_git/index.md). diff --git a/doc/articles/index.md b/doc/articles/index.md index 87ee17bb6de..162db11d6ac 100644 --- a/doc/articles/index.md +++ b/doc/articles/index.md @@ -4,8 +4,8 @@ comments: false # Technical articles list (deprecated) -[Technical articles](../development/documentation/index.md#technical-articles) are -topic-related documentation, written with an user-friendly approach and language, aiming +Technical articles are +topic-related documentation, written with a user-friendly approach and language, aiming to provide the community with guidance on specific processes to achieve certain objectives. The list of technical articles was [deprecated](https://gitlab.com/gitlab-org/gitlab-ce/issues/41138) in favor of having them linked from their topic-related documentation: diff --git a/doc/articles/laravel_with_gitlab_and_envoy/index.md b/doc/articles/laravel_with_gitlab_and_envoy/index.md index b092cdb0f7a..fa4f6243410 100644 --- a/doc/articles/laravel_with_gitlab_and_envoy/index.md +++ b/doc/articles/laravel_with_gitlab_and_envoy/index.md @@ -1 +1,5 @@ +--- +redirect_to: '../../ci/examples/laravel_with_gitlab_and_envoy/index.md' +--- + This document was moved to [another location](../../ci/examples/laravel_with_gitlab_and_envoy/index.md). diff --git a/doc/articles/numerous_undo_possibilities_in_git/index.md b/doc/articles/numerous_undo_possibilities_in_git/index.md index 3f46ee9a5e6..83aac82db4e 100644 --- a/doc/articles/numerous_undo_possibilities_in_git/index.md +++ b/doc/articles/numerous_undo_possibilities_in_git/index.md @@ -1 +1,5 @@ +--- +redirect_to: '../../topics/git/numerous_undo_possibilities_in_git/index.md' +--- + This document was moved to [another location](../../topics/git/numerous_undo_possibilities_in_git/index.md). diff --git a/doc/articles/runner_autoscale_aws/index.md b/doc/articles/runner_autoscale_aws/index.md index e2667aebc5f..fb769731256 100644 --- a/doc/articles/runner_autoscale_aws/index.md +++ b/doc/articles/runner_autoscale_aws/index.md @@ -1 +1,5 @@ +--- +redirect_to: 'https://docs.gitlab.com/runner/configuration/runner_autoscale_aws/index.html' +--- + This document was moved to [another location](https://docs.gitlab.com/runner/configuration/runner_autoscale_aws/index.html). diff --git a/doc/ci/README.md b/doc/ci/README.md index 4c3c9920b93..635cce13b4e 100644 --- a/doc/ci/README.md +++ b/doc/ci/README.md @@ -1,114 +1,140 @@ --- comments: false description: "Learn how to use GitLab CI/CD, the GitLab built-in Continuous Integration, Continuous Deployment, and Continuous Delivery toolset to build, test, and deploy your application." +type: index --- -# GitLab Continuous Integration (GitLab CI/CD) +# GitLab CI/CD -GitLab provides tools for continuously integrating and delivering code. +GitLab CI/CD is a tool built into GitLab for software development +through the [continuous methodologies](introduction/index.md#introduction-to-cicd-methodologies): -Within the [entire DevOps lifecycle](../README.md#the-entire-devops-lifecycle), GitLab CI/CD spans -the [Verify (CI)](../README.md#verify) and [Release (CD)](../README.md#release) stages. +- Continuous Integration (CI) +- Continuous Delivery (CD) +- Continuous Deployment (CD) ## Overview -CI/CD is a vast area, so GitLab provides documentation for all levels of expertise. Consult the following table to find the right documentation for you: +Continuous Integration works by pushing small code chunks to your +application's code base hosted in a Git repository, and, to every +push, run a pipeline of scripts to build, test, and validate the +code changes before merging them into the main branch. -| Level of expertise | Resource | -|:------------------------------------|:------------------------------------------------------------------------------------------------------------------------------------------------------| -| New to the concepts of CI and CD | For a high-level overview, see the [GitLab Continuous Integration & Delivery](https://about.gitlab.com/product/continuous-integration/) product page. | -| Familiar with the purpose of CI/CD | Delve into GitLab CI/CD by continuing down the page, starting with our [introduction](#introduction). | -| Familiar with GitLab CI/CD concepts | After getting familiar with GitLab CI/CD, let us walk you through a simple example in our [quick start guide](quick_start/README.md). | -| A GitLab CI/CD expert | Jump straight to our [`.gitlab.yml`](yaml/README.md) reference. | +Continuous Delivery and Deployment consist of a step further CI, +deploying your application to production at every +push to the default branch of the repository. -## Introduction +These methodologies allow you to catch bugs and errors early in +the development cycle, ensuring that all the code deployed to +production complies with the code standards you established for +your app. -The following introduces the process of continuous integration (CI) and continuous delivery (CD): +For a complete overview of these methodologies and GitLab CI/CD, +read the [Introduction to CI/CD with GitLab](introduction/index.md). - +## Getting started -In this illustration: +GitLab CI/CD is configured by a file called `.gitlab-ci.yml` placed +at the repository's root. The scripts set in this file are executed +by the [GitLab Runner](https://docs.gitlab.com/runner/). -- New code is combined with existing code through a commit to a project's [repository](../user/project/repository/index.md). -- The newly combined code is sent to a CI [pipeline](pipelines.md) where: - - The code is [built](../user/project/pipelines/job_artifacts.md). - - Unit and integration tests are run over the built code. -- Assuming the build and tests are successful, a CD pipeline is triggered to allow for: - - Review using [Review Apps](review_apps/index.md). - - Deploying to configured [environments](environments.md). +To get started with GitLab CI/CD, we recommend you read through +the following documents: -The benefits of CI/CD are vast, allowing automation to be an integral part of your workflow for testing, building, deploying, and monitoring your code. +- [How GitLab CI/CD works](introduction/index.md#how-gitlab-cicd-works). +- [GitLab CI/CD basic workflow](introduction/index.md#basic-cicd-workflow). +- [Step-by-step guide for writing `.gitlab-ci.yml` for the first time](../user/project/pages/getting_started_part_four.md). -Because CI and CD with GitLab is broad topic with many possibilities, the rest of this section provides -links to topics and resources needed to make use of GitLab CI/CD. +You can also get started by using one of the +[`.gitlab-ci.yml` templates](https://gitlab.com/gitlab-org/gitlab-ce/tree/master/lib/gitlab/ci/templates) +available through the UI. You can use them by creating a new file, +choosing a template that suits your application, and adjusting it +to your needs: -## Essentials + -The following documentation provides the minimum required knowledge for making use of GitLab CI/CD: +For a broader overview, see the [CI/CD getting started](quick_start/README.md) guide. -| Topic | Description | -|:------------------------------------------------------------------------|:---------------------------------------------------------| -| [Getting started with GitLab CI/CD](quick_start/README.md) | Outlines the first steps for configuring GitLab CI/CD. | -| [Introduction to pipelines and jobs](pipelines.md) | Provides an overview of GitLab CI/CD and jobs. | -| [Configuration of your pipelines with `.gitlab-ci.yml`](yaml/README.md) | A comprehensive reference for the `.gitlab-ci.yml` file. | +Once you're familiar with how GitLab CI/CD works, see the +[`. gitlab-ci.yml` full reference](yaml/README.md) +for all the attributes you can set and use. NOTE: **Note:** -Familiarity with [GitLab Runner](https://docs.gitlab.com/runner/) is useful because it is -responsible for running the jobs in your CI/CD pipeline. On GitLab.com, shared Runners are enabled -by default so you don't need to set up anything to get started. - -### Auto DevOps - -An alternative to manually configuring CI/CD, GitLab supports [Auto DevOps](../topics/autodevops/index.md), -which: - -- Provides simplified setup and execution of CI/CD. -- Allows GitLab to automatically detect, build, test, deploy, and monitor your applications. - -## Basic usage - -With basic knowledge of how GitLab CI/CD works, the following documentation extends your knowledge -into more features: - -| Topic | Description | -|:-------------------------------------------------------------------------------------------------------|:---------------------------------------------------------------------------------| -| [CI/CD Variables](variables/README.md) | How environment variables can be configured and made available in pipelines. | -| [Where variables can be used](variables/where_variables_can_be_used.md) | A deeper look into where and how CI/CD variables can be used. | -| [User](../user/permissions.md#gitlab-ci) and [job](../user/permissions.md#job-permissions) permissions | Learn about the access levels a user can have for performing certain CI actions. | -| [Configuring GitLab Runners](runners/README.md) | Documentation for configuring [GitLab Runner](https://docs.gitlab.com/runner/). | - -## Advanced usage - -Once you get familiar with the basics of GitLab CI/CD, consult the following documentation to make -use of advanced features: - -| Topic | Description | -|:---------------------------------------------------------------------------------|:-----------------------------------------------------------------------------------------------------------------------------| -| [Introduction to environments and deployments](environments.md) | Learn how to separate your jobs into environments and use them for different purposes like testing, building and, deploying. | -| [Job artifacts](../user/project/pipelines/job_artifacts.md) | Learn about the output of jobs. | -| [Cache dependencies in GitLab CI/CD](caching/index.md) | Discover how to speed up pipelines using caching. | -| [Using Git submodules with GitLab CI](git_submodules.md) | How to run your CI jobs when using Git submodules. | -| [Pipelines for merge requests](merge_request_pipelines/index.md) | Create pipelines specifically for merge requests. | -| [Using SSH keys with GitLab CI/CD](ssh_keys/README.md) | Use SSH keys in your build environment. | -| [Triggering pipelines through the API](triggers/README.md) | Use the GitLab API to trigger a pipeline. | -| [Pipeline schedules](../user/project/pipelines/schedules.md) | Trigger pipelines on a schedule. | -| [Connecting GitLab with a Kubernetes cluster](../user/project/clusters/index.md) | Integrate one or more Kubernetes clusters to your project. | -| [ChatOps](chatops/README.md) | Trigger CI jobs from chat, with results sent back to the channel. | -| [Interactive web terminals](interactive_web_terminal/index.md) | Open an interactive web terminal to debug the running jobs. | - -### GitLab Pages - -GitLab CI/CD can be used to build and host static websites. For more information, see the -documentation on [GitLab Pages](../user/project/pages/index.md). +GitLab CI/CD and [shared runners](runners/README.md#shared-specific-and-group-runners) are enabled in GitLab.com and available for all users, limited only to the [user's pipelines quota](https://docs.gitlab.com/ee/user/admin_area/settings/continuous_integration.html#extra-shared-runners-pipeline-minutes-quota). + +## Configuration + +GitLab CI/CD supports numerous configuration options: + +| Configuration | Description | +|:--------------|:-------------| +| [Pipelines](pipelines.md) | Structure your CI/CD process through pipelines. | +| [Environment variables](variables/README.md) | Reuse values based on a variable/value key pair. | +| [Environments](environments.md) | Deploy your application to different environments (e.g., staging, production). | +| [Job artifacts](../user/project/pipelines/job_artifacts.md) | Output, use, and reuse job artifacts. | +| [Cache dependencies](caching/index.md) | Cache your dependencies for a faster execution. | +| [Schedule pipelines](../user/project/pipelines/schedules.md) | Schedule pipelines to run as often as you need. | +| [Custom path for `.gitlab-ci.yml`](../user/project/pipelines/settings.md#custom-ci-config-path) | Define a custom path for the CI/CD configuration file. | +| [Git submodules for CI/CD](git_submodules.md) | Configure jobs for using Git submodules.| +| [SSH keys for CI/CD](ssh_keys/README.md) | Using SSH keys in your CI pipelines. | +| [Pipelines triggers](triggers/README.md) | Trigger pipelines through the API. | +| [Integrate with Kubernetes clusters](../user/project/clusters/index.md) | Connect your project to Google Kubernetes Engine (GKE) or an existing Kubernetes cluster. | +| [GitLab Runner](https://docs.gitlab.com/runner/) | Configure your own GitLab Runners to execute your scripts. | +| [Optimize GitLab and Runner for large repositories](large_repositories/index.md) | Recommended strategies for handling large repos. | +| [`.gitlab-ci.yml` full reference](yaml/README.md) | All the attributes you can use with GitLab CI/CD. | + +Note that certain operations can only be performed according to the +[user](../user/permissions.md#gitlab-cicd-permissions) and [job](../user/permissions.md#job-permissions) permissions. + +## Feature set + +Use the vast GitLab CI/CD to easily configure it for specific purposes. +Its feature set is listed on the table below according to DevOps stages. + +| Feature | Description | +|:--------|:------------| +| **Configure** || +| [Auto DevOps](../topics/autodevops/index.md) | Set up your app's entire lifecycle. | +| [ChatOps](chatops/README.md) | Trigger CI jobs from chat, with results sent back to the channel. | +|---+---| +| **Verify** || +| [Browser Performance Testing](https://docs.gitlab.com/ee/user/project/merge_requests/browser_performance_testing.html) | Quickly determine the performance impact of pending code changes. | +| [CI services](services/README.md) | Link Docker containers with your base image.| +| [Code Quality](https://docs.gitlab.com/ee/user/project/merge_requests/code_quality.html) **[STARTER]** | Analyze your source code quality. | +| [GitLab CI/CD for external repositories](https://docs.gitlab.com/ee/ci/ci_cd_for_external_repos/) **[PREMIUM]** | Get the benefits of GitLab CI/CD combined with repositories in GitHub and BitBucket Cloud. | +| [Interactive Web Terminals](interactive_web_terminal/index.md) **[CORE ONLY]** | Open an interactive web terminal to debug the running jobs. | +| [JUnit tests](junit_test_reports.md) | Identify script failures directly on merge requests. | +| [Using Docker images](docker/using_docker_images.md) | Use GitLab and GitLab Runner with Docker to build and test applications. | +|---+---| +| **Release** || +| [Auto Deploy](../topics/autodevops/index.md#auto-deploy) | Deploy your application to a production environment in a Kubernetes cluster. | +| [Building Docker images](docker/using_docker_build.md) | Maintain Docker-based projects using GitLab CI/CD. | +| [Canary Deployments](https://docs.gitlab.com/ee/user/project/canary_deployments.html) **[PREMIUM]** | Ship features to only a portion of your pods and let a percentage of your user base to visit the temporarily deployed feature. | +| [Deploy Boards](https://docs.gitlab.com/ee/user/project/deploy_boards.html) **[PREMIUM]** | Check the current health and status of each CI/CD environment running on Kubernetes. | +| [Feature Flags](https://docs.gitlab.com/ee/user/project/operations/feature_flags.html) **[PREMIUM]** | Deploy your features behind Feature Flags. | +| [GitLab Pages](../user/project/pages/index.md) | Deploy static websites. | +| [GitLab Releases](../user/project/releases/index.md) | Add release notes to Git tags. | +| [Review Apps](review_apps/index.md) | Configure GitLab CI/CD to preview code changes. | +|---+---| +| **Secure** || +| [Container Scanning](https://docs.gitlab.com/ee/ci/examples/container_scanning.html) **[ULTIMATE]** | Check your Docker containers for known vulnerabilities.| +| [Dependency Scanning](https://docs.gitlab.com/ee/ci/examples/dependency_scanning.html) **[ULTIMATE]** | Analyze your dependencies for known vulnerabilities. | +| [License Management](https://docs.gitlab.com/ee/user/application_security/license_management/) **[ULTIMATE]** | Search your project dependencies for their licenses. | +| [Security Test reports](https://docs.gitlab.com/ee/user/project/merge_requests/#security-reports-ultimate) **[ULTIMATE]** | Check for app vulnerabilities. | ## Examples -Check out the [GitLab CI/CD examples](examples/README.md) for a collection of tutorials and guides on -setting up your CI/CD pipeline for various programming languages, frameworks, and operating systems. +GitLab provides examples of configuring GitLab CI/CD in the form of: + +- A collection of [examples and other resources](examples/README.md). +- Example projects that are available at the [`gitlab-examples`](https://gitlab.com/gitlab-examples) group. For example, see: + - [`multi-project-pipelines`](https://gitlab.com/gitlab-examples/multi-project-pipelines) for examples of implementing multi-project pipelines. + - [`review-apps-nginx`](https://gitlab.com/gitlab-examples/review-apps-nginx/) provides an example of using Review Apps. -## Administration +## Administration **[CORE ONLY]** -As a GitLab administrator, you can change the default behavior of GitLab CI/CD for: +As a GitLab administrator, you can change the default behavior +of GitLab CI/CD for: - An [entire GitLab instance](../user/admin_area/settings/continuous_integration.md). - Specific projects, using [pipelines settings](../user/project/pipelines/settings.md). @@ -118,80 +144,51 @@ See also: - [How to enable or disable GitLab CI/CD](enable_or_disable_ci.md). - Other [CI administration settings](../administration/index.md#continuous-integration-settings). -## Using Docker - -Docker is commonly used with GitLab CI/CD. Learn more about how to to accomplish this with the following -documentation: +## References -| Topic | Description | -|:-------------------------------------------------------------------------|:-------------------------------------------------------------------------| -| [Using Docker images](docker/using_docker_images.md) | Use GitLab and GitLab Runner with Docker to build and test applications. | -| [Building Docker images with GitLab CI/CD](docker/using_docker_build.md) | Maintain Docker-based projects using GitLab CI/CD. | - -Related topics include: - -- [Docker integration](docker/README.md). -- [CI services (linked Docker containers)](services/README.md). -- [Setting up GitLab Runner For Continuous Integration](https://about.gitlab.com/2016/03/01/gitlab-runner-with-docker/) (article). - -## Further resources - -This section provides further resources to help you get familiar with GitLab CI/CD. - -### Articles - -The following table provides a list of articles about CI/CD, sorted in reverse chronological order of publish date: - -| Publish Date | Article | -|:-------------|:----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| 2017-07-13 | [Making CI easier with GitLab](https://about.gitlab.com/2017/07/13/making-ci-easier-with-gitlab/). | -| 2017-05-22 | [Fast and natural continuous integration with GitLab CI](https://about.gitlab.com/2017/05/22/fast-and-natural-continuous-integration-with-gitlab-ci/). | -| 2016-11-22 | [Introducing Review Apps](https://about.gitlab.com/2016/11/22/introducing-review-apps/). | -| 2016-08-26 | [GitLab CI: Deployment & Environments](https://about.gitlab.com/2016/08/26/ci-deployment-and-environments/). | -| 2016-08-05 | [Continuous Integration, Delivery, and Deployment with GitLab](https://about.gitlab.com/2016/08/05/continuous-integration-delivery-and-deployment-with-gitlab/). | -| 2016-07-29 | [GitLab CI: Run jobs sequentially, in parallel or build a custom pipeline](https://about.gitlab.com/2016/07/29/the-basics-of-gitlab-ci/). | -| 2016-06-09 | [Continuous Delivery with GitLab and Convox](https://about.gitlab.com/2016/06/09/continuous-delivery-with-gitlab-and-convox/) | -| 2016-05-23 | [GitLab Container Registry](https://about.gitlab.com/2016/05/23/gitlab-container-registry/). | -| 2016-05-05 | [Getting Started with GitLab and Shippable Continuous Integration](https://about.gitlab.com/2016/05/05/getting-started-gitlab-and-shippable/) | -| 2016-04-19 | [GitLab Partners with DigitalOcean to make Continuous Integration faster, safer, and more affordable](https://about.gitlab.com/2016/04/19/gitlab-partners-with-digitalocean-to-make-continuous-integration-faster-safer-and-more-affordable/) | -| 2015-03-01 | [Setting up GitLab Runner For Continuous Integration](https://about.gitlab.com/2016/03/01/gitlab-runner-with-docker/). | -| 2015-12-14 | [Getting started with GitLab and GitLab CI](https://about.gitlab.com/2015/12/14/getting-started-with-gitlab-and-gitlab-ci/). | - -### Videos - -The following table provides a list of videos about CI/CD, sorted in reverse chronological order of publish date: +### Why GitLab CI/CD? -| Publish Date | Video | -|:-------------|:-------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| 2017-07-17 | [GitLab CI/CD Deep Dive](https://youtu.be/pBe4t1CD8Fc?t=195). | -| 2017-03-13 | [Demo: CI/CD with GitLab in action](https://about.gitlab.com/2017/03/13/ci-cd-demo/). | -| 2016-04-20 | [Webcast Recording and Slides: Getting started with CI in GitLab](https://about.gitlab.com/2016/04/20/webcast-recording-and-slides-introduction-to-ci-in-gitlab/). | +The following articles explain reasons to use GitLab CI/CD +for your CI/CD infrastructure: -In addition, the following third-party videos are available: +- [Why we chose GitLab CI for our CI/CD solution](https://about.gitlab.com/2016/10/17/gitlab-ci-oohlala/) +- [Building our web-app on GitLab CI](https://about.gitlab.com/2016/07/22/building-our-web-app-on-gitlab-ci/) -- [Intégration continue avec GitLab (September 2016)](https://www.youtube.com/watch?v=URcMBXjIr24&t=13s). -- [GitLab CI for Minecraft Plugins (July 2016)](https://www.youtube.com/watch?v=Z4pcI9F8yf8). +See also the [Why CI/CD?](https://docs.google.com/presentation/d/1OGgk2Tcxbpl7DJaIOzCX4Vqg3dlwfELC3u2jEeCBbDk) presentation. -### Example Projects +### Breaking changes -[`review-apps-nginx`](https://gitlab.com/gitlab-examples/review-apps-nginx/) provides an example of using Review Apps. +As GitLab CI/CD has evolved, certain breaking changes have +been necessary. These are: -Other example projects are available at the [`gitlab-examples`](https://gitlab.com/gitlab-examples) group. +#### 12.0 -### Why GitLab CI/CD? +- [Use refspec to clone/fetch git + repository](https://gitlab.com/gitlab-org/gitlab-runner/issues/4069). +- [Old cache + configuration](https://gitlab.com/gitlab-org/gitlab-runner/issues/4070). +- [Old metrics server + configuration](https://gitlab.com/gitlab-org/gitlab-runner/issues/4072). +- [Remove + `FF_K8S_USE_ENTRYPOINT_OVER_COMMAND`](https://gitlab.com/gitlab-org/gitlab-runner/issues/4073). +- [Remove Linux distributions that reach + EOL](https://gitlab.com/gitlab-org/gitlab-runner/merge_requests/1130). +- [Update command line API for helper + images](https://gitlab.com/gitlab-org/gitlab-runner/issues/4013). +- [Remove old `git clean` + flow](https://gitlab.com/gitlab-org/gitlab-runner/issues/4175). -The following articles explain reasons why you might use GitLab CI/CD for your CI/CD infrastructure: +#### 11.0 -- [Why we chose GitLab CI for our CI/CD solution](https://about.gitlab.com/2016/10/17/gitlab-ci-oohlala/). -- [Building our web-app on GitLab CI](https://about.gitlab.com/2016/07/22/building-our-web-app-on-gitlab-ci/). +- No breaking changes. -See also the [Why CI/CD?](https://docs.google.com/presentation/d/1OGgk2Tcxbpl7DJaIOzCX4Vqg3dlwfELC3u2jEeCBbDk) presentation. +#### 10.0 -## Breaking changes +- No breaking changes. -As GitLab CI/CD has evolved, certain breaking changes have been necessary. These are: +#### 9.0 -- [CI variables renaming for GitLab 9.0](variables/README.md#gitlab-90-renaming). Read about the +- [CI variables renaming for GitLab 9.0](variables/deprecated_variables.md#gitlab-90-renamed-variables). Read about the deprecated CI variables and what you should use for GitLab 9.0+. - [New CI job permissions model](../user/project/new_ci_build_permissions_model.md). See what changed in GitLab 8.12 and how that affects your jobs. diff --git a/doc/ci/autodeploy/index.md b/doc/ci/autodeploy/index.md index 985ec4b972c..5221cbf8609 100644 --- a/doc/ci/autodeploy/index.md +++ b/doc/ci/autodeploy/index.md @@ -1 +1,5 @@ +--- +redirect_to: '../../topics/autodevops/index.md#auto-deploy' +--- + This document was moved to [another location](../../topics/autodevops/index.md#auto-deploy). diff --git a/doc/ci/autodeploy/quick_start_guide.md b/doc/ci/autodeploy/quick_start_guide.md index 985ec4b972c..5221cbf8609 100644 --- a/doc/ci/autodeploy/quick_start_guide.md +++ b/doc/ci/autodeploy/quick_start_guide.md @@ -1 +1,5 @@ +--- +redirect_to: '../../topics/autodevops/index.md#auto-deploy' +--- + This document was moved to [another location](../../topics/autodevops/index.md#auto-deploy). diff --git a/doc/ci/build_artifacts/README.md b/doc/ci/build_artifacts/README.md index 22b3872025f..b63659c1878 100644 --- a/doc/ci/build_artifacts/README.md +++ b/doc/ci/build_artifacts/README.md @@ -1 +1,5 @@ +--- +redirect_to: '../../user/project/pipelines/job_artifacts.md' +--- + This document was moved to [pipelines/job_artifacts.md](../../user/project/pipelines/job_artifacts.md). diff --git a/doc/ci/caching/index.md b/doc/ci/caching/index.md index e079483e2b5..9a5a3624c73 100644 --- a/doc/ci/caching/index.md +++ b/doc/ci/caching/index.md @@ -1,3 +1,7 @@ +--- +type: index, concepts, howto +--- + # Cache dependencies in GitLab CI/CD GitLab CI/CD provides a caching mechanism that can be used to save time @@ -60,7 +64,7 @@ In summary: - Caches are disabled if not defined globally or per job (using `cache:`). - Caches are available for all jobs in your `.gitlab-ci.yml` if enabled globally. -- Caches can be used by subsequent pipelines of that very same job (a script in +- Caches can be used by subsequent pipelines of that same job (a script in a stage) in which the cache was created (if not defined globally). - Caches are stored where the Runner is installed **and** uploaded to S3 if [distributed cache is enabled](https://docs.gitlab.com/runner/configuration/autoscale.html#distributed-runners-caching). @@ -87,7 +91,7 @@ cache, when declaring `cache` in your jobs, use one or a mix of the following: that share their cache. - [Use sticky Runners](../runners/README.md#locking-a-specific-runner-from-being-enabled-for-other-projects) that will be only available to a particular project. -- [Use a `key`](../yaml/README.md#cachekey) that fits your workflow (e.g., +- [Use a `key`](../yaml/README.md#cachekey) that fits your workflow (for example, different caches on each branch). For that, you can take advantage of the [CI/CD predefined variables](../variables/README.md#predefined-environment-variables). @@ -100,7 +104,7 @@ From the perspective of the Runner, in order for cache to work effectively, one of the following must be true: - Use a single Runner for all your jobs. -- Use multiple Runners (in autoscale mode or not) that use. +- Use multiple Runners (in autoscale mode or not) that use [distributed caching](https://docs.gitlab.com/runner/configuration/autoscale.html#distributed-runners-caching), where the cache is stored in S3 buckets (like shared Runners on GitLab.com). - Use multiple Runners (not in autoscale mode) of the same architecture that @@ -289,7 +293,7 @@ jobs inherit it. Gems are installed in `vendor/ruby/` and are cached per-branch: # # https://gitlab.com/gitlab-org/gitlab-ce/tree/master/lib/gitlab/ci/templates/Ruby.gitlab-ci.yml # -image: ruby:2.5 +image: ruby:2.6 # Cache gems in between builds cache: @@ -420,7 +424,7 @@ mismatch and a few ideas how to fix it. Let's explore some examples. ---- +#### Examples Let's assume you have only one Runner assigned to your project, so the cache will be stored in the Runner's machine by default. If two jobs, A and B, @@ -462,8 +466,6 @@ job B: To fix that, use different `keys` for each job. ---- - In another case, let's assume you have more than one Runners assigned to your project, but the distributed cache is not enabled. We want the second time the pipeline is run, `job A` and `job B` to re-use their cache (which in this case @@ -526,3 +528,15 @@ Behind the scenes, this works by increasing a counter in the database, and the value of that counter is used to create the key for the cache by appending an integer to it: `-1`, `-2`, etc. After a push, a new key is generated and the old cache is not valid anymore. + +<!-- ## Troubleshooting + +Include any troubleshooting steps that you can foresee. If you know beforehand what issues +one might have when setting this up, or when something is changed, or on upgrading, it's +important to describe those, too. Think of things that may go wrong and include them here. +This is important to minimize requests for support, and to avoid doc comments with +questions that you know someone might ask. + +Each scenario can be a third-level heading, e.g. `### Getting error message X`. +If you have none to add when creating a doc, leave this section in place +but commented out to help encourage others to add to it in the future. --> diff --git a/doc/ci/chatops/README.md b/doc/ci/chatops/README.md index 6ad1df7bb2a..241134783da 100644 --- a/doc/ci/chatops/README.md +++ b/doc/ci/chatops/README.md @@ -1,18 +1,25 @@ +--- +type: index, concepts, howto +--- + # GitLab ChatOps > **Notes:** > -> * [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/4466) in [GitLab Ultimate](https://about.gitlab.com/pricing/) 10.6. [Moved](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/24780) to [GitLab Core](https://about.gitlab.com/pricing/) in 11.9. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/4466) in [GitLab Ultimate](https://about.gitlab.com/pricing/) 10.6. [Moved](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/24780) to [GitLab Core](https://about.gitlab.com/pricing/) in 11.9. > -> * ChatOps is currently in alpha, with some important features missing like access control. +> - ChatOps is currently in alpha, with some important features missing like access control. GitLab ChatOps provides a method to interact with CI/CD jobs through chat services like Slack. Many organizations' discussion, collaboration, and troubleshooting is taking place in chat services these days, and having a method to run CI/CD jobs with output posted back to the channel can significantly augment a team's workflow. ## How it works -GitLab ChatOps is built upon two existing features, [GitLab CI/CD](../README.md) and [Slack Slash Commmands](../../user/project/integrations/slack_slash_commands.md). +GitLab ChatOps is built upon two existing features: + +- [GitLab CI/CD](../README.md). +- [Slack Slash Commands](../../user/project/integrations/slack_slash_commands.md). -A new `run` action has been added to the [slash commands](../../integration/slash_commands.md), which takes two arguments: a `<job name>` to execute and the `<job arguments>`. When executed, ChatOps will look up the specified job name and attempt to match it to a corresponding job in [.gitlab-ci.yml](../yaml/README.md). If a matching job is found on `master`, a pipeline containing just that job is scheduled. Two additional [CI/CD variables](../variables/README.html#predefined-variables-environment-variables) are passed to the job: `CHAT_INPUT` contains any additional arguments, and `CHAT_CHANNEL` is set to the name of channel the action was triggered in. +A new `run` action has been added to the [slash commands](../../integration/slash_commands.md), which takes two arguments: a `<job name>` to execute and the `<job arguments>`. When executed, ChatOps will look up the specified job name and attempt to match it to a corresponding job in [.gitlab-ci.yml](../yaml/README.md). If a matching job is found on `master`, a pipeline containing just that job is scheduled. Two additional [CI/CD variables](../variables/README.md#predefined-environment-variables) are passed to the job: `CHAT_INPUT` contains any additional arguments, and `CHAT_CHANNEL` is set to the name of channel the action was triggered in. After the job has finished, its output is sent back to Slack provided it has completed within 30 minutes. If a job takes more than 30 minutes to run it must use the Slack API to manually send data back to a channel. @@ -22,9 +29,9 @@ After the job has finished, its output is sent back to Slack provided it has com Since ChatOps is built upon GitLab CI/CD, the job has all the same features and functions available. There a few best practices to consider however when creating ChatOps jobs: -* It is strongly recommended to set `only: [chat]` so the job does not run as part of the standard CI pipeline. -* If the job is set to `when: manual`, the pipeline will be created however the job will wait to be started. -* It is important to keep in mind that there is very limited support for access control. If the user who triggered the slash command is a developer in the project, the job will run. The job itself can utilize existing [CI/CD variables](../variables/README.html#predefined-environment-variables) like `GITLAB_USER_ID` to perform additional rights validation, however these variables can be [overridden](https://docs.gitlab.com/ce/ci/variables/README.html#priority-of-variables). +- It is strongly recommended to set `only: [chat]` so the job does not run as part of the standard CI pipeline. +- If the job is set to `when: manual`, the pipeline will be created however the job will wait to be started. +- It is important to keep in mind that there is limited support for access control. If the user who triggered the slash command is a developer in the project, the job will run. The job itself can utilize existing [CI/CD variables](../variables/README.html#predefined-environment-variables) like `GITLAB_USER_ID` to perform additional rights validation, however these variables can be [overridden](../variables/README.html#priority-of-environment-variables). ### Controlling the ChatOps reply @@ -59,3 +66,15 @@ You can find and download the official GitLab ChatOps icon here.  [Download bigger image](img/gitlab-chatops-icon.png) + +<!-- ## Troubleshooting + +Include any troubleshooting steps that you can foresee. If you know beforehand what issues +one might have when setting this up, or when something is changed, or on upgrading, it's +important to describe those, too. Think of things that may go wrong and include them here. +This is important to minimize requests for support, and to avoid doc comments with +questions that you know someone might ask. + +Each scenario can be a third-level heading, e.g. `### Getting error message X`. +If you have none to add when creating a doc, leave this section in place +but commented out to help encourage others to add to it in the future. --> diff --git a/doc/ci/ci_cd_for_external_repos/bitbucket_integration.md b/doc/ci/ci_cd_for_external_repos/bitbucket_integration.md new file mode 100644 index 00000000000..9126eb65f07 --- /dev/null +++ b/doc/ci/ci_cd_for_external_repos/bitbucket_integration.md @@ -0,0 +1,166 @@ +--- +type: howto +--- + +# Using GitLab CI/CD with a Bitbucket Cloud repository **[PREMIUM]** + +GitLab CI/CD can be used with Bitbucket Cloud by: + +1. Creating a [CI/CD project](https://docs.gitlab.com/ee/user/project/ci_cd_for_external_repo.html). +1. Connecting your Git repository via URL. + +To use GitLab CI/CD with a Bitbucket Cloud repository: + +1. In GitLab create a **CI/CD for external repo**, select **Repo by URL** and + create the project. + +  + + GitLab will import the repository and enable [Pull Mirroring][pull-mirroring]. + +1. In GitLab create a + [Personal Access Token](../../user/profile/personal_access_tokens.md) + with `api` scope. This will be used to authenticate requests from the web + hook that will be created in Bitbucket to notify GitLab of new commits. + +1. In Bitbucket, from **Settings > Webhooks**, create a new web hook to notify + GitLab of new commits. + + The web hook URL should be set to the GitLab API to trigger pull mirroring, + using the Personal Access Token we just generated for authentication. + + ```text + https://gitlab.com/api/v4/projects/<NAMESPACE>%2F<PROJECT>/mirror/pull?private_token=<PERSONAL_ACCESS_TOKEN> + ``` + + The web hook Trigger should be set to 'Repository Push'. + +  + + After saving, test the web hook by pushing a change to your Bitbucket + repository. + +1. In Bitbucket, create an **App Password** from **Bitbucket Settings > App + Passwords** to authenticate the build status script setting commit build + statuses in Bitbucket. Repository write permissions are required. + +  + +1. In GitLab, from **Settings > CI/CD > Environment variables**, add variables to allow + communication with Bitbucket via the Bitbucket API: + + `BITBUCKET_ACCESS_TOKEN`: the Bitbucket app password created above. + + `BITBUCKET_USERNAME`: the username of the Bitbucket account. + + `BITBUCKET_NAMESPACE`: set this if your GitLab and Bitbucket namespaces differ. + + `BITBUCKET_REPOSITORY`: set this if your GitLab and Bitbucket project names differ. + +1. In Bitbucket, add a script to push the pipeline status to Bitbucket. + + > Note: changes made in GitLab will be overwritten by any changes made + > upstream in Bitbucket. + + Create a file `build_status` and insert the script below and run + `chmod +x build_status` in your terminal to make the script executable. + + ```bash + #!/usr/bin/env bash + + # Push GitLab CI/CD build status to Bitbucket Cloud + + if [ -z "$BITBUCKET_ACCESS_TOKEN" ]; then + echo "ERROR: BITBUCKET_ACCESS_TOKEN is not set" + exit 1 + fi + if [ -z "$BITBUCKET_USERNAME" ]; then + echo "ERROR: BITBUCKET_USERNAME is not set" + exit 1 + fi + if [ -z "$BITBUCKET_NAMESPACE" ]; then + echo "Setting BITBUCKET_NAMESPACE to $CI_PROJECT_NAMESPACE" + BITBUCKET_NAMESPACE=$CI_PROJECT_NAMESPACE + fi + if [ -z "$BITBUCKET_REPOSITORY" ]; then + echo "Setting BITBUCKET_REPOSITORY to $CI_PROJECT_NAME" + BITBUCKET_REPOSITORY=$CI_PROJECT_NAME + fi + + BITBUCKET_API_ROOT="https://api.bitbucket.org/2.0" + BITBUCKET_STATUS_API="$BITBUCKET_API_ROOT/repositories/$BITBUCKET_NAMESPACE/$BITBUCKET_REPOSITORY/commit/$CI_COMMIT_SHA/statuses/build" + BITBUCKET_KEY="ci/gitlab-ci/$CI_JOB_NAME" + + case "$BUILD_STATUS" in + running) + BITBUCKET_STATE="INPROGRESS" + BITBUCKET_DESCRIPTION="The build is running!" + ;; + passed) + BITBUCKET_STATE="SUCCESSFUL" + BITBUCKET_DESCRIPTION="The build passed!" + ;; + failed) + BITBUCKET_STATE="FAILED" + BITBUCKET_DESCRIPTION="The build failed." + ;; + esac + + echo "Pushing status to $BITBUCKET_STATUS_API..." + curl --request POST $BITBUCKET_STATUS_API \ + --user $BITBUCKET_USERNAME:$BITBUCKET_ACCESS_TOKEN \ + --header "Content-Type:application/json" \ + --silent \ + --data "{ \"state\": \"$BITBUCKET_STATE\", \"key\": \"$BITBUCKET_KEY\", \"description\": + \"$BITBUCKET_DESCRIPTION\",\"url\": \"$CI_PROJECT_URL/-/jobs/$CI_JOB_ID\" }" + ``` + +1. Still in Bitbucket, create a `.gitlab-ci.yml` file to use the script to push + pipeline success and failures to Bitbucket. + + ```yaml + stages: + - test + - ci_status + + unit-tests: + script: + - echo "Success. Add your tests!" + + success: + stage: ci_status + before_script: + - "" + after_script: + - "" + script: + - BUILD_STATUS=passed BUILD_KEY=push ./build_status + when: on_success + + failure: + stage: ci_status + before_script: + - "" + after_script: + - "" + script: + - BUILD_STATUS=failed BUILD_KEY=push ./build_status + when: on_failure + ``` + +GitLab is now configured to mirror changes from Bitbucket, run CI/CD pipelines +configured in `.gitlab-ci.yml` and push the status to Bitbucket. + +[pull-mirroring]: ../../workflow/repository_mirroring.md#pulling-from-a-remote-repository-starter + +<!-- ## Troubleshooting + +Include any troubleshooting steps that you can foresee. If you know beforehand what issues +one might have when setting this up, or when something is changed, or on upgrading, it's +important to describe those, too. Think of things that may go wrong and include them here. +This is important to minimize requests for support, and to avoid doc comments with +questions that you know someone might ask. + +Each scenario can be a third-level heading, e.g. `### Getting error message X`. +If you have none to add when creating a doc, leave this section in place +but commented out to help encourage others to add to it in the future. --> diff --git a/doc/ci/ci_cd_for_external_repos/github_integration.md b/doc/ci/ci_cd_for_external_repos/github_integration.md new file mode 100644 index 00000000000..612dcc93bc1 --- /dev/null +++ b/doc/ci/ci_cd_for_external_repos/github_integration.md @@ -0,0 +1,134 @@ +--- +type: howto +--- + +# Using GitLab CI/CD with a GitHub repository **[PREMIUM]** + +GitLab CI/CD can be used with **GitHub.com** and **GitHub Enterprise** by +creating a [CI/CD project](https://docs.gitlab.com/ee/user/project/ci_cd_for_external_repo.html) to connect your GitHub repository to +GitLab. + +NOTE: **Note:** +To use **GitHub Enterprise** with **GitLab.com** you should use the +manual method. + +## Connect with GitHub integration + +If the [GitHub integration](../../integration/github.md) has been enabled by your GitLab +administrator: + +NOTE: **Note:** +Due to a 10-token limitation on the [GitHub OAuth Implementation](https://developer.github.com/apps/building-oauth-apps/authorizing-oauth-apps/#creating-multiple-tokens-for-oauth-apps), +if you import more than 10 times, your oldest imported project's token will be +revoked. See issue [#9147](https://gitlab.com/gitlab-org/gitlab-ee/issues/9147) +for more information. + +1. In GitLab create a **CI/CD for external repo** project and select + **GitHub**. + +  + +1. Once authenticated, you will be redirected to a list of your repositories to + connect. Click **Connect** to select the repository. + +  + +1. In GitHub, add a `.gitlab-ci.yml` to [configure GitLab CI/CD](../quick_start/README.md). + +GitLab will: + +1. Import the project. +1. Enable [Pull Mirroring](../../workflow/repository_mirroring.md#pulling-from-a-remote-repository-starter). +1. Enable [GitHub project integration](https://docs.gitlab.com/ee/user/project/integrations/github.html). +1. Create a web hook on GitHub to notify GitLab of new commits. + +## Connect with Personal Access Token + +NOTE: **Note:** +Personal access tokens can only be used to connect GitHub.com +repositories to GitLab. + +If you are not using the [GitHub integration](../../integration/github.md), you can +still perform a one-off authorization with GitHub to grant GitLab access your +repositories: + +1. Open <https://github.com/settings/tokens/new> to create a **Personal Access + Token**. This token with be used to access your repository and push commit + statuses to GitHub. + + The `repo` and `admin:repo_hook` should be enable to allow GitLab access to + your project, update commit statuses, and create a web hook to notify + GitLab of new commits. + +1. In GitLab create a **CI/CD for external repo** project and select + **GitHub**. + +  + +1. Paste the token into the **Personal access token** field and click **List + Repositories**. Click **Connect** to select the repository. + +1. In GitHub, add a `.gitlab-ci.yml` to [configure GitLab CI/CD](../quick_start/README.md). + +GitLab will: + +1. Import the project. +1. Enable [Pull Mirroring](../../workflow/repository_mirroring.md#pulling-from-a-remote-repository-starter). +1. Enable [GitHub project integration](https://docs.gitlab.com/ee/user/project/integrations/github.html). +1. Create a web hook on GitHub to notify GitLab of new commits. + +## Connect manually + +If the [GitHub integration](../../integration/github.md) is not enabled, or is enabled +for a different GitHub instance, you GitLab CI/CD can be manually enabled for +your repository: + +1. In GitHub open <https://github.com/settings/tokens/new> create a **Personal + Access Token.** GitLab will use this token to access your repository and + push commit statuses. + + Enter a **Token description** and update the scope to allow: + + `repo` so that GitLab can access your project and update commit statuses + +1. In GitLab create a **CI/CD project** using the Git URL option and the HTTPS + URL for your GitHub repository. If your project is private, use the personal + access token you just created for authentication. + + GitLab will automatically configure polling-based pull mirroring. + +1. Still in GitLab, enable the [GitHub project integration](https://docs.gitlab.com/ee/user/project/integrations/github.html) + from **Settings > Integrations.** + + Check the **Active** checkbox to enable the integration, paste your + personal access token and HTTPS repository URL into the form, and **Save.** + +1. Still in GitLab create a **Personal Access Token** with `API` scope to + authenticate the GitHub web hook notifying GitLab of new commits. + +1. In GitHub from **Settings > Webhooks** create a web hook to notify GitLab of + new commits. + + The web hook URL should be set to the GitLab API to + [trigger pull mirroring](https://docs.gitlab.com/ee/api/projects.html#start-the-pull-mirroring-process-for-a-project-starter), + using the GitLab personal access token we just created. + + ``` + https://gitlab.com/api/v4/projects/<NAMESPACE>%2F<PROJECT>/mirror/pull?private_token=<PERSONAL_ACCESS_TOKEN> + ``` + +  + +1. In GitHub add a `.gitlab-ci.yml` to configure GitLab CI/CD. + +<!-- ## Troubleshooting + +Include any troubleshooting steps that you can foresee. If you know beforehand what issues +one might have when setting this up, or when something is changed, or on upgrading, it's +important to describe those, too. Think of things that may go wrong and include them here. +This is important to minimize requests for support, and to avoid doc comments with +questions that you know someone might ask. + +Each scenario can be a third-level heading, e.g. `### Getting error message X`. +If you have none to add when creating a doc, leave this section in place +but commented out to help encourage others to add to it in the future. --> diff --git a/doc/ci/ci_cd_for_external_repos/img/bitbucket_app_password.png b/doc/ci/ci_cd_for_external_repos/img/bitbucket_app_password.png Binary files differnew file mode 100644 index 00000000000..ac5be3c3058 --- /dev/null +++ b/doc/ci/ci_cd_for_external_repos/img/bitbucket_app_password.png diff --git a/doc/ci/ci_cd_for_external_repos/img/bitbucket_webhook.png b/doc/ci/ci_cd_for_external_repos/img/bitbucket_webhook.png Binary files differnew file mode 100644 index 00000000000..0a3476d9035 --- /dev/null +++ b/doc/ci/ci_cd_for_external_repos/img/bitbucket_webhook.png diff --git a/doc/ci/ci_cd_for_external_repos/img/ci_cd_for_external_repo.png b/doc/ci/ci_cd_for_external_repos/img/ci_cd_for_external_repo.png Binary files differnew file mode 100644 index 00000000000..f068688146b --- /dev/null +++ b/doc/ci/ci_cd_for_external_repos/img/ci_cd_for_external_repo.png diff --git a/doc/ci/ci_cd_for_external_repos/img/external_repository.png b/doc/ci/ci_cd_for_external_repos/img/external_repository.png Binary files differnew file mode 100644 index 00000000000..b850d91f56b --- /dev/null +++ b/doc/ci/ci_cd_for_external_repos/img/external_repository.png diff --git a/doc/ci/ci_cd_for_external_repos/img/github_omniauth.png b/doc/ci/ci_cd_for_external_repos/img/github_omniauth.png Binary files differnew file mode 100644 index 00000000000..71a3a057a41 --- /dev/null +++ b/doc/ci/ci_cd_for_external_repos/img/github_omniauth.png diff --git a/doc/ci/ci_cd_for_external_repos/img/github_push_webhook.png b/doc/ci/ci_cd_for_external_repos/img/github_push_webhook.png Binary files differnew file mode 100644 index 00000000000..e8c17d664e1 --- /dev/null +++ b/doc/ci/ci_cd_for_external_repos/img/github_push_webhook.png diff --git a/doc/ci/ci_cd_for_external_repos/img/github_repo_list.png b/doc/ci/ci_cd_for_external_repos/img/github_repo_list.png Binary files differnew file mode 100644 index 00000000000..73579dd3cf1 --- /dev/null +++ b/doc/ci/ci_cd_for_external_repos/img/github_repo_list.png diff --git a/doc/ci/ci_cd_for_external_repos/index.md b/doc/ci/ci_cd_for_external_repos/index.md new file mode 100644 index 00000000000..5de88412121 --- /dev/null +++ b/doc/ci/ci_cd_for_external_repos/index.md @@ -0,0 +1,39 @@ +--- +type: index, howto +--- + +# GitLab CI/CD for external repositories **[PREMIUM]** + +NOTE: **Note:** +This feature [is available for free](https://about.gitlab.com/2019/03/21/six-more-months-ci-cd-github/) to +GitLab.com users until September 22nd, 2019. + +>[Introduced][ee-4642] in [GitLab Premium][eep] 10.6. + +GitLab CI/CD can be used with: + +- [GitHub](github_integration.md). +- [Bitbucket Cloud](bitbucket_integration.md). +- Any other Git server. + +Instead of moving your entire project to GitLab, you can connect your +external repository to get the benefits of GitLab CI/CD. + +Connecting an external repository will set up [repository mirroring][mirroring] +and create a lightweight project where issues, merge requests, wiki, and +snippets disabled. These features +[can be re-enabled later][settings]. + +To connect to an external repository: + +1. From your GitLab dashboard, click **New project**. +1. Switch to the **CI/CD for external repo** tab. +1. Choose **GitHub** or **Repo by URL**. +1. The next steps are similar to the [import flow](../../user/project/import/index.md). + + + +[ee-4642]: https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/4642 +[eep]: https://about.gitlab.com/pricing/ +[mirroring]: ../../workflow/repository_mirroring.md +[settings]: ../../user/project/settings/index.md#sharing-and-permissions diff --git a/doc/ci/docker/README.md b/doc/ci/docker/README.md index 446f5b54f0c..f76471b50f2 100644 --- a/doc/ci/docker/README.md +++ b/doc/ci/docker/README.md @@ -1,9 +1,13 @@ --- comments: false +type: index --- # Docker integration +GitLab CI/CD can be combined with [Docker](https://www.docker.com) to enable +integration between the two. + The following documentation is available for using GitLab CI/CD with Docker: - [Using Docker images](using_docker_images.md). diff --git a/doc/ci/docker/using_docker_build.md b/doc/ci/docker/using_docker_build.md index a462c75f2f5..d8068bbb7f0 100644 --- a/doc/ci/docker/using_docker_build.md +++ b/doc/ci/docker/using_docker_build.md @@ -1,23 +1,25 @@ +--- +type: concepts, howto +--- + # Building Docker images with GitLab CI/CD GitLab CI/CD allows you to use Docker Engine to build and test docker-based projects. -TIP: **Tip:** -This also allows to you to use `docker-compose` and other docker-enabled tools. One of the new trends in Continuous Integration/Deployment is to: -1. Create an application image -1. Run tests against the created image -1. Push image to a remote registry -1. Deploy to a server from the pushed image +1. Create an application image. +1. Run tests against the created image. +1. Push image to a remote registry. +1. Deploy to a server from the pushed image. It's also useful when your application already has the `Dockerfile` that can be used to create and test an image: ```bash docker build -t my-image dockerfiles/ -docker run my-docker-image /script/to/run/tests +docker run my-image /script/to/run/tests docker tag my-image my-registry:5000/my-image docker push my-registry:5000/my-image ``` @@ -75,7 +77,7 @@ GitLab Runner then executes job scripts as the `gitlab-runner` user. - docker run my-docker-image /script/to/run/tests ``` -1. You can now use `docker` command and install `docker-compose` if needed. +1. You can now use `docker` command (and **install** `docker-compose` if needed). NOTE: **Note:** By adding `gitlab-runner` to the `docker` group you are effectively granting `gitlab-runner` full root permissions. @@ -85,8 +87,10 @@ For more information please read [On Docker security: `docker` group considered The second approach is to use the special docker-in-docker (dind) [Docker image](https://hub.docker.com/_/docker/) with all tools installed -(`docker` and `docker-compose`) and run the job script in context of that -image in privileged mode. +(`docker`) and run the job script in context of that +image in privileged mode. + +NOTE: **Note:** `docker-compose` is not part of docker-in-docker (dind). In case you'd like to use `docker-compose` in your CI builds, please follow the (installation instructions for docker-compose)[https://docs.docker.com/compose/install/] provided by docker. In order to do that, follow the steps: @@ -113,7 +117,7 @@ In order to do that, follow the steps: The above command will create a `config.toml` entry similar to this: - ``` + ```toml [[runners]] url = "https://gitlab.com/" token = TOKEN @@ -142,9 +146,12 @@ In order to do that, follow the steps: # The 'docker' hostname is the alias of the service container as described at # https://docs.gitlab.com/ee/ci/docker/using_docker_images.html#accessing-the-services # - # Note that if you're using Kubernetes executor, the variable should be set to - # tcp://localhost:2375 because of how Kubernetes executor connects services + # Note that if you're using the Kubernetes executor, the variable should be set to + # tcp://localhost:2375/ because of how the Kubernetes executor connects services # to the job container + # DOCKER_HOST: tcp://localhost:2375/ + # + # For non-Kubernetes executors, we use tcp://docker:2375/ DOCKER_HOST: tcp://docker:2375/ # When using dind, it's wise to use the overlayfs driver for # improved performance. @@ -224,7 +231,7 @@ In order to do that, follow the steps: The above command will create a `config.toml` entry similar to this: - ``` + ```toml [[runners]] url = "https://gitlab.com/" token = REGISTRATION_TOKEN @@ -267,9 +274,9 @@ aware of the following implications: create containers with specific names, they may conflict with each other. - Sharing files and directories from the source repo into containers may not work as expected since volume mounting is done in the context of the host - machine, not the build container, e.g.: + machine, not the build container. For example: - ``` + ```sh docker run --rm -t -i -v $(pwd)/src:/home/app/src test-image:latest run_app_tests ``` @@ -303,20 +310,19 @@ services: - docker:dind variables: - CONTAINER_IMAGE: registry.gitlab.com/$CI_PROJECT_PATH DOCKER_HOST: tcp://docker:2375 DOCKER_DRIVER: overlay2 before_script: - - docker login -u gitlab-ci-token -p $CI_JOB_TOKEN registry.gitlab.com + - docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY build: stage: build script: - - docker pull $CONTAINER_IMAGE:latest || true - - docker build --cache-from $CONTAINER_IMAGE:latest --tag $CONTAINER_IMAGE:$CI_COMMIT_SHA --tag $CONTAINER_IMAGE:latest . - - docker push $CONTAINER_IMAGE:$CI_COMMIT_SHA - - docker push $CONTAINER_IMAGE:latest + - docker pull $CI_REGISTRY_IMAGE:latest || true + - docker build --cache-from $CI_REGISTRY_IMAGE:latest --tag $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA --tag $CI_REGISTRY_IMAGE:latest . + - docker push $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA + - docker push $CI_REGISTRY_IMAGE:latest ``` The steps in the `script` section for the `build` stage can be summed up to: @@ -324,7 +330,7 @@ The steps in the `script` section for the `build` stage can be summed up to: 1. The first command tries to pull the image from the registry so that it can be used as a cache for the `docker build` command. 1. The second command builds a Docker image using the pulled image as a - cache (notice the `--cache-from $CONTAINER_IMAGE:latest` argument) if + cache (notice the `--cache-from $CI_REGISTRY_IMAGE:latest` argument) if available, and tags it. 1. The last two commands push the tagged Docker images to the container registry so that they may also be used as cache for subsequent builds. @@ -335,7 +341,7 @@ NOTE: **Note:** The shared Runners on GitLab.com use the `overlay2` driver by default. By default, when using `docker:dind`, Docker uses the `vfs` storage driver which -copies the filesystem on every run. This is a very disk-intensive operation +copies the filesystem on every run. This is a disk-intensive operation which can be avoided if a different driver is used, for example `overlay2`. ### Requirements @@ -343,13 +349,13 @@ which can be avoided if a different driver is used, for example `overlay2`. 1. Make sure a recent kernel is used, preferably `>= 4.2`. 1. Check whether the `overlay` module is loaded: - ``` + ```sh sudo lsmod | grep overlay ``` If you see no result, then it isn't loaded. To load it use: - ``` + ```sh sudo modprobe overlay ``` @@ -357,7 +363,7 @@ which can be avoided if a different driver is used, for example `overlay2`. On Ubuntu systems, this is done by editing `/etc/modules`. Just add the following line into it: - ``` + ```text overlay ``` @@ -365,7 +371,7 @@ which can be avoided if a different driver is used, for example `overlay2`. You can enable the driver for each project individually by editing the project's `.gitlab-ci.yml`: -``` +```yaml variables: DOCKER_DRIVER: overlay2 ``` @@ -389,6 +395,7 @@ If you're running multiple Runners you will have to modify all configuration fil ## Using the GitLab Container Registry > **Notes:** +> > - This feature requires GitLab 8.8 and GitLab Runner 1.2. > - Starting from GitLab 8.12, if you have [2FA] enabled in your account, you need > to pass a [personal access token][pat] instead of your password in order to @@ -420,14 +427,14 @@ and depend on the visibility of your project. For all projects, mostly suitable for public ones: -- **Using the special `gitlab-ci-token` user**: This user is created for you in order to +- **Using the special `$CI_REGISTRY_USER` variable**: The user specified by this variable is created for you in order to push to the Registry connected to your project. Its password is automatically - set with the `$CI_JOB_TOKEN` variable. This allows you to automate building and deploying + set with the `$CI_REGISTRY_PASSWORD` variable. This allows you to automate building and deploying your Docker images and has read/write access to the Registry. This is ephemeral, so it's only valid for one job. You can use the following example as-is: ```sh - docker login -u gitlab-ci-token -p $CI_JOB_TOKEN $CI_REGISTRY + docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY ``` For private and internal projects: @@ -435,8 +442,10 @@ For private and internal projects: - **Using a personal access token**: You can create and use a [personal access token](../../user/profile/personal_access_tokens.md) in case your project is private: - - For read (pull) access, the scope should be `read_registry`. - - For read/write (pull/push) access, use `api`. + + - For read (pull) access, the scope should be `read_registry`. + - For read/write (pull/push) access, use `api`. + Replace the `<username>` and `<access_token>` in the following example: ```sh @@ -468,9 +477,9 @@ could look like: DOCKER_DRIVER: overlay2 stage: build script: - - docker login -u gitlab-ci-token -p $CI_JOB_TOKEN registry.example.com - - docker build -t registry.example.com/group/project/image:latest . - - docker push registry.example.com/group/project/image:latest + - docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY + - docker build -t $CI_REGISTRY/group/project/image:latest . + - docker push $CI_REGISTRY/group/project/image:latest ``` You can also make use of [other variables](../variables/README.md) to avoid hardcoding: @@ -485,7 +494,7 @@ variables: IMAGE_TAG: $CI_REGISTRY_IMAGE:$CI_COMMIT_REF_SLUG before_script: - - docker login -u gitlab-ci-token -p $CI_JOB_TOKEN $CI_REGISTRY + - docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY build: stage: build @@ -525,7 +534,7 @@ variables: CONTAINER_RELEASE_IMAGE: $CI_REGISTRY_IMAGE:latest before_script: - - docker login -u gitlab-ci-token -p $CI_JOB_TOKEN $CI_REGISTRY + - docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY build: stage: build @@ -566,3 +575,15 @@ deploy: [docker-cap]: https://docs.docker.com/engine/reference/run/#runtime-privilege-and-linux-capabilities [2fa]: ../../user/profile/account/two_factor_authentication.md [pat]: ../../user/profile/personal_access_tokens.md + +<!-- ## Troubleshooting + +Include any troubleshooting steps that you can foresee. If you know beforehand what issues +one might have when setting this up, or when something is changed, or on upgrading, it's +important to describe those, too. Think of things that may go wrong and include them here. +This is important to minimize requests for support, and to avoid doc comments with +questions that you know someone might ask. + +Each scenario can be a third-level heading, e.g. `### Getting error message X`. +If you have none to add when creating a doc, leave this section in place +but commented out to help encourage others to add to it in the future. --> diff --git a/doc/ci/docker/using_docker_images.md b/doc/ci/docker/using_docker_images.md index 3314c76d234..d09efce7cc1 100644 --- a/doc/ci/docker/using_docker_images.md +++ b/doc/ci/docker/using_docker_images.md @@ -1,3 +1,7 @@ +--- +type: concepts, howto +--- + # Using Docker images GitLab CI in conjunction with [GitLab Runner](../runners/README.md) can use @@ -45,10 +49,10 @@ The `image` keyword is the name of the Docker image the Docker executor will run to perform the CI tasks. By default, the executor will only pull images from [Docker Hub][hub], -but this can be configured in the `gitlab-runner/config.toml` by setting +however this can be configured in the `gitlab-runner/config.toml` by setting the [Docker pull policy][] to allow using local images. -For more information about images and Docker Hub please read +For more information about images and Docker Hub, please read the [Docker Fundamentals][] documentation. ## What is a service @@ -95,8 +99,8 @@ required for the CI/CD job to proceed and is accessed by network. To make sure this works, the Runner: -1. checks which ports are exposed from the container by default -1. starts a special container that waits for these ports to be accessible +1. Checks which ports are exposed from the container by default. +1. Starts a special container that waits for these ports to be accessible. When the second stage of the check fails, either because there is no opened port in the service, or the service was not started properly before the timeout and the port is not @@ -106,7 +110,7 @@ In most cases it will affect the job, but there may be situations when the job will still succeed even if that warning was printed. For example: - The service was started a little after the warning was raised, and the job is - not using the linked service from the very beginning. In that case, when the + not using the linked service from the beginning. In that case, when the job needed to access the service, it may have been already there waiting for connections. - The service container is not providing any networking service, but it's doing @@ -143,9 +147,9 @@ job: If you need to have `php`, `node` and `go` available for your script, you should either: -- choose an existing Docker image that contains all required tools, or -- create your own Docker image, which will have all the required tools included - and use that in your job +- Choose an existing Docker image that contains all required tools. +- Create your own Docker image, which will have all the required tools included + and use that in your job. ### Accessing the services @@ -167,18 +171,18 @@ access to it from your build container under two hostnames to choose from: - `tutum-wordpress` - `tutum__wordpress` ->**Note:** +NOTE: **Note:** Hostnames with underscores are not RFC valid and may cause problems in 3rd party applications. The default aliases for the service's hostname are created from its image name following these rules: -- Everything after the colon (`:`) is stripped +- Everything after the colon (`:`) is stripped. - Slash (`/`) is replaced with double underscores (`__`) and the primary alias - is created + is created. - Slash (`/`) is replaced with a single dash (`-`) and the secondary alias is - created (requires GitLab Runner v1.1.0 or higher) + created (requires GitLab Runner v1.1.0 or higher). To override the default behavior, you can [specify a service alias](#available-settings-for-services). @@ -333,7 +337,7 @@ services: ``` The Runner will still start two containers using the `mysql:latest` image, -but now each of them will also be accessible with the alias configured +however now each of them will also be accessible with the alias configured in `.gitlab-ci.yml` file. ### Setting a command for the service @@ -408,8 +412,6 @@ you should check which one your Runner is using. Specifically: The syntax of `image:entrypoint` is similar to [Dockerfile's `ENTRYPOINT`][entrypoint]. ----- - Let's assume you have a `super/sql:experimental` image with some SQL database inside it and you would like to use it as a base image for your job because you want to execute some tests with this database binary. Let's also assume that @@ -443,7 +445,7 @@ image: Look for the `[runners.docker]` section: -``` +```toml [runners.docker] image = "ruby:2.1" services = ["mysql:latest", "postgres:latest"] @@ -469,11 +471,11 @@ image which is private and requires you to login into a private container regist Let's also assume that these are the login credentials: -| Key | Value | -|----------|---------------------------| -| registry | registry.example.com:5000 | -| username | my_username | -| password | my_password | +| Key | Value | +|:---------|:----------------------------| +| registry | `registry.example.com:5000` | +| username | `my_username` | +| password | `my_password` | To configure access for `registry.example.com:5000`, follow these steps: @@ -500,7 +502,7 @@ To configure access for `registry.example.com:5000`, follow these steps: bXlfdXNlcm5hbWU6bXlfcGFzc3dvcmQ= ``` -1. Create a [variable] `DOCKER_AUTH_CONFIG` with the content of the +1. Create a [variable](../variables/README.md#gitlab-cicd-environment-variables) `DOCKER_AUTH_CONFIG` with the content of the Docker configuration file as the value: ```json @@ -534,7 +536,8 @@ To configure access for `registry.example.com:5000`, follow these steps: You can add configuration for as many registries as you want, adding more registries to the `"auths"` hash as described above. -NOTE: **Note:** The full `hostname:port` combination is required everywhere +NOTE: **Note:** +The full `hostname:port` combination is required everywhere for the Runner to match the `DOCKER_AUTH_CONFIG`. For example, if `registry.example.com:5000/namespace/image:tag` is specified in `.gitlab-ci.yml`, then the `DOCKER_AUTH_CONFIG` must also specify `registry.example.com:5000`. @@ -551,8 +554,9 @@ service containers. For all possible configuration variables check the documentation of each image provided in their corresponding Docker hub page. -*Note: All variables will be passed to all services containers. It's not -designed to distinguish which variable should go where.* +NOTE: **Note:** +All variables will be passed to all services containers. It's not +designed to distinguish which variable should go where. ### PostgreSQL service example @@ -582,8 +586,9 @@ time. ## How to debug a job locally -*Note: The following commands are run without root privileges. You should be -able to run Docker with your regular user account.* +NOTE: **Note:** +The following commands are run without root privileges. You should be +able to run Docker with your regular user account. First start with creating a file named `build_script`: @@ -602,7 +607,7 @@ is specific to your project. Then create some service containers: -``` +```sh docker run -d --name service-mysql mysql:latest docker run -d --name service-postgres postgres:latest ``` @@ -614,7 +619,7 @@ respectively. They will both run in the background (`-d`). Finally, create a build container by executing the `build_script` file we created earlier: -``` +```sh docker run --name build -i --link=service-mysql:mysql --link=service-postgres:postgres ruby:2.1 /bin/bash < build_script ``` @@ -626,7 +631,7 @@ piped using STDIN to the bash interpreter which in turn executes the When you finish testing and no longer need the containers, you can remove them with: -``` +```sh docker rm -f -v build service-mysql service-postgres ``` @@ -642,7 +647,6 @@ creation. [postgres-hub]: https://hub.docker.com/r/_/postgres/ [mysql-hub]: https://hub.docker.com/r/_/mysql/ [runner-priv-reg]: http://docs.gitlab.com/runner/configuration/advanced-configuration.html#using-a-private-container-registry -[variable]: ../variables/README.md#variables [entrypoint]: https://docs.docker.com/engine/reference/builder/#entrypoint [cmd]: https://docs.docker.com/engine/reference/builder/#cmd [register]: https://docs.gitlab.com/runner/register/ diff --git a/doc/ci/docker/using_kaniko.md b/doc/ci/docker/using_kaniko.md index f354cdb398e..50f1ac3d54a 100644 --- a/doc/ci/docker/using_kaniko.md +++ b/doc/ci/docker/using_kaniko.md @@ -1,3 +1,7 @@ +--- +type: howto +--- + # Building images with kaniko and GitLab CI/CD > [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/issues/45512) in GitLab 11.2. @@ -9,17 +13,18 @@ container images from a Dockerfile, inside a container or Kubernetes cluster. kaniko solves two problems with using the [docker-in-docker build](using_docker_build.md#use-docker-in-docker-executor) method: -1. Docker-in-docker requires [privileged mode](https://docs.docker.com/engine/reference/run/#runtime-privilege-and-linux-capabilities) - in order to function, which is a significant security concern. -1. Docker-in-docker generally incurs a performance penalty and can be quite slow. +- Docker-in-docker requires [privileged mode](https://docs.docker.com/engine/reference/run/#runtime-privilege-and-linux-capabilities) + in order to function, which is a significant security concern. +- Docker-in-docker generally incurs a performance penalty and can be quite slow. ## Requirements In order to utilize kaniko with GitLab, a [GitLab Runner](https://docs.gitlab.com/runner/) -using either the [Kubernetes](https://docs.gitlab.com/runner/executors/kubernetes.html), -[Docker](https://docs.gitlab.com/runner/executors/docker.html), or -[Docker Machine](https://docs.gitlab.com/runner/executors/docker_machine.html) -executors is required. +using one of the following executors is required: + +- [Kubernetes](https://docs.gitlab.com/runner/executors/kubernetes.html). +- [Docker](https://docs.gitlab.com/runner/executors/docker.html). +- [Docker Machine](https://docs.gitlab.com/runner/executors/docker_machine.html). ## Building a Docker image with kaniko @@ -34,14 +39,17 @@ few important details: - A Docker `config.json` file needs to be created with the authentication information for the desired container registry. ---- +In the following example, kaniko is used to: + +1. Build a Docker image. +1. Then push it to [GitLab Container Registry](../../user/project/container_registry.md). -In the following example, kaniko is used to build a Docker image and then push -it to [GitLab Container Registry](../../user/project/container_registry.md). The job will run only when a tag is pushed. A `config.json` file is created under `/kaniko/.docker` with the needed GitLab Container Registry credentials taken from the [environment variables](../variables/README.md#predefined-environment-variables) -GitLab CI/CD provides. In the last step, kaniko uses the `Dockerfile` under the +GitLab CI/CD provides. + +In the last step, kaniko uses the `Dockerfile` under the root directory of the project, builds the Docker image and pushes it to the project's Container Registry while tagging it with the Git tag: @@ -80,3 +88,15 @@ store: ... -----END CERTIFICATE-----" >> /kaniko/ssl/certs/ca-certificates.crt ``` + +<!-- ## Troubleshooting + +Include any troubleshooting steps that you can foresee. If you know beforehand what issues +one might have when setting this up, or when something is changed, or on upgrading, it's +important to describe those, too. Think of things that may go wrong and include them here. +This is important to minimize requests for support, and to avoid doc comments with +questions that you know someone might ask. + +Each scenario can be a third-level heading, e.g. `### Getting error message X`. +If you have none to add when creating a doc, leave this section in place +but commented out to help encourage others to add to it in the future. --> diff --git a/doc/ci/enable_or_disable_ci.md b/doc/ci/enable_or_disable_ci.md index 7aa7de97c43..56200142055 100644 --- a/doc/ci/enable_or_disable_ci.md +++ b/doc/ci/enable_or_disable_ci.md @@ -1,25 +1,32 @@ +--- +type: howto +--- + # How to enable or disable GitLab CI/CD -To effectively use GitLab CI/CD, you need a valid [`.gitlab-ci.yml`](yaml/README.md) -file present at the root directory of your project and a -[runner](runners/README.md) properly set up. You can read our -[quick start guide](quick_start/README.md) to get you started. +To effectively use GitLab CI/CD, you need: + +- A valid [`.gitlab-ci.yml`](yaml/README.md) file present at the root directory + of your project. +- A [runner](runners/README.md) properly set up. + +You can read our [quick start guide](quick_start/README.md) to get you started. If you are using an external CI/CD server like Jenkins or Drone CI, it is advised to disable GitLab CI/CD in order to not have any conflicts with the commits status API. ---- - GitLab CI/CD is exposed via the `/pipelines` and `/jobs` pages of a project. Disabling GitLab CI/CD in a project does not delete any previous jobs. In fact, the `/pipelines` and `/jobs` pages can still be accessed, although it's hidden from the left sidebar menu. -GitLab CI/CD is enabled by default on new installations and can be disabled either -individually under each project's settings, or site-wide by modifying the -settings in `gitlab.yml` and `gitlab.rb` for source and Omnibus installations -respectively. +GitLab CI/CD is enabled by default on new installations and can be disabled +either: + +- Individually under each project's settings. +- Site-wide by modifying the settings in `gitlab.yml` and `gitlab.rb` for source + and Omnibus installations respectively. ## Per-project user setting @@ -36,10 +43,10 @@ and `gitlab.rb` for source and Omnibus installations respectively. Two things to note: -1. Disabling GitLab CI/CD, will affect only newly-created projects. Projects that - had it enabled prior to this modification, will work as before. -1. Even if you disable GitLab CI/CD, users will still be able to enable it in the - project's settings. +- Disabling GitLab CI/CD, will affect only newly-created projects. Projects that + had it enabled prior to this modification, will work as before. +- Even if you disable GitLab CI/CD, users will still be able to enable it in the + project's settings. For installations from source, open `gitlab.yml` with your editor and set `builds` to `false`: @@ -54,12 +61,32 @@ default_projects_features: builds: false ``` -Save the file and restart GitLab: `sudo service gitlab restart`. +Save the file and restart GitLab: + +```sh +sudo service gitlab restart +``` For Omnibus installations, edit `/etc/gitlab/gitlab.rb` and add the line: -``` +```ruby gitlab_rails['gitlab_default_projects_features_builds'] = false ``` -Save the file and reconfigure GitLab: `sudo gitlab-ctl reconfigure`. +Save the file and reconfigure GitLab: + +```sh +sudo gitlab-ctl reconfigure +``` + +<!-- ## Troubleshooting + +Include any troubleshooting steps that you can foresee. If you know beforehand what issues +one might have when setting this up, or when something is changed, or on upgrading, it's +important to describe those, too. Think of things that may go wrong and include them here. +This is important to minimize requests for support, and to avoid doc comments with +questions that you know someone might ask. + +Each scenario can be a third-level heading, e.g. `### Getting error message X`. +If you have none to add when creating a doc, leave this section in place +but commented out to help encourage others to add to it in the future. --> diff --git a/doc/ci/environments.md b/doc/ci/environments.md index fe66f7e3c28..f2661c4bafd 100644 --- a/doc/ci/environments.md +++ b/doc/ci/environments.md @@ -1,45 +1,69 @@ -# Introduction to environments and deployments +--- +type: reference +--- + +# Environments and deployments > Introduced in GitLab 8.9. -During the development of software, there can be many stages until it's ready -for public consumption. You sure want to first test your code and then deploy it -in a testing or staging environment before you release it to the public. That -way you can prevent bugs not only in your software, but in the deployment -process as well. +Environments allow control of the continuous deployment of your software, +all within GitLab. + +## Introduction + +There are many stages required in the software development process before the software is ready +for public consumption. -GitLab CI is capable of not only testing or building your projects, but also +For example: + +1. Develop your code. +1. Test your code. +1. Deploy your code into a testing or staging environment before you release it to the public. + +This helps find bugs in your software, and also in the deployment process as well. + +GitLab CI/CD is capable of not only testing or building your projects, but also deploying them in your infrastructure, with the added benefit of giving you a -way to track your deployments. In other words, you can always know what is +way to track your deployments. In other words, you will always know what is currently being deployed or has been deployed on your servers. -## Overview +It's important to know that: + +- Environments are like tags for your CI jobs, describing where code gets deployed. +- Deployments are created when [jobs](yaml/README.md#introduction) deploy versions of code to environments, + so every environment can have one or more deployments. + +GitLab: -With environments, you can control the Continuous Deployment of your software -all within GitLab. All you need to do is define them in your project's -[`.gitlab-ci.yml`][yaml] as we will explore below. GitLab provides a full -history of your deployments per every environment. +- Provides a full history of your deployments for each environment. +- Keeps track of your deployments, so you always know what is currently being deployed on your + servers. -Environments are like tags for your CI jobs, describing where code gets deployed. -Deployments are created when [jobs] deploy versions of code to environments, -so every environment can have one or more deployments. GitLab keeps track of -your deployments, so you always know what is currently being deployed on your -servers. If you have a deployment service such as [Kubernetes][kube] -enabled for your project, you can use it to assist with your deployments, and +If you have a deployment service such as [Kubernetes](../user/project/clusters/index.md) +associated with your project, you can use it to assist with your deployments, and can even access a [web terminal](#web-terminals) for your environment from within GitLab! -To better understand how environments and deployments work, let's consider an -example. We assume that you have already created a project in GitLab and set up -a Runner. The example will cover the following: +## Configuring environments -- We are developing an application -- We want to run tests and build our app on all branches -- Our default branch is `master` -- We deploy the app only when a pipeline on `master` branch is run +Configuring environments involves: -Let's see how it all ties together. +1. Understanding how [pipelines](pipelines.md) work. +1. Defining environments in your project's [`.gitlab-ci.yml`](yaml/README.md) file. -## Defining environments +The rest of this section illustrates how to configure environments and deployments using +an example scenario. It assumes you have already: + +- Created a [project](../gitlab-basics/create-project.md) in GitLab. +- Set up [a Runner](runners/README.md). + +In the scenario: + +- We are developing an application. +- We want to run tests and build our app on all branches. +- Our default branch is `master`. +- We deploy the app only when a pipeline on `master` branch is run. + +### Defining environments Let's consider the following `.gitlab-ci.yml` example: @@ -68,125 +92,64 @@ deploy_staging: - master ``` -We have defined 3 [stages](yaml/README.md#stages): +We have defined three [stages](yaml/README.md#stages): + +- `test` +- `build` +- `deploy` -- test -- build -- deploy +The jobs assigned to these stages will run in this order. If any job fails, then +the pipeline fails and jobs that are assigned to the next stage won't run. -The jobs assigned to these stages will run in this order. If a job fails, then -the jobs that are assigned to the next stage won't run, rendering the pipeline -as failed. In our case, the `test` job will run first, then the `build` and -lastly the `deploy_staging`. With this, we ensure that first the tests pass, -then our app is able to be built successfully, and lastly we deploy to the -staging server. +In our case: +- The `test` job will run first. +- Then the `build` job. +- Lastly the `deploy_staging` job. + +With this configuration, we: + +- Check that the tests pass. +- Ensure that our app is able to be built successfully. +- Lastly we deploy to the staging server. + +NOTE: **Note:** The `environment` keyword is just a hint for GitLab that this job actually -deploys to this environment's `name`. It can also have a `url` which, as we -will later see, is exposed in various places within GitLab. Each time a job that -has an environment specified and succeeds, a deployment is recorded, remembering +deploys to the `name` environment. It can also have a `url` that is +exposed in various places within GitLab. Each time a job that +has an environment specified succeeds, a deployment is recorded, storing the Git SHA and environment name. -> **Note:** +In summary, with the above `.gitlab-ci.yml` we have achieved the following: + +- All branches will run the `test` and `build` jobs. +- The `deploy_staging` job will run [only](yaml/README.md#onlyexcept-basic) on the `master` + branch, which means all merge requests that are created from branches don't + get deployed to the staging server. +- When a merge request is merged, all jobs will run and the `deploy_staging` + job will deploy our code to a staging server while the deployment + will be recorded in an environment named `staging`. + > Starting with GitLab 8.15, the environment name is exposed to the Runner in > two forms: `$CI_ENVIRONMENT_NAME`, and `$CI_ENVIRONMENT_SLUG`. The first is > the name given in `.gitlab-ci.yml` (with any variables expanded), while the > second is a "cleaned-up" version of the name, suitable for use in URLs, DNS, > etc. > -> **Note:** > Starting with GitLab 9.3, the environment URL is exposed to the Runner via -> `$CI_ENVIRONMENT_URL`. The URL would be expanded from `.gitlab-ci.yml`, or if -> the URL was not defined there, the external URL from the environment would be -> used. - -To sum up, with the above `.gitlab-ci.yml` we have achieved that: - -- All branches will run the `test` and `build` jobs. -- The `deploy_staging` job will run [only](yaml/README.md#only-and-except-simplified) on the `master` - branch which means all merge requests that are created from branches don't - get to deploy to the staging server -- When a merge request is merged, all jobs will run and the `deploy_staging` - in particular will deploy our code to a staging server while the deployment - will be recorded in an environment named `staging`. +> `$CI_ENVIRONMENT_URL`. The URL is expanded from `.gitlab-ci.yml`, or if +> the URL was not defined there, the external URL from the environment is used. -Let's now see how that information is exposed within GitLab. +### Configuring manual deployments -## Viewing the current status of an environment +Adding `when: manual` to an automatically executed job's configuration converts it to +a job requiring manual action. -The environment list under your project's **Operations > Environments**, is -where you can find information of the last deployment status of an environment. +To expand on the [previous example](#defining-environments), the following includes +another job that deploys our app to a production server and is +tracked by a `production` environment. -Here's how the Environments page looks so far. - - - -There's a bunch of information there, specifically you can see: - -- The environment's name with a link to its deployments -- The last deployment ID number and who performed it -- The job ID of the last deployment with its respective job name -- The commit information of the last deployment such as who committed, to what - branch and the Git SHA of the commit -- The exact time the last deployment was performed -- A button that takes you to the URL that you have defined under the - `environment` keyword in `.gitlab-ci.yml` -- A button that re-deploys the latest deployment, meaning it runs the job - defined by the environment name for that specific commit - -> **Notes:** -> -> - While you can create environments manually in the web interface, we recommend -> that you define your environments in `.gitlab-ci.yml` first. They will -> be automatically created for you after the first deploy. -> - The environments page can only be viewed by Reporters and above. For more -> information on the permissions, see the [permissions documentation][permissions]. -> - Only deploys that happen after your `.gitlab-ci.yml` is properly configured -> will show up in the "Environment" and "Last deployment" lists. - -The information shown in the Environments page is limited to the latest -deployments, but as you may have guessed an environment can have multiple -deployments. - -## Viewing the deployment history of an environment - -GitLab keeps track of your deployments, so you always know what is currently -being deployed on your servers. That way you can have the full history of your -deployments per every environment right in your browser. Clicking on an -environment will show the history of its deployments. Assuming you have deployed -multiple times already, here's how a specific environment's page looks like. - - - -We can see the same information as when in the Environments page, but this time -all deployments are shown. As you may have noticed, apart from the **Re-deploy** -button there are now **Rollback** buttons for each deployment. Let's see how -that works. - -## Rolling back changes - -You can't control everything, so sometimes things go wrong. When that unfortunate -time comes GitLab has you covered. Simply by clicking the **Rollback** button -that can be found in the deployments page -(**Operations > Environments > `environment name`**) you can relaunch the -job with the commit associated with it. - ->**Note:** -Bear in mind that your mileage will vary and it's entirely up to how you define -the deployment process in the job's `script` whether the rollback succeeds or not. -GitLab CI is just following orders. - -Thankfully that was the staging server that we had to rollback, and since we -learn from our mistakes, we decided to not make the same again when we deploy -to the production server. Enter manual actions for deployments. - -## Manually deploying to environments - -Turning a job from running automatically to a manual action is as simple as -adding `when: manual` to it. To expand on our previous example, let's add -another job that this time deploys our app to a production server and is -tracked by a `production` environment. The `.gitlab-ci.yml` looks like this -so far: +The `.gitlab-ci.yml` file for this is as follows: ```yaml stages: @@ -224,41 +187,61 @@ deploy_prod: - master ``` -The `when: manual` action exposes a play button in GitLab's UI and the -`deploy_prod` job will only be triggered if and when we click that play button. -You can find it in the pipeline, job, environment, and deployment views. +The `when: manual` action: -| Pipelines | Single pipeline | Environments | Deployments | jobs | -| --------- | ----------------| ------------ | ----------- | -------| -|  |  |  |  |  | +- Exposes a "play" button in GitLab's UI for that job. +- Means the `deploy_prod` job will only be triggered when the "play" button is clicked. -Clicking on the play button in either of these places will trigger the -`deploy_prod` job, and the deployment will be recorded under a new -environment named `production`. +You can find the "play" button in the pipelines, environments, deployments, and jobs views. ->**Note:** -Remember that if your environment's name is `production` (all lowercase), then +| View | Screenshot | +|:----------------|:-------------------------------------------------------------------------------| +| Pipelines |  | +| Single pipeline |  | +| Environments |  | +| Deployments |  | +| Jobs |  | + +Clicking on the play button in any view will trigger the `deploy_prod` job, and the +deployment will be recorded as a new environment named `production`. + +NOTE: **Note:** +If your environment's name is `production` (all lowercase), it will get recorded in [Cycle Analytics](../user/project/cycle_analytics.md). -Double the benefit! -## Dynamic environments +### Configuring dynamic environments -As the name suggests, it is possible to create environments on the fly by just -declaring their names dynamically in `.gitlab-ci.yml`. Dynamic environments is -the basis of [Review apps](review_apps/index.md). +Regular environments are good when deploying to "stable" environments like staging or production. -NOTE: **Note:** -The `name` and `url` parameters can use most of the CI/CD variables, -including [predefined](variables/README.md#predefined-environment-variables), -[project/group ones](variables/README.md#variables) and -[`.gitlab-ci.yml` variables](yaml/README.md#variables). You however cannot use variables -defined under `script` or on the Runner's side. There are also other variables that -are unsupported in the context of `environment:name`. You can read more about -[where variables can be used](variables/where_variables_can_be_used.md). - -GitLab Runner exposes various [environment variables][variables] when a job runs, -and as such, you can use them as environment names. Let's add another job in -our example which will deploy to all branches except `master`: +However, for environments for branches other than `master`, dynamic environments +can be used. Dynamic environments make it possible to create environments on the fly by +declaring their names dynamically in `.gitlab-ci.yml`. + +Dynamic environments are a fundamental part of [Review apps](review_apps/index.md). + +#### Allowed variables + +The `name` and `url` parameters for dynamic environments can use most available CI/CD variables, +including: + +- [Predefined environment variables](variables/README.md#predefined-environment-variables) +- [Project and group variables](variables/README.md#gitlab-cicd-environment-variables) +- [`.gitlab-ci.yml` variables](yaml/README.md#variables) + +However, you cannot use variables defined: + +- Under `script`. +- On the Runner's side. + +There are also other variables that are unsupported in the context of `environment:name`. +For more information, see [Where variables can be used](variables/where_variables_can_be_used.md). + +#### Example configuration + +GitLab Runner exposes various [environment variables](variables/README.md) when a job runs, so +you can use them as environment names. + +In the following example, the job will deploy to all branches except `master`: ```yaml deploy_review: @@ -274,39 +257,54 @@ deploy_review: - master ``` -Let's break it down in pieces. The job's name is `deploy_review` and it runs -on the `deploy` stage. The `script` at this point is fictional, you'd have to -use your own based on your deployment. Then, we set the `environment` with the -`environment:name` being `review/$CI_COMMIT_REF_NAME`. Now that's an interesting -one. Since the [environment name][env-name] can contain slashes (`/`), we can -use this pattern to distinguish between dynamic environments and the regular -ones. - -So, the first part is `review`, followed by a `/` and then `$CI_COMMIT_REF_NAME` -which takes the value of the branch name. Since `$CI_COMMIT_REF_NAME` itself may -also contain `/`, or other characters that would be invalid in a domain name or -URL, we use `$CI_ENVIRONMENT_SLUG` in the `environment:url` so that the -environment can get a specific and distinct URL for each branch. In this case, -given a `$CI_COMMIT_REF_NAME` of `100-Do-The-Thing`, the URL will be something -like `https://100-do-the-4f99a2.example.com`. Again, the way you set up -the web server to serve these requests is based on your setup. - -You could also use `$CI_COMMIT_REF_SLUG` in `environment:url`, e.g.: -`https://$CI_COMMIT_REF_SLUG.example.com`. We use `$CI_ENVIRONMENT_SLUG` -here because it is guaranteed to be unique, but if you're using a workflow like -[GitLab Flow][gitlab-flow], collisions are very unlikely, and you may prefer -environment names to be more closely based on the branch name - the example -above would give you an URL like `https://100-do-the-thing.example.com` - -Last but not least, we tell the job to run [`only`][only] on branches -[`except`][only] master. - ->**Note:** -You are not bound to use the same prefix or only slashes in the dynamic -environments' names (`/`), but as we will see later, this will enable the -[grouping similar environments](#grouping-similar-environments) feature. - -The whole `.gitlab-ci.yml` looks like this so far: +In this example: + +- The job's name is `deploy_review` and it runs on the `deploy` stage. +- We set the `environment` with the `environment:name` as `review/$CI_COMMIT_REF_NAME`. + Since the [environment name](yaml/README.md#environmentname) can contain slashes (`/`), we can + use this pattern to distinguish between dynamic and regular environments. +- We tell the job to run [`only`](yaml/README.md#onlyexcept-basic) on branches, + [`except`](yaml/README.md#onlyexcept-basic) `master`. + +For the value of: + +- `environment:name`, the first part is `review`, followed by a `/` and then `$CI_COMMIT_REF_NAME`, + which receives the value of the branch name. +- `environment:url`, we want a specific and distinct URL for each branch. `$CI_COMMIT_REF_NAME` + may contain a `/` or other characters that would be invalid in a domain name or URL, + so we use `$CI_ENVIRONMENT_SLUG` to guarantee that we get a valid URL. + + For example, given a `$CI_COMMIT_REF_NAME` of `100-Do-The-Thing`, the URL will be something + like `https://100-do-the-4f99a2.example.com`. Again, the way you set up + the web server to serve these requests is based on your setup. + + We have used `$CI_ENVIRONMENT_SLUG` here because it is guaranteed to be unique. If + you're using a workflow like [GitLab Flow](../workflow/gitlab_flow.md), collisions + are unlikely and you may prefer environment names to be more closely based on the + branch name. In that case, you could use `$CI_COMMIT_REF_SLUG` in `environment:url` in + the example above: `https://$CI_COMMIT_REF_SLUG.example.com`, which would give a URL + of `https://100-do-the-thing.example.com`. + +NOTE: **Note:** +You are not required to use the same prefix or only slashes (`/`) in the dynamic environments' +names. However, using this format will enable the [grouping similar environments](#grouping-similar-environments) +feature. + +### Complete example + +The configuration in this section provides a full development workflow where your app is: + +- Tested. +- Built. +- Deployed as a Review App. +- Deployed to a staging server once the merge request is merged. +- Finally, able to be manually deployed to the production server. + +The following combines the previous configuration examples, including: + +- Defining [simple environments](#defining-environments) for testing, building, and deployment to staging. +- Adding [manual actions](#configuring-manual-deployments) for deployment to production. +- Creating [dynamic environments](#configuring-dynamic-environments) for deployments for reviewing. ```yaml stages: @@ -356,9 +354,10 @@ deploy_prod: - master ``` -A more realistic example would include copying files to a location where a -webserver (NGINX) could then read and serve. The example below will copy the -`public` directory to `/srv/nginx/$CI_COMMIT_REF_SLUG/public`: +A more realistic example would also include copying files to a location where a +webserver (for example, NGINX) could then access and serve them. + +The example below will copy the `public` directory to `/srv/nginx/$CI_COMMIT_REF_SLUG/public`: ```yaml review_app: @@ -370,78 +369,158 @@ review_app: url: https://$CI_COMMIT_REF_SLUG.example.com ``` -It is assumed that the user has already set up NGINX and GitLab Runner in the -server this job will run on. +This example requires that NGINX and GitLab Runner are set up on the server this job will run on. ->**Note:** -Be sure to check out the [limitations](#limitations) section for some edge -cases regarding naming of your branches and Review Apps. +NOTE: **Note:** +See the [limitations](#limitations) section for some edge cases regarding the naming of +your branches and Review Apps. ---- +The complete example provides the following workflow to developers: + +- Create a branch locally. +- Make changes and commit them. +- Push the branch to GitLab. +- Create a merge request. + +Behind the scenes, GitLab Runner will: + +- Pick up the changes and start running the jobs. +- Run the jobs sequentially as defined in `stages`: + - First, run the tests. + - If the tests succeed, build the app. + - If the build succeeds, the app is deployed to an environment with a name specific to the + branch. + +So now, every branch: + +- Gets its own environment. +- Is deployed to its own unique location, with the added benefit of: + - Having a [history of deployments](#viewing-deployment-history). + - Being able to [rollback changes](#retrying-and-rolling-back) if needed. + +For more information, see [Using the environment URL](#using-the-environment-url). + +### Protected environments + +Environments can be "protected", restricting access to them. + +For more information, see [Protected environments](environments/protected_environments.md). + +## Working with environments + +Once environments are configured, GitLab provides many features for working with them, +as documented below. + +### Viewing environments and deployments + +A list of environments and deployment statuses is available on each project's **Operations > Environments** page. + +For example: + + + +This example shows: + +- The environment's name with a link to its deployments. +- The last deployment ID number and who performed it. +- The job ID of the last deployment with its respective job name. +- The commit information of the last deployment, such as who committed it, to what + branch, and the Git SHA of the commit. +- The exact time the last deployment was performed. +- A button that takes you to the URL that you defined under the `environment` keyword + in `.gitlab-ci.yml`. +- A button that re-deploys the latest deployment, meaning it runs the job + defined by the environment name for that specific commit. + +The information shown in the **Environments** page is limited to the latest +deployments, but an environment can have multiple deployments. + +> **Notes:** +> +> - While you can create environments manually in the web interface, we recommend +> that you define your environments in `.gitlab-ci.yml` first. They will +> be automatically created for you after the first deploy. +> - The environments page can only be viewed by users with [Reporter permission](../user/permissions.md#project-members-permissions) +> and above. For more information on permissions, see the [permissions documentation](../user/permissions.md). +> - Only deploys that happen after your `.gitlab-ci.yml` is properly configured +> will show up in the **Environment** and **Last deployment** lists. + +### Viewing deployment history + +GitLab keeps track of your deployments, so you: + +- Always know what is currently being deployed on your servers. +- Can have the full history of your deployments for every environment. -The development workflow would now be: +Clicking on an environment shows the history of its deployments. Here's an example **Environments** page +with multiple deployments: -- Developer creates a branch locally -- Developer makes changes, commits and pushes the branch to GitLab -- Developer creates a merge request + + +This view is similar to the **Environments** page, but all deployments are shown. Also in this view +is a **Rollback** button. For more information, see [Retrying and rolling back](#retrying-and-rolling-back). + +### Retrying and rolling back -Behind the scenes: +If there is a problem with a deployment, you can retry it or roll it back. -- GitLab Runner picks up the changes and starts running the jobs -- The jobs run sequentially as defined in `stages` - - First, the tests pass - - Then, the job begins and successfully also passes - - Lastly, the app is deployed to an environment with a name specific to the - branch +To retry or rollback a deployment: -So now, every branch gets its own environment and is deployed to its own place -with the added benefit of having a [history of deployments](#viewing-the-deployment-history-of-an-environment) -and also being able to [rollback changes](#rolling-back-changes) if needed. -Let's briefly see where URL that's defined in the environments is exposed. +1. Navigate to **Operations > Environments**. +1. Click on the environment. +1. In the deployment history list for the environment, click the: + - **Retry** button next to the last deployment, to retry that deployment. + - **Rollback** button next to a previously successful deployment, to roll back to that deployment. + +NOTE: **Note:** +The defined deployment process in the job's `script` determines whether the rollback succeeds or not. -## Making use of the environment URL +### Using the environment URL The [environment URL](yaml/README.md#environmenturl) is exposed in a few -places within GitLab. +places within GitLab: -| In a merge request widget as a link | In the Environments view as a button | In the Deployments view as a button | -| -------------------- | ------------ | ----------- | -|  |  |  | +- In a merge request widget as a link: +  +- In the Environments view as a button: +  +- In the Deployments view as a button: +  -If a merge request is eventually merged to the default branch (in our case -`master`) and that branch also deploys to an environment (in our case `staging` -and/or `production`) you can see this information in the merge request itself. +You can see this information in a merge request itself if: + +- The merge request is eventually merged to the default branch (usually `master`). +- That branch also deploys to an environment (for example, `staging` or `production`). + +For example:  -### Go directly from source files to public pages on the environment +#### Going from source files to public pages With GitLab's [Route Maps](review_apps/index.md#route-maps) you can go directly -from source files to public pages on the environment set for Review Apps. +from source files to public pages in the environment set for Review Apps. + +### Stopping an environment -From then on, you have a full development cycle, where your app is tested, built, deployed -as a Review App, deployed to a staging server once the merge request is merged, -and finally manually deployed to the production server. This is a simple workflow, -but when you have multiple developers working on a project -at the same time, each of them pushing to their own branches, dynamic environments are -created all the time. In which case, you probably want to do some clean up. Read -next how environments can be stopped. +Stopping an environment: -## Stopping an environment +- Moves it from the list of **Available** environments to the list of **Stopped** + environments on the [**Environments** page](#viewing-environments-and-deployments). +- Executes an [`on_stop` action](yaml/README.md#environmenton_stop), if defined. -By stopping an environment, you are effectively terminating its recording of the -deployments that happen in it. +This is often used when multiple developers are working on a project at the same time, +each of them pushing to their own branches, causing many dynamic environments to be created. -A branch is associated with an environment when the CI pipeline that is created -for this branch, was recently deployed to this environment. You can think of -the CI pipeline as the glue between the branch and the environment: -`branch ➔ CI pipeline ➔ environment`. +NOTE: **Note:** +Starting with GitLab 8.14, dynamic environments are stopped automatically +when their associated branch is deleted. -There is a special case where environments can be manually stopped. That can -happen if you provide another job for that matter. The syntax is a little -tricky since a job calls another job to do the job. +#### Automatically stopping an environment -Consider the following example where the `deploy_review` calls the `stop_review` +Environments can be stopped automatically using special configuration. + +Consider the following example where the `deploy_review` job calls `stop_review` to clean up and stop the environment: ```yaml @@ -470,35 +549,31 @@ stop_review: action: stop ``` -Setting the [`GIT_STRATEGY`][git-strategy] to `none` is necessary on the -`stop_review` job so that the [GitLab Runner] won't try to checkout the code -after the branch is deleted. - ->**Note:** -Starting with GitLab 8.14, dynamic environments will be stopped automatically -when their associated branch is deleted. +Setting the [`GIT_STRATEGY`](yaml/README.md#git-strategy) to `none` is necessary in the +`stop_review` job so that the [GitLab Runner](https://docs.gitlab.com/runner/) won't +try to check out the code after the branch is deleted. When you have an environment that has a stop action defined (typically when -the environment describes a review app), GitLab will automatically trigger a +the environment describes a Review App), GitLab will automatically trigger a stop action when the associated branch is deleted. The `stop_review` job must -be in the same `stage` as the `deploy_review` one in order for the environment +be in the same `stage` as the `deploy_review` job in order for the environment to automatically stop. -You can read more in the [`.gitlab-ci.yml` reference][onstop]. +You can read more in the [`.gitlab-ci.yml` reference](yaml/README.md#environmenton_stop). -## Grouping similar environments +### Grouping similar environments -> [Introduced][ce-7015] in GitLab 8.14. +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/7015) in GitLab 8.14. -As we've seen in the [dynamic environments](#dynamic-environments), you can -prepend their name with a word, then followed by a `/` and finally the branch -name which is automatically defined by the `CI_COMMIT_REF_NAME` variable. +As documented in [Configuring dynamic environments](#configuring-dynamic-environments), you can +prepend environment name with a word, followed by a `/`, and finally the branch +name, which is automatically defined by the `CI_COMMIT_REF_NAME` variable. -In short, environments that are named like `type/foo` are presented under a -group named `type`. +In short, environments that are named like `type/foo` are all presented under the same +group, named `type`. -In our minimal example, we name the environments `review/$CI_COMMIT_REF_NAME` -where `$CI_COMMIT_REF_NAME` is the branch name: +In our [minimal example](#example-configuration), we named the environments `review/$CI_COMMIT_REF_NAME` +where `$CI_COMMIT_REF_NAME` is the branch name. Here is a snippet of the example: ```yaml deploy_review: @@ -509,50 +584,49 @@ deploy_review: name: review/$CI_COMMIT_REF_NAME ``` -In that case, if you visit the Environments page, and provided the branches +In this case, if you visit the **Environments** page and the branches exist, you should see something like:  -## Monitoring environments +### Monitoring environments > **Notes:** > > - For the monitoring dashboard to appear, you need to: -> - Have enabled the [Prometheus integration][prom] -> - Configured Prometheus to collect at least one [supported metric](../user/project/integrations/prometheus_library/index.md) -> - With GitLab 9.2, all deployments to an environment are shown directly on the -> monitoring dashboard +> - Enable the [Prometheus integration](../user/project/integrations/prometheus.md). +> - Configure Prometheus to collect at least one [supported metric](../user/project/integrations/prometheus_library/index.md). +> - With GitLab 9.2, all deployments to an environment are shown directly on the monitoring dashboard. -If you have enabled [Prometheus for monitoring system and response metrics](https://docs.gitlab.com/ee/user/project/integrations/prometheus.html), you can monitor the performance behavior of your app running in each environment. +If you have enabled [Prometheus for monitoring system and response metrics](../user/project/integrations/prometheus.md), +you can monitor the behavior of your app running in each environment. -Once configured, GitLab will attempt to retrieve [supported performance metrics](https://docs.gitlab.com/ee/user/project/integrations/prometheus_library/index.html) for any -environment which has had a successful deployment. If monitoring data was -successfully retrieved, a Monitoring button will appear for each environment. +Once configured, GitLab will attempt to retrieve [supported performance metrics](../user/project/integrations/prometheus_library/index.md) +for any environment that has had a successful deployment. If monitoring data was +successfully retrieved, a **Monitoring** button will appear for each environment.  -Clicking on the Monitoring button will display a new page, showing up to the last +Clicking on the **Monitoring** button will display a new page showing up to the last 8 hours of performance data. It may take a minute or two for data to appear after initial deployment. -All deployments to an environment are shown directly on the monitoring dashboard -which allows easy correlation between any changes in performance and a new -version of the app, all without leaving GitLab. +All deployments to an environment are shown directly on the monitoring dashboard, +which allows easy correlation between any changes in performance and new +versions of the app, all without leaving GitLab.  -## Web terminals +### Web terminals ->**Note:** -Web terminals were added in GitLab 8.15 and are only available to project -maintainers and owners. +> Web terminals were added in GitLab 8.15 and are only available to project Maintainers and Owners. -If you deploy to your environments with the help of a deployment service (e.g., -the [Kubernetes integration][kube]), GitLab can open -a terminal session to your environment! This is a very powerful feature that -allows you to debug issues without leaving the comfort of your web browser. To -enable it, just follow the instructions given in the service integration +If you deploy to your environments with the help of a deployment service (for example, +the [Kubernetes integration](../user/project/clusters/index.md)), GitLab can open +a terminal session to your environment. + +This is a powerful feature that allows you to debug issues without leaving the comfort +of your web browser. To enable it, just follow the instructions given in the service integration documentation. Once enabled, your environments will gain a "terminal" button: @@ -568,41 +642,86 @@ establish the terminal session:  -This works just like any other terminal - you'll be in the container created -by your deployment, so you can run shell commands and get responses in real -time, check the logs, try out configuration or code tweaks, etc. You can open -multiple terminals to the same environment - they each get their own shell -session - and even a multiplexer like `screen` or `tmux`! +This works just like any other terminal. You'll be in the container created +by your deployment so you can: ->**Note:** -Container-based deployments often lack basic tools (like an editor), and may -be stopped or restarted at any time. If this happens, you will lose all your -changes! Treat this as a debugging tool, not a comprehensive online IDE. +- Run shell commands and get responses in real time. +- Check the logs. +- Try out configuration or code tweaks etc. ---- +You can open multiple terminals to the same environment, they each get their own shell +session and even a multiplexer like `screen` or `tmux`. -While this is fine for deploying to some stable environments like staging or -production, what happens for branches? So far we haven't defined anything -regarding deployments for branches other than `master`. Dynamic environments -will help us achieve that. +NOTE: **Note:** +Container-based deployments often lack basic tools (like an editor), and may +be stopped or restarted at any time. If this happens, you will lose all your +changes. Treat this as a debugging tool, not a comprehensive online IDE. -## Checkout deployments locally +### Check out deployments locally -Since 8.13, a reference in the git repository is saved for each deployment, so +Since GitLab 8.13, a reference in the Git repository is saved for each deployment, so knowing the state of your current environments is only a `git fetch` away. -In your git config, append the `[remote "<your-remote>"]` block with an extra +In your Git configuration, append the `[remote "<your-remote>"]` block with an extra fetch line: -``` +```text fetch = +refs/environments/*:refs/remotes/origin/environments/* ``` +### Scoping environments with specs **[PREMIUM]** + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/2112) in [GitLab Premium](https://about.gitlab.com/pricing/) 9.4. + +You can limit the environment scope of a variable by +defining which environments it can be available for. + +Wildcards can be used, and the default environment scope is `*`, which means +any jobs will have this variable, not matter if an environment is defined or +not. + +For example, if the environment scope is `production`, then only the jobs +having the environment `production` defined would have this specific variable. +Wildcards (`*`) can be used along with the environment name, therefore if the +environment scope is `review/*` then any jobs with environment names starting +with `review/` would have that particular variable. + +Some GitLab features can behave differently for each environment. +For example, you can +[create a secret variable to be injected only into a production environment](variables/README.md#limiting-environment-scopes-of-environment-variables-premium). **[PREMIUM]** + +In most cases, these features use the _environment specs_ mechanism, which offers +an efficient way to implement scoping within each environment group. + +Let's say there are four environments: + +- `production` +- `staging` +- `review/feature-1` +- `review/feature-2` + +Each environment can be matched with the following environment spec: + +| Environment Spec | `production` | `staging` | `review/feature-1` | `review/feature-2` | +|:-----------------|:-------------|:----------|:-------------------|:-------------------| +| * | Matched | Matched | Matched | Matched | +| production | Matched | | | | +| staging | | Matched | | | +| review/* | | | Matched | Matched | +| review/feature-1 | | | Matched | | + +As you can see, you can use specific matching for selecting a particular environment, +and also use wildcard matching (`*`) for selecting a particular environment group, +such as [Review Apps](review_apps/index.md) (`review/*`). + +NOTE: **Note:** +The most _specific_ spec takes precedence over the other wildcard matching. +In this case, `review/feature-1` spec takes precedence over `review/*` and `*` specs. + ## Limitations -1. You are limited to use only the [CI predefined variables][variables] in the - `environment: name`. If you try to re-use variables defined inside `script` - as part of the environment name, it will not work. +In the `environment: name`, you are limited to only the [predefined environment variables](variables/predefined_variables.md). +Re-using variables defined inside `script` as part of the environment name will not work. ## Further reading @@ -611,20 +730,16 @@ Below are some links you may find interesting: - [The `.gitlab-ci.yml` definition of environments](yaml/README.md#environment) - [A blog post on Deployments & Environments](https://about.gitlab.com/2016/08/26/ci-deployment-and-environments/) - [Review Apps - Use dynamic environments to deploy your code for every branch](review_apps/index.md) +- [Deploy Boards for your applications running on Kubernetes](https://docs.gitlab.com/ee/user/project/deploy_boards.html) **[PREMIUM]** + +<!-- ## Troubleshooting + +Include any troubleshooting steps that you can foresee. If you know beforehand what issues +one might have when setting this up, or when something is changed, or on upgrading, it's +important to describe those, too. Think of things that may go wrong and include them here. +This is important to minimize requests for support, and to avoid doc comments with +questions that you know someone might ask. -[Pipelines]: pipelines.md -[jobs]: yaml/README.md#jobs -[yaml]: yaml/README.md -[environments]: #environments -[deployments]: #deployments -[permissions]: ../user/permissions.md -[variables]: variables/README.md -[env-name]: yaml/README.md#environmentname -[only]: yaml/README.md#only-and-except-simplified -[onstop]: yaml/README.md#environmenton_stop -[ce-7015]: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/7015 -[gitlab-flow]: ../workflow/gitlab_flow.md -[gitlab runner]: https://docs.gitlab.com/runner/ -[git-strategy]: yaml/README.md#git-strategy -[kube]: ../user/project/clusters/index.md -[prom]: ../user/project/integrations/prometheus.md +Each scenario can be a third-level heading, e.g. `### Getting error message X`. +If you have none to add when creating a doc, leave this section in place +but commented out to help encourage others to add to it in the future. --> diff --git a/doc/ci/environments/protected_environments.md b/doc/ci/environments/protected_environments.md new file mode 100644 index 00000000000..b72ebe838b8 --- /dev/null +++ b/doc/ci/environments/protected_environments.md @@ -0,0 +1,65 @@ +--- +type: concepts, howto +--- + +# Protected Environments **[PREMIUM]** + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/6303) in [GitLab Premium](https://about.gitlab.com/pricing/) 11.3. + +## Overview + +[Environments](../environments.md) can be used for different reasons: + +- Some of them are just for testing. +- Others are for production. + +Since deploy jobs can be raised by different users with different roles, it is important that +specific environments are "protected" to prevent unauthorized people from affecting them. + +By default, a protected environment does one thing: it ensures that only people +with the right privileges can deploy to it, thus keeping it safe. + +NOTE: **Note**: +A GitLab admin is always allowed to use environments, even if they are protected. + +To protect, update, or unprotect an environment, you need to have at least +[Maintainer permissions](../../user/permissions.md). + +## Protecting environments + +To protect an environment: + +1. Navigate to your project's **Settings > CI/CD**. +1. Expand the **Protected Environments** section. +1. From the **Environment** dropdown menu, select the environment you want to protect. +1. In the **Allowed to Deploy** dropdown menu, select the role, users, or groups you + want to give deploy access to. Keep in mind that: + - There are two roles to choose from: + - **Maintainers**: will allow access to all maintainers in the project. + - **Developers**: will allow access to all maintainers and all developers in the project. + - You can only select groups that are already associated with the project. + - Only users that have at least Developer permission level will appear in + the **Allowed to Deploy** dropdown menu. +1. Click the **Protect** button. + +The protected environment will now appear in the list of protected environments. + +## Modifying and unprotecting environments + +Maintainers can: + +- Update existing protected environments at any time by changing the access in the + **Allowed to Deploy** dropdown menu. +- Unprotect a protected environment by clicking the **Unprotect** button for that environment. + +<!-- ## Troubleshooting + +Include any troubleshooting steps that you can foresee. If you know beforehand what issues +one might have when setting this up, or when something is changed, or on upgrading, it's +important to describe those, too. Think of things that may go wrong and include them here. +This is important to minimize requests for support, and to avoid doc comments with +questions that you know someone might ask. + +Each scenario can be a third-level heading, e.g. `### Getting error message X`. +If you have none to add when creating a doc, leave this section in place +but commented out to help encourage others to add to it in the future. --> diff --git a/doc/ci/examples/README.md b/doc/ci/examples/README.md index 87e86bef44b..340a41c196b 100644 --- a/doc/ci/examples/README.md +++ b/doc/ci/examples/README.md @@ -4,88 +4,118 @@ comments: false # GitLab CI/CD Examples -A collection of [`.gitlab-ci.yml` template files][gitlab-ci-templates] is maintained in GitLab. When you create a new file via the UI, -GitLab will give you the option to choose one of these templates. -If your favorite programming language or framework are missing we would love your -help by sending a merge request with a new `.gitlab-ci.yml` to this project. - -There's also a collection of repositories with [example projects](https://gitlab.com/gitlab-examples) for various languages. You can fork and adjust them to your own needs. - -## Languages, frameworks, OSs - -- **PHP**: - - [Testing a PHP application](php.md) - - [Run PHP Composer & NPM scripts then deploy them to a staging server](deployment/composer-npm-deploy.md) - - [How to test and deploy Laravel/PHP applications with GitLab CI/CD and Envoy](laravel_with_gitlab_and_envoy/index.md) -- **Ruby**: [Test and deploy a Ruby application to Heroku](test-and-deploy-ruby-application-to-heroku.md) -- **Python**: [Test and deploy a Python application to Heroku](test-and-deploy-python-application-to-heroku.md) -- **Java**: - - [Deploy a Spring Boot application to Cloud Foundry with GitLab CI/CD](deploy_spring_boot_to_cloud_foundry/index.md) - - [Continuous Delivery of a Spring Boot application with GitLab CI and Kubernetes](https://about.gitlab.com/2016/12/14/continuous-delivery-of-a-spring-boot-application-with-gitlab-ci-and-kubernetes/) -- **Scala**: [Test a Scala application](test-scala-application.md) -- **Clojure**: [Test a Clojure application](test-clojure-application.md) -- **Elixir**: - - [Testing a Phoenix application with GitLab CI/CD](test_phoenix_app_with_gitlab_ci_cd/index.md) - - [Building an Elixir Release into a Docker image using GitLab CI](https://about.gitlab.com/2016/08/11/building-an-elixir-release-into-docker-image-using-gitlab-ci-part-1/) -- **iOS and macOS**: - - [Setting up GitLab CI for iOS projects](https://about.gitlab.com/2016/03/10/setting-up-gitlab-ci-for-ios-projects/) - - [How to use GitLab CI and MacStadium to build your macOS or iOS projects](https://about.gitlab.com/2017/05/15/how-to-use-macstadium-and-gitlab-ci-to-build-your-macos-or-ios-projects/) -- **Android**: [Setting up GitLab CI for Android projects](https://about.gitlab.com/2016/11/30/setting-up-gitlab-ci-for-android-projects/) -- **Debian**: [Continuous Deployment with GitLab: how to build and deploy a Debian Package with GitLab CI](https://about.gitlab.com/2016/10/12/automated-debian-package-build-with-gitlab-ci/) -- **Maven**: [How to deploy Maven projects to Artifactory with GitLab CI/CD](artifactory_and_gitlab/index.md) - -### Game development - -- [DevOps and Game Dev with GitLab CI/CD](devops_and_game_dev_with_gitlab_ci_cd/index.md) - -### Miscellaneous +Examples are a useful way of understanding how to implement GitLab CI/CD for your specific use case. + +Examples are available in several forms. As a collection of: + +- `.gitlab-ci.yml` [template files](https://gitlab.com/gitlab-org/gitlab-ce/tree/master/lib/gitlab/ci/templates) maintained in GitLab. When you create a new file via the UI, + GitLab will give you the option to choose one of these templates. This will allow you to quickly bootstrap your project for CI/CD. + If your favorite programming language or framework are missing, we would love your help by sending a merge request with a new `.gitlab-ci.yml` to this project. +- Repositories with [example projects](https://gitlab.com/gitlab-examples) for various languages. You can fork and adjust them to your own needs. +- Examples and [other resources](#other-resources) listed below. + +## CI/CD examples + +The following table lists examples for different use cases: + +| Use case | Resource | +|:-----------------------------------------------|:------------------------------------------------------------------------------------------------------------------------| +| Browser performance testing | [Browser Performance Testing with the Sitespeed.io container](browser_performance.md). | +| Clojure | [Test a Clojure application with GitLab CI/CD](test-clojure-application.md). | +| Code quality analysis | [Analyze your project's Code Quality](code_quality.md). **[STARTER]** | +| Container scanning | [Container Scanning with GitLab CI/CD](../../user/application_security/container_scanning/index.md). **[ULTIMATE]** | +| Dependency scanning | [Dependency Scanning with GitLab CI/CD](../../user/application_security/dependency_scanning/index.md). **[ULTIMATE]** | +| Deployment with `dpl` | [Using `dpl` as deployment tool](deployment/README.md). | +| Dynamic application<br>security testing (DAST) | [Dynamic Application Security Testing with GitLab CI/CD](../../user/application_security/dast/index.md). **[ULTIMATE]** | +| Elixir | [Testing a Phoenix application with GitLab CI/CD](test_phoenix_app_with_gitlab_ci_cd/index.md). | +| Game development | [DevOps and Game Dev with GitLab CI/CD](devops_and_game_dev_with_gitlab_ci_cd/index.md). | +| GitLab Pages | See the [GitLab Pages](../../user/project/pages/index.md) documentation for a complete example. | +| Java | [Deploy a Spring Boot application to Cloud Foundry with GitLab CI/CD](deploy_spring_boot_to_cloud_foundry/index.md). | +| JUnit | [JUnit test reports](../junit_test_reports.md). | +| License management | [Dependencies license management with GitLab CI/CD](../../user/application_security/license_management/index.md). **[ULTIMATE]** | +| Maven | [How to deploy Maven projects to Artifactory with GitLab CI/CD](artifactory_and_gitlab/index.md). | +| PHP | [Testing PHP projects](php.md). | +| PHP | [Running Composer and NPM scripts with deployment via SCP in GitLab CI/CD](deployment/composer-npm-deploy.md). | +| PHP | [Test and deploy Laravel applications with GitLab CI/CD and Envoy](laravel_with_gitlab_and_envoy/index.md). | +| Python | [Test and deploy a Python application with GitLab CI/CD](test-and-deploy-python-application-to-heroku.md). | +| Ruby | [Test and deploy a Ruby application with GitLab CI/CD](test-and-deploy-ruby-application-to-heroku.md). | +| Scala | [Test and deploy a Scala application to Heroku](test-scala-application.md). | +| Static application<br>security testing (SAST) | [Static Application Security Testing with GitLab CI/CD](../../user/application_security/sast/index.md). **[ULTIMATE]** | +| Testing | [End-to-end testing with GitLab CI/CD and WebdriverIO](end_to_end_testing_webdriverio/index.md). | + +### Contributing examples + +Contributions are welcome! You can help your favorite programming +language users and GitLab by sending a merge request with a guide for that language. +You may want to apply for the [GitLab Community Writers Program](https://about.gitlab.com/community-writers/) +to get paid for writing complete articles for GitLab. -- [End-to-end testing with GitLab CI/CD and WebdriverIO](end_to_end_testing_webdriverio/index.md) -- [Using `dpl` as deployment tool](deployment/README.md) -- [The `.gitlab-ci.yml` file for GitLab itself](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/.gitlab-ci.yml) +### Adding templates to your GitLab installation **[PREMIUM ONLY]** -## Test Reports +If you want to have customized examples and templates for your own self-managed GitLab instance available to your team, your GitLab administrator can [designate an instance template repository](https://docs.gitlab.com/ee/user/admin_area/settings/instance_template_repository.html) that contains examples and templates specific to your enterprise. -[Collect test reports in Verify stage](../junit_test_reports.md) +## Other resources -## Code Quality analysis +This section provides further resources to help you get familiar with different aspects of GitLab CI/CD. -**(Starter)** [Analyze your project's Code Quality](code_quality.md) +NOTE: **Note:** +These resources may no longer reflect the current state of GitLab CI/CD. -## Static Application Security Testing (SAST) +### CI/CD in the cloud -**(Ultimate)** [Scan your code for vulnerabilities](https://docs.gitlab.com/ee/ci/examples/sast.html) +For examples of setting up GitLab CI/CD for cloud-based environments, see: -## Dependency Scanning +- [How to set up multi-account AWS SAM deployments with GitLab CI](https://about.gitlab.com/2019/02/04/multi-account-aws-sam-deployments-with-gitlab-ci/) +- [How to autoscale continuous deployment with GitLab Runner on DigitalOcean](https://about.gitlab.com/2018/06/19/autoscale-continuous-deployment-gitlab-runner-digital-ocean/) +- [How to create a CI/CD pipeline with Auto Deploy to Kubernetes using GitLab and Helm](https://about.gitlab.com/2017/09/21/how-to-create-ci-cd-pipeline-with-autodeploy-to-kubernetes-using-gitlab-and-helm/) -**(Ultimate)** [Scan your dependencies for vulnerabilities](https://docs.gitlab.com/ee/ci/examples/dependency_scanning.html) +### Customer stories -## Container Scanning +For some customer experiences with GitLab CI/CD, see: -[Scan your Docker images for vulnerabilities](container_scanning.md) +- [How Verizon Connect reduced datacenter deploys from 30 days to under 8 hours with GitLab](https://about.gitlab.com/2019/02/14/verizon-customer-story/) +- [How Wag! cut their release process from 40 minutes to just 6](https://about.gitlab.com/2019/01/16/wag-labs-blog-post/) +- [How Jaguar Land Rover embraced CI to speed up their software lifecycle](https://about.gitlab.com/2018/07/23/chris-hill-devops-enterprise-summit-talk/) -## Dynamic Application Security Testing (DAST) +### Getting started -Scan your app for vulnerabilities with GitLab [Dynamic Application Security Testing (DAST)](dast.md) +For some examples to help get you started, see: -## Browser Performance Testing with Sitespeed.io +- [GitLab CI/CD's 2018 highlights](https://about.gitlab.com/2019/01/21/gitlab-ci-cd-features-improvements/) +- [A beginner's guide to continuous integration](https://about.gitlab.com/2018/01/22/a-beginners-guide-to-continuous-integration/) +- [Making CI easier with GitLab](https://about.gitlab.com/2017/07/13/making-ci-easier-with-gitlab/) -Analyze your [browser performance with Sitespeed.io](browser_performance.md) +### Implementing GitLab CI/CD -## GitLab CI/CD for Review Apps +For examples of others who have implemented GitLab CI/CD, see: -- [Example project](https://gitlab.com/gitlab-examples/review-apps-nginx/) that shows how to use GitLab CI/CD for [Review Apps](../review_apps/index.html) +- [How to streamline interactions between multiple repositories with multi-project pipelines](https://about.gitlab.com/2018/10/31/use-multiproject-pipelines-with-gitlab-cicd/) +- [How we used GitLab CI to build GitLab faster](https://about.gitlab.com/2018/05/02/using-gitlab-ci-to-build-gitlab-faster/) +- [Test all the things in GitLab CI with Docker by example](https://about.gitlab.com/2018/02/05/test-all-the-things-gitlab-ci-docker-examples/) +- [A Craftsman looks at continuous integration](https://about.gitlab.com/2018/01/17/craftsman-looks-at-continuous-integration/) +- [Go tools and GitLab: How to do continuous integration like a boss](https://about.gitlab.com/2017/11/27/go-tools-and-gitlab-how-to-do-continuous-integration-like-a-boss/) +- [GitBot – automating boring Git operations with CI](https://about.gitlab.com/2017/11/02/automating-boring-git-operations-gitlab-ci/) +- [How to use GitLab CI for Vue.js](https://about.gitlab.com/2017/09/12/vuejs-app-gitlab/) +- Video: [GitLab CI/CD Deep Dive](https://youtu.be/pBe4t1CD8Fc?t=195) - [Dockerizing GitLab Review Apps](https://about.gitlab.com/2017/07/11/dockerizing-review-apps/) +- [Fast and natural continuous integration with GitLab CI](https://about.gitlab.com/2017/05/22/fast-and-natural-continuous-integration-with-gitlab-ci/) +- [Demo: CI/CD with GitLab in action](https://about.gitlab.com/2017/03/13/ci-cd-demo/) -## GitLab CI/CD for GitLab Pages +### Integrating GitLab CI/CD with other systems -See the documentation on [GitLab Pages](../../user/project/pages/index.md) for a complete overview. +To see how you can integrate GitLab CI/CD with third-party systems, see: -## Contributing +- [Streamline and shorten error remediation with Sentry’s new GitLab integration](https://about.gitlab.com/2019/01/25/sentry-integration-blog-post/) +- [How to simplify your smart home configuration with GitLab CI/CD](https://about.gitlab.com/2018/08/02/using-the-gitlab-ci-slash-cd-for-smart-home-configuration-management/) +- [Demo: GitLab + Jira + Jenkins](https://about.gitlab.com/2018/07/30/gitlab-workflow-with-jira-jenkins/) +- [Introducing Auto Breakfast from GitLab (sort of)](https://about.gitlab.com/2018/06/29/introducing-auto-breakfast-from-gitlab/) -Contributions are very welcome! You can help your favorite programming -language users and GitLab by sending a merge request with a guide for that language. -You may want to apply for the [GitLab Community Writers Program](https://about.gitlab.com/community-writers/) -to get paid for writing complete articles for GitLab. +### Mobile development + +For help with using GitLab CI/CD for mobile application development, see: -[gitlab-ci-templates]: https://gitlab.com/gitlab-org/gitlab-ce/tree/master/lib/gitlab/ci/templates +- [How to publish Android apps to the Google Play Store with GitLab and fastlane](https://about.gitlab.com/2019/01/28/android-publishing-with-gitlab-and-fastlane/) +- [Setting up GitLab CI for Android projects](https://about.gitlab.com/2018/10/24/setting-up-gitlab-ci-for-android-projects/) +- [Working with YAML in GitLab CI from the Android perspective](https://about.gitlab.com/2017/11/20/working-with-yaml-gitlab-ci-android/) +- [How to use GitLab CI and MacStadium to build your macOS or iOS projects](https://about.gitlab.com/2017/05/15/how-to-use-macstadium-and-gitlab-ci-to-build-your-macos-or-ios-projects/) +- [Setting up GitLab CI for iOS projects](https://about.gitlab.com/2016/03/10/setting-up-gitlab-ci-for-ios-projects/) diff --git a/doc/ci/examples/artifactory_and_gitlab/index.md b/doc/ci/examples/artifactory_and_gitlab/index.md index 9e657275d50..589912e7a2a 100644 --- a/doc/ci/examples/artifactory_and_gitlab/index.md +++ b/doc/ci/examples/artifactory_and_gitlab/index.md @@ -2,7 +2,7 @@ redirect_from: 'https://docs.gitlab.com/ee/articles/artifactory_and_gitlab/index.html' author: Fabio Busatto author_gitlab: bikebilly -level: intermediary +level: intermediate article_type: tutorial date: 2017-08-15 --- @@ -11,7 +11,7 @@ date: 2017-08-15 ## Introduction -In this article, we will show how you can leverage the power of [GitLab CI/CD](https://about.gitlab.com/features/gitlab-ci-cd/) +In this article, we will show how you can leverage the power of [GitLab CI/CD](https://about.gitlab.com/product/continuous-integration/) to build a [Maven](https://maven.apache.org/) project, deploy it to [Artifactory](https://www.jfrog.com/artifactory/), and then use it from another Maven application as a dependency. You'll create two different projects: @@ -19,7 +19,7 @@ You'll create two different projects: - `simple-maven-dep`: the app built and deployed to Artifactory (available at <https://gitlab.com/gitlab-examples/maven/simple-maven-dep>) - `simple-maven-app`: the app using the previous one as a dependency (available at <https://gitlab.com/gitlab-examples/maven/simple-maven-app>) -We assume that you already have a GitLab account on [GitLab.com](https://gitlab.com/), and that you know the basic usage of Git and [GitLab CI/CD](https://about.gitlab.com/features/gitlab-ci-cd/). +We assume that you already have a GitLab account on [GitLab.com](https://gitlab.com/), and that you know the basic usage of Git and [GitLab CI/CD](https://about.gitlab.com/product/continuous-integration/). We also assume that an Artifactory instance is available and reachable from the internet, and that you have valid credentials to deploy on it. ## Create the simple Maven dependency @@ -102,7 +102,7 @@ parameter in `.gitlab-ci.yml` to use the custom location instead of the default ### Configure GitLab CI/CD for `simple-maven-dep` -Now it's time we set up [GitLab CI/CD](https://about.gitlab.com/features/gitlab-ci-cd/) to automatically build, test and deploy the dependency! +Now it's time we set up [GitLab CI/CD](https://about.gitlab.com/product/continuous-integration/) to automatically build, test and deploy the dependency! GitLab CI/CD uses a file in the root of the repo, named `.gitlab-ci.yml`, to read the definitions for jobs that will be executed by the configured GitLab Runners. You can read more about this file in the [GitLab Documentation](https://docs.gitlab.com/ee/ci/yaml/). @@ -230,7 +230,7 @@ Now you are ready to use the Artifactory repository to resolve dependencies and You need a last step to have everything in place: configure the `.gitlab-ci.yml` file for this project, as you already did for `simple-maven-dep`. -You want to leverage [GitLab CI/CD](https://about.gitlab.com/features/gitlab-ci-cd/) to automatically build, test and run your awesome application, +You want to leverage [GitLab CI/CD](https://about.gitlab.com/product/continuous-integration/) to automatically build, test and run your awesome application, and see if you can get the greeting as expected! All you need to do is to add the following `.gitlab-ci.yml` to the repo: diff --git a/doc/ci/examples/browser_performance.md b/doc/ci/examples/browser_performance.md index b47038011de..442d0788d37 100644 --- a/doc/ci/examples/browser_performance.md +++ b/doc/ci/examples/browser_performance.md @@ -56,7 +56,7 @@ provide a list of URLs to test, please consult TIP: **Tip:** For [GitLab Premium](https://about.gitlab.com/pricing/) users, key metrics are automatically extracted and shown right in the merge request widget. -[Learn more on Browser Performance Testing in merge requests](https://docs.gitlab.com/ee//user/project/merge_requests/browser_performance_testing.html). +[Learn more on Browser Performance Testing in merge requests](https://docs.gitlab.com/ee/user/project/merge_requests/browser_performance_testing.html). ## Performance testing on Review Apps diff --git a/doc/ci/examples/code_quality.md b/doc/ci/examples/code_quality.md index 3e7d6e7e3f7..186d4527bb6 100644 --- a/doc/ci/examples/code_quality.md +++ b/doc/ci/examples/code_quality.md @@ -1,7 +1,7 @@ # Analyze your project's Code Quality CAUTION: **Caution:** -The job definition shown below is supported on GitLab 11.5 and later versions. +The job definition shown below is supported on GitLab 11.11 and later versions. It also requires the GitLab Runner 11.5 or later. For earlier versions, use the [previous job definitions](#previous-job-definitions). @@ -11,27 +11,11 @@ and Docker. First, you need GitLab Runner with [docker-in-docker executor](../docker/using_docker_build.md#use-docker-in-docker-executor). -Once you set up the Runner, add a new job to `.gitlab-ci.yml` that -generates the expected report: +Once you set up the Runner, include the CodeQuality template in your CI config: ```yaml -code_quality: - image: docker:stable - variables: - DOCKER_DRIVER: overlay2 - allow_failure: true - services: - - docker:stable-dind - script: - - export SP_VERSION=$(echo "$CI_SERVER_VERSION" | sed 's/^\([0-9]*\)\.\([0-9]*\).*/\1-\2-stable/') - - docker run - --env SOURCE_CODE="$PWD" - --volume "$PWD":/code - --volume /var/run/docker.sock:/var/run/docker.sock - "registry.gitlab.com/gitlab-org/security-products/codequality:$SP_VERSION" /code - artifacts: - reports: - codequality: gl-code-quality-report.json +include: + - template: Code-Quality.gitlab-ci.yml ``` The above example will create a `code_quality` job in your CI/CD pipeline which @@ -54,6 +38,28 @@ While these old job definitions are still maintained they have been deprecated and may be removed in next major release, GitLab 12.0. You are advised to update your current `.gitlab-ci.yml` configuration to reflect that change. +For GitLab 11.5 and earlier, the job should look like: + +```yaml +code_quality: + image: docker:stable + variables: + DOCKER_DRIVER: overlay2 + allow_failure: true + services: + - docker:stable-dind + script: + - export SP_VERSION=$(echo "$CI_SERVER_VERSION" | sed 's/^\([0-9]*\)\.\([0-9]*\).*/\1-\2-stable/') + - docker run + --env SOURCE_CODE="$PWD" + --volume "$PWD":/code + --volume /var/run/docker.sock:/var/run/docker.sock + "registry.gitlab.com/gitlab-org/security-products/codequality:$SP_VERSION" /code + artifacts: + reports: + codequality: gl-code-quality-report.json +``` + For GitLab 11.4 and earlier, the job should look like: ```yaml diff --git a/doc/ci/examples/container_scanning.md b/doc/ci/examples/container_scanning.md index e8e9c73d1b2..4d41e424f4a 100644 --- a/doc/ci/examples/container_scanning.md +++ b/doc/ci/examples/container_scanning.md @@ -1,119 +1,5 @@ -# Container Scanning with GitLab CI/CD +--- +redirect_to: 'https://docs.gitlab.com/ee/user/application_security/container_scanning/index.html' +--- -CAUTION: **Caution:** -The job definition shown below is supported on GitLab 11.5 and later versions. -It also requires the GitLab Runner 11.5 or later. -For earlier versions, use the [previous job definitions](#previous-job-definitions). - -You can check your Docker images (or more precisely the containers) for known -vulnerabilities by using [Clair](https://github.com/coreos/clair) and -[clair-scanner](https://github.com/arminc/clair-scanner), two open source tools -for Vulnerability Static Analysis for containers. - -First, you need GitLab Runner with -[docker-in-docker executor](../docker/using_docker_build.md#use-docker-in-docker-executor). - -Once you set up the Runner, add a new job to `.gitlab-ci.yml` that -generates the expected report: - -```yaml -container_scanning: - image: docker:stable - variables: - DOCKER_DRIVER: overlay2 - ## Define two new variables based on GitLab's CI/CD predefined variables - ## https://docs.gitlab.com/ee/ci/variables/#predefined-environment-variables - CI_APPLICATION_REPOSITORY: $CI_REGISTRY_IMAGE/$CI_COMMIT_REF_SLUG - CI_APPLICATION_TAG: $CI_COMMIT_SHA - allow_failure: true - services: - - docker:stable-dind - script: - - docker run -d --name db arminc/clair-db:latest - - docker run -p 6060:6060 --link db:postgres -d --name clair --restart on-failure arminc/clair-local-scan:v2.0.1 - - apk add -U wget ca-certificates - - docker pull ${CI_APPLICATION_REPOSITORY}:${CI_APPLICATION_TAG} - - wget https://github.com/arminc/clair-scanner/releases/download/v8/clair-scanner_linux_amd64 - - mv clair-scanner_linux_amd64 clair-scanner - - chmod +x clair-scanner - - touch clair-whitelist.yml - - while( ! wget -q -O /dev/null http://docker:6060/v1/namespaces ) ; do sleep 1 ; done - - retries=0 - - echo "Waiting for clair daemon to start" - - while( ! wget -T 10 -q -O /dev/null http://docker:6060/v1/namespaces ) ; do sleep 1 ; echo -n "." ; if [ $retries -eq 10 ] ; then echo " Timeout, aborting." ; exit 1 ; fi ; retries=$(($retries+1)) ; done - - ./clair-scanner -c http://docker:6060 --ip $(hostname -i) -r gl-container-scanning-report.json -l clair.log -w clair-whitelist.yml ${CI_APPLICATION_REPOSITORY}:${CI_APPLICATION_TAG} || true - artifacts: - reports: - container_scanning: gl-container-scanning-report.json -``` - -The above example will create a `container_scanning` job in your CI/CD pipeline, pull -the image from the [Container Registry](../../user/project/container_registry.md) -(whose name is defined from the two `CI_APPLICATION_` variables) and scan it -for possible vulnerabilities. The report will be saved as a -[Container Scanning report artifact](../yaml/README.md#artifactsreportscontainer_scanning-ultimate) -that you can later download and analyze. -Due to implementation limitations we always take the latest Container Scanning artifact available. - -If you want to whitelist some specific vulnerabilities, you can do so by defining -them in a [YAML file](https://github.com/arminc/clair-scanner/blob/master/README.md#example-whitelist-yaml-file), -in our case its named `clair-whitelist.yml`. - -TIP: **Tip:** -For [GitLab Ultimate][ee] users, this information will -be automatically extracted and shown right in the merge request widget. -[Learn more on Container Scanning in merge requests](https://docs.gitlab.com/ee/user/project/merge_requests/container_scanning.html). - -CAUTION: **Caution:** -Starting with GitLab 11.5, Container Scanning feature is licensed under the name `container_scanning`. -While the old name `sast_container` is still maintained, it has been deprecated with GitLab 11.5 and -may be removed in next major release, GitLab 12.0. You are advised to update your current `.gitlab-ci.yml` -configuration to reflect that change if you are using the `$GITLAB_FEATURES` environment variable. - -## Previous job definitions - -CAUTION: **Caution:** -Before GitLab 11.5, Container Scanning job and artifact had to be named specifically -to automatically extract report data and show it in the merge request widget. -While these old job definitions are still maintained they have been deprecated -and may be removed in next major release, GitLab 12.0. -You are advised to update your current `.gitlab-ci.yml` configuration to reflect that change. - -For GitLab 11.4 and earlier, the job should look like: - -```yaml -container_scanning: - image: docker:stable - variables: - DOCKER_DRIVER: overlay2 - ## Define two new variables based on GitLab's CI/CD predefined variables - ## https://docs.gitlab.com/ee/ci/variables/#predefined-environment-variables - CI_APPLICATION_REPOSITORY: $CI_REGISTRY_IMAGE/$CI_COMMIT_REF_SLUG - CI_APPLICATION_TAG: $CI_COMMIT_SHA - allow_failure: true - services: - - docker:stable-dind - script: - - docker run -d --name db arminc/clair-db:latest - - docker run -p 6060:6060 --link db:postgres -d --name clair --restart on-failure arminc/clair-local-scan:v2.0.1 - - apk add -U wget ca-certificates - - docker pull ${CI_APPLICATION_REPOSITORY}:${CI_APPLICATION_TAG} - - wget https://github.com/arminc/clair-scanner/releases/download/v8/clair-scanner_linux_amd64 - - mv clair-scanner_linux_amd64 clair-scanner - - chmod +x clair-scanner - - touch clair-whitelist.yml - - while( ! wget -q -O /dev/null http://docker:6060/v1/namespaces ) ; do sleep 1 ; done - - retries=0 - - echo "Waiting for clair daemon to start" - - while( ! wget -T 10 -q -O /dev/null http://docker:6060/v1/namespaces ) ; do sleep 1 ; echo -n "." ; if [ $retries -eq 10 ] ; then echo " Timeout, aborting." ; exit 1 ; fi ; retries=$(($retries+1)) ; done - - ./clair-scanner -c http://docker:6060 --ip $(hostname -i) -r gl-container-scanning-report.json -l clair.log -w clair-whitelist.yml ${CI_APPLICATION_REPOSITORY}:${CI_APPLICATION_TAG} || true - artifacts: - paths: [gl-container-scanning-report.json] -``` - -Alternatively the job name could be `sast:container` -and the artifact name could be `gl-sast-container-report.json`. -These names have been deprecated with GitLab 11.0 -and may be removed in next major release, GitLab 12.0. - -[ee]: https://about.gitlab.com/pricing/ +This document was moved to [another location](https://docs.gitlab.com/ee/user/application_security/container_scanning/index.html). diff --git a/doc/ci/examples/dast.md b/doc/ci/examples/dast.md index ab0ca13d2cf..b676c661267 100644 --- a/doc/ci/examples/dast.md +++ b/doc/ci/examples/dast.md @@ -1,102 +1,5 @@ -# Dynamic Application Security Testing with GitLab CI/CD +--- +redirect_to: 'https://docs.gitlab.com/ee/user/application_security/dast/index.html' +--- -CAUTION: **Caution:** -The job definition shown below is supported on GitLab 11.5 and later versions. -It also requires the GitLab Runner 11.5 or later. -For earlier versions, use the [previous job definitions](#previous-job-definitions). - -[Dynamic Application Security Testing (DAST)](https://en.wikipedia.org/wiki/Dynamic_program_analysis) -is using the popular open source tool [OWASP ZAProxy](https://github.com/zaproxy/zaproxy) -to perform an analysis on your running web application. -Since it is based on [ZAP Baseline](https://github.com/zaproxy/zaproxy/wiki/ZAP-Baseline-Scan) -DAST will perform passive scanning only; -it will not actively attack your application. - -It can be very useful combined with [Review Apps](../review_apps/index.md). - -## Example - -First, you need GitLab Runner with -[docker-in-docker executor](../docker/using_docker_build.md#use-docker-in-docker-executor). - -Once you set up the Runner, add a new job to `.gitlab-ci.yml` that -generates the expected report: - -```yaml -dast: - image: registry.gitlab.com/gitlab-org/security-products/zaproxy - variables: - website: "https://example.com" - allow_failure: true - script: - - mkdir /zap/wrk/ - - /zap/zap-baseline.py -J gl-dast-report.json -t $website || true - - cp /zap/wrk/gl-dast-report.json . - artifacts: - reports: - dast: gl-dast-report.json -``` - -The above example will create a `dast` job in your CI/CD pipeline which will run -the tests on the URL defined in the `website` variable (change it to use your -own) and scan it for possible vulnerabilities. The report will be saved as a -[DAST report artifact](../yaml/README.md#artifactsreportsdast-ultimate) -that you can later download and analyze. -Due to implementation limitations we always take the latest DAST artifact available. - -It's also possible to authenticate the user before performing DAST checks: - -```yaml -dast: - image: registry.gitlab.com/gitlab-org/security-products/zaproxy - variables: - website: "https://example.com" - login_url: "https://example.com/sign-in" - username: "john.doe@example.com" - password: "john-doe-password" - allow_failure: true - script: - - mkdir /zap/wrk/ - - /zap/zap-baseline.py -J gl-dast-report.json -t $website - --auth-url $login_url - --auth-username $username - --auth-password $password || true - - cp /zap/wrk/gl-dast-report.json . - artifacts: - reports: - dast: gl-dast-report.json -``` -See [zaproxy documentation](https://gitlab.com/gitlab-org/security-products/zaproxy) -to learn more about authentication settings. - -TIP: **Tip:** -For [GitLab Ultimate][ee] users, this information will -be automatically extracted and shown right in the merge request widget. -[Learn more on DAST in merge requests](https://docs.gitlab.com/ee/user/project/merge_requests/dast.html). - -## Previous job definitions - -CAUTION: **Caution:** -Before GitLab 11.5, DAST job and artifact had to be named specifically -to automatically extract report data and show it in the merge request widget. -While these old job definitions are still maintained they have been deprecated -and may be removed in next major release, GitLab 12.0. -You are advised to update your current `.gitlab-ci.yml` configuration to reflect that change. - -For GitLab 11.4 and earlier, the job should look like: - -```yaml -dast: - image: registry.gitlab.com/gitlab-org/security-products/zaproxy - variables: - website: "https://example.com" - allow_failure: true - script: - - mkdir /zap/wrk/ - - /zap/zap-baseline.py -J gl-dast-report.json -t $website || true - - cp /zap/wrk/gl-dast-report.json . - artifacts: - paths: [gl-dast-report.json] -``` - -[ee]: https://about.gitlab.com/pricing/ +This document was moved to [another location](https://docs.gitlab.com/ee/user/application_security/dast/index.html). diff --git a/doc/ci/examples/dependency_scanning.md b/doc/ci/examples/dependency_scanning.md new file mode 100644 index 00000000000..3a8b53b425c --- /dev/null +++ b/doc/ci/examples/dependency_scanning.md @@ -0,0 +1,5 @@ +--- +redirect_to: 'https://docs.gitlab.com/ee/user/application_security/dependency_scanning/index.html' +--- + +This document was moved to [another location](https://docs.gitlab.com/ee/user/application_security/dependency_scanning/index.html). diff --git a/doc/ci/examples/deploy_spring_boot_to_cloud_foundry/index.md b/doc/ci/examples/deploy_spring_boot_to_cloud_foundry/index.md index 474a481836a..c622dd86828 100644 --- a/doc/ci/examples/deploy_spring_boot_to_cloud_foundry/index.md +++ b/doc/ci/examples/deploy_spring_boot_to_cloud_foundry/index.md @@ -1,9 +1,10 @@ --- author: Dylan Griffith author_gitlab: DylanGriffith -level: intermediary +level: intermediate article_type: tutorial date: 2018-06-07 +last_updated: 2019-04-08 description: "Continuous Deployment of a Spring Boot application to Cloud Foundry with GitLab CI/CD" --- @@ -77,7 +78,10 @@ image: java:8 stages: - build - deploy - + +before_script: + - chmod +x mvnw + build: stage: build script: ./mvnw package @@ -99,7 +103,7 @@ We've used the `java:8` [docker image](../../docker/using_docker_images.md) to build our application as it provides the up-to-date Java 8 JDK on [Docker Hub](https://hub.docker.com/). We've also added the [`only` -clause](../../yaml/README.md#only-and-except-simplified) +clause](../../yaml/README.md#onlyexcept-basic) to ensure our deployments only happen when we push to the master branch. Now, since the steps defined in `.gitlab-ci.yml` require credentials to login diff --git a/doc/ci/examples/devops_and_game_dev_with_gitlab_ci_cd/index.md b/doc/ci/examples/devops_and_game_dev_with_gitlab_ci_cd/index.md index 1e2be2e8475..d6ad00a77da 100644 --- a/doc/ci/examples/devops_and_game_dev_with_gitlab_ci_cd/index.md +++ b/doc/ci/examples/devops_and_game_dev_with_gitlab_ci_cd/index.md @@ -1,9 +1,10 @@ --- author: Ryan Hall author_gitlab: blitzgren -level: intermediary +level: intermediate article_type: tutorial date: 2018-03-07 +last_updated: 2019-03-11 --- # DevOps and Game Dev with GitLab CI/CD @@ -20,7 +21,7 @@ and the basics of game development. Our [demo game](http://gitlab-game-demo.s3-website-us-east-1.amazonaws.com/) consists of a simple spaceship traveling in space that shoots by clicking the mouse in a given direction. -Creating a strong CI/CD pipeline at the beginning of developing another game, [Dark Nova](http://darknova.io/about), +Creating a strong CI/CD pipeline at the beginning of developing another game, [Dark Nova](http://darknova.io/), was essential for the fast pace the team worked at. This tutorial will build upon my [previous introductory article](https://ryanhallcs.wordpress.com/2017/03/15/devops-and-game-dev/) and go through the following steps: @@ -37,8 +38,8 @@ This will also provide boilerplate code for starting a browser-based game with the following components: - Written in [Typescript](https://www.typescriptlang.org/) and [PhaserJs](https://phaser.io) -- Building, running, and testing with [Gulp](http://gulpjs.com/) -- Unit tests with [Chai](http://chaijs.com/) and [Mocha](https://mochajs.org/) +- Building, running, and testing with [Gulp](https://gulpjs.com) +- Unit tests with [Chai](https://www.chaijs.com) and [Mocha](https://mochajs.org/) - CI/CD with GitLab - Hosting the codebase on GitLab.com - Hosting the game on AWS @@ -317,7 +318,7 @@ allowing us to run our tests by every push. Our entire `.gitlab-ci.yml` file should now look like this: ```yml -image: node:6 +image: node:10 build: stage: build @@ -386,7 +387,7 @@ Uploading artifacts to coordinator... ok id=17095874 responseStatus=2 We have our codebase built and tested on every push. To complete the full pipeline with Continuous Deployment, let's set up [free web hosting with AWS S3](https://aws.amazon.com/s/dm/optimization/server-side-test/free-tier/free_np/) and a job through which our build artifacts get -deployed. GitLab also has a free static site hosting service we could use, [GitLab Pages](https://about.gitlab.com/features/pages/), +deployed. GitLab also has a free static site hosting service we could use, [GitLab Pages](https://about.gitlab.com/product/pages/), however Dark Nova specifically uses other AWS tools that necessitates using `AWS S3`. Read through this article that describes [deploying to both S3 and GitLab Pages](https://about.gitlab.com/2016/08/26/ci-deployment-and-environments/) and further delves into the principles of GitLab CI/CD than discussed in this article. @@ -412,7 +413,7 @@ use root security credentials. Proper IAM credential management is beyond the sc article, but AWS will remind you that using root credentials is unadvised and against their best practices, as they should. Feel free to follow best practices and use a custom IAM user's credentials, which will be the same two credentials (Key ID and Secret). It's a good idea to -fully understand [IAM Best Practices in AWS](http://docs.aws.amazon.com/IAM/latest/UserGuide/best-practices.html). We need to add these credentials to GitLab: +fully understand [IAM Best Practices in AWS](https://docs.aws.amazon.com/IAM/latest/UserGuide/best-practices.html). We need to add these credentials to GitLab: 1. Log into your AWS account and go to the [Security Credentials page](https://console.aws.amazon.com/iam/home#/security_credential) 1. Click the **Access Keys** section and **Create New Access Key**. Create the key and keep the id and secret around, you'll need them later @@ -454,7 +455,7 @@ Be sure to update the region and S3 URL in that last script command to fit your Our final configuration file `.gitlab-ci.yml` looks like: ```yml -image: node:6 +image: node:10 build: stage: build @@ -502,7 +503,7 @@ deploy: ## Conclusion Within the [demo repository](https://gitlab.com/blitzgren/gitlab-game-demo) you can also find a handful of boilerplate code to get -[Typescript](https://www.typescriptlang.org/), [Mocha](https://mochajs.org/), [Gulp](http://gulpjs.com/) and [Phaser](https://phaser.io) all playing +[Typescript](https://www.typescriptlang.org/), [Mocha](https://mochajs.org/), [Gulp](https://gulpjs.com/) and [Phaser](https://phaser.io) all playing together nicely with GitLab CI/CD, which is the result of lessons learned while making [Dark Nova](http://darknova.io/). Using a combination of free and open source software, we have a full CI/CD pipeline, a game foundation, and unit tests, all running and deployed at every push to master - with shockingly little code. @@ -510,7 +511,7 @@ Errors can be easily debugged through GitLab's build logs, and within minutes of you can see the changes live on your game. Setting up Continuous Integration and Continuous Deployment from the start with Dark Nova enables -rapid but stable development. We can easily test changes in a separate [environment](../../../ci/environments.md#introduction-to-environments-and-deployments), +rapid but stable development. We can easily test changes in a separate [environment](../../environments.md), or multiple environments if needed. Balancing and updating a multiplayer game can be ongoing and tedious, but having faith in a stable deployment with GitLab CI/CD allows a lot of breathing room in quickly getting changes to players. @@ -520,7 +521,7 @@ a lot of breathing room in quickly getting changes to players. Here are some ideas to further investigate that can speed up or improve your pipeline: - [Yarn](https://yarnpkg.com) instead of npm -- Set up a custom [Docker](../../../ci/docker/using_docker_images.md#define-image-and-services-from-gitlab-ci-yml) image that can preload dependencies and tools (like AWS CLI) -- Forward a [custom domain](http://docs.aws.amazon.com/AmazonS3/latest/dev/website-hosting-custom-domain-walkthrough.html) to your game's S3 static website +- Set up a custom [Docker](../../../ci/docker/using_docker_images.md#define-image-and-services-from-gitlab-ciyml) image that can preload dependencies and tools (like AWS CLI) +- Forward a [custom domain](https:/docs.aws.amazon.com/AmazonS3/latest/dev/website-hosting-custom-domain-walkthrough.html) to your game's S3 static website - Combine jobs if you find it unnecessary for a small project - Avoid the queues and set up your own [custom GitLab CI/CD runner](https://about.gitlab.com/2016/03/01/gitlab-runner-with-docker/) diff --git a/doc/ci/examples/end_to_end_testing_webdriverio/index.md b/doc/ci/examples/end_to_end_testing_webdriverio/index.md index d7afe11f41f..bd221b7145e 100644 --- a/doc/ci/examples/end_to_end_testing_webdriverio/index.md +++ b/doc/ci/examples/end_to_end_testing_webdriverio/index.md @@ -139,7 +139,7 @@ new browser window interacting with your app as you specified. Which brings us to the exciting part: how do we run this in GitLab CI/CD? There are two things we need to do for this: -1. Set up [CI/CD jobs](../../yaml/README.md#jobs) that actually have a browser available. +1. Set up [CI/CD jobs](../../yaml/README.md#introduction) that actually have a browser available. 2. Update our WebdriverIO configuration to use those browsers to visit the review apps. For the scope of this article, we've defined an additional [CI/CD stage](../../yaml/README.md#stages) @@ -184,7 +184,7 @@ option as an argument to `npm run confidence-check` on the command line. However, we still need to tell WebdriverIO which browser is available for it to use. [GitLab CI/CD makes -a number of variables available](../../variables/README.html#predefined-variables-environment-variables) +a number of variables available](../../variables/README.html#predefined-environment-variables) with information about the current CI job. We can use this information to dynamically set up our WebdriverIO configuration according to the job that is running. More specifically, we can tell WebdriverIO what browser to execute the test on depending on the name of the currently running diff --git a/doc/ci/examples/laravel_with_gitlab_and_envoy/img/laravel_with_gitlab_and_envoy.png b/doc/ci/examples/laravel_with_gitlab_and_envoy/img/laravel_with_gitlab_and_envoy.png Binary files differdeleted file mode 100644 index bc188f83fb1..00000000000 --- a/doc/ci/examples/laravel_with_gitlab_and_envoy/img/laravel_with_gitlab_and_envoy.png +++ /dev/null diff --git a/doc/ci/examples/laravel_with_gitlab_and_envoy/img/variables_page.png b/doc/ci/examples/laravel_with_gitlab_and_envoy/img/variables_page.png Binary files differindex 4675e20ef79..edeaa011ada 100644 --- a/doc/ci/examples/laravel_with_gitlab_and_envoy/img/variables_page.png +++ b/doc/ci/examples/laravel_with_gitlab_and_envoy/img/variables_page.png diff --git a/doc/ci/examples/laravel_with_gitlab_and_envoy/index.md b/doc/ci/examples/laravel_with_gitlab_and_envoy/index.md index d3b6650b0f4..f56d5429fb7 100644 --- a/doc/ci/examples/laravel_with_gitlab_and_envoy/index.md +++ b/doc/ci/examples/laravel_with_gitlab_and_envoy/index.md @@ -2,9 +2,10 @@ redirect_from: 'https://docs.gitlab.com/ee/articles/laravel_with_gitlab_and_envoy/index.html' author: Mehran Rasulian author_gitlab: mehranrasulian -level: intermediary +level: intermediate article_type: tutorial date: 2017-08-31 +last_updated: 2019-03-06 --- # Test and deploy Laravel applications with GitLab CI/CD and Envoy @@ -115,7 +116,7 @@ cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys cat ~/.ssh/id_rsa ``` -Now, let's add it to your GitLab project as a [variable](../../variables/README.md#variables). +Now, let's add it to your GitLab project as a [variable](../../variables/README.md#gitlab-cicd-environment-variables). Variables are user-defined variables and are stored out of `.gitlab-ci.yml`, for security purposes. They can be added per project by navigating to the project's **Settings** > **CI/CD**. @@ -268,7 +269,7 @@ The `releases` directory will hold all our deployments: echo 'Cloning repository' [ -d {{ $releases_dir }} ] || mkdir {{ $releases_dir }} git clone --depth 1 {{ $repository }} {{ $new_release_dir }} - cd {{ $releases_dir }} + cd {{ $new_release_dir }} git reset --hard {{ $commit }} @endtask @@ -346,7 +347,7 @@ At the end, our `Envoy.blade.php` file will look like this: echo 'Cloning repository' [ -d {{ $releases_dir }} ] || mkdir {{ $releases_dir }} git clone --depth 1 {{ $repository }} {{ $new_release_dir }} - cd {{ $releases_dir }} + cd {{ $new_release_dir }} git reset --hard {{ $commit }} @endtask @@ -374,7 +375,7 @@ You might want to create another Envoy task to do that for you. We also create the `.env` file in the same path to set up our production environment variables for Laravel. These are persistent data and will be shared to every new release. -Now, we would need to deploy our app by running `envoy run deploy`, but it won't be necessary since GitLab can handle that for us with CI's [environments](../../environments.md), which will be described [later](#setting-up-gitlab-ci-cd) in this tutorial. +Now, we would need to deploy our app by running `envoy run deploy`, but it won't be necessary since GitLab can handle that for us with CI's [environments](../../environments.md), which will be described [later](#setting-up-gitlab-cicd) in this tutorial. Now it's time to commit [Envoy.blade.php](https://gitlab.com/mehranrasulian/laravel-sample/blob/master/Envoy.blade.php) and push it to the `master` branch. To keep things simple, we commit directly to `master`, without using [feature-branches](../../../workflow/gitlab_flow.md#github-flow-as-a-simpler-alternative) since collaboration is beyond the scope of this tutorial. @@ -557,7 +558,7 @@ So we should adjust the configuration of MySQL instance by defining `MYSQL_DATAB Find out more about MySQL variables at the [official MySQL Docker Image](https://hub.docker.com/r/_/mysql/). Also set the variables `DB_HOST` to `mysql` and `DB_USERNAME` to `root`, which are Laravel specific variables. -We define `DB_HOST` as `mysql` instead of `127.0.0.1`, as we use MySQL Docker image as a service which [is linked to the main Docker image](../../docker/using_docker_images.md#how-services-are-linked-to-the-build). +We define `DB_HOST` as `mysql` instead of `127.0.0.1`, as we use MySQL Docker image as a service which [is linked to the main Docker image](../../docker/using_docker_images.md#how-services-are-linked-to-the-job). ```yaml ... diff --git a/doc/ci/examples/license_management.md b/doc/ci/examples/license_management.md new file mode 100644 index 00000000000..08704425a75 --- /dev/null +++ b/doc/ci/examples/license_management.md @@ -0,0 +1,5 @@ +--- +redirect_to: 'https://docs.gitlab.com/ee/user/application_security/license_management/index.html' +--- + +This document was moved to [another location](https://docs.gitlab.com/ee/user/application_security/license_management/index.html). diff --git a/doc/ci/examples/sast.md b/doc/ci/examples/sast.md new file mode 100644 index 00000000000..688cc79d0f6 --- /dev/null +++ b/doc/ci/examples/sast.md @@ -0,0 +1,5 @@ +--- +redirect_to: 'https://docs.gitlab.com/ee/user/application_security/sast/index.html' +--- + +This document was moved to [another location](https://docs.gitlab.com/ee/user/application_security/sast/index.html). diff --git a/doc/ci/examples/sast_docker.md b/doc/ci/examples/sast_docker.md index 3a657b3a3d5..570898b2d23 100644 --- a/doc/ci/examples/sast_docker.md +++ b/doc/ci/examples/sast_docker.md @@ -1 +1,5 @@ -This document was moved to [another location](./container_scanning.md). +--- +redirect_to: 'https://docs.gitlab.com/ee/user/application_security/container_scanning/index.html' +--- + +This document was moved to [another location](../../user/application_security/container_scanning/index.html). diff --git a/doc/ci/examples/test-and-deploy-python-application-to-heroku.md b/doc/ci/examples/test-and-deploy-python-application-to-heroku.md index 61bf68fa0e8..47d20a4e1c1 100644 --- a/doc/ci/examples/test-and-deploy-python-application-to-heroku.md +++ b/doc/ci/examples/test-and-deploy-python-application-to-heroku.md @@ -1,4 +1,4 @@ -# Test and Deploy a python application with GitLab CI/CD +# Test and deploy a Python application with GitLab CI/CD This example will guide you how to run tests in your Python application and deploy it automatically as Heroku application. @@ -9,7 +9,12 @@ You can checkout the [example source](https://gitlab.com/ayufan/python-getting-s This is what the `.gitlab-ci.yml` file looks like for this project: ```yaml +stages: + - test + - deploy + test: + stage: test script: # this configures Django application to use attached postgres database that is run on `postgres` host - export DATABASE_URL=postgres://postgres:@postgres:5432/python-test-app @@ -19,7 +24,7 @@ test: - python manage.py test staging: - type: deploy + stage: deploy script: - apt-get update -qy - apt-get install -y ruby-dev @@ -29,7 +34,7 @@ staging: - master production: - type: deploy + stage: deploy script: - apt-get update -qy - apt-get install -y ruby-dev @@ -65,7 +70,7 @@ First install [Docker Engine](https://docs.docker.com/installation/). To build this project you also need to have [GitLab Runner](https://docs.gitlab.com/runner). You can use public runners available on `gitlab.com`, but you can register your own: -``` +```sh gitlab-runner register \ --non-interactive \ --url "https://gitlab.com/" \ diff --git a/doc/ci/examples/test-and-deploy-ruby-application-to-heroku.md b/doc/ci/examples/test-and-deploy-ruby-application-to-heroku.md index 46e6efccaf8..3a0ddf001b8 100644 --- a/doc/ci/examples/test-and-deploy-ruby-application-to-heroku.md +++ b/doc/ci/examples/test-and-deploy-ruby-application-to-heroku.md @@ -1,4 +1,4 @@ -# Test and Deploy a ruby application with GitLab CI/CD +# Test and deploy a Ruby application with GitLab CI/CD This example will guide you how to run tests in your Ruby on Rails application and deploy it automatically as Heroku application. @@ -58,10 +58,10 @@ You can do this through the [Dashboard](https://dashboard.heroku.com/). ## Create Runner First install [Docker Engine](https://docs.docker.com/installation/). -To build this project you also need to have [GitLab Runner](https://about.gitlab.com/gitlab-ci/#gitlab-runner). +To build this project you also need to have [GitLab Runner](https://docs.gitlab.com/runner/). You can use public runners available on `gitlab.com`, but you can register your own: -``` +```sh gitlab-runner register \ --non-interactive \ --url "https://gitlab.com/" \ diff --git a/doc/ci/examples/test-scala-application.md b/doc/ci/examples/test-scala-application.md index 66bfa41cad9..e1164b8d03a 100644 --- a/doc/ci/examples/test-scala-application.md +++ b/doc/ci/examples/test-scala-application.md @@ -1,4 +1,4 @@ -# Test and deploy to Heroku a Scala application +# Test and deploy a Scala application to Heroku This example demonstrates the integration of GitLab CI with Scala applications using SBT. Checkout the example @@ -55,8 +55,8 @@ You can use other versions of Scala and SBT by defining them in Add the `Coverage was \[\d+.\d+\%\]` regular expression in the **Settings ➔ Pipelines ➔ Coverage report** project setting to -retrieve the [test coverage] rate from the build trace and have it -displayed with your jobs. +retrieve the [test coverage](../../user/project/pipelines/settings.md#test-coverage-report-badge) +rate from the build trace and have it displayed with your jobs. **Pipelines** must be enabled for this option to appear. @@ -69,8 +69,5 @@ in the `.gitlab-ci.yml` file with your application's name. ## Heroku API key You can look up your Heroku API key in your -[account](https://dashboard.heroku.com/account). Add a secure [variable] with +[account](https://dashboard.heroku.com/account). Add a [protected variable](../variables/README.md#protected-environment-variables) with this value in **Project ➔ Variables** with key `HEROKU_API_KEY`. - -[variable]: ../variables/README.md#user-defined-variables-secure-variables -[test coverage]: ../../user/project/pipelines/settings.md#test-coverage-report-badge diff --git a/doc/ci/examples/test_phoenix_app_with_gitlab_ci_cd/img/job-succeeded.png b/doc/ci/examples/test_phoenix_app_with_gitlab_ci_cd/img/job-succeeded.png Binary files differdeleted file mode 100644 index 77b05f55f88..00000000000 --- a/doc/ci/examples/test_phoenix_app_with_gitlab_ci_cd/img/job-succeeded.png +++ /dev/null diff --git a/doc/ci/examples/test_phoenix_app_with_gitlab_ci_cd/index.md b/doc/ci/examples/test_phoenix_app_with_gitlab_ci_cd/index.md index f24e79355aa..4a5fda661df 100644 --- a/doc/ci/examples/test_phoenix_app_with_gitlab_ci_cd/index.md +++ b/doc/ci/examples/test_phoenix_app_with_gitlab_ci_cd/index.md @@ -4,13 +4,14 @@ author_gitlab: Hostert level: beginner article_type: tutorial date: 2018-02-20 +last_updated: 2019-03-06 --- # Testing a Phoenix application with GitLab CI/CD [Phoenix][phoenix-site] is a web development framework written in [Elixir][elixir-site], which is a functional language designed for productivity and maintainability that runs on the -[Erlang VM][erlang-site]. Erlang VM is really really fast and can handle very large numbers of +[Erlang VM](https://www.erlang.org). Erlang VM is really really fast and can handle very large numbers of simultaneous users. That's why we're hearing so much about Phoenix today. @@ -26,7 +27,7 @@ and GitLab UI._ ### What is Phoenix? [Phoenix][phoenix-site] is a web development framework written in [Elixir][elixir-site] very useful - to build fast, reliable, and high-performance applications, as it uses [Erlang VM][erlang-site]. + to build fast, reliable, and high-performance applications, as it uses [Erlang VM](https://www.erlang.org). Many components and concepts are similar to Ruby on Rails or Python's Django. High developer productivity and high application performance are only a few advantages on learning how to use it. @@ -177,7 +178,7 @@ environment it can run. Since we will work with a single environment, we'll edit configuration file (`test.exs`). But, why do we need to adjust our configuration? Well, GitLab CI/CD builds and tests our code in one -isolated virtual machine, called [Runner][runner-site], using Docker technology. In this Runner, +isolated virtual machine, called [Runner](../../runners/README.md), using Docker technology. In this Runner, GitLab CI/CD has access to everything our Phoenix application need to run, exactly as we have in our `localhost`, but we have to tell GitLab CI/CD where to create and find this database using system variables. This way, GitLab CI/CD will create our test database inside the Runner, just like we do @@ -405,7 +406,6 @@ other reasons][ci-reasons] to keep using GitLab CI/CD. The benefits to our teams [elixir-site]: http://elixir-lang.org/ "Elixir" [elixir-mix]: http://elixir-lang.org/getting-started/mix-otp/introduction-to-mix.html "Introduction to mix" [elixir-docs]: http://elixir-lang.org/getting-started/introduction.html "Elixir Documentation" -[erlang-site]: http://erlang.org "Erlang" [elixir-install]: https://elixir-lang.org/install.html "Elixir Installation" [ecto]: http://hexdocs.pm/ecto "Ecto" [ecto-repo]: https://hexdocs.pm/ecto/Ecto.html#module-repositories "Ecto Repositories" @@ -417,7 +417,6 @@ other reasons][ci-reasons] to keep using GitLab CI/CD. The benefits to our teams [ci-docs]: ../../README.md "GitLab CI/CD Documentation" [skipping-jobs]: ../../yaml/README.md#skipping-jobs "Skipping Jobs" [gitlab-runners]: ../../runners/README.md "GitLab Runners Documentation" -[runner-site]: ../../runners/README.md#runners "Runners" [docker-image]: https://hub.docker.com/r/trenpixster/elixir/ "Elixir Docker Image" [using-docker]: ../../docker/using_docker_images.md "Using Docker Images" [hello-gitlab]: https://gitlab.com/Hostert/hello_gitlab_ci "Hello GitLab CI/CD" diff --git a/doc/ci/git_submodules.md b/doc/ci/git_submodules.md index 37078230b34..551044dd76f 100644 --- a/doc/ci/git_submodules.md +++ b/doc/ci/git_submodules.md @@ -1,3 +1,7 @@ +--- +type: reference +--- + # Using Git submodules with GitLab CI > **Notes:** diff --git a/doc/ci/img/add_file_template_11_10.png b/doc/ci/img/add_file_template_11_10.png Binary files differnew file mode 100644 index 00000000000..ca04d72615b --- /dev/null +++ b/doc/ci/img/add_file_template_11_10.png diff --git a/doc/ci/img/cicd_pipeline_infograph.png b/doc/ci/img/cicd_pipeline_infograph.png Binary files differdeleted file mode 100644 index 9ddd4aa828b..00000000000 --- a/doc/ci/img/cicd_pipeline_infograph.png +++ /dev/null diff --git a/doc/ci/img/deployments_view.png b/doc/ci/img/deployments_view.png Binary files differindex 45d882b536c..12090434bef 100644 --- a/doc/ci/img/deployments_view.png +++ b/doc/ci/img/deployments_view.png diff --git a/doc/ci/img/environments_available.png b/doc/ci/img/environments_available.png Binary files differindex 7ab92838ece..48fc6effc2d 100644 --- a/doc/ci/img/environments_available.png +++ b/doc/ci/img/environments_available.png diff --git a/doc/ci/img/environments_mr_review_app.png b/doc/ci/img/environments_mr_review_app.png Binary files differindex 61b7e9fe77c..6a7b7ce5679 100644 --- a/doc/ci/img/environments_mr_review_app.png +++ b/doc/ci/img/environments_mr_review_app.png diff --git a/doc/ci/img/metrics_reports.png b/doc/ci/img/metrics_reports.png Binary files differnew file mode 100644 index 00000000000..ffd9f6830a2 --- /dev/null +++ b/doc/ci/img/metrics_reports.png diff --git a/doc/ci/img/multi_pipeline_mini_graph.gif b/doc/ci/img/multi_pipeline_mini_graph.gif Binary files differnew file mode 100644 index 00000000000..de49ba5aa12 --- /dev/null +++ b/doc/ci/img/multi_pipeline_mini_graph.gif diff --git a/doc/ci/img/multi_project_pipeline_graph.png b/doc/ci/img/multi_project_pipeline_graph.png Binary files differnew file mode 100644 index 00000000000..723a455cb4a --- /dev/null +++ b/doc/ci/img/multi_project_pipeline_graph.png diff --git a/doc/ci/img/pipelines-goal.png b/doc/ci/img/pipelines-goal.png Binary files differdeleted file mode 100644 index f15716d0b8f..00000000000 --- a/doc/ci/img/pipelines-goal.png +++ /dev/null diff --git a/doc/ci/img/types-of-pipelines.png b/doc/ci/img/types-of-pipelines.png Binary files differdeleted file mode 100644 index 829a53d5d52..00000000000 --- a/doc/ci/img/types-of-pipelines.png +++ /dev/null diff --git a/doc/ci/interactive_web_terminal/index.md b/doc/ci/interactive_web_terminal/index.md index 2a4160f62b0..1387d4df500 100644 --- a/doc/ci/interactive_web_terminal/index.md +++ b/doc/ci/interactive_web_terminal/index.md @@ -1,3 +1,7 @@ +--- +type: reference +--- + # Interactive Web Terminals > [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/issues/50144) in GitLab 11.3. @@ -54,3 +58,8 @@ terminal will block the job from finishing for the duration configured in close the terminal window.  + +## Interactive Web Terminals for the Web IDE **[ULTIMATE ONLY]** + +Read the Web IDE docs to learn how to run [Interactive Terminals through the Web IDE](../../user/project/web_ide/index.md). + diff --git a/doc/ci/introduction/img/gitlab_workflow_example_11_9.png b/doc/ci/introduction/img/gitlab_workflow_example_11_9.png Binary files differnew file mode 100644 index 00000000000..204e9c462e5 --- /dev/null +++ b/doc/ci/introduction/img/gitlab_workflow_example_11_9.png diff --git a/doc/ci/introduction/img/gitlab_workflow_example_extended_11_11.png b/doc/ci/introduction/img/gitlab_workflow_example_extended_11_11.png Binary files differnew file mode 100644 index 00000000000..5089a1088c5 --- /dev/null +++ b/doc/ci/introduction/img/gitlab_workflow_example_extended_11_11.png diff --git a/doc/ci/introduction/img/job_running.png b/doc/ci/introduction/img/job_running.png Binary files differnew file mode 100644 index 00000000000..d5f922ceb8c --- /dev/null +++ b/doc/ci/introduction/img/job_running.png diff --git a/doc/ci/introduction/img/pipeline_status.png b/doc/ci/introduction/img/pipeline_status.png Binary files differnew file mode 100644 index 00000000000..96881f072e1 --- /dev/null +++ b/doc/ci/introduction/img/pipeline_status.png diff --git a/doc/ci/introduction/img/rollback.png b/doc/ci/introduction/img/rollback.png Binary files differnew file mode 100644 index 00000000000..38e0552f4f1 --- /dev/null +++ b/doc/ci/introduction/img/rollback.png diff --git a/doc/ci/introduction/index.md b/doc/ci/introduction/index.md new file mode 100644 index 00000000000..bd2b9b099f2 --- /dev/null +++ b/doc/ci/introduction/index.md @@ -0,0 +1,223 @@ +--- +description: "An overview of Continuous Integration, Continuous Delivery, and Continuous Deployment, as well as an introduction to GitLab CI/CD." +type: concepts +--- + +# Introduction to CI/CD with GitLab + +In this document we'll present an overview of the concepts of Continuous Integration, +Continuous Delivery, and Continuous Deployment, as well as an introduction to +GitLab CI/CD. + +## Introduction to CI/CD methodologies + +The continuous methodologies of software development are based on +automating the execution of scripts to minimize the chance of +introducing errors while developing applications. They require +less human intervention or even no intervention at all, from the +development of new code until its deployment. + +It involves continuously building, testing, and deploying code +changes at every small iteration, reducing the chance of developing +new code based on bugged or failed previous versions. + +There are three main approaches to this methodology, each of them +to be applied according to what best suits your strategy. + +### Continuous Integration + +Consider an application which has its code stored in a Git +repository in GitLab. Developers push code changes every day, +multiple times a day. For every push to the repository, you +can create a set of scripts to build and test your application +automatically, decreasing the chance of introducing errors to your app. + +This practice is known as [Continuous Integration](https://en.wikipedia.org/wiki/Continuous_integration); +for every change submitted to an application - even to development branches - +it's built and tested automatically and continuously, ensuring the +introduced changes pass all tests, guidelines, and code compliance +standards you established for your app. + +[GitLab itself](https://gitlab.com/gitlab-org/gitlab-ce) is an +example of using Continuous Integration as a software +development method. For every push to the project, there's a set +of scripts the code is checked against. + +### Continuous Delivery + +[Continuous Delivery](https://continuousdelivery.com/) is a step +beyond Continuous Integration. Your application is not only +built and tested at every code change pushed to the codebase, +but, as an additional step, it's also deployed continuously, though +the deployments are triggered manually. + +This method ensures the code is checked automatically but requires +human intervention to manually and strategically trigger the deployment +of the changes. + +### Continuous Deployment + +[Continuous Deployment](https://www.airpair.com/continuous-deployment/posts/continuous-deployment-for-practical-people) +is also a further step beyond Continuous Integration, similar to +Continuous Delivery. The difference is that instead of deploying your +application manually, you set it to be deployed automatically. It does +not require human intervention at all to have your application +deployed. + +## Introduction to GitLab CI/CD + +GitLab CI/CD is a powerful tool built into GitLab that allows you +to apply all the continuous methods (Continuous Integration, +Delivery, and Deployment) to your software with no third-party +application or integration needed. + +### How GitLab CI/CD works + +To use GitLab CI/CD, all you need is an application codebase hosted in a +Git repository, and for your build, test, and deployment +scripts to be specified in a file called [`.gitlab-ci.yml`](../yaml/README.md), +located in the root path of your repository. + +In this file, you can define the scripts you want to run, define include and +cache dependencies, choose commands you want to run in sequence +and those you want to run in parallel, define where you want to +deploy your app, and specify whether you will want to run the scripts automatically +or trigger any of them manually. Once you're familiar with +GitLab CI/CD you can add more advanced steps into the configuration file. + +To add scripts to that file, you'll need to organize them in a +sequence that suits your application and are in accordance with +the tests you wish to perform. To visualize the process, imagine +that all the scripts you add to the configuration file are the +same as the commands you run on a terminal in your computer. + +Once you've added your `.gitlab-ci.yml` configuration file to your +repository, GitLab will detect it and run your scripts with the +tool called [GitLab Runner](https://docs.gitlab.com/runner/), which +works similarly to your terminal. + +The scripts are grouped into **jobs**, and together they compose +a **pipeline**. A minimalist example of `.gitlab-ci.yml` file +could contain: + +```yml +before_script: + - apt-get install rubygems ruby-dev -y + +run-test: + script: + - ruby --version +``` + +The `before_script` attribute would install the dependencies +for your app before running anything, and a **job** called +`run-test` would print the Ruby version of the current system. +Both of them compose a **pipeline** triggered at every push +to any branch of the repository. + +GitLab CI/CD not only executes the jobs you've +set, but also shows you what's happening during execution, as you +would see in your terminal: + + + +You create the strategy for your app and GitLab runs the pipeline +for you according to what you've defined. Your pipeline status is also +displayed by GitLab: + + + +At the end, if anything goes wrong, you can easily +[roll back](../environments.md#retrying-and-rolling-back) all the changes: + + + +### Basic CI/CD workflow + +Consider the following example for how GitLab CI/CD fits in a +common development workflow. + +Assume that you have discussed a code implementation in an issue +and worked locally on your proposed changes. Once you push your +commits to a feature branch in a remote repository in GitLab, +the CI/CD pipeline set for your project is triggered. By doing +so, GitLab CI/CD: + +- Runs automated scripts (sequential or parallel) to: + - Build and test your app. + - Preview the changes per merge request with Review Apps, as you + would see in your `localhost`. + +Once you're happy with your implementation: + +- Get your code reviewed and approved. +- Merge the feature branch into the default branch. + - GitLab CI/CD deploys your changes automatically to a production environment. +- And finally, you and your team can easily roll it back if something goes wrong. + + + +GitLab CI/CD is capable of doing a lot more, but this workflow +exemplifies GitLab's ability to track the entire process, +without the need of any external tool to deliver your software. +And, most usefully, you can visualize all the steps through +the GitLab UI. + +#### A deeper look into the CI/CD basic workflow + +If we take a deeper look into the basic workflow, we can see +the features available in GitLab at each stage of the DevOps +lifecycle, as shown on the illustration below. + + + +If you look at the image from the left to the right, +you'll see some of the features available in GitLab +according to each stage (Verify, Package, Release). + +1. **Verify**: + - Automatically build and test your application with Continuous Integration. + - Analyze your source code quality with [GitLab Code Quality](https://docs.gitlab.com/ee/user/project/merge_requests/code_quality.html). **[STARTER]** + - Determine the performance impact of code changes with [Browser Performance Testing](https://docs.gitlab.com/ee/user/project/merge_requests/browser_performance_testing.html). **[PREMIUM]** + - Perform a series of tests, such as [Container Scanning](https://docs.gitlab.com/ee/ci/examples/container_scanning.html) **[ULTIMATE]**, [Dependency Scanning](https://docs.gitlab.com/ee/ci/examples/dependency_scanning.html) **[ULTIMATE]**, and [JUnit tests](../junit_test_reports.md). + - Deploy your changes with [Review Apps](../review_apps/index.md) to preview the app changes on every branch. +1. **Package**: + - Store Docker images with [Container Registry](../../user/project/container_registry.md). + - Store NPM packages with [NPM Registry](https://docs.gitlab.com/ee/user/project/packages/npm_registry.html). **[PREMIUM]** + - Store Maven artifacts with [Maven Repository](https://docs.gitlab.com/ee/user/project/packages/maven_repository.html). **[PREMIUM]** +1. **Release**: + - Continuous Deployment, automatically deploying your app to production. + - Continuous Delivery, manually click to deploy your app to production. + - Deploy static websites with [GitLab Pages](../../user/project/pages/index.md). + - Ship features to only a portion of your pods and let a percentage of your user base to visit the temporarily deployed feature with [Canary Deployments](https://docs.gitlab.com/ee/user/project/canary_deployments.html). **[PREMIUM]** + - Deploy your features behind [Feature Flags](https://docs.gitlab.com/ee/user/project/operations/feature_flags.html). **[PREMIUM]** + - Add release notes to any Git tag with [GitLab Releases](../../user/project/releases/index.md). + - View of the current health and status of each CI environment running on Kubernetes with [Deploy Boards](https://docs.gitlab.com/ee/user/project/deploy_boards.html). **[PREMIUM]** + - Deploy your application to a production environment in a Kubernetes cluster with [Auto Deploy](../../topics/autodevops/index.md#auto-deploy). + +With GitLab CI/CD you can also: + +- Easily set up your app's entire lifecycle with [Auto DevOps](../../topics/autodevops/index.md). +- Deploy your app to different [environments](../environments.md). +- Install your own [GitLab Runner](https://docs.gitlab.com/runner/). +- [Schedule pipelines](../../user/project/pipelines/schedules.md). +- Check for app vulnerabilities with [Security Test reports](https://docs.gitlab.com/ee/user/project/merge_requests/#security-reports-ultimate). **[ULTIMATE]** + +To see all CI/CD features, navigate back to the [CI/CD index](../README.md). + +### Setting up GitLab CI/CD for the first time + +To get started with GitLab CI/CD, you need to familiarize yourself +with the [`.gitlab-ci.yml`](../yaml/README.md) configuration file +syntax and with its attributes. + +This document [introduces the concepts of GitLab CI/CD in the scope of GitLab Pages](../../user/project/pages/getting_started_part_four.md), for deploying static websites. +Although it's meant for users who want to write their own Pages +script from scratch, it also serves as an introduction to the setup process for GitLab CI/CD. +It covers the very first general steps of writing a CI/CD configuration +file, so we recommend you read through it to understand GitLab's CI/CD +logic, and learn how to write your own script (or tweak an +existing one) for any application. + +For a deep view of GitLab's CI/CD configuration options, check the +[`.gitlab-ci.yml` full reference](../yaml/README.md). diff --git a/doc/ci/junit_test_reports.md b/doc/ci/junit_test_reports.md index cf18c6d9660..fa78f53f563 100644 --- a/doc/ci/junit_test_reports.md +++ b/doc/ci/junit_test_reports.md @@ -1,3 +1,7 @@ +--- +type: reference +--- + # JUnit test reports > [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/issues/45318) in GitLab 11.2. @@ -71,11 +75,11 @@ merge request widget. NOTE: **Note:** If you also want the ability to browse JUnit output files, include the -[`artifacts:paths`](yaml/README.md#artifactspaths) keyword. +[`artifacts:paths`](yaml/README.md#artifactspaths) keyword. An example of this is shown in the Ruby example below. ### Ruby example -Use the following job in `.gitlab-ci.yml`: +Use the following job in `.gitlab-ci.yml`. This includes the `artifacts:paths` keyword to provide a link to the JUnit output file. ```yaml ## Use https://github.com/sj26/rspec_junit_formatter to generate a JUnit report with rspec @@ -85,6 +89,8 @@ ruby: - bundle install - rspec spec/lib/ --format RspecJunitFormatter --out rspec.xml artifacts: + paths: + - rspec.xml reports: junit: rspec.xml ``` @@ -113,8 +119,8 @@ There are a few tools that can produce JUnit reports in Java. In the following example, `gradle` is used to generate the test reports. If there are multiple test tasks defined, `gradle` will generate multiple -directories under `build/test-results/`. In that case, you can leverage regex -matching by defining the following path: `build/test-results/test/TEST-*.xml`: +directories under `build/test-results/`. In that case, you can leverage glob +matching by defining the following path: `build/test-results/test/**/TEST-*.xml`: ```yaml java: @@ -123,7 +129,7 @@ java: - gradle test artifacts: reports: - junit: build/test-results/test/TEST-*.xml + junit: build/test-results/test/**/TEST-*.xml ``` #### Maven diff --git a/doc/ci/large_repositories/index.md b/doc/ci/large_repositories/index.md new file mode 100644 index 00000000000..29d649ad717 --- /dev/null +++ b/doc/ci/large_repositories/index.md @@ -0,0 +1,242 @@ +--- +type: reference +--- + +# Optimizing GitLab for large repositories + +Large repositories consisting of more than 50k files in a worktree +often require special consideration because of +the time required to clone and check out. + +GitLab and GitLab Runner handle this scenario well +but require optimized configuration to efficiently perform its +set of operations. + +The general guidelines for handling big repositories are simple. +Each guideline is described in more detail in the sections below: + +- Always fetch incrementally. Do not clone in a way that results in recreating all of the worktree. +- Always use shallow clone to reduce data transfer. Be aware that this puts more burden + on GitLab instance due to higher CPU impact. +- Control the clone directory if you heavily use a fork-based workflow. +- Optimize `git clean` flags to ensure that you remove or keep data that might affect or speed-up your build. + +## Shallow cloning + +> Introduced in GitLab Runner 8.9. + +GitLab and GitLab Runner always perform a full clone by default. +While it means that all changes from GitLab are received, +it often results in receiving extra commit logs. + +Ideally, you should always use `GIT_DEPTH` with a small number +like 10. This will instruct GitLab Runner to perform shallow clones. +Shallow clones makes Git request only the latest set of changes for a given branch, +up to desired number of commits as defined by the `GIT_DEPTH` variable. + +This significantly speeds up fetching of changes from Git repositories, +especially if the repository has a very long backlog consisting of number +of big files as we effectively reduce amount of data transfer. + +The following example makes GitLab Runner shallow clone to fetch only a given branch, +it does not fetch any other branches nor tags. + +```yaml +variables: + GIT_DEPTH: 10 + +test: + script: + - ls -al +``` + +## Git strategy + +> Introduced in GitLab Runner 8.9. + +By default, GitLab is configured to always prefer the `GIT_STRATEGY: fetch` strategy. +The `GIT_STRATEGY: fetch` strategy will re-use existing worktrees if found +on disk. This is different to the `GIT_STRATEGY: clone` strategy +as in case of clones, if a worktree is found, it is removed before clone. + +Usage of `fetch` is preferred because it reduces the amount of data to transfer and +does not really impact the operations that you might do on a repository from CI. + +However, `fetch` does require access to the previous worktree. This works +well when using the `shell` or `docker` executor because these +try to preserve worktrees and try to re-use them by default. + +This does not work today for `kubernetes` executor and has limitations when using +`docker+machine`. `kubernetes` executor today always clones into ephemeral directory. + +GitLab also offers the `GIT_STRATEGY: none` strategy. This disables any `fetch` and `checkout` commands +done by GitLab, requiring you to do them. + +## Git clone path + +> Introduced in GitLab Runner 11.10. + +[`GIT_CLONE_PATH`](../yaml/README.md#custom-build-directories) allows you to +control where you clone your sources. This can have implications if you +heavily use big repositories with fork workflow. + +Fork workflow from GitLab Runner's perspective is stored as a separate repository +with separate worktree. That means that GitLab Runner cannot optimize the usage +of worktrees and you might have to instruct GitLab Runner to use that. + +In such cases, ideally you want to make the GitLab Runner executor be used only used only +for the given project and not shared across different projects to make this +process more efficient. + +The [`GIT_CLONE_PATH`](../yaml/README.md#custom-build-directories) has to be +within the `$CI_BUILDS_DIR`. Currently, it is impossible to pick any path +from disk. + +## Git clean flags + +> Introduced in GitLab Runner 11.10. + +[`GIT_CLEAN_FLAGS`](../yaml/README.md#git-clean-flags) allows you to control +whether or not you require the `git clean` command to be executed for each CI +job. By default, GitLab ensures that you have your worktree on the given SHA, +and that your repository is clean. + +[`GIT_CLEAN_FLAGS`](../yaml/README.md#git-clean-flags) is disabled when set +to `none`. On very big repositories, this might be desired because `git +clean` is disk I/O intensive. Controlling that with `GIT_CLEAN_FLAGS: -ffdx +-e .build/`, for example, allows you to control and disable removal of some +directories within the worktree between subsequent runs, which can speed-up +the incremental builds. This has the biggest effect if you re-use existing +machines, and have an existing worktree that you can re-use for builds. + +For exact parameters accepted by +[`GIT_CLEAN_FLAGS`](../yaml/README.md#git-clean-flags), see the documentation +for [git clean](https://git-scm.com/docs/git-clean). The available parameters +are dependent on Git version. + +## Fork-based workflow + +> Introduced in GitLab Runner 11.10. + +Following the guidelines above, lets imagine that we want to: + +- Optimize for a big project (more than 50k files in directory). +- Use forks-based workflow for contributing. +- Reuse existing worktrees. Have preconfigured runners that are pre-cloned with repositories. +- Runner assigned only to project and all forks. + +Lets consider the following two examples, one using `shell` executor and +other using `docker` executor. + +### `shell` executor example + +Lets assume that you have the following [config.toml](https://docs.gitlab.com/runner/configuration/advanced-configuration.html). + +```toml +concurrent = 4 + +[[runners]] + url = "GITLAB_URL" + token = "TOKEN" + executor = "shell" + builds_dir = "/builds" + cache_dir = "/cache" + + [runners.custom_build_dir] + enabled = true +``` + +This `config.toml`: + +- Uses the `shell` executor, +- Specifies a custom `/builds` directory where all clones will be stored. +- Enables the ability to specify `GIT_CLONE_PATH`, +- Runs at most 4 jobs at once. + +### `docker` executor example + +Lets assume that you have the following [config.toml](https://docs.gitlab.com/runner/configuration/advanced-configuration.html). + +```toml +concurrent = 4 + +[[runners]] + url = "GITLAB_URL" + token = "TOKEN" + executor = "docker" + builds_dir = "/builds" + cache_dir = "/cache" + + [runners.docker] + volumes = ["/builds:/builds", "/cache:/cache"] +``` + +This `config.toml`: + +- Uses the `docker` executor, +- Specifies a custom `/builds` directory on disk where all clones will be stored. + We host mount the `/builds` directory to make it reusable between subsequent runs + and be allowed to override the cloning strategy. +- Doesn't enable the ability to specify `GIT_CLONE_PATH` as it is enabled by default. +- Runs at most 4 jobs at once. + +### Our `.gitlab-ci.yml` + +Once we have the executor configured, we need to fine tune our `.gitlab-ci.yml`. + +Our pipeline will be most performant if we use the following `.gitlab-ci.yml`: + +```yaml +variables: + GIT_DEPTH: 10 + GIT_CLONE_PATH: $CI_BUILDS_DIR/$CI_CONCURRENT_ID/$CI_PROJECT_NAME + +build: + script: ls -al +``` + +The above configures a: + +- Shallow clone of 10, to speed up subsequent `git fetch` commands. +- Custom clone path to make it possible to re-use worktrees between parent project and all forks + because we use the same clone path for all forks. + +Why use `$CI_CONCURRENT_ID`? The main reason is to ensure that worktrees used are not conflicting +between projects. The `$CI_CONCURRENT_ID` represents a unique identifier within the given executor, +so as long as we use it to construct the path, it is guaranteed that this directory will not conflict +with other concurrent jobs running. + +### Store custom clone options in `config.toml` + +Ideally, all job-related configuration should be stored in `.gitlab-ci.yml`. +However, sometimes it is desirable to make these schemes part of Runner configuration. + +In the above example of Forks, making this configuration discoverable for users may be preferred, +but this brings administrative overhead as the `.gitlab-ci.yml` needs to be updated for each branch. +In such cases, it might be desirable to keep the `.gitlab-ci.yml` clone path agnostic, but make it +a configuration of Runner. + +We can extend our [config.toml](https://docs.gitlab.com/runner/configuration/advanced-configuration.html) +with the following specification that will be used by Runner if `.gitlab-ci.yml` will not override it: + +```toml +concurrent = 4 + +[[runners]] + url = "GITLAB_URL" + token = "TOKEN" + executor = "docker" + builds_dir = "/builds" + cache_dir = "/cache" + + environment = [ + "GIT_DEPTH=10", + "GIT_CLONE_PATH=$CI_BUILDS_DIR/$CI_CONCURRENT_ID/$CI_PROJECT_NAME" + ] + + [runners.docker] + volumes = ["/builds:/builds", "/cache:/cache"] +``` + +This makes the cloning configuration to be part of given Runner, +and does not require us to update each `.gitlab-ci.yml`. diff --git a/doc/ci/merge_request_pipelines/img/merge_request.png b/doc/ci/merge_request_pipelines/img/merge_request.png Binary files differindex cf9c628e9a0..d03fdc6a885 100644 --- a/doc/ci/merge_request_pipelines/img/merge_request.png +++ b/doc/ci/merge_request_pipelines/img/merge_request.png diff --git a/doc/ci/merge_request_pipelines/img/merge_request_pipeline.png b/doc/ci/merge_request_pipelines/img/merge_request_pipeline.png Binary files differnew file mode 100644 index 00000000000..58d5581f628 --- /dev/null +++ b/doc/ci/merge_request_pipelines/img/merge_request_pipeline.png diff --git a/doc/ci/merge_request_pipelines/img/merge_request_pipeline_config.png b/doc/ci/merge_request_pipelines/img/merge_request_pipeline_config.png Binary files differnew file mode 100644 index 00000000000..0a84e61d284 --- /dev/null +++ b/doc/ci/merge_request_pipelines/img/merge_request_pipeline_config.png diff --git a/doc/ci/merge_request_pipelines/img/pipeline_detail.png b/doc/ci/merge_request_pipelines/img/pipeline_detail.png Binary files differdeleted file mode 100644 index 6094a0975fb..00000000000 --- a/doc/ci/merge_request_pipelines/img/pipeline_detail.png +++ /dev/null diff --git a/doc/ci/merge_request_pipelines/index.md b/doc/ci/merge_request_pipelines/index.md index 2af0a03cbe4..fe2fc790505 100644 --- a/doc/ci/merge_request_pipelines/index.md +++ b/doc/ci/merge_request_pipelines/index.md @@ -1,21 +1,38 @@ +--- +type: reference +--- + # Pipelines for merge requests +NOTE: **Note**: +As of GitLab 11.10, pipelines for merge requests require GitLab Runner 11.9 +or higher due to the [recent refspecs +changes](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/25504). +Anything lower will cause the pipeline to fail. + > [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/issues/15310) in GitLab 11.6. -Usually, when you create a new merge request, a pipeline runs on the +Usually, when you create a new merge request, a pipeline runs with the new change and checks if it's qualified to be merged into a target branch. This -pipeline should contain only necessary jobs for checking the new changes. +pipeline should contain only necessary jobs for validating the new changes. For example, unit tests, lint checks, and [Review Apps](../review_apps/index.md) are often used in this cycle. With pipelines for merge requests, you can design a specific pipeline structure -for merge requests. All you need to do is just adding `only: [merge_requests]` to -the jobs that you want it to run for only merge requests. -Every time, when developers create or update merge requests, a pipeline runs on -their new commits at every push to GitLab. +for when you are running a pipeline in a merge request. This +could be either adding or removing steps in the pipeline, to make sure that +your pipelines are as efficient as possible. + +## Configuring pipelines for merge requests + +To configure pipelines for merge requests, add the `only: merge_requests` parameter to +the jobs that you want to run only for merge requests. + +Then, when developers create or update merge requests, a pipeline runs +every time a commit is pushed to GitLab. NOTE: **Note**: -If you use both this feature and [Merge When Pipeline Succeeds](../../user/project/merge_requests/merge_when_pipeline_succeeds.md), +If you use this feature with [merge when pipeline succeeds](../../user/project/merge_requests/merge_when_pipeline_succeeds.md), pipelines for merge requests take precedence over the other regular pipelines. For example, consider the following [`.gitlab-ci.yml`](../yaml/README.md): @@ -25,9 +42,7 @@ build: stage: build script: ./build only: - - branches - - tags - - merge_requests + - master test: stage: test @@ -38,33 +53,89 @@ test: deploy: stage: deploy script: ./deploy + only: + - master ``` -After the merge request is updated with new commits, GitLab detects that changes -have occurred and creates a new pipeline for the merge request. -The pipeline fetches the latest code from the source branch and run tests against it. -In the above example, the pipeline contains only `build` and `test` jobs. -Since the `deploy` job doesn't have the `only: [merge_requests]` rule, -deployment jobs will not happen in the merge request. +After the merge request is updated with new commits: -Pipelines tagged as **merge request** indicate that they were triggered -when a merge request was created or updated. +- GitLab detects that changes have occurred and creates a new pipeline for the merge request. +- The pipeline fetches the latest code from the source branch and run tests against it. + +In the above example, the pipeline contains only a `test` job. +Since the `build` and `deploy` jobs don't have the `only: merge_requests` parameter, +they will not run in the merge request. + +Pipelines tagged with the **detached** badge indicate that they were triggered +when a merge request was created or updated. For example:  -The same tag is shown on the pipeline's details: +## Pipelines for Merged Results **[PREMIUM]** + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/issues/7380) in [GitLab Premium](https://about.gitlab.com/pricing/) 11.10. +> This feature is disabled by default until we resolve issues with [contention handling](https://gitlab.com/gitlab-org/gitlab-ee/issues/9186), but [can be enabled manually](#enabling-pipelines-for-merged-results). + +It's possible for your source and target branches to diverge, which can result +in the scenario that source branch's pipeline was green, the target's pipeline was green, +but the combined output fails. + +By having your merge request pipeline automatically +create a new ref that contains the merge result of the source and target branch +(then running a pipeline on that ref), we can better test that the combined result +is also valid. + +GitLab can run pipelines for merge requests +on this merged result. That is, where the source and target branches are combined into a +new ref and a pipeline for this ref validates the result prior to merging. + + - +There are some cases where creating a combined ref is not possible or not wanted. +For example, a source branch that has conflicts with the target branch +or a merge request that is still in WIP status. In this case, the merge request pipeline falls back to a "detached" state +and runs on the source branch ref as if it was a regular pipeline. + +The detached state serves to warn you that you are working in a situation +subjected to merge problems, and helps to highlight that you should +get out of WIP status or resolve merge conflicts as soon as possible. + +### Enabling Pipelines for Merged Results + +To enable pipelines on merged results at the project level: + +1. Visit your project's **Settings > General** and expand **Merge requests**. +1. Check **Merge pipelines will try to validate the post-merge result prior to merging**. +1. Click **Save changes** button. + + + +CAUTION: **Warning:** +Make sure your `gitlab-ci.yml` file is [configured properly for pipelines for merge requests](#configuring-pipelines-for-merge-requests), +otherwise pipelines for merged results won't run and your merge requests will be stuck in an unresolved state. + +### Pipelines for Merged Result's limitations + +- This feature requires [GitLab Runner](https://gitlab.com/gitlab-org/gitlab-runner) 11.9 or newer. +- This feature requires [Gitaly](https://gitlab.com/gitlab-org/gitaly) 1.21.0 or newer. +- After the merge request pipeline succeeds, if the target branch has moved forward, the result of the pipeline is stale and must be retried. In busy repos, this can become a problem as it is highly probable that the target branch will have moved ahead. Improvements are [planned](https://gitlab.com/gitlab-org/gitlab-ee/issues/9186) for future versions of GitLab. +- Forking/cross-repo workflows are not currently supported. To follow progress, see [#9713](https://gitlab.com/gitlab-org/gitlab-ee/issues/9713). +- This feature is not available for [fast forward merges](../../user/project/merge_requests/fast_forward_merge.md) yet. To follow progress, see [#58226](https://gitlab.com/gitlab-org/gitlab-ce/issues/58226). ## Excluding certain jobs -The behavior of the `only: merge_requests` rule is such that _only_ jobs with -that rule are run in the context of a merge request; no other jobs will be run. +The behavior of the `only: merge_requests` parameter is such that _only_ jobs with +that parameter are run in the context of a merge request; no other jobs will be run. -However, you may want to reverse this behaviour, having all of your jobs to run _except_ -for one or two. Consider the following pipeline, with jobs `A`, `B`, and `C`. If you want -all pipelines to always run `A` and `B`, but only want `C` to run for a merge request, -you can configure your `.gitlab-ci.yml` file as follows: +However, you may want to reverse this behavior, having all of your jobs to run _except_ +for one or two. + +Consider the following pipeline, with jobs `A`, `B`, and `C`. Imagine you want: + +- All pipelines to always run `A` and `B`. +- `C` to run only for merge requests. + +To achieve this, you can configure your `.gitlab-ci.yml` file as follows: ``` yaml .only-default: &only-default @@ -90,9 +161,11 @@ C: - merge_requests ``` -Since `A` and `B` are getting the `only:` rule to execute in all cases, they will -always run. `C` specifies that it should only run for merge requests, so for any -pipeline except a merge request pipeline, it will not run. +Therefore: + +- Since `A` and `B` are getting the `only:` rule to execute in all cases, they will always run. +- Since `C` specifies that it should only run for merge requests, it will not run for any pipeline + except a merge request pipeline. As you can see, this will help you avoid a lot of boilerplate where you'd need to add that `only:` rule to all of your jobs in order to make them always run. You @@ -124,3 +197,12 @@ External users could steal secret variables from the parent project by modifying We're discussing a secure solution of running pipelines for merge requests that submitted from forked projects, see [the issue about the permission extension](https://gitlab.com/gitlab-org/gitlab-ce/issues/23902). + +## Additional predefined variables + +By using pipelines for merge requests, GitLab exposes additional predefined variables to the pipeline jobs. +Those variables contain information of the associated merge request, so that it's useful +to integrate your job with [GitLab Merge Request API](../../api/merge_requests.md). + +You can find the list of available variables in [the reference sheet](../variables/predefined_variables.md). +The variable names begin with the `CI_MERGE_REQUEST_` prefix. diff --git a/doc/ci/metrics_reports.md b/doc/ci/metrics_reports.md new file mode 100644 index 00000000000..b7824402d45 --- /dev/null +++ b/doc/ci/metrics_reports.md @@ -0,0 +1,44 @@ +--- +type: reference +--- + +# Metrics Reports **[PREMIUM]** + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/issues/9788) in [GitLab Premium](https://about.gitlab.com/pricing) 11.10. +Requires GitLab Runner 11.10 and above. + +## Overview + +GitLab provides a lot of great reporting tools for [merge requests](../user/project/merge_requests/index.md) - [JUnit reports](./junit_test_reports.md), [codequality](https://docs.gitlab.com/ee/user/project/merge_requests/code_quality.html), performance tests, etc. While JUnit is a great open framework for tests that "pass" or "fail", it is also important to see other types of metrics from a given change. + +You can configure your job to use custom Metrics Reports, and GitLab will display a report on the merge request so that it's easier and faster to identify changes without having to check the entire log. + + + +## Use cases + +Consider the following examples of data that can utilize Metrics Reports: + +1. Memory usage +1. Load testing results +1. Code complexity +1. Code coverage stats + +## How it works + +Metrics are read from the metrics report (default: `metrics.txt`). They are parsed and displayed in the MR widget. + +## How to set it up + +Add a job that creates a [metrics report](yaml/README.md#artifactsreportsmetrics-premium) (default filename: `metrics.txt`). The file should conform to the [OpenMetrics](https://openmetrics.io/) format. + +For example: + +```yaml +metrics: + script: + - echo 'metric_name metric_value' > metrics.txt + artifacts: + reports: + metrics: metrics.txt +``` diff --git a/doc/ci/multi_project_pipeline_graphs.md b/doc/ci/multi_project_pipeline_graphs.md new file mode 100644 index 00000000000..af10e5b6126 --- /dev/null +++ b/doc/ci/multi_project_pipeline_graphs.md @@ -0,0 +1,5 @@ +--- +redirect_to: 'multi_project_pipelines.md' +--- + +This document was moved to [another location](multi_project_pipelines.md). diff --git a/doc/ci/multi_project_pipelines.md b/doc/ci/multi_project_pipelines.md new file mode 100644 index 00000000000..50c8d82602b --- /dev/null +++ b/doc/ci/multi_project_pipelines.md @@ -0,0 +1,189 @@ +--- +type: reference +--- + +# Multi-project pipelines **[PREMIUM]** + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/issues/2121) in +[GitLab Premium 9.3](https://about.gitlab.com/2017/06/22/gitlab-9-3-released/#multi-project-pipeline-graphs). + +When you set up [GitLab CI/CD](README.md) across multiple projects, you can visualize +the entire pipeline, including all cross-project inter-dependencies. + +## Overview + +GitLab CI/CD is a powerful continuous integration tool that works not only per project, but also across projects. When you +configure GitLab CI for your project, you can visualize the stages +of your [jobs](pipelines.md#configuring-pipelines) on a [pipeline graph](pipelines.md#visualizing-pipelines). + + + +In the Merge Request Widget, multi-project pipeline mini-graphs are displayed, +and when hovering or tapping (on touchscreen devices) they will expand and be shown adjacent to each other. + + + +Multi-project pipelines are useful for larger products that require cross-project inter-dependencies, such as those +adopting a [microservices architecture](https://about.gitlab.com/2016/08/16/trends-in-version-control-land-microservices/). + +For a demonstration of how cross-functional development teams can use cross-pipeline +triggering to trigger multiple pipelines for different microservices projects, see +[Cross-project Pipeline Triggering and Visualization](https://about.gitlab.com/handbook/marketing/product-marketing/demo/#cross-project-pipeline-triggering-and-visualization-may-2019---1110). + +## Use cases + +Let's assume you deploy your web app from different projects in GitLab: + +- One for the free version, which has its own pipeline that builds and tests your app +- One for the paid version add-ons, which also pass through builds and tests +- One for the documentation, which also builds, tests, and deploys with an SSG + +With Multi-Project Pipelines, you can visualize the entire pipeline, including all stages of builds and tests for the three projects. + +## Triggering multi-project pipelines through API + +When you use the [`CI_JOB_TOKEN` to trigger pipelines](triggers/README.md#ci-job-token), GitLab +recognizes the source of the job token, and thus internally ties these pipelines +together, allowing you to visualize their relationships on pipeline graphs. + +These relationships are displayed in the pipeline graph by showing inbound and +outbound connections for upstream and downstream pipeline dependencies. + +## Creating multi-project pipelines from `.gitlab-ci.yml` + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/issues/8997) in [GitLab Premium](https://about.gitlab.com/pricing/) 11.8. + +### Triggering a downstream pipeline using a bridge job + +Before GitLab 11.8, it was necessary to implement a pipeline job that was +responsible for making the API request [to trigger a pipeline](#triggering-multi-project-pipelines-through-api) +in a different project. + +In GitLab 11.8, GitLab provides a new CI/CD configuration syntax to make this +task easier, and avoid needing GitLab Runner for triggering cross-project +pipelines. The following illustrates configuring a bridge job: + +```yaml +rspec: + stage: test + script: bundle exec rspec + +staging: + variables: + ENVIRONMENT: staging + stage: deploy + trigger: my/deployment +``` + +In the example above, as soon as `rspec` job succeeds in the `test` stage, +the `staging` bridge job is going to be started. The initial status of this +job will be `pending`. GitLab will create a downstream pipeline in the +`my/deployment` project and, as soon as the pipeline gets created, the +`staging` job will succeed. `my/deployment` is a full path to that project. + +The user that created the upstream pipeline needs to have access rights to the +downstream project (`my/deployment` in this case). If a downstream project can +not be found, or a user does not have access rights to create pipeline there, +the `staging` job is going to be marked as _failed_. + +CAUTION: **Caution:** +`staging` will succeed as soon as a downstream pipeline gets created. +GitLab does not support status attribution yet, however adding first-class +`trigger` configuration syntax is ground work for implementing +[status attribution](https://gitlab.com/gitlab-org/gitlab-ce/issues/39640). + +NOTE: **Note:** +Bridge jobs do not support every configuration entry that a user can use +in the case of regular jobs. Bridge jobs will not to be picked by a Runner, +thus there is no point in adding support for `script`, for example. If a user +tries to use unsupported configuration syntax, YAML validation will fail upon +pipeline creation. + +### Specifying a downstream pipeline branch + +It is possible to specify a branch name that a downstream pipeline will use: + +```yaml +rspec: + stage: test + script: bundle exec rspec + +staging: + stage: deploy + trigger: + project: my/deployment + branch: stable-11-2 +``` + +Use a `project` keyword to specify full path to a downstream project. Use +a `branch` keyword to specify a branch name. + +GitLab will use a commit that is currently on the HEAD of the branch when +creating a downstream pipeline. + +### Passing variables to a downstream pipeline + +Sometimes you might want to pass variables to a downstream pipeline. +You can do that using the `variables` keyword, just like you would when +defining a regular job. + +```yaml +rspec: + stage: test + script: bundle exec rspec + +staging: + variables: + ENVIRONMENT: staging + stage: deploy + trigger: my/deployment +``` + +The `ENVIRONMENT` variable will be passed to every job defined in a downstream +pipeline. It will be available as an environment variable when GitLab Runner picks a job. + +In the following configuration, the `MY_VARIABLE` variable will be passed to the downstream pipeline +that is created when the `trigger-downstream` job is queued. This is because `trigger-downstream` +job inherits variables declared in global variables blocks, and then we pass these variables to a downstream pipeline. + +```yaml +variables: + MY_VARIABLE: my-value + +trigger-downstream: + variables: + ENVIRONMENT: something + trigger: my/project +``` + +You might want to pass some information about the upstream pipeline using, for +example, predefined variables. In order to do that, you can use interpolation +to pass any variable. For example: + +```yaml +downstream-job: + variables: + UPSTREAM_BRANCH: $CI_COMMIT_REF_NAME + trigger: my/project +``` + +In this scenario, the `UPSTREAM_BRANCH` variable with a value related to the +upstream pipeline will be passed to the `downstream-job` job, and will be available +within the context of all downstream builds. + +### Limitations + +Because bridge jobs are a little different to regular jobs, it is not +possible to use exactly the same configuration syntax here, as one would +normally do when defining a regular job that will be picked by a runner. + +Some features are not implemented yet. For example, support for environments. + +[Configuration keywords](yaml/README.md) available for bridge jobs are: + +- `trigger` (to define a downstream pipeline trigger) +- `stage` +- `allow_failure` +- `only` and `except` +- `when` +- `extends` diff --git a/doc/ci/permissions/README.md b/doc/ci/permissions/README.md index 80d8e46f29c..bc1e6ce3e0b 100644 --- a/doc/ci/permissions/README.md +++ b/doc/ci/permissions/README.md @@ -1 +1,5 @@ -This document was moved to [user/permissions.md](../../user/permissions.md#gitlab-ci). +--- +redirect_to: '../../user/permissions.md#gitlab-cicd-permissions' +--- + +This document was moved to [user/permissions.md](../../user/permissions.md#gitlab-cicd-permissions). diff --git a/doc/ci/pipelines.md b/doc/ci/pipelines.md index 4f3106c6dc6..1ad3d516df9 100644 --- a/doc/ci/pipelines.md +++ b/doc/ci/pipelines.md @@ -1,271 +1,348 @@ -# Introduction to pipelines and jobs +--- +type: reference +--- + +# Creating and using CI/CD pipelines > Introduced in GitLab 8.8. +## Introduction + +Pipelines are the top-level component of continuous integration, delivery, and deployment. + +Pipelines comprise: + +- Jobs that define what to run. For example, code compilation or test runs. +- Stages that define when and how to run. For example, that tests run only after code compilation. + +Multiple jobs in the same stage are executed by [Runners](runners/README.md) in parallel, if there are enough concurrent [Runners](runners/README.md). + +If all the jobs in a stage: + +- Succeed, the pipeline moves on to the next stage. +- Fail, the next stage is not (usually) executed and the pipeline ends early. + NOTE: **Note:** -If you have a [mirrored repository where GitLab pulls from](https://docs.gitlab.com/ee/workflow/repository_mirroring.html#pulling-from-a-remote-repository-starter), +If you have a [mirrored repository that GitLab pulls from](../workflow/repository_mirroring.md#pulling-from-a-remote-repository-starter), you may need to enable pipeline triggering in your project's **Settings > Repository > Pull from a remote repository > Trigger pipelines for mirror updates**. -## Pipelines +### Simple pipeline example -A pipeline is a group of [jobs] that get executed in [stages]. -All of the jobs in a stage are executed in parallel (if there are enough -concurrent [Runners]), and if they all succeed, the pipeline moves on to the -next stage. If one of the jobs fails, the next stage is not (usually) -executed. You can access the pipelines page in your project's **Pipelines** tab. +As an example, imagine a pipeline consisting of four stages, executed in the following order: -In the following image you can see that the pipeline consists of four stages -(`build`, `test`, `staging`, `production`) each one having one or more jobs. +- `build`, with a job called `compile`. +- `test`, with two jobs called `test` and `test2`. +- `staging`, with a job called `deploy-to-stage`. +- `production`, with a job called `deploy-to-prod`. ->**Note:** -GitLab capitalizes the stages' names when shown in the [pipeline graphs](#pipeline-graphs). +## Visualizing pipelines - +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/5742) in GitLab 8.11. -## Types of pipelines +Pipelines can be complex structures with many sequential and parallel jobs. -There are three types of pipelines that often use the single shorthand of "pipeline". People often talk about them as if each one is "the" pipeline, but really, they're just pieces of a single, comprehensive pipeline. +To make it easier to understand the flow of a pipeline, GitLab has pipeline graphs for viewing pipelines +and their statuses. - +Pipeline graphs can be displayed in two different ways, depending on the page you +access the graph from. -1. **CI Pipeline**: Build and test stages defined in `.gitlab-ci.yml`. -1. **Deploy Pipeline**: Deploy stage(s) defined in `.gitlab-ci.yml` The flow of deploying code to servers through various stages: e.g. development to staging to production. -1. **Project Pipeline**: Cross-project CI dependencies [triggered via API][triggers], particularly for micro-services, but also for complicated build dependencies: e.g. api -> front-end, ce/ee -> omnibus. +NOTE: **Note:** +GitLab capitalizes the stages' names when shown in the pipeline graphs (below). -## Development workflows +### Regular pipeline graphs -Pipelines accommodate several development workflows: +Regular pipeline graphs show the names of the jobs of each stage. Regular pipeline graphs can +be found when you are on a [single pipeline page](#accessing-pipelines). For example: -1. **Branch Flow** (e.g. different branch for dev, qa, staging, production). -1. **Trunk-based Flow** (e.g. feature branches and single master branch, possibly with tags for releases). -1. **Fork-based Flow** (e.g. merge requests come from forks). + -Example continuous delivery flow: +### Pipeline mini graphs - +Pipeline mini graphs take less space and can tell you at a +quick glance if all jobs passed or something failed. The pipeline mini graph can +be found when you navigate to: -## Jobs +- The pipelines index page. +- A single commit page. +- A merge request page. -Jobs can be defined in the [`.gitlab-ci.yml`][jobs-yaml] file. Not to be -confused with a `build` job or `build` stage. +Pipeline mini graphs allow you to see all related jobs for a single commit and the net result +of each stage of your pipeline. This allows you to quickly see what failed and +fix it. -## Defining pipelines +Stages in pipeline mini graphs are collapsible. Hover your mouse over them and click to expand their jobs. -Pipelines are defined in `.gitlab-ci.yml` by specifying [jobs] that run in -[stages]. +| Mini graph | Mini graph expanded | +|:-------------------------------------------------------------|:---------------------------------------------------------------| +|  |  | -See the reference [documentation for jobs](yaml/README.md#jobs). +### Job ordering in pipeline graphs -## Manually executing pipelines +Job ordering depends on the type of pipeline graph. For [regular pipeline graphs](#regular-pipeline-graphs), jobs are sorted by name. -Pipelines can be manually executed, with predefined or manually-specified [variables](variables/README.md). +For [pipeline mini graphs](#pipeline-mini-graphs) ([introduced](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/9760) +in GitLab 9.0), jobs are sorted by severity and then by name. -To execute a pipeline manually: +The order of severity is: -1. Navigate to your project's **CI/CD > Pipelines**. -1. Click on the **Run Pipeline** button. -1. Select the branch to run the pipeline for and enter any environment variables required for the pipeline run. +- failed +- warning +- pending +- running +- manual +- scheduled +- canceled +- success +- skipped +- created -## Seeing pipeline status +For example: -You can find the current and historical pipeline runs under your project's -**Pipelines** tab. Clicking on a pipeline will show the jobs that were run for -that pipeline. - - + -## Seeing job status +### How pipeline duration is calculated -When you visit a single pipeline you can see the related jobs for that pipeline. -Clicking on an individual job will show you its job trace, and allow you to -cancel the job, retry it, or erase the job trace. +Total running time for a given pipeline excludes retries and pending +(queued) time. - +Each job is represented as a `Period`, which consists of: -## Seeing the failure reason for jobs +- `Period#first` (when the job started). +- `Period#last` (when the job finished). -> [Introduced][ce-17782] in GitLab 10.7. +A simple example is: -When a pipeline fails or is allowed to fail, there are several places where you -can quickly check the reason it failed: +- A (1, 3) +- B (2, 4) +- C (6, 7) -- **In the pipeline graph** present on the pipeline detail view. -- **In the pipeline widgets** present in the merge requests and commit pages. -- **In the job views** present in the global and detailed views of a job. +In the example: -In any case, if you hover over the failed job you can see the reason it failed. +- A begins at 1 and ends at 3. +- B begins at 2 and ends at 4. +- C begins at 6 and ends at 7. - +Visually, it can be viewed as: -From [GitLab 10.8][ce-17814] you can also see the reason it failed on the Job detail page. +```text +0 1 2 3 4 5 6 7 + AAAAAAA + BBBBBBB + CCCC +``` -## Pipeline graphs +The union of A, B, and C is (1, 4) and (6, 7). Therefore, the total running time is: -> [Introduced][ce-5742] in GitLab 8.11. +```text +(4 - 1) + (7 - 6) => 4 +``` -Pipelines can be complex structures with many sequential and parallel jobs. -To make it a little easier to see what is going on, you can view a graph -of a single pipeline and its status. +## Configuring pipelines -A pipeline graph can be shown in two different ways depending on what page you -are on. +Pipelines, and their component jobs and stages, are defined in the [`.gitlab-ci.yml`](yaml/README.md) file for each project. ---- +In particular: -The regular pipeline graph that shows the names of the jobs of each stage can -be found when you are on a [single pipeline page](#seeing-pipeline-status). +- Jobs are the [basic configuration](yaml/README.html#introduction) component. +- Stages are defined using the [`stages`](yaml/README.html#stages) keyword. - +For all available configuration options, see the [GitLab CI/CD Pipeline Configuration Reference](yaml/README.md). -Then, there is the pipeline mini graph which takes less space and can give you a -quick glance if all jobs pass or something failed. The pipeline mini graph can -be found when you visit: +### Settings and schedules -- The pipelines index page. -- A single commit page. -- A merge request page. +In addition to configuring jobs through `.gitlab-ci.yml`, additional configuration options are available +through the GitLab UI: -That way, you can see all related jobs for a single commit and the net result -of each stage of your pipeline. This allows you to quickly see what failed and -fix it. Stages in pipeline mini graphs are collapsible. Hover your mouse over -them and click to expand their jobs. +- Pipeline settings for each project. For more information, see [Pipeline settings](../user/project/pipelines/settings.md). +- Schedules for pipelines. For more information, see [Pipeline schedules](../user/project/pipelines/schedules.md). -| **Mini graph** | **Mini graph expanded** | -| :------------: | :---------------------: | -|  |  | +### Grouping jobs -### Grouping similar jobs in the pipeline graph +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/6242) in GitLab 8.12. -> [Introduced][ce-6242] in GitLab 8.12. +If you have many similar jobs, your [pipeline graph](#visualizing-pipelines) becomes long and hard +to read. -If you have many similar jobs, your pipeline graph becomes very long and hard -to read. For that reason, similar jobs can automatically be grouped together. +For that reason, similar jobs can automatically be grouped together. If the job names are formatted in certain ways, they will be collapsed into a single group in regular pipeline graphs (not the mini graphs). + You'll know when a pipeline has grouped jobs if you don't see the retry or cancel button inside them. Hovering over them will show the number of grouped jobs. Click to expand them.  -The basic requirements is that there are two numbers separated with one of +#### Configuring grouping + +In the pipeline [configuration file](yaml/README.md), job names must include two numbers separated with one of the following (you can even use them interchangeably): -- A space (` `) -- A slash (`/`) -- A colon (`:`) +- A space. +- A slash (`/`). +- A colon (`:`). + +NOTE: **Note:** +More specifically, it uses [this](https://gitlab.com/gitlab-org/gitlab-ce/blob/2f3dc314f42dbd79813e6251792853bc231e69dd/app/models/commit_status.rb#L99) regular expression: `\d+[\s:\/\\]+\d+\s*`. ->**Note:** -More specifically, [it uses][regexp] this regular expression: `\d+[\s:\/\\]+\d+\s*`. +#### How grouping works The jobs will be ordered by comparing those two numbers from left to right. You usually want the first to be the index and the second the total. For example, the following jobs will be grouped under a job named `test`: -- `test 0 3` => `test` -- `test 1 3` => `test` -- `test 2 3` => `test` +- `test 0 3` +- `test 1 3` +- `test 2 3` The following jobs will be grouped under a job named `test ruby`: -- `test 1:2 ruby` => `test ruby` -- `test 2:2 ruby` => `test ruby` +- `test 1:2 ruby` +- `test 2:2 ruby` The following jobs will be grouped under a job named `test ruby` as well: -- `1/3 test ruby` => `test ruby` -- `2/3 test ruby` => `test ruby` -- `3/3 test ruby` => `test ruby` +- `1/3 test ruby` +- `2/3 test ruby` +- `3/3 test ruby` -### Manual actions from the pipeline graph +### Pipelines for merge requests -> [Introduced][ce-7931] in GitLab 8.15. +GitLab supports configuring pipelines that run only for merge requests. For more information, see +[Pipelines for merge requests](merge_request_pipelines/index.md). -[Manual actions][manual] allow you to require manual interaction before moving -forward with a particular job in CI. Your entire pipeline can run automatically, -but the actual [deploy to production][env-manual] will require a click. +### Badges -You can do this straight from the pipeline graph. Just click on the play button -to execute that particular job. For example, in the image below, the `production` -stage has a job with a manual action. +Pipeline status and test coverage report badges are available and configurable for each project. - +For information on adding pipeline badges to projects, see [Pipeline badges](../user/project/pipelines/settings.md#pipeline-badges). -### Delay a particular job in the pipeline graph +## Multi-project pipelines **[PREMIUM]** -> [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/21767) in GitLab 11.4. +Pipelines for different projects can be combined and visualized together. -When you do not want to run a job immediately, you can [delay the job to run after a certain period](yaml/README.md#whendelayed). -This is especially useful for timed incremental rollout that new code is rolled out gradually. -For example, if you start rolling out new code and users do not experience trouble, GitLab automatically completes the deployment from 0% to 100%. -Alternatively, if you start rolling out and you noticed that a few users experience trouble with the version, -you can stop the timed incremental rollout by canceling the pipeline, and [rolling](environments.md#rolling-back-changes) it back to the stable version. +For more information, see [Multi-project pipelines](multi_project_pipelines.md). - +## Working with pipelines -### Ordering of jobs in pipeline graphs +In general, pipelines are executed automatically and require no intervention once created. -**Regular pipeline graph** +However, there are instances where you'll need to interact with pipelines. These are documented below. -In the single pipeline page, jobs are sorted by name. +### Manually executing pipelines -**Mini pipeline graph** +Pipelines can be manually executed, with predefined or manually-specified [variables](variables/README.md). -> [Introduced][ce-9760] in GitLab 9.0. +You might do this if the results of a pipeline (for example, a code build) is required outside the normal +operation of the pipeline. -In the pipeline mini graphs, the jobs are sorted first by severity and then -by name. The order of severity is: +To execute a pipeline manually: -- failed -- warning -- pending -- running -- manual -- scheduled -- canceled -- success -- skipped -- created +1. Navigate to your project's **CI/CD > Pipelines**. +1. Click on the **Run Pipeline** button. +1. On the **Run Pipeline** page: + 1. Select the branch to run the pipeline for in the **Create for** field. + 1. Enter any [environment variables](variables/README.md) required for the pipeline run. + 1. Click the **Create pipeline** button. - +The pipeline will execute the jobs as configured. -## How the pipeline duration is calculated +### Accessing pipelines -Total running time for a given pipeline would exclude retries and pending -(queue) time. We could reduce this problem down to finding the union of -periods. +You can find the current and historical pipeline runs under your project's +**CI/CD > Pipelines** page. Clicking on a pipeline will show the jobs that were run for +that pipeline. -So each job would be represented as a `Period`, which consists of -`Period#first` as when the job started and `Period#last` as when the -job was finished. A simple example here would be: + -- A (1, 3) -- B (2, 4) -- C (6, 7) +You can also access pipelines for a merge request by navigating to its **Pipelines** tab. -Here A begins from 1, and ends to 3. B begins from 2, and ends to 4. -C begins from 6, and ends to 7. Visually it could be viewed as: +### Accessing individual jobs -``` -0 1 2 3 4 5 6 7 - AAAAAAA - BBBBBBB - CCCC -``` +When you access a pipeline, you can see the related jobs for that pipeline. -The union of A, B, and C would be (1, 4) and (6, 7), therefore the -total running time should be: +Clicking on an individual job will show you its job trace, and allow you to: -``` -(4 - 1) + (7 - 6) => 4 -``` +- Cancel the job. +- Retry the job. +- Erase the job trace. -## Badges +### Seeing the failure reason for jobs -Pipeline status and test coverage report badges are available. You can find their -respective link in the [Pipelines settings] page. +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/17782) in GitLab 10.7. + +When a pipeline fails or is allowed to fail, there are several places where you +can quickly check the reason it failed: + +- In the pipeline graph, on the pipeline detail view. +- In the pipeline widgets, in the merge requests and commit pages. +- In the job views, in the global and detailed views of a job. + +In each place, if you hover over the failed job you can see the reason it failed. + + + +From [GitLab 10.8](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/17814), +you can also see the reason it failed on the Job detail page. + +### Manual actions from pipeline graphs + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/7931) in GitLab 8.15. + +Manual actions, configured using the [`when:manual`](yaml/README.md#whenmanual) parameter, +allow you to require manual interaction before moving forward in the pipeline. + +You can do this straight from the pipeline graph. Just click on the play button +to execute that particular job. + +For example, your pipeline start automatically, but require manual action to +[deploy to production](environments.md#configuring-manual-deployments). In the example below, the `production` +stage has a job with a manual action. + + + +### Delay a job in a pipeline graph + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/21767) in GitLab 11.4. + +When you do not want to run a job immediately, you can use the [`when:delayed`](yaml/README.md#whendelayed) parameter to +delay a job's execution for a certain period. + +This is especially useful for timed incremental rollout where new code is rolled out gradually. + +For example, if you start rolling out new code and: + +- Users do not experience trouble, GitLab can automatically complete the deployment from 0% to 100%. +- Users experience trouble with the new code, you can stop the timed incremental rollout by canceling the pipeline + and [rolling](environments.md#retrying-and-rolling-back) back to the last stable version. + + + +### Using the API + +GitLab provides API endpoints to: + +- Perform basic functions. For more information, see [Pipelines API](../api/pipelines.md). +- Maintain pipeline schedules. For more information, see [Pipeline schedules API](../api/pipeline_schedules.md). +- Trigger pipeline runs. For more information, see: + - [Triggering pipelines through the API](triggers/README.md). + - [Pipeline triggers API](../api/pipeline_triggers.md). + +### Start multiple manual actions in a stage + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/27188) in GitLab 11.11. + +Multiple manual actions in a single stage can be started at the same time using the "Play all manual" button. +Once the user clicks this button, each individual manual action will be triggered and refreshed +to an updated status. + +This functionality is only available: + +- For users with at least Developer access. +- If the the stage contains [manual actions](#manual-actions-from-pipeline-graphs). ## Security on protected branches @@ -276,14 +353,14 @@ The following actions are allowed on protected branches only if the user is [allowed to merge or push](../user/project/protected_branches.md#using-the-allowed-to-merge-and-allowed-to-push-settings) on that specific branch: -- Run **manual pipelines** (using [Web UI](#manually-executing-pipelines) or Pipelines API). -- Run **scheduled pipelines**. -- Run pipelines using **triggers**. -- Trigger **manual actions** on existing pipelines. -- **Retry/cancel** existing jobs (using Web UI or Pipelines API). +- Run manual pipelines (using the [Web UI](#manually-executing-pipelines) or pipelines API). +- Run scheduled pipelines. +- Run pipelines using triggers. +- Trigger manual actions on existing pipelines. +- Retry or cancel existing jobs (using the Web UI or pipelines API). **Variables** marked as **protected** are accessible only to jobs that -run on protected branches, avoiding untrusted users to get unintended access to +run on protected branches, preventing untrusted users getting unintended access to sensitive information like deployment credentials and tokens. **Runners** marked as **protected** can run jobs only on protected @@ -291,19 +368,3 @@ branches, avoiding untrusted code to be executed on the protected runner and preserving deployment keys and other credentials from being unintentionally accessed. In order to ensure that jobs intended to be executed on protected runners will not use regular runners, they must be tagged accordingly. - -[jobs]: #jobs -[jobs-yaml]: yaml/README.md#jobs -[manual]: yaml/README.md#whenmanual -[env-manual]: environments.md#manually-deploying-to-environments -[stages]: yaml/README.md#stages -[runners]: runners/README.html -[pipelines settings]: ../user/project/pipelines/settings.md -[triggers]: triggers/README.md -[ce-5742]: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/5742 -[ce-6242]: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/6242 -[ce-7931]: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/7931 -[ce-9760]: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/9760 -[ce-17782]: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/17782 -[ce-17814]: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/17814 -[regexp]: https://gitlab.com/gitlab-org/gitlab-ce/blob/2f3dc314f42dbd79813e6251792853bc231e69dd/app/models/commit_status.rb#L99 diff --git a/doc/ci/quick_start/README.md b/doc/ci/quick_start/README.md index 9684cb6ed98..11bcfd5dc2c 100644 --- a/doc/ci/quick_start/README.md +++ b/doc/ci/quick_start/README.md @@ -1,9 +1,18 @@ +--- +type: reference +--- + # Getting started with GitLab CI/CD ->**Note:** Starting from version 8.0, GitLab [Continuous Integration][ci] (CI) +NOTE: **Note:** +Starting from version 8.0, GitLab [Continuous Integration][ci] (CI) is fully integrated into GitLab itself and is [enabled] by default on all projects. +NOTE: **Note:** +Please keep in mind that only project Maintainers and Admin users have +the permissions to access a project's settings. + GitLab offers a [continuous integration][ci] service. If you [add a `.gitlab-ci.yml` file][yaml] to the root directory of your repository, and configure your GitLab project to use a [Runner], then each commit or @@ -35,11 +44,12 @@ project's **Pipelines** page. --- -This guide assumes that you: +This guide assumes that you have: -- have a working GitLab instance of version 8.0+r or are using - [GitLab.com](https://gitlab.com) -- have a project in GitLab that you would like to use CI for +- A working GitLab instance of version 8.0+r or are using + [GitLab.com](https://gitlab.com). +- A project in GitLab that you would like to use CI for. +- Maintainer or owner access to the project Let's break it down to pieces and work on solving the GitLab CI puzzle. @@ -73,6 +83,8 @@ You need to create a file named `.gitlab-ci.yml` in the root directory of your repository. Below is an example for a Ruby on Rails project. ```yaml +image: "ruby:2.5" + before_script: - apt-get update -qq && apt-get install -y -qq sqlite3 libsqlite3-dev nodejs - ruby -v @@ -105,7 +117,7 @@ Jobs are used to create jobs, which are then picked by What is important is that each job is run independently from each other. If you want to check whether the `.gitlab-ci.yml` of your project is valid, there is a -Lint tool under the page `/ci/lint` of your project namespace. You can also find +Lint tool under the page `/-/ci/lint` of your project namespace. You can also find a "CI Lint" button to go to this page under **CI/CD ➔ Pipelines** and **Pipelines ➔ Jobs** in your project. @@ -127,7 +139,7 @@ Now if you go to the **Pipelines** page you will see that the pipeline is pending. NOTE: **Note:** -If you have a [mirrored repository where GitLab pulls from](https://docs.gitlab.com/ee/workflow/repository_mirroring.html#pulling-from-a-remote-repository-starter), +If you have a [mirrored repository where GitLab pulls from](../../workflow/repository_mirroring.md#pulling-from-a-remote-repository-starter), you may need to enable pipeline triggering in your project's **Settings > Repository > Pull from a remote repository > Trigger pipelines for mirror updates**. diff --git a/doc/ci/review_apps/index.md b/doc/ci/review_apps/index.md index f4d7b9ad194..7b039fe6654 100644 --- a/doc/ci/review_apps/index.md +++ b/doc/ci/review_apps/index.md @@ -1,120 +1,79 @@ +--- +type: reference +--- + # Review Apps > - [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/issues/21971) in GitLab 8.12. Further additions were made in GitLab 8.13 and 8.14. > - Inspired by [Heroku's Review Apps](https://devcenter.heroku.com/articles/github-integration-review-apps), which itself was inspired by [Fourchette](https://github.com/rainforestapp/fourchette). -For a video introduction to Review Apps, see [8.14 Webcast: Review Apps & Time Tracking Beta (EE) - GitLab Release](https://www.youtube.com/watch?v=CteZol_7pxo). - -## Overview +Review Apps is a collaboration tool that takes the hard work out of providing an environment to showcase product changes. -Review Apps are a collaboration tool that takes the hard work out of providing an environment to showcase product changes. +## Introduction Review Apps: - Provide an automatic live preview of changes made in a feature branch by spinning up a dynamic environment for your merge requests. -- Allow designers and product manages to see your changes without needing to check out your branch and run your changes in a sandbox environment. -- Are fully integrated with the [GitLab DevOps LifeCycle](../../README.md#complete-devops-with-gitlab). +- Allow designers and product managers to see your changes without needing to check out your branch and run your changes in a sandbox environment. +- Are fully integrated with the [GitLab DevOps LifeCycle](../../README.md#the-entire-devops-lifecycle). - Allow you to deploy your changes wherever you want.  -Reviewing anything, from performance to interface changes, becomes much easier with a live environment and so Review Apps can make a large impact on your development flow. +In the above example: + +- A Review App is built every time a commit is pushed to `topic branch`. +- The reviewer fails two reviews before passing the third review. +- Once the review as passed, `topic branch` is merged into `master` where it's deploy to staging. +- After been approved in staging, the changes that were merged into `master` are deployed in to production. -## What are Review Apps? +### How Review Apps work -A Review App is a mapping of a branch with an [environment](../environments.md). The following is an example of a merge request with an environment set dynamically. +A Review App is a mapping of a branch with an [environment](../environments.md). +Access to the Review App is made available as a link on the [merge request](../../user/project/merge_requests.md) relevant to the branch. + +The following is an example of a merge request with an environment set dynamically.  -In this example, you can see a branch was: +In this example, a branch was: - Successfully built. - Deployed under a dynamic environment that can be reached by clicking on the **View app** button. -## How do Review Apps work? - -The basis of Review Apps in GitLab is [dynamic environments](../environments.md#dynamic-environments), which allow you to dynamically create a new environment for each branch. - -Access to the Review App is made available as a link on the [merge request](../../user/project/merge_requests.md) relevant to the branch. Review Apps enable you to review all changes proposed by the merge request in live environment. - -## Use cases - -Some supported use cases include the: - -- Simple case of deploying a simple static HTML website. -- More complicated case of an application that uses a database. Deploying a branch on a temporary instance and booting up this instance with all required software and services automatically on the fly is not a trivial task. However, it is possible, especially if you use Docker or a configuration management tool like Chef, Puppet, Ansible, or Salt. - -Review Apps usually make sense with web applications, but you can use them any way you'd like. - -## Implementing Review Apps - -Implementing Review Apps depends on your: - -- Technology stack. -- Deployment process. - -### Prerequisite Knowledge - -To get a better understanding of Review Apps, review documentation on how environments and deployments work. Before you implement your own Review Apps: - -1. Learn about [environments](../environments.md) and their role in the development workflow. -1. Learn about [CI variables](../variables/README.md) and how they can be used in your CI jobs. -1. Explore the [`environment` syntax](../yaml/README.md#environment) as defined in `.gitlab-ci.yml`. This will become a primary reference. -1. Additionally, find out about [manual actions](../environments.md#manually-deploying-to-environments) and how you can use them to deploy to critical environments like production with the push of a button. -1. Follow the [example tutorials](#examples). These will guide you through setting up infrastructure and using Review Apps. - -### Configuring dynamic environments - -Configuring Review Apps dynamic environments depends on your technology stack and infrastructure. - -For more information, see [dynamic environments](../environments.md#dynamic-environments) documentation to understand how to define and create them. - -### Creating and destroying Review Apps - -Creating and destroying Review Apps is defined in `.gitlab-ci.yml` at a job level under the `environment` keyword. +## Configuring Review Apps -For more information, see [Introduction to environments and deployments](../environments.md). +Review Apps are built on [dynamic environments](../environments.md#configuring-dynamic-environments), which allow you to dynamically create a new environment for each branch. -### Adding Review Apps to your workflow - -The process of adding Review Apps in your workflow is as follows: +The process of configuring Review Apps is as follows: 1. Set up the infrastructure to host and deploy the Review Apps. 1. [Install](https://docs.gitlab.com/runner/install/) and [configure](https://docs.gitlab.com/runner/commands/) a Runner to do deployment. -1. Set up a job in `.gitlab-ci.yml` that uses the predefined [predefined CI environment variable](../variables/README.md) `${CI_COMMIT_REF_NAME}` to create dynamic environments and restrict it to run only on branches. +1. Set up a job in `.gitlab-ci.yml` that uses the [predefined CI environment variable](../variables/README.md) `${CI_COMMIT_REF_NAME}` to create dynamic environments and restrict it to run only on branches. 1. Optionally, set a job that [manually stops](../environments.md#stopping-an-environment) the Review Apps. -After adding Review Apps to your workflow, you follow the branched Git flow. That is: +### Examples -1. Push a branch and let the Runner deploy the Review App based on the `script` definition of the dynamic environment job. -1. Wait for the Runner to build and deploy your web application. -1. Click on the link that provided in the merge request related to the branch to see the changes live. - -## Limitations - -Check the [environments limitations](../environments.md#limitations). - -## Examples - -The following are example projects that use Review Apps with: +The following are example projects that demonstrate Review App configuration: - [NGINX](https://gitlab.com/gitlab-examples/review-apps-nginx). - [OpenShift](https://gitlab.com/gitlab-examples/review-apps-openshift). See also the video [Demo: Cloud Native Development with GitLab](https://www.youtube.com/watch?v=jfIyQEwrocw), which includes a Review Apps example. -## Route Maps +### Route Maps -> Introduced in GitLab 8.17. In GitLab 11.5 the file links -are surfaced to the merge request widget. +> Introduced in GitLab 8.17. In GitLab 11.5, the file links are available in the merge request widget. Route Maps allows you to go directly from source files to public pages on the [environment](../environments.md) defined for -Review Apps. Once set up, the review app link in the merge request +Review Apps. + +Once set up, the review app link in the merge request widget can take you directly to the pages changed, making it easier and faster to preview proposed modifications. -All you need to do is to tell GitLab how the paths of files +Configuring Route Maps involves telling GitLab how the paths of files in your repository map to paths of pages on your website using a Route Map. Once set, GitLab will display **View on ...** buttons, which will take you to the pages changed directly from merge requests. @@ -123,9 +82,9 @@ To set up a route map, add a a file inside the repository at `.gitlab/route-map. which contains a YAML array that maps `source` paths (in the repository) to `public` paths (on the website). -### Route Maps example +#### Route Maps example -Below there's an example of a route map for [Middleman](https://middlemanapp.com), +The following is an example of a route map for [Middleman](https://middlemanapp.com), a static site generator (SSG) used to build [GitLab's website](https://about.gitlab.com), deployed from its [project on GitLab.com](https://gitlab.com/gitlab-com/www-gitlab-com): @@ -147,18 +106,17 @@ deployed from its [project on GitLab.com](https://gitlab.com/gitlab-com/www-gitl public: '\1' # images/blogimages/around-the-world-in-6-releases-cover.png ``` -Mappings are defined as entries in the root YAML array, and are identified by a `-` prefix. Within an entry, we have a hash map with two keys: +Mappings are defined as entries in the root YAML array, and are identified by a `-` prefix. Within an entry, there is a hash map with two keys: - `source` - - a string, starting and ending with `'`, for an exact match - - a regular expression, starting and ending with `/`, for a pattern match - - The regular expression needs to match the entire source path - `^` and `$` anchors are implied. - - Can include capture groups denoted by `()` that can be referred to in the `public` path. - - Slashes (`/`) can, but don't have to, be escaped as `\/`. - - Literal periods (`.`) should be escaped as `\.`. -- `public` - - a string, starting and ending with `'`. - - Can include `\N` expressions to refer to capture groups in the `source` regular expression in order of their occurrence, starting with `\1`. + - A string, starting and ending with `'`, for an exact match. + - A regular expression, starting and ending with `/`, for a pattern match: + - The regular expression needs to match the entire source path - `^` and `$` anchors are implied. + - Can include capture groups denoted by `()` that can be referred to in the `public` path. + - Slashes (`/`) can, but don't have to, be escaped as `\/`. + - Literal periods (`.`) should be escaped as `\.`. +- `public`, a string starting and ending with `'`. + - Can include `\N` expressions to refer to capture groups in the `source` regular expression in order of their occurrence, starting with `\1`. The public path for a source path is determined by finding the first `source` expression that matches it, and returning the corresponding @@ -171,12 +129,12 @@ will match `/source\/(.+?\.html).*/` instead of `/source\/(.*)/`, and will result in a public path of `index.html`, instead of `index.html.haml`. -Once you have the route mapping set up, it will be exposed in a few places: +Once you have the route mapping set up, it will take effect in the following locations: -- In the merge request widget. The **View app** button will take you to the - environment URL you have set up in `.gitlab-ci.yml`. The dropdown will render - the first 5 matched items from the route map, but you can filter them if more - than 5 are available. +- In the merge request widget. The: + - **View app** button will take you to the environment URL set in `.gitlab-ci.yml`. + - Dropdown will list the first 5 matched items from the route map, but you can filter them if more + than 5 are available.  @@ -187,3 +145,15 @@ Once you have the route mapping set up, it will be exposed in a few places: - In the blob file view.  + +## Working with Review Apps + +After adding Review Apps to your workflow, you follow the branched Git flow. That is: + +1. Push a branch and let the Runner deploy the Review App based on the `script` definition of the dynamic environment job. +1. Wait for the Runner to build and deploy your web application. +1. Click on the link that provided in the merge request related to the branch to see the changes live. + +## Limitations + +Review App limitations are the same as [environments limitations](../environments.md#limitations). diff --git a/doc/ci/runners/README.md b/doc/ci/runners/README.md index ce55b231666..b089229ab58 100644 --- a/doc/ci/runners/README.md +++ b/doc/ci/runners/README.md @@ -1,3 +1,7 @@ +--- +type: reference +--- + # Configuring GitLab Runners In GitLab CI, Runners run the code defined in [`.gitlab-ci.yml`](../yaml/README.md). diff --git a/doc/ci/runners/shared_to_specific_admin.png b/doc/ci/runners/shared_to_specific_admin.png Binary files differdeleted file mode 100644 index 8f4010a5849..00000000000 --- a/doc/ci/runners/shared_to_specific_admin.png +++ /dev/null diff --git a/doc/ci/services/README.md b/doc/ci/services/README.md index d94b472b768..7fe12eb53e7 100644 --- a/doc/ci/services/README.md +++ b/doc/ci/services/README.md @@ -1,13 +1,18 @@ --- comments: false +type: index --- -# GitLab CI Services +# GitLab CI services examples -GitLab CI uses the `services` keyword to define what docker containers should -be linked with your base image. Below is a list of examples you may use. +The [`services`](../docker/using_docker_images.md#what-is-a-service) +keyword defines a Docker image that runs during a `job` linked to the +Docker image that the image keyword defines. This allows you to access +the service image during build time. + +The service image can run any application, but the most common use +case is to run a database container, for example: - [Using MySQL](mysql.md) - [Using PostgreSQL](postgres.md) - [Using Redis](redis.md) -- [Using Other Services](../docker/using_docker_images.md#how-to-use-other-images-as-services) diff --git a/doc/ci/services/mysql.md b/doc/ci/services/mysql.md index 2902c30c7c0..697452cee83 100644 --- a/doc/ci/services/mysql.md +++ b/doc/ci/services/mysql.md @@ -1,3 +1,7 @@ +--- +type: reference +--- + # Using MySQL As many applications depend on MySQL as their database, you will eventually @@ -17,8 +21,8 @@ services: variables: # Configure mysql environment variables (https://hub.docker.com/_/mysql/) - MYSQL_DATABASE: el_duderino - MYSQL_ROOT_PASSWORD: mysql_strong_password + MYSQL_DATABASE: "<your_mysql_database>" + MYSQL_ROOT_PASSWORD: "<your_mysql_password>" ``` And then configure your application to use the database, for example: @@ -26,18 +30,18 @@ And then configure your application to use the database, for example: ```yaml Host: mysql User: root -Password: mysql_strong_password -Database: el_duderino +Password: <your_mysql_password> +Database: <your_mysql_database> ``` If you are wondering why we used `mysql` for the `Host`, read more at [How services are linked to the job](../docker/using_docker_images.md#how-services-are-linked-to-the-job). -You can also use any other docker image available on [Docker Hub][hub-mysql]. +You can also use any other docker image available on [Docker Hub](https://hub.docker.com/_/mysql/). For example, to use MySQL 5.5 the service becomes `mysql:5.5`. The `mysql` image can accept some environment variables. For more details -check the documentation on [Docker Hub][hub-mysql]. +check the documentation on [Docker Hub](https://hub.docker.com/_/mysql/). ## Use MySQL with the Shell executor @@ -74,13 +78,13 @@ mysql> CREATE USER 'runner'@'localhost' IDENTIFIED BY '$password'; Create the database: ```bash -mysql> CREATE DATABASE IF NOT EXISTS `el_duderino` DEFAULT CHARACTER SET `utf8` COLLATE `utf8_unicode_ci`; +mysql> CREATE DATABASE IF NOT EXISTS `<your_mysql_database>` DEFAULT CHARACTER SET `utf8` COLLATE `utf8_unicode_ci`; ``` Grant the necessary permissions on the database: ```bash -mysql> GRANT SELECT, INSERT, UPDATE, DELETE, CREATE, CREATE TEMPORARY TABLES, DROP, INDEX, ALTER, LOCK TABLES ON `el_duderino`.* TO 'runner'@'localhost'; +mysql> GRANT SELECT, INSERT, UPDATE, DELETE, CREATE, CREATE TEMPORARY TABLES, DROP, INDEX, ALTER, LOCK TABLES ON `<your_mysql_database>`.* TO 'runner'@'localhost'; ``` If all went well you can now quit the database session: @@ -93,7 +97,7 @@ Now, try to connect to the newly created database to check that everything is in place: ```bash -mysql -u runner -p -D el_duderino +mysql -u runner -p -D <your_mysql_database> ``` As a final step, configure your application to use the database, for example: @@ -102,17 +106,14 @@ As a final step, configure your application to use the database, for example: Host: localhost User: runner Password: $password -Database: el_duderino +Database: <your_mysql_database> ``` ## Example project -We have set up an [Example MySQL Project][mysql-example-repo] for your +We have set up an [Example MySQL Project](https://gitlab.com/gitlab-examples/mysql) for your convenience that runs on [GitLab.com](https://gitlab.com) using our publicly available [shared runners](../runners/README.md). -Want to hack on it? Simply fork it, commit and push your changes. Within a few +Want to hack on it? Simply fork it, commit and push your changes. Within a few moments the changes will be picked by a public runner and the job will begin. - -[hub-mysql]: https://hub.docker.com/_/mysql/ -[mysql-example-repo]: https://gitlab.com/gitlab-examples/mysql diff --git a/doc/ci/services/postgres.md b/doc/ci/services/postgres.md index 3899b555f32..211eea26eb0 100644 --- a/doc/ci/services/postgres.md +++ b/doc/ci/services/postgres.md @@ -1,3 +1,7 @@ +--- +type: reference +--- + # Using PostgreSQL As many applications depend on PostgreSQL as their database, you will @@ -31,7 +35,7 @@ Database: nice_marmot ``` If you are wondering why we used `postgres` for the `Host`, read more at -[How is service linked to the job](../docker/using_docker_images.md#how-is-service-linked-to-the-job). +[How services are linked to the job](../docker/using_docker_images.md#how-services-are-linked-to-the-job). You can also use any other docker image available on [Docker Hub][hub-pg]. For example, to use PostgreSQL 9.3 the service becomes `postgres:9.3`. diff --git a/doc/ci/services/redis.md b/doc/ci/services/redis.md index 36f71427ae7..8b227154b06 100644 --- a/doc/ci/services/redis.md +++ b/doc/ci/services/redis.md @@ -1,3 +1,7 @@ +--- +type: reference +--- + # Using Redis As many applications depend on Redis as their key-value store, you will diff --git a/doc/ci/ssh_keys/README.md b/doc/ci/ssh_keys/README.md index ff63d0bd69f..69591ed605c 100644 --- a/doc/ci/ssh_keys/README.md +++ b/doc/ci/ssh_keys/README.md @@ -1,5 +1,6 @@ --- last_updated: 2017-12-13 +type: tutorial --- # Using SSH keys with GitLab CI/CD @@ -49,7 +50,7 @@ to access it. This is where an SSH key pair comes in handy. **Do not** add a passphrase to the SSH key, or the `before_script` will prompt for it. -1. Create a new [variable](../variables/README.md#variables). +1. Create a new [variable](../variables/README.md#gitlab-cicd-environment-variables). As **Key** enter the name `SSH_PRIVATE_KEY` and in the **Value** field paste the content of your _private_ key that you created earlier. @@ -157,7 +158,7 @@ ssh-keyscan example.com ssh-keyscan 1.2.3.4 ``` -Create a new [variable](../variables/README.md#variables) with +Create a new [variable](../variables/README.md#gitlab-cicd-environment-variables) with `SSH_KNOWN_HOSTS` as "Key", and as a "Value" add the output of `ssh-keyscan`. NOTE: **Note:** diff --git a/doc/ci/triggers/README.md b/doc/ci/triggers/README.md index 61037360326..04c541fefe7 100644 --- a/doc/ci/triggers/README.md +++ b/doc/ci/triggers/README.md @@ -1,10 +1,14 @@ +--- +type: tutorial +--- + # Triggering pipelines through the API > **Notes**: > > - [Introduced](https://about.gitlab.com/2015/08/22/gitlab-7-14-released/) in GitLab 7.14. > - GitLab 8.12 has a completely redesigned job permissions system. Read all -> about the [new model and its implications](../../user/project/new_ci_build_permissions_model.md#job-triggers). +> about the [new model and its implications](../../user/project/new_ci_build_permissions_model.md#pipeline-triggers). Triggers can be used to force a pipeline rerun of a specific `ref` (branch or tag) with an API call. @@ -17,6 +21,69 @@ The following methods of authentication are supported. A unique trigger token can be obtained when [adding a new trigger](#adding-a-new-trigger). +DANGER: **Danger:** +Passing plain text tokens in public projects is a security issue. Potential +attackers can impersonate the user that exposed their trigger token publicly in +their `.gitlab-ci.yml` file. Use [variables](../variables/README.md#gitlab-cicd-environment-variables) +to protect trigger tokens. + +### CI job token + +You can use the `CI_JOB_TOKEN` [variable][predef] (used to authenticate +with the [GitLab Container Registry][registry]) in the following cases. + +#### When used with multi-project pipelines **[PREMIUM]** + +> **Note**: +The use of `CI_JOB_TOKEN` for multi-project pipelines was [introduced][ee-2017] +in [GitLab Premium][ee] 9.3. + +This way of triggering can only be used when invoked inside `.gitlab-ci.yml`, +and it creates a dependent pipeline relation visible on the +[pipeline graph](../multi_project_pipelines.md#overview). For example: + +```yaml +build_docs: + stage: deploy + script: + - curl --request POST --form "token=$CI_JOB_TOKEN" --form ref=master https://gitlab.example.com/api/v4/projects/9/trigger/pipeline + only: + - tags +``` + +Pipelines triggered that way also expose a special variable: +`CI_PIPELINE_SOURCE=pipeline`. + +Read more about the [pipelines trigger API][trigapi]. + +#### When a pipeline depends on the artifacts of another pipeline **[PREMIUM]** + +> The use of `CI_JOB_TOKEN` in the artifacts download API was [introduced][ee-2346] + in [GitLab Premium][ee] 9.5. + +With the introduction of dependencies between different projects, one of +them may need to access artifacts created by a previous one. This process +must be granted for authorized accesses, and it can be done using the +`CI_JOB_TOKEN` variable that identifies a specific job. For example: + +```yaml +build_submodule: + image: debian + stage: test + script: + - apt update && apt install -y unzip + - curl --location --output artifacts.zip "https://gitlab.example.com/api/v4/projects/1/jobs/artifacts/master/download?job=test&job_token=$CI_JOB_TOKEN" + - unzip artifacts.zip + only: + - tags +``` + +This allows you to use that for multi-project pipelines and download artifacts +from any project to which you have access as this follows the same principles +with the [permission model][permissions]. + +Read more about the [jobs API](../../api/jobs.md#download-the-artifacts-archive). + ## Adding a new trigger You can add a new trigger by going to your project's @@ -53,9 +120,6 @@ The action is irreversible. > > - Valid refs are only the branches and tags. If you pass a commit SHA as a ref, > it will not trigger a job. -> - If your project is public, passing the token in plain text is probably not the -> wisest idea, so you might want to use a -> [variable](../variables/README.md#variables) for that purpose. To trigger a job you need to send a `POST` request to GitLab's API endpoint: @@ -219,10 +283,13 @@ Old triggers, created before GitLab 9.0 will be marked as legacy. Triggers with the legacy label do not have an associated user and only have access to the current project. They are considered deprecated and will be removed with one of the future versions of GitLab. You are advised to -[take ownership](#taking-ownership) of any legacy triggers. +[take ownership](#taking-ownership-of-a-trigger) of any legacy triggers. [ee-2017]: https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/2017 +[ee-2346]: https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/2346 [ee]: https://about.gitlab.com/pricing/ [variables]: ../variables/README.md [predef]: ../variables/README.md#predefined-environment-variables [registry]: ../../user/project/container_registry.md +[permissions]: ../../user/permissions.md#job-permissions +[trigapi]: ../../api/pipeline_triggers.md diff --git a/doc/ci/triggers/img/trigger_variables.png b/doc/ci/triggers/img/trigger_variables.png Binary files differindex d273b1fe3a2..910622d74cc 100644 --- a/doc/ci/triggers/img/trigger_variables.png +++ b/doc/ci/triggers/img/trigger_variables.png diff --git a/doc/ci/variables/README.md b/doc/ci/variables/README.md index ca668ac526f..df455857dee 100644 --- a/doc/ci/variables/README.md +++ b/doc/ci/variables/README.md @@ -1,169 +1,291 @@ --- table_display_block: true +type: reference --- -# GitLab CI/CD Variables +# GitLab CI/CD environment variables -When receiving a job from GitLab CI, the [Runner](https://docs.gitlab.com/runner/) prepares the build environment. -It starts by setting a list of: +After a brief overview over the use of environment +variables, this document teaches you how to use GitLab CI/CD's +variables, presents the full reference for predefined variables, +and dives into more advanced applications. -- [Predefined environment variables](#predefined-environment-variables). -- Other variables. +## Overview -## Priority of variables +An environment variable is a dynamic-named value that can +affect the way running processes will behave on an operating +system. -Variables of different types can take precedence over other variables, depending on where they are defined. +They are part of the environment in which a process runs. +For example, a running process can query the value of the +`TEMP` environment variable to discover a suitable location +to store temporary files, or to define a `URL` for a database +that can be reused in different scripts. -The order of precedence for variables is (from highest to lowest): +Variables are useful for customizing your jobs in GitLab +CI/CD's pipelines. Using variables means no hardcoded values. -1. [Trigger variables](../triggers/README.md#pass-job-variables-to-a-trigger) or [scheduled pipeline variables](../../user/project/pipelines/schedules.md#making-use-of-scheduled-pipeline-variables). -1. Project-level [variables](#variables) or [protected variables](#protected-variables). -1. Group-level [variables](#variables) or [protected variables](#protected-variables). -1. YAML-defined [job-level variables](../yaml/README.md#variables). -1. YAML-defined [global variables](../yaml/README.md#variables). -1. [Deployment variables](#deployment-variables). -1. [Predefined environment variables](#predefined-environment-variables). +### Predefined environment variables -For example, you define: +GitLab CI/CD has a default set of +[predefined variables](predefined_variables.md) +which can be used without any specification needed. +You can call issues numbers, user names, branch names, +pipeline and commit IDs, and much more. -- `API_TOKEN=secure` as a project variable. -- `API_TOKEN=yaml` in your `.gitlab-ci.yml`. +Predefined environment variables are the ones that GitLab +provides out of the box for the local environment of the Runner. -`API_TOKEN` will take the value `secure` as the project variables take precedence over those defined -in `.gitlab-ci.yml`. +GitLab reads the .gitlab-ci.yml file, sends the information +to the Runner (which runs the script commands), under which +the variables are exposed. -## Unsupported variables +For example, two jobs under the same pipeline can share the same +`CI_PIPELINE_ID` variable, but each one has its own `CI_JOB_ID` +variable. -There are cases where some variables cannot be used in the context of a -`.gitlab-ci.yml` definition (for example under `script`). Read more -about which variables are [not supported](where_variables_can_be_used.md). +### Custom environment variables -## Predefined environment variables +When your use case requires a specific variable, you can +[set them up easily from the UI](#creating-a-custom-environment-variable) +or directly in the `.gitlab-ci.yml` file and reuse them as you wish. -Some of the predefined environment variables are available only if a minimum -version of [GitLab Runner](https://docs.gitlab.com/runner/) is used. Consult the table below to find the -version of Runner required. +That can be very powerful as it can be used for scripting without +the need to specify the value itself. -NOTE: **Note:** -Starting with GitLab 9.0, we have deprecated some variables. Read the -[9.0 Renaming](#9-0-renaming) section to find out their replacements. **You are -strongly advised to use the new variables as we will remove the old ones in -future GitLab releases.** - -| Variable | GitLab | Runner | Description | -|-------------------------------------------|--------|--------|-------------| -| **ARTIFACT_DOWNLOAD_ATTEMPTS** | 8.15 | 1.9 | Number of attempts to download artifacts running a job | -| **CHAT_INPUT** | 10.6 | all | Additional arguments passed in the [ChatOps](../chatops/README.md) command | -| **CHAT_CHANNEL** | 10.6 | all | Source chat channel which triggered the [ChatOps](../chatops/README.md) command | -| **CI** | all | 0.4 | Mark that job is executed in CI environment | -| **CI_COMMIT_BEFORE_SHA** | 11.2 | all | The previous latest commit present on a branch before a push request. | -| **CI_COMMIT_DESCRIPTION** | 10.8 | all | The description of the commit: the message without first line, if the title is shorter than 100 characters; full message in other case. | -| **CI_COMMIT_MESSAGE** | 10.8 | all | The full commit message. | -| **CI_COMMIT_REF_NAME** | 9.0 | all | The branch or tag name for which project is built | -| **CI_COMMIT_REF_SLUG** | 9.0 | all | `$CI_COMMIT_REF_NAME` lowercased, shortened to 63 bytes, and with everything except `0-9` and `a-z` replaced with `-`. No leading / trailing `-`. Use in URLs, host names and domain names. | -| **CI_COMMIT_SHA** | 9.0 | all | The commit revision for which project is built | -| **CI_COMMIT_SHORT_SHA** | 11.7 | all | The first eight characters of `CI_COMMIT_SHA` | -| **CI_COMMIT_TAG** | 9.0 | 0.5 | The commit tag name. Present only when building tags. | -| **CI_COMMIT_TITLE** | 10.8 | all | The title of the commit - the full first line of the message | -| **CI_CONFIG_PATH** | 9.4 | 0.5 | The path to CI config file. Defaults to `.gitlab-ci.yml` | -| **CI_DEBUG_TRACE** | all | 1.7 | Whether [debug tracing](#debug-tracing) is enabled | -| **CI_DEPLOY_PASSWORD** | 10.8 | all | Authentication password of the [GitLab Deploy Token][gitlab-deploy-token], only present if the Project has one related.| -| **CI_DEPLOY_USER** | 10.8 | all | Authentication username of the [GitLab Deploy Token][gitlab-deploy-token], only present if the Project has one related.| -| **CI_DISPOSABLE_ENVIRONMENT** | all | 10.1 | Marks that the job is executed in a disposable environment (something that is created only for this job and disposed of/destroyed after the execution - all executors except `shell` and `ssh`). If the environment is disposable, it is set to true, otherwise it is not defined at all. | -| **CI_ENVIRONMENT_NAME** | 8.15 | all | The name of the environment for this job. Only present if [`environment:name`](../yaml/README.md#environmentname) is set. | -| **CI_ENVIRONMENT_SLUG** | 8.15 | all | A simplified version of the environment name, suitable for inclusion in DNS, URLs, Kubernetes labels, etc. Only present if [`environment:name`](../yaml/README.md#environmentname) is set. | -| **CI_ENVIRONMENT_URL** | 9.3 | all | The URL of the environment for this job. Only present if [`environment:url`](../yaml/README.md#environmenturl) is set. | -| **CI_JOB_ID** | 9.0 | all | The unique id of the current job that GitLab CI uses internally | -| **CI_JOB_MANUAL** | 8.12 | all | The flag to indicate that job was manually started | -| **CI_JOB_NAME** | 9.0 | 0.5 | The name of the job as defined in `.gitlab-ci.yml` | -| **CI_JOB_STAGE** | 9.0 | 0.5 | The name of the stage as defined in `.gitlab-ci.yml` | -| **CI_JOB_TOKEN** | 9.0 | 1.2 | Token used for authenticating with the [GitLab Container Registry][registry] and downloading [dependent repositories][dependent-repositories] | -| **CI_JOB_URL** | 11.1 | 0.5 | Job details URL | -| **CI_MERGE_REQUEST_ID** | 11.6 | all | The ID of the merge request if it's [pipelines for merge requests](../merge_request_pipelines/index.md) | -| **CI_MERGE_REQUEST_IID** | 11.6 | all | The IID of the merge request if it's [pipelines for merge requests](../merge_request_pipelines/index.md) | -| **CI_MERGE_REQUEST_PROJECT_ID** | 11.6 | all | The ID of the project of the merge request if it's [pipelines for merge requests](../merge_request_pipelines/index.md) | -| **CI_MERGE_REQUEST_PROJECT_PATH** | 11.6 | all | The path of the project of the merge request if it's [pipelines for merge requests](../merge_request_pipelines/index.md) (e.g. `namespace/awesome-project`) | -| **CI_MERGE_REQUEST_PROJECT_URL** | 11.6 | all | The URL of the project of the merge request if it's [pipelines for merge requests](../merge_request_pipelines/index.md) (e.g. `http://192.168.10.15:3000/namespace/awesome-project`) | -| **CI_MERGE_REQUEST_REF_PATH** | 11.6 | all | The ref path of the merge request if it's [pipelines for merge requests](../merge_request_pipelines/index.md). (e.g. `refs/merge-requests/1/head`) | -| **CI_MERGE_REQUEST_SOURCE_BRANCH_NAME** | 11.6 | all | The source branch name of the merge request if it's [pipelines for merge requests](../merge_request_pipelines/index.md) | -| **CI_MERGE_REQUEST_SOURCE_PROJECT_ID** | 11.6 | all | The ID of the source project of the merge request if it's [pipelines for merge requests](../merge_request_pipelines/index.md) | -| **CI_MERGE_REQUEST_SOURCE_PROJECT_PATH** | 11.6 | all | The path of the source project of the merge request if it's [pipelines for merge requests](../merge_request_pipelines/index.md) | -| **CI_MERGE_REQUEST_SOURCE_PROJECT_URL** | 11.6 | all | The URL of the source project of the merge request if it's [pipelines for merge requests](../merge_request_pipelines/index.md) | -| **CI_MERGE_REQUEST_TARGET_BRANCH_NAME** | 11.6 | all | The target branch name of the merge request if it's [pipelines for merge requests](../merge_request_pipelines/index.md) | -| **CI_NODE_INDEX** | 11.5 | all | Index of the job in the job set. If the job is not parallelized, this variable is not set. | -| **CI_NODE_TOTAL** | 11.5 | all | Total number of instances of this job running in parallel. If the job is not parallelized, this variable is set to `1`. | -| **CI_API_V4_URL** | 11.7 | all | The GitLab API v4 root URL | -| **CI_PAGES_DOMAIN** | 11.8 | all | The configured domain that hosts GitLab Pages. | -| **CI_PAGES_URL** | 11.8 | all | URL to GitLab Pages-built pages. Always belongs to a subdomain of `CI_PAGES_DOMAIN`. | -| **CI_PIPELINE_ID** | 8.10 | all | The unique id of the current pipeline that GitLab CI uses internally | -| **CI_PIPELINE_IID** | 11.0 | all | The unique id of the current pipeline scoped to project | -| **CI_PIPELINE_SOURCE** | 10.0 | all | Indicates how the pipeline was triggered. Possible options are: `push`, `web`, `trigger`, `schedule`, `api`, and `pipeline`. For pipelines created before GitLab 9.5, this will show as `unknown` | -| **CI_PIPELINE_TRIGGERED** | all | all | The flag to indicate that job was [triggered] | -| **CI_PIPELINE_URL** | 11.1 | 0.5 | Pipeline details URL | -| **CI_PROJECT_DIR** | all | all | The full path where the repository is cloned and where the job is run. If the GitLab Runner `builds_dir` parameter is set, this variable is set relative to the value of `builds_dir`. For more information, see [Advanced configuration](https://docs.gitlab.com/runner/configuration/advanced-configuration.html#the-runners-section) for GitLab Runner. | -| **CI_PROJECT_ID** | all | all | The unique id of the current project that GitLab CI uses internally | -| **CI_PROJECT_NAME** | 8.10 | 0.5 | The project name that is currently being built (actually it is project folder name) | -| **CI_PROJECT_NAMESPACE** | 8.10 | 0.5 | The project namespace (username or groupname) that is currently being built | -| **CI_PROJECT_PATH** | 8.10 | 0.5 | The namespace with project name | -| **CI_PROJECT_PATH_SLUG** | 9.3 | all | `$CI_PROJECT_PATH` lowercased and with everything except `0-9` and `a-z` replaced with `-`. Use in URLs and domain names. | -| **CI_PROJECT_URL** | 8.10 | 0.5 | The HTTP(S) address to access project | -| **CI_PROJECT_VISIBILITY** | 10.3 | all | The project visibility (internal, private, public) | -| **CI_REGISTRY** | 8.10 | 0.5 | If the Container Registry is enabled it returns the address of GitLab's Container Registry | -| **CI_REGISTRY_IMAGE** | 8.10 | 0.5 | If the Container Registry is enabled for the project it returns the address of the registry tied to the specific project | -| **CI_REGISTRY_PASSWORD** | 9.0 | all | The password to use to push containers to the GitLab Container Registry | -| **CI_REGISTRY_USER** | 9.0 | all | The username to use to push containers to the GitLab Container Registry | -| **CI_REPOSITORY_URL** | 9.0 | all | The URL to clone the Git repository | -| **CI_RUNNER_DESCRIPTION** | 8.10 | 0.5 | The description of the runner as saved in GitLab | -| **CI_RUNNER_EXECUTABLE_ARCH** | all | 10.6 | The OS/architecture of the GitLab Runner executable (note that this is not necessarily the same as the environment of the executor) | -| **CI_RUNNER_ID** | 8.10 | 0.5 | The unique id of runner being used | -| **CI_RUNNER_REVISION** | all | 10.6 | GitLab Runner revision that is executing the current job | -| **CI_RUNNER_TAGS** | 8.10 | 0.5 | The defined runner tags | -| **CI_RUNNER_VERSION** | all | 10.6 | GitLab Runner version that is executing the current job | -| **CI_SERVER** | all | all | Mark that job is executed in CI environment | -| **CI_SERVER_NAME** | all | all | The name of CI server that is used to coordinate jobs | -| **CI_SERVER_REVISION** | all | all | GitLab revision that is used to schedule jobs | -| **CI_SERVER_VERSION** | all | all | GitLab version that is used to schedule jobs | -| **CI_SERVER_VERSION_MAJOR** | 11.4 | all | GitLab version major component | -| **CI_SERVER_VERSION_MINOR** | 11.4 | all | GitLab version minor component | -| **CI_SERVER_VERSION_PATCH** | 11.4 | all | GitLab version patch component | -| **CI_SHARED_ENVIRONMENT** | all | 10.1 | Marks that the job is executed in a shared environment (something that is persisted across CI invocations like `shell` or `ssh` executor). If the environment is shared, it is set to true, otherwise it is not defined at all. | -| **GET_SOURCES_ATTEMPTS** | 8.15 | 1.9 | Number of attempts to fetch sources running a job | -| **GITLAB_CI** | all | all | Mark that job is executed in GitLab CI environment | -| **GITLAB_USER_EMAIL** | 8.12 | all | The email of the user who started the job | -| **GITLAB_USER_ID** | 8.12 | all | The id of the user who started the job | -| **GITLAB_USER_LOGIN** | 10.0 | all | The login username of the user who started the job | -| **GITLAB_USER_NAME** | 10.0 | all | The real name of the user who started the job | -| **RESTORE_CACHE_ATTEMPTS** | 8.15 | 1.9 | Number of attempts to restore the cache running a job | - -## GitLab 9.0 renaming - -To follow conventions of naming across GitLab, and to further move away from the -`build` term and toward `job` CI variables have been renamed for the 9.0 -release. +#### Variable types + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/issues/46806) in GitLab 11.11. + +There are two types of variables supported by GitLab: + +- "Variable": the Runner will create an environment variable named same as the variable key and set its value to the variable value. +- "File": the Runner will write the variable value to a temporary file and set the path to this file as the value of an environment variable named same as the variable key. + +Many tools (like [AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-envvars.html) and [kubectl](https://kubernetes.io/docs/concepts/configuration/organize-cluster-access-kubeconfig/#the-kubeconfig-environment-variable)) provide the ability to customise configuration using files by either providing the file path as a command line argument or an environment variable. Prior to the introduction of variable types, the common pattern was to use the value of a CI variable, save it in a file, and then use the newly created file in your script: + +```bash +# Save the content of variable in a file +echo "$KUBE_CA_PEM" > "$(pwd)/kube.ca.pem" + # Use the newly created file +kubectl config set-cluster e2e --server="$KUBE_URL" --certificate-authority="$(pwd)/kube.ca.pem" +``` + +This can be simplified by creating a variable of type "File" and using it directly. For example, let's say we have the following variables. + + + +We can then call them from `.gitlab-ci.yml` like this: + +```bash +kubectl config set-cluster e2e --server="$KUBE_URL" --certificate-authority="$KUBE_CA_PEM" +``` + +Variable types can be set via the [UI](#via-the-ui) or the [API](../../api/project_level_variables.md#create-variable), but not in `.gitlab-ci.yml`. + +#### Masked variables + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/issues/13784) in GitLab 11.10 + +Variables can be created as masked variables. +This means that the value of the variable will be hidden in job logs, +though it must match certain requirements to do so: + +- The value must be in a single line. +- The value must only consist of characters from the Base64 alphabet, defined in [RFC4648](https://tools.ietf.org/html/rfc4648). +- The value must be at least 8 characters long. +- The value must not use variables. + +If the value does not meet the requirements above, then the CI variable will fail to save. +In order to save, either alter the value to meet the masking requirements +or disable **Masked** for the variable. + +## Getting started + +To get started with environment variables in the scope of GitLab +CI/CD, let's go over a few examples. + +### Using predefined environment variables + +To get started, choose one of the existing +[predefined variables](predefined_variables.md) +to be output by the Runner. For example, let's say that you want +a given job you're running through your script to output the +stage that job is running for. In your `.gitlab-ci.yml` file, +call the variable from your script according to the [syntaxes](#syntax-of-environment-variables-in-job-scripts) available. To +output the job stage, use the predefined variable `CI_JOB_STAGE`: + +```yaml +test_variable: + stage: test + script: + - echo $CI_JOB_STAGE +``` + +For this case, the Runner will output the `stage` for the +job `test_variable`, which is `test`: + + + +As another example, let's say you're using your own GitLab +instance you want to know what domain your GitLab Pages are +served under. You can easily call it with the predefined +variable `$CI_PAGES_DOMAIN` in your script: + +```yaml +pages: + script: + - ... + - echo $CI_PAGES_DOMAIN +``` + +For GitLab.com users, the output will be `gitlab.io`. For your +private instance, the output will be whatever your sysadmin has +defined. + +### Creating a custom environment variable + +Assume you have something you want to repeat through your scripts +in GitLab CI/CD's configuration file. To keep this example simple, +let's say you want to output `HELLO WORLD` for a `TEST` variable. + +You can either set the variable directly in the `.gitlab-ci.yml` +file or through the UI. + +#### Via `.gitlab-ci.yml` + +To create a new custom `env_var` variable via [`.gitlab-ci.yml`](../yaml/README.md#variables), define their variable/value pair under +`variables`: + +```yaml +variables: + TEST: "HELLO WORLD" +``` + +For a deeper look into them, see [`.gitlab-ci.yml` defined variables](#gitlab-ciyml-defined-variables). + +#### Via the UI + +From the UI, navigate to your project's **Settings > CI/CD** and +expand **Variables**. Create a new variable by choosing its **type**, naming +it in the field **Input variable key**, and defining its value in the +**Input variable value** field: + + + +You'll also see the option to mask and/or protect your variables. + +Once you've set the variables, call them from the `.gitlab-ci.yml` file: + +```yaml +test_variable: + stage: test + script: + - echo $CI_JOB_STAGE # calls a predefined variable + - echo $TEST # calls a custom variable of type `env_var` + - echo $GREETING # calls a custom variable of type `file` that contains the path to the temp file + - cat $GREETING # the temp file itself contains the variable value +``` + +The output will be: + + + +### Syntax of environment variables in job scripts + +All variables are set as environment variables in the build environment, and +they are accessible with normal methods that are used to access such variables. +In most cases `bash` or `sh` is used to execute the job script. + +To access environment variables, use the syntax for your Runner's [shell][shellexecutors]. + +| Shell | Usage | +|----------------------|-----------------| +| bash/sh | `$variable` | +| windows batch | `%variable%` | +| PowerShell | `$env:variable` | + +To access environment variables in bash, prefix the variable name with (`$`): + +```yaml +job_name: + script: + - echo $CI_JOB_ID +``` + +To access environment variables in **Windows Batch**, surround the variable +with (`%`): + +```yaml +job_name: + script: + - echo %CI_JOB_ID% +``` + +To access environment variables in a **Windows PowerShell** environment, prefix +the variable name with (`$env:`): + +```yaml +job_name: + script: + - echo $env:CI_JOB_ID +``` + +You can also list all environment variables with the `export` command, +but be aware that this will also expose the values of all the variables +you set, in the job log: + +```yaml +job_name: + script: + - export +``` + +Example values: + +```bash +export CI_JOB_ID="50" +export CI_COMMIT_SHA="1ecfd275763eff1d6b4844ea3168962458c9f27a" +export CI_COMMIT_SHORT_SHA="1ecfd275" +export CI_COMMIT_REF_NAME="master" +export CI_REPOSITORY_URL="https://gitlab-ci-token:abcde-1234ABCD5678ef@example.com/gitlab-org/gitlab-ce.git" +export CI_COMMIT_TAG="1.0.0" +export CI_JOB_NAME="spec:other" +export CI_JOB_STAGE="test" +export CI_JOB_MANUAL="true" +export CI_JOB_TRIGGERED="true" +export CI_JOB_TOKEN="abcde-1234ABCD5678ef" +export CI_PIPELINE_ID="1000" +export CI_PIPELINE_IID="10" +export CI_PAGES_DOMAIN="gitlab.io" +export CI_PAGES_URL="https://gitlab-org.gitlab.io/gitlab-ce" +export CI_PROJECT_ID="34" +export CI_PROJECT_DIR="/builds/gitlab-org/gitlab-ce" +export CI_PROJECT_NAME="gitlab-ce" +export CI_PROJECT_NAMESPACE="gitlab-org" +export CI_PROJECT_PATH="gitlab-org/gitlab-ce" +export CI_PROJECT_URL="https://example.com/gitlab-org/gitlab-ce" +export CI_REGISTRY="registry.example.com" +export CI_REGISTRY_IMAGE="registry.example.com/gitlab-org/gitlab-ce" +export CI_RUNNER_ID="10" +export CI_RUNNER_DESCRIPTION="my runner" +export CI_RUNNER_TAGS="docker, linux" +export CI_SERVER="yes" +export CI_SERVER_NAME="GitLab" +export CI_SERVER_REVISION="70606bf" +export CI_SERVER_VERSION="8.9.0" +export CI_SERVER_VERSION_MAJOR="8" +export CI_SERVER_VERSION_MINOR="9" +export CI_SERVER_VERSION_PATCH="0" +export GITLAB_USER_ID="42" +export GITLAB_USER_EMAIL="user@example.com" +export CI_REGISTRY_USER="gitlab-ci-token" +export CI_REGISTRY_PASSWORD="longalfanumstring" +``` + +### `.gitlab-ci.yml` defined variables NOTE: **Note:** -Starting with GitLab 9.0, we have deprecated the `$CI_BUILD_*` variables. **You are -strongly advised to use the new variables as we will remove the old ones in -future GitLab releases.** - -| 8.x name | 9.0+ name | -| --------------------- |------------------------ | -| `CI_BUILD_ID` | `CI_JOB_ID` | -| `CI_BUILD_REF` | `CI_COMMIT_SHA` | -| `CI_BUILD_TAG` | `CI_COMMIT_TAG` | -| `CI_BUILD_BEFORE_SHA` | `CI_COMMIT_BEFORE_SHA` | -| `CI_BUILD_REF_NAME` | `CI_COMMIT_REF_NAME` | -| `CI_BUILD_REF_SLUG` | `CI_COMMIT_REF_SLUG` | -| `CI_BUILD_NAME` | `CI_JOB_NAME` | -| `CI_BUILD_STAGE` | `CI_JOB_STAGE` | -| `CI_BUILD_REPO` | `CI_REPOSITORY_URL` | -| `CI_BUILD_TRIGGERED` | `CI_PIPELINE_TRIGGERED` | -| `CI_BUILD_MANUAL` | `CI_JOB_MANUAL` | -| `CI_BUILD_TOKEN` | `CI_JOB_TOKEN` | - -## `.gitlab-ci.yml` defined variables - -NOTE **Note:** This feature requires GitLab Runner 0.5.0 or higher and GitLab 7.14 or higher. GitLab CI allows you to add to `.gitlab-ci.yml` variables that are set in the @@ -201,42 +323,60 @@ script: - 'eval $LS_CMD' # will execute 'ls -al $TMP_DIR' ``` -## Variables +### Group-level environment variables -> Group-level variables were introduced in GitLab 9.4. +> Introduced in GitLab 9.4. -CAUTION: **Important:** -Be aware that variables are not masked, and their values can be shown -in the job logs if explicitly asked to do so. If your project is public or -internal, you can set the pipelines private from your [project's Pipelines -settings](../../user/project/pipelines/settings.md#visibility-of-pipelines). -Follow the discussion in issue [#13784][ce-13784] for masking the variables. - -GitLab CI allows you to define per-project or per-group variables -that are set in the pipeline environment. The variables are stored out of +GitLab CI/CD allows you to define per-project or per-group variables +that are set in the pipeline environment. Group-level variables are stored out of the repository (not in `.gitlab-ci.yml`) and are securely passed to GitLab Runner -making them available during a pipeline run. It's the recommended method to +making them available during a pipeline run. It's the **recommended method** to use for storing things like passwords, SSH keys, and credentials. -Project-level variables can be added by: - -1. Navigating to your project's **Settings > CI/CD** page. -1. Inputing variable keys and values in the **Environment variables** section. - Group-level variables can be added by: 1. Navigating to your group's **Settings > CI/CD** page. -1. Inputing variable keys and values in the **Environment variables** section. Any variables of - [subgroups](../../user/group/subgroups/index.md) will be inherited recursively. +1. Inputing variable types, keys, and values in the **Variables** section. +Any variables of [subgroups](../../user/group/subgroups/index.md) will be inherited recursively. + +Once you set them, they will be available for all subsequent pipelines. -Once you set them, they will be available for all subsequent pipelines. You can also -[protect your variables](#protected-variables). +## Priority of environment variables -### Protected variables +Variables of different types can take precedence over other +variables, depending on where they are defined. + +The order of precedence for variables is (from highest to lowest): + +1. [Trigger variables](../triggers/README.md#making-use-of-trigger-variables) or [scheduled pipeline variables](../../user/project/pipelines/schedules.md#using-variables). +1. Project-level [variables](#creating-a-custom-environment-variable) or [protected variables](#protected-environment-variables). +1. Group-level [variables](#group-level-environment-variables) or [protected variables](#protected-environment-variables). +1. YAML-defined [job-level variables](../yaml/README.md#variables). +1. YAML-defined [global variables](../yaml/README.md#variables). +1. [Deployment variables](#deployment-environment-variables). +1. [Predefined environment variables](predefined_variables.md). + +For example, if you define: + +- `API_TOKEN=secure` as a project variable. +- `API_TOKEN=yaml` in your `.gitlab-ci.yml`. + +`API_TOKEN` will take the value `secure` as the project +variables take precedence over those defined in `.gitlab-ci.yml`. + +## Unsupported variables + +There are cases where some variables cannot be used in the context of a +`.gitlab-ci.yml` definition (for example under `script`). Read more +about which variables are [not supported](where_variables_can_be_used.md). + +## Advanced use + +### Protected environment variables > Introduced in GitLab 9.3. -Variables could be protected. Whenever a variable is +Variables can be protected. Whenever a variable is protected, it would only be securely passed to pipelines running on the [protected branches] or [protected tags]. The other pipelines would not get any protected variables. @@ -247,13 +387,14 @@ Protected variables can be added by going to your project's Once you set them, they will be available for all subsequent pipelines. -### Manually-specified variables +### Limiting environment scopes of environment variables **[PREMIUM]** -> [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/issues/44059) in GitLab 10.8. +You can limit the environment scope of a variable by +[defining which environments][envs] it can be available for. -Variables can be specified for a single pipeline run when a [manual pipeline](../pipelines.md#manually-executing-pipelines) is created. +To learn more about about scoping environments, see [Scoping environments with specs](../environments.md#scoping-environments-with-specs-premium). -## Deployment variables +### Deployment environment variables > Introduced in GitLab 8.15. @@ -266,7 +407,7 @@ the project services that you are using to learn which variables they define. An example project service that defines deployment variables is the [Kubernetes integration](../../user/project/clusters/index.md#deployment-variables). -## Auto DevOps application variables +### Auto DevOps environment variables > [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/issues/49056) in GitLab 11.7. @@ -283,6 +424,136 @@ CAUTION: **Caution:** Variables with multiline values are not currently supported due to limitations with the current Auto DevOps scripting environment. +### Environment variables triggered manually + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/issues/44059) in GitLab 10.8. + +[Manually triggered pipelines](../pipelines.md#manually-executing-pipelines) allow you to override the value of a current variable. + +For instance, suppose you added a +[custom variable `$TEST`](#creating-a-custom-environment-variable) +as exemplified above and you want to override it in a manual pipeline. +Navigate to your project's **CI/CD > Pipelines** and click **Run pipeline**. +Choose the branch you want to run the pipeline for, then add a new variable through the UI: + + + +The Runner will override the value previously set and use the custom +value you set for this specific pipeline: + + + +## Environment variables expressions + +> Introduced in GitLab 10.7. + +It is possible to use variables expressions with only / except policies in +`.gitlab-ci.yml`. By using this approach you can limit what jobs are going to +be created within a pipeline after pushing a code to GitLab. + +This is particularly useful in combination with variables and triggered +pipeline variables. + +```yaml +deploy: + script: cap staging deploy + environment: staging + only: + variables: + - $RELEASE == "staging" + - $STAGING +``` + +Each expression provided is going to be evaluated before creating a pipeline. + +If any of the conditions in `variables` evaluates to truth when using `only`, +a new job is going to be created. If any of the expressions evaluates to truth +when `except` is being used, a job is not going to be created. + +This follows usual rules for [`only` / `except` policies](../yaml/README.md#onlyexcept-advanced). + +### Supported syntax + +Below you can find supported syntax reference: + +1. Equality matching using a string + + > Example: `$VARIABLE == "some value"` + + > Example: `$VARIABLE != "some value"` _(added in 11.11)_ + + You can use equality operator `==` or `!=` to compare a variable content to a + string. We support both, double quotes and single quotes to define a string + value, so both `$VARIABLE == "some value"` and `$VARIABLE == 'some value'` + are supported. `"some value" == $VARIABLE` is correct too. + +1. Checking for an undefined value + + > Example: `$VARIABLE == null` + + > Example: `$VARIABLE != null` _(added in 11.11)_ + + It sometimes happens that you want to check whether a variable is defined + or not. To do that, you can compare a variable to `null` keyword, like + `$VARIABLE == null`. This expression is going to evaluate to truth if + variable is not defined when `==` is used, or to falsey if `!=` is used. + +1. Checking for an empty variable + + > Example: `$VARIABLE == ""` + + > Example: `$VARIABLE != ""` _(added in 11.11)_ + + If you want to check whether a variable is defined, but is empty, you can + simply compare it against an empty string, like `$VAR == ''` or non-empty + string `$VARIABLE != ""`. + +1. Comparing two variables + + > Example: `$VARIABLE_1 == $VARIABLE_2` + + > Example: `$VARIABLE_1 != $VARIABLE_2` _(added in 11.11)_ + + It is possible to compare two variables. This is going to compare values + of these variables. + +1. Variable presence check + + > Example: `$STAGING` + + If you only want to create a job when there is some variable present, + which means that it is defined and non-empty, you can simply use + variable name as an expression, like `$STAGING`. If `$STAGING` variable + is defined, and is non empty, expression will evaluate to truth. + `$STAGING` value needs to a string, with length higher than zero. + Variable that contains only whitespace characters is not an empty variable. + +1. Pattern matching _(added in 11.0)_ + + > Example: `$VARIABLE =~ /^content.*/` + + > Example: `$VARIABLE_1 !~ /^content.*/` _(added in 11.11)_ + + It is possible perform pattern matching against a variable and regular + expression. Expression like this evaluates to truth if matches are found + when using `=~`. It evaluates to truth if matches are not found when `!~` is used. + + Pattern matching is case-sensitive by default. Use `i` flag modifier, like + `/pattern/i` to make a pattern case-insensitive. + +1. Conjunction / Disjunction + + > Example: `$VARIABLE1 =~ /^content.*/ && $VARIABLE2 == "something"` + + > Example: `$VARIABLE1 =~ /^content.*/ && $VARIABLE2 =~ /thing$/ && $VARIABLE3` + + > Example: `$VARIABLE1 =~ /^content.*/ || $VARIABLE2 =~ /thing$/ && $VARIABLE3` + + It is possible to join multiple conditions using `&&` or `||`. Any of the otherwise + supported syntax may be used in a conjunctive or disjunctive statement. + Precedence of operators follows standard Ruby 2.5 operation + [precedence](https://ruby-doc.org/core-2.5.0/doc/syntax/precedence_rdoc.html). + ## Debug tracing > Introduced in GitLab Runner 1.7. @@ -305,7 +576,7 @@ variables that were set, etc. Before enabling this, you should ensure jobs are visible to [team members only](../../user/permissions.md#project-features). You should -also [erase](../pipelines.md#seeing-build-status) all generated job traces +also [erase](../pipelines.md#accessing-individual-jobs) all generated job traces before making them visible again. To enable debug traces, set the `CI_DEBUG_TRACE` variable to `true`: @@ -347,6 +618,8 @@ $'\''git'\'' "checkout" "-f" "-q" "dd648b2e48ce6518303b0bb580b2ee32fadaf045" Running on runner-8a2f473d-project-1796893-concurrent-0 via runner-8a2f473d-machine-1480971377-317a7d0f-digital-ocean-4gb... ++ export CI=true ++ CI=true +++ export CI_API_V4_URL=https://example.com:3000/api/v4 +++ CI_API_V4_URL=https://example.com:3000/api/v4 ++ export CI_DEBUG_TRACE=false ++ CI_DEBUG_TRACE=false ++ export CI_COMMIT_SHA=dd648b2e48ce6518303b0bb580b2ee32fadaf045 @@ -385,6 +658,8 @@ Running on runner-8a2f473d-project-1796893-concurrent-0 via runner-8a2f473d-mach ++ GITLAB_CI=true ++ export CI=true ++ CI=true +++ export CI_API_V4_URL=https://example.com:3000/api/v4 +++ CI_API_V4_URL=https://example.com:3000/api/v4 ++ export GITLAB_CI=true ++ GITLAB_CI=true ++ export CI_JOB_ID=7046507 @@ -448,192 +723,14 @@ MIIFQzCCBCugAwIBAgIRAL/ElDjuf15xwja1ZnCocWAwDQYJKoZIhvcNAQELBQAw' ... ``` -## Using the CI variables in your job scripts - -All variables are set as environment variables in the build environment, and -they are accessible with normal methods that are used to access such variables. -In most cases `bash` or `sh` is used to execute the job script. - -To access environment variables, use the syntax for your Runner's [shell][shellexecutors]. - -| Shell | Usage | -|----------------------|-----------------| -| bash/sh | `$variable` | -| windows batch | `%variable%` | -| PowerShell | `$env:variable` | - -To access environment variables in bash, prefix the variable name with (`$`): - -```yaml -job_name: - script: - - echo $CI_JOB_ID -``` - -To access environment variables in **Windows Batch**, surround the variable -with (`%`): - -```yaml -job_name: - script: - - echo %CI_JOB_ID% -``` - -To access environment variables in a **Windows PowerShell** environment, prefix -the variable name with (`$env:`): - -```yaml -job_name: - script: - - echo $env:CI_JOB_ID -``` - -You can also list all environment variables with the `export` command, -but be aware that this will also expose the values of all the variables -you set, in the job log: - -```yaml -job_name: - script: - - export -``` - -Example values: - -```bash -export CI_JOB_ID="50" -export CI_COMMIT_SHA="1ecfd275763eff1d6b4844ea3168962458c9f27a" -export CI_COMMIT_SHORT_SHA="1ecfd275" -export CI_COMMIT_REF_NAME="master" -export CI_REPOSITORY_URL="https://gitlab-ci-token:abcde-1234ABCD5678ef@example.com/gitlab-org/gitlab-ce.git" -export CI_COMMIT_TAG="1.0.0" -export CI_JOB_NAME="spec:other" -export CI_JOB_STAGE="test" -export CI_JOB_MANUAL="true" -export CI_JOB_TRIGGERED="true" -export CI_JOB_TOKEN="abcde-1234ABCD5678ef" -export CI_PIPELINE_ID="1000" -export CI_PIPELINE_IID="10" -export CI_PAGES_DOMAIN="gitlab.io" -export CI_PAGES_URL="https://gitlab-org.gitlab.io/gitlab-ce" -export CI_PROJECT_ID="34" -export CI_PROJECT_DIR="/builds/gitlab-org/gitlab-ce" -export CI_PROJECT_NAME="gitlab-ce" -export CI_PROJECT_NAMESPACE="gitlab-org" -export CI_PROJECT_PATH="gitlab-org/gitlab-ce" -export CI_PROJECT_URL="https://example.com/gitlab-org/gitlab-ce" -export CI_REGISTRY="registry.example.com" -export CI_REGISTRY_IMAGE="registry.example.com/gitlab-org/gitlab-ce" -export CI_RUNNER_ID="10" -export CI_RUNNER_DESCRIPTION="my runner" -export CI_RUNNER_TAGS="docker, linux" -export CI_SERVER="yes" -export CI_SERVER_NAME="GitLab" -export CI_SERVER_REVISION="70606bf" -export CI_SERVER_VERSION="8.9.0" -export CI_SERVER_VERSION_MAJOR="8" -export CI_SERVER_VERSION_MINOR="9" -export CI_SERVER_VERSION_PATCH="0" -export GITLAB_USER_ID="42" -export GITLAB_USER_EMAIL="user@example.com" -export CI_REGISTRY_USER="gitlab-ci-token" -export CI_REGISTRY_PASSWORD="longalfanumstring" -``` - -## Variables expressions - -> Introduced in GitLab 10.7. - -It is possible to use variables expressions with only / except policies in -`.gitlab-ci.yml`. By using this approach you can limit what jobs are going to -be created within a pipeline after pushing a code to GitLab. - -This is particularly useful in combination with variables and triggered -pipeline variables. - -```yaml -deploy: - script: cap staging deploy - environment: staging - only: - variables: - - $RELEASE == "staging" - - $STAGING -``` - -Each expression provided is going to be evaluated before creating a pipeline. - -If any of the conditions in `variables` evaluates to truth when using `only`, -a new job is going to be created. If any of the expressions evaluates to truth -when `except` is being used, a job is not going to be created. - -This follows usual rules for [`only` / `except` policies][builds-policies]. - -### Supported syntax - -Below you can find supported syntax reference: - -1. Equality matching using a string - - > Example: `$VARIABLE == "some value"` - - You can use equality operator `==` to compare a variable content to a - string. We support both, double quotes and single quotes to define a string - value, so both `$VARIABLE == "some value"` and `$VARIABLE == 'some value'` - are supported. `"some value" == $VARIABLE` is correct too. - -1. Checking for an undefined value - - > Example: `$VARIABLE == null` - - It sometimes happens that you want to check whether a variable is defined - or not. To do that, you can compare a variable to `null` keyword, like - `$VARIABLE == null`. This expression is going to evaluate to truth if - variable is not defined. - -1. Checking for an empty variable - - > Example: `$VARIABLE == ""` - - If you want to check whether a variable is defined, but is empty, you can - simply compare it against an empty string, like `$VAR == ''`. - -1. Comparing two variables - - > Example: `$VARIABLE_1 == $VARIABLE_2` - - It is possible to compare two variables. This is going to compare values - of these variables. - -1. Variable presence check - - > Example: `$STAGING` - - If you only want to create a job when there is some variable present, - which means that it is defined and non-empty, you can simply use - variable name as an expression, like `$STAGING`. If `$STAGING` variable - is defined, and is non empty, expression will evaluate to truth. - `$STAGING` value needs to a string, with length higher than zero. - Variable that contains only whitespace characters is not an empty variable. - -1. Pattern matching _(added in 11.0)_ - - > Example: `$VARIABLE =~ /^content.*/` - - It is possible perform pattern matching against a variable and regular - expression. Expression like this evaluates to truth if matches are found. - - Pattern matching is case-sensitive by default. Use `i` flag modifier, like - `/pattern/i` to make a pattern case-insensitive. - [ce-13784]: https://gitlab.com/gitlab-org/gitlab-ce/issues/13784 "Simple protection of CI variables" -[eep]: https://about.gitlab.com/pricing/ "Available only in GitLab Premium" [envs]: ../environments.md [protected branches]: ../../user/project/protected_branches.md [protected tags]: ../../user/project/protected_tags.md [shellexecutors]: https://docs.gitlab.com/runner/executors/ [triggered]: ../triggers/README.md -[builds-policies]: ../yaml/README.md#only-and-except-complex +[trigger-job-token]: ../triggers/README.md#ci-job-token [gitlab-deploy-token]: ../../user/project/deploy_tokens/index.md#gitlab-deploy-token [registry]: ../../user/project/container_registry.md [dependent-repositories]: ../../user/project/new_ci_build_permissions_model.md#dependent-repositories +[get-job-artifacts]: ../../api/jobs.html#get-job-artifacts diff --git a/doc/ci/variables/deprecated_variables.md b/doc/ci/variables/deprecated_variables.md new file mode 100644 index 00000000000..cdca5bf27fc --- /dev/null +++ b/doc/ci/variables/deprecated_variables.md @@ -0,0 +1,34 @@ +--- +type: reference +--- + +# Deprecated GitLab CI/CD variables + +Read through this document to learn what predefined variables +were deprecated and their new references. + +## GitLab 9.0 renamed variables + +To follow conventions of naming across GitLab, and to further move away from the +`build` term and toward `job`, some [CI/CD environment variables](README.md#predefined-environment-variables) were renamed for GitLab 9.0 +release. + +NOTE: **Note:** +Starting with GitLab 9.0, we have deprecated the `$CI_BUILD_*` variables. **You are +strongly advised to use the new variables as we will remove the old ones in +future GitLab releases.** + +| 8.x name | 9.0+ name | +| --------------------- |------------------------ | +| `CI_BUILD_ID` | `CI_JOB_ID` | +| `CI_BUILD_REF` | `CI_COMMIT_SHA` | +| `CI_BUILD_TAG` | `CI_COMMIT_TAG` | +| `CI_BUILD_BEFORE_SHA` | `CI_COMMIT_BEFORE_SHA` | +| `CI_BUILD_REF_NAME` | `CI_COMMIT_REF_NAME` | +| `CI_BUILD_REF_SLUG` | `CI_COMMIT_REF_SLUG` | +| `CI_BUILD_NAME` | `CI_JOB_NAME` | +| `CI_BUILD_STAGE` | `CI_JOB_STAGE` | +| `CI_BUILD_REPO` | `CI_REPOSITORY_URL` | +| `CI_BUILD_TRIGGERED` | `CI_PIPELINE_TRIGGERED` | +| `CI_BUILD_MANUAL` | `CI_JOB_MANUAL` | +| `CI_BUILD_TOKEN` | `CI_JOB_TOKEN` | diff --git a/doc/ci/variables/img/ci_job_stage_output_example.png b/doc/ci/variables/img/ci_job_stage_output_example.png Binary files differnew file mode 100644 index 00000000000..056238d5693 --- /dev/null +++ b/doc/ci/variables/img/ci_job_stage_output_example.png diff --git a/doc/ci/variables/img/custom_variables_output.png b/doc/ci/variables/img/custom_variables_output.png Binary files differnew file mode 100644 index 00000000000..29f5c63b3d9 --- /dev/null +++ b/doc/ci/variables/img/custom_variables_output.png diff --git a/doc/ci/variables/img/new_custom_variables_example.png b/doc/ci/variables/img/new_custom_variables_example.png Binary files differnew file mode 100644 index 00000000000..efe104efe4c --- /dev/null +++ b/doc/ci/variables/img/new_custom_variables_example.png diff --git a/doc/ci/variables/img/override_value_via_manual_pipeline_output.png b/doc/ci/variables/img/override_value_via_manual_pipeline_output.png Binary files differnew file mode 100644 index 00000000000..02369d57fb8 --- /dev/null +++ b/doc/ci/variables/img/override_value_via_manual_pipeline_output.png diff --git a/doc/ci/variables/img/override_variable_manual_pipeline.png b/doc/ci/variables/img/override_variable_manual_pipeline.png Binary files differnew file mode 100644 index 00000000000..3c8c59720cf --- /dev/null +++ b/doc/ci/variables/img/override_variable_manual_pipeline.png diff --git a/doc/ci/variables/img/variable_types_usage_example.png b/doc/ci/variables/img/variable_types_usage_example.png Binary files differnew file mode 100644 index 00000000000..0e8bde891fe --- /dev/null +++ b/doc/ci/variables/img/variable_types_usage_example.png diff --git a/doc/ci/variables/predefined_variables.md b/doc/ci/variables/predefined_variables.md new file mode 100644 index 00000000000..4655eec51de --- /dev/null +++ b/doc/ci/variables/predefined_variables.md @@ -0,0 +1,122 @@ +--- +type: reference +--- + +# Predefined environment variables reference + +For an introduction on this subject, read through the +[getting started with environment variables](README.md) document. + +## Overview + +Some of the predefined environment variables are available only if a minimum +version of [GitLab Runner](https://docs.gitlab.com/runner/) is used. Consult the table below to find the +version of Runner required. + +NOTE: **Note:** +Starting with GitLab 9.0, we have deprecated some variables. Read the +[9.0 Renaming](deprecated_variables.md#gitlab-90-renamed-variables) section to find out their replacements. **You are +strongly advised to use the new variables as we will remove the old ones in +future GitLab releases.** + +## Variables reference + +| Variable | GitLab | Runner | Description | +|-------------------------------------------|--------|--------|-------------| +| `ARTIFACT_DOWNLOAD_ATTEMPTS` | 8.15 | 1.9 | Number of attempts to download artifacts running a job | +| `CHAT_INPUT` | 10.6 | all | Additional arguments passed in the [ChatOps](../chatops/README.md) command | +| `CHAT_CHANNEL` | 10.6 | all | Source chat channel which triggered the [ChatOps](../chatops/README.md) command | +| `CI` | all | 0.4 | Mark that job is executed in CI environment | +| `CI_BUILDS_DIR` | all | 11.10 | Top-level directory where builds are executed. | +| `CI_CONCURRENT_ID` | all | 11.10 | Unique ID of build execution within a single executor. | +| `CI_CONCURRENT_PROJECT_ID` | all | 11.10 | Unique ID of build execution within a single executor and project. | +| `CI_COMMIT_BEFORE_SHA` | 11.2 | all | The previous latest commit present on a branch before a push request. | +| `CI_COMMIT_DESCRIPTION` | 10.8 | all | The description of the commit: the message without first line, if the title is shorter than 100 characters; full message in other case. | +| `CI_COMMIT_MESSAGE` | 10.8 | all | The full commit message. | +| `CI_COMMIT_REF_NAME` | 9.0 | all | The branch or tag name for which project is built | +| `CI_COMMIT_REF_SLUG` | 9.0 | all | `$CI_COMMIT_REF_NAME` lowercased, shortened to 63 bytes, and with everything except `0-9` and `a-z` replaced with `-`. No leading / trailing `-`. Use in URLs, host names and domain names. | +| `CI_COMMIT_SHA` | 9.0 | all | The commit revision for which project is built | +| `CI_COMMIT_SHORT_SHA` | 11.7 | all | The first eight characters of `CI_COMMIT_SHA` | +| `CI_COMMIT_TAG` | 9.0 | 0.5 | The commit tag name. Present only when building tags. | +| `CI_COMMIT_TITLE` | 10.8 | all | The title of the commit - the full first line of the message | +| `CI_CONFIG_PATH` | 9.4 | 0.5 | The path to CI config file. Defaults to `.gitlab-ci.yml` | +| `CI_DEBUG_TRACE` | all | 1.7 | Whether [debug tracing](README.md#debug-tracing) is enabled | +| `CI_DEPLOY_PASSWORD` | 10.8 | all | Authentication password of the [GitLab Deploy Token][gitlab-deploy-token], only present if the Project has one related.| +| `CI_DEPLOY_USER` | 10.8 | all | Authentication username of the [GitLab Deploy Token][gitlab-deploy-token], only present if the Project has one related.| +| `CI_DISPOSABLE_ENVIRONMENT` | all | 10.1 | Marks that the job is executed in a disposable environment (something that is created only for this job and disposed of/destroyed after the execution - all executors except `shell` and `ssh`). If the environment is disposable, it is set to true, otherwise it is not defined at all. | +| `CI_ENVIRONMENT_NAME` | 8.15 | all | The name of the environment for this job. Only present if [`environment:name`](../yaml/README.md#environmentname) is set. | +| `CI_ENVIRONMENT_SLUG` | 8.15 | all | A simplified version of the environment name, suitable for inclusion in DNS, URLs, Kubernetes labels, etc. Only present if [`environment:name`](../yaml/README.md#environmentname) is set. | +| `CI_ENVIRONMENT_URL` | 9.3 | all | The URL of the environment for this job. Only present if [`environment:url`](../yaml/README.md#environmenturl) is set. | +| `CI_JOB_ID` | 9.0 | all | The unique id of the current job that GitLab CI uses internally | +| `CI_JOB_MANUAL` | 8.12 | all | The flag to indicate that job was manually started | +| `CI_JOB_NAME` | 9.0 | 0.5 | The name of the job as defined in `.gitlab-ci.yml` | +| `CI_JOB_STAGE` | 9.0 | 0.5 | The name of the stage as defined in `.gitlab-ci.yml` | +| `CI_JOB_TOKEN` | 9.0 | 1.2 | Token used for authenticating with the [GitLab Container Registry][registry] and downloading [dependent repositories][dependent-repositories] | +| `CI_JOB_URL` | 11.1 | 0.5 | Job details URL | +| `CI_MERGE_REQUEST_ID` | 11.6 | all | The ID of the merge request if [the pipelines are for merge requests](../merge_request_pipelines/index.md). Available only if `only: [merge_requests]` is used and the merge request is created. | +| `CI_MERGE_REQUEST_IID` | 11.6 | all | The IID of the merge request if [the pipelines are for merge requests](../merge_request_pipelines/index.md). Available only if `only: [merge_requests]` is used and the merge request is created. | +| `CI_MERGE_REQUEST_PROJECT_ID` | 11.6 | all | The ID of the project of the merge request if [the pipelines are for merge requests](../merge_request_pipelines/index.md). Available only if `only: [merge_requests]` is used and the merge request is created. | +| `CI_MERGE_REQUEST_PROJECT_PATH` | 11.6 | all | The path of the project of the merge request if [the pipelines are for merge requests](../merge_request_pipelines/index.md) (e.g. `namespace/awesome-project`). Available only if `only: [merge_requests]` is used and the merge request is created. | +| `CI_MERGE_REQUEST_PROJECT_URL` | 11.6 | all | The URL of the project of the merge request if [the pipelines are for merge requests](../merge_request_pipelines/index.md) (e.g. `http://192.168.10.15:3000/namespace/awesome-project`). Available only if `only: [merge_requests]` is used and the merge request is created. | +| `CI_MERGE_REQUEST_REF_PATH` | 11.6 | all | The ref path of the merge request if [the pipelines are for merge requests](../merge_request_pipelines/index.md). (e.g. `refs/merge-requests/1/head`). Available only if `only: [merge_requests]` is used and the merge request is created. | +| `CI_MERGE_REQUEST_SOURCE_BRANCH_NAME` | 11.6 | all | The source branch name of the merge request if [the pipelines are for merge requests](../merge_request_pipelines/index.md). Available only if `only: [merge_requests]` is used and the merge request is created. | +| `CI_MERGE_REQUEST_SOURCE_BRANCH_SHA` | 11.9 | all | The HEAD sha of the source branch of the merge request if [the pipelines are for merge requests](../merge_request_pipelines/index.md). Available only if `only: [merge_requests]` is used and the merge request is created. | +| `CI_MERGE_REQUEST_SOURCE_PROJECT_ID` | 11.6 | all | The ID of the source project of the merge request if [the pipelines are for merge requests](../merge_request_pipelines/index.md). Available only if `only: [merge_requests]` is used and the merge request is created. | +| `CI_MERGE_REQUEST_SOURCE_PROJECT_PATH` | 11.6 | all | The path of the source project of the merge request if [the pipelines are for merge requests](../merge_request_pipelines/index.md). Available only if `only: [merge_requests]` is used and the merge request is created. | +| `CI_MERGE_REQUEST_SOURCE_PROJECT_URL` | 11.6 | all | The URL of the source project of the merge request if [the pipelines are for merge requests](../merge_request_pipelines/index.md). Available only if `only: [merge_requests]` is used and the merge request is created. | +| `CI_MERGE_REQUEST_TARGET_BRANCH_NAME` | 11.6 | all | The target branch name of the merge request if [the pipelines are for merge requests](../merge_request_pipelines/index.md). Available only if `only: [merge_requests]` is used and the merge request is created. | +| `CI_MERGE_REQUEST_TARGET_BRANCH_SHA` | 11.9 | all | The HEAD sha of the target branch of the merge request if [the pipelines are for merge requests](../merge_request_pipelines/index.md). Available only if `only: [merge_requests]` is used and the merge request is created. | +| `CI_MERGE_REQUEST_TITLE` | 11.9 | all | The title of the merge request if [the pipelines are for merge requests](../merge_request_pipelines/index.md). Available only if `only: [merge_requests]` is used and the merge request is created. | +| `CI_MERGE_REQUEST_ASSIGNEES` | 11.9 | all | Comma-separated list of username(s) of assignee(s) for the merge request if [the pipelines are for merge requests](../merge_request_pipelines/index.md). Available only if `only: [merge_requests]` is used and the merge request is created. | +| `CI_MERGE_REQUEST_MILESTONE` | 11.9 | all | The milestone title of the merge request if [the pipelines are for merge requests](../merge_request_pipelines/index.md). Available only if `only: [merge_requests]` is used and the merge request is created. | +| `CI_MERGE_REQUEST_LABELS` | 11.9 | all | Comma-separated label names of the merge request if [the pipelines are for merge requests](../merge_request_pipelines/index.md). Available only if `only: [merge_requests]` is used and the merge request is created. | +| `CI_NODE_INDEX` | 11.5 | all | Index of the job in the job set. If the job is not parallelized, this variable is not set. | +| `CI_NODE_TOTAL` | 11.5 | all | Total number of instances of this job running in parallel. If the job is not parallelized, this variable is set to `1`. | +| `CI_API_V4_URL` | 11.7 | all | The GitLab API v4 root URL | +| `CI_PAGES_DOMAIN` | 11.8 | all | The configured domain that hosts GitLab Pages. | +| `CI_PAGES_URL` | 11.8 | all | URL to GitLab Pages-built pages. Always belongs to a subdomain of `CI_PAGES_DOMAIN`. | +| `CI_PIPELINE_ID` | 8.10 | all | The unique id of the current pipeline that GitLab CI uses internally | +| `CI_PIPELINE_IID` | 11.0 | all | The unique id of the current pipeline scoped to project | +| `CI_PIPELINE_SOURCE` | 10.0 | all | Indicates how the pipeline was triggered. Possible options are: `push`, `web`, `trigger`, `schedule`, `api`, and `pipeline`. For pipelines created before GitLab 9.5, this will show as `unknown` | +| `CI_PIPELINE_TRIGGERED` | all | all | The flag to indicate that job was [triggered] | +| `CI_PIPELINE_URL` | 11.1 | 0.5 | Pipeline details URL | +| `CI_PROJECT_DIR` | all | all | The full path where the repository is cloned and where the job is run. If the GitLab Runner `builds_dir` parameter is set, this variable is set relative to the value of `builds_dir`. For more information, see [Advanced configuration](https://docs.gitlab.com/runner/configuration/advanced-configuration.html#the-runners-section) for GitLab Runner. | +| `CI_PROJECT_ID` | all | all | The unique id of the current project that GitLab CI uses internally | +| `CI_PROJECT_NAME` | 8.10 | 0.5 | The project name that is currently being built (actually it is project folder name) | +| `CI_PROJECT_NAMESPACE` | 8.10 | 0.5 | The project namespace (username or groupname) that is currently being built | +| `CI_PROJECT_PATH` | 8.10 | 0.5 | The namespace with project name | +| `CI_PROJECT_PATH_SLUG` | 9.3 | all | `$CI_PROJECT_PATH` lowercased and with everything except `0-9` and `a-z` replaced with `-`. Use in URLs and domain names. | +| `CI_PROJECT_URL` | 8.10 | 0.5 | The HTTP(S) address to access project | +| `CI_PROJECT_VISIBILITY` | 10.3 | all | The project visibility (internal, private, public) | +| `CI_COMMIT_REF_PROTECTED` | 11.11 | all | If the job is running on a protected branch | +| `CI_REGISTRY` | 8.10 | 0.5 | If the Container Registry is enabled it returns the address of GitLab's Container Registry | +| `CI_REGISTRY_IMAGE` | 8.10 | 0.5 | If the Container Registry is enabled for the project it returns the address of the registry tied to the specific project | +| `CI_REGISTRY_PASSWORD` | 9.0 | all | The password to use to push containers to the GitLab Container Registry | +| `CI_REGISTRY_USER` | 9.0 | all | The username to use to push containers to the GitLab Container Registry | +| `CI_REPOSITORY_URL` | 9.0 | all | The URL to clone the Git repository | +| `CI_RUNNER_DESCRIPTION` | 8.10 | 0.5 | The description of the runner as saved in GitLab | +| `CI_RUNNER_EXECUTABLE_ARCH` | all | 10.6 | The OS/architecture of the GitLab Runner executable (note that this is not necessarily the same as the environment of the executor) | +| `CI_RUNNER_ID` | 8.10 | 0.5 | The unique id of runner being used | +| `CI_RUNNER_REVISION` | all | 10.6 | GitLab Runner revision that is executing the current job | +| `CI_RUNNER_TAGS` | 8.10 | 0.5 | The defined runner tags | +| `CI_RUNNER_VERSION` | all | 10.6 | GitLab Runner version that is executing the current job | +| `CI_SERVER` | all | all | Mark that job is executed in CI environment | +| `CI_SERVER_NAME` | all | all | The name of CI server that is used to coordinate jobs | +| `CI_SERVER_REVISION` | all | all | GitLab revision that is used to schedule jobs | +| `CI_SERVER_VERSION` | all | all | GitLab version that is used to schedule jobs | +| `CI_SERVER_VERSION_MAJOR` | 11.4 | all | GitLab version major component | +| `CI_SERVER_VERSION_MINOR` | 11.4 | all | GitLab version minor component | +| `CI_SERVER_VERSION_PATCH` | 11.4 | all | GitLab version patch component | +| `CI_SHARED_ENVIRONMENT` | all | 10.1 | Marks that the job is executed in a shared environment (something that is persisted across CI invocations like `shell` or `ssh` executor). If the environment is shared, it is set to true, otherwise it is not defined at all. | +| `GET_SOURCES_ATTEMPTS` | 8.15 | 1.9 | Number of attempts to fetch sources running a job | +| `GITLAB_CI` | all | all | Mark that job is executed in GitLab CI environment | +| `GITLAB_USER_EMAIL` | 8.12 | all | The email of the user who started the job | +| `GITLAB_USER_ID` | 8.12 | all | The id of the user who started the job | +| `GITLAB_USER_LOGIN` | 10.0 | all | The login username of the user who started the job | +| `GITLAB_USER_NAME` | 10.0 | all | The real name of the user who started the job | +| `RESTORE_CACHE_ATTEMPTS` | 8.15 | 1.9 | Number of attempts to restore the cache running a job | +| `GITLAB_FEATURES` | 10.6 | all | The comma separated list of licensed features available for your instance and plan | + +[gitlab-deploy-token]: ../../user/project/deploy_tokens/index.md#gitlab-deploy-token +[registry]: ../../user/project/container_registry.md +[dependent-repositories]: ../../user/project/new_ci_build_permissions_model.md#dependent-repositories diff --git a/doc/ci/variables/where_variables_can_be_used.md b/doc/ci/variables/where_variables_can_be_used.md index 1d98e8426fe..8009b1d5e8a 100644 --- a/doc/ci/variables/where_variables_can_be_used.md +++ b/doc/ci/variables/where_variables_can_be_used.md @@ -1,3 +1,7 @@ +--- +type: reference +--- + # Where variables can be used As it's described in the [CI/CD variables](README.md) docs, you can @@ -15,26 +19,26 @@ There are two places defined variables can be used. On the: ### `.gitlab-ci.yml` file -| Definition | Can be expanded? | Expansion place | Description | -|--------------------------------------|-------------------|-----------------|--------------| -| `environment:url` | yes | GitLab | The variable expansion is made by GitLab's [internal variable expansion mechanism](#gitlab-internal-variable-expansion-mechanism).<ul><li>Supported: all variables defined for a job (project/group variables, variables from `.gitlab-ci.yml`, variables from triggers, variables from pipeline schedules)</li><li>Not supported: variables defined in Runner's `config.toml` and variables created in job's `script`</li></ul> | -| `environment:name` | yes | GitLab | Similar to `environment:url`, but the variables expansion doesn't support: <ul><li>variables that are based on the environment's name (`CI_ENVIRONMENT_NAME`, `CI_ENVIRONMENT_SLUG`)</li><li>any other variables related to environment (currently only `CI_ENVIRONMENT_URL`)</li><li>[persisted variables](#persisted-variables)</li></ul> | -| `variables` | yes | Runner | The variable expansion is made by GitLab Runner's [internal variable expansion mechanism](#gitlab-runner-internal-variable-expansion-mechanism) | -| `image` | yes | Runner | The variable expansion is made by GitLab Runner's [internal variable expansion mechanism](#gitlab-runner-internal-variable-expansion-mechanism) | -| `services:[]` | yes | Runner | The variable expansion is made by GitLab Runner's [internal variable expansion mechanism](#gitlab-runner-internal-variable-expansion-mechanism) | -| `services:[]:name` | yes | Runner | The variable expansion is made by GitLab Runner's [internal variable expansion mechanism](#gitlab-runner-internal-variable-expansion-mechanism) | -| `cache:key` | yes | Runner | The variable expansion is made by GitLab Runner's [internal variable expansion mechanism](#gitlab-runner-internal-variable-expansion-mechanism) | -| `artifacts:name` | yes | Runner | The variable expansion is made by GitLab Runner's shell environment | -| `script`, `before_script`, `after_script` | yes | Script execution shell | The variable expansion is made by the [execution shell environment](#execution-shell-environment) | -| `only:variables:[]`, `except:variables:[]` | no | n/a | The variable must be in the form of `$variable`.<br/>Not supported:<ul><li>variables that are based on the environment's name (`CI_ENVIRONMENT_NAME`, `CI_ENVIRONMENT_SLUG`)</li><li>any other variables related to environment (currently only `CI_ENVIRONMENT_URL`)</li><li>[persisted variables](#persisted-variables)</li></ul> | +| Definition | Can be expanded? | Expansion place | Description | +|:-------------------------------------------|:-----------------|:-----------------------|:----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| `environment:url` | yes | GitLab | The variable expansion is made by GitLab's [internal variable expansion mechanism](#gitlab-internal-variable-expansion-mechanism).<br/><br/>Supported are all variables defined for a job (project/group variables, variables from `.gitlab-ci.yml`, variables from triggers, variables from pipeline schedules).<br/><br/>Not supported are variables defined in Runner's `config.toml` and variables created in job's `script`. | +| `environment:name` | yes | GitLab | Similar to `environment:url`, but the variables expansion doesn't support the following:<br/><br/>- Variables that are based on the environment's name (`CI_ENVIRONMENT_NAME`, `CI_ENVIRONMENT_SLUG`).<br/>- Any other variables related to environment (currently only `CI_ENVIRONMENT_URL`).<br/>- [Persisted variables](#persisted-variables). | +| `variables` | yes | Runner | The variable expansion is made by GitLab Runner's [internal variable expansion mechanism](#gitlab-runner-internal-variable-expansion-mechanism) | +| `image` | yes | Runner | The variable expansion is made by GitLab Runner's [internal variable expansion mechanism](#gitlab-runner-internal-variable-expansion-mechanism) | +| `services:[]` | yes | Runner | The variable expansion is made by GitLab Runner's [internal variable expansion mechanism](#gitlab-runner-internal-variable-expansion-mechanism) | +| `services:[]:name` | yes | Runner | The variable expansion is made by GitLab Runner's [internal variable expansion mechanism](#gitlab-runner-internal-variable-expansion-mechanism) | +| `cache:key` | yes | Runner | The variable expansion is made by GitLab Runner's [internal variable expansion mechanism](#gitlab-runner-internal-variable-expansion-mechanism) | +| `artifacts:name` | yes | Runner | The variable expansion is made by GitLab Runner's shell environment | +| `script`, `before_script`, `after_script` | yes | Script execution shell | The variable expansion is made by the [execution shell environment](#execution-shell-environment) | +| `only:variables:[]`, `except:variables:[]` | no | n/a | The variable must be in the form of `$variable`. Not supported are the following:<br/><br/>- Variables that are based on the environment's name (`CI_ENVIRONMENT_NAME`, `CI_ENVIRONMENT_SLUG`).<br/>- Any other variables related to environment (currently only `CI_ENVIRONMENT_URL`).<br/>- [Persisted variables](#persisted-variables). | ### `config.toml` file NOTE: **Note:** You can read more about `config.toml` in the [Runner's docs](https://docs.gitlab.com/runner/configuration/advanced-configuration.html). -| Definition | Can be expanded? | Description | -|--------------------------------------|------------------|-------------| +| Definition | Can be expanded? | Description | +|:-------------------------------------|:-----------------|:---------------------------------------------------------------------------------------------------------------------------------------------| | `runners.environment` | yes | The variable expansion is made by the Runner's [internal variable expansion mechanism](#gitlab-runner-internal-variable-expansion-mechanism) | | `runners.kubernetes.pod_labels` | yes | The Variable expansion is made by the Runner's [internal variable expansion mechanism](#gitlab-runner-internal-variable-expansion-mechanism) | | `runners.kubernetes.pod_annotations` | yes | The Variable expansion is made by the Runner's [internal variable expansion mechanism](#gitlab-runner-internal-variable-expansion-mechanism) | @@ -106,9 +110,28 @@ The following variables are known as "persisted": They are: -- Supported for definitions where the ["Expansion place"](#gitlab-ci-yml-file) is: +- Supported for definitions where the ["Expansion place"](#gitlab-ciyml-file) is: - Runner. - Script execution shell. - Not supported: - - For definitions where the ["Expansion place"](#gitlab-ci-yml-file) is GitLab. - - In the `only` and `except` [variables expressions](README.md#variables-expressions). + - For definitions where the ["Expansion place"](#gitlab-ciyml-file) is GitLab. + - In the `only` and `except` [variables expressions](README.md#environment-variables-expressions). + +## Variables with an environment scope + +Variables defined with an environment scope are supported. Given that +there is a variable `$STAGING_SECRET` defined in a scope of +`review/staging/*`, the following job that is using dynamic environments +is going to be created, based on the matching variable expression: + +```yaml +my-job: + stage: staging + environment: + name: review/$CI_JOB_STAGE/deploy + script: + - 'deploy staging' + only: + variables: + - $STAGING_SECRET == 'something' +``` diff --git a/doc/ci/yaml/README.md b/doc/ci/yaml/README.md index e5668c140fa..fa4b0378f61 100644 --- a/doc/ci/yaml/README.md +++ b/doc/ci/yaml/README.md @@ -1,26 +1,43 @@ -# Configuration of your pipelines with .gitlab-ci.yml +--- +type: reference +--- + +# GitLab CI/CD Pipeline Configuration Reference + +GitLab CI/CD [pipelines](../pipelines.md) are configured using a YAML file called `.gitlab-ci.yml` within each project. + +The `.gitlab-ci.yml` file defines the structure and order of the pipelines and determines: -This document describes the usage of `.gitlab-ci.yml`, the file that is used by -GitLab Runner to manage your project's pipelines. +- What to execute using [GitLab Runner](https://docs.gitlab.com/runner/). +- What decisions to make when specific conditions are encountered. For example, when a process succeeds or fails. -From version 7.12, GitLab CI uses a [YAML](https://en.wikipedia.org/wiki/YAML) -file (`.gitlab-ci.yml`) for the project configuration. It is placed in the root -of your repository and contains definitions of how your project should be built. +This topic covers CI/CD pipeline configuration. For other CI/CD configuration information, see: -If you want a quick introduction to GitLab CI, follow our -[quick start guide](../quick_start/README.md). +- [GitLab CI/CD Variables](../variables/README.md), for configuring the environment the pipelines run in. +- [GitLab Runner advanced configuration](https://docs.gitlab.com/runner/configuration/advanced-configuration.html), for configuring GitLab Runner. + +We have complete examples of configuring pipelines: + +- For a quick introduction to GitLab CI, follow our [quick start guide](../quick_start/README.md). +- For a collection of examples, see [GitLab CI/CD Examples](../examples/README.md). +- To see a large `.gitlab-ci.yml` file used in an enterprise, see the [`.gitlab-ci.yml` file for `gitlab-ce`](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/.gitlab-ci.yml). NOTE: **Note:** -If you have a [mirrored repository where GitLab pulls from](https://docs.gitlab.com/ee/workflow/repository_mirroring.html#pulling-from-a-remote-repository-starter), +If you have a [mirrored repository where GitLab pulls from](../../workflow/repository_mirroring.md#pulling-from-a-remote-repository-starter), you may need to enable pipeline triggering in your project's **Settings > Repository > Pull from a remote repository > Trigger pipelines for mirror updates**. -## Jobs +## Introduction + +Pipeline configuration begins with jobs. Jobs are the most fundamental element of a `.gitlab-ci.yml` file. -The YAML file defines a set of jobs with constraints stating when they should -be run. You can specify an unlimited number of jobs which are defined as -top-level elements with an arbitrary name and always have to contain at least -the `script` clause. +Jobs are: + +- Defined with constraints stating under what conditions they should be executed. +- Top-level elements with an arbitrary name and must contain at least the [`script`](#script) clause. +- Not limited in how many can be defined. + +For example: ```yaml job1: @@ -39,6 +56,14 @@ Jobs are picked up by [Runners](../runners/README.md) and executed within the environment of the Runner. What is important, is that each job is run independently from each other. +### Validate the .gitlab-ci.yml + +Each instance of GitLab CI has an embedded debug tool called Lint, which validates the +content of your `.gitlab-ci.yml` files. You can find the Lint under the page `ci/lint` of your +project namespace. For example, `https://gitlab.example.com/gitlab-org/project-123/-/ci/lint`. + +### Unavailable names for jobs + Each job must have a unique name, but there are a few **reserved `keywords` that cannot be used as job names**: @@ -51,42 +76,137 @@ cannot be used as job names**: - `variables` - `cache` -A job is defined by a list of parameters that define the job behavior. - -| Keyword | Required | Description | -|---------------|----------|-------------| -| [script](#script) | yes | Defines a shell script which is executed by Runner | -| [extends](#extends) | no | Defines a configuration entry that this job is going to inherit from | -| [include](#include) | no | Defines a configuration entry that allows this job to include external YAML files | -| [image](#image-and-services) | no | Use docker image, covered in [Using Docker Images](../docker/using_docker_images.md#define-image-and-services-from-gitlab-ciyml) | -| [services](#image-and-services) | no | Use docker services, covered in [Using Docker Images](../docker/using_docker_images.md#define-image-and-services-from-gitlab-ciyml) | -| [stage](#stage) | no | Defines a job stage (default: `test`) | -| type | no | Alias for `stage` | -| [variables](#variables) | no | Define job variables on a job level | -| [only](#only-and-except-simplified) | no | Defines a list of git refs for which job is created | -| [except](#only-and-except-simplified) | no | Defines a list of git refs for which job is not created | -| [tags](#tags) | no | Defines a list of tags which are used to select Runner | -| [allow_failure](#allow_failure) | no | Allow job to fail. Failed job doesn't contribute to commit status | -| [when](#when) | no | Define when to run job. Can be `on_success`, `on_failure`, `always` or `manual` | -| [dependencies](#dependencies) | no | Define other jobs that a job depends on so that you can pass artifacts between them| -| [artifacts](#artifacts) | no | Define list of [job artifacts](#artifacts) | -| [cache](#cache) | no | Define list of files that should be cached between subsequent runs | -| [before_script](#before_script-and-after_script) | no | Override a set of commands that are executed before job | -| [after_script](#before_script-and-after_script) | no | Override a set of commands that are executed after job | -| [environment](#environment) | no | Defines a name of environment to which deployment is done by this job | -| [coverage](#coverage) | no | Define code coverage settings for a given job | -| [retry](#retry) | no | Define when and how many times a job can be auto-retried in case of a failure | -| [parallel](#parallel) | no | Defines how many instances of a job should be run in parallel | - -## `image` and `services` - -This allows to specify a custom Docker image and a list of services that can be -used for time of the job. The configuration of this feature is covered in -[a separate document](../docker/README.md). - -## `before_script` and `after_script` - -> Introduced in GitLab 8.7 and requires GitLab Runner v1.2 +### Using reserved keywords + +If you get validation error when using specific values (for example, `true` or `false`), try to: + +- Quote them. +- Change them to a different form. For example, `/bin/true`. + +## Configuration parameters + +A job is defined as a list of parameters that define the job's behavior. + +The following table lists available parameters for jobs: + +| Keyword | Description | +|:---------------------------------------------------|:----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| [`script`](#script) | Shell script which is executed by Runner. | +| [`image`](#image) | Use docker images. Also available: `image:name` and `image:entrypoint`. | +| [`services`](#services) | Use docker services images. Also available: `services:name`, `services:alias`, `services:entrypoint`, and `services:command`. | +| [`before_script`](#before_script-and-after_script) | Override a set of commands that are executed before job. | +| [`after_script`](#before_script-and-after_script) | Override a set of commands that are executed after job. | +| [`stages`](#stages) | Define stages in a pipeline. | +| [`stage`](#stage) | Defines a job stage (default: `test`). | +| [`only`](#onlyexcept-basic) | Limit when jobs are created. Also available: [`only:refs`, `only:kubernetes`, `only:variables`, and `only:changes`](#onlyexcept-advanced). | +| [`except`](#onlyexcept-basic) | Limit when jobs are not created. Also available: [`except:refs`, `except:kubernetes`, `except:variables`, and `except:changes`](#onlyexcept-advanced). | +| [`tags`](#tags) | List of tags which are used to select Runner. | +| [`allow_failure`](#allow_failure) | Allow job to fail. Failed job doesn't contribute to commit status. | +| [`when`](#when) | When to run job. Also available: `when:manual` and `when:delayed`. | +| [`environment`](#environment) | Name of an environment to which the job deploys. Also available: `environment:name`, `environment:url`, `environment:on_stop`, and `environment:action`. | +| [`cache`](#cache) | List of files that should be cached between subsequent runs. Also available: `cache:paths`, `cache:key`, `cache:untracked`, and `cache:policy`. | +| [`artifacts`](#artifacts) | List of files and directories to attach to a job on success. Also available: `artifacts:paths`, `artifacts:name`, `artifacts:untracked`, `artifacts:when`, `artifacts:expire_in`, `artifacts:reports`, and `artifacts:reports:junit`.<br><br>In GitLab [Enterprise Edition](https://about.gitlab.com/pricing/), these are available: `artifacts:reports:codequality`, `artifacts:reports:sast`, `artifacts:reports:dependency_scanning`, `artifacts:reports:container_scanning`, `artifacts:reports:dast`, `artifacts:reports:license_management`, `artifacts:reports:performance` and `artifacts:reports:metrics`. | +| [`dependencies`](#dependencies) | Other jobs that a job depends on so that you can pass artifacts between them. | +| [`coverage`](#coverage) | Code coverage settings for a given job. | +| [`retry`](#retry) | When and how many times a job can be auto-retried in case of a failure. | +| [`parallel`](#parallel) | How many instances of a job should be run in parallel. | +| [`trigger`](#trigger-premium) | Defines a downstream pipeline trigger. | +| [`include`](#include) | Allows this job to include external YAML files. Also available: `include:local`, `include:file`, `include:template`, and `include:remote`. | +| [`extends`](#extends) | Configuration entries that this job is going to inherit from. | +| [`pages`](#pages) | Upload the result of a job to use with GitLab Pages. | +| [`variables`](#variables) | Define job variables on a job level. | + +NOTE: **Note:** +Parameters `types` and `type` are [deprecated](#deprecated-parameters). + +## Parameter details + +The following are detailed explanations for parameters used to configure CI/CD pipelines. + +### `script` + +`script` is the only required keyword that a job needs. It's a shell script +which is executed by the Runner. For example: + +```yaml +job: + script: "bundle exec rspec" +``` + +This parameter can also contain several commands using an array: + +```yaml +job: + script: + - uname -a + - bundle exec rspec +``` + +NOTE: **Note:** +Sometimes, `script` commands will need to be wrapped in single or double quotes. +For example, commands that contain a colon (`:`) need to be wrapped in quotes so +that the YAML parser knows to interpret the whole thing as a string rather than +a "key: value" pair. Be careful when using special characters: +`:`, `{`, `}`, `[`, `]`, `,`, `&`, `*`, `#`, `?`, `|`, `-`, `<`, `>`, `=`, `!`, `%`, `@`, `` ` ``. + +### `image` + +Used to specify [a Docker image](../docker/using_docker_images.md#what-is-an-image) to use for the job. + +For: + +- Simple definition examples, see [Define `image` and `services` from .gitlab-ci.yml](../docker/using_docker_images.md#define-image-and-services-from-gitlab-ciyml). +- Detailed usage information, refer to [Docker integration](../docker/README.md) documentation. + +#### `image:name` + +An [extended docker configuration option](../docker/using_docker_images.md#extended-docker-configuration-options). + +For more information, see [Available settings for `image`](../docker/using_docker_images.md#available-settings-for-image). + +#### `image:entrypoint` + +An [extended docker configuration option](../docker/using_docker_images.md#extended-docker-configuration-options). + +For more information, see [Available settings for `image`](../docker/using_docker_images.md#available-settings-for-image). + +### `services` + +Used to specify a [service Docker image](../docker/using_docker_images.md#what-is-a-service), linked to a base image specified in [`image`](#image). + +For: + +- Simple definition examples, see [Define `image` and `services` from .gitlab-ci.yml](../docker/using_docker_images.md#define-image-and-services-from-gitlab-ciyml). +- Detailed usage information, refer to [Docker integration](../docker/README.md) documentation. +- For example services, see [GitLab CI Services](../services/README.md). + +#### `services:name` + +An [extended docker configuration option](../docker/using_docker_images.md#extended-docker-configuration-options). + +For more information, see see [Available settings for `services`](../docker/using_docker_images.md#available-settings-for-services). + +#### `services:alias` + +An [extended docker configuration option](../docker/using_docker_images.md#extended-docker-configuration-options). + +For more information, see see [Available settings for `services`](../docker/using_docker_images.md#available-settings-for-services). + +#### `services:entrypoint` + +An [extended docker configuration option](../docker/using_docker_images.md#extended-docker-configuration-options). + +For more information, see see [Available settings for `services`](../docker/using_docker_images.md#available-settings-for-services). + +#### `services:command` + +An [extended docker configuration option](../docker/using_docker_images.md#extended-docker-configuration-options). + +For more information, see see [Available settings for `services`](../docker/using_docker_images.md#available-settings-for-services). + +### `before_script` and `after_script` + +> Introduced in GitLab 8.7 and requires GitLab Runner v1.2. `before_script` is used to define the command that should be run before all jobs, including deploy jobs, but after the restoration of [artifacts](#artifacts). @@ -116,7 +236,7 @@ job: - execute this after my script ``` -## `stages` +### `stages` `stages` is used to define stages that can be used by jobs and is defined globally. @@ -150,11 +270,11 @@ There are also two edge cases worth mentioning: `test` and `deploy` are allowed to be used as job's stage by default. 1. If a job doesn't specify a `stage`, the job is assigned the `test` stage. -## `stage` +### `stage` `stage` is defined per-job and relies on [`stages`](#stages) which is defined globally. It allows to group jobs into different stages, and jobs of the same -`stage` are executed in `parallel`. For example: +`stage` are executed in parallel (subject to [certain conditions](#using-your-own-runners)). For example: ```yaml stages: @@ -179,38 +299,18 @@ job 4: script: make deploy ``` -## `types` +#### Using your own Runners -CAUTION: **Deprecated:** -`types` is deprecated, and could be removed in one of the future releases. -Use [stages](#stages) instead. - -## `script` - -`script` is the only required keyword that a job needs. It's a shell script -which is executed by the Runner. For example: - -```yaml -job: - script: "bundle exec rspec" -``` - -This parameter can also contain several commands using an array: +When using your own Runners, GitLab Runner runs only one job at a time by default (see the +`concurrent` flag in [Runner global settings](https://docs.gitlab.com/runner/configuration/advanced-configuration.html#the-global-section) +for more information). -```yaml -job: - script: - - uname -a - - bundle exec rspec -``` +Jobs will run on your own Runners in parallel only if: -Sometimes, `script` commands will need to be wrapped in single or double quotes. -For example, commands that contain a colon (`:`) need to be wrapped in quotes so -that the YAML parser knows to interpret the whole thing as a string rather than -a "key: value" pair. Be careful when using special characters: -`:`, `{`, `}`, `[`, `]`, `,`, `&`, `*`, `#`, `?`, `|`, `-`, `<`, `>`, `=`, `!`, `%`, `@`, `` ` ``. +- Run on different Runners. +- The Runner's `concurrent` setting has been changed. -## `only` and `except` (simplified) +### `only`/`except` (basic) `only` and `except` are two parameters that set a job policy to limit when jobs are created: @@ -223,7 +323,7 @@ There are a few rules that apply to the usage of job policy: - `only` and `except` are inclusive. If both `only` and `except` are defined in a job specification, the ref is filtered by `only` and `except`. -- `only` and `except` allow the use of regular expressions (using [Ruby regexp syntax](https://ruby-doc.org/core/Regexp.html)). +- `only` and `except` allow the use of regular expressions ([supported regexp syntax](#supported-onlyexcept-regexp-syntax)). - `only` and `except` allow to specify a repository path to filter jobs for forks. @@ -241,6 +341,7 @@ In addition, `only` and `except` allow the use of special keywords: | `triggers` | For pipelines created using a trigger token. | | `web` | For pipelines created using **Run pipeline** button in GitLab UI (under your project's **Pipelines**). | | `merge_requests` | When a merge request is created or updated (See [pipelines for merge requests](../merge_request_pipelines/index.md)). | +| `chats` | For jobs created using a [GitLab ChatOps](../chatops/README.md) command. | In the example below, `job` will run only for refs that start with `issue-`, whereas all branches will be skipped: @@ -255,6 +356,19 @@ job: - branches ``` +Pattern matching is case-sensitive by default. Use `i` flag modifier, like +`/pattern/i` to make a pattern case-insensitive: + +```yaml +job: + # use regexp + only: + - /^issue-.*$/i + # use special keyword + except: + - branches +``` + In this example, `job` will run only for refs that are tagged, or if a build is explicitly requested via an API trigger or a [Pipeline Schedule][schedules]: @@ -276,10 +390,11 @@ job: - branches@gitlab-org/gitlab-ce except: - master@gitlab-org/gitlab-ce + - /^release/.*$/@gitlab-org/gitlab-ce ``` The above example will run `job` for all branches on `gitlab-org/gitlab-ce`, -except master. +except `master` and those with names prefixed with `release/`. If a job does not have an `only` rule, `only: ['branches', 'tags']` is set by default. If it doesn't have an `except` rule, it is empty. @@ -299,11 +414,50 @@ job: only: ['branches', 'tags'] ``` -## `only` and `except` (complex) +#### Regular expressions + +Because `@` is used to denote the beginning of a ref's repository path, +matching a ref name containing the `@` character in a regular expression +requires the use of the hex character code match `\x40`. + +Only the tag or branch name can be matched by a regular expression. +The repository path, if given, is always matched literally. + +If a regular expression shall be used to match the tag or branch name, +the entire ref name part of the pattern has to be a regular expression, +and must be surrounded by `/`. +(With regular expression flags appended after the closing `/`.) +So `issue-/.*/` won't work to match all tag names or branch names +that begin with `issue-`. -> - `refs` and `kubernetes` policies introduced in GitLab 10.0. -> - `variables` policy introduced in GitLab 10.7. -> - `changes` policy [introduced](https://gitlab.com/gitlab-org/gitlab-ce/issues/19232) in GitLab 11.4. +TIP: **Tip** +Use anchors `^` and `$` to avoid the regular expression +matching only a substring of the tag name or branch name. +For example, `/^issue-.*$/` is equivalent to `/^issue-/`, +while just `/issue/` would also match a branch called `severe-issues`. + +### Supported `only`/`except` regexp syntax + +CAUTION: **Warning:** +This is a breaking change that was introduced with GitLab 11.9.4. + +In GitLab 11.9.4, GitLab begun internally converting regexp used +in `only` and `except` parameters to [RE2](https://github.com/google/re2/wiki/Syntax). + +This means that only subset of features provided by [Ruby Regexp](https://ruby-doc.org/core/Regexp.html) +is supported. [RE2](https://github.com/google/re2/wiki/Syntax) limits the set of features +provided due to computational complexity, which means some features became unavailable in GitLab 11.9.4. +For example, negative lookaheads. + +For GitLab versions from 11.9.7 and up to GitLab 12.0, GitLab provides a feature flag that can be +enabled by administrators that allows users to use unsafe regexp syntax. This brings compatibility +with previously allowed syntax version and allows users to gracefully migrate to the new syntax. + +```ruby +Feature.enable(:allow_unsafe_ruby_regexp) +``` + +### `only`/`except` (advanced) CAUTION: **Warning:** This an _alpha_ feature, and it is subject to change at any time without @@ -323,10 +477,12 @@ If you use multiple keys under `only` or `except`, they act as an AND. The logic > (any of refs) AND (any of variables) AND (any of changes) AND (if kubernetes is active) -### `only:refs` and `except:refs` +#### `only:refs`/`except:refs` + +> `refs` policy introduced in GitLab 10.0. The `refs` strategy can take the same values as the -[simplified only/except configuration](#only-and-except-simplified). +[simplified only/except configuration](#onlyexcept-basic). In the example below, the `deploy` job is going to be created only when the pipeline has been [scheduled][schedules] or runs for the `master` branch: @@ -339,7 +495,9 @@ deploy: - schedules ``` -### `only:kubernetes` and `except:kubernetes` +#### `only:kubernetes`/`except:kubernetes` + +> `kubernetes` policy introduced in GitLab 10.0. The `kubernetes` strategy accepts only the `active` keyword. @@ -352,7 +510,9 @@ deploy: kubernetes: active ``` -### `only:variables` and `except:variables` +#### `only:variables`/`except:variables` + +> `variables` policy introduced in GitLab 10.7. The `variables` keyword is used to define variables expressions. In other words, you can use predefined variables / project / group or @@ -382,11 +542,13 @@ end-to-end: - $CI_COMMIT_MESSAGE =~ /skip-end-to-end-tests/ ``` -Learn more about [variables expressions](../variables/README.md#variables-expressions). +Learn more about [variables expressions](../variables/README.md#environment-variables-expressions). -### `only:changes` and `except:changes` +#### `only:changes`/`except:changes` -Using the `changes` keyword with `only` or `except`, makes it possible to define if +> `changes` policy [introduced][ce-19232] in GitLab 11.4. + +Using the `changes` keyword with `only` or `except` makes it possible to define if a job should be created based on files modified by a git push event. For example: @@ -402,36 +564,59 @@ docker build: - more_scripts/*.{rb,py,sh} ``` -In the scenario above, if you are pushing multiple commits to GitLab to an -existing branch, GitLab creates and triggers the `docker build` job, provided that -one of the commits contains changes to either: +In the scenario above, when pushing multiple commits to GitLab to an existing +branch, GitLab creates and triggers the `docker build` job, provided that one of the +commits contains changes to any of the following: - The `Dockerfile` file. - Any of the files inside `docker/scripts/` directory. - Any of the files and subdirectories inside the `dockerfiles` directory. - Any of the files with `rb`, `py`, `sh` extensions inside the `more_scripts` directory. +You can also use glob patterns to match multiple files in either the root directory of the repo, or in _any_ directory within the repo. For example: + +```yaml +test: + script: npm run test + only: + changes: + - "*.json" + - "**/*.sql" +``` + +NOTE: **Note:** +In the example above, the expressions are wrapped double quotes because they are glob patterns. GitLab will fail to parse `.gitlab-ci.yml` files with unwrapped glob patterns. + +The following example will skip the CI job if a change is detected in any file in the root directory of the repo with a `.md` extension: + +```yaml +build: + script: npm run build + except: + changes: + - "*.md" +``` + CAUTION: **Warning:** There are some caveats when using this feature with new branches and tags. See the section below. -#### Using `changes` with new branches and tags +##### Using `changes` with new branches and tags -If you are pushing a **new** branch or a **new** tag to GitLab, the policy -always evaluates to true and GitLab will create a job. This feature is not -connected with merge requests yet, and because GitLab is creating pipelines -before an user can create a merge request we don't know a target branch at -this point. +When pushing a **new** branch or a **new** tag to GitLab, the policy always +evaluates to true and GitLab will create a job. This feature is not connected +with merge requests yet and, because GitLab is creating pipelines before a user +can create a merge request, it is unknown what the target branch is at this point. -#### Using `changes` with `merge_requests` +##### Using `changes` with `merge_requests` With [pipelines for merge requests](../merge_request_pipelines/index.md), -make it possible to define if a job should be created base on files modified +it is possible to define a job to be created based on files modified in a merge request. For example: -``` +```yaml docker build service one: script: docker build -t my-service-one-image:$CI_COMMIT_REF_SLUG . only: @@ -442,11 +627,11 @@ docker build service one: - service-one/**/* ``` -In the scenario above, if you create or update a merge request that changes -either files in `service-one` folder or `Dockerfile`, GitLab creates and triggers -the `docker build service one` job. +In the scenario above, if a merge request is created or updated that changes +either files in `service-one` directory or the `Dockerfile`, GitLab creates +and triggers the `docker build service one` job. -## `tags` +### `tags` `tags` is used to select specific Runners from the list of all Runners that are allowed to run this project. @@ -489,7 +674,7 @@ osx job: - echo "Hello, $USER!" ``` -## `allow_failure` +### `allow_failure` `allow_failure` allows a job to fail without impacting the rest of the CI suite. @@ -525,7 +710,7 @@ job3: - deploy_to_staging ``` -## `when` +### `when` `when` is used to implement jobs that are run in case of failure or despite the failure. @@ -587,10 +772,8 @@ The above script will: success or failure. 1. Allow you to manually execute `deploy_job` from GitLab's UI. -### `when:manual` +#### `when:manual` -> **Notes:** -> > - Introduced in GitLab 8.10. > - Blocking manual actions were introduced in GitLab 9.0. > - Protected actions were introduced in GitLab 9.2. @@ -599,7 +782,7 @@ Manual actions are a special type of job that are not executed automatically, they need to be explicitly started by a user. An example usage of manual actions would be a deployment to a production environment. Manual actions can be started from the pipeline, job, environment, and deployment views. Read more at the -[environments documentation](../environments.md#manually-deploying-to-environments). +[environments documentation](../environments.md#configuring-manual-deployments). Manual actions can be either optional or blocking. Blocking manual actions will block the execution of the pipeline at the stage this action is defined in. It's @@ -618,11 +801,11 @@ action fails, the pipeline will eventually succeed. Manual actions are considered to be write actions, so permissions for [protected branches](../../user/project/protected_branches.md) are used when -user wants to trigger an action. In other words, in order to trigger a manual -action assigned to a branch that the pipeline is running for, user needs to -have ability to merge to this branch. +a user wants to trigger an action. In other words, in order to trigger a manual +action assigned to a branch that the pipeline is running for, the user needs to +have the ability to merge to this branch. -### `when:delayed` +#### `when:delayed` > [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/21767) in GitLab 11.4. @@ -658,10 +841,8 @@ This job will never be executed in the future unless you execute the job manuall You can start a delayed job immediately by clicking the **Play** button. GitLab runner will pick your job soon and start the job. -## `environment` +### `environment` -> **Notes:** -> > - Introduced in GitLab 8.9. > - You can read more about environments and find more examples in the > [documentation about environments][environment]. @@ -683,10 +864,8 @@ deploy to production: In the above example, the `deploy to production` job will be marked as doing a deployment to the `production` environment. -### `environment:name` +#### `environment:name` -> **Notes:** -> > - Introduced in GitLab 8.11. > - Before GitLab 8.11, the name of an environment could be defined as a string like > `environment: production`. The recommended way now is to define it under the @@ -722,10 +901,8 @@ deploy to production: name: production ``` -### `environment:url` +#### `environment:url` -> **Notes:** -> > - Introduced in GitLab 8.11. > - Before GitLab 8.11, the URL could be added only in GitLab's UI. The > recommended way now is to define it in `.gitlab-ci.yml`. @@ -749,10 +926,8 @@ deploy to production: url: https://prod.example.com ``` -### `environment:on_stop` +#### `environment:on_stop` -> **Notes:** -> > - [Introduced][ce-6669] in GitLab 8.13. > - Starting with GitLab 8.14, when you have an environment that has a stop action > defined, GitLab will automatically trigger a stop action when the associated @@ -764,7 +939,7 @@ the environment. Read the `environment:action` section for an example. -### `environment:action` +#### `environment:action` > [Introduced][ce-6669] in GitLab 8.13. @@ -805,10 +980,8 @@ The `stop_review_app` job is **required** to have the following keywords defined - `stage` should be the same as the `review_app` in order for the environment to stop automatically when the branch is deleted -### Dynamic environments +#### Dynamic environments -> **Notes:** -> > - [Introduced][ce-6323] in GitLab 8.12 and GitLab Runner 1.6. > - The `$CI_ENVIRONMENT_SLUG` was [introduced][ce-7983] in GitLab 8.15. > - The `name` and `url` parameters can use any of the defined CI variables, @@ -841,10 +1014,8 @@ The common use case is to create dynamic environments for branches and use them as Review Apps. You can see a simple example using Review Apps at <https://gitlab.com/gitlab-examples/review-apps-nginx/>. -## `cache` +### `cache` -> **Notes:** -> > - Introduced in GitLab Runner v0.7.0. > - `cache` can be set globally and per-job. > - From GitLab 9.0, caching is enabled and shared between pipelines and jobs @@ -862,7 +1033,7 @@ workspace. If `cache` is defined outside the scope of jobs, it means it is set globally and all jobs will use that definition. -### `cache:paths` +#### `cache:paths` Use the `paths` directive to choose which files or directories will be cached. Wildcards can be used as well. @@ -898,7 +1069,7 @@ Note that since cache is shared between jobs, if you're using different paths for different jobs, you should also set a different **cache:key** otherwise cache content can be overwritten. -### `cache:key` +#### `cache:key` > Introduced in GitLab Runner v1.0.0. @@ -939,7 +1110,7 @@ cache: - binaries/ ``` -### `cache:untracked` +#### `cache:untracked` Set `untracked: true` to cache all files that are untracked in your Git repository: @@ -962,7 +1133,7 @@ rspec: - binaries/ ``` -### `cache:policy` +#### `cache:policy` > Introduced in GitLab 9.4. @@ -1009,10 +1180,8 @@ Additionally, if you have a job that unconditionally recreates the cache without reference to its previous contents, you can use `policy: push` in that job to skip the download step. -## `artifacts` +### `artifacts` -> **Notes:** -> > - Introduced in GitLab Runner v0.7.0 for non-Windows platforms. > - Windows support was added in GitLab Runner v.1.0.0. > - From GitLab 9.2, caches are restored before artifacts. @@ -1020,14 +1189,14 @@ skip the download step. > - Job artifacts are only collected for successful jobs by default. `artifacts` is used to specify a list of files and directories which should be -attached to the job after success. +attached to the job when it [succeeds, fails, or always](#artifactswhen). -The artifacts will be sent to GitLab after the job finishes successfully and will +The artifacts will be sent to GitLab after the job finishes and will be available for download in the GitLab UI. -[Read more about artifacts.](../../user/project/pipelines/job_artifacts.md) +[Read more about artifacts](../../user/project/pipelines/job_artifacts.md). -### `artifacts:paths` +#### `artifacts:paths` You can only use paths that are within the project workspace. To pass artifacts between different jobs, see [dependencies](#dependencies). @@ -1072,7 +1241,7 @@ release-job: - tags ``` -### `artifacts:name` +#### `artifacts:name` > Introduced in GitLab 8.6 and GitLab Runner v1.1.0. @@ -1153,14 +1322,13 @@ job: - binaries/ ``` -### `artifacts:untracked` +#### `artifacts:untracked` `artifacts:untracked` is used to add all Git untracked files as artifacts (along to the paths defined in `artifacts:paths`). NOTE: **Note:** -To exclude the folders/files which should not be a part of `untracked` just -add them to `.gitignore`. +`artifacts:untracked` ignores configuration in the repository's `.gitignore` file. Send all Git untracked files: @@ -1178,7 +1346,7 @@ artifacts: - binaries/ ``` -### `artifacts:when` +#### `artifacts:when` > Introduced in GitLab 8.9 and GitLab Runner v1.3.0. @@ -1199,7 +1367,7 @@ job: when: on_failure ``` -### `artifacts:expire_in` +#### `artifacts:expire_in` > Introduced in GitLab 8.9 and GitLab Runner v1.3.0. @@ -1234,7 +1402,7 @@ job: expire_in: 1 week ``` -### `artifacts:reports` +#### `artifacts:reports` > [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/20390) in GitLab 11.2. Requires GitLab Runner 11.2 and above. @@ -1252,7 +1420,7 @@ NOTE: **Note:** If you also want the ability to browse the report output files, include the [`artifacts:paths`](#artifactspaths) keyword. -#### `artifacts:reports:junit` +##### `artifacts:reports:junit` > [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/20390) in GitLab 11.2. Requires GitLab Runner 11.2 and above. @@ -1286,7 +1454,7 @@ concatenated into a single file. Use a filename pattern (`junit: rspec-*.xml`), an array of filenames (`junit: [rspec-1.xml, rspec-2.xml, rspec-3.xml]`), or a combination thereof (`junit: [rspec.xml, test-results/TEST-*.xml]`). -#### `artifacts:reports:codequality` **[STARTER]** +##### `artifacts:reports:codequality` **[STARTER]** > Introduced in GitLab 11.5. Requires GitLab Runner 11.5 and above. @@ -1296,51 +1464,51 @@ as artifacts. The collected Code Quality report will be uploaded to GitLab as an artifact and will be automatically shown in merge requests. -#### `artifacts:reports:sast` **[ULTIMATE]** +##### `artifacts:reports:sast` **[ULTIMATE]** > Introduced in GitLab 11.5. Requires GitLab Runner 11.5 and above. -The `sast` report collects [SAST vulnerabilities](https://docs.gitlab.com/ee/user/project/merge_requests/sast.html) +The `sast` report collects [SAST vulnerabilities](https://docs.gitlab.com/ee/user/application_security/sast/index.html) as artifacts. The collected SAST report will be uploaded to GitLab as an artifact and will be automatically shown in merge requests, pipeline view and provide data for security dashboards. -#### `artifacts:reports:dependency_scanning` **[ULTIMATE]** +##### `artifacts:reports:dependency_scanning` **[ULTIMATE]** > Introduced in GitLab 11.5. Requires GitLab Runner 11.5 and above. -The `dependency_scanning` report collects [Dependency Scanning vulnerabilities](https://docs.gitlab.com/ee/user/project/merge_requests/dependency_scanning.html) +The `dependency_scanning` report collects [Dependency Scanning vulnerabilities](https://docs.gitlab.com/ee/user/application_security/dependency_scanning/index.html) as artifacts. The collected Dependency Scanning report will be uploaded to GitLab as an artifact and will be automatically shown in merge requests, pipeline view and provide data for security dashboards. -#### `artifacts:reports:container_scanning` **[ULTIMATE]** +##### `artifacts:reports:container_scanning` **[ULTIMATE]** > Introduced in GitLab 11.5. Requires GitLab Runner 11.5 and above. -The `container_scanning` report collects [Container Scanning vulnerabilities](https://docs.gitlab.com/ee/user/project/merge_requests/container_scanning.html) +The `container_scanning` report collects [Container Scanning vulnerabilities](https://docs.gitlab.com/ee/user/application_security/container_scanning/index.html) as artifacts. The collected Container Scanning report will be uploaded to GitLab as an artifact and will be automatically shown in merge requests, pipeline view and provide data for security dashboards. -#### `artifacts:reports:dast` **[ULTIMATE]** +##### `artifacts:reports:dast` **[ULTIMATE]** > Introduced in GitLab 11.5. Requires GitLab Runner 11.5 and above. -The `dast` report collects [DAST vulnerabilities](https://docs.gitlab.com/ee/user/project/merge_requests/dast.html) +The `dast` report collects [DAST vulnerabilities](https://docs.gitlab.com/ee/user/application_security/dast/index.html) as artifacts. The collected DAST report will be uploaded to GitLab as an artifact and will be automatically shown in merge requests, pipeline view and provide data for security dashboards. -#### `artifacts:reports:license_management` **[ULTIMATE]** +##### `artifacts:reports:license_management` **[ULTIMATE]** > Introduced in GitLab 11.5. Requires GitLab Runner 11.5 and above. @@ -1351,7 +1519,7 @@ The collected License Management report will be uploaded to GitLab as an artifac be automatically shown in merge requests, pipeline view and provide data for security dashboards. -#### `artifacts:reports:performance` **[PREMIUM]** +##### `artifacts:reports:performance` **[PREMIUM]** > Introduced in GitLab 11.5. Requires GitLab Runner 11.5 and above. @@ -1361,7 +1529,17 @@ as artifacts. The collected Performance report will be uploaded to GitLab as an artifact and will be automatically shown in merge requests. -## `dependencies` +##### `artifacts:reports:metrics` **[PREMIUM]** + +> Introduced in GitLab 11.10. + +The `metrics` report collects [Metrics](../../ci/metrics_reports.md) +as artifacts. + +The collected Metrics report will be uploaded to GitLab as an artifact and will +be automatically shown in merge requests. + +### `dependencies` > Introduced in GitLab 8.6 and GitLab Runner v1.1.1. @@ -1420,7 +1598,7 @@ deploy: script: make deploy ``` -### When a dependent job will fail +#### When a dependent job will fail > Introduced in GitLab 10.3. @@ -1434,7 +1612,7 @@ You can ask your administrator to [flip this switch](../../administration/job_artifacts.md#validation-for-dependencies) and bring back the old behavior. -## `coverage` +### `coverage` > [Introduced][ce-7447] in GitLab 8.17. @@ -1454,7 +1632,7 @@ job1: coverage: '/Code coverage: \d+\.\d+/' ``` -## `retry` +### `retry` > [Introduced][ce-12909] in GitLab 9.5. > [Behaviour expanded](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/21758) @@ -1528,7 +1706,7 @@ Possible values for `when` are: - `missing_dependency_failure`: Retry if a dependency was missing. - `runner_unsupported`: Retry if the runner was unsupported. -## `parallel` +### `parallel` > [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/22631) in GitLab 11.5. @@ -1548,7 +1726,49 @@ test: parallel: 5 ``` -## `include` +### `trigger` **[PREMIUM]** + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/issues/8997) in [GitLab Premium](https://about.gitlab.com/pricing/) 11.8. + +`trigger` allows you to define downstream pipeline trigger. When a job created +from `trigger` definition is started by GitLab, a downstream pipeline gets +created. + +Learn more about [multi-project pipelines](../multi_project_pipelines.md#creating-multi-project-pipelines-from-gitlab-ciyml). + +#### Simple `trigger` syntax + +The most simple way to configure a downstream trigger to use `trigger` keyword +with a full path to a downstream project: + +```yaml +rspec: + stage: test + script: bundle exec rspec + +staging: + stage: deploy + trigger: my/deployment +``` + +#### Complex `trigger` syntax + +It is possible to configure a branch name that GitLab will use to create +a downstream pipeline with: + +```yaml +rspec: + stage: test + script: bundle exec rspec + +staging: + stage: deploy + trigger: + project: my/deployment + branch: stable +``` + +### `include` > - Introduced in [GitLab Premium](https://about.gitlab.com/pricing/) 10.5. > - Available for Starter, Premium and Ultimate since 10.6. @@ -1568,9 +1788,6 @@ TIP: **Tip:** Use merging to customize and override included CI/CD configurations with local definitions. -Recursive includes are not supported. Your external files should not use the -`include` keyword as it will be ignored. - NOTE: **Note:** Using YAML aliases across different YAML files sourced by `include` is not supported. You must only refer to aliases in the same file. Instead @@ -1585,7 +1802,12 @@ of using YAML anchors, you can use the [`extends` keyword](#extends). See [usage examples](#include-examples). -### `include:local` +NOTE: **Note:** +`.gitlab-ci.yml` configuration included by all methods is evaluated at pipeline creation. +The configuration is a snapshot in time and persisted in the database. Any changes to +referenced `.gitlab-ci.yml` configuration will not be reflected in GitLab until the next pipeline is created. + +#### `include:local` `include:local` includes a file from the same repository as `.gitlab-ci.yml`. It's referenced using full paths relative to the root directory (`/`). @@ -1594,6 +1816,9 @@ You can only use files that are currently tracked by Git on the same branch your configuration file is on. In other words, when using a `include:local`, make sure that both `.gitlab-ci.yml` and the local file are on the same branch. +All [nested includes](#nested-includes) will be executed in the scope of the same project, +so it is possible to use local, project, remote or template includes. + NOTE: **Note:** Including local files through Git submodules paths is not supported. @@ -1604,7 +1829,7 @@ include: - local: '/templates/.gitlab-ci-template.yml' ``` -### `include:file` +#### `include:file` > [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/issues/53903) in GitLab 11.7. @@ -1635,7 +1860,11 @@ include: file: '/templates/.gitlab-ci-template.yml' ``` -### `include:template` +All [nested includes](#nested-includes) will be executed in the scope of the target project, +so it is possible to use local (relative to target project), project, remote +or template includes. + +#### `include:template` > [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/issues/53445) in GitLab 11.7. @@ -1650,7 +1879,10 @@ include: - template: Auto-DevOps.gitlab-ci.yml ``` -### `include:remote` +All [nested includes](#nested-includes) will be executed only with the permission of the user, +so it is possible to use project, remote or template includes. + +#### `include:remote` `include:remote` can be used to include a file from a different location, using HTTP/HTTPS, referenced by using the full URL. The remote file must be @@ -1662,16 +1894,22 @@ include: - remote: 'https://gitlab.com/awesome-project/raw/master/.gitlab-ci-template.yml' ``` -NOTE: **Note for GitLab admins:** -In order to include files from another repository inside your local network, -you may need to enable the **Allow requests to the local network from hooks and services** checkbox -located in the **Admin area > Settings > Network > Outbound requests** section. +All nested includes will be executed without context as public user, so only another remote, +or public project, or template is allowed. + +#### Nested includes + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/issues/56836) in GitLab 11.9. -### `include` examples +Nested includes allow you to compose a set of includes. +A total of 50 includes is allowed. +Duplicate includes are considered a configuration error. + +#### `include` examples Here are a few more `include` examples. -#### Single string or array of multiple values +##### Single string or array of multiple values You can include your extra YAML file(s) either as a single string or an array of multiple values. The following examples are all valid. @@ -1725,7 +1963,7 @@ include: file: '/templates/.gitlab-ci-template.yml' ``` -#### Re-using a `before_script` template +##### Re-using a `before_script` template In the following example, the content of `.before-script-template.yml` will be automatically fetched and evaluated along with the content of `.gitlab-ci.yml`. @@ -1749,7 +1987,7 @@ rspec: - bundle exec rspec ``` -#### Overriding external template values +##### Overriding external template values The following example shows specific YAML-defined variables and details of the `production` job from an include file being customized in `.gitlab-ci.yml`. @@ -1769,7 +2007,7 @@ production: - deploy environment: name: production - url: https://$CI_PROJECT_PATH_SLUG.$AUTO_DEVOPS_DOMAIN + url: https://$CI_PROJECT_PATH_SLUG.$KUBE_INGRESS_BASE_DOMAIN only: - master ``` @@ -1834,11 +2072,56 @@ In this case, if `install_dependencies` and `deploy` were not repeated in `.gitlab-ci.yml`, they would not be part of the script for the `production` job in the combined CI configuration. -## `extends` +##### Using nested includes + +The examples below show how includes can be nested from different sources +using a combination of different methods. + +In this example, `.gitlab-ci.yml` includes local the file `/.gitlab-ci/another-config.yml`: + +```yaml +include: + - local: /.gitlab-ci/another-config.yml +``` + +The `/.gitlab-ci/another-config.yml` includes a template and the `/templates/docker-workflow.yml` file +from another project: + +```yaml +include: + - template: Bash.gitlab-ci.yml + - project: group/my-project + file: /templates/docker-workflow.yml +``` + +The `/templates/docker-workflow.yml` present in `group/my-project` includes two local files +of the `group/my-project`: + +```yaml +include: + - local: /templates/docker-build.yml + - local: /templates/docker-testing.yml +``` + +Our `/templates/docker-build.yml` present in `group/my-project` adds a `docker-build` job: + +```yaml +docker-build: + script: docker build -t my-image . +``` + +Our second `/templates/docker-test.yml` present in `group/my-project` adds a `docker-test` job: + +```yaml +docker-test: + script: docker run my-image /run/tests.sh +``` + +### `extends` > Introduced in GitLab 11.3. -`extends` defines an entry name that a job that uses `extends` is going to +`extends` defines entry names that a job that uses `extends` is going to inherit from. It is an alternative to using [YAML anchors](#anchors) and is a little @@ -1915,7 +2198,47 @@ spinach: script: rake spinach ``` -## Using `extends` and `include` together +It's also possible to use multiple parents for `extends`. +The algorithm used for merge is "closest scope wins", so keys +from the last member will always shadow anything defined on other levels. +For example: + +```yaml +.only-important: + only: + - master + - stable + tags: + - production + +.in-docker: + tags: + - docker + image: alpine + +rspec: + extends: + - .only-important + - .in-docker + script: + - rake rspec +``` + +This results in the following `rspec` job: + +```yaml +rspec: + only: + - master + - stable + tags: + - docker + image: alpine + script: + - rake rspec +``` + +### Using `extends` and `include` together `extends` works across configuration files combined with `include`. @@ -1940,7 +2263,7 @@ useTemplate: This will run a job called `useTemplate` that runs `echo Hello!` as defined in the `.template` job, and uses the `alpine` Docker image as defined in the local job. -## `pages` +### `pages` `pages` is a special job that is used to upload static content to GitLab that can be used to serve your website. It has a special syntax, so the two @@ -1969,7 +2292,7 @@ pages: Read more on [GitLab Pages user documentation](../../user/project/pages/index.md). -## `variables` +### `variables` > Introduced in GitLab Runner v0.5.0. @@ -1999,12 +2322,12 @@ Runner itself](../variables/README.md#predefined-environment-variables). One example would be `CI_COMMIT_REF_NAME` which has the value of the branch or tag name for which project is built. Apart from the variables you can set in `.gitlab-ci.yml`, there are also the so called -[Variables](../variables/README.md#variables) +[Variables](../variables/README.md#gitlab-cicd-environment-variables) which can be set in GitLab's UI. -[Learn more about variables and their priority.][variables] +Learn more about [variables and their priority][variables]. -### Git strategy +#### Git strategy > Introduced in GitLab 8.9 as an experimental feature. May change or be removed > completely in future releases. `GIT_STRATEGY=none` requires GitLab Runner @@ -2049,7 +2372,7 @@ NOTE: **Note:** `GIT_STRATEGY` is not supported for but may be in the future. See the [support Git strategy with Kubernetes executor feature proposal](https://gitlab.com/gitlab-org/gitlab-runner/issues/3847) for updates. -### Git submodule strategy +#### Git submodule strategy > Requires GitLab Runner v1.10+. @@ -2087,9 +2410,9 @@ Note that for this feature to work correctly, the submodules must be configured - a relative path to another repository on the same GitLab server. See the [Git submodules](../git_submodules.md) documentation. -### Git checkout +#### Git checkout -> Introduced in GitLab Runner 9.3 +> Introduced in GitLab Runner 9.3. The `GIT_CHECKOUT` variable can be used when the `GIT_STRATEGY` is set to either `clone` or `fetch` to specify whether a `git checkout` should be run. If not @@ -2112,11 +2435,38 @@ variables: GIT_STRATEGY: clone GIT_CHECKOUT: "false" script: - - git checkout master - - git merge $CI_BUILD_REF_NAME + - git checkout -B master origin/master + - git merge $CI_COMMIT_SHA +``` + +#### Git clean flags + +> Introduced in GitLab Runner 11.10 + +The `GIT_CLEAN_FLAGS` variable is used to control the default behavior of +`git clean` after checking out the sources. You can set it globally or per-job in the +[`variables`](#variables) section. + +`GIT_CLEAN_FLAGS` accepts all possible options of the [git clean](https://git-scm.com/docs/git-clean) +command. + +`git clean` is disabled if `GIT_CHECKOUT: "false"` is specified. + +If `GIT_CLEAN_FLAGS` is: + +- Not specified, `git clean` flags default to `-ffdx`. +- Given the value `none`, `git clean` is not executed. + +For example: + +```yaml +variables: + GIT_CLEAN_FLAGS: -ffdx -e cache/ +script: + - ls -al cache/ ``` -### Job stages attempts +#### Job stages attempts > Introduced in GitLab, it requires GitLab Runner v1.9+. @@ -2140,7 +2490,7 @@ variables: You can set them globally or per-job in the [`variables`](#variables) section. -### Shallow cloning +#### Shallow cloning > Introduced in GitLab 8.9 as an experimental feature. May change in future releases or be removed completely. @@ -2150,7 +2500,7 @@ shallow cloning of the repository which can significantly speed up cloning for repositories with a large number of commits or old, large binaries. The value is passed to `git fetch` and `git clone`. ->**Note:** +NOTE: **Note:** If you use a depth of 1 and have a queue of jobs or retry jobs, jobs may fail. @@ -2173,6 +2523,88 @@ variables: You can set it globally or per-job in the [`variables`](#variables) section. +## Deprecated parameters + +The following parameters are deprecated. + +### `types` + +CAUTION: **Deprecated:** +`types` is deprecated, and could be removed in a future release. +Use [`stages`](#stages) instead. + +### `type` + +CAUTION: **Deprecated:** +`type` is deprecated, and could be removed in one of the future releases. +Use [`stage`](#stage) instead. + +## Custom build directories + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-runner/merge_requests/1267) in Gitlab Runner 11.10 + +NOTE: **Note:** +This can only be used when `custom_build_dir` is enabled in the [Runner's +configuration](https://docs.gitlab.com/runner/configuration/advanced-configuration.html#the-runnerscustom_build_dir-section). +This is the default configuration for `docker` and `kubernetes` executor. + +By default, GitLab Runner clones the repository in a unique subpath of the +`$CI_BUILDS_DIR` directory. However, your project might require the code in a +specific directory (Go projects, for example). In that case, you can specify +the `GIT_CLONE_PATH` variable to tell the Runner in which directory to clone the +repository: + +```yml +variables: + GIT_CLONE_PATH: $CI_BUILDS_DIR/project-name + +test: + script: + - pwd +``` + +The `GIT_CLONE_PATH` has to always be within `$CI_BUILDS_DIR`. The directory set in `$CI_BUILDS_DIR` +is dependent on executor and configuration of [runners.builds_dir](https://docs.gitlab.com/runner/configuration/advanced-configuration.html#the-runners-section) +setting. + +### Handling concurrency + +An executor using a concurrency greater than `1` might lead +to failures because multiple jobs might be working on the same directory if the `builds_dir` +is shared between jobs. +GitLab Runner does not try to prevent this situation. It is up to the administrator +and developers to comply with the requirements of Runner configuration. + +To avoid this scenario, you can use a unique path within `$CI_BUILDS_DIR`, because Runner +exposes two additional variables that provide a unique `ID` of concurrency: + +- `$CI_CONCURRENT_ID`: Unique ID for all jobs running within the given executor. +- `$CI_CONCURRENT_PROJECT_ID`: Unique ID for all jobs running within the given executor and project. + +The most stable configuration that should work well in any scenario and on any executor +is to use `$CI_CONCURRENT_ID` in the `GIT_CLONE_PATH`. For example: + +```yml +variables: + GIT_CLONE_PATH: $CI_BUILDS_DIR/$CI_CONCURRENT_ID/project-name + +test: + script: + - pwd +``` + +The `$CI_CONCURRENT_PROJECT_ID` should be used in conjunction with `$CI_PROJECT_PATH` +as the `$CI_PROJECT_PATH` provides a path of a repository. That is, `group/subgroup/project`. For example: + +```yml +variables: + GIT_CLONE_PATH: $CI_BUILDS_DIR/$CI_CONCURRENT_ID/$CI_PROJECT_PATH + +test: + script: + - pwd +``` + ## Special YAML features It's possible to use special YAML features like anchors (`&`), aliases (`*`) @@ -2332,10 +2764,21 @@ You can see that the hidden keys are conveniently used as templates. ## Triggers Triggers can be used to force a rebuild of a specific branch, tag or commit, -with an API call. +with an API call when a pipeline gets created using a trigger token. + +Not to be confused with [`trigger`](#trigger-premium). [Read more in the triggers documentation.](../triggers/README.md) +## Processing Git pushes + +GitLab will create at most 4 branch and tags pipelines when +doing pushing multiple changes in single `git push` invocation. + +This limitation does not affect any of the updated Merge Request pipelines, +all updated Merge Requests will have a pipeline created when using +[pipelines for merge requests](../merge_request_pipelines/index.md). + ## Skipping jobs If your commit message contains `[ci skip]` or `[skip ci]`, using any @@ -2348,28 +2791,25 @@ using Git 2.10 or newer: git push -o ci.skip ``` -## Validate the .gitlab-ci.yml - -Each instance of GitLab CI has an embedded debug tool called Lint, which validates the -content of your `.gitlab-ci.yml` files. You can find the Lint under the page `ci/lint` of your -project namespace (e.g, `http://gitlab-example.com/gitlab-org/project-123/-/ci/lint`) - -## Using reserved keywords - -If you get validation error when using specific values (e.g., `true` or `false`), -try to quote them, or change them to a different form (e.g., `/bin/true`). +<!-- ## Troubleshooting -## Examples +Include any troubleshooting steps that you can foresee. If you know beforehand what issues +one might have when setting this up, or when something is changed, or on upgrading, it's +important to describe those, too. Think of things that may go wrong and include them here. +This is important to minimize requests for support, and to avoid doc comments with +questions that you know someone might ask. -See a [list of examples](../examples/README.md "CI/CD examples") for using -GitLab CI/CD with various languages. +Each scenario can be a third-level heading, e.g. `### Getting error message X`. +If you have none to add when creating a doc, leave this section in place +but commented out to help encourage others to add to it in the future. --> [ce-6323]: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/6323 [ce-6669]: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/6669 [ce-7983]: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/7983 [ce-7447]: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/7447 [ce-12909]: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/12909 +[ce-19232]: https://gitlab.com/gitlab-org/gitlab-ce/issues/19232 [environment]: ../environments.md "CI/CD environments" [schedules]: ../../user/project/pipelines/schedules.md "Pipelines schedules" [variables]: ../variables/README.md "CI/CD variables" -[push-option]: https://git-scm.com/docs/git-push#git-push--oltoptiongt +[push-option]: https://git-scm.com/docs/git-push#Documentation/git-push.txt--oltoptiongt diff --git a/doc/container_registry/README.md b/doc/container_registry/README.md index 5d2f5edcb18..b31870df36d 100644 --- a/doc/container_registry/README.md +++ b/doc/container_registry/README.md @@ -1 +1,5 @@ +--- +redirect_to: '../user/project/container_registry.md' +--- + This document was moved to [another location](../user/project/container_registry.md). diff --git a/doc/container_registry/troubleshooting.md b/doc/container_registry/troubleshooting.md index 2f8cd37b488..c99d7011ac2 100644 --- a/doc/container_registry/troubleshooting.md +++ b/doc/container_registry/troubleshooting.md @@ -1 +1,5 @@ +--- +redirect_to: '../user/project/container_registry.md' +--- + This document was moved to [user/project/container_registry](../user/project/container_registry.md). diff --git a/doc/customization/issue_and_merge_request_template.md b/doc/customization/issue_and_merge_request_template.md new file mode 100644 index 00000000000..adaa120a37e --- /dev/null +++ b/doc/customization/issue_and_merge_request_template.md @@ -0,0 +1,5 @@ +--- +redirect_to: '../user/project/description_templates.md#setting-a-default-template-for-issues-and-merge-requests--starter' +--- + +This document was moved to [description_templates](../user/project/description_templates.md#setting-a-default-template-for-issues-and-merge-requests--starter). diff --git a/doc/customization/system_header_and_footer_messages.md b/doc/customization/system_header_and_footer_messages.md index cf7d8b3f3e8..9d6931c730d 100644 --- a/doc/customization/system_header_and_footer_messages.md +++ b/doc/customization/system_header_and_footer_messages.md @@ -7,6 +7,10 @@ Navigate to the **Admin** area and go to the **Appearance** page. Under **System header and footer** insert your header message and/or footer message. Both background and font color of the header and footer are customizable. +You can also apply the header and footer messages to gitlab emails, +by checking the **Enable header and footer in emails** checkbox. +Note that color settings will only be applied within the app interface and not to emails +  After saving, all GitLab pages will contain the custom system header and/or footer messages: diff --git a/doc/customization/system_header_and_footer_messages/appearance.png b/doc/customization/system_header_and_footer_messages/appearance.png Binary files differindex 14667f751c4..fd315bb6c07 100644 --- a/doc/customization/system_header_and_footer_messages/appearance.png +++ b/doc/customization/system_header_and_footer_messages/appearance.png diff --git a/doc/customization/system_header_and_footer_messages/custom_header_footer.png b/doc/customization/system_header_and_footer_messages/custom_header_footer.png Binary files differindex 691ca8a3484..15d19c87dd7 100644 --- a/doc/customization/system_header_and_footer_messages/custom_header_footer.png +++ b/doc/customization/system_header_and_footer_messages/custom_header_footer.png diff --git a/doc/customization/system_header_and_footer_messages/sign_up_custom_header_and_footer.png b/doc/customization/system_header_and_footer_messages/sign_up_custom_header_and_footer.png Binary files differindex a9c59bc9f62..51349ddf6f4 100644 --- a/doc/customization/system_header_and_footer_messages/sign_up_custom_header_and_footer.png +++ b/doc/customization/system_header_and_footer_messages/sign_up_custom_header_and_footer.png diff --git a/doc/customization/welcome_message.md b/doc/customization/welcome_message.md index 9194f847cdf..45f1fed355e 100644 --- a/doc/customization/welcome_message.md +++ b/doc/customization/welcome_message.md @@ -1 +1,5 @@ +--- +redirect_to: 'branded_login_page.md' +--- + This document was moved to [another location](branded_login_page.md). diff --git a/doc/development/README.md b/doc/development/README.md index 13646cbfe48..d2f09fc01de 100644 --- a/doc/development/README.md +++ b/doc/development/README.md @@ -20,6 +20,7 @@ description: 'Learn how to contribute to GitLab.' - [Automatic CE->EE merge](automatic_ce_ee_merge.md) - [Guidelines for implementing Enterprise Edition features](ee_features.md) - [Security process for developers](https://gitlab.com/gitlab-org/release/docs/blob/master/general/security/developer.md#security-releases-critical-non-critical-as-a-developer) +- [Requesting access to Chatops on GitLab.com](chatops_on_gitlabcom.md#requesting-access) (for GitLabbers) ## UX and frontend guides @@ -38,22 +39,28 @@ description: 'Learn how to contribute to GitLab.' - [Sidekiq guidelines](sidekiq_style_guide.md) for working with Sidekiq workers - [Working with Gitaly](gitaly.md) - [Manage feature flags](feature_flags.md) +- [Licensed feature availability](licensed_feature_availability.md) - [View sent emails or preview mailers](emails.md) - [Shell commands](shell_commands.md) in the GitLab codebase - [`Gemfile` guidelines](gemfile.md) - [Pry debugging](pry_debugging.md) - [Sidekiq debugging](sidekiq_debugging.md) +- [Accessing session data](session.md) - [Gotchas](gotchas.md) to avoid - [Avoid modules with instance variables](module_with_instance_variables.md) if possible - [How to dump production data to staging](db_dump.md) - [Working with the GitHub importer](github_importer.md) - [Import/Export development documentation](import_export.md) +- [Elasticsearch integration docs](elasticsearch.md) - [Working with Merge Request diffs](diffs.md) - [Kubernetes integration guidelines](kubernetes.md) - [Permissions](permissions.md) - [Prometheus metrics](prometheus_metrics.md) - [Guidelines for reusing abstractions](reusing_abstractions.md) - [DeclarativePolicy framework](policies.md) +- [How Git object deduplication works in GitLab](git_object_deduplication.md) +- [Geo development](geo.md) +- [Routing](routing.md) ## Performance guides @@ -100,6 +107,10 @@ description: 'Learn how to contribute to GitLab.' - [Query Count Limits](query_count_limits.md) - [Database helper modules](database_helpers.md) +## Integration guides + +- [Jira Connect app](integrations/jira_connect.md) + ## Testing guides - [Testing standards and style guidelines](testing_guide/index.md) @@ -128,3 +139,8 @@ description: 'Learn how to contribute to GitLab.' ## Go guides - [Go Guidelines](go_guide/index.md) + +## Other GitLab Development Kit (GDK) guides + +- [Run full Auto DevOps cycle in a GDK instance](https://gitlab.com/gitlab-org/gitlab-development-kit/blob/master/doc/howto/auto_devops.md) +- [Using GitLab Runner with GDK](https://gitlab.com/gitlab-org/gitlab-development-kit/blob/master/doc/howto/runner.md) diff --git a/doc/development/api_graphql_styleguide.md b/doc/development/api_graphql_styleguide.md index 95722c027ba..38270af682e 100644 --- a/doc/development/api_graphql_styleguide.md +++ b/doc/development/api_graphql_styleguide.md @@ -9,28 +9,6 @@ can be shared. It is also possible to add a `private_token` to the querystring, or add a `HTTP_PRIVATE_TOKEN` header. -### Authorization - -Fields can be authorized using the same abilities used in the Rails -app. This can be done using the `authorize` helper: - -```ruby -module Types - class QueryType < BaseObject - graphql_name 'Query' - - field :project, Types::ProjectType, null: true, resolver: Resolvers::ProjectResolver do - authorize :read_project - end - end -``` - -The object found by the resolve call is used for authorization. - -This works for authorizing a single record, for authorizing -collections, we should only load what the currently authenticated user -is allowed to view. Preferably we use our existing finders for that. - ## Types When exposing a model through the GraphQL API, we do so by creating a @@ -54,6 +32,21 @@ a new presenter specifically for GraphQL. The presenter is initialized using the object resolved by a field, and the context. +### Exposing Global ids + +When exposing an `id` field on a type, we will by default try to +expose a global id by calling `to_global_id` on the resource being +rendered. + +To override this behaviour, you can implement an `id` method on the +type for which you are exposing an id. Please make sure that when +exposing a `GraphQL::ID_TYPE` using a custom method that it is +globally unique. + +The records that are exposing a `full_path` as an `ID_TYPE` are one of +these exceptions. Since the full path is a unique identifier for a +`Project` or `Namespace`. + ### Connection Types GraphQL uses [cursor based @@ -101,14 +94,14 @@ look like this: { "cursor": "Nzc=", "node": { - "id": "77", + "id": "gid://gitlab/Pipeline/77", "status": "FAILED" } }, { "cursor": "Njc=", "node": { - "id": "67", + "id": "gid://gitlab/Pipeline/67", "status": "FAILED" } } @@ -187,6 +180,114 @@ end policies at once. The fields for these will all have be non-nullable booleans with a default description. +## Authorization + +Authorizations can be applied to both types and fields using the same +abilities as in the Rails app. + +If the: + +- Currently authenticated user fails the authorization, the authorized +resource will be returned as `null`. +- Resource is part of a collection, the collection will be filtered to +exclude the objects that the user's authorization checks failed against. + +TIP: **Tip:** +Try to load only what the currently authenticated user is allowed to +view with our existing finders first, without relying on authorization +to filter the records. This minimizes database queries and unnecessary +authorization checks of the loaded records. + +### Type authorization + +Authorize a type by passing an ability to the `authorize` method. All +fields with the same type will be authorized by checking that the +currently authenticated user has the required ability. + +For example, the following authorization ensures that the currently +authenticated user can only see projects that they have the +`read_project` ability for (so long as the project is returned in a +field that uses `Types::ProjectType`): + +```ruby +module Types + class ProjectType < BaseObject + authorize :read_project + end +end +``` + +You can also authorize against multiple abilities, in which case all of +the ability checks must pass. + +For example, the following authorization ensures that the currently +authenticated user must have `read_project` and `another_ability` +abilities to see a project: + +```ruby +module Types + class ProjectType < BaseObject + authorize [:read_project, :another_ability] + end +end +``` + +### Field authorization + +Fields can be authorized with the `authorize` option. + +For example, the following authorization ensures that the currently +authenticated user must have the `owner_access` ability to see the +project: + +```ruby +module Types + class MyType < BaseObject + field :project, Types::ProjectType, null: true, resolver: Resolvers::ProjectResolver, authorize: :owner_access + end +end +``` + +Fields can also be authorized against multiple abilities, in which case +all of ability checks must pass. **Note:** This requires explicitly +passing a block to `field`: + +```ruby +module Types + class MyType < BaseObject + field :project, Types::ProjectType, null: true, resolver: Resolvers::ProjectResolver do + authorize [:owner_access, :another_ability] + end + end +end +``` + +NOTE: **Note:** If the field's type already [has a particular +authorization](#type-authorization) then there is no need to add that +same authorization to the field. + +### Type and Field authorizations together + +Authorizations are cumulative, so where authorizations are defined on +a field, and also on the field's type, then the currently authenticated +user would need to pass all ability checks. + +In the following simplified example the currently authenticated user +would need both `first_permission` and `second_permission` abilities in +order to see the author of the issue. + +```ruby +class UserType + authorize :first_permission +end +``` + +```ruby +class IssueType + field :author, UserType, authorize: :second_permission +end +``` + ## Resolvers To find objects to display in a field, we can add resolvers to @@ -244,7 +345,7 @@ argument :project_path, GraphQL::ID_TYPE, required: true, description: "The project the merge request to mutate is in" -argument :iid, GraphQL::ID_TYPE, +argument :iid, GraphQL::STRING_TYPE, required: true, description: "The iid of the merge request to mutate" diff --git a/doc/development/architecture.md b/doc/development/architecture.md index e22552fd3a3..650325121b2 100644 --- a/doc/development/architecture.md +++ b/doc/development/architecture.md @@ -10,34 +10,147 @@ For information, see the [GitLab Release Process](https://gitlab.com/gitlab-org/ Both EE and CE require some add-on components called gitlab-shell and Gitaly. These components are available from the [gitlab-shell](https://gitlab.com/gitlab-org/gitlab-shell/tree/master) and [gitaly](https://gitlab.com/gitlab-org/gitaly/tree/master) repositories respectively. New versions are usually tags but staying on the master branch will give you the latest stable version. New releases are generally around the same time as GitLab CE releases with exception for informal security updates deemed critical. -## GitLab Omnibus Component by Component +## Components -This document is designed to be consumed by systems adminstrators and GitLab Support Engineers who want to understand more about the internals of GitLab and how they work together. +A typical install of GitLab will be on GNU/Linux. It uses Nginx or Apache as a web front end to proxypass the Unicorn web server. By default, communication between Unicorn and the front end is via a Unix domain socket but forwarding requests via TCP is also supported. The web front end accesses `/home/git/gitlab/public` bypassing the Unicorn server to serve static pages, uploads (e.g. avatar images or attachments), and precompiled assets. GitLab serves web pages and a [GitLab API](https://gitlab.com/gitlab-org/gitlab-ce/tree/master/doc/api) using the Unicorn web server. It uses Sidekiq as a job queue which, in turn, uses redis as a non-persistent database backend for job information, meta data, and incoming jobs. -When deployed, GitLab should be considered the amalgamation of the below processes. When troubleshooting or debugging, be as specific as possible as to which component you are referencing. That should increase clarity and reduce confusion. +We also support deploying GitLab on Kubernetes using our [gitlab Helm chart](https://docs.gitlab.com/charts/). -### GitLab Process Descriptions +The GitLab web app uses MySQL or PostgreSQL for persistent database information (e.g. users, permissions, issues, other meta data). GitLab stores the bare git repositories it serves in `/home/git/repositories` by default. It also keeps default branch and hook information with the bare repository. -As of this writing, a fresh GitLab 11.3.0 install will show the following processes with `gitlab-ctl status`: +When serving repositories over HTTP/HTTPS GitLab utilizes the GitLab API to resolve authorization and access as well as serving git objects. +The add-on component gitlab-shell serves repositories over SSH. It manages the SSH keys within `/home/git/.ssh/authorized_keys` which should not be manually edited. gitlab-shell accesses the bare repositories through Gitaly to serve git objects and communicates with redis to submit jobs to Sidekiq for GitLab to process. gitlab-shell queries the GitLab API to determine authorization and access. + +Gitaly executes git operations from gitlab-shell and the GitLab web app, and provides an API to the GitLab web app to get attributes from git (e.g. title, branches, tags, other meta data), and to get blobs (e.g. diffs, commits, files). + +You may also be interested in the [production architecture of GitLab.com](https://about.gitlab.com/handbook/engineering/infrastructure/production-architecture/). + +### Component diagram + +```mermaid +graph TB + + HTTP[HTTP/HTTPS] -- TCP 80, 443 --> NGINX[NGINX] + SSH -- TCP 22 --> GitLabShell[GitLab Shell] + SMTP[SMTP Gateway] + Geo[GitLab Geo Node] -- TCP 22, 80, 443 --> NGINX + + GitLabShell --TCP 8080 -->Unicorn["Unicorn (GitLab Rails)"] + GitLabShell --> Gitaly + GitLabShell --> Redis + Unicorn --> PgBouncer[PgBouncer] + Unicorn --> Redis + Unicorn --> Gitaly + Redis --> Sidekiq + Sidekiq["Sidekiq (GitLab Rails, ES Indexer)"] --> PgBouncer + GitLabWorkhorse[GitLab Workhorse] --> Unicorn + GitLabWorkhorse --> Redis + GitLabWorkhorse --> Gitaly + Gitaly --> Redis + NGINX --> GitLabWorkhorse + NGINX -- TCP 8090 --> GitLabPages[GitLab Pages] + NGINX --> Grafana[Grafana] + Grafana -- TCP 9090 --> Prometheus[Prometheus] + Prometheus -- TCP 80, 443 --> Unicorn + RedisExporter[Redis Exporter] --> Redis + Prometheus -- TCP 9121 --> RedisExporter + PostgreSQLExporter[PostgreSQL Exporter] --> PostgreSQL + PgBouncerExporter[PgBouncer Exporter] --> PgBouncer + Prometheus -- TCP 9187 --> PostgreSQLExporter + Prometheus -- TCP 9100 --> NodeExporter[Node Exporter] + Prometheus -- TCP 9168 --> GitLabMonitor[GitLab Monitor] + Prometheus -- TCP 9127 --> PgBouncerExporter + GitLabMonitor --> PostgreSQL + GitLabMonitor --> GitLabShell + GitLabMonitor --> Sidekiq + PgBouncer --> Consul + PostgreSQL --> Consul + PgBouncer --> PostgreSQL + NGINX --> Registry + Unicorn --> Registry + NGINX --> Mattermost + Mattermost --- Unicorn + Prometheus --> Alertmanager + Migrations --> PostgreSQL + Runner -- TCP 443 --> NGINX + Unicorn -- TCP 9200 --> ElasticSearch + Sidekiq -- TCP 9200 --> ElasticSearch + Sidekiq -- TCP 80, 443 --> Sentry + Unicorn -- TCP 80, 443 --> Sentry + Sidekiq -- UDP 6831 --> Jaeger + Unicorn -- UDP 6831 --> Jaeger + Gitaly -- UDP 6831 --> Jaeger + GitLabShell -- UDP 6831 --> Jaeger + GitLabWorkhorse -- UDP 6831 --> Jaeger + Alertmanager -- TCP 25 --> SMTP + Sidekiq -- TCP 25 --> SMTP + Unicorn -- TCP 25 --> SMTP + Unicorn -- TCP 369 --> LDAP + Sidekiq -- TCP 369 --> LDAP + Unicorn -- TCP 443 --> ObjectStorage["Object Storage"] + Sidekiq -- TCP 443 --> ObjectStorage + GitLabWorkhorse -- TCP 443 --> ObjectStorage + Registry -- TCP 443 --> ObjectStorage + Geo -- TCP 5432 --> PostgreSQL ``` -run: alertmanager: (pid 30829) 14207s; run: log: (pid 13906) 2432044s -run: gitaly: (pid 30771) 14210s; run: log: (pid 13843) 2432046s -run: gitlab-monitor: (pid 30788) 14209s; run: log: (pid 13868) 2432045s -run: gitlab-workhorse: (pid 30758) 14210s; run: log: (pid 13855) 2432046s -run: logrotate: (pid 30246) 3407s; run: log: (pid 13825) 2432047s -run: nginx: (pid 30849) 14207s; run: log: (pid 13856) 2432046s -run: node-exporter: (pid 30929) 14206s; run: log: (pid 13877) 2432045s -run: postgres-exporter: (pid 30935) 14206s; run: log: (pid 13931) 2432044s -run: postgresql: (pid 13133) 2432214s; run: log: (pid 13848) 2432046s -run: prometheus: (pid 30807) 14209s; run: log: (pid 13884) 2432045s -run: redis: (pid 30560) 14274s; run: log: (pid 13807) 2432047s -run: redis-exporter: (pid 30946) 14205s; run: log: (pid 13869) 2432045s -run: sidekiq: (pid 30953) 14205s; run: log: (pid 13810) 2432047s -run: unicorn: (pid 30960) 14204s; run: log: (pid 13809) 2432047s -``` -### Layers +### Component legend + +* ✅ - Installed by default +* ⚙ - Requires additional configuration, or GitLab Managed Apps +* ⤓ - Manual installation required +* ❌ - Not supported or no instructions available + +Component statuses are linked to configuration documentation for each component. + +### Component list + +| Component | Description | [Omnibus GitLab](https://docs.gitlab.com/omnibus/) | [GitLab chart](https://docs.gitlab.com/charts/) | [Minikube Minimal](https://docs.gitlab.com/charts/development/minikube/#deploying-gitlab-with-minimal-settings) | [GitLab.com](https://gitlab.com) | [Source](../install/installation.md) | [GDK](https://gitlab.com/gitlab-org/gitlab-development-kit) | CE/EE | +| --------- | ----------- |:--------------------:|:------------------:|:-----:|:--------:|:--------:|:-------:|:-------:| +| [NGINX](#nginx) | Routes requests to appropriate components, terminates SSL | [✅][nginx-omnibus] | [✅][nginx-charts] | [⚙][nginx-charts] | [✅](https://about.gitlab.com/handbook/engineering/infrastructure/production-architecture/#service-architecture) | [⤓][nginx-source] | ❌ | CE & EE | +| [Unicorn (GitLab Rails)](#unicorn) | Handles requests for the web interface and API | [✅][unicorn-omnibus] | [✅][unicorn-charts] | [✅][unicorn-charts] | [✅](../user/gitlab_com/index.md#unicorn) | [⚙][unicorn-source] | [✅][gitlab-yml] | CE & EE | +| [Sidekiq](#sidekiq) | Background jobs processor | [✅][sidekiq-omnibus] | [✅][sidekiq-charts] | [✅](https://docs.gitlab.com/charts/charts/gitlab/sidekiq/index.html) | [✅](../user/gitlab_com/index.md#sidekiq) | [✅][gitlab-yml] | [✅][gitlab-yml] | CE & EE | +| [Gitaly](#gitaly) | Git RPC service for handling all git calls made by GitLab | [✅][gitaly-omnibus] | [✅][gitaly-charts] | [✅][gitaly-charts] | [✅](https://about.gitlab.com/handbook/engineering/infrastructure/production-architecture/#service-architecture) | [⚙][gitaly-source] | ✅ | CE & EE | +| [GitLab Workhorse](#gitlab-workhorse) | Smart reverse proxy, handles large HTTP requests | [✅][workhorse-omnibus] | [✅][workhorse-charts] | [✅][workhorse-charts] | [✅](https://about.gitlab.com/handbook/engineering/infrastructure/production-architecture/#service-architecture) | [⚙][workhorse-source] | ✅ | CE & EE | +| [GitLab Shell](#gitlab-shell) | Handles `git` over SSH sessions | [✅][shell-omnibus] | [✅][shell-charts] | [✅][shell-charts] | [✅](https://about.gitlab.com/handbook/engineering/infrastructure/production-architecture/#service-architecture) | [⚙][shell-source] | [✅][gitlab-yml] | CE & EE | +| [GitLab Pages](#gitlab-pages) | Hosts static websites | [⚙][pages-omnibus] | [❌][pages-charts] | [❌][pages-charts] | [✅](../user/gitlab_com/index.md#gitlab-pages) | [⚙][pages-source] | [⚙][pages-gdk] | CE & EE | +| [Registry](#registry) | Container registry, allows pushing and pulling of images | [⚙][registry-omnibus] | [✅][registry-charts] | [✅][registry-charts] | [✅](../user/project/container_registry.md#build-and-push-images) | [⤓][registry-source] | [⚙][registry-gdk] | CE & EE | +| [Redis](#redis) | Caching service | [✅][redis-omnibus] | [✅][redis-omnibus] | [✅][redis-charts] | [✅](https://about.gitlab.com/handbook/engineering/infrastructure/production-architecture/#service-architecture) | [⤓][redis-source] | ✅ | CE & EE | +| [PostgreSQL](#postgresql) | Database | [✅][postgres-omnibus] | [✅][postgres-charts] | [✅][postgres-charts] | [✅](../user/gitlab_com/index.md#postgresql) | [⤓][postgres-source] | ✅ | CE & EE | +| [PgBouncer](#pgbouncer) | Database connection pooling, failover | [⚙][pgbouncer-omnibus] | [❌][pgbouncer-charts] | [❌][pgbouncer-charts] | [✅](https://about.gitlab.com/handbook/engineering/infrastructure/production-architecture/#database-architecture) | ❌ | ❌ | EE Only | +| [Consul](#consul) | Database node discovery, failover | [⚙][consul-omnibus] | [❌][consul-charts] | [❌][consul-charts] | [✅](../user/gitlab_com/index.md#consul) | ❌ | ❌ | EE Only | +| [GitLab self-monitoring: Prometheus](#prometheus) | Time-series database, metrics collection, and query service | [✅][prometheus-omnibus] | [✅][prometheus-charts] | [⚙][prometheus-charts] | [✅](../user/gitlab_com/index.md#prometheus) | ❌ | ❌ | CE & EE | +| [GitLab self-monitoring: Alertmanager](#alertmanager) | Deduplicates, groups, and routes alerts from Prometheus | [✅][alertmanager-omnibus] | [✅][alertmanager-charts] | [⚙][alertmanager-charts] | [✅](https://about.gitlab.com/handbook/engineering/monitoring/) | ❌ | ❌ | CE & EE | +| [GitLab self-monitoring: Grafana](#grafana) | Metrics dashboard | [⚙][grafana-omnibus] | [⤓][grafana-charts] | [⤓][grafana-charts] | [✅](https://dashboards.gitlab.com/d/RZmbBr7mk/gitlab-triage?refresh=30s) | ❌ | ❌ | CE & EE | +| [GitLab self-monitoring: Sentry](#sentry) | Track errors generated by the GitLab instance | [⤓][sentry-omnibus] | [❌][sentry-charts] | [❌][sentry-charts] | [✅](https://about.gitlab.com/handbook/support/workflows/services/gitlab_com/500_errors.html#searching-sentry) | [⤓][gitlab-yml] | [⤓][gitlab-yml] | CE & EE | +| [GitLab self-monitoring: Jaeger](#jaeger) | View traces generated by the GitLab instance | [❌][jaeger-omnibus] | [❌][jaeger-charts] | [❌][jaeger-charts] | [❌](https://gitlab.com/gitlab-org/omnibus-gitlab/issues/4104) | [⤓][jaeger-source] | [⚙][jaeger-gdk] | CE & EE | +| [Redis Exporter](#redis-exporter) | Prometheus endpoint with Redis metrics | [✅][redis-exporter-omnibus] | [✅][redis-exporter-charts] | [✅][redis-exporter-charts] | [✅](https://about.gitlab.com/handbook/engineering/monitoring/) | ❌ | ❌ | CE & EE | +| [Postgres Exporter](#postgres-exporter) | Prometheus endpoint with PostgreSQL metrics | [✅][postgres-exporter-omnibus] | [✅][postgres-exporter-charts] | [✅][postgres-exporter-charts] | [✅](https://about.gitlab.com/handbook/engineering/monitoring/) | ❌ | ❌ | CE & EE | +| [PgBouncer Exporter](#pgbouncer-exporter) | Prometheus endpoint with PgBouncer metrics | [⚙][pgbouncer-exporter-omnibus] | [❌][pgbouncer-exporter-charts] | [❌][pgbouncer-exporter-charts] | [✅](https://about.gitlab.com/handbook/engineering/monitoring/) | ❌ | ❌ | CE & EE | +| [GitLab Monitor](#gitlab-monitor) | Generates a variety of GitLab metrics | [✅][gitlab-monitor-omnibus] | [❌][gitab-monitor-charts] | [❌][gitab-monitor-charts] | [✅](https://about.gitlab.com/handbook/engineering/monitoring/) | ❌ | ❌ | CE & EE | +| [Node Exporter](#node-exporter) | Prometheus endpoint with system metrics | [✅][node-exporter-omnibus] | [❌][node-exporter-charts] | [❌][node-exporter-charts] | [✅](https://about.gitlab.com/handbook/engineering/monitoring/) | ❌ | ❌ | CE & EE | +| [Mattermost](#mattermost) | Open-source Slack alternative | [⚙][mattermost-omnibus] | [⤓][mattermost-charts] | [⤓][mattermost-charts] | [⤓](../user/project/integrations/mattermost_slash_commands.md#manual-configuration), [⤓](../user/project/integrations/mattermost.html) | ❌ | ❌ | CE & EE | +| [MinIO](#minio) | Object storage service | [⤓][minio-omnibus] | [✅][minio-charts] | [✅][minio-charts] | [✅](https://about.gitlab.com/handbook/engineering/infrastructure/production-architecture/#storage-architecture) | ❌ | [⚙][minio-gdk] | CE & EE | +| [Runner](#gitlab-runner) | Executes GitLab CI jobs | [⤓][runner-omnibus] | [✅][runner-charts] | [⚙][runner-charts] | [✅](../user/gitlab_com/index.md#shared-runners) | [⚙][runner-source] | [⚙][runner-gdk] | CE & EE | +| [Database Migrations](#database-migrations) | Database migrations | [✅][database-migrations-omnibus] | [✅][database-migrations-charts] | [✅][database-migrations-charts] | ✅ | [⚙][database-migrations-source] | ✅ | CE & EE | +| [Certificate Management](#certificate-management) | TLS Settings, Let's Encrypt | [✅][certificate-management-omnibus] | [✅][certificate-management-charts] | [⚙][certificate-management-charts] | [✅](https://about.gitlab.com/handbook/engineering/infrastructure/production-architecture/#secrets-management) | [⚙][certificate-management-source] | [⚙][certificate-management-gdk] | CE & EE | +| [GitLab Geo Node](#gitlab-geo) | Geographically distributed GitLab nodes | [⚙][geo-omnibus] | [❌][geo-charts] | [❌][geo-charts] | ✅ | ❌ | [⚙][geo-gdk] | EE Only | +| [LDAP Authentication](#ldap-authentication) | Authenticate users against centralized LDAP directory | [⤓][ldap-omnibus] | [⤓][ldap-charts] | [⤓][ldap-charts] | [❌](https://about.gitlab.com/pricing/#gitlab-com) | [⤓][gitlab-yml] | [⤓][ldap-gdk] | CE & EE | +| [Outbound email (SMTP)](#outbound-email) | Send email messages to users | [⤓][outbound-email-omnibus] | [⤓][outbound-email-charts] | [⤓][outbound-email-charts] | [✅](../user/gitlab_com/index.md#mail-configuration) | [⤓][gitlab-yml] | [⤓][gitlab-yml] | CE & EE | +| [Inbound email (SMTP)](#inbound-email) | Receive messages to update issues | [⤓][inbound-email-omnibus] | [⤓][inbound-email-charts] | [⤓][inbound-email-charts] | [✅](../user/gitlab_com/index.md#mail-configuration) | [⤓][gitlab-yml] | [⤓][gitlab-yml] | CE & EE | +| [ElasticSearch](#elasticsearch) | Improved search within GitLab | [⤓][elasticsearch-omnibus] | [⤓][elasticsearch-charts] | [⤓][elasticsearch-charts] | [❌](https://gitlab.com/groups/gitlab-org/-/epics/153) | [⤓][elasticsearch-source] | [⤓][elasticsearch-gdk] | EE Only | +| [Sentry integration](#sentry) | Error tracking for deployed apps | [⤓][sentry-integration] | [⤓][sentry-integration] | [⤓][sentry-integration] | [⤓][sentry-integration] | [⤓][sentry-integration] | [⤓][sentry-integration] | CE & EE | +| [Jaeger integration](#jaeger) | Distributed tracing for deployed apps | [⤓][jaeger-integration] | [⤓][jaeger-integration] | [⤓][jaeger-integration] | [⤓][jaeger-integration] | [⤓][jaeger-integration] | [⤓][jaeger-integration] | EE Only | +| [GitLab Managed Apps](#gitlab-managed-apps) | Deploy [Helm](https://docs.helm.sh/), [Ingress](https://kubernetes.io/docs/concepts/services-networking/ingress/), [Cert-Manager](https://docs.cert-manager.io/en/latest/), [Prometheus](https://prometheus.io/docs/introduction/overview/), a [Runner](https://docs.gitlab.com/runner/), [JupyterHub](http://jupyter.org/), [Knative](https://cloud.google.com/knative) to a cluster | [⤓][managed-k8s-apps] | [⤓][managed-k8s-apps] | [⤓][managed-k8s-apps] | [⤓][managed-k8s-apps] | [⤓][managed-k8s-apps] | [⤓][managed-k8s-apps] | CE & EE | + +### Component details + +This document is designed to be consumed by systems administrators and GitLab Support Engineers who want to understand more about the internals of GitLab and how they work together. + +When deployed, GitLab should be considered the amalgamation of the below processes. When troubleshooting or debugging, be as specific as possible as to which component you are referencing. That should increase clarity and reduce confusion. + +**Layers** GitLab can be considered to have two layers from a process perspective: @@ -46,117 +159,288 @@ GitLab can be considered to have two layers from a process perspective: - **Processors**: These processes are responsible for actually performing operations and presenting the service. - **Data**: These services store/expose structured data for the GitLab service. -### alertmanager +#### Alertmanager -- Omnibus configuration options +- [Project page](https://github.com/prometheus/alertmanager/blob/master/README.md) +- Configuration: [Omnibus][alertmanager-omnibus], [Charts][alertmanager-charts] - Layer: Monitoring +- Process: `alertmanager` + +[Alert manager](https://prometheus.io/docs/alerting/alertmanager/) is a tool provided by Prometheus that _"handles alerts sent by client applications such as the Prometheus server. It takes care of deduplicating, grouping, and routing them to the correct receiver integration such as email, PagerDuty, or OpsGenie. It also takes care of silencing and inhibition of alerts."_ You can read more in [issue gitlab-ce#45740](https://gitlab.com/gitlab-org/gitlab-ce/issues/45740) about what we will be alerting on. + +#### Certificate management + +- Project page: [Omnibus](https://github.com/certbot/certbot/blob/master/README.rst), [Charts](https://github.com/jetstack/cert-manager/blob/master/README.md) +- Configuration: [Omnibus][certificate-management-omnibus], [Charts][certificate-management-charts], [Source][certificate-management-source], [GDK][certificate-management-gdk] +- Layer: Core Service (Processor) + +#### Consul + +- [Project page](https://github.com/hashicorp/consul/blob/master/README.md) +- Configuration: [Omnibus][consul-omnibus], [Charts][consul-charts] +- Layer: Core Service (Data) + +Consul is a tool for service discovery and configuration. Consul is distributed, highly available, and extremely scalable. + +#### Database migrations + +- Configuration: [Omnibus][registry-omnibus], [Charts][registry-charts], [Source][database-migrations-source] +- Layer: Core Service (Data) + +#### Elasticsearch -[Alert manager](https://prometheus.io/docs/alerting/alertmanager/) is a tool provided by prometheus that _"handles alerts sent by client applications such as the Prometheus server. It takes care of deduplicating, grouping, and routing them to the correct receiver integration such as email, PagerDuty, or OpsGenie. It also takes care of silencing and inhibition of alerts."_ You can read more in [issue gitlab-ce#45740](https://gitlab.com/gitlab-org/gitlab-ce/issues/45740) about what we will be alerting on. +- [Project page](https://github.com/elastic/elasticsearch/blob/master/README.textile) +- Configuration: [Omnibus][elasticsearch-omnibus], [Charts][elasticsearch-charts], [Source][elasticsearch-source], [GDK][elasticsearch-gdk] +- Layer: Core Service (Data) + +Elasticsearch is a distributed RESTful search engine built for the cloud. -### gitaly +#### Gitaly -- [Omnibus configuration options](https://gitlab.com/gitlab-org/gitaly/tree/master/doc/configuration) +- [Project page](https://gitlab.com/gitlab-org/gitaly/blob/master/README.md) +- Configuration: [Omnibus][gitaly-omnibus], [Charts][gitaly-charts], [Source][gitaly-source] - Layer: Core Service (Data) +- Process: `gitaly` -Gitaly is a service designed by GitLab to remove our need for NFS for Git storage in distributed deployments of GitLab (Think GitLab.com or High Availability Deployments). As of 11.3.0, this service handles all Git level access in GitLab. You can read more about the project [in the project's readme](https://gitlab.com/gitlab-org/gitaly). +Gitaly is a service designed by GitLab to remove our need for NFS for Git storage in distributed deployments of GitLab (think GitLab.com or High Availability Deployments). As of 11.3.0, this service handles all Git level access in GitLab. You can read more about the project [in the project's readme](https://gitlab.com/gitlab-org/gitaly). -### gitlab-monitor +#### Gitlab Geo + +- Configuration: [Omnibus][geo-omnibus], [Charts][geo-charts], [GDK][geo-gdk] +- Layer: Core Service (Processor) -- Omnibus configuration options +#### Gitlab Monitor + +- [Project page](https://gitlab.com/gitlab-org/gitlab-monitor) +- Configuration: [Omnibus][gitlab-monitor-omnibus], [Charts][gitlab-monitor-charts] - Layer: Monitoring +- Process: `gitlab-monitor` + +GitLab Monitor is a process designed in house that allows us to export metrics about GitLab application internals to Prometheus. You can read more [in the project's readme](https://gitlab.com/gitlab-org/gitlab-monitor). + +#### Gitlab Pages + +- Configuration: [Omnibus][pages-omnibus], [Charts][pages-charts], [Source][pages-source], [GDK][pages-gdk] +- Layer: Core Service (Processor) + +GitLab Pages is a feature that allows you to publish static websites directly from a repository in GitLab. + +You can use it either for personal or business websites, such as portfolios, documentation, manifestos, and business presentations. You can also attribute any license to your content. + +#### Gitlab Runner + +- [Project page](https://gitlab.com/gitlab-org/gitlab-runner/blob/master/README.md) +- Configuration: [Omnibus][runner-omnibus], [Charts][runner-charts], [Source][runner-source], [GDK][runner-gdk] +- Layer: Core Service (Processor) + +GitLab Runner runs tests and sends the results to GitLab. -GitLab Monitor is a process disigned in house that allows us to export metrics about GitLab application internals to prometheus. You can read more [in the project's readme](https://gitlab.com/gitlab-org/gitlab-monitor) +GitLab CI is the open-source continuous integration service included with GitLab that coordinates the testing. The old name of this project was GitLab CI Multi Runner but please use "GitLab Runner" (without CI) from now on. -### gitlab-workhorse +#### Gitlab Shell -- Omnibus configuration options +- [Project page](https://gitlab.com/gitlab-org/gitlab-shell/blob/master/README.md) +- Configuration: [Omnibus][shell-omnibus], [Charts][shell-charts], [Source][shell-source], [GDK][gitlab-yml] - Layer: Core Service (Processor) -[GitLab Workhorse](https://gitlab.com/gitlab-org/gitlab-workhorse) is a program designed at GitLab to help alieviate pressure from unicorn. You can read more about the [historical reasons for developing](https://about.gitlab.com/2016/04/12/a-brief-history-of-gitlab-workhorse/). It's designed to act as a smart reverse proxy to help speed up GitLab as a whole. +[GitLab Shell](https://gitlab.com/gitlab-org/gitlab-shell) is a program designed at GitLab to handle ssh-based `git` sessions, and modifies the list of authorized keys. GitLab Shell is not a Unix shell nor a replacement for Bash or Zsh. + +#### Gitlab Workhorse + +- [Project page](https://gitlab.com/gitlab-org/gitlab-workhorse/blob/master/README.md) +- Configuration: [Omnibus][gitlab-workhorse-omnibus], [Charts][gitlab-workhorse-charts], [Source][workhorse-source] +- Layer: Core Service (Processor) +- Process: `gitlab-workhorse` + +[GitLab Workhorse](https://gitlab.com/gitlab-org/gitlab-workhorse) is a program designed at GitLab to help alleviate pressure from Unicorn. You can read more about the [historical reasons for developing](https://about.gitlab.com/2016/04/12/a-brief-history-of-gitlab-workhorse/). It's designed to act as a smart reverse proxy to help speed up GitLab as a whole. + +#### Grafana + +- [Project page](https://github.com/grafana/grafana/blob/master/README.md) +- Configuration: [Omnibus][grafana-omnibus], [Charts][grafana-charts] +- Layer: Monitoring + +Grafana is an open source, feature rich metrics dashboard and graph editor for Graphite, Elasticsearch, OpenTSDB, Prometheus and InfluxDB. + +#### Jaeger + +- [Project page](https://github.com/jaegertracing/jaeger/blob/master/README.md) +- Configuration: [Omnibus][jaeger-omnibus], [Charts][jaeger-charts], [Source][jaeger-source], [GDK][jaeger-gdk] +- Layer: Monitoring + +Jaeger, inspired by Dapper and OpenZipkin, is a distributed tracing system. It can be used for monitoring microservices-based distributed systems. -### logrotate +#### Logrotate -- [Omnibus configuration options](https://docs.gitlab.com/omnibus/settings/logs.html#logrotate) +- [Project page](https://github.com/logrotate/logrotate/blob/master/README.md) +- Configuration: [Omnibus](https://docs.gitlab.com/omnibus/settings/logs.html#logrotate) - Layer: Core Service +- Process: `logrotate` -GitLab is comprised of a large number of services that all log. We started bundling our own logrotate as of 7.4 to make sure we were logging responsibly. This is just a packaged version of the common opensource offering. +GitLab is comprised of a large number of services that all log. We started bundling our own logrotate as of 7.4 to make sure we were logging responsibly. This is just a packaged version of the common open source offering. -### nginx +#### Mattermost -- [Omnibus configuration options](https://docs.gitlab.com/omnibus/settings/nginx.html) +- [Project page](https://github.com/mattermost/mattermost-server/blob/master/README.md) +- Configuration: [Omnibus][mattermost-omnibus], [Charts][mattermost-charts] - Layer: Core Service (Processor) +Mattermost is an open source, private cloud, Slack-alternative from https://mattermost.com. + +#### MinIO + +- [Project page](https://github.com/minio/minio/blob/master/README.md) +- Configuration: [Omnibus][minio-omnibus], [Charts][minio-charts], [GDK][minio-gdk] +- Layer: Core Service (Data) + +MinIO is an object storage server released under Apache License v2.0. It is compatible with Amazon S3 cloud storage service. It is best suited for storing unstructured data such as photos, videos, log files, backups and container / VM images. Size of an object can range from a few KBs to a maximum of 5TB. + +#### NGINX + +- Project page: [Omnibus](https://github.com/nginx/nginx), [Charts](https://github.com/kubernetes/ingress-nginx/blob/master/README.md) +- Configuration: [Omnibus][nginx-omnibus], [Charts][nginx-charts], [Source][nginx-source] +- Layer: Core Service (Processor) +- Process: `nginx` + Nginx as an ingress port for all HTTP requests and routes them to the approriate sub-systems within GitLab. We are bundling an unmodified version of the popular open source webserver. -### node-exporter +#### Node Exporter -- [Omnibus configuration options](https://docs.gitlab.com/ee/administration/monitoring/prometheus/node_exporter.html) +- [Project page](https://github.com/prometheus/node_exporter/blob/master/README.md) +- Configuration: [Omnibus][node-exporter-omnibus], [Charts][node-exporter-charts] - Layer: Monitoring +- Process: `node-exporter` + +[Node Exporter](https://github.com/prometheus/node_exporter) is a Prometheus tool that gives us metrics on the underlying machine (think CPU/Disk/Load). It's just a packaged version of the common open source offering from the Prometheus project. + +#### PgBouncer + +- [Project page](https://github.com/pgbouncer/pgbouncer/blob/master/README.md) +- Configuration: [Omnibus][pgbouncer-omnibus], [Charts][pgbouncer-charts] +- Layer: Core Service (Data) -[Node Exporter](https://github.com/prometheus/node_exporter) is a Prometheus tool that gives us metrics on the underlying machine. (Think CPU/Disk/Load) It's just a packaged version of the common open source offering from the Prometheus project. +Lightweight connection pooler for PostgreSQL. -### postgres-exporter +#### PgBouncer Exporter -- [Omnibus configuration options](https://docs.gitlab.com/ee/administration/monitoring/prometheus/postgres_exporter.html) +- [Project page](https://github.com/stanhu/pgbouncer_exporter/blob/master/README.md) +- Configuration: [Omnibus][pgbouncer-exporter-omnibus], [Charts][pgbouncer-exporter-charts] - Layer: Monitoring -[Postgres-exporter](https://github.com/wrouesnel/postgres_exporter) is the community provided Prometheus exporter that will deliver data about Postgres to prometheus for use in Grafana Dashboards. +Prometheus exporter for PgBouncer. Exports metrics at 9127/metrics. -### postgresql +#### Postgresql -- [Omnibus configuration options](https://docs.gitlab.com/omnibus/settings/database.html) +- [Project page](https://github.com/postgres/postgres/blob/master/README) +- Configuration: [Omnibus][postgres-omnibus], [Charts][postgres-charts], [Source][postgres-source] - Layer: Core Service (Data) +- Process: `postgresql` GitLab packages the popular Database to provide storage for Application meta data and user information. -### prometheus +#### Postgres Exporter -- [Omnibus configuration options](https://docs.gitlab.com/ee/administration/monitoring/prometheus/) +- [Project page](https://github.com/wrouesnel/postgres_exporter/blob/master/README.md) +- Configuration: [Omnibus][postgres-exporter-omnibus], [Charts][postgres-exporter-charts] - Layer: Monitoring +- Process: `postgres-exporter` + +[Postgres-exporter](https://github.com/wrouesnel/postgres_exporter) is the community provided Prometheus exporter that will deliver data about Postgres to Prometheus for use in Grafana Dashboards. + +#### Prometheus + +- [Project page](https://github.com/prometheus/prometheus/blob/master/README.md) +- Configuration: [Omnibus][prometheus-omnibus], [Charts][prometheus-charts] +- Layer: Monitoring +- Process: `prometheus` Prometheus is a time-series tool that helps GitLab administrators expose metrics about the individual processes used to provide GitLab the service. -### redis +#### Redis -- [Omnibus configuration options](https://docs.gitlab.com/omnibus/settings/redis.html) +- [Project page](https://github.com/antirez/redis/blob/unstable/README.md) +- Configuration: [Omnibus][redis-omnibus], [Charts][redis-charts], [Source][redis-source] - Layer: Core Service (Data) +- Process: `redis` Redis is packaged to provide a place to store: - session data - temporary cache information -- background job queues. +- background job queues + +#### Redis Exporter + +- [Project page](https://github.com/oliver006/redis_exporter/blob/master/README.md) +- Configuration: [Omnibus][redis-exporter-omnibus], [Charts][redis-exporter-charts] +- Layer: Monitoring +- Process: `redis-exporter` + +[Redis Exporter](https://github.com/oliver006/redis_exporter) is designed to give specific metrics about the Redis process to Prometheus so that we can graph these metrics in Grafana. + +#### Registry + +- [Project page](https://github.com/docker/distribution/blob/master/README.md) +- Configuration: [Omnibus][registry-omnibus], [Charts][registry-charts], [Source][registry-source], [GDK][registry-gdk] +- Layer: Core Service (Processor) + +The registry is what users use to store their own Docker images. The bundled +registry uses nginx as a load balancer and GitLab as an authentication manager. +Whenever a client requests to pull or push an image from the registry, it will +return a `401` response along with a header detailing where to get an +authentication token, in this case the GitLab instance. The client will then +request a pull or push auth token from GitLab and retry the original request +to the registry. Learn more about [token authentication](https://docs.docker.com/registry/spec/auth/token/). + +An external registry can also be configured to use GitLab as an auth endpoint. -### redis-exporter +#### Sentry -- [Omnibus configuration options](https://docs.gitlab.com/ee/administration/monitoring/prometheus/redis_exporter.html) +- [Project page](https://github.com/getsentry/sentry/blob/master/README.rst) +- Configuration: [Omnibus][sentry-omnibus], [Charts][sentry-charts], [Source][gitlab-yml], [GDK][gitlab-yml] - Layer: Monitoring -[Redis Exporter](https://github.com/oliver006/redis_exporter) is designed to give specific metrics about the Redis process to Prometheus so that we can graph these metrics in Graphana. +Sentry fundamentally is a service that helps you monitor and fix crashes in realtime. The server is in Python, but it contains a full API for sending events from any language, in any application. -### sidekiq +#### Sidekiq -- Omnibus configuration options +- [Project page](https://github.com/mperham/sidekiq/blob/master/README.md) +- Configuration: [Omnibus][sidekiq-omnibus], [Charts][sidekiq-charts], [Source][gitlab-yml], [GDK][gitlab-yml] - Layer: Core Service (Processor) +- Process: `sidekiq` Sidekiq is a Ruby background job processor that pulls jobs from the redis queue and processes them. Background jobs allow GitLab to provide a faster request/response cycle by moving work into the background. -### unicorn +#### Unicorn -- [Omnibus configuration options](https://docs.gitlab.com/omnibus/settings/unicorn.html) +- [Project page](https://gitlab.com/gitlab-org/gitlab-ee/blob/master/README.md) +- Configuration: [Omnibus][unicorn-omnibus], [Charts][unicorn-charts], [Source][unicorn-source], [GDK][gitlab-yml] - Layer: Core Service (Processor) +- Process: `unicorn` [Unicorn](https://bogomips.org/unicorn/) is a Ruby application server that is used to run the core Rails Application that provides the user facing features in GitLab. Often process output you will see this as `bundle` or `config.ru` depending on the GitLab version. -### Additional Processes +#### LDAP Authentication -### GitLab Pages +- Configuration: [Omnibus][ldap-omnibus], [Charts][ldap-charts], [Source][gitlab-yml], [GDK][ldap-gdk] +- Layer: Core Service (Processor) -TODO +#### Outbound Email -### Mattermost +- Configuration: [Omnibus][outbound-email-omnibus], [Charts][outbound-email-charts], [Source][gitlab-yml], [GDK][gitlab-yml] +- Layer: Core Service (Processor) -TODO +#### Inbound Email + +- Configuration: [Omnibus][inbound-email-omnibus], [Charts][inbound-email-charts], [Source][gitlab-yml], [GDK][gitlab-yml] +- Layer: Core Service (Processor) + +#### GitLab Managed Apps + +- Configuration: [Omnibus][managed-k8s-apps], [Charts][managed-k8s-apps], [Source][managed-k8s-apps], [GDK][managed-k8s-apps] +- Layer: Core Service (Processor) + +GitLab provides [GitLab Managed Apps](../user/project/clusters/index.md#installing-applications), a one-click install for various applications which can be added directly to your configured cluster. These applications are needed for Review Apps and deployments when using Auto DevOps. You can install them after you create a cluster. ## GitLab by Request Type @@ -169,10 +453,10 @@ It's important to understand the distinction as some processes are used in both ### GitLab Web HTTP Request Cycle -When making a request to an HTTP Endpoint (Think `/users/sign_in`) the request will take the following path through the GitLab Service: +When making a request to an HTTP Endpoint (think `/users/sign_in`) the request will take the following path through the GitLab Service: -- nginx - Acts as our first line reverse proxy -- gitlab-workhorse - This determines if it needs to go to the Rails application or somewhere else to reduce load on unicorn. +- nginx - Acts as our first line reverse proxy. +- gitlab-workhorse - This determines if it needs to go to the Rails application or somewhere else to reduce load on Unicorn. - unicorn - Since this is a web request, and it needs to access the application it will go to Unicorn. - Postgres/Gitaly/Redis - Depending on the type of request, it may hit these services to store or retrieve data. @@ -188,7 +472,7 @@ TODO ## System Layout -When referring to `~git` in the pictures it means the home directory of the git user which is typically /home/git. +When referring to `~git` in the pictures it means the home directory of the git user which is typically `/home/git`. GitLab is primarily installed within the `/home/git` user home directory as `git` user. Within the home directory is where the gitlabhq server software resides as well as the repositories (though the repository location is configurable). @@ -196,24 +480,6 @@ The bare repositories are located in `/home/git/repositories`. GitLab is a ruby To serve repositories over SSH there's an add-on application called gitlab-shell which is installed in `/home/git/gitlab-shell`. -### Components - -<img src="https://docs.google.com/drawings/d/1fBzAyklyveF-i-2q-OHUIqDkYfjjxC4mq5shwKSZHLs/pub?w=987&h=797"> - -_[edit diagram (for GitLab team members only)](https://docs.google.com/drawings/d/1fBzAyklyveF-i-2q-OHUIqDkYfjjxC4mq5shwKSZHLs/edit)_ - -A typical install of GitLab will be on GNU/Linux. It uses Nginx or Apache as a web front end to proxypass the Unicorn web server. By default, communication between Unicorn and the front end is via a Unix domain socket but forwarding requests via TCP is also supported. The web front end accesses `/home/git/gitlab/public` bypassing the Unicorn server to serve static pages, uploads (e.g. avatar images or attachments), and precompiled assets. GitLab serves web pages and a [GitLab API](https://gitlab.com/gitlab-org/gitlab-ce/tree/master/doc/api) using the Unicorn web server. It uses Sidekiq as a job queue which, in turn, uses redis as a non-persistent database backend for job information, meta data, and incoming jobs. - -The GitLab web app uses MySQL or PostgreSQL for persistent database information (e.g. users, permissions, issues, other meta data). GitLab stores the bare git repositories it serves in `/home/git/repositories` by default. It also keeps default branch and hook information with the bare repository. - -When serving repositories over HTTP/HTTPS GitLab utilizes the GitLab API to resolve authorization and access as well as serving git objects. - -The add-on component gitlab-shell serves repositories over SSH. It manages the SSH keys within `/home/git/.ssh/authorized_keys` which should not be manually edited. gitlab-shell accesses the bare repositories through Gitaly to serve git objects and communicates with redis to submit jobs to Sidekiq for GitLab to process. gitlab-shell queries the GitLab API to determine authorization and access. - -Gitaly executes git operations from gitlab-shell and the GitLab web app, and provides an API to the GitLab web app to get attributes from git (e.g. title, branches, tags, other meta data), and to get blobs (e.g. diffs, commits, files). - -You may also be interested in the [production architecture of GitLab.com](https://about.gitlab.com/handbook/engineering/infrastructure/production-architecture/). - ### Installation Folder Summary To summarize here's the [directory structure of the `git` user home directory](../install/structure.md). @@ -340,3 +606,95 @@ Note: It is recommended to log into the `git` user using `sudo -i -u git` or `su ## GitLab.com We've also detailed [our architecture of GitLab.com](https://about.gitlab.com/handbook/engineering/infrastructure/production-architecture/) but this is probably over the top unless you have millions of users. + +[alertmanager-omnibus]: https://gitlab.com/gitlab-org/omnibus-gitlab/blob/master/files/gitlab-config-template/gitlab.rb.template +[alertmanager-charts]: https://github.com/helm/charts/tree/master/stable/prometheus +[nginx-omnibus]: https://docs.gitlab.com/omnibus/settings/ +[nginx-charts]: https://docs.gitlab.com/charts/charts/nginx/ +[nginx-source]: ../install/installation.md#9-nginx +[unicorn-omnibus]: https://docs.gitlab.com/omnibus/settings/unicorn.html +[unicorn-charts]: https://docs.gitlab.com/charts/charts/gitlab/unicorn/ +[unicorn-source]: ../install/installation.md#configure-it +[gitlab-yml]: https://gitlab.com/gitlab-org/gitlab-ce/blob/master/config/gitlab.yml.example +[sidekiq-omnibus]: https://gitlab.com/gitlab-org/omnibus-gitlab/blob/master/files/gitlab-config-template/gitlab.rb.template +[sidekiq-charts]: https://docs.gitlab.com/charts/charts/gitlab/sidekiq/ +[gitaly-omnibus]: ../administration/gitaly/index.md +[gitaly-charts]: https://docs.gitlab.com/charts/charts/gitlab/gitaly/ +[gitaly-source]: ../install/installation.md#install-gitaly +[workhorse-omnibus]: https://gitlab.com/gitlab-org/omnibus-gitlab/blob/master/files/gitlab-config-template/gitlab.rb.template +[workhorse-charts]: https://docs.gitlab.com/charts/charts/gitlab/unicorn/ +[workhorse-source]: ../install/installation.md#install-gitlab-workhorse +[shell-omnibus]: https://gitlab.com/gitlab-org/omnibus-gitlab/blob/master/files/gitlab-config-template/gitlab.rb.template +[shell-charts]: https://docs.gitlab.com/charts/charts/gitlab/gitlab-shell/ +[shell-source]: ../install/installation.md#install-gitlab-shell +[pages-omnibus]: ../administration/pages/index.md +[pages-charts]: https://gitlab.com/charts/gitlab/issues/37 +[pages-source]: ../install/installation.md#install-gitlab-pages +[pages-gdk]: https://gitlab.com/gitlab-org/gitlab-development-kit/blob/master/doc/howto/pages.md +[registry-omnibus]: ../administration/container_registry.md#container-registry-domain-configuration +[registry-charts]: https://docs.gitlab.com/charts/charts/registry/ +[registry-source]: ../administration/container_registry.md#enable-the-container-registry +[registry-gdk]: https://gitlab.com/gitlab-org/gitlab-development-kit/blob/master/doc/howto/registry.md +[redis-omnibus]: https://docs.gitlab.com/omnibus/settings/redis.html +[redis-charts]: https://docs.gitlab.com/charts/charts/redis/ +[redis-source]: ../install/installation.md#7-redis +[postgres-omnibus]: https://docs.gitlab.com/omnibus/settings/database.html +[postgres-charts]: https://github.com/helm/charts/tree/master/stable/postgresql +[postgres-source]: ../install/installation.md#6-database +[pgbouncer-omnibus]: ../administration/high_availability/pgbouncer.md +[pgbouncer-charts]: https://docs.gitlab.com/charts/installation/deployment.html#postgresql +[consul-omnibus]: ../administration/high_availability/consul.md +[consul-charts]: https://docs.gitlab.com/charts/installation/deployment.html#postgresql +[prometheus-omnibus]: ../administration/monitoring/prometheus/index.md +[prometheus-charts]: https://github.com/helm/charts/tree/master/stable/prometheus +[grafana-omnibus]: ../administration/monitoring/performance/grafana_configuration.md +[grafana-charts]: https://github.com/helm/charts/tree/master/stable/grafana +[sentry-omnibus]: https://docs.gitlab.com/omnibus/settings/configuration.html#error-reporting-and-logging-with-sentry +[sentry-charts]: https://gitlab.com/charts/gitlab/issues/1319 +[jaeger-omnibus]: https://gitlab.com/gitlab-org/omnibus-gitlab/issues/4104 +[jaeger-charts]: https://gitlab.com/charts/gitlab/issues/1320 +[jaeger-source]: ../development/distributed_tracing.md#enabling-distributed-tracing +[jaeger-gdk]: ../development/distributed_tracing.html#using-jaeger-in-the-gitlab-development-kit +[redis-exporter-omnibus]: ../administration/monitoring/prometheus/redis_exporter.md +[redis-exporter-charts]: https://docs.gitlab.com/charts/charts/redis/ +[postgres-exporter-omnibus]: ../administration/monitoring/prometheus/postgres_exporter.md +[postgres-exporter-charts]: https://github.com/helm/charts/tree/master/stable/postgresql +[pgbouncer-exporter-omnibus]: ../administration/monitoring/prometheus/pgbouncer_exporter.md +[pgbouncer-exporter-charts]: https://docs.gitlab.com/charts/installation/deployment.html#postgresql +[gitlab-monitor-omnibus]: ../administration/monitoring/prometheus/gitlab_monitor_exporter.md +[gitab-monitor-charts]: https://gitlab.com/charts/gitlab/issues/319 +[node-exporter-omnibus]: ../administration/monitoring/prometheus/node_exporter.md +[node-exporter-charts]: https://gitlab.com/charts/gitlab/issues/1332 +[mattermost-omnibus]: https://docs.gitlab.com/omnibus/gitlab-mattermost/ +[mattermost-charts]: https://docs.mattermost.com/install/install-mmte-helm-gitlab-helm.html +[minio-omnibus]: https://min.io/download +[minio-charts]: https://docs.gitlab.com/charts/charts/minio/ +[minio-gdk]: https://gitlab.com/gitlab-org/gitlab-development-kit/blob/master/doc/howto/object_storage.md +[runner-omnibus]: https://docs.gitlab.com/runner/ +[runner-charts]: https://docs.gitlab.com/runner/install/kubernetes.html +[runner-source]: https://docs.gitlab.com/runner/ +[runner-gdk]: https://gitlab.com/gitlab-org/gitlab-development-kit/blob/master/doc/howto/runner.md +[database-migrations-omnibus]: https://docs.gitlab.com/omnibus/settings/database.html#disabling-automatic-database-migration +[database-migrations-charts]: https://docs.gitlab.com/charts/charts/gitlab/migrations/ +[database-migrations-source]: ../update/upgrading_from_source.md#14-install-libs-migrations-etc +[certificate-management-omnibus]: https://docs.gitlab.com/omnibus/settings/ssl.html +[certificate-management-charts]: https://docs.gitlab.com/charts/installation/tls.html +[certificate-management-source]: ../install/installation.md#using-https +[certificate-management-gdk]: https://gitlab.com/gitlab-org/gitlab-development-kit/blob/master/doc/howto/https.md +[geo-omnibus]: ../administration/geo/replication/index.md#setup-instructions +[geo-charts]: https://gitlab.com/charts/gitlab/issues/8 +[geo-gdk]: https://gitlab.com/gitlab-org/gitlab-development-kit/blob/master/doc/howto/geo.md +[ldap-omnibus]: ../administration/auth/ldap.md +[ldap-charts]: https://docs.gitlab.com/charts/charts/globals.html#ldap +[ldap-gdk]: https://gitlab.com/gitlab-org/gitlab-development-kit/blob/master/doc/howto/ldap.md +[outbound-email-omnibus]: https://docs.gitlab.com/omnibus/settings/smtp.html +[outbound-email-charts]: https://docs.gitlab.com/charts/installation/command-line-options.html#outgoing-email-configuration +[inbound-email-omnibus]: ../administration/incoming_email.md +[inbound-email-charts]: https://docs.gitlab.com/charts/installation/command-line-options.html#incoming-email-configuration +[elasticsearch-omnibus]: ../integration/elasticsearch.md +[elasticsearch-charts]: ../integration/elasticsearch.md +[elasticsearch-source]: ../integration/elasticsearch.md +[elasticsearch-gdk]: https://gitlab.com/gitlab-org/gitlab-development-kit/blob/master/doc/howto/elasticsearch.md +[sentry-integration]: ../user/project/operations/error_tracking.md +[jaeger-integration]: ../user/project/operations/tracing.md +[managed-k8s-apps]: ../user/project/clusters/index.md#installing-applications diff --git a/doc/development/changelog.md b/doc/development/changelog.md index cd0a1f46d27..45b3d5a23a1 100644 --- a/doc/development/changelog.md +++ b/doc/development/changelog.md @@ -35,6 +35,7 @@ the `author` field. GitLab team members **should not**. - Any user-facing change **should** have a changelog entry. Example: "GitLab now uses system fonts for all text." +- Any change behind a feature flag **should not** have a changelog entry. The entry should be added [in the merge request removing the feature flags](https://docs.gitlab.com/ee/development/feature_flags.html#developing-with-feature-flags). - A fix for a regression introduced and then fixed in the same release (i.e., fixing a bug introduced during a monthly release candidate) **should not** have a changelog entry. @@ -43,7 +44,7 @@ the `author` field. GitLab team members **should not**. database records created during Cycle Analytics model spec." - _Any_ contribution from a community member, no matter how small, **may** have a changelog entry regardless of these guidelines if the contributor wants one. - Example: "Fixed a typo on the search results page. (Jane Smith)" + Example: "Fixed a typo on the search results page." - Performance improvements **should** have a changelog entry. - Any change that introduces a database migration **must** have a changelog entry. @@ -135,21 +136,13 @@ If you're working on the GitLab EE repository, the entry will be added to | Argument | Shorthand | Purpose | | ----------------- | --------- | --------------------------------------------------------------------------------------------------------------------------------------- | -| [`--amend`] | | Amend the previous commit | -| [`--force`] | `-f` | Overwrite an existing entry | -| [`--merge-request`] | `-m` | Set merge request ID | -| [`--dry-run`] | `-n` | Don't actually write anything, just print | -| [`--git-username`] | `-u` | Use Git user.name configuration as the author | -| [`--type`] | `-t` | The category of the change, valid options are: `added`, `fixed`, `changed`, `deprecated`, `removed`, `security`, `performance`, `other` | -| [`--help`] | `-h` | Print help message | - -[`--amend`]: #-amend -[`--force`]: #-force-or-f -[`--merge-request`]: #-merge-request-or-m -[`--dry-run`]: #-dry-run-or-n -[`--git-username`]: #-git-username-or-u -[`--type`]: #-type-or-t -[`--help`]: #-help +| [`--amend`](#--amend) | | Amend the previous commit | +| [`--force`](#--force-or--f) | `-f` | Overwrite an existing entry | +| [`--merge-request`](#--merge-request-or--m) | `-m` | Set merge request ID | +| [`--dry-run`](#--dry-run-or--n) | `-n` | Don't actually write anything, just print | +| [`--git-username`](#--git-username-or--u) | `-u` | Use Git user.name configuration as the author | +| [`--type`](#--type-or--t) | `-t` | The category of the change, valid options are: `added`, `fixed`, `changed`, `deprecated`, `removed`, `security`, `performance`, `other` | +| `--help` | `-h` | Print help message | ##### `--amend` diff --git a/doc/development/chatops_on_gitlabcom.md b/doc/development/chatops_on_gitlabcom.md new file mode 100644 index 00000000000..c63ec53414c --- /dev/null +++ b/doc/development/chatops_on_gitlabcom.md @@ -0,0 +1,21 @@ +# Chatops on GitLab.com + +Chatops on GitLab.com allows GitLabbers to run various automation tasks on GitLab.com using Slack. + +## Requesting access + +GitLabbers may need access to Chatops on GitLab.com for administration tasks such as: + +- Configuring feature flags on staging. +- Running `EXPLAIN` queries against the GitLab.com production replica. + +To request access to Chatops on GitLab.com: + +1. Log into <https://ops.gitlab.net/users/sign_in> using the same username as for GitLab.com. +1. Ask [anyone in the `chatops` project](https://gitlab.com/gitlab-com/chatops/project_members) to add you by running `/chatops run member add <username> gitlab-com/chatops --ops`. + +## See also + + - [Chatops Usage](https://docs.gitlab.com/ee/ci/chatops/README.html) + - [Understanding EXPLAIN plans](understanding_explain_plans.md) + - [Feature Groups](feature_flags.md#feature-groups) diff --git a/doc/development/code_review.md b/doc/development/code_review.md index f115045dbb7..29e2aa1a581 100644 --- a/doc/development/code_review.md +++ b/doc/development/code_review.md @@ -20,23 +20,38 @@ importance of involving reviewer(s) in the section on the responsibility of the If you need some guidance (e.g. it's your first merge request), feel free to ask one of the [Merge request coaches][team]. -If you need assistance with security scans or comments, feel free to include the +If you need assistance with security scans or comments, feel free to include the Security Team (`@gitlab-com/gl-security`) in the review. -The `danger-review` CI job will randomly pick a reviewer and a maintainer for -each area of the codebase that your merge request seems to touch. It only makes -recommendations - feel free to override it if you think someone else is a better -fit! - Depending on the areas your merge request touches, it must be **approved** by one or more [maintainers](https://about.gitlab.com/handbook/engineering/workflow/code-review/#maintainer): For approvals, we use the approval functionality found in the merge request -widget. Reviewers can add their approval by [approving additionally](https://docs.gitlab.com/ee/user/project/merge_requests/merge_request_approvals.html#adding-or-removing-an-approval). +widget. Reviewers can add their approval by [approving additionally](../user/project/merge_requests/merge_request_approvals.md#adding-or-removing-an-approval). Getting your merge request **merged** also requires a maintainer. If it requires more than one approval, the last maintainer to review and approve it will also merge it. +### Reviewer roulette + +The `danger-review` CI job will randomly pick a reviewer and a maintainer for +each area of the codebase that your merge request seems to touch. It only makes +recommendations - feel free to override it if you think someone else is a better +fit! + +It picks reviewers and maintainers from the list at the +[engineering projects](https://about.gitlab.com/handbook/engineering/projects/) +page, with these behaviours: + +1. It will not pick people whose [GitLab status](../user/profile/index.md#current-status) + contains the string 'OOO'. +2. [Trainee maintainers](https://about.gitlab.com/handbook/engineering/workflow/code-review/#trainee-maintainer) + are three times as likely to be picked as other reviewers. +3. It always picks the same reviewers and maintainers for the same + branch name (unless their OOO status changes, as in point 1). It + removes leading `ce-` and `ee-`, and trailing `-ce` and `-ee`, so + that it can be stable for backport branches. + ### Approval guidelines As described in the section on the responsibility of the maintainer below, you @@ -58,12 +73,7 @@ from teams other than your own. #### Security requirements - 1. If your merge request is processing, storing, or transferring any kind of [RED or ORANGE data](https://docs.google.com/document/d/15eNKGA3zyZazsJMldqTBFbYMnVUSQSpU14lo22JMZQY/edit) (this is a confidential document), it must be - **approved by a [Security Engineer][team]**. - 1. If your merge request involves implementing, utilizing, or is otherwise related to any type of authentication, authorization, or session handling mechanism, it must be - **approved by a [Security Engineer][team]**. - 1. If your merge request has a goal which requires a cryptographic function such as: confidentiality, integrity, authentication, or non-repudiation, it must be - **approved by a [Security Engineer][team]**. +View the updated documentation regarding [internal application security reviews](https://about.gitlab.com/handbook/engineering/security/index.html#internal-application-security-reviews) for **when** and **how** to request a security review. ### The responsibility of the merge request author @@ -137,21 +147,13 @@ If a developer who happens to also be a maintainer was involved in a merge reque as a domain expert and/or reviewer, it is recommended that they are not also picked as the maintainer to ultimately approve and merge it. -Try to review in a timely manner; doing so allows everyone involved in the merge -request to iterate faster as the context is fresh in memory. Further, this -improves contributors' experiences significantly. Reviewers should aim to review -within two working days from the date they were assigned the merge request. If -you don't think you'll be able to review a merge request within that time, let -the author know as soon as possible. When the author of the merge request has not -heard anything after two days, a new reviewer should be assigned. - Maintainers should check before merging if the merge request is approved by the required approvers. Maintainers must check before merging if the merge request is introducing new vulnerabilities, by inspecting the list in the Merge Request [Security -Widget](https://docs.gitlab.com/ee/user/project/merge_requests/#security-reports-ultimate). -When in doubt, a [Security Engineer][team] can be involved. The list of detected +Widget](../user/project/merge_requests/index.md#security-reports-ultimate). +When in doubt, a [Security Engineer][team] can be involved. The list of detected vulnerabilities must be either empty or containing: - dismissed vulnerabilities in case of false positives @@ -225,6 +227,37 @@ It is responsibility of the author of a merge request that the merge request is Developers who have capacity can regularly check the list of [merge requests to review](https://gitlab.com/groups/gitlab-org/-/merge_requests?scope=all&utf8=%E2%9C%93&state=opened&label_name%5B%5D=ready%20for%20review) and assign any merge request they want to review. +### Review turnaround time + +Since [unblocking others is always a top priority](https://about.gitlab.com/handbook/values/#global-optimization), +reviewers are expected to review assigned merge requests in a timely manner, +even when this may negatively impact their other tasks and priorities. +Doing so allows everyone involved in the merge request to iterate faster as the +context is fresh in memory, improves contributors' experiences significantly, +and gives authors more time to address feedback and iterate on their work before +the [feature freeze](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/PROCESS.md#feature-freeze-on-the-7th-for-the-release-on-the-22nd). + +A turnaround time of two working days is usually acceptable, since engineers +will typically have other things to work on while they're waiting for review, +but don't hesitate to ask the author if it's unclear what time frame would be +acceptable, how urgent the review is, or how significant the blockage. Authors +are also encouraged to provide this information up-front to reviewers, but are +expected to be mindful of the [guidelines on when to ask for review on MRs that +are intended to go in before the feature freeze](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/PROCESS.md#between-the-1st-and-the-7th), +and realistic in their expectations if these were not followed. + +If you don't think you'll be able to review a merge request within a reasonable +time frame, let the author know as soon as possible and try to help them find +another reviewer or maintainer who will be able to, so that they can be unblocked +and get on with their work quickly. Of course, if you are out of office and have +[communicated](https://about.gitlab.com/handbook/paid-time-off/#communicating-your-time-off) +this through your GitLab.com Status, authors are expected to realize this and +find a different reviewer themselves. + +When a merge request author feels like they have been blocked for longer than +is reasonable, they are free to remind the reviewer through Slack or assign +another reviewer. + ### Reviewing code Understand why the change is necessary (fixes a bug, improves the user @@ -253,7 +286,7 @@ experience, refactors the existing code). Then: author has already set this option or if the merge request clearly contains a messy commit history that is intended to be squashed. -[squash-and-merge]: https://docs.gitlab.com/ee/user/project/merge_requests/squash_and_merge.html#squash-and-merge +[squash-and-merge]: ../user/project/merge_requests/squash_and_merge.md#squash-and-merge ### The right balance @@ -286,7 +319,7 @@ reviewee. GitLab is used in a lot of places. Many users use our [Omnibus packages](https://about.gitlab.com/installation/), but some use the [Docker images](https://docs.gitlab.com/omnibus/docker/), some are -[installed from source](https://docs.gitlab.com/ce/install/installation.html), +[installed from source](../install/installation.md), and there are other installation methods available. GitLab.com itself is a large Enterprise Edition instance. This has some implications: diff --git a/doc/development/contributing/index.md b/doc/development/contributing/index.md index f7a0fdbeb40..59cf5014da4 100644 --- a/doc/development/contributing/index.md +++ b/doc/development/contributing/index.md @@ -3,9 +3,6 @@ Thank you for your interest in contributing to GitLab. This guide details how to contribute to GitLab in a way that is easy for everyone. -We want to create a welcoming environment for everyone who is interested in contributing. -Please visit our [Code of Conduct page](https://about.gitlab.com/contributing/code-of-conduct) to learn more about our commitment to an open and welcoming environment. - For a first-time step-by-step guide to the contribution process, please see ["Contributing to GitLab"](https://about.gitlab.com/contributing/). @@ -33,8 +30,8 @@ vulnerabilities. ## Code of conduct -Our code of conduct can be found on the -["Contributing to GitLab"](https://about.gitlab.com/contributing/) page. +We want to create a welcoming environment for everyone who is interested in contributing. +Please visit our [Code of Conduct page](https://about.gitlab.com/community/contribute/code-of-conduct/) to learn more about our commitment to an open and welcoming environment. ## Closing policy for issues and merge requests @@ -52,9 +49,12 @@ for audiences of all ages. If a contributor is no longer actively working on a submitted merge request we can decide that the merge request will be finished by one of our [Merge request coaches][team] or close the merge request. We make this decision -based on how important the change is for our product vision. If a Merge request +based on how important the change is for our product vision. If a merge request coach is going to finish the merge request we assign the -~"coach will finish" label. +~"coach will finish" label. When a team member picks up a community contribution, +we credit the original author by adding a changelog entry crediting the author +and optionally include the original author on at least one of the commits +within the MR. ## Helping others @@ -87,7 +87,7 @@ Sometimes style guides will be followed but the code will lack structural integr GitLab will do its best to review community contributions as quickly as possible. Specially appointed developers review community contributions daily. You may take a look at the [team page](https://about.gitlab.com/team/) for the merge request coach who specializes in the type of code you have written and mention them in the merge request. For example, if you have written some JavaScript in your code then you should mention the frontend merge request coach. If your code has multiple disciplines you may mention multiple merge request coaches. -GitLab receives a lot of community contributions, so if your code has not been reviewed within 4 days of its initial submission feel free to re-mention the appropriate merge request coach. +GitLab receives a lot of community contributions, so if your code has not been reviewed within two days (excluding weekend and public holidays) of its initial submission feel free to re-mention the appropriate merge request coach. When submitting code to GitLab, you may feel that your contribution requires the aid of an external library. If your code includes an external library please provide a link to the library, as well as reasons for including it. @@ -119,6 +119,7 @@ This [documentation](merge_request_workflow.md) outlines the current merge reque - [Merge request guidelines](merge_request_workflow.md#merge-request-guidelines) - [Contribution acceptance criteria](merge_request_workflow.md#contribution-acceptance-criteria) - [Definition of done](merge_request_workflow.md#definition-of-done) +- [Dependencies](merge_request_workflow.md#dependencies) ## Style guides diff --git a/doc/development/contributing/issue_workflow.md b/doc/development/contributing/issue_workflow.md index 4c53643ed9c..e3a1dc711fd 100644 --- a/doc/development/contributing/issue_workflow.md +++ b/doc/development/contributing/issue_workflow.md @@ -7,7 +7,7 @@ scheduling into milestones. Labelling is a task for everyone. Most issues will have labels for at least one of the following: - Type: ~feature, ~bug, ~customer, etc. -- Subject: ~wiki, ~"container registry", ~ldap, ~api, ~frontend, etc. +- Subject: ~wiki, ~"Container Registry", ~ldap, ~api, ~frontend, etc. - Team: ~Plan, ~Manage, ~Quality, etc. - Stage: ~"devops:plan", ~"devops:create", etc. - Release Scoping: ~Deliverable, ~Stretch, ~"Next Patch Release" @@ -44,7 +44,7 @@ Subject labels are labels that define what area or feature of GitLab this issue hits. They are not always necessary, but very convenient. Examples of subject labels are ~wiki, ~ldap, ~api, -~issues, ~"merge requests", ~labels, and ~"container registry". +~issues, ~"merge requests", ~labels, and ~"Container Registry". If you are an expert in a particular area, it makes it easier to find issues to work on. You can also subscribe to those labels to receive an email each time an @@ -62,10 +62,12 @@ The current team labels are: - ~Configure - ~Create +- ~Defend - ~Distribution - ~Documentation - ~Geo - ~Gitaly +- ~Growth - ~Manage - ~Monitor - ~Plan @@ -90,18 +92,21 @@ Stage labels specify which [DevOps stage][devops-stages] the issue belongs to. The current stage labels are: -- ~"devops:manage" -- ~"devops:plan" -- ~"devops:create" -- ~"devops:verify" -- ~"devops:package" -- ~"devops:release" -- ~"devops:configure" -- ~"devops:monitor" -- ~"devops:secure" - -These labels should be mutually exclusive. If an issue belongs to multiple -stages, the most relevant should be used. +- ~"devops::manage" +- ~"devops::plan" +- ~"devops::create" +- ~"devops::verify" +- ~"devops::package" +- ~"devops::release" +- ~"devops::configure" +- ~"devops::monitor" +- ~"devops::secure" +- ~"devops::defend" +- ~"devops::growth" +- ~"devops::enablement" + +These labels are [scoped labels](../../user/project/labels.md#scoped-labels-premium) +and thus are mutually exclusive. They differ from the [Team labels](#team-labels) because teams may work on issues outside their stage. @@ -117,6 +122,25 @@ The Stage labels are used to generate the [direction pages][direction-pages] aut [devops-stages]: https://about.gitlab.com/direction/#devops-stages [direction-pages]: https://about.gitlab.com/direction/ +## Group labels + +Group labels specify which [groups][structure-groups] the issue belongs to. + +Examples include: + +- ~"group::control" +- ~"group::editor" + +These labels are [scoped labels](../../user/project/labels.md#scoped-labels-premium) +and thus are mutually exclusive. + +Groups are nested beneath a particular stage, so only one stage label and one group label +can be applied to a single issue. You can find the groups listed in the +[Product Categories pages][product-categories]. + +[structure-groups]: https://about.gitlab.com/company/team/structure/#groups +[product-categories]: https://about.gitlab.com/handbook/product/categories/ + ## Release Scoping labels Release Scoping labels help us clearly communicate expectations of the work for the @@ -336,13 +360,14 @@ addressed. In order to track things that can be improved in GitLab's codebase, we use the ~"technical debt" label in [GitLab's issue tracker][ce-tracker]. -For user experience improvements, we use the ~"UX debt" label. +For missed user experience requirements, we use the ~"UX debt" label. These labels should be added to issues that describe things that can be improved, shortcuts that have been taken, features that need additional attention, and all other things that have been left behind due to high velocity of development. For example, code that needs refactoring should use the ~"technical debt" label, -user experience refinements should use the ~"UX debt" label. +something that didn't ship according to our Design System guidelines should +use the ~"UX debt" label. Everyone can create an issue, though you may need to ask for adding a specific label, if you do not have permissions to do it by yourself. Additional labels diff --git a/doc/development/contributing/merge_request_workflow.md b/doc/development/contributing/merge_request_workflow.md index 4e766f37871..8a4aa5dfa7f 100644 --- a/doc/development/contributing/merge_request_workflow.md +++ b/doc/development/contributing/merge_request_workflow.md @@ -1,91 +1,95 @@ # Merge requests -We welcome merge requests with fixes and improvements to GitLab code, tests, -and/or documentation. The issues that are specifically suitable for -community contributions are listed with -[the `Accepting merge requests` label](issue_workflow.md#label-for-community-contributors), -but you are free to contribute to any other issue you want. - -Please note that if an issue is marked for the current milestone either before -or while you are working on it, a team member may take over the merge request +We welcome merge requests from everyone, with fixes and improvements +to GitLab code, tests, and documentation. The issues that are specifically suitable +for community contributions are listed with the [`Accepting merge requests`](issue_workflow.md#label-for-community-contributors) +label, but you are free to contribute to any issue you want. + +Please note that if an issue is marked for the current milestone at any time, even +when you are working on it, a GitLab Inc. team member may take over the merge request in order to ensure the work is finished before the release date. -If you want to add a new feature that is not labeled it is best to first create -a feedback issue (if there isn't one already) and leave a comment asking for it +If you want to add a new feature that is not labeled, it is best to first create +an issue (if there isn't one already) and leave a comment asking for it to be marked as `Accepting Merge Requests`. Please include screenshots or -wireframes if the feature will also change the UI. +wireframes of the proposed feature if it will also change the UI. -Merge requests should be opened at [GitLab.com][gitlab-mr-tracker]. +Merge requests should be submitted to the appropriate project at GitLab.com, for example +[GitLab CE](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests), +[GitLab EE](https://gitlab.com/gitlab-org/gitlab-ee/merge_requests), +[GitLab Runner](https://gitlab.com/gitlab-org/gitlab-runner/merge_requests), +[GitLab Omnibus](https://gitlab.com/gitlab-org/omnibus-gitlab/merge_requests), etc. If you are new to GitLab development (or web development in general), see the -[I want to contribute!](index.md#i-want-to-contribute) section to get you started with +[I want to contribute!](index.md#i-want-to-contribute) section to get started with some potentially easy issues. -To start with GitLab development download the [GitLab Development Kit][gdk] and -see the [Development section](../../README.md) for some guidelines. - -[gitlab-mr-tracker]: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests -[gdk]: https://gitlab.com/gitlab-org/gitlab-development-kit +To start developing GitLab, download the [GitLab Development Kit](https://gitlab.com/gitlab-org/gitlab-development-kit) +and see the [Development section](../../README.md) for the required guidelines. ## Merge request guidelines -If you can, please submit a merge request with the fix or improvements -including tests. If you don't know how to fix the issue but can write a test -that exposes the issue we will accept that as well. In general bug fixes that -include a regression test are merged quickly while new features without proper -tests are least likely to receive timely feedback. The workflow to make a merge +If you find an issue, please submit a merge request with a fix or improvement, if +you can, and include tests. If you don't know how to fix the issue but can write a test +that exposes the issue, we will accept that as well. In general, bug fixes that +include a regression test are merged quickly, while new features without proper +tests might be slower to receive feedback. The workflow to make a merge request is as follows: -1. Fork the project into your personal space on GitLab.com -1. Create a feature branch, branch away from `master` -1. Write [tests](https://docs.gitlab.com/ee/development/rake_tasks.html#run-tests) and code -1. [Generate a changelog entry with `bin/changelog`][changelog] +1. [Fork](../../workflow/forking_workflow.md#creating-a-fork) the project into + your personal namespace (or group) on GitLab.com. +1. Create a feature branch in your fork (don't work off `master`). +1. Write [tests](../rake_tasks.md#run-tests) and code. +1. [Generate a changelog entry with `bin/changelog`](../changelog.md) 1. If you are writing documentation, make sure to follow the - [documentation guidelines][doc-guidelines] -1. If you have multiple commits please combine them into a few logically - organized commits by [squashing them][git-squash] -1. Push the commit(s) to your fork -1. Submit a merge request (MR) to the `master` branch - 1. Your merge request needs at least 1 approval but feel free to require more. - For instance if you're touching backend and frontend code, it's a good idea + [documentation guidelines](../documentation/index.md). +1. Follow the [commit messages guidelines](#commit-messages-guidelines). +1. If you have multiple commits, combine them into a few logically organized + commits by [squashing them](https://git-scm.com/book/en/v2/Git-Tools-Rewriting-History#_squashing), + but do not change the commit history if you're working on shared branches though. +1. Push the commit(s) to your working branch in your fork. +1. Submit a merge request (MR) to the `master` branch in the main GitLab project. + 1. Your merge request needs at least 1 approval, but feel free to require more. + For instance if you're touching both backend and frontend code, it's a good idea to require 2 approvals: 1 from a backend maintainer and 1 from a frontend - maintainer - 1. You don't have to select any approvers, but you can if you really want - specific people to approve your merge request -1. The MR title should describe the change you want to make -1. The MR description should give a motive for your change and the method you - used to achieve it. - 1. If you are contributing code, fill in the template already provided in the - "Description" field. + maintainer. + 1. If you're submitting changes to documentation, you'll need approval from a technical + writer, based on the appropriate [product category](https://about.gitlab.com/handbook/product/categories/). + Only assign the MR to them when it's ready for docs review. + 1. You don't have to select any specific approvers, but you can if you really want + specific people to approve your merge request. +1. The MR title should describe the change you want to make. +1. The MR description should give a reason for your change. + 1. If you are contributing code, fill in the description according to the default + template already provided in the "Description" field. 1. If you are contributing documentation, choose `Documentation` from the - "Choose a template" menu and fill in the template. + "Choose a template" menu and fill in the description according to the template. 1. Mention the issue(s) your merge request solves, using the `Solves #XXX` or - `Closes #XXX` syntax to auto-close the issue(s) once the merge request will - be merged. -1. If you're allowed to, set a relevant milestone and labels -1. If the MR changes the UI it should include *Before* and *After* screenshots -1. If the MR changes CSS classes please include the list of affected pages, - `grep css-class ./app -R` -1. Be prepared to answer questions and incorporate feedback even if requests - for this arrive weeks or months after your MR submission - 1. If a discussion has been addressed, select the "Resolve discussion" button - beneath it to mark it resolved. -1. If your MR touches code that executes shell commands, reads or opens files or + `Closes #XXX` syntax to [auto-close](../../user/project/issues/automatic_issue_closing.md) + the issue(s) once the merge request is merged. +1. If you're allowed to (Core team members, for example), set a relevant milestone + and [labels](issue_workflow.md). +1. If the MR changes the UI, it should include *Before* and *After* screenshots. +1. If the MR changes CSS classes, please include the list of affected pages, which + can be found by running `grep css-class ./app -R`. +1. Be prepared to answer questions and incorporate feedback into your MR with new + commits. Once you have fully addressed a suggestion from a reviewer, click the + "Resolve discussion" button beneath it to mark it resolved. + 1. The merge request author resolves only the discussions they have fully addressed. + If there's an open reply or discussion, a suggestion, a question, or anything else, + the discussion should be left to be resolved by the reviewer. +1. If your MR touches code that executes shell commands, reads or opens files, or handles paths to files on disk, make sure it adheres to the [shell command guidelines](../shell_commands.md) 1. If your code creates new files on disk please read the [shared files guidelines](../shared_files.md). -1. When writing commit messages please follow - [these](http://tbaggery.com/2008/04/19/a-note-about-git-commit-messages.html) - [guidelines](http://chris.beams.io/posts/git-commit/). 1. If your merge request adds one or more migrations, make sure to execute all migrations on a fresh database before the MR is reviewed. If the review leads - to large changes in the MR, do this again once the review is complete. -1. For more complex migrations, write tests. -1. Merge requests **must** adhere to the [merge request performance - guidelines](../merge_request_performance_guidelines.md). -1. For tests that use Capybara or PhantomJS, see this [article on how - to write reliable asynchronous tests](https://robots.thoughtbot.com/write-reliable-asynchronous-integration-tests-with-capybara). + to large changes in the MR, execute the migrations again once the review is complete. +1. Write tests for more complex migrations. +1. Merge requests **must** adhere to the [merge request performance guidelines](../merge_request_performance_guidelines.md). +1. For tests that use Capybara, read + [how to write reliable, asynchronous integration tests](https://robots.thoughtbot.com/write-reliable-asynchronous-integration-tests-with-capybara). 1. If your merge request introduces changes that require additional steps when installing GitLab from source, add them to `doc/install/installation.md` in the same merge request. @@ -95,109 +99,117 @@ request is as follows: instructions are specific to a version, add them to the "Version specific upgrading instructions" section. -Please keep the change in a single MR **as small as possible**. If you want to -contribute a large feature think very hard what the minimum viable change is. -Can you split the functionality? Can you only submit the backend/API code? Can -you start with a very simple UI? Can you do part of the refactor? The increased -reviewability of small MRs that leads to higher code quality is more important -to us than having a minimal commit log. The smaller an MR is the more likely it -is it will be merged (quickly). After that you can send more MRs to enhance it. -The ['How to get faster PR reviews' document of Kubernetes](https://github.com/kubernetes/community/blob/master/contributors/devel/faster_reviews.md) also has some great points regarding this. - -For examples of feedback on merge requests please look at already -[closed merge requests][closed-merge-requests]. If you would like quick feedback -on your merge request feel free to mention someone from the [core team] or one -of the [Merge request coaches][team]. -Please ensure that your merge request meets the contribution acceptance criteria. - -When having your code reviewed and when reviewing merge requests please take the -[code review guidelines](../code_review.md) into account. - -[git-squash]: https://git-scm.com/book/en/Git-Tools-Rewriting-History#Squashing-Commits -[closed-merge-requests]: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests?assignee_id=&label_name=&milestone_id=&scope=&sort=&state=closed -[team]: https://about.gitlab.com/team/ +If you would like quick feedback on your merge request feel free to mention someone +from the [core team](https://about.gitlab.com/community/core-team/) or one of the +[merge request coaches](https://about.gitlab.com/team/). When having your code reviewed +and when reviewing merge requests, please keep the [code review guidelines](../code_review.md) +in mind. + +### Keep it simple + +*Live by smaller iterations.* Please keep the amount of changes in a single MR **as small as possible**. +If you want to contribute a large feature, think very carefully about what the +[minimum viable change](https://about.gitlab.com/handbook/product/#the-minimally-viable-change) +is. Can you split the functionality into two smaller MRs? Can you submit only the +backend/API code? Can you start with a very simple UI? Can you do just a part of the +refactor? + +Small MRs which are more easily reviewed, lead to higher code quality which is +more important to GitLab than having a minimal commit log. The smaller an MR is, +the more likely it will be merged quickly. After that you can send more MRs to +enhance and expand the feature. The [How to get faster PR reviews](https://github.com/kubernetes/kubernetes/blob/release-1.5/docs/devel/faster_reviews.md) +document from the Kubernetes team also has some great points regarding this. + +### Commit messages guidelines + +When writing commit messages, please follow the guidelines below: + +- The commit subject must contain at least 3 words. +- The commit subject should ideally contain up to 50 characters, +and must not be longer than 72 characters. +- The commit subject must start with a capital letter. +- The commit subject must not end with a period. +- The commit subject and body must be separated by a blank line. +- The commit body must not contain more than 72 characters per line. +- Commits that change 30 or more lines across at least 3 files must +describe these changes in the commit body. +- The commit subject or body must not contain Emojis. +- Use issues and merge requests' full URLs instead of short references, +as they are displayed as plain text outside of GitLab. +- The merge request must not contain more than 10 commit messages. + +If the guidelines are not met, the MR will not pass the +[Danger checks](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/danger/commit_messages/Dangerfile). +For more information see [How to Write a Git Commit Message](https://chris.beams.io/posts/git-commit/). ## Contribution acceptance criteria -1. The change is as small as possible +To make sure that your merge request can be approved, please ensure that it meets +the contribution acceptance criteria below: + +1. The change is as small as possible. 1. Include proper tests and make all tests pass (unless it contains a test exposing a bug in existing code). Every new class should have corresponding unit tests, even if the class is exercised at a higher level, such as a feature test. -1. If you suspect a failing CI build is unrelated to your contribution, you may - try and restart the failing CI job or ask a developer to fix the - aforementioned failing test -1. Your MR initially contains a single commit (please use `git rebase -i` to - squash commits) -1. Your changes can merge without problems (if not please rebase if you're the - only one working on your feature branch, otherwise, merge `master`) -1. Does not break any existing functionality -1. Fixes one specific issue or implements one specific feature (do not combine - things, send separate merge requests if needed) -1. Migrations should do only one thing (e.g., either create a table, move data - to a new table or remove an old table) to aid retrying on failure -1. Keeps the GitLab code base clean and well structured -1. Contains functionality we think other users will benefit from too -1. Doesn't add configuration options or settings options since they complicate - making and testing future changes -1. Changes do not adversely degrade performance. - - Avoid repeated polling of endpoints that require a significant amount of overhead - - Check for N+1 queries via the SQL log or [`QueryRecorder`](https://docs.gitlab.com/ce/development/merge_request_performance_guidelines.html) - - Avoid repeated access of filesystem -1. If you need polling to support real-time features, please use - [polling with ETag caching][polling-etag]. -1. Changes after submitting the merge request should be in separate commits - (no squashing). -1. It conforms to the [style guides](style_guides.md) and the following: - - If your change touches a line that does not follow the style, modify the - entire line to follow it. This prevents linting tools from generating warnings. - - Don't touch neighbouring lines. As an exception, automatic mass - refactoring modifications may leave style non-compliant. -1. If the merge request adds any new libraries (gems, JavaScript libraries, - etc.), they should conform to our [Licensing guidelines][license-finder-doc]. - See the instructions in that document for help if your MR fails the - "license-finder" test with a "Dependencies that need approval" error. -1. The merge request meets the [definition of done](#definition-of-done). - -[license-finder-doc]: ../licensing.md -[polling-etag]: ../polling.md + - If a failing CI build seems to be unrelated to your contribution, you can try + restarting the failing CI job, rebasing from master to bring in updates that + may resolve the failure, or if it has not been fixed yet, ask a developer to + help you fix the test. +1. The MR initially contains a few logically organized commits. +1. The changes can merge without problems. If not, you should rebase if you're the + only one working on your feature branch, otherwise merge `master`. +1. Only one specific issue is fixed or one specific feature is implemented. Do not + combine things; send separate merge requests for each issue or feature. +1. Migrations should do only one thing (e.g., create a table, move data to a new + table, or remove an old table) to aid retrying on failure. +1. Contains functionality that other users will benefit from. +1. Doesn't add configuration options or settings options since they complicate making + and testing future changes. +1. Changes do not degrade performance: + - Avoid repeated polling of endpoints that require a significant amount of overhead. + - Check for N+1 queries via the SQL log or [`QueryRecorder`](../merge_request_performance_guidelines.md). + - Avoid repeated access of the filesystem. + - Use [polling with ETag caching](../polling.md) if needed to support real-time features. +1. If the merge request adds any new libraries (gems, JavaScript libraries, etc.), + they should conform to our [Licensing guidelines](../licensing.md). See those + instructions for help if the "license-finder" test fails with a + `Dependencies that need approval` error. Also, make the reviewer aware of the new + library and explain why you need it. +1. The merge request meets GitLab's [definition of done](#definition-of-done), below. ## Definition of done If you contribute to GitLab please know that changes involve more than just -code. We have the following [definition of done][definition-of-done]. Please ensure you support -the feature you contribute through all of these steps. - -1. Description explaining the relevancy (see following item) -1. Working and clean code that is commented where needed -1. [Unit, integration, and system tests][testing] that pass on the CI server -1. Performance/scalability implications have been considered, addressed, and tested -1. [Documented][doc-guidelines] in the `/doc` directory -1. [Changelog entry added][changelog], if necessary -1. Reviewed by UX/FE/BE and any concerns are addressed -1. Merged by a project maintainer -1. Added to the release blog article, if relevant -1. Added to [the website](https://gitlab.com/gitlab-com/www-gitlab-com/), if relevant -1. Community questions answered -1. Answers to questions radiated (in docs/wiki/support etc.) -1. [Black-box tests/end-to-end tests](../testing_guide/testing_levels.md#black-box-tests-or-end-to-end-tests) added if required. Please contact [the quality team](https://about.gitlab.com/handbook/engineering/quality/#teams) with any questions +code. We use the following [definition of done](https://www.agilealliance.org/glossary/definition-of-done). +Your contribution is not *done* until you have made sure it meets all of these +requirements. + +1. Clear description explaining the relevancy of the contribution. +1. Working and clean code that is commented where needed. +1. [Unit, integration, and system tests](../testing_guide/index.md) that all pass + on the CI server. +1. Performance/scalability implications have been considered, addressed, and tested. +1. [Documented](../documentation/index.md) in the `/doc` directory. +1. [Changelog entry added](../changelog.md), if necessary. +1. Reviewed by relevant (UX/FE/BE/tech writing) reviewers and all concerns are addressed. +1. Merged by a project maintainer. +1. Added to the [release post](https://about.gitlab.com/handbook/marketing/blog/release-posts/), + if relevant. +1. Added to [the website](https://gitlab.com/gitlab-com/www-gitlab-com/blob/master/data/features.yml), if relevant. +1. [Black-box tests/end-to-end tests](../testing_guide/testing_levels.md#black-box-tests-at-the-system-level-aka-end-to-end-tests) + added if required. Please contact [the quality team](https://about.gitlab.com/handbook/engineering/quality/#teams) + with any questions. + +## Dependencies If you add a dependency in GitLab (such as an operating system package) please -consider updating the following and note the applicability of each in your -merge request: - -1. Note the addition in the release blog post (create one if it doesn't exist yet) <https://gitlab.com/gitlab-com/www-gitlab-com/merge_requests/> -1. Upgrade guide, for example <https://gitlab.com/gitlab-org/gitlab-ce/blob/master/doc/update/7.5-to-7.6.md> -1. Installation guide <https://gitlab.com/gitlab-org/gitlab-ce/blob/master/doc/install/installation.md#1-packages-dependencies> -1. GitLab Development Kit <https://gitlab.com/gitlab-org/gitlab-development-kit> -1. Test suite <https://gitlab.com/gitlab-org/gitlab-ce/blob/master/scripts/prepare_build.sh> -1. Omnibus package creator <https://gitlab.com/gitlab-org/omnibus-gitlab> - -[definition-of-done]: http://guide.agilealliance.org/guide/definition-of-done.html -[testing]: ../testing_guide/index.md - ---- - -[Return to Contributing documentation](index.md) - -[changelog]: ../changelog.md "Generate a changelog entry" -[doc-guidelines]: ../documentation/index.md "Documentation guidelines" +consider updating the following, and note the applicability of each in your merge +request: + +1. Note the addition in the [release blog post](https://about.gitlab.com/handbook/marketing/blog/release-posts/) + (create one if it doesn't exist yet). +1. [The upgrade guide](../../update/upgrading_from_source.md). +1. The [GitLab Installation Guide](../../install/installation.md#1-packages-and-dependencies). +1. The [GitLab Development Kit](https://gitlab.com/gitlab-org/gitlab-development-kit). +1. The [CI environment preparation](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/scripts/prepare_build.sh). +1. The [Omnibus package creator](https://gitlab.com/gitlab-org/omnibus-gitlab). diff --git a/doc/development/contributing/style_guides.md b/doc/development/contributing/style_guides.md index 0eedef5e14f..f319d00d7fe 100644 --- a/doc/development/contributing/style_guides.md +++ b/doc/development/contributing/style_guides.md @@ -10,7 +10,7 @@ 1. [Testing][testing] 1. [JavaScript styleguide][js-styleguide] 1. [SCSS styleguide][scss-styleguide] -1. [Shell commands](../shell_commands.md) created by GitLab +1. [Shell commands (Ruby)](../shell_commands.md) created by GitLab contributors to enhance security 1. [Database Migrations](../migration_style_guide.md) 1. [Markdown](http://www.cirosantilli.com/markdown-styleguide) @@ -22,6 +22,7 @@ text should be _sorry, we could not create your account because:_ 1. Code should be written in [US English][us-english] 1. [Go](../go_guide/index.md) +1. [Python](../python_guide/index.md) This is also the style used by linting tools such as [RuboCop](https://github.com/bbatsov/rubocop) and [Hound CI](https://houndci.com). diff --git a/doc/development/database_debugging.md b/doc/development/database_debugging.md index f00c5ccb9e9..68d33a9d8e0 100644 --- a/doc/development/database_debugging.md +++ b/doc/development/database_debugging.md @@ -56,7 +56,7 @@ bundle exec rails db RAILS_ENV=development ### `ActiveRecord::PendingMigrationError` with Spring -When running specs with the [Spring preloader](./rake_tasks.md#speed-up-tests-rake-tasks-and-migrations), +When running specs with the [Spring preloader](rake_tasks.md#speed-up-tests-rake-tasks-and-migrations), the test database can get into a corrupted state. Trying to run the migration or dropping/resetting the test database has no effect. diff --git a/doc/development/distributed_tracing.md b/doc/development/distributed_tracing.md new file mode 100644 index 00000000000..bfce7488a8d --- /dev/null +++ b/doc/development/distributed_tracing.md @@ -0,0 +1,182 @@ +# Distributed Tracing - development guidelines + +GitLab is instrumented for distributed tracing. + +According to [Open Tracing](https://opentracing.io/docs/overview/what-is-tracing/): + +> Distributed tracing, also called distributed request tracing, is a method used to profile and +> monitor applications, especially those built using a microservices architecture. Distributed +> tracing helps to pinpoint where failures occur and what causes poor performance. + +Distributed tracing is especially helpful in understanding the lifecycle of a request as it passes +through the different components of the GitLab application. At present, Workhorse, Rails, Sidekiq, +and Gitaly support tracing instrumentation. + +Distributed tracing adds minimal overhead when disabled, but imposes only small overhead when +enabled and is therefore capable in any environment, including production. For this reason, it can +be useful in diagnosing production issues, particularly performance problems. + +## Enabling distributed tracing + +GitLab uses the `GITLAB_TRACING` environment variable to configure distributed tracing. The same +configuration is used for all components (e.g., Workhorse, Rails, etc). + +When `GITLAB_TRACING` is not set, the application will not be instrumented, meaning that there is +no overhead at all. + +To enable `GITLAB_TRACING`, a valid _"configuration-string"_ value should be set, with a URL-like +form: + +```console +GITLAB_TRACING=opentracing://<driver>?<param_name>=<param_value>&<param_name_2>=<param_value_2> +``` + +In this example, we have the following hypothetical values: + +- `driver`: the driver. [GitLab supports + `jaeger`](../user/project/operations/tracing.md). In future, other + tracing implementations may also be supported. +- `param_name`, `param_value`: these are driver specific configuration values. Configuration + parameters for Jaeger are documented [further on in this + document](#2-configure-the-gitlab_tracing-environment-variable) they should be URL encoded. + Multiple values should be separated by `&` characters like a URL. + +## Using Jaeger in the GitLab Development Kit + +The first tracing implementation that GitLab supports is Jaeger, and the [GitLab Development +Kit](https://gitlab.com/gitlab-org/gitlab-development-kit/) supports distributed tracing with +Jaeger out-of-the-box. + +The easiest way to access tracing from a GDK environment is through the +[performance-bar](../administration/monitoring/performance/performance_bar.md). This can be shown +by typing `p` `b` in the browser window. + + + +Once the performance bar is enabled, click on the **Trace** link in the performance bar to go to +the Jaeger UI. + +The Jaeger search UI will return a query for the `Correlation-ID` of the current request. Normally, +this search should return a single trace result. Clicking this result will show the detail of the +trace in a hierarchical time-line. + + + +## Using Jaeger without the GitLab Developer Kit + +Distributed Tracing can be enabled in non-GDK development environments as well as production or +staging environments, for troubleshooting. Please note that at this time, this functionality is +experimental, and not supported in production environments at present. In this first release, it is intended to be +used for debugging in development environments only. + +Jaeger tracing can be enabled through a three-step process: + +1. [Start Jaeger](#1-start-jaeger). +1. [Configure the `GITLAB_TRACING` environment variable](#2-configure-the-gitlab_tracing-environment-variable). +1. [Start the GitLab application](#3-start-the-gitlab-application). +1. [Go to the Jaeger Search UI in your browser](#4-open-the-jaeger-search-ui). + +### 1. Start Jaeger + +Jaeger has many configuration options, but is very easy to start in an "all-in-one" mode which uses +memory for trace storage (and is therefore non-persistent). The main advantage of "all-in-one" mode +being ease of use. + +For more detailed configuration options, refer to the [Jaeger +documentation](https://www.jaegertracing.io/docs/1.9/getting-started/). + +#### Using Docker + +If you have Docker available, the easier approach to running the Jaeger all-in-one is through +Docker, using the following command: + +```console +$ docker run \ + --rm \ + -e COLLECTOR_ZIPKIN_HTTP_PORT=9411 \ + -p 5775:5775/udp \ + -p 6831:6831/udp \ + -p 6832:6832/udp \ + -p 5778:5778 \ + -p 16686:16686 \ + -p 14268:14268 \ + -p 9411:9411 \ + jaegertracing/all-in-one:latest +``` + +#### Using the Jaeger process + +Without Docker, the all-in-one process is still easy to setup. + +1. Download the [latest Jaeger release](https://github.com/jaegertracing/jaeger/releases) for your + platform. +1. Extract the archive and run the `bin/all-in-one` process. + +This should start the process with the default listening ports. + +### 2. Configure the `GITLAB_TRACING` environment variable + +Once you have Jaeger running, you'll need to configure the `GITLAB_TRACING` variable with the +appropriate configuration string. + +**TL;DR:** If you are running everything on the same host, use the following value: + +```console +$ export GITLAB_TRACING="opentracing://jaeger?http_endpoint=http%3A%2F%2Flocalhost%3A14268%2Fapi%2Ftraces&sampler=const&sampler_param=1" +``` + +This configuration string uses the Jaeger driver `opentracing://jaeger` with the following options: + +| Name | Value | Description | +|------|-------|-------------| +| `http_endpoint` | `http://localhost:14268/api/traces` | Configures Jaeger to send trace information to the HTTP endpoint running on `http://localhost:14268/`. Alternatively, the `upd_endpoint` can be used. | +| `sampler` | `const` | Configures Jaeger to use the constant sampler (either on or off). | +| `sampler_param` | `1` | Configures the `const` sampler to sample _all_ traces. Using `0` would sample _no_ traces. | + +**Other parameter values are also possible:** + +| Name | Example | Description | +|------|-------|-------------| +| `udp_endpoint` | `localhost:6831` | This is the default. Configures Jaeger to send trace information to the UDP listener on port `6831` using compact thrift protocol. Note that we've experienced some issues with the [Jaeger Client for Ruby](https://github.com/salemove/jaeger-client-ruby) when using this protocol. | +| `sampler` | `probabalistic` | Configures Jaeger to use a probabilistic random sampler. The rate of samples is configured by the `sampler_param` value. | +| `sampler_param` | `0.01` | Use a ratio of `0.01` to configure the `probabalistic` sampler to randomly sample _1%_ of traces. | + +NOTE: **Note:** +The same `GITLAB_TRACING` value should to be configured in the environment +variables for all GitLab processes, including Workhorse, Gitaly, Rails, and Sidekiq. + +### 3. Start the GitLab application + +Once the `GITLAB_TRACING` environment variable is exported to all GitLab services, start the +application. + +When `GITLAB_TRACING` is configured properly, the application will log this on startup: + +```console +13:41:53 gitlab-workhorse.1 | 2019/02/12 13:41:53 Tracing enabled +... +13:41:54 gitaly.1 | 2019/02/12 13:41:54 Tracing enabled +... +``` + +If `GITLAB_TRACING` is not configured correctly, this will also be logged: + +```console +13:43:45 gitaly.1 | 2019/02/12 13:43:45 skipping tracing configuration step: tracer: unable to load driver mytracer +``` + +By default, GitLab ships with the Jaeger tracer, but other tracers can be included at compile time. +Details of how this can be done are included in the [LabKit tracing +documentation](https://godoc.org/gitlab.com/gitlab-org/labkit/tracing). + +If no log messages about tracing are emitted, the `GITLAB_TRACING` environment variable is likely +not set. + +### 4. Open the Jaeger Search UI + +By default, the Jaeger search UI is available at <http://localhost:16686/search>. + +TIP: **Tip:** +Don't forget that you will need to generate traces by using the application before +they appear in the Jaeger UI. + diff --git a/doc/development/documentation/feature-change-workflow.md b/doc/development/documentation/feature-change-workflow.md index 3f31fe5ca19..ca29353ecbe 100644 --- a/doc/development/documentation/feature-change-workflow.md +++ b/doc/development/documentation/feature-change-workflow.md @@ -78,7 +78,7 @@ For issues requiring any new or updated documentation, the Product Manager (PM) must: - Add the `Documentation` label. -- Confirm or add the [documentation requirements](#documentation-requirements). +- Confirm or add the [documentation requirements](#documentation-requirements-in-feature-issues). - Ensure the issue contains any new or updated feature name, overview/description, and use cases, as required per the [documentation structure and template](structure.md), when applicable. @@ -86,7 +86,7 @@ Everyone is encouraged to draft the requirements in the issue, but a product man do the following: - When the issue is assigned a release milestone, review and update the Documentation details. -- By the kickoff, finalizie the Documentation details. +- By the kickoff, finalize the Documentation details. ### Developer and maintainer roles @@ -117,7 +117,7 @@ Follow the process below unless otherwise agreed with the product manager and te #### Reviews and merging -All reviewers can help ensure accuracy, clarity, completeness, and adherence to the plans in the issue, as well as the [Documentation Guidelines](https://docs.gitlab.com/ee/development/documentation/) and [Style Guide](https://docs.gitlab.com/ee/development/documentation/styleguide.html). +All reviewers can help ensure accuracy, clarity, completeness, and adherence to the plans in the issue, as well as the [Documentation Guidelines](index.md) and [Style Guide](styleguide.md). - **Prior to merging**, documentation changes committed by the developer must be reviewed by: @@ -136,7 +136,7 @@ All reviewers can help ensure accuracy, clarity, completeness, and adherence to 1. **The maintainer** who is assigned to merge the MR, to verify clarity, completeness, and quality, to the best of their ability. - Upon merging, if a technical writer review has not been performed and there is not yet a linked issue for a follow-up review, the maintainer should [create an issue using the Doc Review template](https://gitlab.com/gitlab-org/gitlab-ce/issues/new?issuable_template=Doc%20Review), link it from the MR, and - mention the original MR author in the new issue. Alternatively, the mainitainer can ask the MR author to create and link this issue before the MR is merged. + mention the original MR author in the new issue. Alternatively, the maintainer can ask the MR author to create and link this issue before the MR is merged. - After merging, documentation changes are reviewed by: @@ -157,14 +157,14 @@ All reviewers can help ensure accuracy, clarity, completeness, and adherence to #### Collaboration By default, the developer will work on documentation changes independently, but -the developer, PM, or technicial writer can propose a broader collaboration for +the developer, PM, or technical writer can propose a broader collaboration for any given issue. Additionally, technical writers are available for questions at any time. #### Review -- Techncial writers provide non-blocking reviews of all documentation changes, +- Technical writers provide non-blocking reviews of all documentation changes, before or after the change is merged. However, if the docs are ready in the MR while there's time before the freeze, the technical writer's review can commence early, on request. - The technical writer will confirm that the doc is clear, grammatically correct, @@ -173,7 +173,7 @@ Additionally, technical writers are available for questions at any time. the developer and code reviewer should have already made a good-faith effort to ensure: - Clarity. - Adherence to the plans and goals in the issue. - - Location (make sure the docs are in the correct directorkes and has the correct name). + - Location (make sure the docs are in the correct directories and has the correct name). - Syntax, typos, and broken links. - Improvements to the content. - Accordance with the [Documentation Style Guide](styleguide.md), and [Structure and Template](structure.md) doc. diff --git a/doc/development/documentation/index.md b/doc/development/documentation/index.md index 652fe3ea711..c7fa40af930 100644 --- a/doc/development/documentation/index.md +++ b/doc/development/documentation/index.md @@ -16,13 +16,13 @@ In addition to this page, the following resources to help craft and contribute d - [Markdown Guide](https://about.gitlab.com/handbook/product/technical-writing/markdown-guide/) - A reference for the markdown implementation used by GitLab's documentation site and about.gitlab.com. - [Site architecture](site_architecture/index.md) - How docs.gitlab.com is built. -## Source and rendered locations +## Source files and rendered web locations Documentation for GitLab Community Edition (CE) and Enterprise Edition (EE), along with GitLab Runner and Omnibus, is published to [docs.gitlab.com](https://docs.gitlab.com). The documentation for CE and EE is also published within the application at `/help` on the domain of the GitLab instance. At `/help`, only content for your current edition and version is included, whereas multiple versions' content is available at docs.gitlab.com. -The source of the documentation is maintained in the following repository locations: +The source of the documentation exists within the codebase of each GitLab application in the following repository locations: | Project | Path | | --- | --- | @@ -48,95 +48,67 @@ as its markdown rendering engine. See the [GitLab Markdown Guide](https://about. Adhere to the [Documentation Style Guide](styleguide.md). If a style standard is missing, you are welcome to suggest one via a merge request. -## Documentation directory structure - -The documentation is structured based on the GitLab UI structure itself, -separated by [`user`](https://gitlab.com/gitlab-org/gitlab-ce/tree/master/doc/user), -[`administrator`](https://gitlab.com/gitlab-org/gitlab-ce/tree/master/doc/administration), and [`contributor`](https://gitlab.com/gitlab-org/gitlab-ce/tree/master/doc/development). - -In order to have a [solid site structure](https://searchengineland.com/seo-benefits-developing-solid-site-structure-277456) for our documentation, -all docs should be linked. Every new document should be cross-linked to its related documentation, and linked from its topic-related index, when existent. - -The directories `/workflow/`, `/gitlab-basics/`, `/university/`, and `/articles/` have -been **deprecated** and the majority their docs have been moved to their correct location -in small iterations. Please do not create new docs in these folders. Organize docs by product area and subject, not type. - -### Documentation files - -- When you create a new directory, always start with an `index.md` file. - Do not use another file name and **do not** create `README.md` files. -- **Do not** use special chars and spaces, or capital letters in file names, - directory names, branch names, and anything that generates a path. -- Max screenshot size: 100KB. -- We do not support videos (yet). - -### Location and naming documents - -Our goal is to have a clear hierarchical structure with meaningful URLs -like `docs.gitlab.com/user/project/merge_requests/`. With this pattern, -you can immediately tell that you are navigating to user-related documentation -about project features; specifically about merge requests. Our site's paths match -those of our repository, so the clear structure also makes documentation easier to update. - -While the documentation is home to a variety of content types, we do not organize by content type. -For example, do not create groupings of similar media types (e.g. indexes of all articles, videos, etc.). -Similarly, we do not use glossaries or FAQs. Such grouping of content by type makes -it difficult to browse for the information you need and difficult to maintain up-to-date content. -Instead, organize content by its subject (e.g. everything related to CI goes together) -and cross-link between any related content. - -Do not simply link out to GitLab technical blog posts. There should be an up-to-date -single source of truth on the topic within the documentation, and the top of the -blog post should be updated to link to that doc. - -The table below shows what kind of documentation goes where. - -| Directory | What belongs here | -|:----------------------|:---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| `doc/user/` | User related documentation. Anything that can be done within the GitLab UI goes here including `/admin`. | -| `doc/administration/` | Documentation that requires the user to have access to the server where GitLab is installed. The admin settings that can be accessed via GitLab's interface go under `doc/user/admin_area/`. | -| `doc/api/` | API related documentation. | -| `doc/development/` | Documentation related to the development of GitLab. Related process and style guides should go here. | -| `doc/legal/` | Legal documents about contributing to GitLab. | -| `doc/install/` | Probably the most visited directory, since `installation.md` is there. Ideally this should go under `doc/administration/`, but it's best to leave it as-is in order to avoid confusion (still debated though). | -| `doc/update/` | Same with `doc/install/`. Should be under `administration/`, but this is a well known location, better leave as-is, at least for now. | -| `doc/topics/` | Indexes per Topic (`doc/topics/topic-name/index.md`): all resources for that topic (user and admin documentation, articles, and third-party docs) | - -**General rules & best practices:** - -1. When creating a new document and it has more than one word in its name, - make sure to use underscores instead of spaces or dashes (`-`). For example, - a proper naming would be `import_projects_from_github.md`. The same rule - applies to images. -1. Start a new directory with an `index.md` file. -1. There are four main directories, `user`, `administration`, `api` and `development`. -1. The `doc/user/` directory has five main subdirectories: `project/`, `group/`, - `profile/`, `dashboard/` and `admin_area/`. - 1. `doc/user/project/` should contain all project related documentation. - 1. `doc/user/group/` should contain all group related documentation. - 1. `doc/user/profile/` should contain all profile related documentation. - Every page you would navigate under `/profile` should have its own document, - i.e. `account.md`, `applications.md`, `emails.md`, etc. - 1. `doc/user/dashboard/` should contain all dashboard related documentation. - 1. `doc/user/admin_area/` should contain all admin related documentation - describing what can be achieved by accessing GitLab's admin interface - (_not to be confused with `doc/administration` where server access is - required_). - 1. Every category under `/admin/application_settings` should have its - own document located at `doc/user/admin_area/settings/`. For example, - the **Visibility and Access Controls** category should have a document - located at `doc/user/admin_area/settings/visibility_and_access_controls.md`. -1. The `doc/topics/` directory holds topic-related technical content. Create - `doc/topics/topic-name/subtopic-name/index.md` when subtopics become necessary. - General user- and admin- related documentation, should be placed accordingly. - -If you are unsure where a document or a content addition should live, this should -not stop you from authoring and contributing. You can use your best judgment and -then ask the reviewer of your MR to confirm your decision, and/or ask a technical writer -at any stage in the process. The techncial writing team will review all documentation -changes, regardless, and can move content if there is a better place for it. - -### Changing document location +## Folder structure and files + +See the [Structure](styleguide.md#structure) section of the [Documentation Style Guide](styleguide.md). + +## Single codebase + +We currently maintain two sets of docs: one in the +[gitlab-ce](https://gitlab.com/gitlab-org/gitlab-ce/tree/master/doc) repo and +one in [gitlab-ee](https://gitlab.com/gitlab-org/gitlab-ee/tree/master/doc). +They are similar, and most pages are identical, but they are different repositories. +With the single codebase effort, we want to make those two sets identical, so when the +time comes to have only one codebase, we'll be ready. + +Here are some links to get you up to speed with the current effort: + +- [CE/EE codebases blueprint](https://about.gitlab.com/handbook/engineering/infrastructure/blueprint/ce-ee-codebases/) +- [CE/EE codebases merge design](https://about.gitlab.com/handbook/engineering/infrastructure/design/merge-ce-ee-codebases/) +- [Single docs codebase epic](https://gitlab.com/groups/gitlab-org/-/epics/199) +- [Issue board of related issues](https://gitlab.com/groups/gitlab-org/-/boards/981090?&label_name[]=Documentation&label_name[]=single%20codebase) +- [Related merge requests](https://gitlab.com/groups/gitlab-org/-/merge_requests?scope=all&utf8=%E2%9C%93&state=all&label_name[]=Documentation&label_name[]=single%20codebase) +- [Visualize the existing diffs](https://leipert-projects.gitlab.io/is-gitlab-pretty-yet/diff/?search=%5Edoc) + +### CE first + +After a given documentation path is aligned across CE and EE, all merge requests +affecting that path must be submitted to CE, regardless of the content it has. +This means that: + +* For **EE-only docs changes**, you only have to submit a CE MR. +* For **EE-only features** that touch both the code and the docs, you have to submit +an EE MR containing all changes, and a CE MR containing only the docs changes +and without a changelog entry. + +This might seem like a duplicate effort, but it's only for the short term. +A list of the already aligned docs can be found in +[the epic description](https://gitlab.com/groups/gitlab-org/-/epics/199#ee-specific-lines-check). + +Since the docs will be combined, it's crucial to add the relevant +[product badges](styleguide.md#product-badges) for all EE documentation, so that +we can discern which features belong to which tier. + +### EE specific lines check + +There's a special test in place +([`ee_specific_check.rb`](https://gitlab.com/gitlab-org/gitlab-ee/blob/master/scripts/ee_specific_check/ee_specific_check.rb)), +which, among others, checks and prevents creating/editing new files and directories +in EE under `doc/`. + +We have a long list of documentation paths that are either whitelisted or not. +Paths in the whitelist (not commented out) will not be subject to the test, +which means you are allowed to create/change docs content in EE for the time +being. The goal is to not have any doc whitelisted. + +At the time of this writing, the only items left to be aligned are the API docs: + +- `doc/api/*` ([issue](https://gitlab.com/gitlab-org/gitlab-ce/issues/60045) / [merge request](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/27491)) + +Eventually, once all docs are aligned, we'll remove any doc reference from that +script, so it catches everything. + +## Changing document location Changing a document's location requires specific steps to be followed to ensure that users can seamlessly access the new doc page, whether they are accesing content @@ -194,7 +166,7 @@ Things to note: built-in help page, that's why we omit it in `git grep`. - Use the checklist on the "Change documentation location" MR description template. -#### Alternative redirection method +### Alternative redirection method Alternatively to the method described above, you can simply replace the content of the old file with a frontmatter containing a redirect link: @@ -212,7 +184,7 @@ This redirection method will not provide a redirect fallback on GitLab `/help`. it, make sure to add a link to the new page on the doc, otherwise it's a dead end for users that land on the doc via `/help`. -#### Redirections for pages with Disqus comments +### Redirections for pages with Disqus comments If the documentation page being relocated already has any Disqus comments, we need to preserve the Disqus thread. @@ -397,7 +369,7 @@ to merge changes that will break `master` from a merge request with a successful ## Docs site architecture See the [Docs site architecture](site_architecture/index.md) page to learn -how we build and deploy the site at [docs.gitlab.com](https://docs.gitlab.com), and +how we build and deploy the site at [docs.gitlab.com](https://docs.gitlab.com) and to review all the assets and libraries in use. ### Global navigation @@ -499,7 +471,7 @@ If you want to know the in-depth details, here's what's really happening: The following GitLab features are used among others: - [Manual actions](../../ci/yaml/README.md#whenmanual) -- [Multi project pipelines](https://docs.gitlab.com/ee/ci/multi_project_pipeline_graphs.html) +- [Multi project pipelines](../../ci/multi_project_pipeline_graphs.md) - [Review Apps](../../ci/review_apps/index.md) - [Artifacts](../../ci/yaml/README.md#artifacts) - [Specific Runner](../../ci/runners/README.md#locking-a-specific-runner-from-being-enabled-for-other-projects) @@ -511,10 +483,10 @@ Currently, the following tests are in place: 1. `docs lint`: Check that all internal (relative) links work correctly and that all cURL examples in API docs use the full switches. It's recommended - to [check locally](#previewing-locally) before pushing to GitLab by executing the command + to [check locally](#previewing-the-changes-live) before pushing to GitLab by executing the command `bundle exec nanoc check internal_links` on your local [`gitlab-docs`](https://gitlab.com/gitlab-com/gitlab-docs) directory. -1. [`ee_compat_check`](../automatic_ce_ee_merge.md#avoiding-ce-gt-ee-merge-conflicts-beforehand) (runs on CE only): +1. [`ee_compat_check`](../automatic_ce_ee_merge.md#avoiding-ce-ee-merge-conflicts-beforehand) (runs on CE only): When you submit a merge request to GitLab Community Edition (CE), there is this additional job that runs against Enterprise Edition (EE) and checks if your changes can apply cleanly to the EE codebase. @@ -522,6 +494,10 @@ Currently, the following tests are in place: As CE is merged into EE once a day, it's important to avoid merge conflicts. Submitting an EE-equivalent merge request cherry-picking all commits from CE to EE is essential to avoid them. +1. [`ee-files-location-check`/`ee-specific-lines-check`](#ee-specific-lines-check) (runs on EE only): + This test ensures that no new files/directories are created/changed in EE. + All docs should be submitted in CE instead, regardless the tier they are on. + This is for the [single codebase](#single-codebase) effort. 1. In a full pipeline, tests for [`/help`](#gitlab-help-tests). ### Linting diff --git a/doc/development/documentation/site_architecture/global_nav.md b/doc/development/documentation/site_architecture/global_nav.md index 0aa3c41a225..f2f4f5f0e1c 100644 --- a/doc/development/documentation/site_architecture/global_nav.md +++ b/doc/development/documentation/site_architecture/global_nav.md @@ -62,7 +62,7 @@ the consent of one of the technical writers. The global nav is built from two files: - [Data](#data-file) -- [Layout](#layout-file) +- [Layout](#layout-file-logic) The data file feeds the layout with the links to the docs. The layout organizes the data among the nav in containers properly [styled](#css-classes). diff --git a/doc/development/documentation/structure.md b/doc/development/documentation/structure.md index 95b5fcd99a1..fe676efa94d 100644 --- a/doc/development/documentation/structure.md +++ b/doc/development/documentation/structure.md @@ -77,8 +77,8 @@ for use (e.g. variations on the main use case), but if that's not applicable, th Examples of use cases on feature pages: - CE and EE: [Issues](../../user/project/issues/index.md#use-cases) - CE and EE: [Merge Requests](../../user/project/merge_requests/index.md) -- EE-only: [Geo](https://docs.gitlab.com/ee/administration/geo/replication/index.html) -- EE-only: [Jenkins integration](https://docs.gitlab.com/ee/integration/jenkins.html) +- EE-only: [Geo](../../administration/geo/replication/index.md) +- EE-only: [Jenkins integration](../../integration/jenkins.md) ## Requirements @@ -121,8 +121,8 @@ but commented out to help encourage others to add to it in the future. --> Notes: -- (1): Apply the [tier badges](https://docs.gitlab.com/ee/development/documentation/styleguide.html#product-badges) accordingly -- (2): Apply the correct format for the [GitLab version introducing the feature](https://docs.gitlab.com/ee/development/documentation/styleguide.html#gitlab-versions-and-tiers) +- (1): Apply the [tier badges](styleguide.md#product-badges) accordingly +- (2): Apply the correct format for the [GitLab version introducing the feature](styleguide.md#gitlab-versions-and-tiers) ``` ## Help and feedback section diff --git a/doc/development/documentation/styleguide.md b/doc/development/documentation/styleguide.md index 7a3a8f25c2d..5caca846cc9 100644 --- a/doc/development/documentation/styleguide.md +++ b/doc/development/documentation/styleguide.md @@ -4,65 +4,192 @@ description: 'Writing styles, markup, formatting, and other standards for GitLab # Documentation Style Guide -The documentation style guide defines the markup structure used in -GitLab documentation. Check the -[documentation guidelines](index.md) for general development instructions. +This document defines the standards for GitLab's documentation content and files. -See the GitLab handbook for the [writing style guidelines](https://about.gitlab.com/handbook/communication/#writing-style-guidelines). +For broader information about the documentation, see the [Documentation guidelines](index.md). For programmatic help adhering to the guidelines, see [linting](index.md#linting). -## Files +See the GitLab handbook for further [writing style guidelines](https://about.gitlab.com/handbook/communication/#writing-style-guidelines) +that apply to all GitLab content, not just documentation. -- [Directory structure](index.md#location-and-naming-documents): place the docs - in the correct location. -- [Documentation files](index.md#documentation-files): name the files accordingly. +## Documentation is the single source of truth (SSOT) -DANGER: **Attention:** -**Do not** use capital letters, spaces, or special chars in file names, -branch names, directory names, headings, or in anything that generates a path. +### Why a single source of truth -NOTE: **Note:** -**Do not** create new `README.md` files, name them `index.md` instead. There's -a test that will fail if it spots a new `README.md` file. +The documentation is the SSOT for all information related to the implementation, usage, and troubleshooting of GitLab products and features. It evolves continually, in keeping with new products and features, and with improvements for clarity, accuracy, and completeness. + +This policy prevents information silos, ensuring that it remains easy to find information about GitLab products. + +It also informs decisions about the kinds of content we include in our documentation. + +The documentation is a continually evolving SSOT for all information related to the implementation, usage, and troubleshooting of GitLab products and features. + +### All information + +Include problem-solving actions that may address rare cases or be considered 'risky', so long as proper context is provided in the form of fully detailed warnings and caveats. This kind of content should be included as it could be helpful to others and, when properly explained, its benefits outweigh the risks. If you think you have found an exception to this rule, contact the Technical Writing team. + +We will add all troubleshooting information to the documentation, no matter how unlikely a user is to encounter a situation. +For the Troubleshooting sections, people in GitLab Support can merge additions themselves. + +### All media types + +Include any media types/sources if the content is relevant to readers. You can freely include or link presentations, diagrams, videos, etc.; no matter who it was originally composed for, if it is helpful to any of our audiences, we can include it. + + - If you use an image that has a separate source file (for example, a vector or diagram format), link the image to the source file so that it may be reused or updated by anyone. + - Do not copy and paste content from other sources unless it is a limited quotation with the source cited. Typically it is better to either rephrase relevant information in your own words or link out to the other source. + +### No special types + +In the software industry, it is a best practice to organize documentatioin in different types. For example, [Divio recommends](https://www.divio.com/blog/documentation/): + +1. Tutorials +2. How-to guides +3. Explanation +4. Reference (for example, a glossary) + +At GitLab, we have so many product changes in our monthly releases that we can't afford to continually update multiple types of information. +If we have multiple types, the information will become outdated. Therefore, we have a [single template](structure.md) for documentation. + +We currently do not distinguish specific document types, although we are open to reconsidering this policy +once the documentation has reached a future stage of maturity and quality. If you are reading this, then despite our +continual improvement efforts, that point hasn't been reached. + +### Link instead of summarize + +There is a temptation to summarize the information on another page. +This will cause the information to live in two places. +Instead, link to the SSOT and explain why it is important to consume the information. + +### Organize by topic, not by type + +Beyond top-level audience-type folders (e.g. `administration`), we organize content by topic, not by type, so that it can be located as easily as possible within the single-source-of-truth (SSOT) section for the subject matter. + +For example, do not create groupings of similar media types (e.g. glossaries, FAQs, or sets of all articles or videos). + +Such grouping of content by type makes +it difficult to browse for the information you need and difficult to maintain up-to-date content. +Instead, organize content by its subject (e.g. everything related to CI goes together) +and cross-link between any related content. + +### Docs-first methodology + +We employ a **docs-first methodology** to help ensure that the docs remain a complete and trusted resource, and to make communicating about the use of GitLab more efficient. + +* If the answer to a question exists in documentation, share the link to the docs instead of rephrasing the information. +* When you encounter new information not available in GitLab’s documentation (for example, when working on a support case or testing a feature), your first step should be to create a merge request to add this information to the docs. You can then share the MR in order to communicate this information. + +New information that would be useful toward the future usage or troubleshooting of GitLab should not be written directly in a forum or other messaging system, but added to a docs MR and then referenced, as described above. Note that among any other doc changes, you can always add a Troubleshooting section to a doc if none exists, or un-comment and use the placeholder Troubleshooting section included as part of our [doc template](structure.md#template-for-new-docs), if present. + +The more we reflexively add useful information to the docs, the more (and more successfully) the docs will be used to efficiently accomplish tasks and solve problems. -### Markdown +If you have questions when considering, authoring, or editing docs, ask the Technical Writing team on Slack in `#docs` or in GitLab by mentioning the writer for the applicable [DevOps stage](https://about.gitlab.com/handbook/product/categories/#devops-stages). Otherwise, forge ahead with your best effort. It does not need to be perfect; the team is happy to review and improve upon your content. Please review the [Documentation guidelines](index.md) before you begin your first documentation MR. -The [documentation website](https://docs.gitlab.com) had its markdown engine migrated from [Redcarpet to GitLab Kramdown](https://gitlab.com/gitlab-com/gitlab-docs/merge_requests/108) -in October 2018. +Having a knowledge base is any form that is separate from the documentation would be against the docs-first methodology because the content would overlap with the documentation. + +## Markdown + +All GitLab documentation is written using [Markdown](https://en.wikipedia.org/wiki/Markdown). + +The [documentation website](https://docs.gitlab.com) uses GitLab Kramdown as its Markdown rendering engine. For a complete Kramdown reference, see the [GitLab Markdown Kramdown Guide](https://about.gitlab.com/handbook/product/technical-writing/markdown-guide/). The [`gitlab-kramdown`](https://gitlab.com/gitlab-org/gitlab_kramdown) -gem will support all [GFM markup](../../user/markdown.md) in the future. For now, -use regular markdown markup, following the rules on this style guide. For a complete -Kramdown reference, check the [GitLab Markdown Kramdown Guide](https://about.gitlab.com/handbook/product/technical-writing/markdown-guide/). -Use Kramdown markup wisely: do not overuse its specific markup (e.g., `{:.class}`) as it will not render properly in -[`/help`](#gitlab-help). - -## Content - -These guidelines help toward the goal of having every user's search of documentation -yield a useful result, and ensuring content is helpful and easy to consume. - -- What to include: - - Any and all helpful information, processes, and tips for implementing, - using, and troubleshooting GitLab features. [The documentation is the single source of truth](https://about.gitlab.com/handbook/documentation/#documentation-as-single-source-of-truth-ssot) - for this information. - - 'Risky' or niche problem-solving steps. There is no reason to withhold these or - store them elsewhere; simply include them along with the rest of the docs including all necessary - detail, such as specific warnings and caveats about potential ramifications. - - Any content types/sources, if relevant to users or admins. You can freely - include presentations, videos, etc.; no matter who it was originally written for, - if it is helpful to any of our audiences, we can include it. If an outside source - that's under copyright, rephrase, or summarize and link out; do not copy and paste. - - All applicable subsections as described on the [structure and template](structure.md) page, - with files organized in the [correct directory](index.md#documentation-directory-structure). -- To ensure discoverability, link to each doc from its higher-level index page and other related pages. -- When referencing other GitLab products and features, link to their - respective docs; when referencing third-party products or technologies, - link out to their external sites, documentation, and resources. -- Do not duplicate information. +Ruby gem will support all [GFM markup](../../user/markdown.md) in the future. That is, +all markup that is supported for display in the GitLab application itself. For now, +use regular Markdown markup, following the rules in the linked style guide. + +Note that Kramdown-specific markup (e.g., `{:.class}`) will not render properly on GitLab instances under [`/help`](index.md#gitlab-help). + +## Structure + +### Organize by topic, not by type + +Because we want documentation to be a SSOT, we should [organize by topic, not by type](#organize-by-topic-not-by-type). + +### Folder structure overview + +The documentation is separated by top-level audience folders [`user`](https://gitlab.com/gitlab-org/gitlab-ce/tree/master/doc/user), +[`administration`](https://gitlab.com/gitlab-org/gitlab-ce/tree/master/doc/administration), and [`development`](https://gitlab.com/gitlab-org/gitlab-ce/tree/master/doc/development) (contributing) folders. + +Beyond that, we primarily follow the structure of the GitLab user interface or API. + +Our goal is to have a clear hierarchical structure with meaningful URLs +like `docs.gitlab.com/user/project/merge_requests/`. With this pattern, +you can immediately tell that you are navigating to user-related documentation +about Project features; specifically about Merge Requests. Our site's paths match +those of our repository, so the clear structure also makes documentation easier to update. + +The table below shows what kind of documentation goes where. + +| Directory | What belongs here | +|:----------------------|:---------------------------------------------------------------------------------------------------------------------------------| +| `doc/user/` | User related documentation. Anything that can be done within the GitLab UI goes here, including usage of the `/admin` interface. | +| `doc/administration/` | Documentation that requires the user to have access to the server where GitLab is installed. The admin settings that can be accessed via GitLab's interface exist under `doc/user/admin_area/`. | +| `doc/api/` | API related documentation. | +| `doc/development/` | Documentation related to the development of GitLab, whether contributing code or docs. Related process and style guides should go here. | +| `doc/legal/` | Legal documents about contributing to GitLab. | +| `doc/install/` | Contains instructions for installing GitLab. | +| `doc/update/` | Contains instructions for updating GitLab. | +| `doc/topics/` | Indexes per topic (`doc/topics/topic-name/index.md`): all resources for that topic. | + +### Working with directories and files + +1. When you create a new directory, always start with an `index.md` file. + Do not use another file name and **do not** create `README.md` files. +1. **Do not** use special characters and spaces, or capital letters in file names, + directory names, branch names, and anything that generates a path. +1. When creating a new document and it has more than one word in its name, + make sure to use underscores instead of spaces or dashes (`-`). For example, + a proper naming would be `import_projects_from_github.md`. The same rule + applies to images. +1. For image files, do not exceed 100KB. +1. We do not yet support embedded videos. Please link out. +1. There are four main directories, `user`, `administration`, `api` and `development`. +1. The `doc/user/` directory has five main subdirectories: `project/`, `group/`, + `profile/`, `dashboard/` and `admin_area/`. + 1. `doc/user/project/` should contain all project related documentation. + 1. `doc/user/group/` should contain all group related documentation. + 1. `doc/user/profile/` should contain all profile related documentation. + Every page you would navigate under `/profile` should have its own document, + i.e. `account.md`, `applications.md`, `emails.md`, etc. + 1. `doc/user/dashboard/` should contain all dashboard related documentation. + 1. `doc/user/admin_area/` should contain all admin related documentation + describing what can be achieved by accessing GitLab's admin interface + (_not to be confused with `doc/administration` where server access is + required_). + 1. Every category under `/admin/application_settings` should have its + own document located at `doc/user/admin_area/settings/`. For example, + the **Visibility and Access Controls** category should have a document + located at `doc/user/admin_area/settings/visibility_and_access_controls.md`. +1. The `doc/topics/` directory holds topic-related technical content. Create + `doc/topics/topic-name/subtopic-name/index.md` when subtopics become necessary. + General user- and admin- related documentation, should be placed accordingly. +1. The directories `/workflow/`, `/university/`, and `/articles/` have +been **deprecated** and the majority their docs have been moved to their correct location +in small iterations. + +If you are unsure where a document or a content addition should live, this should +not stop you from authoring and contributing. You can use your best judgment and +then ask the reviewer of your MR to confirm your decision, and/or ask a technical writer +at any stage in the process. The techncial writing team will review all documentation +changes, regardless, and can move content if there is a better place for it. + +### Avoid duplication + +Do not include the same information in multiple places. [Link to a SSOT instead.](#link-instead-of-summarize) + +### References across documents + +- Give each folder an index.md page that introduces the topic, introduces the pages within, and links to the pages within (including to the index pages of any next-level subpaths). +- To ensure discoverability, ensure each new or renamed doc is linked from its higher-level index page and other related pages. +- When making reference to other GitLab products and features, link to their respective docs, at least on first mention. +- When making reference to third-party products or technologies, link out to their external sites, documentation, and resources. + +### Structure within documents + +- Include any and all applicable subsections as described on the [structure and template](structure.md) page. - Structure content in alphabetical order in tables, lists, etc., unless there is - a logical reason not to (for example, when mirroring the UI or an ordered sequence). + a logical reason not to (for example, when mirroring the UI or an otherwise ordered sequence). ## Language @@ -80,11 +207,10 @@ yield a useful result, and ensuring content is helpful and easy to consume. ## Text -- Split up long lines (wrap text), this makes it much easier to review and edit. Only - double line breaks are shown as a full line break by creating new paragraphs. - 80-100 characters is the recommended line length. +- Splitting long lines (preferably up to 100 characters) can make it easier to provide feedback on small chunks of text. +- Insert an empty line for new paragraphs. - Use sentence case for titles, headings, labels, menu items, and buttons. -- Jump a line between different markups (e.g., after every paragraph, header, list, etc). Example: +- Insert an empty line between different markups (e.g., after every paragraph, header, list, etc). Example: ```md ## Header @@ -95,7 +221,7 @@ yield a useful result, and ensuring content is helpful and easy to consume. - List item 2 ``` -### Tables overlapping the ToC +### Tables overlapping the TOC By default, all tables have a width of 100% on docs.gitlab.com. In a few cases, the table will overlap the table of contents (ToC). @@ -236,6 +362,31 @@ For other punctuation rules, please refer to the E.g., instead of writing something like `Read more about GitLab Issue Boards [here](LINK)`, write `Read more about [GitLab Issue Boards](LINK)`. +### Links requiring permissions + +Don't link directly to: + +- [Confidential issues](../../user/project/issues/confidential_issues.md). +- Project features that require [special permissions](../../user/permissions.md) to view. + +These will fail for: + +- Those without sufficient permissions. +- Automated link checkers. + +Instead: + +- To reduce confusion, mention in the text that the information is either: + - Contained in a confidential issue. + - Requires special permission to a project to view. +- Provide a link in back ticks (`` ` ``) so that those with access to the issue can easily navigate to it. + +Example: + +```md +For more information, see the [confidential issue](https://docs.gitlab.com/ee/user/project/issues/confidential_issues.html) `https://gitlab.com/gitlab-org/gitlab-ce/issues/<issue_number>`. +``` + ### Unlinking emails By default, all email addresses will render in an email tag on docs.gitlab.com. @@ -453,41 +604,49 @@ The following are recommended verbs for specific uses. ## GitLab versions and tiers -- Every piece of documentation that comes with a new feature should declare the - GitLab version that feature got introduced. Right below the heading add a - blockquote: +Tagged and released versions of GitLab documentation are available: + +- In the [documentation archives](https://docs.gitlab.com/archives/). +- At the `/help` URL for any GitLab installation. + +The version introducing a new feature is added to the top of the topic in the documentation to provide +a helpful link back to how the feature was developed. + +### Text for documentation requiring version text + +- For features that need to declare the GitLab version that the feature was introduced. Text similar + to the following should be added immediately below the heading as a blockquote: ```md - > Introduced in GitLab 8.3. + > Introduced in GitLab 11.3. ``` -- Whenever possible, every feature should have a link to the issue, MR or epic - (in that order) that introduced it. The above quote would be then transformed to: +- Whenever possible, version text should have a link to the issue, merge request, or epic that introduced the feature. + An issue is preferred over a merge request, and a merge request is preferred over an epic. For example: ```md - > [Introduced](<link-to-issue>) in GitLab 8.3. + > [Introduced](<link-to-issue>) in GitLab 11.3. ``` -- If the feature is only available in GitLab Enterprise Edition, don't forget to mention +- If the feature is only available in GitLab Enterprise Edition, mention the [paid tier](https://about.gitlab.com/handbook/marketing/product-marketing/#tiers) the feature is available in: ```md - > [Introduced](<link-to-issue>) in [GitLab Starter](https://about.gitlab.com/pricing/) 10.3. + > [Introduced](<link-to-issue>) in [GitLab Starter](https://about.gitlab.com/pricing/) 11.3. ``` -### Early versions of EE - -If the feature was created before GitLab 9.2 (before [different EE tiers were introduced](https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/1851)): +### Removing version text -- Declare it as "Introduced in GitLab Enterprise Edition X.Y". -- Note which tier the feature is available in. +Over time, version text will reference a progressively older version of GitLab. In cases where version text +refers to versions of GitLab four or more major versions back, consider removing the text. -For example: +For example, if the current major version is 11.x, version text referencing versions of GitLab 7.x +and older are candidates for removal. -```md -> [Introduced](<link-to-issue>) in GitLab Enterprise Edition 9.0. Available in [GitLab Premium](https://about.gitlab.com/pricing/). -``` +NOTE: **Note:** +This guidance applies to any text that mentions a GitLab version, not just "Introduced in... " text. +Other text includes deprecation notices and version-specific how-to information. ## Product badges @@ -507,6 +666,12 @@ keyword "only": - For GitLab Premium: `**[PREMIUM ONLY]**`. - For GitLab Ultimate: `**[ULTIMATE ONLY]**`. +For GitLab.com only tiers (when the feature is not available for self-hosted instances): + +- For GitLab Bronze and higher tiers: `**[BRONZE ONLY]**`. +- For GitLab Silver and higher tiers: `**[SILVER ONLY]**`. +- For GitLab Gold: `**[GOLD ONLY]**`. + The tier should be ideally added to headers, so that the full badge will be displayed. However, it can be also mentioned from paragraphs, list items, and table cells. For these cases, the tier mention will be represented by an orange question mark that will show the tiers on hover. @@ -515,6 +680,7 @@ For example: - `**[STARTER]**` renders as **[STARTER]** - `**[STARTER ONLY]**` renders as **[STARTER ONLY]** +- `**[SILVER ONLY]**` renders as **[SILVER ONLY]** The absence of tiers' mentions mean that the feature is available in GitLab Core, GitLab.com Free, and all higher tiers. @@ -612,7 +778,7 @@ In this case: - The code blocks are indented one or more spaces under the list item to render correctly. - Different highlighting languages are used for each config in the code block. -- The [references](#references) guide is used for reconfigure/restart. +- The [GitLab Restart](#gitlab-restart) section is used to explain a required restart/reconfigure of GitLab. ## API diff --git a/doc/development/ee_features.md b/doc/development/ee_features.md index 3e85c0e1995..cca52706ddc 100644 --- a/doc/development/ee_features.md +++ b/doc/development/ee_features.md @@ -19,6 +19,11 @@ CE specs should remain untouched as much as possible and extra specs should be added for EE. Licensed features can be stubbed using the spec helper `stub_licensed_features` in `EE::LicenseHelpers`. +You can force Webpack to act as CE by either deleting the `ee/` directory or by +setting the [`IS_GITLAB_EE` environment variable](https://gitlab.com/gitlab-org/gitlab-ee/blob/master/config/helpers/is_ee_env.js) +to something that evaluates as `false`. The same works for running tests +(for example `IS_GITLAB_EE=0 yarn jest`). + [ee-as-ce]: https://gitlab.com/gitlab-org/gitlab-ee/issues/2500 ## Separation of EE code @@ -161,7 +166,7 @@ still having access the class's implementation with `super`. There are a few gotchas with it: -- you should always [`extend ::Gitlab::Utils::Override`] and use `override` to +- you should always [`extend ::Gitlab::Utils::Override`](utilities.md#overridehttpsgitlabcomgitlab-orggitlab-ceblobmasterlibgitlabutilsoverriderb) and use `override` to guard the "overrider" method to ensure that if the method gets renamed in CE, the EE override won't be silently forgotten. - when the "overrider" would add a line in the middle of the CE @@ -273,8 +278,6 @@ module EE end ``` -[`extend ::Gitlab::Utils::Override`]: utilities.md#override - ##### Overriding CE class methods The same applies to class methods, except we want to use @@ -443,6 +446,19 @@ The disadvantage of this: port `render_if_exists` to CE. - If we have typos in the partial name, it would be silently ignored. + +##### Caveats + +The `render_if_exists` view path argument must be relative to `app/views/` and `ee/app/views`. +Resolving an EE template path that is relative to the CE view path will not work. + +```haml +- # app/views/projects/index.html.haml + += render_if_exists 'button' # Will not render `ee/app/views/projects/_button` and will quietly fail += render_if_exists 'projects/button' # Will render `ee/app/views/projects/_button` +``` + #### Using `render_ce` For `render` and `render_if_exists`, they search for the EE partial first, @@ -532,40 +548,56 @@ due to `prepend`, but Grape is complex internally and we couldn't easily do that, so we'll follow regular object-oriented practices that we define the interface first here. -For example, suppose we have a few more optional params for EE, given this CE -API code: +For example, suppose we have a few more optional params for EE. We can move the +params out of the `Grape::API` class to a helper module, so we can `prepend` it +before it would be used in the class. ```ruby module API - class MergeRequests < Grape::API - # EE::API::MergeRequests would override the following helpers - helpers do - params :optional_params_ee do + class Projects < Grape::API + helpers Helpers::ProjectsHelpers + end +end +``` + +Given this CE API `params`: + +```ruby +module API + module Helpers + module ProjectsHelpers + extend ActiveSupport::Concern + extend Grape::API::Helpers + + params :optional_project_params_ce do + # CE specific params go here... end - end - params :optional_params do - # CE specific params go here... + params :optional_project_params_ee do + end - use :optional_params_ee + params :optional_project_params do + use :optional_project_params_ce + use :optional_project_params_ee + end end end end -API::MergeRequests.prepend(EE::API::MergeRequests) +API::Helpers::ProjectsHelpers.prepend(EE::API::Helpers::ProjectsHelpers) ``` -And then we could override it in EE module: +We could override it in EE module: ```ruby module EE module API - module MergeRequests - extend ActiveSupport::Concern + module Helpers + module ProjectsHelpers + extend ActiveSupport::Concern - prepended do - helpers do - params :optional_params_ee do + prepended do + params :optional_project_params_ee do # EE specific params go here... end end @@ -575,9 +607,6 @@ module EE end ``` -This way, the only difference between CE and EE for that API file would be -`prepend EE::API::MergeRequests`. - #### EE helpers To make it easy for an EE module to override the CE helpers, we need to define @@ -877,15 +906,88 @@ import bundle from 'ee/protected_branches/protected_branches_bundle.js'; import bundle from 'ee_else_ce/protected_branches/protected_branches_bundle.js'; ``` -See the frontend guide [performance section](./fe_guide/performance.md) for +See the frontend guide [performance section](fe_guide/performance.md) for information on managing page-specific javascript within EE. + +## Vue code in `assets/javascript` +### script tag + +#### Child Component only used in EE +To separate Vue template differences we should [async import the components](https://vuejs.org/v2/guide/components-dynamic-async.html#Async-Components). + +Doing this allows for us to load the correct component in EE whilst in CE +we can load a empty component that renders nothing. This code **should** +exist in the CE repository as well as the EE repository. + +```html +<script> +export default { + components: { + EEComponent: () => import('ee_component/components/test.vue'), + }, +}; +</script> + +<template> + <div> + <ee-component /> + </div> +</template> +``` + +#### For JS code that is EE only, like props, computed properties, methods, etc, we will keep the current approach + - Since we [can't async load a mixin](https://github.com/vuejs/vue-loader/issues/418#issuecomment-254032223) we will use the [`ee_else_ce`](../development/ee_features.md#javascript-code-in-assetsjavascripts) alias we already have for webpack. + - This means all the EE specific props, computed properties, methods, etc that are EE only should be in a mixin in the `ee/` folder and we need to create a CE counterpart of the mixin + +##### Example: +```javascript +import mixin from 'ee_else_ce/path/mixin'; + +{ + mixins: [mixin] +} +``` + +- Computed Properties/methods and getters only used in the child import still need a counterpart in CE + +- For store modules, we will need a CE counterpart too. +- You can see an MR with an example [here](https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/9762) + +#### `template` tag +* **EE Child components** + - Since we are using the async loading to check which component to load, we'd still use the component's name, check [this example](#child-component-only-used-in-ee). + +* **EE extra HTML** + - For the templates that have extra HTML in EE we should move it into a new component and use the `ee_else_ce` dynamic import + +### Non Vue Files +For regular JS files, the approach is similar. + +1. We will keep using the [`ee_else_ce`](../development/ee_features.md#javascript-code-in-assetsjavascripts) helper, this means that EE only code should be inside the `ee/` folder. + 1. An EE file should be created with the EE only code, and it should extend the CE counterpart. + 1. For code inside functions that can't be extended, the code should be moved into a new file and we should use `ee_else_ce` helper: + +##### Example: + +```javascript + import eeCode from 'ee_else_ce/ee_code'; + + function test() { + const test = 'a'; + + eeCode(); + + return test; + } +``` + ## SCSS code in `assets/stylesheets` To separate EE-specific styles in SCSS files, if a component you're adding styles for is limited to only EE, it is better to have a separate SCSS file in appropriate directory within `app/assets/stylesheets`. -See [backporting changes](#backporting-changes) for instructions on how to merge changes safely. +See [backporting changes](#backporting-changes-from-ee-to-ce) for instructions on how to merge changes safely. In some cases, this is not entirely possible or creating dedicated SCSS file is an overkill, e.g. a text style of some component is different for EE. In such cases, diff --git a/doc/development/elasticsearch.md b/doc/development/elasticsearch.md new file mode 100644 index 00000000000..8b0f4f02d19 --- /dev/null +++ b/doc/development/elasticsearch.md @@ -0,0 +1,166 @@ +# Elasticsearch knowledge **[STARTER ONLY]** + +This area is to maintain a compendium of useful information when working with elasticsearch. + +Information on how to enable ElasticSearch and perform the initial indexing is kept in ../integration/elasticsearch.md#enabling-elasticsearch + +## Initial installation on OS X + +It is recommended to use the Docker image. After installing docker you can immediately spin up an instance with + +``` +docker run --name elastic56 -p 9200:9200 -p 9300:9300 -e "discovery.type=single-node" docker.elastic.co/elasticsearch/elasticsearch:5.6.12 +``` + +and use `docker stop elastic56` and `docker start elastic56` to stop/start it. + +### Installing on the host + +We currently only support Elasticsearch [5.6 to 6.x](../integration/elasticsearch.md#version-requirements) + +Version 5.6 is available on homebrew and is the recommended version to use in order to test compatibility. + +``` +brew install elasticsearch@5.6 +``` + +There is no need to install any plugins + +## New repo indexer (beta) + +If you're interested on working with the new beta repo indexer, all you need to do is: + +- git clone git@gitlab.com:gitlab-org/gitlab-elasticsearch-indexer.git +- make +- make install + +this adds `gitlab-elasticsearch-indexer` to `$GOPATH/bin`, please make sure that is in your `$PATH`. After that GitLab will find it and you'll be able to enable it in the admin settings area. + +**note:** `make` will not recompile the executable unless you do `make clean` beforehand + +## Helpful rake tasks + +- `gitlab:elastic:test:index_size`: Tells you how much space the current index is using, as well as how many documents are in the index. +- `gitlab:elastic:test:index_size_change`: Outputs index size, reindexes, and outputs index size again. Useful when testing improvements to indexing size. + +Additionally, if you need large repos or multiple forks for testing, please consider [following these instructions](rake_tasks.md#extra-project-seed-options) + +## How does it work? + +The ElasticSearch integration depends on an external indexer. We ship a [ruby indexer](https://gitlab.com/gitlab-org/gitlab-ee/blob/master/bin/elastic_repo_indexer) by default but are also working on an [indexer written in Go](https://gitlab.com/gitlab-org/gitlab-elasticsearch-indexer). The user must trigger the initial indexing via a rake task, but after this is done GitLab itself will trigger reindexing when required via `after_` callbacks on create, update, and destroy that are inherited from [/ee/app/models/concerns/elastic/application_search.rb](https://gitlab.com/gitlab-org/gitlab-ee/blob/master/ee/app/models/concerns/elastic/application_search.rb). + +All indexing after the initial one is done via `ElasticIndexerWorker` (sidekiq jobs). + +Search queries are generated by the concerns found in [ee/app/models/concerns/elastic](https://gitlab.com/gitlab-org/gitlab-ee/tree/master/ee/app/models/concerns/elastic). These concerns are also in charge of access control, and have been a historic source of security bugs so please pay close attention to them! + +## Existing Analyzers/Tokenizers/Filters +These are all defined in https://gitlab.com/gitlab-org/gitlab-ee/blob/master/ee/lib/elasticsearch/git/model.rb + +### Analyzers +#### `path_analyzer` +Used when indexing blobs' paths. Uses the `path_tokenizer` and the `lowercase` and `asciifolding` filters. + +Please see the `path_tokenizer` explanation below for an example. + +#### `sha_analyzer` +Used in blobs and commits. Uses the `sha_tokenizer` and the `lowercase` and `asciifolding` filters. + +Please see the `sha_tokenizer` explanation later below for an example. + +#### `code_analyzer` +Used when indexing a blob's filename and content. Uses the `whitespace` tokenizer and the filters: `code`, `edgeNGram_filter`, `lowercase`, and `asciifolding` + +The `whitespace` tokenizer was selected in order to have more control over how tokens are split. For example the string `Foo::bar(4)` needs to generate tokens like `Foo` and `bar(4)` in order to be properly searched. + +Please see the `code` filter for an explanation on how tokens are split. + +#### `code_search_analyzer` +Not directly used for indexing, but rather used to transform a search input. Uses the `whitespace` tokenizer and the `lowercase` and `asciifolding` filters. + +### Tokenizers +#### `sha_tokenizer` +This is a custom tokenizer that uses the [`edgeNGram` tokenizer](https://www.elastic.co/guide/en/elasticsearch/reference/5.5/analysis-edgengram-tokenizer.html) to allow SHAs to be searcheable by any sub-set of it (minimum of 5 chars). + +example: + +`240c29dc7e` becomes: +- `240c2` +- `240c29` +- `240c29d` +- `240c29dc` +- `240c29dc7` +- `240c29dc7e` + +#### `path_tokenizer` +This is a custom tokenizer that uses the [`path_hierarchy` tokenizer](https://www.elastic.co/guide/en/elasticsearch/reference/5.5/analysis-pathhierarchy-tokenizer.html) with `reverse: true` in order to allow searches to find paths no matter how much or how little of the path is given as input. + +example: + +`'/some/path/application.js'` becomes: +- `'/some/path/application.js'` +- `'some/path/application.js'` +- `'path/application.js'` +- `'application.js'` + +### Filters +#### `code` +Uses a [Pattern Capture token filter](https://www.elastic.co/guide/en/elasticsearch/reference/5.5/analysis-pattern-capture-tokenfilter.html) to split tokens into more easily searched versions of themselves. + +Patterns: +- `"(\\p{Ll}+|\\p{Lu}\\p{Ll}+|\\p{Lu}+)"`: captures CamelCased and lowedCameCased strings as separate tokens +- `"(\\d+)"`: extracts digits +- `"(?=([\\p{Lu}]+[\\p{L}]+))"`: captures CamelCased strings recursively. Ex: `ThisIsATest` => `[ThisIsATest, IsATest, ATest, Test]` +- `'"((?:\\"|[^"]|\\")*)"'`: captures terms inside quotes, removing the quotes +- `"'((?:\\'|[^']|\\')*)'"`: same as above, for single-quotes +- `'\.([^.]+)(?=\.|\s|\Z)'`: separate terms with periods in-between +- `'\/?([^\/]+)(?=\/|\b)'`: separate path terms `like/this/one` + +#### `edgeNGram_filter` +Uses an [Edge NGram token filter](https://www.elastic.co/guide/en/elasticsearch/reference/5.5/analysis-edgengram-tokenfilter.html) to allow inputs with only parts of a token to find the token. For example it would turn `glasses` into permutations starting with `gl` and ending with `glasses`, which would allow a search for "`glass`" to find the original token `glasses` + +## Gotchas + +- Searches can have their own analyzers. Remember to check when editing analyzers +- `Character` filters (as opposed to token filters) always replace the original character, so they're not a good choice as they can hinder exact searches + +## Troubleshooting + +### Getting "flood stage disk watermark [95%] exceeded" + +You might get an error such as + +``` +[2018-10-31T15:54:19,762][WARN ][o.e.c.r.a.DiskThresholdMonitor] [pval5Ct] + flood stage disk watermark [95%] exceeded on + [pval5Ct7SieH90t5MykM5w][pval5Ct][/usr/local/var/lib/elasticsearch/nodes/0] free: 56.2gb[3%], + all indices on this node will be marked read-only +``` + +This is because you've exceeded the disk space threshold - it thinks you don't have enough disk space left, based on the default 95% threshold. + +In addition, the `read_only_allow_delete` setting will be set to `true`. It will block indexing, `forcemerge`, etc + +``` +curl "http://localhost:9200/gitlab-development/_settings?pretty" +``` + +Add this to your `elasticsearch.yml` file: + +``` +# turn off the disk allocator +cluster.routing.allocation.disk.threshold_enabled: false +``` + +_or_ + +``` +# set your own limits +cluster.routing.allocation.disk.threshold_enabled: true +cluster.routing.allocation.disk.watermark.flood_stage: 5gb # ES 6.x only +cluster.routing.allocation.disk.watermark.low: 15gb +cluster.routing.allocation.disk.watermark.high: 10gb +``` + +Restart ElasticSearch, and the `read_only_allow_delete` will clear on it's own. + +_from "Disk-based Shard Allocation | Elasticsearch Reference" [5.6](https://www.elastic.co/guide/en/elasticsearch/reference/5.6/disk-allocator.html#disk-allocator) and [6.x](https://www.elastic.co/guide/en/elasticsearch/reference/6.x/disk-allocator.html)_ diff --git a/doc/development/fe_guide/architecture.md b/doc/development/fe_guide/architecture.md index aebb22caa15..c67389b169e 100644 --- a/doc/development/fe_guide/architecture.md +++ b/doc/development/fe_guide/architecture.md @@ -11,12 +11,9 @@ Architectural decisions should be accessible to everyone, so please document them in the relevant Merge Request discussion or by updating our documentation when appropriate. -You can find the Frontend Architecture experts on the [team page][team-page]. +You can find the Frontend Architecture experts on the [team page](https://about.gitlab.com/team). ## Examples You can find documentation about the desired architecture for a new feature -built with Vue.js [here][vue-section]. - -[team-page]: https://about.gitlab.com/team -[vue-section]: vue.md#frontend.html#how-to-build-a-new-feature-with-vue-js +built with Vue.js [here](vue.md). diff --git a/doc/development/new_fe_guide/event_tracking.md b/doc/development/fe_guide/event_tracking.md index 1958f1ce528..6ab3fa4acf3 100644 --- a/doc/development/new_fe_guide/event_tracking.md +++ b/doc/development/fe_guide/event_tracking.md @@ -1,6 +1,6 @@ # Event Tracking -We use [Snowplow](https://github.com/snowplow/snowplow) for tracking custom events. +We use [Snowplow](https://github.com/snowplow/snowplow) for tracking custom events (available in GitLab [Enterprise Edition](https://about.gitlab.com/pricing/) only). ## Generic tracking function @@ -72,3 +72,20 @@ Below is a list of supported `data-track-*` attributes: | `data-track-value` | The `value` in `trackEvent`. If omitted, this will be `target.value` or empty string. For checkboxes, the default value being tracked will be the element's checked attribute if `data-track-value` is omitted. | false | Since Snowplow is an Enterprise Edition feature, it's necessary to create a CE backport when adding `data-track-*` attributes to HAML templates in most cases. + +## Testing + +Snowplow can be enabled by navigating to: + +- **Admin area > Settings > Integrations** in the UI. +- `admin/application_settings/integrations` in your browser. + +The following configuration is required: + +| Name | Value | +| ------------- | ------------------------- | +| Collector | `snowplow.trx.gitlab.net` | +| Site ID | `gitlab` | +| Cookie domain | `.gitlab.com` | + +Now the implemented tracking events can be inspected locally by looking at the network panel of the browser's development tools. diff --git a/doc/development/fe_guide/frontend_faq.md b/doc/development/fe_guide/frontend_faq.md new file mode 100644 index 00000000000..e4225f2bc39 --- /dev/null +++ b/doc/development/fe_guide/frontend_faq.md @@ -0,0 +1,41 @@ +# Frontend FAQ + +## Rules of Frontend FAQ + +1. **You talk about Frontend FAQ.** + Please share links to it whenever applicable, so more eyes catch when content + gets outdated. +2. **Keep it short and simple.** + Whenever an answer needs more than two sentences it does not belong here. +3. **Provide background when possible.** + Linking to relevant source code, issue / epic, or other documentation helps + to understand the answer. +4. **If you see something, do something.** + Please remove or update any content that is outdated as soon as you see it. + +## FAQ + +### How do I find the Rails route for a page? + +The easiest way is to type the following in the browser while on the page in +question: + +```javascript +document.body.dataset.page +``` + +Find here the [source code setting the attribute](https://gitlab.com/gitlab-org/gitlab-ce/blob/cc5095edfce2b4d4083a4fb1cdc7c0a1898b9921/app/views/layouts/application.html.haml#L4). + +### `modal_copy_button` vs `clipboard_button` + +The `clipboard_button` uses the `copy_to_clipboard.js` behaviour, which is +initialized on page load, so if there are vue-based clipboard buttons that +don't exist at page load (such as ones in a `GlModal`), they do not have the +click handlers associated with the clipboard package. + +`modal_copy_button` was added that manages an instance of the +[`clipboard` plugin](https://www.npmjs.com/package/clipboard) specific to +the instance of that component, which means that clipboard events are +bound on mounting and destroyed when the button is, mitigating the above +issue. It also has bindings to a particular container or modal ID +available, to work with the focus trap created by our GlModal. diff --git a/doc/development/fe_guide/graphql.md b/doc/development/fe_guide/graphql.md index 3290f29530a..8e06aa5d173 100644 --- a/doc/development/fe_guide/graphql.md +++ b/doc/development/fe_guide/graphql.md @@ -43,10 +43,12 @@ new Vue({ Read more about [Vue Apollo][vue-apollo] in the [Vue Apollo documentation][vue-apollo-docs]. -### Local state with `apollo-link-state` +### Local state with Apollo + +It is possible to manage an application state with Apollo by passing +in a resolvers object when creating the default client. The default state can be set by writing +to the cache after setting up the default client. -It is possible to use our Apollo setup with [apollo-link-state][apollo-link-state] by passing -in the client state object when creating the default client. ```javascript import Vue from 'vue'; @@ -54,18 +56,28 @@ import VueApollo from 'vue-apollo'; import createDefaultClient from '~/lib/graphql'; Vue.use(VueApollo); +const defaultClient = createDefaultClient({ + Query: { + ... + }, + Mutations: { + ... + }, +}); + +defaultClient.cache.writeData({ + data: { + isLoading: true, + }, +}); + const apolloProvider = new VueApollo({ - defaultClient: createDefaultClient({ - defaults: { - testing: true, - }, - resolvers: { - ... - }, - }), + defaultClient, }); ``` +Read more about local state management with Apollo in the [Vue Apollo documentation](https://vue-apollo.netlify.com/guide/local-state.html#local-state). + ### Testing With [Vue test utils][vue-test-utils] it is easy to quickly test components that @@ -82,6 +94,8 @@ it('tests apollo component', () => { }); ``` +Another possible way is testing queries with mocked GraphQL schema. Read more about this way in [Vue Apollo testing documentation](https://vue-apollo.netlify.com/guide/testing.html#tests-with-mocked-graqhql-schema) + ## Usage outside of Vue It is also possible to use GraphQL outside of Vue by directly importing diff --git a/doc/development/fe_guide/index.md b/doc/development/fe_guide/index.md index 86b8972a69e..36d5e4ab96b 100644 --- a/doc/development/fe_guide/index.md +++ b/doc/development/fe_guide/index.md @@ -1,8 +1,5 @@ # Frontend Development Guidelines -> **Notice:** -> We are currently in the process of re-writing our development guide to make it easier to find information. The new guide is still WIP but viewable in [development/new_fe_guide](../new_fe_guide/index.md) - This document describes various guidelines to ensure consistency and quality across GitLab's frontend team. @@ -12,15 +9,6 @@ GitLab is built on top of [Ruby on Rails][rails] using [Haml][haml] and also a J Be wary of [the limitations that come with using Hamlit][hamlit-limits]. We also use [SCSS][scss] and plain JavaScript with modern ECMAScript standards supported through [Babel][babel] and ES module support through [webpack][webpack]. -### Javascript development - -[Vue.js][vue] is used for particularly advanced, dynamic elements and based on previous iterations [jQuery][jquery] is used in lot of places through the application's JavaScript. - -We also use [Axios][axios] to handle all of our network requests. - -We also utilize [webpack][webpack] to handle the bundling, minification, and -compression of our assets. - Working with our frontend assets requires Node (v8.10.0 or greater) and Yarn (v1.10.0 or greater). You can find information on how to install these on our [installation guide][install]. @@ -31,6 +19,14 @@ For our currently-supported browsers, see our [requirements][requirements]. --- +## Initiatives + +Current high-level frontend goals are listed on [Frontend Epics](https://gitlab.com/groups/gitlab-org/-/epics?label_name%5B%5D=frontend). + +## [Principles](principles.md) + +High-level guidelines for contributing to GitLab. + ## [Development Process](development_process.md) How we plan and execute the work on the frontend. @@ -73,6 +69,14 @@ How we use SVG for our Icons and Illustrations. How we use UI components. +## [Event Tracking](event_tracking.md) + +How we use Snowplow to track custom events. + +## Frontend FAQ + +Read the [frontend's FAQ](frontend_faq.md) for common small pieces of helpful information. + --- ## Style Guides @@ -124,14 +128,3 @@ The [externalization part of the guide](../i18n/externalization.md) explains the [scss-lint]: https://github.com/brigade/scss-lint [install]: ../../install/installation.md#4-node [requirements]: ../../install/requirements.md#supported-web-browsers - ---- - -## [DropLab](droplab/droplab.md) - -Our internal `DropLab` dropdown library. - -- [DropLab](droplab/droplab.md) -- [Ajax plugin](droplab/plugins/ajax.md) -- [Filter plugin](droplab/plugins/filter.md) -- [InputSetter plugin](droplab/plugins/input_setter.md) diff --git a/doc/development/fe_guide/principles.md b/doc/development/fe_guide/principles.md new file mode 100644 index 00000000000..6fb3456222f --- /dev/null +++ b/doc/development/fe_guide/principles.md @@ -0,0 +1,15 @@ +# Principles + +These principles will ensure that your frontend contribution starts off in the right direction. + +## Discuss architecture before implementation + +Discuss your architecture design in an issue before writing code. This helps decrease the review time and also provides good practice for writing and thinking about system design. + +## Be consistent + +There are multiple ways of writing code to accomplish the same results. We should be as consistent as possible in how we write code across our codebases. This will make it more easier us to maintain our code across GitLab. + +## Improve code [iteratively](https://about.gitlab.com/handbook/values/#iteration) + +Whenever you see with existing code that does not follow our current style guide, update it proactively. You don't need to fix everything, but each merge request should iteratively improve our codebase, and reduce technical debt where possible. diff --git a/doc/development/fe_guide/style_guide_js.md b/doc/development/fe_guide/style_guide_js.md index 060cd8baf7f..b50159c2b75 100644 --- a/doc/development/fe_guide/style_guide_js.md +++ b/doc/development/fe_guide/style_guide_js.md @@ -95,6 +95,7 @@ See [our current .eslintrc](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/ #### Modules, Imports, and Exports 1. Use ES module syntax to import modules + ```javascript // bad const SomeClass = require('some_class'); @@ -168,6 +169,7 @@ See [our current .eslintrc](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/ Do not use them anymore and feel free to remove them when refactoring legacy code. 1. Avoid adding to the global namespace. + ```javascript // bad window.MyClass = class { /* ... */ }; @@ -176,7 +178,8 @@ See [our current .eslintrc](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/ export default class MyClass { /* ... */ } ``` -1. Side effects are forbidden in any script which contains exports +1. Side effects are forbidden in any script which contains export + ```javascript // bad export default class MyClass { /* ... */ } @@ -449,6 +452,7 @@ Please check this [rules][eslint-plugin-vue-rules] for more documentation. #### Props 1. Props should be declared as an object + ```javascript // bad props: ['foo'] @@ -544,14 +548,24 @@ Please check this [rules][eslint-plugin-vue-rules] for more documentation. <component @click="eventHandler"/> ``` -1. Shorthand `:` is preferable over `v-bind` +2. Shorthand `:` is preferable over `v-bind` ```javascript // bad <component v-bind:class="btn"/> // good - <component :class="btsn"/> + <component :class="btn"/> + ``` + +3. Shorthand `#` is preferable over `v-slot` + + ```javascript + // bad + <template v-slot:header></template> + + // good + <template #header></template> ``` #### Closing tags diff --git a/doc/development/fe_guide/style_guide_scss.md b/doc/development/fe_guide/style_guide_scss.md index b09243598d5..b25dce65ffe 100644 --- a/doc/development/fe_guide/style_guide_scss.md +++ b/doc/development/fe_guide/style_guide_scss.md @@ -5,6 +5,49 @@ easy to maintain, and performant for the end-user. ## Rules +### Utility Classes +As part of the effort for [cleaning up our CSS and moving our components into GitLab-UI](https://gitlab.com/groups/gitlab-org/-/epics/950) +led by the [GitLab UI WG](https://gitlab.com/gitlab-com/www-gitlab-com/merge_requests/20623) we prefer the use of utility classes over adding new CSS. However, complex CSS can be addressed by adding component classes. + +#### Where are utility classes defined? + +- [Bootstrap's Utility Classes](https://getbootstrap.com/docs/4.3/utilities/) +- [`common.scss`](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/app/assets/stylesheets/framework/common.scss) (old) +- [`utilities.scss`](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/app/assets/stylesheets/utilities.scss) (new) + +#### Where should I put new utility classes? + +New utility classes should be added to [`utilities.scss`](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/app/assets/stylesheets/utilities.scss). Existing classes include: + +| Name | Pattern | Example | +|------|---------|---------| +| Background color | `.bg-{variant}-{shade}` | `.bg-warning-400` | +| Text color | `.text-{variant}-{shade}` | `.text-success-500` | +| Font size | `.text-{size}` | `.text-2` | + +- `{variant}` is one of 'primary', 'secondary', 'success', 'warning', 'error' +- `{shade}` is on of the shades listed on [colors](https://design.gitlab.com/foundations/colors/) +- `{size}` is a number from 1-6 from our [Type scale](https://design.gitlab.com/foundations/typography) + +#### When should I create component classes? + +We recommend a "utility-first" approach. + +1. Start with utility classes. +2. If composing utility classes into a component class removes code duplication and encapsulates a clear responsibility, do it. + +This encourages an organic growth of component classes and prevents the creation of one-off unreusable classes. Also, the kind of classes that emerge from "utility-first" tend to be design-centered (e.g. `.button`, `.alert`, `.card`) rather than domain-centered (e.g. `.security-report-widget`, `.commit-header-icon`). + +Examples of component classes that were created using "utility-first" include: + +- [`.circle-icon-container`](https://gitlab.com/gitlab-org/gitlab-ce/blob/579fa8b8ec7eb38d40c96521f517c9dab8c3b97a/app/assets/stylesheets/framework/icons.scss#L85) +- [`.d-flex-center`](https://gitlab.com/gitlab-org/gitlab-ce/blob/900083d89cd6af391d26ab7922b3f64fa2839bef/app/assets/stylesheets/framework/common.scss#L425) + +Inspiration: + +- https://tailwindcss.com/docs/utility-first +- https://tailwindcss.com/docs/extracting-components + ### Naming Filenames should use `snake_case`. diff --git a/doc/development/fe_guide/testing.md b/doc/development/fe_guide/testing.md index 98e499b8c0f..b23e37d1eef 100644 --- a/doc/development/fe_guide/testing.md +++ b/doc/development/fe_guide/testing.md @@ -1 +1,5 @@ -This document was moved to [../testing_guide/frontend_testing.md](../testing_guide/frontend_testing.md). +--- +redirect_to: '../testing_guide/frontend_testing.md' +--- + +This document was moved to [another location](../testing_guide/frontend_testing.md). diff --git a/doc/development/fe_guide/vue.md b/doc/development/fe_guide/vue.md index 3cd70bd63fa..8c6a73c6824 100644 --- a/doc/development/fe_guide/vue.md +++ b/doc/development/fe_guide/vue.md @@ -38,7 +38,7 @@ _For consistency purposes, we recommend you to follow the same structure._ Let's look into each of them: -### A `index.js` file +### An `index.js` file This is the index file of your new feature. This is where the root Vue instance of the new feature should be. @@ -46,7 +46,7 @@ of the new feature should be. The Store and the Service should be imported and initialized in this file and provided as a prop to the main component. -Don't forget to follow [these steps][page_specific_javascript]. +Be sure to read about [page-specific JavaScript][page_specific_javascript]. ### Bootstrapping Gotchas #### Providing data from HAML to JavaScript @@ -120,9 +120,16 @@ You can read more about components in Vue.js site, [Component System][component- #### Vuex Check this [page](vuex.md) for more details. +### Mixing Vue and jQuery + +- Mixing Vue and jQuery is not recommended. +- If you need to use a specific jQuery plugin in Vue, [create a wrapper around it][https://vuejs.org/v2/examples/select2.html]. +- It is acceptable for Vue to listen to existing jQuery events using jQuery event listeners. +- It is not recommended to add new jQuery events for Vue to interact with jQuery. + ## Style guide -Please refer to the Vue section of our [style guide](style_guide_js.md#vue-js) +Please refer to the Vue section of our [style guide](style_guide_js.md#vuejs) for best practices while writing your Vue components and templates. ## Testing Vue Components @@ -132,7 +139,7 @@ Each Vue component has a unique output. This output is always present in the ren Although we can test each method of a Vue component individually, our goal must be to test the output of the render/template function, which represents the state at all times. -Make use of the [axios mock adapter](axios.md#mock-axios-response-on-tests) to mock data returned. +Make use of the [axios mock adapter](axios.md#mock-axios-response-in-tests) to mock data returned. Here's how we would test the Todo App above: @@ -227,13 +234,13 @@ One should apply to be a Vue.js expert by opening an MR when the Merge Request's - Deep understanding of Vue and Vuex reactivy - Vue and Vuex code are structured according to both official and our guidelines - Full understanding of testing a Vue and Vuex application -- Vuex code follows the [documented pattern](./vuex.md#actions-pattern-request-and-receive-namespaces) +- Vuex code follows the [documented pattern](vuex.md#actions-pattern-request-and-receive-namespaces) - Knowledge about the existing Vue and Vuex applications and existing reusable components [vue-docs]: http://vuejs.org/guide/index.html [issue-boards]: https://gitlab.com/gitlab-org/gitlab-ce/tree/master/app/assets/javascripts/boards [environments-table]: https://gitlab.com/gitlab-org/gitlab-ce/tree/master/app/assets/javascripts/environments -[page_specific_javascript]: https://docs.gitlab.com/ce/development/frontend.html#page-specific-javascript +[page_specific_javascript]: ./performance.md#page-specific-javascript [component-system]: https://vuejs.org/v2/guide/#Composing-with-Components [state-management]: https://vuejs.org/v2/guide/state-management.html#Simple-State-Management-from-Scratch [one-way-data-flow]: https://vuejs.org/v2/guide/components.html#One-Way-Data-Flow diff --git a/doc/development/fe_guide/vuex.md b/doc/development/fe_guide/vuex.md index 4b60ec80cb8..bf248b7f8af 100644 --- a/doc/development/fe_guide/vuex.md +++ b/doc/development/fe_guide/vuex.md @@ -52,7 +52,7 @@ The first thing you should do before writing any code is to design the state. Often we need to provide data from haml to our Vue application. Let's store it in the state for better access. ```javascript - export default { + export default () => ({ endpoint: null, isLoading: false, @@ -62,7 +62,7 @@ Often we need to provide data from haml to our Vue application. Let's store it i errorAddingUser: false, users: [], - }; + }); ``` #### Access `state` properties @@ -83,7 +83,7 @@ In this file, we will write the actions that will call the respective mutations: export const requestUsers = ({ commit }) => commit(types.REQUEST_USERS); export const receiveUsersSuccess = ({ commit }, data) => commit(types.RECEIVE_USERS_SUCCESS, data); - export const receiveUsersError = ({ commit }, error) => commit(types.REQUEST_USERS_ERROR, error); + export const receiveUsersError = ({ commit }, error) => commit(types.RECEIVE_USERS_ERROR, error); export const fetchUsers = ({ state, dispatch }) => { dispatch('requestUsers'); @@ -175,7 +175,7 @@ Remember that actions only describe that something happened, they don't describe state.users = data; state.isLoading = false; }, - [types.REQUEST_USERS_ERROR](state, error) { + [types.RECEIVE_USERS_ERROR](state, error) { state.isLoading = false; }, [types.REQUEST_ADD_USER](state, user) { @@ -186,7 +186,7 @@ Remember that actions only describe that something happened, they don't describe state.users.push(user); }, [types.REQUEST_ADD_USER_ERROR](state, error) { - state.isAddingUser = true; + state.isAddingUser = false; state.errorAddingUser = error; }, }; @@ -231,7 +231,7 @@ The store should be included in the main component of your application: ```javascript // app.vue - import store from 'store'; // it will include the index.js file + import store from './store'; // it will include the index.js file export default { name: 'application', diff --git a/doc/development/feature_flags.md b/doc/development/feature_flags.md index 67356a12eba..13f0c5cc33e 100644 --- a/doc/development/feature_flags.md +++ b/doc/development/feature_flags.md @@ -20,14 +20,15 @@ dynamic (querying the DB etc.). Once defined in `lib/feature.rb`, you will be able to activate a feature for a given feature group via the [`feature_group` param of the features API](../api/features.md#set-or-create-a-feature) -For GitLab.com, team members have access to feature flags through chatops. Only +For GitLab.com, [team members have access to feature flags through Chatops](chatops_on_gitlabcom.md). Only percentage gates are supported at this time. Setting a feature to be used 50% of the time, you should execute `/chatops run feature set my_feature_flag 50`. ## Feature flags for user applications -GitLab does not yet support the use of feature flags in deployed user applications. -You can follow the progress on that [in the issue on our issue tracker](https://gitlab.com/gitlab-org/gitlab-ee/issues/779). +This document only covers feature flags used in the development of GitLab +itself. Feature flags in deployed user applications can be found at +[Feature Flags](../user/project/operations/feature_flags.md) ## Developing with feature flags @@ -107,11 +108,11 @@ so we make sure behavior under feature flag doesn't go untested in some non-spec contexts. Whenever a feature flag is present, make sure to test _both_ states of the -feature flag. You can stub a feature flag as follows: +feature flag. -```ruby -stub_feature_flags(my_feature_flag: false) -``` +See the +[testing guide](testing_guide/best_practices.md#feature-flags-in-tests) +for information and examples on how to stub feature flags in tests. ## Enabling a feature flag (in development) diff --git a/doc/development/geo.md b/doc/development/geo.md new file mode 100644 index 00000000000..6e59fab34c7 --- /dev/null +++ b/doc/development/geo.md @@ -0,0 +1,529 @@ +# Geo (development) **[PREMIUM ONLY]** + +Geo connects GitLab instances together. One GitLab instance is +designated as a **primary** node and can be run with multiple +**secondary** nodes. Geo orchestrates quite a few components that can be seen on +the diagram below and are described in more detail within this document. + + + +## Replication layer + +Geo handles replication for different components: + +- [Database](#database-replication): includes the entire application, except cache and jobs. +- [Git repositories](#repository-replication): includes both projects and wikis. +- [Uploaded blobs](#uploads-replication): includes anything from images attached on issues +to raw logs and assets from CI. + +With the exception of the Database replication, on a *secondary* node, everything is coordinated +by the [Geo Log Cursor](#geo-log-cursor). + +### Geo Log Cursor daemon + +The [Geo Log Cursor daemon](#geo-log-cursor-daemon) is a separate process running on +each **secondary** node. It monitors the [Geo Event Log](#geo-event-log) +for new events and creates background jobs for each specific event type. + +For example when a repository is updated, the Geo **primary** node creates +a Geo event with an associated repository updated event. The Geo Log Cursor daemon +picks the event up and schedules a `Geo::ProjectSyncWorker` job which will +use the `Geo::RepositorySyncService` and `Geo::WikiSyncService` classes +to update the repository and the wiki respectively. + +The Geo Log Cursor daemon can operate in High Availability mode automatically. +The daemon will try to acquire a lock from time to time and once acquired, it +will behave as the *active* daemon. + +Any additional running daemons on the same node, will be in standby +mode, ready to resume work if the *active* daemon releases its lock. + +We use the [`ExclusiveLease`](https://www.rubydoc.info/github/gitlabhq/gitlabhq/Gitlab/ExclusiveLease) lock type with a small TTL, that is renewed at every +pooling cycle. That allows us to implement this global lock with a timeout. + +At the end of the pooling cycle, if the daemon can't renew and/or reacquire +the lock, it switches to standby mode. + +### Database replication + +Geo uses [streaming replication](#streaming-replication) to replicate +the database from the **primary** to the **secondary** nodes. This +replication gives the **secondary** nodes access to all the data saved +in the database. So users can log in on the **secondary** and read all +the issues, merge requests, etc. on the **secondary** node. + +### Repository replication + +Geo also replicates repositories. Each **secondary** node keeps track of +the state of every repository in the [tracking database](#tracking-database). + +There are a few ways a repository gets replicated by the: + +- [Repository Sync worker](#repository-sync-worker). +- [Geo Log Cursor](#geo-log-cursor). + +#### Project Registry + +The `Geo::ProjectRegistry` class defines the model used to track the +state of repository replication. For each project in the main +database, one record in the tracking database is kept. + +It records the following about repositories: + +- The last time they were synced. +- The last time they were successfully synced. +- If they need to be resynced. +- When a retry should be attempted. +- The number of retries. +- If and when they were verified. + +It also stores these attributes for project wikis in dedicated columns. + +#### Repository Sync worker + +The `Geo::RepositorySyncWorker` class runs periodically in the +background and it searches the `Geo::ProjectRegistry` model for +projects that need updating. Those projects can be: + +- Unsynced: Projects that have never been synced on the **secondary** + node and so do not exist yet. +- Updated recently: Projects that have a `last_repository_updated_at` + timestamp that is more recent than the `last_repository_successful_sync_at` + timestamp in the `Geo::ProjectRegistry` model. +- Manual: The admin can manually flag a repository to resync in the + [Geo admin panel](../user/admin_area/geo_nodes.md). + +When we fail to fetch a repository on the secondary `RETRIES_BEFORE_REDOWNLOAD` +times, Geo does a so-called _redownload_. It will do a clean clone +into the `@geo-temporary` directory in the root of the storage. When +it's successful, we replace the main repo with the newly cloned one. + +### Uploads replication + +File uploads are also being replicated to the **secondary** node. To +track the state of syncing, the `Geo::FileRegistry` model is used. + +#### File Registry + +Similar to the [Project Registry](#project-registry), there is a +`Geo::FileRegistry` model that tracks the synced uploads. + +CI Job Artifacts are synced in a similar way as uploads or LFS +objects, but they are tracked by `Geo::JobArtifactRegistry` model. + +#### File Download Dispatch worker + +Also similar to the [Repository Sync worker](#repository-sync-worker), +there is a `Geo::FileDownloadDispatchWorker` class that is run +periodically to sync all uploads that aren't synced to the Geo +**secondary** node yet. + +Files are copied via HTTP(s) and initiated via the +`/api/v4/geo/transfers/:type/:id` endpoint, +e.g. `/api/v4/geo/transfers/lfs/123`. + +## Authentication + +To authenticate file transfers, each `GeoNode` record has two fields: + +- A public access key (`access_key` field). +- A secret access key (`secret_access_key` field). + +The **secondary** node authenticates itself via a [JWT request](https://jwt.io/). +When the **secondary** node wishes to download a file, it sends an +HTTP request with the `Authorization` header: + +``` +Authorization: GL-Geo <access_key>:<JWT payload> +``` + +The **primary** node uses the `access_key` field to look up the +corresponding **secondary** node and decrypts the JWT payload, +which contains additional information to identify the file +request. This ensures that the **secondary** node downloads the right +file for the right database ID. For example, for an LFS object, the +request must also include the SHA256 sum of the file. An example JWT +payload looks like: + +``` +{ "data": { sha256: "31806bb23580caab78040f8c45d329f5016b0115" }, iat: "1234567890" } +``` + +If the requested file matches the requested SHA256 sum, then the Geo +**primary** node sends data via the [X-Sendfile](https://www.nginx.com/resources/wiki/start/topics/examples/xsendfile/) +feature, which allows NGINX to handle the file transfer without tying +up Rails or Workhorse. + +NOTE: **Note:** +JWT requires synchronized clocks between the machines +involved, otherwise it may fail with an encryption error. + +## Git Push to Geo secondary + +The Git Push Proxy exists as a functionality built inside the `gitlab-shell` component. +It is active on a **secondary** node only. It allows the user that has cloned a repository +from the secondary node to push to the same URL. + +Git `push` requests directed to a **secondary** node will be sent over to the **primary** node, +while `pull` requests will continue to be served by the **secondary** node for maximum efficiency. + +HTTPS and SSH requests are handled differently: + +- With HTTPS, we will give the user a `HTTP 302 Redirect` pointing to the project on the **primary** node. +The git client is wise enough to understand that status code and process the redirection. +- With SSH, because there is no equivalent way to perform a redirect, we have to proxy the request. +This is done inside [`gitlab-shell`](https://gitlab.com/gitlab-org/gitlab-shell), by first translating the request +to the HTTP protocol, and then proxying it to the **primary** node. + +The [`gitlab-shell`](https://gitlab.com/gitlab-org/gitlab-shell) daemon knows when to proxy based on the response +from `/api/v4/allowed`. A special `HTTP 300` status code is returned and we execute a "custom action", +specified in the response body. The response contains additional data that allows the proxied `push` operation +to happen on the **primary** node. + +## Using the Tracking Database + +Along with the main database that is replicated, a Geo **secondary** +node has its own separate [Tracking database](#tracking-database). + +The tracking database contains the state of the **secondary** node. + +Any database migration that needs to be run as part of an upgrade +needs to be applied to the tracking database on each **secondary** node. + +### Configuration + +The database configuration is set in [`config/database_geo.yml`](https://gitlab.com/gitlab-org/gitlab-ee/blob/master/config/database_geo.yml.postgresql). +The directory [`ee/db/geo`](https://gitlab.com/gitlab-org/gitlab-ee/tree/master/ee/db/geo) +contains the schema and migrations for this database. + +To write a migration for the database, use the `GeoMigrationGenerator`: + +``` +rails g geo_migration [args] [options] +``` + +To migrate the tracking database, run: + +``` +bundle exec rake geo:db:migrate +``` + +### Foreign Data Wrapper + +> Introduced in GitLab 10.1. + +Foreign Data Wrapper ([FDW](#fdw)) is used by the [Geo Log Cursor](#geo-log-cursor) and improves +the performance of many synchronization operations. + +FDW is a PostgreSQL extension ([`postgres_fdw`](https://www.postgresql.org/docs/current/postgres-fdw.html)) that is enabled within +the Geo Tracking Database (on a **secondary** node), which allows it +to connect to the readonly database replica and perform queries and filter +data from both instances. + +While FDW is available in older versions of PostgreSQL, we needed to +raise the minimum required version to 9.6 as this includes many +performance improvements to the FDW implementation. + +This persistent connection is configured as an FDW server +named `gitlab_secondary`. This configuration exists within the database's user +context only. To access the `gitlab_secondary`, GitLab needs to use the +same database user that had previously been configured. + +The Geo Tracking Database accesses the readonly database replica via FDW as a regular user, +limited by its own restrictions. The credentials are configured as a +`USER MAPPING` associated with the `SERVER` mapped previously +(`gitlab_secondary`). + +FDW configuration and credentials definition are managed automatically by the +Omnibus GitLab `gitlab-ctl reconfigure` command. + +#### Refeshing the Foreign Tables + +Whenever a new Geo node is configured or the database schema changes on the +**primary** node, you must refresh the foreign tables on the **secondary** node +by running the following: + +```sh +bundle exec rake geo:db:refresh_foreign_tables +``` + +Failure to do this will prevent the **secondary** node from +functioning properly. The **secondary** node will generate error +messages, as the following PostgreSQL error: + +``` +ERROR: relation "gitlab_secondary.ci_job_artifacts" does not exist at character 323 +STATEMENT: SELECT a.attname, format_type(a.atttypid, a.atttypmod), + pg_get_expr(d.adbin, d.adrelid), a.attnotnull, a.atttypid, a.atttypmod + FROM pg_attribute a LEFT JOIN pg_attrdef d + ON a.attrelid = d.adrelid AND a.attnum = d.adnum + WHERE a.attrelid = '"gitlab_secondary"."ci_job_artifacts"'::regclass + AND a.attnum > 0 AND NOT a.attisdropped + ORDER BY a.attnum +``` + +#### Accessing data from a Foreign Table + +At the SQL level, all you have to do is `SELECT` data from `gitlab_secondary.*`. + +Here's an example of how to access all projects from the Geo Tracking Database's FDW: + +```sql +SELECT * FROM gitlab_secondary.projects; +``` + +As a more real-world example, this is how you filter for unarchived projects +on the Tracking Database: + +```sql +SELECT project_registry.* + FROM project_registry + JOIN gitlab_secondary.projects + ON (project_registry.project_id = gitlab_secondary.projects.id + AND gitlab_secondary.projects.archived IS FALSE) +``` + +At the ActiveRecord level, we have additional Models that represent the +foreign tables. They must be mapped in a slightly different way, and they are read-only. + +Check the existing FDW models in `ee/app/models/geo/fdw` for reference. + +From a developer's perspective, it's no different than creating a model that +represents a Database View. + +With the examples above, you can access the projects with: + +```ruby +Geo::Fdw::Project.all +``` + +and to access the `ProjectRegistry` filtering by unarchived projects: + +```ruby +# We have to use Arel here: +project_registry_table = Geo::ProjectRegistry.arel_table +fdw_project_table = Geo::Fdw::Project.arel_table + +project_registry_table.join(fdw_project_table) + .on(project_registry_table[:project_id].eq(fdw_project_table[:id])) + .where((fdw_project_table[:archived]).eq(true)) # if you append `.to_sql` you can check generated query +``` + +## Finders + +Geo uses [Finders](https://gitlab.com/gitlab-org/gitlab-ee/tree/master/app/finders), +which are classes take care of the heavy lifting of looking up +projects/attachments/etc. in the tracking database and main database. + +### Finders Performance + +The Finders need to compare data from the main database with data in +the tracking database. For example, counting the number of synced +projects normally involves retrieving the project IDs from one +database and checking their state in the other database. This is slow +and requires a lot of memory. + +To overcome this, the Finders use [FDW](#fdw), or Foreign Data +Wrappers. This allows a regular `JOIN` between the main database and +the tracking database. + +## Redis + +Redis on the **secondary** node works the same as on the **primary** +node. It is used for caching, storing sessions, and other persistent +data. + +Redis data replication between **primary** and **secondary** node is +not used, so sessions etc. aren't shared between nodes. + +## Object Storage + +GitLab can optionally use Object Storage to store data it would +otherwise store on disk. These things can be: + + - LFS Objects + - CI Job Artifacts + - Uploads + +Objects that are stored in object storage, are not handled by Geo. Geo +ignores items in object storage. Either: + +- The object storage layer should take care of its own geographical + replication. +- All secondary nodes should use the same storage node. + +## Verification + +### Repository verification + +Repositories are verified with a checksum. + +The **primary** node calculates a checksum on the repository. It +basically hashes all Git refs together and stores that hash in the +`project_repository_states` table of the database. + +The **secondary** node does the same to calculate the hash of its +clone, and compares the hash with the value the **primary** node +calculated. If there is a mismatch, Geo will mark this as a mismatch +and the administrator can see this in the [Geo admin panel](../user/admin_area/geo_nodes.md). + +## Glossary + +### Primary node + +A **primary** node is the single node in a Geo setup that read-write +capabilities. It's the single source of truth and the Geo +**secondary** nodes replicate their data from there. + +In a Geo setup, there can only be one **primary** node. All +**secondary** nodes connect to that **primary**. + +### Secondary node + +A **secondary** node is a read-only replica of the **primary** node +running in a different geographical location. + +### Streaming replication + +Geo depends on the streaming replication feature of PostgreSQL. It +completely replicates the database data and the database schema. The +database replica is a read-only copy. + +Streaming replication depends on the Write Ahead Logs, or WAL. Those +logs are copied over to the replica and replayed there. + +Since streaming replication also replicates the schema, the database +migration do not need to run on the secondary nodes. + +### Tracking database + +A database on each Geo **secondary** node that keeps state for the node +on which it resides. Read more in [Using the Tracking database](#using-the-tracking-database). + +### FDW + +Foreign Data Wrapper, or FDW, is a feature built-in in PostgreSQL. It +allows data to be queried from different data sources. In Geo, it's +used to query data from different PostgreSQL instances. + +## Geo Event Log + +The Geo **primary** stores events in the `geo_event_log` table. Each +entry in the log contains a specific type of event. These type of +events include: + + - Repository Deleted event + - Repository Renamed event + - Repositories Changed event + - Repository Created event + - Hashed Storage Migrated event + - Lfs Object Deleted event + - Hashed Storage Attachments event + - Job Artifact Deleted event + - Upload Deleted event + +### Geo Log Cursor + +The process running on the **secondary** node that looks for new +`Geo::EventLog` rows. + +## Code features + +### `Gitlab::Geo` utilities + +Small utility methods related to Geo go into the +[`ee/lib/gitlab/geo.rb`](https://gitlab.com/gitlab-org/gitlab-ee/blob/master/ee/lib/gitlab/geo.rb) +file. + +Many of these methods are cached using the `RequestStore` class, to +reduce the performance impact of using the methods throughout the +codebase. + +#### Current node + +The class method `.current_node` returns the `GeoNode` record for the +current node. + +We use the `host`, `port`, and `relative_url_root` values from +`gitlab.yml` and search in the database to identify which node we are +in (see `GeoNode.current_node`). + +#### Primary or secondary + +To determine whether the current node is a **primary** node or a +**secondary** node use the `.primary?` and `.secondary?` class +methods. + +It is possible for these methods to both return `false` on a node when +the node is not enabled. See [Enablement](#enablement). + +#### Geo Database configured? + +There is also an additional gotcha when dealing with things that +happen during initialization time. In a few places, we use the +`Gitlab::Geo.geo_database_configured?` method to check if the node has +the tracking database, which only exists on the **secondary** +node. This overcomes race conditions that could happen during +bootstrapping of a new node. + +#### Enablement + +We consider Geo feature enabled when the user has a valid license with the +feature included, and they have at least one node defined at the Geo Nodes +screen. + +See `Gitlab::Geo.enabled?` and `Gitlab::Geo.license_allows?` methods. + +#### Read-only + +All Geo **secondary** nodes are read-only. + +The general principle of a [read-only database](verifying_database_capabilities.md#read-only-database) +applies to all Geo **secondary** nodes. So the +`Gitlab::Database.read_only?` method will always return `true` on a +**secondary** node. + +When some write actions are not allowed because the node is a +**secondary**, consider adding the `Gitlab::Database.read_only?` or +`Gitlab::Database.read_write?` guard, instead of `Gitlab::Geo.secondary?`. + +The database itself will already be read-only in a replicated setup, +so we don't need to take any extra step for that. + +## History of communication channel + +The communication channel has changed since first iteration, you can +check here historic decisions and why we moved to new implementations. + +### Custom code (GitLab 8.6 and earlier) + +In GitLab versions before 8.6, custom code is used to handle +notification from **primary** node to **secondary** nodes by HTTP +requests. + +### System hooks (GitLab 8.7 to 9.5) + +Later, it was decided to move away from custom code and begin using +system hooks. More people were using them, so +many would benefit from improvements made to this communication layer. + +There is a specific **internal** endpoint in our API code (Grape), +that receives all requests from this System Hooks: +`/api/v4/geo/receive_events`. + +We switch and filter from each event by the `event_name` field. + +### Geo Log Cursor (GitLab 10.0 and up) + +Since GitLab 10.0, [System Webhooks](#system-hooks-gitlab-87-to-95) are no longer +used and Geo Log Cursor is used instead. The Log Cursor traverses the +`Geo::EventLog` rows to see if there are changes since the last time +the log was checked and will handle repository updates, deletes, +changes, and renames. + +The table is within the replicated database. This has two advantages over the +old method: + +- Replication is synchronous and we preserve the order of events. +- Replication of the events happen at the same time as the changes in the + database. diff --git a/doc/development/git_object_deduplication.md b/doc/development/git_object_deduplication.md new file mode 100644 index 00000000000..81c5f69c7b8 --- /dev/null +++ b/doc/development/git_object_deduplication.md @@ -0,0 +1,261 @@ +# How Git object deduplication works in GitLab + +When a GitLab user [forks a project](../workflow/forking_workflow.md), +GitLab creates a new Project with an associated Git repository that is a +copy of the original project at the time of the fork. If a large project +gets forked often, this can lead to a quick increase in Git repository +storage disk use. To counteract this problem, we are adding Git object +deduplication for forks to GitLab. In this document, we will describe how +GitLab implements Git object deduplication. + +## Enabling Git object deduplication via feature flags + +As of GitLab 11.9, Git object deduplication in GitLab is in beta. In this +document, you can read about the caveats of enabling the feature. Also, +note that Git object deduplication is limited to forks of public +projects on hashed repository storage. + +You can enable deduplication globally by setting the `object_pools` +feature flag to `true`: + +``` {.ruby} +Feature.enable(:object_pools) +``` + +Or just for forks of a specific project: + +``` {.ruby} +fork_parent = Project.find(MY_PROJECT_ID) +Feature.enable(:object_pools, fork_parent) +``` + +To check if a project uses Git object deduplication, look in a Rails +console if `project.pool_repository` is present. + +## Pool repositories + +### Understanding Git alternates + +At the Git level, we achieve deduplication by using [Git +alternates](https://git-scm.com/docs/gitrepository-layout#gitrepository-layout-objects). +Git alternates is a mechanism that lets a repository borrow objects from +another repository on the same machine. + +If we want repository A to borrow from repository B, we first write a +path that resolves to `B.git/objects` in the special file +`A.git/objects/info/alternates`. This establishes the alternates link. +Next, we must perform a Git repack in A. After the repack, any objects +that are duplicated between A and B will get deleted from A. Repository +A is now no longer self-contained, but it still has its own refs and +configuration. Objects in A that are not in B will remain in A. For this +to work, it is of course critical that **no objects ever get deleted from +B** because A might need them. + +### Git alternates in GitLab: pool repositories + +GitLab organizes this object borrowing by creating special **pool +repositories** which are hidden from the user. We then use Git +alternates to let a collection of project repositories borrow from a +single pool repository. We call such a collection of project +repositories a pool. Pools form star-shaped networks of repositories +that borrow from a single pool, which will resemble (but not be +identical to) the fork networks that get formed when users fork +projects. + +At the Git level, pool repositories are created and managed using Gitaly +RPC calls. Just like with normal repositories, the authority on which +pool repositories exist, and which repositories borrow from them, lies +at the Rails application level in SQL. + +In conclusion, we need three things for effective object deduplication +across a collection of GitLab project repositories at the Git level: + +1. A pool repository must exist. +2. The participating project repositories must be linked to the pool + repository via their respective `objects/info/alternates` files. +3. The pool repository must contain Git object data common to the + participating project repositories. + +### Deduplication factor + +The effectiveness of Git object deduplication in GitLab depends on the +amount of overlap between the pool repository and each of its +participants. As of GitLab 11.9, we have a somewhat optimistic system. +The only data that will be deduplicated is the data in the source +project repository at the time the pool repository is created. That is, +the data in the source project at the time of the first fork *after* the +deduplication feature has been enabled. + +When we enable the object deduplication feature for +gitlab.com/gitlab-org/gitlab-ce, which is about 1GB at the time of +writing, all new forks of that project would be 1GB smaller than they +would have been without Git object deduplication. So even in its current +optimistic form, we expect Git object deduplication in GitLab to make a +difference. + +However, if a lot of Git objects get added to the project repositories +in a pool after the pool repository was created these new Git objects +will currently (GitLab 11.9) not get deduplicated. Over time, the +deduplication factor of the pool will get worse and worse. + +As an extreme example, if we create an empty repository A, and fork that +to repository B, behind the scenes we get an object pool P with no +objects in it at all. If we then push 1GB of Git data to A, and push the +same Git data to B, it will not get deduplicated, because that data was +not in A at the time P was created. + +This also matters in less extreme examples. Consider a pool P with +source project A and 500 active forks B1, B2,...,B500. Suppose, +optimistically, that the forks are fully deduplicated at the start of +our scenario. Now some time passes and 200MB of new Git data gets added +to project A. Because of the forking workflow, this data makes also its way +into the forks B1, ..., B500. That means we would now have 100GB of Git +data sitting around (500 \* 200MB) across the forks, that could have +been deduplicated. But because of the way we do deduplication this new +data will not be deduplicated. + +> TODO Add periodic maintenance of object pools to prevent gradual loss +> of deduplication over time. +> https://gitlab.com/groups/gitlab-org/-/epics/524 + +## SQL model + +As of GitLab 11.8, project repositories in GitLab do not have their own +SQL table. They are indirectly identified by columns on the `projects` +table. In other words, the only way to look up a project repository is to +first look up its project, and then call `project.repository`. + +With pool repositories we made a fresh start. These live in their own +`pool_repositories` SQL table. The relations between these two tables +are as follows: + +- a `Project` belongs to at most one `PoolRepository` + (`project.pool_repository`) +- as an automatic consequence of the above, a `PoolRepository` has + many `Project`s +- a `PoolRepository` has exactly one "source `Project`" + (`pool.source_project`) + +### Assumptions + +- All repositories in a pool must use [hashed + storage](../administration/repository_storage_types.md). This is so + that we don't have to ever worry about updating paths in + `object/info/alternates` files. +- All repositories in a pool must be on the same Gitaly storage shard. + The Git alternates mechanism relies on direct disk access across + multiple repositories, and we can only assume direct disk access to + be possible within a Gitaly storage shard. +- All project repositories in a pool must have "Public" visibility in + GitLab at the time they join. There are gotchas around visibility of + Git objects across alternates links. This restriction is a defense + against accidentally leaking private Git data. +- The only two ways to remove a member project from a pool are (1) to + delete the project or (2) to move the project to another Gitaly + storage shard. + +### Creating pools and pool memberships + +- When a pool gets created, it must have a source project. The initial + contents of the pool repository are a Git clone of the source + project repository. +- The occasion for creating a pool is when an existing eligible + (public, hashed storage, non-forked) GitLab project gets forked and + this project does not belong to a pool repository yet. The fork + parent project becomes the source project of the new pool, and both + the fork parent and the fork child project become members of the new + pool. +- Once project A has become the source project of a pool, all future + eligible forks of A will become pool members. +- If the fork source is itself a fork, the resulting repository will + neither join the repository nor will a new pool repository be + seeded. + + eg: + + Suppose fork A is part of a pool repository, any forks created off + of fork A *will not* be a part of the pool repository that fork A is + a part of. + + Suppose B is a fork of A, and A does not belong to an object pool. + Now C gets created as a fork of B. C will not be part of a pool + repository. + +> TODO should forks of forks be deduplicated? +> https://gitlab.com/gitlab-org/gitaly/issues/1532 + +### Consequences + +- If a normal Project participating in a pool gets moved to another + Gitaly storage shard, its "belongs to PoolRepository" relation must + be broken. Because of the way moving repositories between shard is + implemented, we will automatically get a fresh self-contained copy + of the project's repository on the new storage shard. +- If the source project of a pool gets moved to another Gitaly storage + shard or is deleted, we may have to break the "PoolRepository has + one source Project" relation? + +> TODO What happens, or should happen, if a source project changes +> visibility, is deleted, or moves to another storage shard? +> https://gitlab.com/gitlab-org/gitaly/issues/1488 + +## Consistency between the SQL pool relation and Gitaly + +As far as Gitaly is concerned, the SQL pool relations make two types of +claims about the state of affairs on the Gitaly server: pool repository +existence, and the existence of an alternates connection between a +repository and a pool. + +### Pool existence + +If GitLab thinks a pool repository exists (i.e. it exists according to +SQL), but it does not on the Gitaly server, then certain RPC calls that +take the object pool as an argument will fail. + +> TODO What happens if SQL says the pool repo exists but Gitaly says it +> does not? https://gitlab.com/gitlab-org/gitaly/issues/1533 + +If GitLab thinks a pool does not exist, while it does exist on disk, +that has no direct consequences on its own. However, if other +repositories on disk borrow objects from this unknown pool repository +then we risk data loss, see below. + +### Pool relation existence + +There are three different things that can go wrong here. + +#### 1. SQL says repo A belongs to pool P but Gitaly says A has no alternate objects + +In this case, we miss out on disk space savings but all RPC's on A itself +will function fine. As long as Git can find all its objects, it does not +matter exactly where those objects are. + +#### 2. SQL says repo A belongs to pool P1 but Gitaly says A has alternate objects in pool P2 + +If we are not careful, this situation can lead to data loss. During some +operations (repository maintenance), GitLab will try to re-link A to its +pool P1. If this clobbers the existing link to P2, then A will loose Git +objects and become invalid. + +Also, keep in mind that if GitLab's database got messed up, it may not +even know that P2 exists. + +> TODO Ensure that Gitaly will not clobber existing, unexpected +> alternates links. https://gitlab.com/gitlab-org/gitaly/issues/1534 + +#### 3. SQL says repo A does not belong to any pool but Gitaly says A belongs to P + +This has the same data loss possibility as scenario 2 above. + +## Git object deduplication and GitLab Geo + +When a pool repository record is created in SQL on a Geo primary, this +will eventually trigger an event on the Geo secondary. The Geo secondary +will then create the pool repository in Gitaly. This leads to an +"eventually consistent" situation because as each pool participant gets +synchronized, Geo will eventuall trigger garbage collection in Gitaly on +the secondary, at which stage Git objects will get deduplicated. + +> TODO How do we handle the edge case where at the time the Geo +> secondary tries to create the pool repository, the source project does +> not exist? https://gitlab.com/gitlab-org/gitaly/issues/1533 diff --git a/doc/development/gitaly.md b/doc/development/gitaly.md index d5fc403bf8b..4ef20d5fd74 100644 --- a/doc/development/gitaly.md +++ b/doc/development/gitaly.md @@ -3,10 +3,17 @@ [Gitaly](https://gitlab.com/gitlab-org/gitaly) is a high-level Git RPC service used by GitLab CE/EE, Workhorse and GitLab-Shell. +## Beginner's guide + +Start by reading the gitaly repository's +[Beginner's guide to Gitaly contributions](https://gitlab.com/gitlab-org/gitaly/blob/master/doc/beginners_guide.md). +It describes how to setup gitaly, the various components of gitaly and what they do, and how to run its test suites. + ## Developing new Git features -Starting with GitLab 10.8, all new Git features should be developed in -Gitaly. +To read or write Git data, a request has to be made to Gitaly. This means that +if you're developing a new feature where you need data that's not yet available +in `lib/gitlab/git` changes have to be made to Gitaly. > This is a new process that is not clearly defined yet. If you want to contribute a Git feature and you're getting stuck, reach out to the @@ -56,6 +63,48 @@ If your test-suite is failing with Gitaly issues, as a first step, try running: rm -rf tmp/tests/gitaly ``` +During rspec tests, the Gitaly instance will write logs to `gitlab/log/gitaly-test.log`. + +## Legacy Rugged code + +While Gitaly can handle all Git access, many of GitLab customers still +run Gitaly atop NFS. The legacy Rugged implementation for Git calls may +be faster than the Gitaly RPC due to N+1 Gitaly calls and other +reasons. See [the +issue](https://gitlab.com/gitlab-org/gitlab-ce/issues/57317) for more +details. + +Until GitLab has eliminated most of these inefficiencies or the use of +NFS is discontinued for Git data, Rugged implementations of some of the +most commonly-used RPCs can be enabled via feature flags: + +* `rugged_find_commit` +* `rugged_get_tree_entries` +* `rugged_tree_entry` +* `rugged_commit_is_ancestor` +* `rugged_commit_tree_entry` +* `rugged_list_commits_by_oid` + +A convenience Rake task can be used to enable or disable these flags +all together. To enable: + +```sh +bundle exec rake gitlab:features:enable_rugged +``` + +To disable: + +```sh +bundle exec rake gitlab:features:disable_rugged +``` + +Most of this code exists in the `lib/gitlab/git/rugged_impl` directory. + +NOTE: **Note:** You should NOT need to add or modify code related to +Rugged unless explicitly discussed with the [Gitaly +Team](https://gitlab.com/groups/gl-gitaly/group_members). This code will +NOT work on GitLab.com or other GitLab instances that do not use NFS. + ## `TooManyInvocationsError` errors During development and testing, you may experience `Gitlab::GitalyClient::TooManyInvocationsError` failures. @@ -147,8 +196,87 @@ GITALY_REPO_URL=https://gitlab+deploy-token-1000:token-here@gitlab.com/nick.thom To use a custom Gitaly repository in CI, for instance if you want your GitLab fork to always use your own Gitaly fork, set `GITALY_REPO_URL` -as a [CI environment variable](../ci/variables/README.md#variables). +as a [CI environment variable](../ci/variables/README.md#gitlab-cicd-environment-variables). --- [Return to Development documentation](README.md) + +## Wrapping RPCs in Feature Flags + +Here are the steps to gate a new feature in Gitaly behind a feature flag. + +### Gitaly + +1. Create a package scoped flag name: + + ```go + var findAllTagsFeatureFlag = "go-find-all-tags" + ``` + +1. Create a switch in the code using the `featureflag` package: + + ```go + if featureflag.IsEnabled(ctx, findAllTagsFeatureFlag) { + // go implementation + } else { + // ruby implementation + } + ``` + +1. Create prometheus metrics: + + ```go + var findAllTagsRequests = prometheus.NewCounterVec( + prometheus.CounterOpts{ + Name: "gitaly_find_all_tags_requests_total", + Help: "Counter of go vs ruby implementation of FindAllTags", + }, + []string{"implementation"}, + ) + ) + + func init() { + prometheus.Register(findAllTagsRequests) + } + + if featureflag.IsEnabled(ctx, findAllTagsFeatureFlag) { + findAllTagsRequests.WithLabelValues("go").Inc() + // go implementation + } else { + findAllTagsRequests.WithLabelValues("ruby").Inc() + // ruby impelmentation + } + ``` + +1. Set headers in tests: + + ```go + import ( + "google.golang.org/grpc/metadata" + + "gitlab.com/gitlab-org/gitaly/internal/featureflag" + ) + + //... + + md := metadata.New(map[string]string{featureflag.HeaderKey(findAllTagsFeatureFlag): "true"}) + ctx = metadata.NewOutgoingContext(context.Background(), md) + + c, err = client.FindAllTags(ctx, rpcRequest) + require.NoError(t, err) + ``` + +### Gitlab-Rails + +1. Add feature flag to `lib/gitlab/gitaly_client.rb` (in gitlab-rails): + + ```ruby + SERVER_FEATURE_FLAGS = %w[go-find-all-tags].freeze + ``` + +1. Test in rails console by setting feature flag: + + ```ruby + Feature.enable('gitaly_go-find-all-tags') + ``` diff --git a/doc/development/gitlab_architecture_diagram.png b/doc/development/gitlab_architecture_diagram.png Binary files differdeleted file mode 100644 index 90e27d5462a..00000000000 --- a/doc/development/gitlab_architecture_diagram.png +++ /dev/null diff --git a/doc/development/go_guide/index.md b/doc/development/go_guide/index.md index cdc806a2d31..4dad8815fcb 100644 --- a/doc/development/go_guide/index.md +++ b/doc/development/go_guide/index.md @@ -26,7 +26,7 @@ Reviewers and maintainers should pay attention to: - `defer` functions: ensure the presence when needed, and after `err` check. - Inject dependencies as parameters. -- Void structs when marshalling to JSON (generates `null` instead of `[]`). +- Void structs when marshaling to JSON (generates `null` instead of `[]`). ### Security @@ -40,7 +40,7 @@ of possible security breaches in our code: - SQL injections Remember to run -[SAST](https://docs.gitlab.com/ee/user/project/merge_requests/sast.html) +[SAST](../../user/application_security/sast/index.md) **[ULTIMATE]** on your project (or at least the [gosec analyzer](https://gitlab.com/gitlab-org/security-products/analyzers/gosec)), and to follow our [Security @@ -93,10 +93,10 @@ become available, you will be able to share job templates like this Dependencies should be kept to the minimum. The introduction of a new dependency should be argued in the merge request, as per our [Approval -Guidelines](../code_review.html#approval-guidelines). Both [License -Management](https://docs.gitlab.com/ee/user/project/merge_requests/license_management.html) +Guidelines](../code_review.md#approval-guidelines). Both [License +Management](../../user/project/merge_requests/license_management.md) **[ULTIMATE]** and [Dependency -Scanning](https://docs.gitlab.com/ee/user/project/merge_requests/dependency_scanning.html) +Scanning](../../user/application_security/dependency_scanning/index.md) **[ULTIMATE]** should be activated on all projects to ensure new dependencies security status and license compatibility. @@ -161,12 +161,39 @@ in the code. ### Logging -The usage of a logging library is strongly recommended for daemons. Even though -there is a `log` package in the standard library, we generally use -[logrus](https://github.com/sirupsen/logrus). Its plugin ("hooks") system +The usage of a logging library is strongly recommended for daemons. Even +though there is a `log` package in the standard library, we generally use +[Logrus](https://github.com/sirupsen/logrus). Its plugin ("hooks") system makes it a powerful logging library, with the ability to add notifiers and formatters at the logger level directly. +#### Structured (JSON) logging + +Every binary ideally must have structured (JSON) logging in place as it helps +with searching and filtering the logs. At GitLab we use structured logging in +JSON format, as all our infrastructure assumes that. When using +[Logrus](https://github.com/sirupsen/logrus) you can turn on structured +logging simply by using the build in [JSON +formatter](https://github.com/sirupsen/logrus#formatters). This follows the +same logging type we use in our [Ruby +applications](../logging.md#use-structured-json-logging). + +#### How to use Logrus + +There are a few guidelines one should follow when using the +[Logrus](https://github.com/sirupsen/logrus) package: + +- When printing an error use + [WithError](https://godoc.org/github.com/sirupsen/logrus#WithError). For + example, `logrus.WithError(err).Error("Failed to do something")`. +- Since we use [structured logging](#structured-json-logging) we can log + fields in the context of that code path, such as the URI of the request using + [`WithField`](https://godoc.org/github.com/sirupsen/logrus#WithField) or + [`WithFields`](https://godoc.org/github.com/sirupsen/logrus#WithFields). For + example, `logrus.WithField("file", "/app/go).Info("Opening dir")`. If you + have to log multiple keys, always use `WithFields` instead of calling + `WithField` more than once. + ### Tracing and Correlation [LabKit](https://gitlab.com/gitlab-org/labkit) is a place to keep common diff --git a/doc/development/i18n/externalization.md b/doc/development/i18n/externalization.md index 223585ebb55..9fb8ea542d9 100644 --- a/doc/development/i18n/externalization.md +++ b/doc/development/i18n/externalization.md @@ -174,7 +174,7 @@ For example use `%{created_at}` in Ruby but `%{createdAt}` in JavaScript. # => When size == 2: 'There are 2 mice.' ``` - Avoid using `%d` or count variables in sigular strings. This allows more natural translation in some languages. + Avoid using `%d` or count variables in singular strings. This allows more natural translation in some languages. - In JavaScript: @@ -195,6 +195,7 @@ For example use `%{created_at}` in Ruby but `%{createdAt}` in JavaScript. Sometimes you need to add some context to the text that you want to translate (if the word occurs in a sentence and/or the word is ambiguous). +Namespaces should be PascalCase. - In Ruby/HAML: @@ -211,7 +212,7 @@ Sometimes you need to add some context to the text that you want to translate ``` Note: The namespace should be removed from the translation. See the [translation -guidelines for more details](./translation.md#namespaced-strings). +guidelines for more details](translation.md#namespaced-strings). ### Dates / times @@ -300,7 +301,7 @@ file in. Once the changes are on master, they will be picked up by [Crowdin](http://translate.gitlab.com) and be presented for translation. If there are merge conflicts in the `gitlab.pot` file, you can delete the file -and regenerate it using the same command. Confirm that you are not deleting any strings accidentally by looking over the diff. +and regenerate it using the same command. ### Validating PO files @@ -331,7 +332,7 @@ Errors in `locale/zh_HK/gitlab.po`: Syntax error in msgstr Syntax error in message_line There should be only whitespace until the end of line after the double quote character of a message text. - Parseing result before error: '{:msgid=>["", "You are going to remove %{project_name_with_namespace}.\\n", "Removed project CANNOT be restored!\\n", "Are you ABSOLUTELY sure?"]}' + Parsing result before error: '{:msgid=>["", "You are going to remove %{project_name_with_namespace}.\\n", "Removed project CANNOT be restored!\\n", "Are you ABSOLUTELY sure?"]}' SimplePoParser filtered backtrace: SimplePoParser::ParserError Errors in `locale/zh_TW/gitlab.po`: 1 pipeline diff --git a/doc/development/i18n/merging_translations.md b/doc/development/i18n/merging_translations.md index 2fa7558d30b..eadf1cd74c5 100644 --- a/doc/development/i18n/merging_translations.md +++ b/doc/development/i18n/merging_translations.md @@ -32,8 +32,7 @@ clicking `Pause sync` on the [Crowdin integration settings page](https://translate.gitlab.com/project/gitlab-ee/settings#integration). When all failures are resolved, the translations need to be double -checked once more [as discussed in this -issue](https://gitlab.com/gitlab-org/gitlab-ce/issues/37850). +checked once more as discussed in [confidential issue](../../user/project/issues/confidential_issues.md) `https://gitlab.com/gitlab-org/gitlab-ce/issues/37850`. ## Merging translations diff --git a/doc/development/i18n/proofreader.md b/doc/development/i18n/proofreader.md index 6054873cb46..35c5b155594 100644 --- a/doc/development/i18n/proofreader.md +++ b/doc/development/i18n/proofreader.md @@ -13,18 +13,21 @@ are very appreciative of the work done by translators and proofreaders! - Lyubomir Vasilev - [Crowdin](https://crowdin.com/profile/lyubomirv) - Catalan - David Planella - [GitLab](https://gitlab.com/dplanella), [Crowdin](https://crowdin.com/profile/dplanella) -- Chinese Simplified - - Huang Tao - [GitLab](https://gitlab.com/htve), [Crowdin](https://crowdin.com/profile/htve) -- Chinese Traditional +- Chinese Simplified 简体中文 - Huang Tao - [GitLab](https://gitlab.com/htve), [Crowdin](https://crowdin.com/profile/htve) + - Victor Wu - [GitLab](https://gitlab.com/victorwuky), [Crowdin](https://crowdin.com/profile/victorwu) + - Xiaogang Wen - [GitLab](https://gitlab.com/xiaogang_gitlab), [Crowdin](https://crowdin.com/profile/xiaogang_gitlab) +- Chinese Traditional 繁體中文 - Weizhe Ding - [GitLab](https://gitlab.com/d.weizhe), [Crowdin](https://crowdin.com/profile/d.weizhe) - Yi-Jyun Pan - [GitLab](https://gitlab.com/pan93412), [Crowdin](https://crowdin.com/profile/pan93412) -- Chinese Traditional, Hong Kong - - Huang Tao - [GitLab](https://gitlab.com/htve), [Crowdin](https://crowdin.com/profile/htve) + - Victor Wu - [GitLab](https://gitlab.com/victorwuky), [Crowdin](https://crowdin.com/profile/victorwu) +- Chinese Traditional, Hong Kong 繁體中文 (香港) + - Victor Wu - [GitLab](https://gitlab.com/victorwuky), [Crowdin](https://crowdin.com/profile/victorwu) + - Ivan Ip - [GitLab](https://gitlab.com/lifehome), [Crowdin](https://crowdin.com/profile/lifehome) - Czech - Proofreaders needed. - Danish - - Proofreaders needed. + - Saederup92 - [GitLab](https://gitlab.com/Saederup92), [Crowdin](https://crowdin.com/profile/Saederup92) - Dutch - Emily Hendle - [GitLab](https://gitlab.com/pundachan), [Crowdin](https://crowdin.com/profile/pandachan) - Esperanto @@ -40,6 +43,7 @@ are very appreciative of the work done by translators and proofreaders! - Pedro Garcia - [GitLab](https://gitlab.com/pedgarrod), [Crowdin](https://crowdin.com/profile/breaking_pitt) - German - Michael Hahnle - [GitLab](https://gitlab.com/mhah), [Crowdin](https://crowdin.com/profile/mhah) + - Katrin Leinweber - [GitLab](https://gitlab.com/katrinleinweber/), [Crowdin](https://crowdin.com/profile/katrinleinweber) - Greek - Proofreaders needed. - Hebrew @@ -47,15 +51,15 @@ are very appreciative of the work done by translators and proofreaders! - Hungarian - Proofreaders needed. - Indonesian + - Adi Ferdian - [GitLab](https://gitlab.com/adiferd), [Crowdin](https://crowdin.com/profile/adiferd) - Ahmad Naufal Mukhtar - [GitLab](https://gitlab.com/anaufalm), [Crowdin](https://crowdin.com/profile/anaufalm) - Italian - Paolo Falomo - [GitLab](https://gitlab.com/paolofalomo), [Crowdin](https://crowdin.com/profile/paolo.falomo) - Japanese - - Yamana Tokiuji - [GitLab](https://gitlab.com/tokiuji), [Crowdin](https://crowdin.com/profile/yamana) - Hiroyuki Sato - [GitLab](https://gitlab.com/hiroponz), [Crowdin](https://crowdin.com/profile/hiroponz) + - Tomo Dote - [Gitlab](https://gitlab.com/fu7mu4), [Crowdin](https://crowdin.com/profile/fu7mu4) - Korean - Chang-Ho Cha - [GitLab](https://gitlab.com/changho-cha), [Crowdin](https://crowdin.com/profile/zzazang) - - Huang Tao - [GitLab](https://gitlab.com/htve), [Crowdin](https://crowdin.com/profile/htve) - Ji Hun Oh - [GitLab](https://gitlab.com/Baw-Appie), [Crowdin](https://crowdin.com/profile/BawAppie) - Jeongwhan Choi - [GitLab](https://gitlab.com/jeongwhanchoi), [Crowdin](https://crowdin.com/profile/jeongwhanchoi) - Mongolian @@ -67,6 +71,7 @@ are very appreciative of the work done by translators and proofreaders! - Maksymilian Roman - [GitLab](https://gitlab.com/villaincandle), [Crowdin](https://crowdin.com/profile/villaincandle) - Portuguese - Proofreaders needed. + - Diogo Trindade - [GitLab](https://gitlab.com/luisdiogo2071317), [Crowdin](https://crowdin.com/profile/ldiogotrindade) - Portuguese, Brazilian - Paulo George Gomes Bezerra - [GitLab](https://gitlab.com/paulobezerra), [Crowdin](https://crowdin.com/profile/paulogomes.rep) - André Gama - [GitLab](https://gitlab.com/andregamma), [Crowdin](https://crowdin.com/profile/ToeOficial) @@ -125,5 +130,8 @@ are very appreciative of the work done by translators and proofreaders! your previous translations by [GitLab team members](https://about.gitlab.com/team/) or [Core team members](https://about.gitlab.com/core-team/) who are fluent in the language or current proofreaders. + - When a request is made for the first proofreader for a language and there are no [GitLab team members](https://about.gitlab.com/team/) + or [Core team members](https://about.gitlab.com/core-team/) who speak the language, we will request links to previous translation work in other communities or projects. + [proofreader-src]: https://gitlab.com/gitlab-org/gitlab-ce/blob/master/doc/development/i18n/proofreader.md diff --git a/doc/development/i18n/translation.md b/doc/development/i18n/translation.md index 99c0fe6db1d..62be3786549 100644 --- a/doc/development/i18n/translation.md +++ b/doc/development/i18n/translation.md @@ -89,9 +89,7 @@ To propose additions to the glossary please ### Inclusive language in French -In French, we should follow the guidelines from [ecriture-inclusive.fr]. For -instance: +In French, the "écriture inclusive" is now over (see on [Legifrance](https://www.legifrance.gouv.fr/affichTexte.do?cidTexte=JORFTEXT000036068906&categorieLien=id)). +So, to include both genders, write “Utilisateurs et utilisatrices” instead of “Utilisateur·rice·s”. +When space is missing, the male gender should be used alone. -- Utilisateur•rice•s - -[ecriture-inclusive.fr]: http://www.ecriture-inclusive.fr/ diff --git a/doc/development/i18n_guide.md b/doc/development/i18n_guide.md index f6e949b5fd8..e588b47e203 100644 --- a/doc/development/i18n_guide.md +++ b/doc/development/i18n_guide.md @@ -1 +1,5 @@ -This document was moved to [a new location](i18n/index.md). +--- +redirect_to: 'i18n/index.md' +--- + +This document was moved to [another location](i18n/index.md). diff --git a/doc/development/img/distributed_tracing_jaeger_ui.png b/doc/development/img/distributed_tracing_jaeger_ui.png Binary files differnew file mode 100644 index 00000000000..57517dacced --- /dev/null +++ b/doc/development/img/distributed_tracing_jaeger_ui.png diff --git a/doc/development/img/distributed_tracing_performance_bar.png b/doc/development/img/distributed_tracing_performance_bar.png Binary files differnew file mode 100644 index 00000000000..c9998cedd2d --- /dev/null +++ b/doc/development/img/distributed_tracing_performance_bar.png diff --git a/doc/development/import_export.md b/doc/development/import_export.md index f7f48b03651..fd067b80c16 100644 --- a/doc/development/import_export.md +++ b/doc/development/import_export.md @@ -27,7 +27,7 @@ Read through the current performance problems using the Import/Export below. ### OOM errors -Out of memory (OOM) errors are normally caused by the [Sidekiq Memory Killer](https://docs.gitlab.com/ee/administration/operations/sidekiq_memory_killer.html): +Out of memory (OOM) errors are normally caused by the [Sidekiq Memory Killer](../administration/operations/sidekiq_memory_killer.md): ```bash SIDEKIQ_MEMORY_KILLER_MAX_RSS = 2GB in GitLab.com diff --git a/doc/development/integrations/jira_connect.md b/doc/development/integrations/jira_connect.md new file mode 100644 index 00000000000..5bf43d320c6 --- /dev/null +++ b/doc/development/integrations/jira_connect.md @@ -0,0 +1,41 @@ +# Setting up a development environment + +The following are required to install and test the app: + +1. A Jira Cloud instance + + Atlassian provides free instances for development and testing. [Click here to sign up](http://go.atlassian.com/cloud-dev). + +1. A GitLab instance available over the internet + + For the app to work, Jira Cloud should be able to connect to the GitLab instance through the internet. + + To easily expose your local development environment, you can use tools like [serveo](https://serveo.net) or [ngrok](https://ngrok.com). + These also take care of SSL for you because Jira requires all connections to the app host to be over SSL. + +> This feature is currently behind the `:jira_connect_app` feature flag + +# Installing the app in Jira + +1. Enable Jira development mode to install apps that are not from the Atlassian Marketplace + + 1. Navigate to **Jira settings** (cog icon) > **Apps** > **Manage apps**. + 1. Scroll to the bottom of the **Manage apps** page and click **Settings**. + 1. Select **Enable development mode** and click **Apply**. + +1. Install the app + + 1. Navigate to Jira, then choose **Jira settings** (cog icon) > **Apps** > **Manage apps**. + 1. Click **Upload app**. + 1. In the **From this URL** field, provide a link to the app descriptor. The host and port must point to your GitLab instance. + + For example: + ``` + https://xxxx.serveo.net/-/jira_connect/app_descriptor.json + ``` + 1. Click **Upload**. + + If the install was successful, you should see the **GitLab for Jira** app under **Manage apps**. + You can also click **Getting Started** to open the configuration page rendered from your GitLab instance. + + _Note that any changes to the app descriptor requires you to uninstall then reinstall the app._ diff --git a/doc/development/licensed_feature_availability.md b/doc/development/licensed_feature_availability.md new file mode 100644 index 00000000000..6f3dd59b2c3 --- /dev/null +++ b/doc/development/licensed_feature_availability.md @@ -0,0 +1,37 @@ +# Licensed feature availability **[STARTER]** + +As of GitLab 9.4, we've been supporting a simplified version of licensed +feature availability checks via `ee/app/models/license.rb`, both for +on-premise or GitLab.com plans and features. + +## Restricting features scoped by namespaces or projects + +GitLab.com plans are persisted on user groups and namespaces, therefore, if you're adding a +feature such as [Related issues](../user/project/issues/related_issues.md) or +[Service desk](../user/project/service_desk.md), +it should be restricted on namespace scope. + +1. Add the feature symbol on `EES_FEATURES`, `EEP_FEATURES` or `EEU_FEATURES` constants in + `ee/app/models/license.rb`. Note on `ee/app/models/ee/namespace.rb` that _Bronze_ GitLab.com + features maps to on-premise _EES_, _Silver_ to _EEP_ and _Gold_ to _EEU_. +2. Check using: + +```ruby +project.feature_available?(:feature_symbol) +``` + +## Restricting global features (instance) + +However, for features such as [Geo](../administration/geo/replication/index.md) and +[Load balancing](../administration/database_load_balancing.md), which cannot be restricted +to only a subset of projects or namespaces, the check will be made directly in +the instance license. + +1. Add the feature symbol on `EES_FEATURES`, `EEP_FEATURES` or `EEU_FEATURES` constants in + `ee/app/models/license.rb`. +2. Add the same feature symbol to `GLOBAL_FEATURES` +3. Check using: + +```ruby +License.feature_available?(:feature_symbol) +``` diff --git a/doc/development/licensing.md b/doc/development/licensing.md index 0e71cd47481..0db90d2872f 100644 --- a/doc/development/licensing.md +++ b/doc/development/licensing.md @@ -88,9 +88,11 @@ Definitions GitLab means GitLab Inc. and its affiliates and subsidiaries. -## Requesting Approval for Licenses +## Requesting Approval for Licenses or any other Intellectual Property -Libraries that are not listed in the [Acceptable Licenses][Acceptable-Licenses] or [Unacceptable Licenses][Unacceptable-Licenses] list can be submitted to the legal team for review. Please email `legal@gitlab.com` with the details. After a decision has been made, the original requestor is responsible for updating this document. +Libraries that are not already approved and listed on the [Acceptable Licenses][Acceptable-Licenses] list or that may be listed on the [Unacceptable Licenses][Unacceptable-Licenses] list may be submitted to the legal team for review and use on a case-by-case basis. Please email `legal@gitlab.com` with the details of how the software will be used, whether or not it will be modified, and how it will be distributed (if at all). After a decision has been made, the original requestor is responsible for updating this document, if applicable. Not all approvals will be approved for universal use and may continue to remain on the Unacceptable License list. + +All inquiries relating to patents should be directed to the Legal team. ## Notes diff --git a/doc/development/migration_style_guide.md b/doc/development/migration_style_guide.md index bb40c0d32b4..9b26f691b55 100644 --- a/doc/development/migration_style_guide.md +++ b/doc/development/migration_style_guide.md @@ -186,7 +186,11 @@ end When adding a foreign-key constraint to either an existing or new column remember to also add a index on the column. -This is _required_ for all foreign-keys. +This is **required** for all foreign-keys, e.g., to support efficient cascading +deleting: when a lot of rows in a table get deleted, the referenced records need +to be deleted too. The database has to look for corresponding records in the +referenced table. Without an index, this will result in a sequential scan on the +table which can take a long time. Here's an example where we add a new column with a foreign key constraint. Note it includes `index: true` to create an index for it. @@ -423,3 +427,9 @@ _namespaces_ that have a `project_id`. The `path` column for these rows will be renamed to their previous value followed by an integer. For example: `users` would turn into `users0` + +### Moving migrations from EE to CE + +When migrations need to be moved from GitLab Enterprise Edition to GitLab Community Edition, +a migration file should be moved from `ee/db/{post_,}migrate` directory in the `gitlab-ee` project to `db/{post_,}migrate` directory in the `gitlab-ce` project. This way +the schema number remains intact, there is no need to modify old migrations, and proper columns, tables or data are added in the Community Edition. diff --git a/doc/development/new_fe_guide/development/components.md b/doc/development/new_fe_guide/development/components.md index 8ae58d30c35..963ce53423b 100644 --- a/doc/development/new_fe_guide/development/components.md +++ b/doc/development/new_fe_guide/development/components.md @@ -17,5 +17,5 @@ D3 is very popular across many projects outside of GitLab: Within GitLab, D3 has been used for the following notable features -- [Prometheus graphs](https://docs.gitlab.com/ee/user/project/integrations/prometheus.html) +- [Prometheus graphs](../../../user/project/integrations/prometheus.md) - Contribution calendars diff --git a/doc/development/new_fe_guide/development/design_patterns.md b/doc/development/new_fe_guide/development/design_patterns.md deleted file mode 100644 index ee06566ed30..00000000000 --- a/doc/development/new_fe_guide/development/design_patterns.md +++ /dev/null @@ -1,3 +0,0 @@ -# Design patterns - -> TODO: Add content diff --git a/doc/development/new_fe_guide/development/index.md b/doc/development/new_fe_guide/development/index.md index cee8e43ebad..5dced3dc466 100644 --- a/doc/development/new_fe_guide/development/index.md +++ b/doc/development/new_fe_guide/development/index.md @@ -1,9 +1,5 @@ # Development -## [Design patterns](design_patterns.md) - -Examples of proven design patterns used in our codebase. - ## [Components](components.md) Documentation on existing components and how to best create a new component. @@ -12,14 +8,6 @@ Documentation on existing components and how to best create a new component. Learn how to implement an accessible frontend. -## [Network requests](network_requests.md) - -Learn how to handle network requests in our codebase. - -## [Security](security.md) - -Learn how to ensure that our frontend is secure. - ## [Performance](performance.md) Learn how to keep our frontend performant. diff --git a/doc/development/new_fe_guide/development/network_requests.md b/doc/development/new_fe_guide/development/network_requests.md deleted file mode 100644 index 047c00313bc..00000000000 --- a/doc/development/new_fe_guide/development/network_requests.md +++ /dev/null @@ -1,3 +0,0 @@ -# Network requests - -> TODO: Add content diff --git a/doc/development/new_fe_guide/development/security.md b/doc/development/new_fe_guide/development/security.md deleted file mode 100644 index 5bb38f17988..00000000000 --- a/doc/development/new_fe_guide/development/security.md +++ /dev/null @@ -1,14 +0,0 @@ -# Security - -## Avoid inline scripts and styles - -Inline scripts and styles should be avoided in almost all cases. In an effort to protect users from [XSS vulnerabilities](https://en.wikipedia.org/wiki/Cross-site_scripting), we will be disabling inline scripts using Content Security Policy. - -## Including external resources - -External fonts, CSS, and JavaScript should never be used with the exception of Google Analytics and Piwik - and only when the instance has enabled it. Assets should always be hosted and served locally from the GitLab instance. Embedded resources via `iframes` should never be used except in certain circumstances such as with ReCaptcha, which cannot be used without an `iframe`. - -## Resources for security testing - -- [Mozilla's HTTP Observatory CLI](https://github.com/mozilla/http-observatory-cli) -- [Qualys SSL Labs Server Test](https://www.ssllabs.com/ssltest/analyze.html) diff --git a/doc/development/new_fe_guide/index.md b/doc/development/new_fe_guide/index.md index 5fd5af252ef..0e8f5486861 100644 --- a/doc/development/new_fe_guide/index.md +++ b/doc/development/new_fe_guide/index.md @@ -3,14 +3,6 @@ This guide contains all the information to successfully contribute to GitLab's frontend. This is a living document, and we welcome contributions, feedback and suggestions. -## [Principles](principles.md) - -Ensure that your frontend contribution starts off in the right direction. - -## [Initiatives](initiatives.md) - -High level overview of where we are going from a frontend perspective. - ## [Development](development/index.md) Guidance on topics related to development. @@ -27,9 +19,6 @@ Learn about all the internal JavaScript modules that make up our frontend. Style guides to keep our code consistent. -## [Event Tracking with Snowplow](event_tracking.md) - -How we use Snowplow to track custom events. ## [Tips](tips.md) diff --git a/doc/development/new_fe_guide/initiatives.md b/doc/development/new_fe_guide/initiatives.md deleted file mode 100644 index c81ed3579f0..00000000000 --- a/doc/development/new_fe_guide/initiatives.md +++ /dev/null @@ -1,3 +0,0 @@ -# Initiatives - -> TODO: Add Initiatives diff --git a/doc/development/new_fe_guide/principles.md b/doc/development/new_fe_guide/principles.md deleted file mode 100644 index 0af5f506a91..00000000000 --- a/doc/development/new_fe_guide/principles.md +++ /dev/null @@ -1,35 +0,0 @@ -# Principles - -These principles will ensure that your frontend contribution starts off in the right direction. - -## Discuss architecture before implementation - -Discuss your architecture design in an issue before writing code. This helps decrease the review time and also provides good practice for writing and thinking about system design. - -## Be consistent - -There are multiple ways of writing code to accomplish the same results. We should be as consistent as possible in how we write code across our codebases. This will make it more easier us to maintain our code across GitLab. - -## Enhance progressively - -Whenever you see with existing code that does not follow our current style guide, update it proactively. Refrain from changing everything but each merge request should progressively enhance our codebase and reduce technical debt. - -## When to use Vue - -- Use Vue for feature that make use of heavy DOM manipulation -- Use Vue for reusable components - -## When to use jQuery - -- Use jQuery to interact with Bootstrap JavaScript components -- Avoid jQuery when a better alternative exists. We are slowly moving away from it [#43559][jquery-future] - -## Mixing Vue and jQuery - -- Mixing Vue and jQuery is not recommended. -- If you need to use a specific jQuery plugin in Vue, [create a wrapper around it][select2]. -- It is acceptable for Vue to listen to existing jQuery events using jQuery event listeners. -- It is not recommended to add new jQuery events for Vue to interact with jQuery. - -[jquery-future]: https://gitlab.com/gitlab-org/gitlab-ce/issues/43559 -[select2]: https://vuejs.org/v2/examples/select2.html diff --git a/doc/development/new_fe_guide/style/html.md b/doc/development/new_fe_guide/style/html.md index 035fcbb28df..e8c9c2ccebf 100644 --- a/doc/development/new_fe_guide/style/html.md +++ b/doc/development/new_fe_guide/style/html.md @@ -2,11 +2,11 @@ ## Buttons -<a name="button-type"></a><a name="1.1"></a> +### Button type -- [1.1](#button-type) **Use button type** Button tags requires a `type` attribute according to the [W3C HTML specification][button-type-spec]. +Button tags requires a `type` attribute according to the [W3C HTML specification](https://www.w3.org/TR/2011/WD-html5-20110525/the-button-element.html#dom-button-type). -``` +```html // bad <button></button> @@ -14,11 +14,11 @@ <button type="button"></button> ``` -<a name="button-role"></a><a name="1.2"></a> +### Button role -- [1.2](#button-role) **Use button role for non buttons** If an HTML element has an onClick handler but is not a button, it should have `role="button"`. This is more [accessible][button-role-accessible]. +If an HTML element has an `onClick` handler but is not a button, it should have `role="button"`. This is [more accessible](https://developer.mozilla.org/en-US/docs/Web/Accessibility/ARIA/ARIA_Techniques/Using_the_button_role). -``` +```html // bad <div onClick="doSomething"></div> @@ -28,11 +28,11 @@ ## Links -<a name="blank-links"></a><a name="2.1"></a> +### Blank target -- [2.1](#blank-links) **Use rel for target blank** Use `rel="noopener noreferrer"` whenever your links open in a new window i.e. `target="_blank"`. This prevents [the following][jitbit-target-blank] security vulnerability documented by JitBit +Use `rel="noopener noreferrer"` whenever your links open in a new window, i.e. `target="_blank"`. This prevents a security vulnerability [documented by JitBit](https://www.jitbit.com/alexblog/256-targetblank---the-most-underestimated-vulnerability-ever/). -``` +```html // bad <a href="url" target="_blank"></a> @@ -40,18 +40,14 @@ <a href="url" target="_blank" rel="noopener noreferrer"></a> ``` -<a name="fake-links"></a><a name="2.2"></a> +### Fake links -- [2.2](#fake-links) **Do not use fake links** Use a button tag if a link only invokes JavaScript click event handlers. This is more semantic. +**Do not use fake links.** Use a button tag if a link only invokes JavaScript click event handlers, which is more semantic. -``` +```html // bad <a class="js-do-something" href="#"></a> // good <button class="js-do-something" type="button"></button> ``` - -[button-type-spec]: https://www.w3.org/TR/2011/WD-html5-20110525/the-button-element.html#dom-button-type -[button-role-accessible]: https://developer.mozilla.org/en-US/docs/Web/Accessibility/ARIA/ARIA_Techniques/Using_the_button_role -[jitbit-target-blank]: https://www.jitbit.com/alexblog/256-targetblank---the-most-underestimated-vulnerability-ever/ diff --git a/doc/development/new_fe_guide/style/javascript.md b/doc/development/new_fe_guide/style/javascript.md index 7985b893c9e..3019eaa089c 100644 --- a/doc/development/new_fe_guide/style/javascript.md +++ b/doc/development/new_fe_guide/style/javascript.md @@ -1,208 +1,196 @@ # JavaScript style guide -We use [Airbnb's JavaScript Style Guide][airbnb-style-guide] and it's accompanying linter to manage most of our JavaScript style guidelines. +We use [Airbnb's JavaScript Style Guide](https://github.com/airbnb/javascript) and it's accompanying +linter to manage most of our JavaScript style guidelines. -In addition to the style guidelines set by Airbnb, we also have a few specific rules listed below. +In addition to the style guidelines set by Airbnb, we also have a few specific rules +listed below. > **Tip:** You can run eslint locally by running `yarn eslint` -## Arrays +## Avoid forEach + +Avoid forEach when mutating data. Use `map`, `reduce` or `filter` instead of `forEach` +when mutating data. This will minimize mutations in functions, +which aligns with [Airbnb's style guide](https://github.com/airbnb/javascript#testing--for-real). + +```javascript +// bad +users.forEach((user, index) => { + user.id = index; +}); + +// good +const usersWithId = users.map((user, index) => { + return Object.assign({}, user, { id: index }); +}); +``` + +## Limit number of parameters + +If your function or method has more than 3 parameters, use an object as a parameter +instead. + +```javascript +// bad +function a(p1, p2, p3) { + // ... +}; + +// good +function a(p) { + // ... +}; +``` + +## Avoid side effects in constructors + +Avoid making asynchronous calls, API requests or DOM manipulations in the `constructor`. +Move them into separate functions instead. This will make tests easier to write and +code easier to maintain. -<a name="avoid-foreach"></a><a name="1.1"></a> +```javascript +// bad +class myClass { + constructor(config) { + this.config = config; + axios.get(this.config.endpoint) + } +} -- [1.1](#avoid-foreach) **Avoid ForEach when mutating data** Use `map`, `reduce` or `filter` instead of `forEach` when mutating data. This will minimize mutations in functions ([which is aligned with Airbnb's style guide][airbnb-minimize-mutations]) +// good +class myClass { + constructor(config) { + this.config = config; + } - ``` - // bad - users.forEach((user, index) => { - user.id = index; - }); + makeRequest() { + axios.get(this.config.endpoint) + } +} +const instance = new myClass(); +instance.makeRequest(); - // good - const usersWithId = users.map((user, index) => { - return Object.assign({}, user, { id: index }); - }); - ``` +``` -## Functions +## Avoid classes to handle DOM events -<a name="limit-params"></a><a name="2.1"></a> +If the only purpose of the class is to bind a DOM event and handle the callback, prefer +using a function. -- [2.1](#limit-params) **Limit number of parameters** If your function or method has more than 3 parameters, use an object as a parameter instead. +```javascript +// bad +class myClass { + constructor(config) { + this.config = config; + } - ``` - // bad - function a(p1, p2, p3) { - // ... - }; + init() { + document.addEventListener('click', () => {}); + } +} - // good - function a(p) { - // ... - }; - ``` +// good -## Classes & constructors +const myFunction = () => { + document.addEventListener('click', () => { + // handle callback here + }); +} +``` -<a name="avoid-constructor-side-effects"></a><a name="3.1"></a> +## Pass element container to constructor -- [3.1](#avoid-constructor-side-effects) **Avoid side effects in constructors** Avoid making some operations in the `constructor`, such as asynchronous calls, API requests and DOM manipulations. Prefer moving them into separate functions. This will make tests easier to write and code easier to maintain. +When your class manipulates the DOM, receive the element container as a parameter. +This is more maintainable and performant. - ```javascript - // bad - class myClass { - constructor(config) { - this.config = config; - axios.get(this.config.endpoint) - } - } +```javascript +// bad +class a { + constructor() { + document.querySelector('.b'); + } +} - // good - class myClass { - constructor(config) { - this.config = config; - } +// good +class a { + constructor(options) { + options.container.querySelector('.b'); + } +} +``` - makeRequest() { - axios.get(this.config.endpoint) - } - } - const instance = new myClass(); - instance.makeRequest(); +## Use ParseInt - ``` +Use `ParseInt` when converting a numeric string into a number. -<a name="avoid-classes-to-handle-dom-events"></a><a name="3.2"></a> +```javascript +// bad +Number('10') -- [3.2](#avoid-classes-to-handle-dom-events) **Avoid classes to handle DOM events** If the only purpose of the class is to bind a DOM event and handle the callback, prefer using a function. +// good +parseInt('10', 10); +``` - ``` - // bad - class myClass { - constructor(config) { - this.config = config; - } +## CSS Selectors - Use `js-` prefix - init() { - document.addEventListener('click', () => {}); - } - } +If a CSS class is only being used in JavaScript as a reference to the element, prefix +the class name with `js-`. - // good +```html +// bad +<button class="add-user"></button> - const myFunction = () => { - document.addEventListener('click', () => { - // handle callback here - }); - } - ``` +// good +<button class="js-add-user"></button> +``` -<a name="element-container"></a><a name="3.3"></a> +## Absolute vs relative paths for modules -- [3.3](#element-container) **Pass element container to constructor** When your class manipulates the DOM, receive the element container as a parameter. - This is more maintainable and performant. +Use relative paths if the module you are importing is less than two levels up. - ``` - // bad - class a { - constructor() { - document.querySelector('.b'); - } - } +```javascript +// bad +import GitLabStyleGuide from '~/guides/GitLabStyleGuide'; - // good - class a { - constructor(options) { - options.container.querySelector('.b'); - } - } - ``` +// good +import GitLabStyleGuide from '../GitLabStyleGuide'; +``` -## Type Casting & Coercion +If the module you are importing is two or more levels up, use an absolute path instead: -<a name="use-parseint"></a><a name="4.1"></a> +```javascript +// bad +import GitLabStyleGuide from '../../../guides/GitLabStyleGuide'; -- [4.1](#use-parseint) **Use ParseInt** Use `ParseInt` when converting a numeric string into a number. +// good +import GitLabStyleGuide from '~/GitLabStyleGuide'; +``` - ``` - // bad - Number('10') +Additionally, **do not add to global namespace**. - // good - parseInt('10', 10); - ``` +## Do not use `DOMContentLoaded` in non-page modules -## CSS Selectors +Imported modules should act the same each time they are loaded. `DOMContentLoaded` +events are only allowed on modules loaded in the `/pages/*` directory because those +are loaded dynamically with webpack. -<a name="use-js-prefix"></a><a name="5.1"></a> +## Avoid XSS -- [5.1](#use-js-prefix) **Use js prefix** If a CSS class is only being used in JavaScript as a reference to the element, prefix the class name with `js-` +Do not use `innerHTML`, `append()` or `html()` to set content. It opens up too many +vulnerabilities. - ``` - // bad - <button class="add-user"></button> +## Disabling ESLint in new files - // good - <button class="js-add-user"></button> - ``` +Do not disable ESLint when creating new files. Existing files may have existing rules +disabled due to legacy compatibility reasons but they are in the process of being refactored. -## Modules +Do not disable specific ESLint rules. Due to technical debt, you may disable the following +rules only if you are invoking/instantiating existing code modules. -<a name="use-absolute-paths"></a><a name="6.1"></a> + - [no-new](http://eslint.org/docs/rules/no-new) + - [class-method-use-this](http://eslint.org/docs/rules/class-methods-use-this) -- [6.1](#use-absolute-paths) **Use absolute paths for nearby modules** Use absolute paths if the module you are importing is less than two levels up. - - ``` - // bad - import GitLabStyleGuide from '~/guides/GitLabStyleGuide'; - - // good - import GitLabStyleGuide from '../GitLabStyleGuide'; - ``` - -<a name="use-relative-paths"></a><a name="6.2"></a> - -- [6.2](#use-relative-paths) **Use relative paths for distant modules** If the module you are importing is two or more levels up, use a relative path instead of an absolute path. - - ``` - // bad - import GitLabStyleGuide from '../../../guides/GitLabStyleGuide'; - - // good - import GitLabStyleGuide from '~/GitLabStyleGuide'; - ``` - -<a name="global-namespace"></a><a name="6.3"></a> - -- [6.3](#global-namespace) **Do not add to global namespace** - -<a name="domcontentloaded"></a><a name="6.4"></a> - -- [6.4](#domcontentloaded) **Do not use DOMContentLoaded in non-page modules** Imported modules should act the same each time they are loaded. `DOMContentLoaded` events are only allowed on modules loaded in the `/pages/*` directory because those are loaded dynamically with webpack. - -## Security - -<a name="avoid-xss"></a><a name="7.1"></a> - -- [7.1](#avoid-xss) **Avoid XSS** Do not use `innerHTML`, `append()` or `html()` to set content. It opens up too many vulnerabilities. - -## ESLint - -<a name="disable-eslint-file"></a><a name="8.1"></a> - -- [8.1](#disable-eslint-file) **Disabling ESLint in new files** Do not disable ESLint when creating new files. Existing files may have existing rules disabled due to legacy compatibility reasons but they are in the process of being refactored. - -<a name="disable-eslint-rule"></a><a name="8.2"></a> - -- [8.2](#disable-eslint-rule) **Disabling ESLint rule** Do not disable specific ESLint rules. Due to technical debt, you may disable the following rules only if you are invoking/instantiating existing code modules - - - [no-new][no-new] - - [class-method-use-this][class-method-use-this] - -> Note: Disable these rules on a per line basis. This makes it easier to refactor in the future. E.g. use `eslint-disable-next-line` or `eslint-disable-line` - -[airbnb-style-guide]: https://github.com/airbnb/javascript -[airbnb-minimize-mutations]: https://github.com/airbnb/javascript#testing--for-real -[no-new]: http://eslint.org/docs/rules/no-new -[class-method-use-this]: http://eslint.org/docs/rules/class-methods-use-this +> Note: Disable these rules on a per line basis. This makes it easier to refactor + in the future. E.g. use `eslint-disable-next-line` or `eslint-disable-line`. diff --git a/doc/development/new_fe_guide/tips.md b/doc/development/new_fe_guide/tips.md index 881ad1662ae..4564f678ec0 100644 --- a/doc/development/new_fe_guide/tips.md +++ b/doc/development/new_fe_guide/tips.md @@ -7,3 +7,19 @@ To clear production compiled assets created with `yarn webpack-prod` you can run ``` yarn clean ``` + +## Creating feature flags in development + +The process for creating a feature flag is the same as [enabling a feature flag in development](../feature_flags.md#enabling-a-feature-flag-in-development). + +Your feature flag can now be: + +- [made available to the frontend](../feature_flags.md#frontend) via the `gon` +- queried in [tests](../feature_flags.md#specs) +- queried in HAML templates and ruby files via the `Feature.enabled?(:my_shiny_new_feature_flag)` method + +### More on feature flags + +- [Deleting a feature flag](../../api/features.md#delete-a-feature) +- [Manage feature flags](../feature_flags.md) +- [Feature flags API](../../api/features.md) diff --git a/doc/development/packages.md b/doc/development/packages.md new file mode 100644 index 00000000000..ab0c5f9904d --- /dev/null +++ b/doc/development/packages.md @@ -0,0 +1,68 @@ +# Packages **[PREMIUM]** + +This document will guide you through adding another [package management system](../administration/packages.md) support to GitLab. + +See already supported package types in [Packages documentation](../administration/packages.md) + +Since GitLab packages' UI is pretty generic, it is possible to add new +package system support by solely backend changes. This guide is superficial and does +not cover the way the code should be written. However, you can find a good example +by looking at existing merge requests with Maven and NPM support: + +- [NPM registry support](https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/8673). +- [Maven repository](https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/6607). +- [Instance level endpoint for Maven repository](https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/8757) + +## General information + +The existing database model requires the following: + +- Every package belongs to a project. +- Every package file belongs to a package. +- A package can have one or more package files. +- The package model is based on storing information about the package and its version. + +## API endpoints + +Package systems work with GitLab via API. For example `ee/lib/api/npm_packages.rb` +implements API endpoints to work with NPM clients. So, the first thing to do is to +add a new `ee/lib/api/your_name_packages.rb` file with API endpoints that are +necessary to make the package system client to work. Usually that means having +endpoints like: + +- GET package information. +- GET package file content. +- PUT upload package. + +Since the packages belong to a project, it's expected to have project-level endpoint +for uploading and downloading them. For example: + +``` +GET https://gitlab.com/api/v4/projects/<your_project_id>/packages/npm/ +PUT https://gitlab.com/api/v4/projects/<your_project_id>/packages/npm/ +``` + +Group-level and instance-level endpoints are good to have but are optional. + +NOTE: **Note:** +To avoid name conflict for instance-level endpoints we use +[the package naming convention](../user/project/packages/npm_registry.md#package-naming-convention) + +## Configuration + +GitLab has a `packages` section in its configuration file (`gitlab.rb`). +It applies to all package systems supported by GitLab. Usually you don't need +to add anything there. + +Packages can be configured to use object storage, therefore your code must support it. + +## Database + +The current database model allows you to store a name and a version for each package. +Every time you upload a new package, you can either create a new record of `Package` +or add files to existing record. `PackageFile` should be able to store all file-related +information like the file `name`, `side`, `sha1`, etc. + +If there is specific data necessary to be stored for only one package system support, +consider creating a separate metadata model. See `packages_maven_metadata` table +and `Packages::MavenMetadatum` model as example for package specific data. diff --git a/doc/development/performance.md b/doc/development/performance.md index 2a287611bdf..0e21d45f57c 100644 --- a/doc/development/performance.md +++ b/doc/development/performance.md @@ -38,6 +38,7 @@ GitLab provides built-in tools to help improve performance and availability: - [Profiling](profiling.md). - [Sherlock](profiling.md#sherlock). +- [Distributed Tracing](distributed_tracing.md) - [GitLab Performance Monitoring](../administration/monitoring/performance/index.md). - [Request Profiling](../administration/monitoring/performance/request_profiling.md). - [QueryRecoder](query_recorder.md) for preventing `N+1` regressions. diff --git a/doc/development/profiling.md b/doc/development/profiling.md index f41d635de43..b2f3a105b23 100644 --- a/doc/development/profiling.md +++ b/doc/development/profiling.md @@ -10,6 +10,10 @@ There is a `Gitlab::Profiler.profile` method, and corresponding `bin/profile-url` script, that enable profiling a GET or POST request to a specific URL, either as an anonymous user (the default) or as a specific user. +NOTE: **Note:** The first argument to the profiler is either a full URL +(including the instance hostname) or an absolute path, including the +leading slash. + When using the script, command-line documentation is available by passing no arguments. @@ -86,10 +90,8 @@ that builds on this to add some additional niceties, such as allowing configuration with a single Yaml file for multiple URLs, and uploading of the profile and log output to S3. -For GitLab.com, currently the latest profiling data has been [moved from -Redash to Looker](https://gitlab.com/gitlab-com/Product/issues/5#note_121194467). -We are [currently investigating how to make this data -public](https://gitlab.com/meltano/looker/issues/294). +For GitLab.com, you can find the latest results here: +<http://redash.gitlab.com/dashboard/gitlab-profiler-statistics> ## Sherlock diff --git a/doc/development/python_guide/index.md b/doc/development/python_guide/index.md new file mode 100644 index 00000000000..6025dc9ebf2 --- /dev/null +++ b/doc/development/python_guide/index.md @@ -0,0 +1,79 @@ +# Python Development Guidelines + +GitLab requires Python as a dependency for [reStructuredText](http://docutils.sourceforge.net/rst.html) +markup rendering. + +As of GitLab 11.10, we require Python 3. + +## Installation + +There are several ways of installing python on your system. To be able to use the same version we use in production, +we suggest you use [pyenv](https://github.com/pyenv/pyenv). It works and behave similar to its counterpart in the +ruby world: [rbenv](https://github.com/rbenv/rbenv). + +### macOS + +To install `pyenv` on macOS, you can use [Homebrew](https://brew.sh/) with: + +```bash +brew install pyenv +``` + +### Linux + +To install `pyenv` on Linux, you can run the command below: + +```bash +curl https://pyenv.run | bash +``` + +Alternatively, you may find `pypenv` available as a system package via your distro package manager. + +You can read more about it in: <https://github.com/pyenv/pyenv-installer#prerequisites>. + +### Shell integration + +Pyenv installation will add required changes to Bash. If you use a different shell, +check for any additional steps required for it. + +For Fish, you can install a plugin for [Fisherman](https://github.com/fisherman/fisherman): + +```bash +fisher add fisherman/pyenv +``` + +Or for [Oh My Fish](https://github.com/oh-my-fish/oh-my-fish): + +```bash +omf install pyenv +``` + +## Dependency management + +While GitLab doesn't directly contain any Python scripts, because we depend on Python to render +[reStructuredText](http://docutils.sourceforge.net/rst.html) markup, we need to keep track on dependencies +on the main project level, so we can run that on our development machines. + +Recently, an equivalent to the `Gemfile` and the [Bundler](https://bundler.io/) project has been introduced to Python: +`Pipfile` and [Pipenv](https://pipenv.readthedocs.io/en/latest/). + +You will now find a `Pipfile` with the dependencies in the root folder. To install them, run: + +```bash +pipenv install +``` + +Running this command will install both the required Python version as well as required pip dependencies. + +## Use instructions + +To run any python code under the Pipenv environment, you need to first start a `virtualenv` based on the dependencies +of the application. With Pipenv, this is a simple as running: + +```bash +pipenv shell +``` + +After running that command, you can run GitLab on the same shell and it will be using the Python and dependencies +installed from the `pipenv install` command. + diff --git a/doc/development/rake_tasks.md b/doc/development/rake_tasks.md index dcb32c89f65..28a12572961 100644 --- a/doc/development/rake_tasks.md +++ b/doc/development/rake_tasks.md @@ -12,6 +12,40 @@ The `setup` task is an alias for `gitlab:setup`. This tasks calls `db:reset` to create the database, calls `add_limits_mysql` that adds limits to the database schema in case of a MySQL database and finally it calls `db:seed_fu` to seed the database. Note: `db:setup` calls `db:seed` but this does nothing. +### Seeding issues for all or a given project + +You can seed issues for all or a given project with the `gitlab:seed:issues` +task: + +```shell +# All projects +bin/rake gitlab:seed:issues + +# A specific project +bin/rake "gitlab:seed:issues[group-path/project-path]" +``` + +By default, this seeds an average of 2 issues per week for the last 5 weeks per +project. + +#### Seeding issues for Insights charts **[ULTIMATE]** + +You can seed issues specifically for working with the +[Insights charts](../user/group/insights/index.md) with the +`gitlab:seed:insights:issues` task: + +```shell +# All projects +bin/rake gitlab:seed:insights:issues + +# A specific project +bin/rake "gitlab:seed:insights:issues[group-path/project-path]" +``` + +By default, this seeds an average of 10 issues per week for the last 52 weeks +per project. All issues will also be randomly labeled with team, type, severity, +and priority. + ### Automation If you're very sure that you want to **wipe the current database** and refill @@ -74,11 +108,13 @@ To make sure that indices still fit. You could find great details in: In order to run the test you can use the following commands: -- `rake spec` to run the rspec suite -- `rake karma` to run the karma test suite -- `rake gitlab:test` to run all the tests +- `bin/rake spec` to run the rspec suite +- `bin/rake spec:unit` to run the only the unit tests +- `bin/rake spec:integration` to run the only the integration tests +- `bin/rake spec:system` to run the only the system tests +- `bin/rake karma` to run the karma test suite -Note: `rake spec` takes significant time to pass. +Note: `bin/rake spec` takes significant time to pass. Instead of running full test suite locally you can save a lot of time by running a single test or directory related to your changes. After you submit merge request CI will run full test suite for you. Green CI status in the merge request means @@ -87,6 +123,9 @@ full test suite is passed. Note: You can't run `rspec .` since this will try to run all the `_spec.rb` files it can find, also the ones in `/tmp` +Note: You can pass RSpec command line options to the `spec:unit`, +`spec:integration`, and `spec:system` tasks, e.g. `bin/rake "spec:unit[--tag ~geo --dry-run]"`. + To run a single test file you can use: - `bin/rspec spec/controllers/commit_controller_spec.rb` for a rspec test diff --git a/doc/development/rolling_out_changes_using_feature_flags.md b/doc/development/rolling_out_changes_using_feature_flags.md index ef1aba95712..84028b1b342 100644 --- a/doc/development/rolling_out_changes_using_feature_flags.md +++ b/doc/development/rolling_out_changes_using_feature_flags.md @@ -65,15 +65,14 @@ the worst case scenario, which we should optimise for, our total cost is now 20. If we had used a feature flag, things would have been very different. We don't need to revert a release, and because feature flags are disabled by default we don't need to revert and pick any Git commits. In fact, all we have to do is -disable the feature, and _maybe_ perform some cleanup. Let's say that the cost -of this is 1. In this case, our best case cost is 11: 10 to build the feature, -and 1 to add the feature flag. The worst case cost is now 12: 10 to build the -feature, 1 to add the feature flag, and 1 to disable it. +disable the feature, and in the worst case, perform cleanup. Let's say that +the cost of this is 2. In this case, our best case cost is 11: 10 to build the +feature, and 1 to add the feature flag. The worst case cost is now 13: 10 to +build the feature, 1 to add the feature flag, and 2 to disable and clean up. Here we can see that in the best case scenario the work necessary is only a tiny bit more compared to not using a feature flag. Meanwhile, the process of -reverting our changes has been made significantly cheaper, to the point of being -trivial. +reverting our changes has been made significantly and reliably cheaper. In other words, feature flags do not slow down the development process. Instead, they speed up the process as managing incidents now becomes _much_ easier. Once @@ -103,7 +102,21 @@ GitLab's feature library (using [Flipper](https://github.com/jnunemaker/flipper), and covered in the [Feature Flags](feature_flags.md) guide) supports rolling out changes to a percentage of users. This in turn can be controlled using [GitLab -chatops](https://docs.gitlab.com/ee/ci/chatops/). +chatops](../ci/chatops/README.md). + +For an up to date list of feature flag commands please see [the source +code](https://gitlab.com/gitlab-com/chatops/blob/master/lib/chatops/commands/feature.rb). +Note that all the examples in that file must be preceded by +`/chatops run`. + +If you get an error "Whoops! This action is not allowed. This incident +will be reported." that means your Slack account is not allowed to +change feature flags. To test if you are allowed to do anything at all, +run: + +``` +/chatops run feature --help +``` For example, to enable a feature for 25% of all users, run the following in Slack: @@ -135,6 +148,20 @@ want to wait several hours or even days. This is entirely up to you, just make sure it is clearly communicated to your team, and the Production team if you anticipate any potential problems. +Feature gates can also be actor based, for example a feature could first be +enabled for only the `gitlab-ce` project. The project is passed by supplying a +`--project` flag: + +``` +/chatops run feature set --project=gitlab-org/gitlab-ce some_feature true +``` + +For groups the `--group` flag is available: + +``` +/chatops run feature set --group=gitlab-org some_feature true +``` + Once a change is deemed stable, submit a new merge request to remove the feature flag. This ensures the change is available to all users and self-hosted instances. Make sure to add the ~"feature flag" label to this merge request so @@ -173,10 +200,9 @@ isn't gated by a License or Plan. ### Undefined feature flags default to "on" -An important side-effect of the [implicit feature -flags][#implicit-feature-flags] mentioned above is that unless the feature is -explicitly disabled or limited to a percentage of users, the feature flag check -will default to `true`. +An important side-effect of the [implicit feature flags](#implicit-feature-flags) +mentioned above is that unless the feature is explicitly disabled or limited to a +percentage of users, the feature flag check will default to `true`. As an example, if you were to ship the backend half of a feature behind a flag, you'd want to explicitly disable that flag until the frontend half is also ready @@ -188,3 +214,12 @@ to be shipped. You can do this via ChatOps: Note that you can do this at any time, even before the merge request using the flag has been merged! + +### Cleaning up + +When a feature gate has been removed from the code base, the value still exists +in the database. This can be removed through ChatOps: + +``` +/chatops run feature delete some_feature +``` diff --git a/doc/development/routing.md b/doc/development/routing.md new file mode 100644 index 00000000000..e9c0ad8d4e8 --- /dev/null +++ b/doc/development/routing.md @@ -0,0 +1,63 @@ +# Routing + +The GitLab backend is written primarily with Rails so it uses [Rails +routing](https://guides.rubyonrails.org/routing.html). Beside Rails best +practices, there are few rules unique to the GitLab application. To +support subgroups, GitLab project and group routes use the wildcard +character to match project and group routes. For example, we might have +a path such as: + + /gitlab-com/customer-success/north-america/west/customerA + +However, paths can be ambiguous. Consider the following example: + + /gitlab-com/edit + +It's ambiguous whether there is a subgroup named `edit` or whether +this is a special endpoint to edit the `gitlab-com` group. + +To eliminate the ambiguity and to make the backend easier to maintain, +we introduced the `/-/` scope. The purpose of it is to separate group or +project paths from the rest of the routes. Also it helps to reduce the +number of [reserved names](../user/reserved_names.md). + +## Global routes + +We have a number of global routes. For example: + + /-/health + /-/metrics + +## Group routes + +Every group route must be under the `/-/` scope. + +Examples: + + gitlab-org/-/edit + gitlab-org/-/activity + gitlab-org/-/security/dashboard + gitlab-org/serverless/-/activity + +To achieve that, use the `scope '-'` method. + +## Project routes + +Every project route must be under the `/-/` scope, except cases where a Git +client or other software requires something different. + +Examples: + + gitlab-org/gitlab-ce/-/activity + gitlab-org/gitlab-ce/-/jobs/123 + gitlab-org/gitlab-ce/-/settings/repository + gitlab-org/serverless/runtimes/-/settings/repository + +Currently, only some project routes are placed under the `/-/` scope. However, +you can help us migrate more of them! To migrate project routes: + +1. Modify existing routes by adding `-` scope. +1. Add redirects for legacy routes by using `Gitlab::Routing.redirect_legacy_paths`. +1. Create a technical debt issue to remove deprecated routes in later releases. + +To get started, see an [example merge request](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/28435). diff --git a/doc/development/session.md b/doc/development/session.md new file mode 100644 index 00000000000..9edce3dbda0 --- /dev/null +++ b/doc/development/session.md @@ -0,0 +1,65 @@ +# Accessing session data + +Session data in GitLab is stored in Redis and can be accessed in a variety of ways. + +During a web request, for example: + +- Rails provides access to the session from within controllers through [`ActionDispatch::Session`](https://guides.rubyonrails.org/action_controller_overview.html#session). +- Outside of controllers, it is possible to access the session through `Gitlab::Session`. + +Outside of a web request it is still possible to access sessions stored in Redis. For example: + +- Session IDs and contents can be [looked up directly in Redis](#redis). +- Data about the UserAgent associated with the session can be accessed through `ActiveSession`. + +When storing values in a session it is best to: + +- Use simple primitives and avoid storing objects to avoid marshaling complications. +- Clean up after unneeded variables to keep memory usage in Redis down. + +## Gitlab::Session + +Sometimes you might want to persist data in the session instead of another store like the database. `Gitlab::Session` lets you access this without passing the session around extensively. For example, you could access it from within a policy without having to pass the session through to each place permissions are checked from. + +The session has a hash-like interface, just like when using it from a controller. There is also `NamespacedSessionStore` for storing key-value data in a hash. + +```ruby +# Lookup a value stored in the current session +Gitlab::Session.current[:my_feature] + +# Modify the current session stored in redis +Gitlab::Session.current[:my_feature] = value + +# Store key-value data namespaced under a key +Gitlab::NamespacedSessionStore.new(:my_feature)[some_key] = value + +# Set the session for a block of code, such as for tests +Gitlab::Session.with_session(my_feature: value) do + # Code that uses Session.current[:my_feature] +end +``` + +## Redis + +Session data can be accessed directly through Redis. This can let you check up on a browser session when debugging. + +```ruby +# Get a list of sessions +session_ids = Gitlab::Redis::SharedState.with do |redis| + redis.smembers("#{Gitlab::Redis::SharedState::USER_SESSIONS_LOOKUP_NAMESPACE}:#{user.id}") +end + +# Retrieve a specific session +session_data = Gitlab::Redis::SharedState.with { |redis| redis.get("#{Gitlab::Redis::SharedState::SESSION_NAMESPACE}:#{session_id}") } +Marshal.load(session_data) +``` + +## Getting device information with ActiveSession + +The [**Active Sessions** page on a user's profile](../user/profile/active_sessions.md) displays information about the device used to access each session. The methods used there to list sessions can also be useful for development. + +```ruby +# Get list of sessions for a given user +# Includes session_id and data from the UserAgent +ActiveSession.list(user) +``` diff --git a/doc/development/sql.md b/doc/development/sql.md index 47519d39e74..edeca7fb298 100644 --- a/doc/development/sql.md +++ b/doc/development/sql.md @@ -155,6 +155,21 @@ The _only_ time you should use `pluck` is when you actually need to operate on the values in Ruby itself (e.g. write them to a file). In almost all other cases you should ask yourself "Can I not just use a sub-query?". +In line with our `CodeReuse/ActiveRecord` cop, you should only use forms like +`pluck(:id)` or `pluck(:user_id)` within model code. In the former case, you can +use the `ApplicationRecord`-provided `.pluck_primary_key` helper method instead. +In the latter, you should add a small helper method to the relevant model. + +## Inherit from ApplicationRecord + +Most models in the GitLab codebase should inherit from `ApplicationRecord`, +rather than from `ActiveRecord::Base`. This allows helper methods to be easily +added. + +An exception to this rule exists for models created in database migrations. As +these should be isolated from application code, they should continue to subclass +from `ActiveRecord::Base`. + ## Use UNIONs UNIONs aren't very commonly used in most Rails applications but they're very diff --git a/doc/development/testing.md b/doc/development/testing.md index 45b1519ece8..79ef8e75432 100644 --- a/doc/development/testing.md +++ b/doc/development/testing.md @@ -1 +1,5 @@ -This document was moved to [testing_guide/index.md](testing_guide/index.md). +--- +redirect_to: 'testing_guide/index.md' +--- + +This document was moved to [another location](testing_guide/index.md). diff --git a/doc/development/testing_guide/best_practices.md b/doc/development/testing_guide/best_practices.md index 2c8d488877b..82439c94c5a 100644 --- a/doc/development/testing_guide/best_practices.md +++ b/doc/development/testing_guide/best_practices.md @@ -1,5 +1,30 @@ # Testing best practices +## Test Design + +Testing at GitLab is a first class citizen, not an afterthought. It's important we consider the design of our tests +as we do the design of our features. + +When implementing a feature, we think about developing the right capabilities the right way, which helps us +narrow our scope to a manageable level. When implementing tests for a feature, we must think about developing +the right tests, but then cover _all_ the important ways the test may fail, which can quickly widen our scope to +a level that is difficult to manage. + +Test heuristics can help solve this problem. They concisely address many of the common ways bugs +manifest themselves within our code. When designing our tests, take time to review known test heuristics to inform +our test design. We can find some helpful heuristics documented in the Handbook in the +[Test Design](https://about.gitlab.com/handbook/engineering/quality/guidelines/test-engineering/test-design/) section. + +## Run tests against MySQL + +By default, tests are only run againts PostgreSQL, but you can run them on +demand against MySQL by following one of the following conventions: + +| Convention | Valid example | +|:----------------------|:-----------------------------| +| Include `mysql` in your branch name | `enhance-mysql-support` | +| Include `[run mysql]` in your commit message | `Fix MySQL support<br><br>[run mysql]` | + ## Test speed GitLab has a massive test suite that, without [parallelization], can take hours @@ -40,7 +65,7 @@ bundle exec rspec spec/[path]/[to]/[spec].rb to separate phases. - Use `Gitlab.config.gitlab.host` rather than hard coding `'localhost'` - Don't assert against the absolute value of a sequence-generated attribute (see - [Gotchas](../gotchas.md#dont-assert-against-the-absolute-value-of-a-sequence-generated-attribute)). + [Gotchas](../gotchas.md#do-not-assert-against-the-absolute-value-of-a-sequence-generated-attribute)). - Don't supply the `:each` argument to hooks since it's the default. - On `before` and `after` hooks, prefer it scoped to `:context` over `:all` - When using `evaluate_script("$('.js-foo').testSomething()")` (or `execute_script`) which acts on a given element, @@ -168,12 +193,13 @@ instead of 30+ seconds in case of a regular `spec_helper`. ### `let` variables -GitLab's RSpec suite has made extensive use of `let` variables to reduce -duplication. However, this sometimes [comes at the cost of clarity][lets-not], +GitLab's RSpec suite has made extensive use of `let`(along with it strict, non-lazy +version `let!`) variables to reduce duplication. However, this sometimes [comes at the cost of clarity][lets-not], so we need to set some guidelines for their use going forward: -- `let` variables are preferable to instance variables. Local variables are - preferable to `let` variables. +- `let!` variables are preferable to instance variables. `let` variables + are preferable to `let!` variables. Local variables are preferable to + `let` variables. - Use `let` to reduce duplication throughout an entire spec file. - Don't use `let` to define variables used by a single test; define them as local variables inside the test's `it` block. @@ -183,6 +209,9 @@ so we need to set some guidelines for their use going forward: - Try to avoid overriding the definition of one `let` variable with another. - Don't define a `let` variable that's only used by the definition of another. Use a helper method instead. +- `let!` variables should be used only in case if strict evaluation with defined + order is required, otherwise `let` will suffice. Remember that `let` is lazy and won't + be evaluated until it is referenced. [lets-not]: https://robots.thoughtbot.com/lets-not @@ -197,8 +226,10 @@ project, one project will do for the entire file. This can be achieved by using reloads or recreates the model, _only_ if needed. That is, when you changed properties or destroyed the object. -There is one gotcha; you can't reference a model defined in a `let` block in a -`set` block. +Note that you can't reference a model defined in a `let` block in a `set` block. + +Also, `set` is not supported in `:js` specs since those don't use transactions +to clean up database state after each example. ### Time-sensitive tests @@ -219,6 +250,36 @@ it 'is overdue' do end ``` +### Feature flags in tests + +All feature flags are stubbed to be enabled by default in our Ruby-based +tests. + +To disable a feature flag in a test, use the `stub_feature_flags` +helper. For example, to globally disable the `ci_live_trace` feature +flag in a test: + +```ruby +stub_feature_flags(ci_live_trace: false) + +Feature.enabled?(:ci_live_trace) # => false +``` + +If you wish to set up a test where a feature flag is disabled for some +actors and not others, you can specify this in options passed to the +helper. For example, to disable the `ci_live_trace` feature flag for a +specifc project: + +```ruby +project1, project2 = build_list(:project, 2) + +# Feature will only be disabled for project1 +stub_feature_flags(ci_live_trace: { enabled: false, thing: project1 }) + +Feature.enabled?(:ci_live_trace, project1) # => false +Feature.enabled?(:ci_live_trace, project2) # => true +``` + ### Pristine test environments The code exercised by a single GitLab test may access and modify many items of diff --git a/doc/development/testing_guide/ci.md b/doc/development/testing_guide/ci.md index 5aa668290b4..7a7fca46534 100644 --- a/doc/development/testing_guide/ci.md +++ b/doc/development/testing_guide/ci.md @@ -31,7 +31,11 @@ After that, the next pipeline will use the up-to-date The GitLab test suite is [monitored] for the `master` branch, and any branch that includes `rspec-profile` in their name. +A [public dashboard] is available for everyone to see. Feel free to look at the +slowest test files and try to improve them. + [monitored]: ../performance.md#rspec-profiling +[public dashboard]: https://redash.gitlab.com/public/dashboards/l1WhHXaxrCWM5Ai9D7YDqHKehq6OU3bx5gssaiWe?org_slug=default ## CI setup diff --git a/doc/development/testing_guide/end_to_end/best_practices.md b/doc/development/testing_guide/end_to_end/best_practices.md new file mode 100644 index 00000000000..89500ef9a90 --- /dev/null +++ b/doc/development/testing_guide/end_to_end/best_practices.md @@ -0,0 +1,38 @@ +# Best practices when writing end-to-end tests + +The majority of the end-to-end tests require some state to be built in the application for the tests to happen. + +A good example is a user being logged in as a pre-condition for testing the feature. + +But if the login feature is already covered with end-to-end tests through the GUI, there is no reason to perform such an expensive task to test the functionality of creating a project, or importing a repo, even if these features depend on a user being logged in. Let's see an example to make things clear. + +Let's say that, on average, the process to perform a successful login through the GUI takes 2 seconds. + +Now, realize that almost all tests need the user to be logged in, and that we need every test to run in isolation, meaning that tests cannot interfere with each other. This would mean that for every test the user needs to log in, and "waste 2 seconds". + +Now, multiply the number of tests per 2 seconds, and as your test suite grows, the time to run it grows with it, and this is not sustainable. + +An alternative to perform a login in a cheaper way would be having an endpoint (available only for testing) where we could pass the user's credentials as encrypted values as query strings, and then we would be redirected to the logged in home page if the credentials are valid. Let's say that, on average, this process takes only 200 miliseconds. + +You see the point right? + +Performing a login through the GUI for every test would cost a lot in terms of tests' execution. + +And there is another reason. + +Let's say that you don't follow the above suggestion, and depend on the GUI for the creation of every application state in order to test a specific feature. In this case we could be talking about the **Issues** feature, that depends on a project to exist, and the user to be logged in. + +What would happen if there was a bug in the project creation page, where the 'Create' button is disabled, not allowing for the creation of a project through the GUI, but the API logic is still working? + +In this case, instead of having only the project creation test failing, we would have many tests that depend on a project to be failing too. + +But, if we were following the best practices, only one test would be failing, and tests for other features that depend on a project to exist would continue to pass, since they could be creating the project behind the scenes interacting directly with the public APIs, ensuring a more reliable metric of test failure rate. + +Finally, interacting with the application only by its GUI generates a higher rate of test flakiness, and we want to avoid that at max. + +**The takeaways here are:** + +- Building state through the GUI is time consuming and it's not sustainable as the test suite grows. +- When depending only on the GUI to create the application's state and tests fail due to front-end issues, we can't rely on the test failures rate, and we generate a higher rate of test flakiness. + +Now that we are aware of all of it, [let's go create some tests](quick_start_guide.md). diff --git a/doc/development/testing_guide/end_to_end/dynamic_element_validation.md b/doc/development/testing_guide/end_to_end/dynamic_element_validation.md new file mode 100644 index 00000000000..f7b3ca8bc89 --- /dev/null +++ b/doc/development/testing_guide/end_to_end/dynamic_element_validation.md @@ -0,0 +1,113 @@ +# Dynamic Element Validation + +We devised a solution to solve common test automation problems such as the dreaded `NoSuchElementException`. + +Other problems that dynamic element validations solve are... + +- When we perform an action with the mouse, we expect something to occur. +- When our test is navigating to (or from) a page, we ensure that we are on the page we expect before +test continuation. + +## How it works + +We interpret user actions on the page to have some sort of effect. These actions are + +- [Navigation](#navigation) +- [Clicks](#clicks) + +### Navigation + +When a page is navigated to, there are elements that will always appear on the page unconditionally. + +Dynamic element validation is instituted when using + +```ruby +Runtime::Browser.visit(:gitlab, Some::Page) +``` + +### Clicks + +When we perform a click within our tests, we expect something to occur. That something could be a component to now +appear on the webpage, or the test to navigate away from the page entirely. + +Dynamic element validation is instituted when using + +```ruby +click_element :my_element, Some::Page +``` + +### Required Elements + +#### Definition + +First it is important to define what a "required element" is. + +Simply put, a required element is a visible HTML element that appears on a UI component without any user input. + +"Visible" can be defined as + +- Not having any CSS preventing its display. E.g.: `display: none` or `width: 0px; height: 0px;` +- Being able to be interacted with by the user + +"UI component" can be defined as + +- Anything the user sees +- A button, a text field +- A layer that sits atop the page + +#### Application + +Requiring elements is very easy. By adding `required: true` as a parameter to an `element`, you've now made it +a requirement that the element appear on the page upon navigation. + +## Examples + +Given ... + +```ruby +class MyPage < Page::Base + view 'app/views/view.html.haml' do + element :my_element, required: true + element :another_element, required: true + element :conditional_element + end + + def open_layer + click_element :my_element, Layer::MyLayer + end +end + +class Layer < Page::Component + view 'app/views/mylayer/layer.html.haml' do + element :message_content, required: true + end +end +``` + +### Navigating + +Given the [source](#examples) ... + +```ruby +Runtime::Browser.visit(:gitlab, Page::MyPage) + +execute_stuff +``` + +will invoke GitLab QA to scan `MyPage` for `my_element` and `another_element` to be on the page before continuing to +`execute_stuff` + +### Clicking + +Given the [source](#examples) ... + +```ruby +def open_layer + click_element :my_element, Layer::MyLayer +end +``` + +will invoke GitLab QA to ensure that `message_content` appears on +the Layer upon clicking `my_element`. + +This will imply that the Layer is indeed rendered before we continue our test. diff --git a/doc/development/testing_guide/end_to_end_tests.md b/doc/development/testing_guide/end_to_end/index.md index a239dc84a1c..afd81ff00b2 100644 --- a/doc/development/testing_guide/end_to_end_tests.md +++ b/doc/development/testing_guide/end_to_end/index.md @@ -7,6 +7,24 @@ as expected across the entire software stack and architecture, including integration of all micro-services and components that are supposed to work together. +## Branch naming + +If your contribution contains **only** changes under the +[`qa/` folder](https://gitlab.com/gitlab-org/gitlab-ce/tree/master/qa), you can +speed up the CI process by following some branch naming conventions. You have +three choices: + +| Branch name | Valid example | +|:----------------------|:-----------------------------| +| Starting with `qa/` | `qa/new-oauth-login-test` | +| Starting with `qa-` | `qa-new-oauth-login-test` | +| Ending in `-qa` | `123-new-oauth-login-test-qa` | + +If your branch name matches any of the above, it will run only the QA-related +jobs. +If it does not, the whole application test suite will run (including QA-related +jobs). + ## How do we test GitLab? We use [Omnibus GitLab][omnibus-gitlab] to build GitLab packages and then we @@ -17,18 +35,25 @@ a black-box testing framework for the API and the UI. We run scheduled pipeline each night to test nightly builds created by Omnibus. You can find these nightly pipelines at [gitlab-org/quality/nightly/pipelines][quality-nightly-pipelines]. +Results are reported in the `#qa-nightly` Slack channel. ### Testing staging We run scheduled pipeline each night to test staging. You can find these nightly pipelines at [gitlab-org/quality/staging/pipelines][quality-staging-pipelines]. +Results are reported in the `#qa-staging` Slack channel. ### Testing code in merge requests -It is possible to run end-to-end tests (eventually being run within a -[GitLab QA pipeline][gitlab-qa-pipelines]) for a merge request by triggering -the `package-and-qa` manual action in the `test` stage, that should be present -in a merge request widget (unless the merge request is from a fork). +#### Using the `package-and-qa` job + +It is possible to run end-to-end tests for a merge request, eventually being run in +a pipeline in the [`gitlab-qa`](https://gitlab.com/gitlab-org/gitlab-qa/) project, +by triggering the `package-and-qa` manual action in the `test` stage (not +available for forks). + +**This runs end-to-end tests against a custom Omnibus package built from your +merge request's changes.** Manual action that starts end-to-end tests is also available in merge requests in [Omnibus GitLab][omnibus-gitlab]. @@ -40,6 +65,29 @@ Below you can read more about how to use it and how does it work. Currently, we are using _multi-project pipeline_-like approach to run QA pipelines. + + +<details> +<summary>Show mermaid source</summary> +<pre> +graph LR + A1 -.->|1. Triggers an omnibus-gitlab pipeline and wait for it to be done| A2 + B2[<b>`Trigger-qa` stage</b><br />`Trigger:qa-test` job] -.->|2. Triggers a gitlab-qa pipeline and wait for it to be done| A3 + +subgraph gitlab-ce/ee pipeline + A1[<b>`test` stage</b><br />`package-and-qa` job] + end + +subgraph omnibus-gitlab pipeline + A2[<b>`Trigger-docker` stage</b></b><br />`Trigger:gitlab-docker` job] -->|once done| B2 + end + +subgraph gitlab-qa pipeline + A3>QA jobs run] -.->|3. Reports back the pipeline result to the `package-and-qa` job<br />and post the result on the original commit tested| A1 + end +</pre> +</details> + 1. Developer triggers a manual action, that can be found in CE / EE merge requests. This starts a chain of pipelines in multiple projects. @@ -51,16 +99,34 @@ pipelines. 1. When packages are ready, and available in the registry, a final step in the [Omnibus GitLab][omnibus-gitlab] pipeline, triggers a new - [GitLab QA pipeline][gitlab-qa-pipelines]. It also waits for a resulting status. + GitLab QA pipeline (those with access can view them at `https://gitlab.com/gitlab-org/gitlab-qa/pipelines`). It also waits for a resulting status. 1. GitLab QA pulls images from the registry, spins-up containers and runs tests against a test environment that has been just orchestrated by the `gitlab-qa` tool. -1. The result of the [GitLab QA pipeline][gitlab-qa-pipelines] is being +1. The result of the GitLab QA pipeline is being propagated upstream, through Omnibus, back to the CE / EE merge request. -#### How do I write tests? +#### Using the `review-qa-all` jobs + +On every pipeline during the `test` stage, the `review-qa-smoke` job is +automatically started: it runs the QA smoke suite against the +[Review App][review-apps]. + +You can also manually start the `review-qa-all`: it runs the full QA suite +against the [Review App][review-apps]. + +**This runs end-to-end tests against a Review App based on [the official GitLab +Helm chart][helm-chart], itself deployed with custom +[Cloud Native components][cng] built from your merge request's changes.** + +See [Review Apps][review-apps] for more details about Review Apps. + +[helm-chart]: https://gitlab.com/charts/gitlab/ +[cng]: https://gitlab.com/gitlab-org/build/CNG + +## How do I write tests? In order to write new tests, you first need to learn more about GitLab QA architecture. See the [documentation about it][gitlab-qa-architecture]. @@ -70,6 +136,12 @@ Once you decided where to put [test environment orchestration scenarios] and the [GitLab QA orchestrator README][gitlab-qa-readme], and [the already existing instance-level scenarios][instance-level scenarios]. +Continued reading: + +- [Quick Start Guide](quick_start_guide.md) +- [Style Guide](style_guide.md) +- [Best Practices](best_practices.md) + ## Where can I ask for help? You can ask question in the `#quality` channel on Slack (GitLab internal) or @@ -81,9 +153,9 @@ you can find an issue you would like to work on in [omnibus-gitlab]: https://gitlab.com/gitlab-org/omnibus-gitlab [gitlab-qa]: https://gitlab.com/gitlab-org/gitlab-qa [gitlab-qa-readme]: https://gitlab.com/gitlab-org/gitlab-qa/tree/master/README.md -[gitlab-qa-pipelines]: https://gitlab.com/gitlab-org/gitlab-qa/pipelines [quality-nightly-pipelines]: https://gitlab.com/gitlab-org/quality/nightly/pipelines [quality-staging-pipelines]: https://gitlab.com/gitlab-org/quality/staging/pipelines +[review-apps]: ../review_apps.md [gitlab-qa-architecture]: https://gitlab.com/gitlab-org/gitlab-qa/blob/master/docs/architecture.md [gitlab-qa-issues]: https://gitlab.com/gitlab-org/gitlab-qa/issues?label_name%5B%5D=new+scenario [gitlab-ce-issues]: https://gitlab.com/gitlab-org/gitlab-ce/issues?label_name[]=QA&label_name[]=test diff --git a/doc/development/testing_guide/end_to_end/page_objects.md b/doc/development/testing_guide/end_to_end/page_objects.md new file mode 100644 index 00000000000..73e1fd862c1 --- /dev/null +++ b/doc/development/testing_guide/end_to_end/page_objects.md @@ -0,0 +1,167 @@ +# Page objects in GitLab QA + +In GitLab QA we are using a known pattern, called _Page Objects_. + +This means that we have built an abstraction for all GitLab pages that we use +to drive GitLab QA scenarios. Whenever we do something on a page, like filling +in a form, or clicking a button, we do that only through a page object +associated with this area of GitLab. + +For example, when GitLab QA test harness signs in into GitLab, it needs to fill +in a user login and user password. In order to do that, we have a class, called +`Page::Main::Login` and `sign_in_using_credentials` methods, that is the only +piece of the code, that has knowledge about `user_login` and `user_password` +fields. + +## Why do we need that? + +We need page objects, because we need to reduce duplication and avoid problems +whenever someone changes some selectors in GitLab's source code. + +Imagine that we have a hundred specs in GitLab QA, and we need to sign into +GitLab each time, before we make assertions. Without a page object one would +need to rely on volatile helpers or invoke Capybara methods directly. Imagine +invoking `fill_in :user_login` in every `*_spec.rb` file / test example. + +When someone later changes `t.text_field :login` in the view associated with +this page to `t.text_field :username` it will generate a different field +identifier, what would effectively break all tests. + +Because we are using `Page::Main::Login.act { sign_in_using_credentials }` +everywhere, when we want to sign into GitLab, the page object is the single +source of truth, and we will need to update `fill_in :user_login` +to `fill_in :user_username` only in a one place. + +## What problems did we have in the past? + +We do not run QA tests for every commit, because of performance reasons, and +the time it would take to build packages and test everything. + +That is why when someone changes `t.text_field :login` to +`t.text_field :username` in the _new session_ view we won't know about this +change until our GitLab QA nightly pipeline fails, or until someone triggers +`package-and-qa` action in their merge request. + +Obviously such a change would break all tests. We call this problem a _fragile +tests problem_. + +In order to make GitLab QA more reliable and robust, we had to solve this +problem by introducing coupling between GitLab CE / EE views and GitLab QA. + +## How did we solve fragile tests problem? + +Currently, when you add a new `Page::Base` derived class, you will also need to +define all selectors that your page objects depends on. + +Whenever you push your code to CE / EE repository, `qa:selectors` sanity test +job is going to be run as a part of a CI pipeline. + +This test is going to validate all page objects that we have implemented in +`qa/page` directory. When it fails, you will be notified about missing +or invalid views / selectors definition. + +## How to properly implement a page object? + +We have built a DSL to define coupling between a page object and GitLab views +it is actually implemented by. See an example below. + +```ruby +module Page + module Main + class Login < Page::Base + view 'app/views/devise/passwords/edit.html.haml' do + element :password_field + element :password_confirmation + element :change_password_button + end + + view 'app/views/devise/sessions/_new_base.html.haml' do + element :login_field + element :password_field + element :sign_in_button + end + + # ... + end + end +end +``` + +### Defining Elements + +The `view` DSL method will correspond to the rails View, partial, or vue component that renders the elements. + +The `element` DSL method in turn declares an element for which a corresponding +`qa-element-name-dasherized` CSS class will need to be added to the view file. + +You can also define a value (String or Regexp) to match to the actual view +code but **this is deprecated** in favor of the above method for two reasons: + +- Consistency: there is only one way to define an element +- Separation of concerns: QA uses dedicated CSS classes instead of reusing code + or classes used by other components (e.g. `js-*` classes etc.) + +```ruby +view 'app/views/my/view.html.haml' do + # Implicitly require `.qa-logout-button` CSS class to be present in the view + element :logout_button + + ## This is deprecated and forbidden by the `QA/ElementWithPattern` RuboCop cop. + # Require `f.submit "Sign in"` to be present in `my/view.html.haml + element :my_button, 'f.submit "Sign in"' # rubocop:disable QA/ElementWithPattern + + ## This is deprecated and forbidden by the `QA/ElementWithPattern` RuboCop cop. + # Match every line in `my/view.html.haml` against + # `/link_to .* "My Profile"/` regexp. + element :profile_link, /link_to .* "My Profile"/ # rubocop:disable QA/ElementWithPattern +end +``` + +### Adding Elements to a View + +Given the following elements... + +```ruby +view 'app/views/my/view.html.haml' do + element :login_field + element :password_field + element :sign_in_button +end +``` + +To add these elements to the view, you must change the rails View, partial, or vue component by adding a `qa-element-descriptor` class +for each element defined. + +In our case, `qa-login-field`, `qa-password-field` and `qa-sign-in-button` + +**app/views/my/view.html.haml** + +```haml += f.text_field :login, class: "form-control top qa-login-field", autofocus: "autofocus", autocapitalize: "off", autocorrect: "off", required: true, title: "This field is required." += f.password_field :password, class: "form-control bottom qa-password-field", required: true, title: "This field is required." += f.submit "Sign in", class: "btn btn-success qa-sign-in-button" +``` + +Things to note: + +- The CSS class must be `kebab-cased` (separated with hyphens "`-`") +- If the element appears on the page unconditionally, add `required: true` to the element. See +[Dynamic element validation](dynamic_element_validation.md) + +## Running the test locally + +During development, you can run the `qa:selectors` test by running + +```shell +bin/qa Test::Sanity::Selectors +``` + +from within the `qa` directory. + +## Where to ask for help? + +If you need more information, ask for help on `#quality` channel on Slack +(internal, GitLab Team only). + +If you are not a Team Member, and you still need help to contribute, please +open an issue in GitLab CE issue tracker with the `~QA` label. diff --git a/doc/development/testing_guide/end_to_end/quick_start_guide.md b/doc/development/testing_guide/end_to_end/quick_start_guide.md new file mode 100644 index 00000000000..afe76acf9c9 --- /dev/null +++ b/doc/development/testing_guide/end_to_end/quick_start_guide.md @@ -0,0 +1,585 @@ +# Writing end-to-end tests step-by-step + +In this tutorial, you will find different examples, and the steps involved, in the creation of end-to-end (_e2e_) tests for GitLab CE and GitLab EE, using GitLab QA. + +> When referring to end-to-end tests in this document, this means testing a specific feature end-to-end, such as a user logging in, the creation of a project, the management of labels, breaking down epics into sub-epics and issues, etc. + +## Important information before we start writing tests + +It's important to understand that end-to-end tests of isolated features, such as the ones described in the above note, doesn't mean that everything needs to happen through the GUI. + +If you don't exactly understand what we mean by **not everything needs to happen through the GUI,** please make sure you've read the [best practices](best_practices.md) before moving on. + +## This document covers the following items: + +- [0.](#0-are-end-to-end-tests-needed) Identifying if end-to-end tests are really needed +- [1.](#1-identifying-the-devops-stage) Identifying the [DevOps stage](https://about.gitlab.com/stages-devops-lifecycle/) of the feature that you are going to cover with end-to-end tests +- [2.](#2-test-skeleton) Creating the skeleton of the test file (`*_spec.rb`) +- [3.](#3-test-cases-mvc) The [MVC](https://about.gitlab.com/handbook/values/#minimum-viable-change-mvc) of the test cases' logic +- [4.](#4-extracting-duplicated-code) Extracting duplicated code into methods +- [5.](#5-tests-pre-conditions-using-resources-and-page-objects) Tests' pre-conditions (`before :context` and `before`) using resources and [Page Objects] +- [6.](#6-optimization) Optimizing the test suite +- [7.](#7-resources) Using and implementing resources +- [8.](#8-page-objects) Moving element definitions and methods to [Page Objects] + +### 0. Are end-to-end tests needed? + +At GitLab we respect the [test pyramid](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/doc/development/testing_guide/testing_levels.md), and so, we recommend you check the code coverage of a specific feature before writing end-to-end tests, for both [CE](https://gitlab-org.gitlab.io/gitlab-ce/coverage-ruby/#_AllFiles) and [EE](https://gitlab-org.gitlab.io/gitlab-ee/coverage-ruby/#_AllFiles) projects. + +Sometimes you may notice that there is already good coverage in other test levels, and we can stay confident that if we break a feature, we will still have quick feedback about it, even without having end-to-end tests. + +If after this analysis you still think that end-to-end tests are needed, keep reading. + +### 1. Identifying the DevOps stage + +The GitLab QA end-to-end tests are organized by the different [stages in the DevOps lifecycle](https://gitlab.com/gitlab-org/gitlab-ce/tree/master/qa/qa/specs/features/browser_ui), and so, if you are creating tests for issue creation, for instance, you would locate the spec files under the `qa/qa/specs/features/browser_ui/2_plan/` directory since issue creation is part of the Plan stage. + + In another case of a test for listing merged merge requests (MRs), the test should go under the `qa/qa/specs/features/browser_ui/3_create/` directory since merge requests are a feature from the Create stage. + +> There may be sub-directories inside the stages directories, for different features. For example: `.../browser_ui/2_plan/ee_epics/` and `.../browser_ui/2_plan/issues/`. + +Now, let's say we want to create tests for the [scoped labels](https://about.gitlab.com/2019/04/22/gitlab-11-10-released/#scoped-labels) feature, available on GitLab EE Premium (this feature is part of the Plan stage.) + +> Because these tests are for a feature available only on GitLab EE, we need to create them in the [EE repository](https://gitlab.com/gitlab-org/gitlab-ee). + +Since [there is no specific directory for this feature](https://gitlab.com/gitlab-org/gitlab-ee/tree/master/qa/qa/specs/features/browser_ui/2_plan), we should create a sub-directory for it. + +Under `.../browser_ui/2_plan/`, let's create a sub-directory called `ee_scoped_labels/`. + +> Notice that since this feature is only available for GitLab EE we prefix the sub-directory with `ee_`. + +### 2. Test skeleton + +Inside the newly created sub-directory, let's create a file describing the test suite (e.g. `editing_scoped_labels_spec.rb`.) + +#### The `context` and `describe` blocks + +Specs have an outer `context` that indicates the DevOps stage. The next level is the `describe` block, that briefly states the subject of the test suite. See the following example: + +```ruby +module QA + context 'Plan' do + describe 'Editing scoped labels on issues' do + end + end +end +``` + +#### The `it` blocks + +Every test suite is composed of at least one `it` block, and a good way to start writing end-to-end tests is by writing test cases descriptions as `it` blocks. These might help you to think of different test scenarios. Take a look at the following example: + +```ruby +module QA + context 'Plan' do + describe 'Editing scoped labels on issues' do + it 'replaces an existing label if it has the same key' do + end + + it 'keeps both scoped labels when adding a label with a different key' do + end + end + end +end +``` + +### 3. Test cases MVC + +For the [MVC](https://about.gitlab.com/handbook/values/#minimum-viable-change-mvc) of our test cases, let's say that we already have the application in the state needed for the tests, and then let's focus on the logic of the test cases only. + +To evolve the test cases drafted on step 2, let's imagine that the user is already logged into a GitLab EE instance, they already have at least a Premium license in use, there is already a project created, there is already an issue opened in the project, the issue already has a scoped label (e.g. `animal::fox`), there are other scoped labels (for the same scope and for a different scope (e.g. `animal::dolphin` and `plant::orchid`), and finally, the user is already on the issue's page. Let's also suppose that for every test case the application is in a clean state, meaning that one test case won't affect another. + +> Note: there are different approaches to creating an application state for end-to-end tests. Some of them are very time consuming and subject to failures, such as when using the GUI for all the pre-conditions of the tests. On the other hand, other approaches are more efficient, such as using the public APIs. The latter is more efficient since it doesn't depend on the GUI. We won't focus on this part yet, but it's good to keep it in mind. + +Let's now focus on the first test case. + +```ruby +it 'replaces an existing label if it has the same key' do + # This implementation is only for tutorial purposes. We normally encapsulate elements in Page Objects (which we cover on section 8). + page.find('.block.labels .edit-link').click + page.find('.dropdown-menu-labels .dropdown-input-field').send_keys ['animal::dolphin', :enter] + page.find('#content-body').click + page.refresh + + labels_block = page.find('.qa-labels-block') + + expect(labels_block).to have_content('animal::dolphin') + expect(labels_block).not_to have_content('animal::fox') + expect(page).to have_content('added animal::dolphin label and removed animal::fox') +end +``` + +> Notice that the test itself is simple. The most challenging part is the creation of the application state, which will be covered later. + +> The exemplified test case's MVC is not enough for the change to be merged, but it helps to build up the test logic. The reason is that we do not want to use locators directly in the tests, and tests **must** use [Page Objects] before they can be merged. This way we better separate the responsibilities, where the Page Objects encapsulate elements and methods that allow us to interact with pages, while the spec files describe the test cases in more business-related language. + +Below are the steps that the test covers: + +1. The test finds the 'Edit' link for the labels and clicks on it. +2. Then it fills in the 'Assign labels' input field with the value 'animal::dolphin' and press enters. +3. Then it clicks in the content body to apply the label and refreshes the page. +4. Finally, the expectations check that the previous scoped label was removed and that the new one was added. + +Let's now see how the second test case would look. + +```ruby +it 'keeps both scoped labels when adding a label with a different key' do + # This implementation is only for tutorial purposes. We normally encapsulate elements in Page Objects (which we cover on section 8). + page.find('.block.labels .edit-link').click + page.find('.dropdown-menu-labels .dropdown-input-field').send_keys ['plant::orchid', :enter] + page.find('#content-body').click + page.refresh + + labels_block = page.find('.qa-labels-block') + + expect(labels_block).to have_content('animal::fox') + expect(labels_block).to have_content('plant::orchid') + expect(page).to have_content('added animal::fox') + expect(page).to have_content('added plant::orchid') +end +``` + +> Note that elements are always located using CSS selectors, and a good practice is to add test-specific selectors (this is called adding testability to the application and we will talk more about it later.) For example, the `labels_block` element uses the selector `.qa-labels-block`, which was added specifically for testing purposes. + +Below are the steps that the test covers: + +1. The test finds the 'Edit' link for the labels and clicks on it. +2. Then it fills in the 'Assign labels' input field with the value 'plant::orchid' and press enters. +3. Then it clicks in the content body to apply the label and refreshes the page. +4. Finally, the expectations check that both scoped labels are present. + +> Similar to the previous test, this one is also very straightforward, but there is some code duplication. Let's address it. + +### 4. Extracting duplicated code + +If we refactor the tests created on step 3 we could come up with something like this: + +```ruby +before do + ... + + @initial_label = 'animal::fox' + @new_label_same_scope = 'animal::dolphin' + @new_label_different_scope = 'plant::orchid' + + ... +end + +it 'replaces an existing label if it has the same key' do + select_label_and_refresh @new_label_same_scope + + labels_block = page.find('.qa-labels-block') + + expect(labels_block).to have_content(@new_label_same_scope) + expect(labels_block).not_to have_content(@initial_label) + expect(page).to have_content("added #{@new_label_same_scope}") + expect(page).to have_content("and removed #{@initial_label}") +end + +it 'keeps both scoped label when adding a label with a different key' do + select_label_and_refresh @new_label_different_scope + + labels_block = page.find('.qa-labels-block') + + expect(labels_blocks).to have_content(@new_label_different_scope) + expect(labels_blocks).to have_content(@initial_label) + expect(page).to have_content("added #{@new_label_different_scope}") + expect(page).to have_content("added #{@initial_label}") +end + +def select_label_and_refresh(label) + page.find('.block.labels .edit-link').click + page.find('.dropdown-menu-labels .dropdown-input-field').send_keys [label, :enter] + page.find('#content-body').click + page.refresh +end +``` + +First, we remove the duplication of strings by defining the global variables `@initial_label`, `@new_label_same_scope` and `@new_label_different_scope` in the `before` block, and by using them in the expectations. + +Then, by creating a reusable `select_label_and_refresh` method, we remove the code duplication of this action, and later we can move this method to a Page Object class that will be created for easier maintenance purposes. + +> Notice that the reusable method is created at the bottom of the file. The reason for that is that reading the code should be similar to reading a newspaper, where high-level information is at the top, like the title and summary of the news, while low level, or more specific information, is at the bottom (this helps readability). + +### 5. Tests' pre-conditions using resources and Page Objects + +In this section, we will address the previously mentioned subject of creating the application state for the tests, using the `before :context` and `before` blocks, together with resources and Page Objects. + +#### `before :context` + +A pre-condition for the entire test suite is defined in the `before :context` block. + +> For our test suite, due to the need of the tests being completely independent of each other, we won't use the `before :context` block. The `before :context` block would make the tests dependent on each other because the first test changes the label of the issue, and the second one depends on the `'animal::fox'` label being set. + +> **Tip:** In case of a test suite with only one `it` block it's ok to use only the `before` block (see below) with all the test's pre-conditions. + +#### `before` + +As the pre-conditions for our test suite, the things that needs to happen before each test starts are: + +- The user logging in; +- A premium license already being set; +- A project being created with an issue and labels already set; +- The issue page being opened with only one scoped label applied to the it. + +> When running end-to-end tests as part of the GitLab's continuous integration process [a license is already set as an environment variable](https://gitlab.com/gitlab-org/gitlab-ee/blob/1a60d926740db10e3b5724713285780a4f470531/qa/qa/ee/strategy.rb#L20). For running tests locally you can set up such license by following the document [what tests can be run?](https://gitlab.com/gitlab-org/gitlab-qa/blob/master/docs/what_tests_can_be_run.md#supported-remote-grid-environment-variables), based on the [supported GitLab environment variables](https://gitlab.com/gitlab-org/gitlab-qa/blob/master/docs/what_tests_can_be_run.md#supported-gitlab-environment-variables). + +#### Implementation + +In the following code we will focus only on the test suite's pre-conditions: + +```ruby +module QA + context 'Plan' do + describe 'Editing scoped labels on issues' do + before do + Runtime::Browser.visit(:gitlab, Page::Main::Login) + Page::Main::Login.perform(&:sign_in_using_credentials) + + @initial_label = 'animal::fox' + @new_label_same_scope = 'animal::dolphin' + @new_label_different_scope = 'plant::orchid' + + issue = Resource::Issue.fabricate_via_api! do |issue| + issue.title = 'Issue to test the scoped labels' + issue.labels = @initial_label + end + + [@new_label_same_scope, @new_label_different_scope].each do |label| + Resource::Label.fabricate_via_api! do |l| + l.project = issue.project.id + l.title = label + end + end + + issue.visit! + end + + it 'replaces an existing label if it has the same key' do + ... + end + + it 'keeps both scoped labels when adding a label with a different key' do + ... + end + + def select_label_and_refresh(label) + ... + end + end + end +end +``` + +In the `before` block we create all the application state needed for the tests to run. We do that by using the `Runtime::Browser.visit` method to go to the login page, by performing a `sign_in_using_credentials` from the `Login` Page Object, by fabricating resources via APIs (`issue`, and `Resource::Label`), and by using the `issue.visit!` to visit the issue page. + +> A project is created in the background by creating the `issue` resource. + +> When creating the [Resources], notice that when calling the `fabricate_via_api` method, we pass some attribute:values, like `title`, and `labels` for the `issue` resource; and `project` and `title` for the `label` resource. + +> What's important to understand here is that by creating the application state mostly using the public APIs we save a lot of time in the test suite setup stage. + +> Soon we will cover the use of the already existing resources' methods and the creation of your own `fabricate_via_api` methods for resources where this is still not available, but first, let's optimize our implementation. + +### 6. Optimization + +As already mentioned in the [best practices](best_practices.md) document, end-to-end tests are very costly in terms of execution time, and it's our responsibility as software engineers to ensure that we optimize them as much as possible. + +> Note that end-to-end tests are slow to run and so they can have several actions and assertions in a single test, which helps us get feedback from the tests sooner. In comparison, unit tests are much faster to run and can exercise every little piece of the application in isolation, and so they usually have only one assertion per test. + +Some improvements that we could make in our test suite to optimize its time to run are: + +1. Having a single test case (an `it` block) that exercises both scenarios to avoid "wasting" time in the tests' pre-conditions, instead of having two different test cases. +2. Making the selection of labels more performant by allowing for the selection of more than one label in the same reusable method. + +Let's look at a suggestion that addresses the above points, one by one: + +```ruby +module QA + context 'Plan' do + describe 'Editing scoped labels on issues' do + before do + ... + end + + it 'correctly applies scoped labels depending on if they are from the same or a different scope' do + select_labels_and_refresh [@new_label_same_scope, @new_label_different_scope] + + labels_block = page.all('.qa-labels-block') + + expect(labels_block).to have_content(@new_label_same_scope) + expect(labels_block).to have_content(@new_label_different_scope) + expect(labels_block).not_to have_content(@initial_label) + expect(page).to have_content("added #{@initial_label}") + expect(page).to have_content("added #{@new_label_same_scope} #{@new_label_different_scope} labels and removed #{@initial_label}") + end + + def select_labels_and_refresh(labels) + find('.block.labels .edit-link').click + labels.each do |label| + find('.dropdown-menu-labels .dropdown-input-field').send_keys [label, :enter] + end + find('#content-body').click + refresh + end + end + end + end +``` + +To address point 1, we changed the test implementation from two `it` blocks into a single one that exercises both scenarios. Now the new test description is: `'correctly applies the scoped labels depending if they are from the same or a different scope'`. It's a long description, but it describes well what the test does. + +> Notice that the implementation of the new and unique `it` block had to change a little bit. Below we describe in details what it does. + +1. It selects two scoped labels simultaneously, one from the same scope of the one already applied in the issue during the setup phase (in the `before` block), and another one from a different scope. +2. It asserts that the correct labels are visible in the `labels_block`, and that the labels were correctly added and removed; +3. Finally, the `select_label_and_refresh` method is changed to `select_labels_and_refresh`, which accepts an array of labels instead of a single label, and it iterates on them for faster label selection (this is what is used in step 1 explained above.) + +### 7. Resources + +**Note:** When writing this document, some code that is now merged to master was not implemented yet, but we left them here for the readers to understand the whole process of end-to-end test creation. + +You can think of [Resources] as anything that can be created on GitLab CE or EE, either through the GUI, the API, or the CLI. + +With that in mind, resources can be a project, an epic, an issue, a label, a commit, etc. + +As you saw in the tests' pre-conditions and the optimization sections, we're already creating some of these resources, and we are doing that by calling the `fabricate_via_api!` method. + +> We could be using the `fabricate!` method instead, which would use the `fabricate_via_api!` method if it exists, and fallback to GUI fabrication otherwise, but we recommend being explicit to make it clear what the test does. Also, we always recommend fabricating resources via API since this makes tests faster and more reliable. + +For our test suite example, the resources that we need to create don't have the necessary code for the `fabricate_via_api!` method to correctly work (e.g., the issue and label resources), so we will have to create them. + +#### Implementation + +In the following we describe the changes needed in each of the resource files mentioned above. + +**Issue resource** + +Now, let's make it possible to create an issue resource through the API. + +First, in the [issue resource](https://gitlab.com/gitlab-org/gitlab-ee/blob/d3584e80b4236acdf393d815d604801573af72cc/qa/qa/resource/issue.rb), let's expose its labels attribute. + +Add the following `attribute :labels` right below the [`attribute :title`](https://gitlab.com/gitlab-org/gitlab-ee/blob/d3584e80b4236acdf393d815d604801573af72cc/qa/qa/resource/issue.rb#L15). + +> This line is needed to allow for labels to be automatically added to an issue when fabricating it via API. + +Next, add the following code right below the [`fabricate!`](https://gitlab.com/gitlab-org/gitlab-ee/blob/d3584e80b4236acdf393d815d604801573af72cc/qa/qa/resource/issue.rb#L27) method. + +```ruby +def api_get_path + "/projects/#{project.id}/issues/#{id}" +end + +def api_post_path + "/projects/#{project.id}/issues" +end + +def api_post_body + { + title: title, + labels: [labels] + } +end +``` + +By defining the `api_get_path` method, we allow the [`ApiFabricator`](https://gitlab.com/gitlab-org/gitlab-ee/blob/master/qa/qa/resource/api_fabricator.rb) module to know which path to use to get a single issue. + +> This `GET` path can be found in the [public API documentation](https://docs.gitlab.com/ee/api/issues.html#single-issue). + +By defining the `api_post_path` method, we allow the [`ApiFabricator`](https://gitlab.com/gitlab-org/gitlab-ee/blob/master/qa/qa/resource/api_fabricator.rb) module to know which path to use to create a new issue in a specific project. + +> This `POST` path can be found in the [public API documentation](https://docs.gitlab.com/ee/api/issues.html#new-issue). + +By defining the `api_post_body` method, we allow the [`ApiFabricator.api_post`](https://gitlab.com/gitlab-org/gitlab-ee/blob/a9177ca1812bac57e2b2fa4560e1d5dd8ffac38b/qa/qa/resource/api_fabricator.rb#L68) method to know which data to send when making the `POST` request. + +> Notice that we pass both `title` and `labels` attributes in the `api_post_body`, where `labels` receives an array of labels, and [`title` is required](https://docs.gitlab.com/ee/api/issues.html#new-issue). + +**Label resource** + +Finally, let's make it possible to create label resources through the API. + +Add the following code right below the [`fabricate!`](https://gitlab.com/gitlab-org/gitlab-ee/blob/a9177ca1812bac57e2b2fa4560e1d5dd8ffac38b/qa/qa/resource/label.rb#L36) method. + +```ruby +def resource_web_url(resource) + super +rescue ResourceURLMissingError + # this particular resource does not expose a web_url property +end + +def api_get_path + raise NotImplementedError, "The Labels API doesn't expose a single-resource endpoint so this method cannot be properly implemented." +end + +def api_post_path + "/projects/#{project}/labels" +end + +def api_post_body + { + name: @title, + color: @color + } +end +``` + +By defining the `resource_web_url(resource)` method, we override the one from the [`ApiFabricator`](https://gitlab.com/gitlab-org/gitlab-ee/blob/master/qa/qa/resource/api_fabricator.rb#L44) module. We do that to avoid failing the test due to this particular resource not exposing a `web_url` property. + +By defining the `api_get_path` method, we **would** allow for the [`ApiFabricator`](https://gitlab.com/gitlab-org/gitlab-ee/blob/master/qa/qa/resource/api_fabricator.rb) module to know which path to use to get a single label, but since there's no path available for that in the publich API, we raise a `NotImplementedError` instead. + +By defining the `api_post_path` method, we allow for the [`ApiFabricator `](https://gitlab.com/gitlab-org/gitlab-ee/blob/master/qa/qa/resource/api_fabricator.rb) module to know which path to use to create a new label in a specific project. + +By defining the `api_post_body` method, we we allow for the [`ApiFabricator.api_post`](https://gitlab.com/gitlab-org/gitlab-ee/blob/a9177ca1812bac57e2b2fa4560e1d5dd8ffac38b/qa/qa/resource/api_fabricator.rb#L68) method to know which data to send when making the `POST` request. + +> Notice that we pass both `name` and `color` attributes in the `api_post_body` since [those are required](https://docs.gitlab.com/ee/api/labels.html#create-a-new-label). + +### 8. Page Objects + +Page Objects are used in end-to-end tests for maintenance reasons, where a page's elements and methods are defined to be reused in any test. + +> Page Objects are auto-loaded in the `qa/qa.rb` file and available in all the test files (`*_spec.rb`). + +Take a look at the [Page Objects] documentation. + +Now, let's go back to our example. + +As you may have noticed, we are defining elements with CSS selectors and the `select_labels_and_refresh` method directly in the test file, and this is an anti-pattern since we need to better separate the responsibilities. + +To address this issue, we will move the implementation to Page Objects, and the test suite will only focus on the business rules that we are testing. + +#### Updates in the test file + +As in a test-driven development approach, let's start changing the test file even before the Page Object implementation is in place. + +Replace the code of the `it` block in the test file by the following: + +```ruby +module QA + context 'Plan' do + describe 'Editing scoped labels on issues' do + before do + ... + end + + it 'correctly applies scoped labels depending on if they are from the same or a different scope' do + Page::Project::Issue::Show.perform do |issue_page| + issue_page.select_labels_and_refresh [@new_label_same_scope, @new_label_different_scope] + + expect(page).to have_content("added #{@initial_label}") + expect(page).to have_content("added #{@new_label_same_scope} #{@new_label_different_scope} labels and removed #{@initial_label}") + expect(issue_page.text_of_labels_block).to have_content(@new_label_same_scope) + expect(issue_page.text_of_labels_block).to have_content(@new_label_different_scope) + expect(issue_page.text_of_labels_block).not_to have_content(@initial_label) + end + end + end + end +end +``` + +Notice that `select_labels_and_refresh` is now a method from the issue Page Object (which is not yet implemented), and that we verify the labels' text by using `text_of_labels_block`, instead of via the `labels_block` element. The `text_of_labels_block` method will also be implemented in the issue Page Object. + +Let's now update the Issue Page Object. + +#### Updates in the Issue Page Object + +> Page Objects are located in the `qa/qa/page/` directory, and its sub-directories. + +The file we will have to change is the [Issue Page Object](https://gitlab.com/gitlab-org/gitlab-ee/blob/master/qa/qa/page/project/issue/show.rb). + +First, add the following code right below the definition of an already implemented view: + +```ruby +view 'app/views/shared/issuable/_sidebar.html.haml' do + element :labels_block + element :edit_link_labels + element :dropdown_menu_labels +end + +view 'app/helpers/dropdowns_helper.rb' do + element :dropdown_input_field +end +``` + +Similarly to what we did before, let's first change the Page Object even without the elements being defined in the view (`_sidebar.html.haml`) and the `dropdowns_helper.rb` files, and later we will update them by adding the appropriate CSS selectors. + +Now, let's implement the methods `select_labels_and_refresh` and `text_of_labels_block`. + +Somewhere between the definition of the views and the private methods, add the following snippet of code: + +```ruby +def select_labels_and_refresh(labels) + click_element(:edit_link_labels) + labels.each do |label| + within_element(:dropdown_menu_labels, text: label) do + send_keys_to_element(:dropdown_input_field, [label, :enter]) + end + end + click_body + labels.each do |label| + has_element?(:labels_block, text: label) + end + refresh +end + +def text_of_labels_block + find_element(:labels_block) +end +``` + +##### Details of `select_labels_and_refresh` + +Notice that we have not only moved the `select_labels_and_refresh` method, but we have also changed its implementation to: +1. Click the `:edit_link_labels` element previously defined, instead of using `find('.block.labels .edit-link').click` +2. Use `within_element(:dropdown_menu_labels, text: label)`, and inside of it, we call `send_keys_to_element(:dropdown_input_field, [label, :enter])`, which is a method that we will implement in the `QA::Page::Base` class to replace `find('.dropdown-menu-labels .dropdown-input-field').send_keys [label, :enter]` +3. Use `click_body` after iterating on each label, instead of using `find('#content-body').click` +4. Iterate on every label again, and then we use `has_element?(:labels_block, text: label)` after clicking the page body (which applies the labels), and before refreshing the page, to avoid test flakiness due to refreshing too fast. + +##### Details of `text_of_labels_block` + +The `text_of_labels_block` method is a simple method that returns the `:labels_block` element (`find_element(:labels_block)`). + +#### Updates in the view (*.html.haml) and `dropdowns_helper.rb` files + +Now let's change the view and the `dropdowns_helper` files to add the selectors that relate to the Page Object. + +In the [app/views/shared/issuable/_sidebar.html.haml](https://gitlab.com/gitlab-org/gitlab-ee/blob/master/app/views/shared/issuable/_sidebar.html.haml) file, on [line 105 ](https://gitlab.com/gitlab-org/gitlab-ee/blob/84043fa72ca7f83ae9cde48ad670e6d5d16501a3/app/views/shared/issuable/_sidebar.html.haml#L105), add an extra class `qa-edit-link-labels`. + +The code should look like this: `= link_to _('Edit'), '#', class: 'js-sidebar-dropdown-toggle edit-link qa-edit-link-labels float-right'`. + +In the same file, on [line 121](https://gitlab.com/gitlab-org/gitlab-ee/blob/84043fa72ca7f83ae9cde48ad670e6d5d16501a3/app/views/shared/issuable/_sidebar.html.haml#L121), add an extra class `.qa-dropdown-menu-labels`. + +The code should look like this: `.dropdown-menu.dropdown-select.dropdown-menu-paging.dropdown-menu-labels.qa-dropdown-menu-labels.dropdown-menu-selectable`. + +In the [`dropdowns_helper.rb`](https://gitlab.com/gitlab-org/gitlab-ee/blob/master/app/helpers/dropdowns_helper.rb) file, on [line 94](https://gitlab.com/gitlab-org/gitlab-ee/blob/99e51a374f2c20bee0989cac802e4b5621f72714/app/helpers/dropdowns_helper.rb#L94), add an extra class `qa-dropdown-input-field`. + +The code should look like this: `filter_output = search_field_tag search_id, nil, class: "dropdown-input-field qa-dropdown-input-field", placeholder: placeholder, autocomplete: 'off'`. + +> Classes starting with `qa-` are used for testing purposes only, and by defining such classes in the elements we add **testability** in the application. + +> When defining a class like `qa-labels-block`, it is transformed into `:labels_block` for usage in the Page Objects. So, `qa-edit-link-labels` is tranformed into `:edit_link_labels`, `qa-dropdown-menu-labels` is transformed into `:dropdown_menu_labels`, and `qa-dropdown-input-field` is transformed into `:dropdown_input_field`. Also, we use a [sanity test](https://gitlab.com/gitlab-org/gitlab-ce/tree/master/qa/qa/page#how-did-we-solve-fragile-tests-problem) to check that defined elements have their respective `qa-` selectors in the specified views. + +> We did not define the `qa-labels-block` class in the `app/views/shared/issuable/_sidebar.html.haml` file because it was already there to be used. + +#### Updates in the `QA::Page::Base` class + +The last thing that we have to do is to update `QA::Page::Base` class to add the `send_keys_to_element` method on it. + +Add the following snippet of code somewhere where class methods are defined: + +```ruby +def send_keys_to_element(name, keys) + find_element(name).send_keys(keys) +end +``` + +This method receives an element (`name`) and the `keys` that it will send to that element, and the keys are an array that can receive strings, or "special" keys, like `:enter`. + +As you might remember, in the Issue Page Object we call this method like this: `send_keys_to_element(:dropdown_input_field, [label, :enter])`. + +___ + +With that, you should be able to start writing end-to-end tests yourself. *Congratulations!* + +[Page Objects]: page_objects.md +[Resources]: resources.md diff --git a/doc/development/testing_guide/end_to_end/resources.md b/doc/development/testing_guide/end_to_end/resources.md new file mode 100644 index 00000000000..1e32db4f633 --- /dev/null +++ b/doc/development/testing_guide/end_to_end/resources.md @@ -0,0 +1,392 @@ +# Resource class in GitLab QA + +Resources are primarily created using Browser UI steps, but can also +be created via the API or the CLI. + +## How to properly implement a resource class? + +All resource classes should inherit from `Resource::Base`. + +There is only one mandatory method to implement to define a resource class. +This is the `#fabricate!` method, which is used to build the resource via the +browser UI. Note that you should only use [Page objects](page_objects.md) to +interact with a Web page in this method. + +Here is an imaginary example: + +```ruby +module QA + module Resource + class Shirt < Base + attr_accessor :name + + def fabricate! + Page::Dashboard::Index.perform do |dashboard_index| + dashboard_index.go_to_new_shirt + end + + Page::Shirt::New.perform do |shirt_new| + shirt_new.set_name(name) + shirt_new.create_shirt! + end + end + end + end +end +``` + +### Define API implementation + +A resource class may also implement the three following methods to be able to +create the resource via the public GitLab API: + +- `#api_get_path`: The `GET` path to fetch an existing resource. +- `#api_post_path`: The `POST` path to create a new resource. +- `#api_post_body`: The `POST` body (as a Ruby hash) to create a new resource. + +Let's take the `Shirt` resource class, and add these three API methods: + +```ruby +module QA + module Resource + class Shirt < Base + attr_accessor :name + + def fabricate! + # ... same as before + end + + def api_get_path + "/shirt/#{name}" + end + + def api_post_path + "/shirts" + end + + def api_post_body + { + name: name + } + end + end + end +end +``` + +The `Project` resource is a good real example of Browser +UI and API implementations. + +#### Resource attributes + +A resource may need another resource to exist first. For instance, a project +needs a group to be created in. + +To define a resource attribute, you can use the `attribute` method with a +block using the other resource class to fabricate the resource. + +That will allow access to the other resource from your resource object's +methods. You would usually use it in `#fabricate!`, `#api_get_path`, +`#api_post_path`, `#api_post_body`. + +Let's take the `Shirt` resource class, and add a `project` attribute to it: + +```ruby +module QA + module Resource + class Shirt < Base + attr_accessor :name + + attribute :project do + Project.fabricate! do |resource| + resource.name = 'project-to-create-a-shirt' + end + end + + def fabricate! + project.visit! + + Page::Project::Show.perform do |project_show| + project_show.go_to_new_shirt + end + + Page::Shirt::New.perform do |shirt_new| + shirt_new.set_name(name) + shirt_new.create_shirt! + end + end + + def api_get_path + "/project/#{project.path}/shirt/#{name}" + end + + def api_post_path + "/project/#{project.path}/shirts" + end + + def api_post_body + { + name: name + } + end + end + end +end +``` + +**Note that all the attributes are lazily constructed. This means if you want +a specific attribute to be fabricated first, you'll need to call the +attribute method first even if you're not using it.** + +#### Product data attributes + +Once created, you may want to populate a resource with attributes that can be +found in the Web page, or in the API response. +For instance, once you create a project, you may want to store its repository +SSH URL as an attribute. + +Again we could use the `attribute` method with a block, using a page object +to retrieve the data on the page. + +Let's take the `Shirt` resource class, and define a `:brand` attribute: + +```ruby +module QA + module Resource + class Shirt < Base + attr_accessor :name + + attribute :project do + Project.fabricate! do |resource| + resource.name = 'project-to-create-a-shirt' + end + end + + # Attribute populated from the Browser UI (using the block) + attribute :brand do + Page::Shirt::Show.perform do |shirt_show| + shirt_show.fetch_brand_from_page + end + end + + # ... same as before + end + end +end +``` + +**Note again that all the attributes are lazily constructed. This means if +you call `shirt.brand` after moving to the other page, it'll not properly +retrieve the data because we're no longer on the expected page.** + +Consider this: + +```ruby +shirt = + QA::Resource::Shirt.fabricate! do |resource| + resource.name = "GitLab QA" + end + +shirt.project.visit! + +shirt.brand # => FAIL! +``` + +The above example will fail because now we're on the project page, trying to +construct the brand data from the shirt page, however we moved to the project +page already. There are two ways to solve this, one is that we could try to +retrieve the brand before visiting the project again: + +```ruby +shirt = + QA::Resource::Shirt.fabricate! do |resource| + resource.name = "GitLab QA" + end + +shirt.brand # => OK! + +shirt.project.visit! + +shirt.brand # => OK! +``` + +The attribute will be stored in the instance therefore all the following calls +will be fine, using the data previously constructed. If we think that this +might be too brittle, we could eagerly construct the data right before +ending fabrication: + +```ruby +module QA + module Resource + class Shirt < Base + # ... same as before + + def fabricate! + project.visit! + + Page::Project::Show.perform do |project_show| + project_show.go_to_new_shirt + end + + Page::Shirt::New.perform do |shirt_new| + shirt_new.set_name(name) + shirt_new.create_shirt! + end + + populate(:brand) # Eagerly construct the data + end + end + end +end +``` + +The `populate` method will iterate through its arguments and call each +attribute respectively. Here `populate(:brand)` has the same effect as +just `brand`. Using the populate method makes the intention clearer. + +With this, it will make sure we construct the data right after we create the +shirt. The drawback is that this will always construct the data when the +resource is fabricated even if we don't need to use the data. + +Alternatively, we could just make sure we're on the right page before +constructing the brand data: + +```ruby +module QA + module Resource + class Shirt < Base + attr_accessor :name + + attribute :project do + Project.fabricate! do |resource| + resource.name = 'project-to-create-a-shirt' + end + end + + # Attribute populated from the Browser UI (using the block) + attribute :brand do + back_url = current_url + visit! + + Page::Shirt::Show.perform do |shirt_show| + shirt_show.fetch_brand_from_page + end + + visit(back_url) + end + + # ... same as before + end + end +end +``` + +This will make sure it's on the shirt page before constructing brand, and +move back to the previous page to avoid breaking the state. + +#### Define an attribute based on an API response + +Sometimes, you want to define a resource attribute based on the API response +from its `GET` or `POST` request. For instance, if the creation of a shirt via +the API returns + +```ruby +{ + brand: 'a-brand-new-brand', + style: 't-shirt', + materials: [[:cotton, 80], [:polyamide, 20]] +} +``` + +you may want to store `style` as-is in the resource, and fetch the first value +of the first `materials` item in a `main_fabric` attribute. + +Let's take the `Shirt` resource class, and define a `:style` and a +`:main_fabric` attributes: + +```ruby +module QA + module Resource + class Shirt < Base + # ... same as before + + # @style from the instance if present, + # or fetched from the API response if present, + # or a QA::Resource::Base::NoValueError is raised otherwise + attribute :style + + # If @main_fabric is not present, + # and if the API does not contain this field, this block will be + # used to construct the value based on the API response, and + # store the result in @main_fabric + attribute :main_fabric do + api_response.&dig(:materials, 0, 0) + end + + # ... same as before + end + end +end +``` + +**Notes on attributes precedence:** + +- resource instance variables have the highest precedence +- attributes from the API response take precedence over attributes from the + block (usually from Browser UI) +- attributes without a value will raise a `QA::Resource::Base::NoValueError` error + +## Creating resources in your tests + +To create a resource in your tests, you can call the `.fabricate!` method on +the resource class. +Note that if the resource class supports API fabrication, this will use this +fabrication by default. + +Here is an example that will use the API fabrication method under the hood +since it's supported by the `Shirt` resource class: + +```ruby +my_shirt = Resource::Shirt.fabricate! do |shirt| + shirt.name = 'my-shirt' +end + +expect(page).to have_text(my_shirt.name) # => "my-shirt" from the resource's instance variable +expect(page).to have_text(my_shirt.brand) # => "a-brand-new-brand" from the API response +expect(page).to have_text(my_shirt.style) # => "t-shirt" from the API response +expect(page).to have_text(my_shirt.main_fabric) # => "cotton" from the API response via the block +``` + +If you explicitly want to use the Browser UI fabrication method, you can call +the `.fabricate_via_browser_ui!` method instead: + +```ruby +my_shirt = Resource::Shirt.fabricate_via_browser_ui! do |shirt| + shirt.name = 'my-shirt' +end + +expect(page).to have_text(my_shirt.name) # => "my-shirt" from the resource's instance variable +expect(page).to have_text(my_shirt.brand) # => the brand name fetched from the `Page::Shirt::Show` page via the block +expect(page).to have_text(my_shirt.style) # => QA::Resource::Base::NoValueError will be raised because no API response nor a block is provided +expect(page).to have_text(my_shirt.main_fabric) # => QA::Resource::Base::NoValueError will be raised because no API response and the block didn't provide a value (because it's also based on the API response) +``` + +You can also explicitly use the API fabrication method, by calling the +`.fabricate_via_api!` method: + +```ruby +my_shirt = Resource::Shirt.fabricate_via_api! do |shirt| + shirt.name = 'my-shirt' +end +``` + +In this case, the result will be similar to calling +`Resource::Shirt.fabricate!`. + +## Where to ask for help? + +If you need more information, ask for help on `#quality` channel on Slack +(internal, GitLab Team only). + +If you are not a Team Member, and you still need help to contribute, please +open an issue in GitLab CE issue tracker with the `~QA` label. diff --git a/doc/development/testing_guide/end_to_end/style_guide.md b/doc/development/testing_guide/end_to_end/style_guide.md new file mode 100644 index 00000000000..0272e1810f2 --- /dev/null +++ b/doc/development/testing_guide/end_to_end/style_guide.md @@ -0,0 +1,99 @@ +# Style guide for writing end-to-end tests + +This document describes the conventions used at GitLab for writing End-to-end (E2E) tests using the GitLab QA project. + +## `click_` versus `go_to_` + +### When to use `click_`? + +When clicking in a single link to navigate, use `click_`. + +E.g.: + +```ruby +def click_ci_cd_pipelines + within_sidebar do + click_element :link_pipelines + end +end +``` + +From a testing perspective, if we want to check that clicking a link, or a button (a single interaction) is working as intended, we would want the test to read as: + +- Click a certain element +- Verify the action took place + +### When to use `go_to_`? + +When interacting with multiple elements to go to a page, use `go_to_`. + +E.g.: + +```ruby +def go_to_operations_environments + hover_operations do + within_submenu do + click_element(:operations_environments_link) + end + end +end +``` + +`go_to_` fits the definition of interacting with multiple elements very well given it's more of a meta-navigation action that includes multiple interactions. + +Notice that in the above example, before clicking the `:operations_environments_link`, another element is hovered over. + +> We can create these methods as helpers to abstract multi-step navigation. + +### Element naming convention + +When adding new elements to a page, it's important that we have a uniform element naming convention. + +We follow a simple formula roughly based on hungarian notation. + +*Formula*: `element :<descriptor>_<type>` + +- `descriptor`: The natural-language description of what the element is. On the login page, this could be `username`, or `password`. +- `type`: A physical control on the page that can be seen by a user. + - `_button` + - `_link` + - `_tab` + - `_dropdown` + - `_field` + - `_checkbox` + - `_radio` + - `_content` + +*Note: This list is a work in progress. This list will eventually be the end-all enumeration of all available types. + I.e., any element that does not end with something in this list is bad form.* + +#### Examples + +**Good** + +```ruby +view '...' do + element :edit_button + element :notes_tab + element :squash_checkbox + element :username_field + element :issue_title_content +end +``` + +**Bad** + +```ruby +view '...' do + # `_confirmation` should be `_field`. what sort of confirmation? a checkbox confirmation? no real way to disambiguate. + # an appropriate replacement would be `element :password_confirmation_field` + element :password_confirmation + + # `clone_options` is too vague. If it's a dropdown menu, it should be `clone_dropdown`. + # If it's a checkbox, it should be `clone_checkbox` + element :clone_options + + # how is this url being displayed? is it a textbox? a simple span? + element :ssh_clone_url +end +``` diff --git a/doc/development/testing_guide/flaky_tests.md b/doc/development/testing_guide/flaky_tests.md index 3d568c37fba..931cbc51cae 100644 --- a/doc/development/testing_guide/flaky_tests.md +++ b/doc/development/testing_guide/flaky_tests.md @@ -8,7 +8,7 @@ eventually. ## Quarantined tests When a test frequently fails in `master`, -[a ~"broken master" issue](https://about.gitlab.com/handbook/engineering/workflow/#broken-master) +[a ~"master:broken" issue](https://about.gitlab.com/handbook/engineering/workflow/#broken-master) should be created. If the test cannot be fixed in a timely fashion, there is an impact on the productivity of all the developers, so it should be placed in quarantine by @@ -21,7 +21,7 @@ bin/rspec --tag quarantine ``` **Before putting a test in quarantine, you should make sure that a -~"broken master" issue exists for it so it won't stay in quarantine forever.** +~"master:broken" issue exists for it so it won't stay in quarantine forever.** Once a test is in quarantine, there are 3 choices: diff --git a/doc/development/testing_guide/frontend_testing.md b/doc/development/testing_guide/frontend_testing.md index 0470a071d39..4c9d1684c00 100644 --- a/doc/development/testing_guide/frontend_testing.md +++ b/doc/development/testing_guide/frontend_testing.md @@ -13,16 +13,60 @@ in the future. See the [Testing Standards and Style Guidelines](index.md) page for more information on general testing practices at GitLab. +## Jest + +We have started to migrate frontend tests to the [Jest](https://jestjs.io) testing framework (see also the corresponding +[epic](https://gitlab.com/groups/gitlab-org/-/epics/895)). + +Jest tests can be found in `/spec/frontend` and `/ee/spec/frontend` in EE. + +It is not yet a requirement to use Jest. You can view the +[epic](https://gitlab.com/groups/gitlab-org/-/epics/873) of issues +we need to solve before being able to use Jest for all our needs. + +### Differences to Karma + +- Jest runs in a Node.js environment, not in a browser. Support for running Jest tests in a browser [is planned](https://gitlab.com/gitlab-org/gitlab-ce/issues/58205). +- Because Jest runs in a Node.js environment, it uses [jsdom](https://github.com/jsdom/jsdom) by default. +- All calls to `setTimeout` and `setInterval` are mocked away. See also [Jest Timer Mocks](https://jestjs.io/docs/en/timer-mocks). +- `rewire` is not required because Jest supports mocking modules. See also [Manual Mocks](https://jestjs.io/docs/en/manual-mocks). +- The following will cause tests to fail in Jest: + - Unmocked requests. + - Unhandled Promise rejections. + - Calls to `console.warn`, including warnings from libraries like Vue. + +### Debugging Jest tests + +Running `yarn jest-debug` will run Jest in debug mode, allowing you to debug/inspect as described in the [Jest docs](https://jestjs.io/docs/en/troubleshooting#tests-are-failing-and-you-don-t-know-why). + +### Timeout error + +The default timeout for Jest is set in +[`/spec/frontend/test_setup.js`](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/spec/frontend/test_setup.js). + +If your test exceeds that time, it will fail. + +If you cannot improve the performance of the tests, you can increase the timeout +for a specific test using +[`setTestTimeout`](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/spec/frontend/helpers/timeout.js). + +```javascript +import { setTestTimeout } from 'helpers/timeout'; + +describe('Component', () => { + it('does something amazing', () => { + setTestTimeout(500); + // ... + }); +}); +``` + +Remember that the performance of each test depends on the environment. + ## Karma test suite GitLab uses the [Karma][karma] test runner with [Jasmine] as its test -framework for our JavaScript unit and integration tests. For integration tests, -we generate HTML files using RSpec (see `spec/javascripts/fixtures/*.rb` for examples). -Some fixtures are still HAML templates that are translated to HTML files using the same mechanism (see `static_fixtures.rb`). -Adding these static fixtures should be avoided as they are harder to keep up to date with real views. -The existing static fixtures will be migrated over time. -Please see [gitlab-org/gitlab-ce#24753](https://gitlab.com/gitlab-org/gitlab-ce/issues/24753) to track our progress. -Fixtures are served during testing by the [jasmine-jquery][jasmine-jquery] plugin. +framework for our JavaScript unit and integration tests. JavaScript tests live in `spec/javascripts/`, matching the folder structure of `app/assets/javascripts/`: `app/assets/javascripts/behaviors/autosize.js` @@ -134,7 +178,7 @@ placeholders, and recalling when they are called and the arguments that are passed to them. These tools should be used liberally, to test for expected behavior, to mock responses, and to block unwanted side effects (such as a method that would generate a network request or alter `window.location`). The -documentation for these methods can be found in the [jasmine introduction page](https://jasmine.github.io/2.0/introduction.html#section-Spies). +documentation for these methods can be found in the [Jasmine introduction page](https://jasmine.github.io/2.0/introduction.html#section-Spies). Sometimes you may need to spy on a method that is directly imported by another module. GitLab has a custom `spyOnDependency` method which utilizes @@ -149,6 +193,7 @@ export default function doSomething() { visitUrl('/foo/bar'); } ``` + ```js // my_module_spec.js import doSomething from '~/my_module'; @@ -168,14 +213,202 @@ export of a module who's import you want to stub, rather than an object which contains a method you wish to stub (if the module does not have a default export, one is be generated by the babel plugin). The second parameter is the name of the import you wish to change. The result of the function is a Spy -object which can be treated like any other jasmine spy object. +object which can be treated like any other Jasmine spy object. Further documentation on the babel rewire pluign API can be found on [its repository Readme doc](https://github.com/speedskater/babel-plugin-rewire#babel-plugin-rewire). #### Waiting in tests -If you cannot avoid using [`setTimeout`](https://developer.mozilla.org/en-US/docs/Web/API/WindowOrWorkerGlobalScope/setTimeout) in tests, please use the [Jasmine mock clock](https://jasmine.github.io/api/2.9/Clock.html). +Sometimes a test needs to wait for something to happen in the application before it continues. +Avoid using [`setTimeout`](https://developer.mozilla.org/en-US/docs/Web/API/WindowOrWorkerGlobalScope/setTimeout) +because it makes the reason for waiting unclear and if passed a time larger than zero it will slow down our test suite. +Instead use one of the following approaches. + +##### Promises and Ajax calls + +Register handler functions to wait for the `Promise` to be resolved. + +```javascript +const askTheServer = () => { + return axios + .get('/endpoint') + .then(response => { + // do something + }) + .catch(error => { + // do something else + }); +}; +``` + +**in Jest:** + +```javascript +it('waits for an Ajax call', () => { + return askTheServer().then(() => { + expect(something).toBe('done'); + }); +}); +``` + +**in Karma:** + +```javascript +it('waits for an Ajax call', done => { + askTheServer() + .then(() => { + expect(something).toBe('done'); + }) + .then(done) + .catch(done.fail); +}); +``` + +If you are not able to register handlers to the `Promise`—for example because it is executed in a synchronous Vue life +cycle hook—you can flush all pending `Promise`s: + +**in Jest:** + +```javascript +it('waits for an Ajax call', () => { + synchronousFunction(); + jest.runAllTicks(); + + expect(something).toBe('done'); +}); +``` + +**in Karma:** + +You are out of luck. The following only works sometimes and may lead to flaky failures: + +```javascript +it('waits for an Ajax call', done => { + synchronousFunction(); + + // create a new Promise and hope that it resolves after the rest + Promise.resolve() + .then(() => { + expect(something).toBe('done'); + }) + .then(done) + .catch(done.fail); +}); +``` + +##### Vue rendering + +To wait until a Vue component is re-rendered, use either of the equivalent +[`Vue.nextTick()`](https://vuejs.org/v2/api/#Vue-nextTick) or `vm.$nextTick()`. + +**in Jest:** + +```javascript +it('renders something', () => { + wrapper.setProps({ value: 'new value' }); + + return wrapper.vm.$nextTick().then(() => { + expect(wrapper.text()).toBe('new value'); + }); +}); +``` + +**in Karma:** + +```javascript +it('renders something', done => { + wrapper.setProps({ value: 'new value' }); + + wrapper.vm + .$nextTick() + .then(() => { + expect(wrapper.text()).toBe('new value'); + }) + .then(done) + .catch(done.fail); +}); +``` + +##### `setTimeout()` / `setInterval()` in application + +If the application itself is waiting for some time, mock await the waiting. In Jest this is already +[done by default](https://gitlab.com/gitlab-org/gitlab-ce/blob/a2128edfee799e49a8732bfa235e2c5e14949c68/jest.config.js#L47) +(see also [Jest Timer Mocks](https://jestjs.io/docs/en/timer-mocks)). In Karma you can use the +[Jasmine mock clock](https://jasmine.github.io/api/2.9/Clock.html). + +```javascript +const doSomethingLater = () => { + setTimeout(() => { + // do something + }, 4000); +}; +``` + +**in Jest:** + +```javascript +it('does something', () => { + doSomethingLater(); + jest.runAllTimers(); + + expect(something).toBe('done'); +}); +``` + +**in Karma:** + +```javascript +it('does something', () => { + jasmine.clock().install(); + + doSomethingLater(); + jasmine.clock().tick(4000); + + expect(something).toBe('done'); + jasmine.clock().uninstall(); +}); +``` + +##### Events + +If the application triggers an event that you need to wait for in your test, register an event handler which contains +the assertions: + +```javascript +it('waits for an event', done => { + eventHub.$once('someEvent', eventHandler); + + someFunction(); + + function eventHandler() { + expect(something).toBe('done'); + done(); + } +}); +``` + +In Jest you can also use a `Promise` for this: + +```javascript +it('waits for an event', () => { + const eventTriggered = new Promise(resolve => eventHub.$once('someEvent', resolve)); + + someFunction(); + + return eventTriggered.then(() => { + expect(something).toBe('done'); + }); +}); +``` + +#### Migrating flaky Karma tests to Jest + +Some of our Karma tests are flaky because they access the properties of a shared scope. +This also means that they are not easily parallelized. + +Migrating flaky Karma tests to Jest will help significantly as each test is executed +in an isolated scope, improving performance and predictability. ### Vue.js unit tests @@ -183,32 +416,30 @@ See this [section][vue-test]. ### Running frontend tests -`rake karma` runs the frontend-only (JavaScript) tests. -It consists of two subtasks: +For running the frontend tests, you need the following commands: -- `rake karma:fixtures` (re-)generates fixtures -- `rake karma:tests` actually executes the tests +- `rake karma:fixtures` (re-)generates [fixtures](#frontend-test-fixtures). +- `yarn test` executes the tests. -As long as the fixtures don't change, `rake karma:tests` (or `yarn karma`) -is sufficient (and saves you some time). +As long as the fixtures don't change, `yarn test` is sufficient (and saves you some time). ### Live testing and focused testing -While developing locally, it may be helpful to keep karma running so that you +While developing locally, it may be helpful to keep Karma running so that you can get instant feedback on as you write tests and modify code. To do this -you can start karma with `yarn run karma-start`. It will compile the javascript +you can start Karma with `yarn run karma-start`. It will compile the javascript assets and run a server at `http://localhost:9876/` where it will automatically run the tests on any browser which connects to it. You can enter that url on multiple browsers at once to have it run the tests on each in parallel. -While karma is running, any changes you make will instantly trigger a recompile +While Karma is running, any changes you make will instantly trigger a recompile and retest of the entire test suite, so you can see instantly if you've broken -a test with your changes. You can use [jasmine focused][jasmine-focus] or -excluded tests (with `fdescribe` or `xdescribe`) to get karma to run only the +a test with your changes. You can use [Jasmine focused][jasmine-focus] or +excluded tests (with `fdescribe` or `xdescribe`) to get Karma to run only the tests you want while you're working on a specific feature, but make sure to remove these directives when you commit your code. -It is also possible to only run karma on specific folders or files by filtering +It is also possible to only run Karma on specific folders or files by filtering the run tests via the argument `--filter-spec` or short `-f`: ```bash @@ -235,26 +466,49 @@ yarn karma -f 'spec/javascripts/ide/**/file_spec.js' Information on setting up and running RSpec integration tests with [Capybara] can be found in the [Testing Best Practices](best_practices.md). -## Gotchas +## Frontend test fixtures -### Errors due to use of unsupported JavaScript features +Code that is added to HAML templates (in `app/views/`) or makes Ajax requests to the backend has tests that require HTML or JSON from the backend. +Fixtures for these tests are located at: -Similar errors will be thrown if you're using JavaScript features not yet -supported by the PhantomJS test runner which is used for both Karma and RSpec -tests. We polyfill some JavaScript objects for older browsers, but some -features are still unavailable: +- `spec/javascripts/fixtures/`, for running tests in CE. +- `ee/spec/javascripts/fixtures/`, for running tests in EE. -- Array.from -- Array.first -- Async functions -- Generators -- Array destructuring -- For..Of -- Symbol/Symbol.iterator -- Spread +Fixture files in: -Until these are polyfilled appropriately, they should not be used. Please -update this list with additional unsupported features. +- The Karma test suite are served by [jasmine-jquery](https://github.com/velesin/jasmine-jquery). +- Jest use `spec/frontend/helpers/fixtures.js`. + +The following are examples of tests that work for both Karma and Jest: + +```javascript +it('makes a request', () => { + const responseBody = getJSONFixture('some/fixture.json'); // loads spec/javascripts/fixtures/some/fixture.json + axiosMock.onGet(endpoint).reply(200, responseBody); + + myButton.click(); + + // ... +}); + +it('uses some HTML element', () => { + loadFixtures('some/page.html'); // loads spec/javascripts/fixtures/some/page.html and adds it to the DOM + + const element = document.getElementById('#my-id'); + + // ... +}); +``` + +HTML and JSON fixtures are generated from backend views and controllers using RSpec (see `spec/javascripts/fixtures/*.rb`). + +For each fixture, the content of the `response` variable is stored in the output file. +This variable gets automagically set if the test is marked as `type: :request` or `type: :controller`. +Fixtures are regenerated using the `bin/rake karma:fixtures` command but you can also generate them individually, +for example `bin/rspec spec/javascripts/fixtures/merge_requests.rb`. +When creating a new fixture, it often makes sense to take a look at the corresponding tests for the endpoint in `(ee/)spec/controllers/` or `(ee/)spec/requests/`. + +## Gotchas ### RSpec errors due to JavaScript @@ -286,7 +540,6 @@ end ``` [jasmine-focus]: https://jasmine.github.io/2.5/focused_specs.html -[jasmine-jquery]: https://github.com/velesin/jasmine-jquery [karma]: http://karma-runner.github.io/ [vue-test]: https://docs.gitlab.com/ce/development/fe_guide/vue.html#testing-vue-components [rspec]: https://github.com/rspec/rspec-rails#feature-specs diff --git a/doc/development/testing_guide/img/qa_on_merge_requests_cicd_architecture.png b/doc/development/testing_guide/img/qa_on_merge_requests_cicd_architecture.png Binary files differnew file mode 100644 index 00000000000..5b93a05db96 --- /dev/null +++ b/doc/development/testing_guide/img/qa_on_merge_requests_cicd_architecture.png diff --git a/doc/development/testing_guide/img/review_apps_cicd_architecture.png b/doc/development/testing_guide/img/review_apps_cicd_architecture.png Binary files differindex 87e472076f3..1ee28d3db91 100644 --- a/doc/development/testing_guide/img/review_apps_cicd_architecture.png +++ b/doc/development/testing_guide/img/review_apps_cicd_architecture.png diff --git a/doc/development/testing_guide/index.md b/doc/development/testing_guide/index.md index 67e4cfeda0e..93ee2a6371a 100644 --- a/doc/development/testing_guide/index.md +++ b/doc/development/testing_guide/index.md @@ -33,7 +33,7 @@ changes should be tested. ## [Testing best practices](best_practices.md) -Everything you should know about how to write good tests: RSpec, FactoryBot, +Everything you should know about how to write good tests: Test Design, RSpec, FactoryBot, system tests, parameterized tests etc. --- @@ -71,7 +71,7 @@ Everything you should know about how to test Rake tasks. --- -## [End-to-end tests](end_to_end_tests.md) +## [End-to-end tests](end_to_end/index.md) Everything you should know about how to run end-to-end tests using [GitLab QA][gitlab-qa] testing framework. diff --git a/doc/development/testing_guide/review_apps.md b/doc/development/testing_guide/review_apps.md index 703e342fc13..b5fde92a23d 100644 --- a/doc/development/testing_guide/review_apps.md +++ b/doc/development/testing_guide/review_apps.md @@ -6,7 +6,7 @@ Review Apps are automatically deployed by each pipeline, both in ## How does it work? -### CD/CD architecture diagram +### CI/CD architecture diagram  @@ -14,23 +14,29 @@ Review Apps are automatically deployed by each pipeline, both in <summary>Show mermaid source</summary> <pre> graph TD - B1 -.->|2. once gitlab:assets:compile is done,<br />triggers a CNG-mirror pipeline and wait for it to be done| A2 - C1 -.->|2. once review-build-cng is done,<br />Helm deploys the Review App using the Cloud<br/>Native images built by the CNG-mirror pipeline| A3 - -subgraph gitlab-ce/ee `test` stage - A1[gitlab:assets:compile] - B1[review-build-cng] -->|1. wait for| A1 - C1[review-deploy] -->|1. wait for| B1 - D1[review-qa-smoke] -->|1. wait for| C1 - D1[review-qa-smoke] -.->|2. once review-deploy is done| E1>gitlab-qa runs the smoke<br/>suite against the Review App] + build-qa-image -.->|once the `prepare` stage is done| gitlab:assets:compile + review-build-cng -->|triggers a CNG-mirror pipeline and wait for it to be done| CNG-mirror + review-build-cng -.->|once the `test` stage is done| review-deploy + review-deploy -.->|once the `review` stage is done| review-qa-smoke + +subgraph 1. gitlab-ce/ee `prepare` stage + build-qa-image end -subgraph CNG-mirror pipeline - A2>Cloud Native images are built]; +subgraph 2. gitlab-ce/ee `test` stage + gitlab:assets:compile -->|plays dependent job once done| review-build-cng + end + +subgraph 3. gitlab-ce/ee `review` stage + review-deploy["review-deploy<br /><br />Helm deploys the Review App using the Cloud<br/>Native images built by the CNG-mirror pipeline.<br /><br />Cloud Native images are deployed to the `review-apps-ce` or `review-apps-ee`<br />Kubernetes (GKE) cluster, in the GCP `gitlab-review-apps` project."] end -subgraph GCP `gitlab-review-apps` project - A3>"Cloud Native images are deployed to the<br />`review-apps-ce` or `review-apps-ee` Kubernetes (GKE) cluster"]; +subgraph 4. gitlab-ce/ee `qa` stage + review-qa-smoke[review-qa-smoke<br /><br />gitlab-qa runs the smoke suite against the Review App.] + end + +subgraph CNG-mirror pipeline + CNG-mirror>Cloud Native images are built]; end </pre> </details> @@ -38,107 +44,120 @@ subgraph GCP `gitlab-review-apps` project ### Detailed explanation 1. On every [pipeline][gitlab-pipeline] during the `test` stage, the - [`review-build-cng`][review-build-cng] and - [`review-deploy`][review-deploy] jobs are automatically started. - - The [`review-deploy`][review-deploy] job waits for the - [`review-build-cng`][review-build-cng] job to finish. - - The [`review-build-cng`][review-build-cng] job waits for the - [`gitlab:assets:compile`][gitlab:assets:compile] job to finish since the - [`CNG-mirror`][cng-mirror] pipeline triggered in the following step depends on it. -1. Once the [`gitlab:assets:compile`][gitlab:assets:compile] job is done, - [`review-build-cng`][review-build-cng] [triggers a pipeline][cng-pipeline] - in the [`CNG-mirror`][cng-mirror] project. - - The [`CNG-mirror`][cng-pipeline] pipeline creates the Docker images of - each component (e.g. `gitlab-rails-ee`, `gitlab-shell`, `gitaly` etc.) - based on the commit from the [GitLab pipeline][gitlab-pipeline] and store - them in its [registry][cng-mirror-registry]. - - We use the [`CNG-mirror`][cng-mirror] project so that the `CNG`, (**C**loud - **N**ative **G**itLab), project's registry is not overloaded with a - lot of transient Docker images. -1. Once the [`review-build-cng`][review-build-cng] job is done, the - [`review-deploy`][review-deploy] job deploys the Review App using - [the official GitLab Helm chart][helm-chart] to the - [`review-apps-ce`][review-apps-ce] / [`review-apps-ee`][review-apps-ee] - Kubernetes cluster on GCP. - - The actual scripts used to deploy the Review App can be found at - [`scripts/review_apps/review-apps.sh`][review-apps.sh]. - - These scripts are basically - [our official Auto DevOps scripts][Auto-DevOps.gitlab-ci.yml] where the - default CNG images are overridden with the images built and stored in the - [`CNG-mirror` project's registry][cng-mirror-registry]. - - Since we're using [the official GitLab Helm chart][helm-chart], this means - you get a dedicated environment for your branch that's very close to what - it would look in production. + [`gitlab:assets:compile`][gitlab:assets:compile] job is automatically started. + - Once it's done, it starts the [`review-build-cng`][review-build-cng] + manual job since the [`CNG-mirror`][cng-mirror] pipeline triggered in the + following step depends on it. +1. The [`review-build-cng`][review-build-cng] job [triggers a pipeline][cng-mirror-pipeline] + in the [`CNG-mirror`][cng-mirror] project. + - The [`CNG-mirror`][cng-mirror-pipeline] pipeline creates the Docker images of + each component (e.g. `gitlab-rails-ee`, `gitlab-shell`, `gitaly` etc.) + based on the commit from the [GitLab pipeline][gitlab-pipeline] and stores + them in its [registry][cng-mirror-registry]. + - We use the [`CNG-mirror`][cng-mirror] project so that the `CNG`, (**C**loud + **N**ative **G**itLab), project's registry is not overloaded with a + lot of transient Docker images. + - Note that the official CNG images are built by the `cloud-native-image` + job, which runs only for tags, and triggers itself a [`CNG`][cng] pipeline. +1. Once the `test` stage is done, the [`review-deploy`][review-deploy] job + deploys the Review App using [the official GitLab Helm chart][helm-chart] to + the [`review-apps-ce`][review-apps-ce] / [`review-apps-ee`][review-apps-ee] + Kubernetes cluster on GCP. + - The actual scripts used to deploy the Review App can be found at + [`scripts/review_apps/review-apps.sh`][review-apps.sh]. + - These scripts are basically + [our official Auto DevOps scripts][Auto-DevOps.gitlab-ci.yml] where the + default CNG images are overridden with the images built and stored in the + [`CNG-mirror` project's registry][cng-mirror-registry]. + - Since we're using [the official GitLab Helm chart][helm-chart], this means + you get a dedicated environment for your branch that's very close to what + it would look in production. 1. Once the [`review-deploy`][review-deploy] job succeeds, you should be able to - use your Review App thanks to the direct link to it from the MR widget. To log - into the Review App, see "Log into my Review App?" below. + use your Review App thanks to the direct link to it from the MR widget. To log + into the Review App, see "Log into my Review App?" below. **Additional notes:** +- If the `review-deploy` job keep failing (note that we already retry it twice), + please post a message in the `#quality` channel and/or create a ~Quality ~bug + issue with a link to your merge request. Note that the deployment failure can + reveal an actual problem introduced in your merge request (i.e. this isn't + necessarily a transient failure)! +- If the `review-qa-smoke` job keep failing (note that we already retry it twice), + please check the job's logs: you could discover an actual problem introduced in + your merge request. You can also download the artifacts to see screenshots of + the page at the time the failures occurred. If you don't find the cause of the + failure or if it seems unrelated to your change, please post a message in the + `#quality` channel and/or create a ~Quality ~bug issue with a link to your + merge request. +- The manual [`review-stop`][gitlab-ci-yml] in the `test` stage can be used to + stop a Review App manually, and is also started by GitLab once a merge + request's branch is deleted after being merged. +- Review Apps are cleaned up regularly via a pipeline schedule that runs + the [`schedule:review-cleanup`][gitlab-ci-yml] job. - The Kubernetes cluster is connected to the `gitlab-{ce,ee}` projects using [GitLab's Kubernetes integration][gitlab-k8s-integration]. This basically allows to have a link to the Review App directly from the merge request widget. -- If the Review App deployment fails, you can simply retry it (there's no need - to run the [`review-stop`][gitlab-ci-yml] job first). -- The manual [`review-stop`][gitlab-ci-yml] in the `test` stage can be used to - stop a Review App manually, and is also started by GitLab once a branch is - deleted. -- Review Apps are cleaned up regularly using a pipeline schedule that runs - the [`schedule:review-cleanup`][gitlab-ci-yml] job. ## QA runs -On every [pipeline][gitlab-pipeline] during the `test` stage, the -`review-qa-smoke` job is automatically started: it runs the smoke QA suite. +On every [pipeline][gitlab-pipeline] in the `qa` stage (which comes after the +`review` stage), the `review-qa-smoke` job is automatically started and it runs +the QA smoke suite. + You can also manually start the `review-qa-all`: it runs the full QA suite. -Note that both jobs first wait for the `review-deploy` job to be finished. +## Performance Metrics -## How to? +On every [pipeline][gitlab-pipeline] in the `qa` stage, the +`review-performance` job is automatically started: this job does basic +browser performance testing using a +[Sitespeed.io Container](../../user/project/merge_requests/browser_performance_testing.md). -### Log into my Review App? +## How to: + +### Log into my Review App The default username is `root` and its password can be found in the 1Password secure note named **gitlab-{ce,ee} Review App's root password**. -### Enable a feature flag for my Review App? +### Enable a feature flag for my Review App 1. Open your Review App and log in as documented above. 1. Create a personal access token. 1. Enable the feature flag using the [Feature flag API](../../api/features.md). -### Find my Review App slug? +### Find my Review App slug 1. Open the `review-deploy` job. 1. Look for `Checking for previous deployment of review-*`. 1. For instance for `Checking for previous deployment of review-qa-raise-e-12chm0`, - your Review App slug would be `review-qa-raise-e-12chm0` in this case. + your Review App slug would be `review-qa-raise-e-12chm0` in this case. -### Run a Rails console? +### Run a Rails console 1. [Filter Workloads by your Review App slug](https://console.cloud.google.com/kubernetes/workload?project=gitlab-review-apps) - , e.g. `review-29951-issu-id2qax`. -1. Find and open the `task-runner` Deployment, e.g. `review-29951-issu-id2qax-task-runner`. -1. Click on the Pod in the "Managed pods" section, e.g. `review-29951-issu-id2qax-task-runner-d5455cc8-2lsvz`. + , e.g. `review-qa-raise-e-12chm0`. +1. Find and open the `task-runner` Deployment, e.g. `review-qa-raise-e-12chm0-task-runner`. +1. Click on the Pod in the "Managed pods" section, e.g. `review-qa-raise-e-12chm0-task-runner-d5455cc8-2lsvz`. 1. Click on the `KUBECTL` dropdown, then `Exec` -> `task-runner`. -1. Replace `-c task-runner -- ls` with `-- /srv/gitlab/bin/rails c` from the - default command or - - Run `kubectl exec --namespace review-apps-ce -it review-29951-issu-id2qax-task-runner-d5455cc8-2lsvz -- /srv/gitlab/bin/rails c` - and - - Replace `review-apps-ce` with `review-apps-ee` if the Review App - is running EE, and - - Replace `review-29951-issu-id2qax-task-runner-d5455cc8-2lsvz` - with your Pod's name. - -### Dig into a Pod's logs? - -1. [Filter Workloads by your Review App slug](https://console.cloud.google.com/kubernetes/workload?project=gitlab-review-apps) - , e.g. `review-1979-1-mul-dnvlhv`. +1. Replace `-c task-runner -- ls` with `-it -- gitlab-rails console` from the + default command or + - Run `kubectl exec --namespace review-apps-ce review-qa-raise-e-12chm0-task-runner-d5455cc8-2lsvz -it -- gitlab-rails console` and + - Replace `review-apps-ce` with `review-apps-ee` if the Review App + is running EE, and + - Replace `review-qa-raise-e-12chm0-task-runner-d5455cc8-2lsvz` + with your Pod's name. + +### Dig into a Pod's logs + +1. [Filter Workloads by your Review App slug](https://console.cloud.google.com/kubernetes/workload?project=gitlab-review-apps), + e.g. `review-qa-raise-e-12chm0`. 1. Find and open the `migrations` Deployment, e.g. - `review-1979-1-mul-dnvlhv-migrations.1`. + `review-qa-raise-e-12chm0-migrations.1`. 1. Click on the Pod in the "Managed pods" section, e.g. - `review-1979-1-mul-dnvlhv-migrations.1-nqwtx`. + `review-qa-raise-e-12chm0-migrations.1-nqwtx`. 1. Click on the `Container logs` link. ## Frequently Asked Questions @@ -174,7 +193,8 @@ find a way to limit it to only us.** [review-build-cng]: https://gitlab.com/gitlab-org/gitlab-ce/-/jobs/149511623 [review-deploy]: https://gitlab.com/gitlab-org/gitlab-ce/-/jobs/149511624 [cng-mirror]: https://gitlab.com/gitlab-org/build/CNG-mirror -[cng-pipeline]: https://gitlab.com/gitlab-org/build/CNG-mirror/pipelines/44364657 +[cng]: https://gitlab.com/gitlab-org/build/CNG +[cng-mirror-pipeline]: https://gitlab.com/gitlab-org/build/CNG-mirror/pipelines/44364657 [cng-mirror-registry]: https://gitlab.com/gitlab-org/build/CNG-mirror/container_registry [helm-chart]: https://gitlab.com/charts/gitlab/ [review-apps-ce]: https://console.cloud.google.com/kubernetes/clusters/details/us-central1-a/review-apps-ce?project=gitlab-review-apps @@ -183,7 +203,7 @@ find a way to limit it to only us.** [automated_cleanup.rb]: https://gitlab.com/gitlab-org/gitlab-ee/blob/master/scripts/review_apps/automated_cleanup.rb [Auto-DevOps.gitlab-ci.yml]: https://gitlab.com/gitlab-org/gitlab-ce/blob/master/lib/gitlab/ci/templates/Auto-DevOps.gitlab-ci.yml [gitlab-ci-yml]: https://gitlab.com/gitlab-org/gitlab-ce/blob/master/.gitlab-ci.yml -[gitlab-k8s-integration]: https://docs.gitlab.com/ee/user/project/clusters/index.html +[gitlab-k8s-integration]: ../../user/project/clusters/index.md [password-bug]: https://gitlab.com/gitlab-org/gitlab-ce/issues/53621 --- diff --git a/doc/development/testing_guide/smoke.md b/doc/development/testing_guide/smoke.md index 3360031c220..c9d3238fbe9 100644 --- a/doc/development/testing_guide/smoke.md +++ b/doc/development/testing_guide/smoke.md @@ -7,13 +7,19 @@ functionality is working. Currently, our suite consists of this basic functionality coverage: -- User Login (Standard Auth) -- Project Creation -- Issue Creation -- Merge Request Creation +- User standard authentication +- SSH Key creation and addition to a user +- Project simple creation +- Project creation with Auto-DevOps enabled +- Issue creation +- Merge Request creation +- Snippet creation Smoke tests have the `:smoke` RSpec metadata. +See [End-to-end Testing](end_to_end/index.md) for more details about +end-to-end tests. + --- [Return to Testing documentation](index.md) diff --git a/doc/development/testing_guide/testing_levels.md b/doc/development/testing_guide/testing_levels.md index a7a3459719b..e1ce4d3b7d1 100644 --- a/doc/development/testing_guide/testing_levels.md +++ b/doc/development/testing_guide/testing_levels.md @@ -4,6 +4,15 @@ _This diagram demonstrates the relative priority of each test type we use. `e2e` stands for end-to-end._ +As of 2019-05-01, we have the following distribution of tests per level: + +| Test level | Community Edition | Enterprise Edition | Community + Enterprise Edition | +| --------- | ---------- | -------------- | ----- | +| Black-box tests at the system level (aka end-to-end or QA tests) | 68 (0.14%) | 31 (0.2%) | 99 (0.17%) | +| White-box tests at the system level (aka system or feature tests) | 5,471 (11.9%) | 969 (7.4%) | 6440 (10.9%) | +| Integration tests | 8,333 (18.2%) | 2,244 (17.2%) | 10,577 (17.9%) | +| Unit tests | 32,031 (69.7%) | 9,778 (75.1%) | 41,809 (71%) | + ## Unit tests Formal definition: <https://en.wikipedia.org/wiki/Unit_testing> @@ -16,19 +25,31 @@ records should use stubs/doubles as much as possible. | Code path | Tests path | Testing engine | Notes | | --------- | ---------- | -------------- | ----- | +| `app/assets/javascripts/` | `spec/javascripts/`, `spec/frontend/` | Karma & Jest | More details in the [Frontend Testing guide](frontend_testing.md) section. | | `app/finders/` | `spec/finders/` | RSpec | | +| `app/graphql/` | `spec/graphql/` | RSpec | | | `app/helpers/` | `spec/helpers/` | RSpec | | -| `app/db/{post_,}migrate/` | `spec/migrations/` | RSpec | More details at [`spec/migrations/README.md`](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/spec/migrations/README.md). | +| `app/models/` | `spec/models/` | RSpec | | | `app/policies/` | `spec/policies/` | RSpec | | | `app/presenters/` | `spec/presenters/` | RSpec | | -| `app/routing/` | `spec/routing/` | RSpec | | | `app/serializers/` | `spec/serializers/` | RSpec | | | `app/services/` | `spec/services/` | RSpec | | -| `app/tasks/` | `spec/tasks/` | RSpec | | | `app/uploaders/` | `spec/uploaders/` | RSpec | | +| `app/validators/` | `spec/validators/` | RSpec | | | `app/views/` | `spec/views/` | RSpec | | | `app/workers/` | `spec/workers/` | RSpec | | -| `app/assets/javascripts/` | `spec/javascripts/` | Karma | More details in the [Frontend Testing guide](frontend_testing.md) section. | +| `bin/` | `spec/bin/` | RSpec | | +| `config/` | `spec/config/` | RSpec | | +| `config/initializers/` | `spec/initializers/` | RSpec | | +| `config/routes.rb`, `config/routes/` | `spec/routing/` | RSpec | | +| `config/puma.example.development.rb`, `config/unicorn.rb.example` | `spec/rack_servers/` | RSpec | | +| `db/` | `spec/db/` | RSpec | | +| `db/{post_,}migrate/` | `spec/migrations/` | RSpec | More details at [`spec/migrations/README.md`](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/spec/migrations/README.md). | +| `Gemfile` | `spec/dependencies/`, `spec/sidekiq/` | RSpec | | +| `lib/` | `spec/lib/` | RSpec | | +| `lib/tasks/` | `spec/tasks/` | RSpec | | +| `rubocop/` | `spec/rubocop/` | RSpec | | +| `spec/factories` | `spec/factories_spec.rb` | RSpec | | ## Integration tests @@ -46,7 +67,7 @@ They're useful to test permissions, redirections, what view is rendered etc. | `app/mailers/` | `spec/mailers/` | RSpec | | | `lib/api/` | `spec/requests/api/` | RSpec | | | `lib/ci/api/` | `spec/requests/ci/api/` | RSpec | | -| `app/assets/javascripts/` | `spec/javascripts/` | Karma | More details in the [JavaScript](#javascript) section. | +| `app/assets/javascripts/` | `spec/javascripts/`, `spec/frontend/` | Karma & Jest | More details in the [Frontend Testing guide](frontend_testing.md) section. | ### About controller tests @@ -157,10 +178,10 @@ Every new feature should come with a [test plan]. | ---------- | -------------- | ----- | | `qa/qa/specs/features/` | [Capybara] + [RSpec] + Custom QA framework | Tests should be placed under their corresponding [Product category] | -> See [end-to-end tests](end_to_end_tests.md) for more information. +> See [end-to-end tests](end_to_end/index.md) for more information. Note that `qa/spec` contains unit tests of the QA framework itself, not to be -confused with the application's [unit tests](#unit-tests) or +confused with the application's [unit tests](#unit-tests) or [end-to-end tests](#black-box-tests-at-the-system-level-aka-end-to-end-tests). [multiple pieces]: ../architecture.md#components @@ -210,7 +231,7 @@ trade-off: - Integration tests are a bit more expensive, but don't abuse them. A system test is often better than an integration test that is stubbing a lot of internals. - System tests are expensive (compared to unit tests), even more if they require - a JavaScript driver. Make sure to follow the guidelines in the [Speed](#test-speed) + a JavaScript driver. Make sure to follow the guidelines in the [Speed](best_practices.md#test-speed) section. Another way to see it is to think about the "cost of tests", this is well @@ -234,6 +255,8 @@ you should write an integration test using Jasmine. [big]: https://twitter.com/timbray/status/822470746773409794 [picture]: https://twitter.com/withzombies/status/829716565834752000 [tests-cost]: https://medium.com/table-xi/high-cost-tests-and-high-value-tests-a86e27a54df#.2ulyh3a4e +[RSpec]: https://github.com/rspec/rspec-rails#feature-specs +[Capybara]: https://github.com/teamcapybara/capybara --- diff --git a/doc/development/understanding_explain_plans.md b/doc/development/understanding_explain_plans.md index 01a0044f096..bfbb7be70e3 100644 --- a/doc/development/understanding_explain_plans.md +++ b/doc/development/understanding_explain_plans.md @@ -80,8 +80,9 @@ Planning time: 2.861 ms Execution time: 3428.596 ms ``` -For more information, refer to the official [EXPLAIN -documentation](https://www.postgresql.org/docs/current/static/sql-explain.html). +For more information, refer to the official +[`EXPLAIN` documentation](https://www.postgresql.org/docs/current/sql-explain.html) +and [using `EXPLAIN` guide](https://www.postgresql.org/docs/current/using-explain.html). ## Nodes @@ -653,9 +654,39 @@ and related tools such as: - <https://explain.depesz.com/> - <http://tatiyants.com/postgres-query-plan-visualization/> -GitLab employees can also use our chatops solution, available in Slack using the -`/chatops` slash command. You can use chatops to get a query plan by running the -following: + +## Producing query plans + +There are a few ways to get the output of a query plan. Of course you +can directly run the `EXPLAIN` query in the `psql` console, or you can +follow one of the other options below. + +### Rails console + +Using the [`activerecord-explain-analyze`](https://github.com/6/activerecord-explain-analyze) +you can directly generate the query plan from the Rails console: + +```ruby +pry(main)> require 'activerecord-explain-analyze' +=> true +pry(main)> Project.where('build_timeout > ?', 3600).explain(analyze: true) + Project Load (1.9ms) SELECT "projects".* FROM "projects" WHERE (build_timeout > 3600) + ↳ (pry):12 +=> EXPLAIN for: SELECT "projects".* FROM "projects" WHERE (build_timeout > 3600) +Seq Scan on public.projects (cost=0.00..2.17 rows=1 width=742) (actual time=0.040..0.041 rows=0 loops=1) + Output: id, name, path, description, created_at, updated_at, creator_id, namespace_id, ... + Filter: (projects.build_timeout > 3600) + Rows Removed by Filter: 14 + Buffers: shared hit=2 +Planning time: 0.411 ms +Execution time: 0.113 ms +``` + +### Chatops + +[GitLab employees can also use our chatops solution, available in Slack using the +`/chatops` slash command](chatops_on_gitlabcom.md). +You can use chatops to get a query plan by running the following: ``` /chatops run explain SELECT COUNT(*) FROM projects WHERE visibility_level IN (0, 20) @@ -674,3 +705,9 @@ For more information about the available options, run: ``` /chatops run explain --help ``` + +## Further reading + +A more extensive guide on understanding query plans can be found in +the [presentation](https://www.dalibo.org/_media/understanding_explain.pdf) +from [Dalibo.org](https://www.dalibo.org/en/). diff --git a/doc/development/ux_guide/img/animation-autoscroll.gif b/doc/development/ux_guide/img/animation-autoscroll.gif Binary files differdeleted file mode 100644 index 155b0234c64..00000000000 --- a/doc/development/ux_guide/img/animation-autoscroll.gif +++ /dev/null diff --git a/doc/development/ux_guide/img/animation-dropdown.gif b/doc/development/ux_guide/img/animation-dropdown.gif Binary files differdeleted file mode 100644 index c9b31d26165..00000000000 --- a/doc/development/ux_guide/img/animation-dropdown.gif +++ /dev/null diff --git a/doc/development/ux_guide/img/animation-hover.gif b/doc/development/ux_guide/img/animation-hover.gif Binary files differdeleted file mode 100644 index 37ad9c76828..00000000000 --- a/doc/development/ux_guide/img/animation-hover.gif +++ /dev/null diff --git a/doc/development/ux_guide/img/animation-quickupdate.gif b/doc/development/ux_guide/img/animation-quickupdate.gif Binary files differdeleted file mode 100644 index 8db70bc3d24..00000000000 --- a/doc/development/ux_guide/img/animation-quickupdate.gif +++ /dev/null diff --git a/doc/development/ux_guide/img/animation-reorder.gif b/doc/development/ux_guide/img/animation-reorder.gif Binary files differdeleted file mode 100644 index ccdeb3d396f..00000000000 --- a/doc/development/ux_guide/img/animation-reorder.gif +++ /dev/null diff --git a/doc/development/ux_guide/img/button-close--active.png b/doc/development/ux_guide/img/button-close--active.png Binary files differdeleted file mode 100644 index 97a5301fb91..00000000000 --- a/doc/development/ux_guide/img/button-close--active.png +++ /dev/null diff --git a/doc/development/ux_guide/img/button-close--hover.png b/doc/development/ux_guide/img/button-close--hover.png Binary files differdeleted file mode 100644 index 6b8fdf5695b..00000000000 --- a/doc/development/ux_guide/img/button-close--hover.png +++ /dev/null diff --git a/doc/development/ux_guide/img/button-close--resting.png b/doc/development/ux_guide/img/button-close--resting.png Binary files differdeleted file mode 100644 index 5679b51687c..00000000000 --- a/doc/development/ux_guide/img/button-close--resting.png +++ /dev/null diff --git a/doc/development/ux_guide/img/button-danger--active.png b/doc/development/ux_guide/img/button-danger--active.png Binary files differdeleted file mode 100644 index 6a9aab0fcc2..00000000000 --- a/doc/development/ux_guide/img/button-danger--active.png +++ /dev/null diff --git a/doc/development/ux_guide/img/button-danger--hover.png b/doc/development/ux_guide/img/button-danger--hover.png Binary files differdeleted file mode 100644 index 13e21c28779..00000000000 --- a/doc/development/ux_guide/img/button-danger--hover.png +++ /dev/null diff --git a/doc/development/ux_guide/img/button-danger--resting.png b/doc/development/ux_guide/img/button-danger--resting.png Binary files differdeleted file mode 100644 index 0ff192bc463..00000000000 --- a/doc/development/ux_guide/img/button-danger--resting.png +++ /dev/null diff --git a/doc/development/ux_guide/img/button-info--active.png b/doc/development/ux_guide/img/button-info--active.png Binary files differdeleted file mode 100644 index 12ecdc72a31..00000000000 --- a/doc/development/ux_guide/img/button-info--active.png +++ /dev/null diff --git a/doc/development/ux_guide/img/button-info--hover.png b/doc/development/ux_guide/img/button-info--hover.png Binary files differdeleted file mode 100644 index 3bf93bf2b32..00000000000 --- a/doc/development/ux_guide/img/button-info--hover.png +++ /dev/null diff --git a/doc/development/ux_guide/img/button-info--resting.png b/doc/development/ux_guide/img/button-info--resting.png Binary files differdeleted file mode 100644 index a37a37033bf..00000000000 --- a/doc/development/ux_guide/img/button-info--resting.png +++ /dev/null diff --git a/doc/development/ux_guide/img/button-primary.png b/doc/development/ux_guide/img/button-primary.png Binary files differdeleted file mode 100644 index eda5ed84aec..00000000000 --- a/doc/development/ux_guide/img/button-primary.png +++ /dev/null diff --git a/doc/development/ux_guide/img/button-secondary.png b/doc/development/ux_guide/img/button-secondary.png Binary files differdeleted file mode 100644 index 26d4e8cf43d..00000000000 --- a/doc/development/ux_guide/img/button-secondary.png +++ /dev/null diff --git a/doc/development/ux_guide/img/button-spam--active.png b/doc/development/ux_guide/img/button-spam--active.png Binary files differdeleted file mode 100644 index a9e115f49c1..00000000000 --- a/doc/development/ux_guide/img/button-spam--active.png +++ /dev/null diff --git a/doc/development/ux_guide/img/button-spam--hover.png b/doc/development/ux_guide/img/button-spam--hover.png Binary files differdeleted file mode 100644 index 3b2c16430a6..00000000000 --- a/doc/development/ux_guide/img/button-spam--hover.png +++ /dev/null diff --git a/doc/development/ux_guide/img/button-spam--resting.png b/doc/development/ux_guide/img/button-spam--resting.png Binary files differdeleted file mode 100644 index 4f9f18ca68a..00000000000 --- a/doc/development/ux_guide/img/button-spam--resting.png +++ /dev/null diff --git a/doc/development/ux_guide/img/button-success--active.png b/doc/development/ux_guide/img/button-success--active.png Binary files differdeleted file mode 100644 index b99f6f5e70e..00000000000 --- a/doc/development/ux_guide/img/button-success--active.png +++ /dev/null diff --git a/doc/development/ux_guide/img/button-success--hover.png b/doc/development/ux_guide/img/button-success--hover.png Binary files differdeleted file mode 100644 index 0d0a61c679a..00000000000 --- a/doc/development/ux_guide/img/button-success--hover.png +++ /dev/null diff --git a/doc/development/ux_guide/img/button-success--resting.png b/doc/development/ux_guide/img/button-success--resting.png Binary files differdeleted file mode 100644 index 53b955c650a..00000000000 --- a/doc/development/ux_guide/img/button-success--resting.png +++ /dev/null diff --git a/doc/development/ux_guide/img/button-success-secondary--active.png b/doc/development/ux_guide/img/button-success-secondary--active.png Binary files differdeleted file mode 100644 index 333a91f2217..00000000000 --- a/doc/development/ux_guide/img/button-success-secondary--active.png +++ /dev/null diff --git a/doc/development/ux_guide/img/button-success-secondary--hover.png b/doc/development/ux_guide/img/button-success-secondary--hover.png Binary files differdeleted file mode 100644 index 0cce59212e3..00000000000 --- a/doc/development/ux_guide/img/button-success-secondary--hover.png +++ /dev/null diff --git a/doc/development/ux_guide/img/button-success-secondary--resting.png b/doc/development/ux_guide/img/button-success-secondary--resting.png Binary files differdeleted file mode 100644 index 2779a4949f8..00000000000 --- a/doc/development/ux_guide/img/button-success-secondary--resting.png +++ /dev/null diff --git a/doc/development/ux_guide/img/button-warning--active.png b/doc/development/ux_guide/img/button-warning--active.png Binary files differdeleted file mode 100644 index f5760cd7c12..00000000000 --- a/doc/development/ux_guide/img/button-warning--active.png +++ /dev/null diff --git a/doc/development/ux_guide/img/button-warning--hover.png b/doc/development/ux_guide/img/button-warning--hover.png Binary files differdeleted file mode 100644 index a1f4c5cbcc6..00000000000 --- a/doc/development/ux_guide/img/button-warning--hover.png +++ /dev/null diff --git a/doc/development/ux_guide/img/button-warning--resting.png b/doc/development/ux_guide/img/button-warning--resting.png Binary files differdeleted file mode 100644 index 3d62fed5930..00000000000 --- a/doc/development/ux_guide/img/button-warning--resting.png +++ /dev/null diff --git a/doc/development/ux_guide/img/color-blue.png b/doc/development/ux_guide/img/color-blue.png Binary files differdeleted file mode 100644 index 77c1a2cab31..00000000000 --- a/doc/development/ux_guide/img/color-blue.png +++ /dev/null diff --git a/doc/development/ux_guide/img/color-green.png b/doc/development/ux_guide/img/color-green.png Binary files differdeleted file mode 100644 index 51600584c96..00000000000 --- a/doc/development/ux_guide/img/color-green.png +++ /dev/null diff --git a/doc/development/ux_guide/img/color-grey.png b/doc/development/ux_guide/img/color-grey.png Binary files differdeleted file mode 100644 index f0f0b9d80bb..00000000000 --- a/doc/development/ux_guide/img/color-grey.png +++ /dev/null diff --git a/doc/development/ux_guide/img/color-orange.png b/doc/development/ux_guide/img/color-orange.png Binary files differdeleted file mode 100644 index f16435c0a64..00000000000 --- a/doc/development/ux_guide/img/color-orange.png +++ /dev/null diff --git a/doc/development/ux_guide/img/color-red.png b/doc/development/ux_guide/img/color-red.png Binary files differdeleted file mode 100644 index 5008e75da78..00000000000 --- a/doc/development/ux_guide/img/color-red.png +++ /dev/null diff --git a/doc/development/ux_guide/img/color-textprimary.png b/doc/development/ux_guide/img/color-textprimary.png Binary files differdeleted file mode 100644 index 90f2821f0cf..00000000000 --- a/doc/development/ux_guide/img/color-textprimary.png +++ /dev/null diff --git a/doc/development/ux_guide/img/color-textsecondary.png b/doc/development/ux_guide/img/color-textsecondary.png Binary files differdeleted file mode 100644 index 61cb8a13c45..00000000000 --- a/doc/development/ux_guide/img/color-textsecondary.png +++ /dev/null diff --git a/doc/development/ux_guide/img/components-alerts.png b/doc/development/ux_guide/img/components-alerts.png Binary files differdeleted file mode 100644 index 66a43ac69e1..00000000000 --- a/doc/development/ux_guide/img/components-alerts.png +++ /dev/null diff --git a/doc/development/ux_guide/img/components-anchorlinks.png b/doc/development/ux_guide/img/components-anchorlinks.png Binary files differdeleted file mode 100644 index bd8d30f5905..00000000000 --- a/doc/development/ux_guide/img/components-anchorlinks.png +++ /dev/null diff --git a/doc/development/ux_guide/img/components-contentblock.png b/doc/development/ux_guide/img/components-contentblock.png Binary files differdeleted file mode 100644 index 58d87729701..00000000000 --- a/doc/development/ux_guide/img/components-contentblock.png +++ /dev/null diff --git a/doc/development/ux_guide/img/components-counts.png b/doc/development/ux_guide/img/components-counts.png Binary files differdeleted file mode 100644 index 19280e988a0..00000000000 --- a/doc/development/ux_guide/img/components-counts.png +++ /dev/null diff --git a/doc/development/ux_guide/img/components-coverblock.png b/doc/development/ux_guide/img/components-coverblock.png Binary files differdeleted file mode 100644 index 61160de5613..00000000000 --- a/doc/development/ux_guide/img/components-coverblock.png +++ /dev/null diff --git a/doc/development/ux_guide/img/components-dateexact.png b/doc/development/ux_guide/img/components-dateexact.png Binary files differdeleted file mode 100644 index cc1fb8216bf..00000000000 --- a/doc/development/ux_guide/img/components-dateexact.png +++ /dev/null diff --git a/doc/development/ux_guide/img/components-daterelative.png b/doc/development/ux_guide/img/components-daterelative.png Binary files differdeleted file mode 100644 index 4954dfb51b3..00000000000 --- a/doc/development/ux_guide/img/components-daterelative.png +++ /dev/null diff --git a/doc/development/ux_guide/img/components-dropdown.png b/doc/development/ux_guide/img/components-dropdown.png Binary files differdeleted file mode 100644 index 7f9a701c089..00000000000 --- a/doc/development/ux_guide/img/components-dropdown.png +++ /dev/null diff --git a/doc/development/ux_guide/img/components-fileholder.png b/doc/development/ux_guide/img/components-fileholder.png Binary files differdeleted file mode 100644 index 5bf8565346a..00000000000 --- a/doc/development/ux_guide/img/components-fileholder.png +++ /dev/null diff --git a/doc/development/ux_guide/img/components-horizontalform.png b/doc/development/ux_guide/img/components-horizontalform.png Binary files differdeleted file mode 100644 index e6cbc69d20a..00000000000 --- a/doc/development/ux_guide/img/components-horizontalform.png +++ /dev/null diff --git a/doc/development/ux_guide/img/components-listinsidepanel.png b/doc/development/ux_guide/img/components-listinsidepanel.png Binary files differdeleted file mode 100644 index 6b773a19954..00000000000 --- a/doc/development/ux_guide/img/components-listinsidepanel.png +++ /dev/null diff --git a/doc/development/ux_guide/img/components-listwithavatar.png b/doc/development/ux_guide/img/components-listwithavatar.png Binary files differdeleted file mode 100644 index f6db575433c..00000000000 --- a/doc/development/ux_guide/img/components-listwithavatar.png +++ /dev/null diff --git a/doc/development/ux_guide/img/components-listwithhover.png b/doc/development/ux_guide/img/components-listwithhover.png Binary files differdeleted file mode 100644 index 0826848ff34..00000000000 --- a/doc/development/ux_guide/img/components-listwithhover.png +++ /dev/null diff --git a/doc/development/ux_guide/img/components-panels.png b/doc/development/ux_guide/img/components-panels.png Binary files differdeleted file mode 100644 index c1391ca07e5..00000000000 --- a/doc/development/ux_guide/img/components-panels.png +++ /dev/null diff --git a/doc/development/ux_guide/img/components-referencehover.png b/doc/development/ux_guide/img/components-referencehover.png Binary files differdeleted file mode 100644 index af5405d3e0b..00000000000 --- a/doc/development/ux_guide/img/components-referencehover.png +++ /dev/null diff --git a/doc/development/ux_guide/img/components-referenceissues.png b/doc/development/ux_guide/img/components-referenceissues.png Binary files differdeleted file mode 100644 index 4e175dc169d..00000000000 --- a/doc/development/ux_guide/img/components-referenceissues.png +++ /dev/null diff --git a/doc/development/ux_guide/img/components-referencelabels.png b/doc/development/ux_guide/img/components-referencelabels.png Binary files differdeleted file mode 100644 index 29a985bbaa0..00000000000 --- a/doc/development/ux_guide/img/components-referencelabels.png +++ /dev/null diff --git a/doc/development/ux_guide/img/components-referencemilestone.png b/doc/development/ux_guide/img/components-referencemilestone.png Binary files differdeleted file mode 100644 index 47c76a9d60f..00000000000 --- a/doc/development/ux_guide/img/components-referencemilestone.png +++ /dev/null diff --git a/doc/development/ux_guide/img/components-referencemrs.png b/doc/development/ux_guide/img/components-referencemrs.png Binary files differdeleted file mode 100644 index 9a5032a1516..00000000000 --- a/doc/development/ux_guide/img/components-referencemrs.png +++ /dev/null diff --git a/doc/development/ux_guide/img/components-referencepeople.png b/doc/development/ux_guide/img/components-referencepeople.png Binary files differdeleted file mode 100644 index f9ef11be853..00000000000 --- a/doc/development/ux_guide/img/components-referencepeople.png +++ /dev/null diff --git a/doc/development/ux_guide/img/components-rowcontentblock.png b/doc/development/ux_guide/img/components-rowcontentblock.png Binary files differdeleted file mode 100644 index c66a50f9564..00000000000 --- a/doc/development/ux_guide/img/components-rowcontentblock.png +++ /dev/null diff --git a/doc/development/ux_guide/img/components-searchbox.png b/doc/development/ux_guide/img/components-searchbox.png Binary files differdeleted file mode 100644 index 5c19024bfb0..00000000000 --- a/doc/development/ux_guide/img/components-searchbox.png +++ /dev/null diff --git a/doc/development/ux_guide/img/components-searchboxscoped.png b/doc/development/ux_guide/img/components-searchboxscoped.png Binary files differdeleted file mode 100644 index d4a35977658..00000000000 --- a/doc/development/ux_guide/img/components-searchboxscoped.png +++ /dev/null diff --git a/doc/development/ux_guide/img/components-simplelist.png b/doc/development/ux_guide/img/components-simplelist.png Binary files differdeleted file mode 100644 index 8d11c674e84..00000000000 --- a/doc/development/ux_guide/img/components-simplelist.png +++ /dev/null diff --git a/doc/development/ux_guide/img/components-table.png b/doc/development/ux_guide/img/components-table.png Binary files differdeleted file mode 100644 index cedc55758a9..00000000000 --- a/doc/development/ux_guide/img/components-table.png +++ /dev/null diff --git a/doc/development/ux_guide/img/components-verticalform.png b/doc/development/ux_guide/img/components-verticalform.png Binary files differdeleted file mode 100644 index 489ae6f862f..00000000000 --- a/doc/development/ux_guide/img/components-verticalform.png +++ /dev/null diff --git a/doc/development/ux_guide/img/cursors-default.png b/doc/development/ux_guide/img/cursors-default.png Binary files differdeleted file mode 100644 index c188ec4e351..00000000000 --- a/doc/development/ux_guide/img/cursors-default.png +++ /dev/null diff --git a/doc/development/ux_guide/img/cursors-ibeam.png b/doc/development/ux_guide/img/cursors-ibeam.png Binary files differdeleted file mode 100644 index 86f28639982..00000000000 --- a/doc/development/ux_guide/img/cursors-ibeam.png +++ /dev/null diff --git a/doc/development/ux_guide/img/cursors-move.png b/doc/development/ux_guide/img/cursors-move.png Binary files differdeleted file mode 100644 index a9c610eaa88..00000000000 --- a/doc/development/ux_guide/img/cursors-move.png +++ /dev/null diff --git a/doc/development/ux_guide/img/cursors-panclosed.png b/doc/development/ux_guide/img/cursors-panclosed.png Binary files differdeleted file mode 100644 index 6d247a765ac..00000000000 --- a/doc/development/ux_guide/img/cursors-panclosed.png +++ /dev/null diff --git a/doc/development/ux_guide/img/cursors-panopened.png b/doc/development/ux_guide/img/cursors-panopened.png Binary files differdeleted file mode 100644 index 76f2eeda831..00000000000 --- a/doc/development/ux_guide/img/cursors-panopened.png +++ /dev/null diff --git a/doc/development/ux_guide/img/cursors-pointer.png b/doc/development/ux_guide/img/cursors-pointer.png Binary files differdeleted file mode 100644 index d86dd955fa7..00000000000 --- a/doc/development/ux_guide/img/cursors-pointer.png +++ /dev/null diff --git a/doc/development/ux_guide/img/features-contextualnav.png b/doc/development/ux_guide/img/features-contextualnav.png Binary files differdeleted file mode 100644 index aa816776fad..00000000000 --- a/doc/development/ux_guide/img/features-contextualnav.png +++ /dev/null diff --git a/doc/development/ux_guide/img/features-emptystates.png b/doc/development/ux_guide/img/features-emptystates.png Binary files differdeleted file mode 100644 index 50f31f5e523..00000000000 --- a/doc/development/ux_guide/img/features-emptystates.png +++ /dev/null diff --git a/doc/development/ux_guide/img/features-filters.png b/doc/development/ux_guide/img/features-filters.png Binary files differdeleted file mode 100644 index 41db76db938..00000000000 --- a/doc/development/ux_guide/img/features-filters.png +++ /dev/null diff --git a/doc/development/ux_guide/img/features-globalnav.png b/doc/development/ux_guide/img/features-globalnav.png Binary files differdeleted file mode 100644 index 73294d1b524..00000000000 --- a/doc/development/ux_guide/img/features-globalnav.png +++ /dev/null diff --git a/doc/development/ux_guide/img/harry-robison.png b/doc/development/ux_guide/img/harry-robison.png Binary files differdeleted file mode 100644 index 702a8b02262..00000000000 --- a/doc/development/ux_guide/img/harry-robison.png +++ /dev/null diff --git a/doc/development/ux_guide/img/icon-add.png b/doc/development/ux_guide/img/icon-add.png Binary files differdeleted file mode 100644 index f66525cc1b4..00000000000 --- a/doc/development/ux_guide/img/icon-add.png +++ /dev/null diff --git a/doc/development/ux_guide/img/icon-close.png b/doc/development/ux_guide/img/icon-close.png Binary files differdeleted file mode 100644 index af6c30ebe6a..00000000000 --- a/doc/development/ux_guide/img/icon-close.png +++ /dev/null diff --git a/doc/development/ux_guide/img/icon-edit.png b/doc/development/ux_guide/img/icon-edit.png Binary files differdeleted file mode 100644 index b9649f4aeec..00000000000 --- a/doc/development/ux_guide/img/icon-edit.png +++ /dev/null diff --git a/doc/development/ux_guide/img/icon-notification.png b/doc/development/ux_guide/img/icon-notification.png Binary files differdeleted file mode 100644 index 5cf8f8ab59a..00000000000 --- a/doc/development/ux_guide/img/icon-notification.png +++ /dev/null diff --git a/doc/development/ux_guide/img/icon-rss.png b/doc/development/ux_guide/img/icon-rss.png Binary files differdeleted file mode 100644 index 7e2987a2656..00000000000 --- a/doc/development/ux_guide/img/icon-rss.png +++ /dev/null diff --git a/doc/development/ux_guide/img/icon-spec.png b/doc/development/ux_guide/img/icon-spec.png Binary files differdeleted file mode 100644 index 5bb85c5be98..00000000000 --- a/doc/development/ux_guide/img/icon-spec.png +++ /dev/null diff --git a/doc/development/ux_guide/img/icon-subscribe.png b/doc/development/ux_guide/img/icon-subscribe.png Binary files differdeleted file mode 100644 index 7e2f5e6a1c6..00000000000 --- a/doc/development/ux_guide/img/icon-subscribe.png +++ /dev/null diff --git a/doc/development/ux_guide/img/icon-trash.png b/doc/development/ux_guide/img/icon-trash.png Binary files differdeleted file mode 100644 index bc46638fb2e..00000000000 --- a/doc/development/ux_guide/img/icon-trash.png +++ /dev/null diff --git a/doc/development/ux_guide/img/illustration-size-large-horizontal.png b/doc/development/ux_guide/img/illustration-size-large-horizontal.png Binary files differdeleted file mode 100644 index 8aa835adccc..00000000000 --- a/doc/development/ux_guide/img/illustration-size-large-horizontal.png +++ /dev/null diff --git a/doc/development/ux_guide/img/illustration-size-large-vertical.png b/doc/development/ux_guide/img/illustration-size-large-vertical.png Binary files differdeleted file mode 100644 index 813b6a065e5..00000000000 --- a/doc/development/ux_guide/img/illustration-size-large-vertical.png +++ /dev/null diff --git a/doc/development/ux_guide/img/illustration-size-medium.png b/doc/development/ux_guide/img/illustration-size-medium.png Binary files differdeleted file mode 100644 index 55cfe1dcb91..00000000000 --- a/doc/development/ux_guide/img/illustration-size-medium.png +++ /dev/null diff --git a/doc/development/ux_guide/img/illustration-size-small.png b/doc/development/ux_guide/img/illustration-size-small.png Binary files differdeleted file mode 100644 index 0124f58f48e..00000000000 --- a/doc/development/ux_guide/img/illustration-size-small.png +++ /dev/null diff --git a/doc/development/ux_guide/img/illustrations-border-radius.png b/doc/development/ux_guide/img/illustrations-border-radius.png Binary files differdeleted file mode 100644 index 4e2fef5c7f5..00000000000 --- a/doc/development/ux_guide/img/illustrations-border-radius.png +++ /dev/null diff --git a/doc/development/ux_guide/img/illustrations-caps-do.png b/doc/development/ux_guide/img/illustrations-caps-do.png Binary files differdeleted file mode 100644 index f1030769b94..00000000000 --- a/doc/development/ux_guide/img/illustrations-caps-do.png +++ /dev/null diff --git a/doc/development/ux_guide/img/illustrations-caps-don't.png b/doc/development/ux_guide/img/illustrations-caps-don't.png Binary files differdeleted file mode 100644 index ab7abcaaf6f..00000000000 --- a/doc/development/ux_guide/img/illustrations-caps-don't.png +++ /dev/null diff --git a/doc/development/ux_guide/img/illustrations-color-grey.png b/doc/development/ux_guide/img/illustrations-color-grey.png Binary files differdeleted file mode 100644 index 63855026c2b..00000000000 --- a/doc/development/ux_guide/img/illustrations-color-grey.png +++ /dev/null diff --git a/doc/development/ux_guide/img/illustrations-color-orange.png b/doc/development/ux_guide/img/illustrations-color-orange.png Binary files differdeleted file mode 100644 index 96765c8c28c..00000000000 --- a/doc/development/ux_guide/img/illustrations-color-orange.png +++ /dev/null diff --git a/doc/development/ux_guide/img/illustrations-color-purple.png b/doc/development/ux_guide/img/illustrations-color-purple.png Binary files differdeleted file mode 100644 index 745d2c853ba..00000000000 --- a/doc/development/ux_guide/img/illustrations-color-purple.png +++ /dev/null diff --git a/doc/development/ux_guide/img/illustrations-geometric.png b/doc/development/ux_guide/img/illustrations-geometric.png Binary files differdeleted file mode 100644 index 33f05547bac..00000000000 --- a/doc/development/ux_guide/img/illustrations-geometric.png +++ /dev/null diff --git a/doc/development/ux_guide/img/illustrations-palette-oragne.png b/doc/development/ux_guide/img/illustrations-palette-oragne.png Binary files differdeleted file mode 100644 index 15f35912646..00000000000 --- a/doc/development/ux_guide/img/illustrations-palette-oragne.png +++ /dev/null diff --git a/doc/development/ux_guide/img/illustrations-palette-purple.png b/doc/development/ux_guide/img/illustrations-palette-purple.png Binary files differdeleted file mode 100644 index e0f5839705e..00000000000 --- a/doc/development/ux_guide/img/illustrations-palette-purple.png +++ /dev/null diff --git a/doc/development/ux_guide/img/james-mackey.png b/doc/development/ux_guide/img/james-mackey.png Binary files differdeleted file mode 100644 index f51a45c437b..00000000000 --- a/doc/development/ux_guide/img/james-mackey.png +++ /dev/null diff --git a/doc/development/ux_guide/img/karolina-plaskaty.png b/doc/development/ux_guide/img/karolina-plaskaty.png Binary files differdeleted file mode 100644 index d1c9528dd5a..00000000000 --- a/doc/development/ux_guide/img/karolina-plaskaty.png +++ /dev/null diff --git a/doc/development/ux_guide/img/matthieu-poirier.png b/doc/development/ux_guide/img/matthieu-poirier.png Binary files differdeleted file mode 100644 index 0ecc2d670d6..00000000000 --- a/doc/development/ux_guide/img/matthieu-poirier.png +++ /dev/null diff --git a/doc/development/ux_guide/img/modals-general-confimation-dialog.png b/doc/development/ux_guide/img/modals-general-confimation-dialog.png Binary files differdeleted file mode 100644 index 4ea0ea10ca7..00000000000 --- a/doc/development/ux_guide/img/modals-general-confimation-dialog.png +++ /dev/null diff --git a/doc/development/ux_guide/img/modals-layout-for-modals.png b/doc/development/ux_guide/img/modals-layout-for-modals.png Binary files differdeleted file mode 100644 index c481edd8250..00000000000 --- a/doc/development/ux_guide/img/modals-layout-for-modals.png +++ /dev/null diff --git a/doc/development/ux_guide/img/modals-special-confimation-dialog.png b/doc/development/ux_guide/img/modals-special-confimation-dialog.png Binary files differdeleted file mode 100644 index d966010158b..00000000000 --- a/doc/development/ux_guide/img/modals-special-confimation-dialog.png +++ /dev/null diff --git a/doc/development/ux_guide/img/modals-three-buttons.png b/doc/development/ux_guide/img/modals-three-buttons.png Binary files differdeleted file mode 100644 index 157d1b650bf..00000000000 --- a/doc/development/ux_guide/img/modals-three-buttons.png +++ /dev/null diff --git a/doc/development/ux_guide/img/monospacefont-sample.png b/doc/development/ux_guide/img/monospacefont-sample.png Binary files differdeleted file mode 100644 index 1cd290b713c..00000000000 --- a/doc/development/ux_guide/img/monospacefont-sample.png +++ /dev/null diff --git a/doc/development/ux_guide/img/nazim-ramesh.png b/doc/development/ux_guide/img/nazim-ramesh.png Binary files differdeleted file mode 100644 index dad2b37010b..00000000000 --- a/doc/development/ux_guide/img/nazim-ramesh.png +++ /dev/null diff --git a/doc/development/ux_guide/img/popover-placement-above.png b/doc/development/ux_guide/img/popover-placement-above.png Binary files differdeleted file mode 100644 index 84c9c878ec2..00000000000 --- a/doc/development/ux_guide/img/popover-placement-above.png +++ /dev/null diff --git a/doc/development/ux_guide/img/popover-placement-below.png b/doc/development/ux_guide/img/popover-placement-below.png Binary files differdeleted file mode 100644 index f6f18199ab6..00000000000 --- a/doc/development/ux_guide/img/popover-placement-below.png +++ /dev/null diff --git a/doc/development/ux_guide/img/skeleton-loading.gif b/doc/development/ux_guide/img/skeleton-loading.gif Binary files differdeleted file mode 100644 index 5877139171d..00000000000 --- a/doc/development/ux_guide/img/skeleton-loading.gif +++ /dev/null diff --git a/doc/development/ux_guide/img/sourcesanspro-sample.png b/doc/development/ux_guide/img/sourcesanspro-sample.png Binary files differdeleted file mode 100644 index f7ecf0c7c66..00000000000 --- a/doc/development/ux_guide/img/sourcesanspro-sample.png +++ /dev/null diff --git a/doc/development/ux_guide/img/steven-lyons.png b/doc/development/ux_guide/img/steven-lyons.png Binary files differdeleted file mode 100644 index 2efe1d0b168..00000000000 --- a/doc/development/ux_guide/img/steven-lyons.png +++ /dev/null diff --git a/doc/development/ux_guide/img/surfaces-contentitemtitle.png b/doc/development/ux_guide/img/surfaces-contentitemtitle.png Binary files differdeleted file mode 100644 index f6cd212ecfd..00000000000 --- a/doc/development/ux_guide/img/surfaces-contentitemtitle.png +++ /dev/null diff --git a/doc/development/ux_guide/img/surfaces-header.png b/doc/development/ux_guide/img/surfaces-header.png Binary files differdeleted file mode 100644 index ba616388003..00000000000 --- a/doc/development/ux_guide/img/surfaces-header.png +++ /dev/null diff --git a/doc/development/ux_guide/img/surfaces-systeminformationblock.png b/doc/development/ux_guide/img/surfaces-systeminformationblock.png Binary files differdeleted file mode 100644 index f3313add2b8..00000000000 --- a/doc/development/ux_guide/img/surfaces-systeminformationblock.png +++ /dev/null diff --git a/doc/development/ux_guide/img/surfaces-ux.png b/doc/development/ux_guide/img/surfaces-ux.png Binary files differdeleted file mode 100644 index eaa7f70c0c7..00000000000 --- a/doc/development/ux_guide/img/surfaces-ux.png +++ /dev/null diff --git a/doc/development/ux_guide/img/tooltip-placement.png b/doc/development/ux_guide/img/tooltip-placement.png Binary files differdeleted file mode 100644 index da49c192878..00000000000 --- a/doc/development/ux_guide/img/tooltip-placement.png +++ /dev/null diff --git a/doc/development/ux_guide/img/tooltip-usage.png b/doc/development/ux_guide/img/tooltip-usage.png Binary files differdeleted file mode 100644 index 4f5884c4b48..00000000000 --- a/doc/development/ux_guide/img/tooltip-usage.png +++ /dev/null diff --git a/doc/development/ux_guide/users.md b/doc/development/ux_guide/users.md index 11bac11257c..3aecbbffd73 100644 --- a/doc/development/ux_guide/users.md +++ b/doc/development/ux_guide/users.md @@ -1,5 +1,5 @@ --- -redirect_to: 'https://design.gitlab.com/getting-started/personas/' +redirect_to: 'https://about.gitlab.com/handbook/marketing/product-marketing/roles-personas/' --- -The content of this document was moved into the [GitLab Design System](https://design.gitlab.com/). +This document was moved to [another location](https://about.gitlab.com/handbook/marketing/product-marketing/roles-personas/). diff --git a/doc/getting-started/subscription.md b/doc/getting-started/subscription.md new file mode 100644 index 00000000000..65999183d4a --- /dev/null +++ b/doc/getting-started/subscription.md @@ -0,0 +1,3 @@ +--- +redirect_to: '../subscriptions/index.md' +---
\ No newline at end of file diff --git a/doc/git_hooks/git_hooks.md b/doc/git_hooks/git_hooks.md new file mode 100644 index 00000000000..b251e58410a --- /dev/null +++ b/doc/git_hooks/git_hooks.md @@ -0,0 +1,5 @@ +--- +redirect_to: '../push_rules/push_rules.md' +--- + +This document was moved to [another location](../push_rules/push_rules.md) diff --git a/doc/gitlab-basics/README.md b/doc/gitlab-basics/README.md index 4e15f7cfd49..00ac6d6b650 100644 --- a/doc/gitlab-basics/README.md +++ b/doc/gitlab-basics/README.md @@ -2,18 +2,36 @@ comments: false --- -# GitLab basics - -Step-by-step guides on the basics of working with Git and GitLab. - -- [Command line basics](command-line-commands.md) -- [Start using Git on the command line](start-using-git.md) -- [Create and add your SSH Keys](create-your-ssh-keys.md) -- [Create a project](create-project.md) -- [Create a group](../user/group/index.md#create-a-new-group) -- [Create a branch](create-branch.md) -- [Fork a project](fork-project.md) -- [Add a file](add-file.md) -- [Add an image](add-image.md) -- [Create an issue](../user/project/issues/create_new_issue.md) -- [Create a merge request](add-merge-request.md) +# GitLab basics guides + +This section provides resources to help you start with GitLab by focusing on basic functionality. + +This documentation is split into the following groups: + +- [GitLab-specific functionality](#gitlab-basics), for basic GitLab features. +- [General Git functionality](#git-basics), for working with Git in conjunction with GitLab. + +## GitLab basics + +The following are guides to basic GitLab functionality: + +- [Create and add your SSH public key](create-your-ssh-keys.md), for enabling Git over SSH. +- [Create a project](create-project.md), to start using GitLab. +- [Create a group](../user/group/index.md#create-a-new-group), to combine and administer projects together. +- [Create a branch](create-branch.md), to make changes to files stored in a project's repository. +- [Fork a project](fork-project.md), to duplicate projects so they can be worked on in parallel. +- [Add a file](add-file.md), to add new files to a project's repository. +- [Add an image](add-image.md), to add new images to a project's repository. +- [Create an issue](../user/project/issues/create_new_issue.md), to start collaborating within a project. +- [Create a merge request](add-merge-request.md), to request changes made in a branch be merged into a project's repository. + +## Git basics + +If you're familiar with Git on the command line, you can interact with your GitLab projects just as you would with any other Git repository. + +These resources will help get further acclimated to working on the command line. + +- [Start using Git on the command line](start-using-git.md), for some simple Git commands. +- [Command line basics](command-line-commands.md), to create and edit files using the command line. + +More Git resources are available at GitLab's [Git documentation](../topics/git/index.md). diff --git a/doc/gitlab-basics/add-merge-request.md b/doc/gitlab-basics/add-merge-request.md index 5cc014419ad..7bb2e014738 100644 --- a/doc/gitlab-basics/add-merge-request.md +++ b/doc/gitlab-basics/add-merge-request.md @@ -8,7 +8,7 @@ request. For more information, check the --- 1. Before you start, you should have already [created a branch](create-branch.md) - and [pushed your changes](basic-git-commands.md) to GitLab. + and [pushed your changes](start-using-git.md#send-changes-to-gitlabcom) to GitLab. 1. Go to the project where you'd like to merge your changes and click on the **Merge requests** tab. 1. Click on **New merge request** on the right side of the screen. diff --git a/doc/gitlab-basics/command-line-commands.md b/doc/gitlab-basics/command-line-commands.md index a0111be0767..1cf883679d7 100644 --- a/doc/gitlab-basics/command-line-commands.md +++ b/doc/gitlab-basics/command-line-commands.md @@ -13,10 +13,12 @@ button (you'll have to paste it on your shell in the next step).  -## On the command line +## Working with project files on the command line This section has examples of some basic shell commands that you might find useful. For more information, search the web for _bash commands_. +Alternatively, you can edit files using your choice of editor (IDE) or the GitLab user interface. + ### Clone your project Go to your computer's shell and type the following command with your SSH or HTTPS URL: diff --git a/doc/gitlab-basics/create-group.md b/doc/gitlab-basics/create-group.md index 985a52d88f5..8c0d3561882 100644 --- a/doc/gitlab-basics/create-group.md +++ b/doc/gitlab-basics/create-group.md @@ -1,2 +1,5 @@ +--- +redirect_to: '../user/group/index.md#create-a-new-group' +--- This document was moved to [another location](../user/group/index.md#create-a-new-group). diff --git a/doc/gitlab-basics/create-issue.md b/doc/gitlab-basics/create-issue.md index abb163dbf18..6e2a09fc030 100644 --- a/doc/gitlab-basics/create-issue.md +++ b/doc/gitlab-basics/create-issue.md @@ -1,2 +1,5 @@ +--- +redirect_to: '../user/project/issues/index.md#issue-actions' +--- -This document was moved to [another location](../user/project/issues/index.md#new-issue). +This document was moved to [another location](../user/project/issues/index.md#issue-actions). diff --git a/doc/gitlab-basics/create-project.md b/doc/gitlab-basics/create-project.md index 978ffde84ad..a9ae4fb23f9 100644 --- a/doc/gitlab-basics/create-project.md +++ b/doc/gitlab-basics/create-project.md @@ -16,6 +16,7 @@ To create a project in GitLab: - [Import a project](../user/project/import/index.md) from a different repository, if enabled on your GitLab instance. Contact your GitLab admin if this is unavailable. + - Run [CI/CD pipelines for external repositories](../ci/ci_cd_for_external_repos/index.md). **[PREMIUM]** ## Blank projects @@ -42,7 +43,9 @@ Project templates can pre-populate your project with necessary files to get you There are two types of project templates: -- [Built-in templates](#builtin-templates), sourced from the [`project-templates`](https://gitlab.com/gitlab-org/project-templates) group. +- [Built-in templates](#built-in-templates), sourced from the following groups: + - [`project-templates`](https://gitlab.com/gitlab-org/project-templates) + - [`pages`](https://gitlab.com/pages) - [Custom project templates](#custom-project-templates-premium-only), for custom templates configured by GitLab administrators and users. ### Built-in templates @@ -50,12 +53,12 @@ There are two types of project templates: Built-in templates are project templates that are: - Developed and maintained in the - [`project-templates`](https://gitlab.com/gitlab-org/project-templates) group. + [`project-templates`](https://gitlab.com/gitlab-org/project-templates) and [`pages`](https://gitlab.com/pages) groups. - Released with GitLab. To use a built-in template on the **New project** page: -1. On the **Create from template** tab. +1. On the **Create from template** tab, select the **Built-in** tab. 1. From the list of available built-in templates, click the: - **Preview** button to look at the template source itself. - **Use template** button to start creating the project. @@ -64,7 +67,7 @@ To use a built-in template on the **New project** page: TIP: **Tip:** You can improve the existing built-in templates or contribute new ones on the -[`project-templates`](https://gitlab.com/gitlab-org/project-templates) group. +[`project-templates`](https://gitlab.com/gitlab-org/project-templates) and [`pages`](https://gitlab.com/pages) groups. ### Custom project templates **[PREMIUM ONLY]** diff --git a/doc/gitlab-basics/create-your-ssh-keys.md b/doc/gitlab-basics/create-your-ssh-keys.md index 8c2f48fb1e2..aac73d4c9c5 100644 --- a/doc/gitlab-basics/create-your-ssh-keys.md +++ b/doc/gitlab-basics/create-your-ssh-keys.md @@ -1,36 +1,22 @@ -# How to create your SSH Keys +# Create and add your SSH public key -1. The first thing you need to do is go to your [command line](start-using-git.md) - and follow the [instructions](../ssh/README.md) to generate your SSH key pair. +This topic describes how to: -1. Once you do that, login to GitLab with your credentials. -1. On the upper right corner, click on your avatar and go to your **Profile settings**. +- Create an SSH key pair to use with GitLab. +- Add the SSH public key to your GitLab account. -  +You do this to use [Git over SSH instead of Git over HTTP](https://git-scm.com/book/en/v2/Git-on-the-Server-The-Protocols). -1. Navigate to the **SSH keys** tab. +## Creating your SSH key pair -  +1. Go to your [command line](start-using-git.md). +1. Follow the [instructions](../ssh/README.md#generating-a-new-ssh-key-pair) to generate your SSH key pair. -1. Paste your **public** key that you generated in the first step in the 'Key' - box. +## Adding your SSH public key to GitLab -  +To add the SSH public key to GitLab, +see [Adding an SSH key to your GitLab account](../ssh/README.md#adding-an-ssh-key-to-your-gitlab-account). -1. Optionally, give it a descriptive title so that you can recognize it in the - event you add multiple keys. - -  - -1. Finally, click on **Add key** to add it to GitLab. You will be able to see - its fingerprint, its title and creation date. - -  - ->**Note:** -Once you add a key, you cannot edit it, only remove it. In case the paste -didn't work, you will have to remove the offending key and re-add it. - ---- - -Congratulations! You are now ready to use Git over SSH, instead of Git over HTTP! +NOTE: **Note:** +Once you add a key, you cannot edit it. If the paste +[didn't work](../ssh/README.md#testing-that-everything-is-set-up-correctly), you need to remove the key and re-add it. diff --git a/doc/gitlab-basics/fork-project.md b/doc/gitlab-basics/fork-project.md index 6c232fe6086..a128a7c7dd3 100644 --- a/doc/gitlab-basics/fork-project.md +++ b/doc/gitlab-basics/fork-project.md @@ -1,8 +1,8 @@ # How to fork a project -A fork is a copy of an original repository that you can put in another namespace -where you can experiment and apply changes that you can later decide if -publishing or not, without affecting your original project. +A fork is a copy of an original repository that you put in another namespace +where you can experiment and apply changes that you can later decide whether or +not to share, without affecting the original project. It takes just a few steps to fork a project in GitLab. diff --git a/doc/gitlab-basics/img/new_issue_button.png b/doc/gitlab-basics/img/new_issue_button.png Binary files differdeleted file mode 100644 index 3b113471f0c..00000000000 --- a/doc/gitlab-basics/img/new_issue_button.png +++ /dev/null diff --git a/doc/gitlab-basics/img/new_issue_page.png b/doc/gitlab-basics/img/new_issue_page.png Binary files differdeleted file mode 100644 index ce3e60df276..00000000000 --- a/doc/gitlab-basics/img/new_issue_page.png +++ /dev/null diff --git a/doc/gitlab-basics/img/profile_settings.png b/doc/gitlab-basics/img/profile_settings.png Binary files differdeleted file mode 100644 index b91b698fb18..00000000000 --- a/doc/gitlab-basics/img/profile_settings.png +++ /dev/null diff --git a/doc/gitlab-basics/img/profile_settings_ssh_keys.png b/doc/gitlab-basics/img/profile_settings_ssh_keys.png Binary files differdeleted file mode 100644 index 8ac603a2af9..00000000000 --- a/doc/gitlab-basics/img/profile_settings_ssh_keys.png +++ /dev/null diff --git a/doc/gitlab-basics/img/profile_settings_ssh_keys_paste_pub.png b/doc/gitlab-basics/img/profile_settings_ssh_keys_paste_pub.png Binary files differdeleted file mode 100644 index 0b1c64a72f3..00000000000 --- a/doc/gitlab-basics/img/profile_settings_ssh_keys_paste_pub.png +++ /dev/null diff --git a/doc/gitlab-basics/img/profile_settings_ssh_keys_single_key.png b/doc/gitlab-basics/img/profile_settings_ssh_keys_single_key.png Binary files differdeleted file mode 100644 index 8014f1d5301..00000000000 --- a/doc/gitlab-basics/img/profile_settings_ssh_keys_single_key.png +++ /dev/null diff --git a/doc/gitlab-basics/img/profile_settings_ssh_keys_title.png b/doc/gitlab-basics/img/profile_settings_ssh_keys_title.png Binary files differdeleted file mode 100644 index 02ca0bf7478..00000000000 --- a/doc/gitlab-basics/img/profile_settings_ssh_keys_title.png +++ /dev/null diff --git a/doc/gitlab-basics/img/public_file_link.png b/doc/gitlab-basics/img/public_file_link.png Binary files differdeleted file mode 100644 index f60df6807f4..00000000000 --- a/doc/gitlab-basics/img/public_file_link.png +++ /dev/null diff --git a/doc/gitlab-basics/start-using-git.md b/doc/gitlab-basics/start-using-git.md index e30afdf8a40..b3c5d32f2f5 100644 --- a/doc/gitlab-basics/start-using-git.md +++ b/doc/gitlab-basics/start-using-git.md @@ -71,6 +71,18 @@ git config --global --list Start using Git via the command line with the most basic commands as described below. +## Initialize a local directory for Git version control + +If you have an existing local directory that you want to *initialize* for version control, use the `init` command to instruct Git to begin tracking the directory: + +```bash +git init +``` + +This creates a `.git` directory that contains the Git configuration files. + +Once the directory has been initialized, you can [add a remote repository](#add-a-remote-repository) and [send changes to GitLab.com](#send-changes-to-gitlabcom). View the instructions on [Create a project](../gitlab-basics/create-project.html#push-to-create-a-new-project) to create a new project on GitLab with your changes. + ### Clone a repository To start working locally on an existing remote repository, @@ -140,6 +152,16 @@ To view your remote repositories, type: git remote -v ``` +### Add a remote repository + +To add a link to a remote repository: + +```bash +git remote add SOURCE-NAME REPOSITORY-PATH +``` + +You'll use this source name every time you [push changes to GitLab.com](#send-changes-to-gitlabcom), so use something easy to remember and type. + ### Create a branch To create a branch, type the following (spaces won't be recognized in the branch name, so you will need to use a hyphen or underscore): @@ -193,7 +215,7 @@ git commit -m "COMMENT TO DESCRIBE THE INTENTION OF THE COMMIT" NOTE: **Note:** The `.` character typically means _all_ in Git. -### Send changes to gitlab.com +### Send changes to GitLab.com To push all local commits to the remote repository: diff --git a/doc/gitlab-geo/README.md b/doc/gitlab-geo/README.md new file mode 100644 index 00000000000..30d21db7de5 --- /dev/null +++ b/doc/gitlab-geo/README.md @@ -0,0 +1,5 @@ +--- +redirect_to: '../administration/geo/replication/index.md' +--- + +This document was moved to [another location](../administration/geo/replication/index.md). diff --git a/doc/gitlab-geo/after_setup.md b/doc/gitlab-geo/after_setup.md new file mode 100644 index 00000000000..c8a7b9d1096 --- /dev/null +++ b/doc/gitlab-geo/after_setup.md @@ -0,0 +1,5 @@ +--- +redirect_to: '../administration/geo/replication/using_a_geo_server.md' +--- + +This document was moved to [another location](../administration/geo/replication/using_a_geo_server.md). diff --git a/doc/gitlab-geo/bring-primary-back.md b/doc/gitlab-geo/bring-primary-back.md new file mode 100644 index 00000000000..8c43f4d805f --- /dev/null +++ b/doc/gitlab-geo/bring-primary-back.md @@ -0,0 +1,5 @@ +--- +redirect_to: '../administration/geo/disaster_recovery/bring_primary_back.md' +--- + +This document was moved to [another location](../administration/geo/disaster_recovery/bring_primary_back.md). diff --git a/doc/gitlab-geo/configuration.md b/doc/gitlab-geo/configuration.md new file mode 100644 index 00000000000..b46a2caea4a --- /dev/null +++ b/doc/gitlab-geo/configuration.md @@ -0,0 +1,5 @@ +--- +redirect_to: '../administration/geo/replication/configuration.md' +--- + +This document was moved to [another location](../administration/geo/replication/configuration.md). diff --git a/doc/gitlab-geo/configuration_source.md b/doc/gitlab-geo/configuration_source.md new file mode 100644 index 00000000000..b46a2caea4a --- /dev/null +++ b/doc/gitlab-geo/configuration_source.md @@ -0,0 +1,5 @@ +--- +redirect_to: '../administration/geo/replication/configuration.md' +--- + +This document was moved to [another location](../administration/geo/replication/configuration.md). diff --git a/doc/gitlab-geo/database.md b/doc/gitlab-geo/database.md new file mode 100644 index 00000000000..b4156dc4ec6 --- /dev/null +++ b/doc/gitlab-geo/database.md @@ -0,0 +1,5 @@ +--- +redirect_to: '../administration/geo/replication/database.md' +--- + +This document was moved to [another location](../administration/geo/replication/database.md). diff --git a/doc/gitlab-geo/database_source.md b/doc/gitlab-geo/database_source.md new file mode 100644 index 00000000000..b4156dc4ec6 --- /dev/null +++ b/doc/gitlab-geo/database_source.md @@ -0,0 +1,5 @@ +--- +redirect_to: '../administration/geo/replication/database.md' +--- + +This document was moved to [another location](../administration/geo/replication/database.md). diff --git a/doc/gitlab-geo/disaster-recovery.md b/doc/gitlab-geo/disaster-recovery.md new file mode 100644 index 00000000000..d42e815a879 --- /dev/null +++ b/doc/gitlab-geo/disaster-recovery.md @@ -0,0 +1,5 @@ +--- +redirect_to: '../administration/geo/disaster_recovery/index.md' +--- + +This document was moved to [another location](../administration/geo/disaster_recovery/index.md). diff --git a/doc/gitlab-geo/docker_registry.md b/doc/gitlab-geo/docker_registry.md new file mode 100644 index 00000000000..26a708f6845 --- /dev/null +++ b/doc/gitlab-geo/docker_registry.md @@ -0,0 +1,5 @@ +--- +redirect_to: '../administration/geo/replication/docker_registry.md' +--- + +This document was moved to [another location](../administration/geo/replication/docker_registry.md). diff --git a/doc/gitlab-geo/faq.md b/doc/gitlab-geo/faq.md new file mode 100644 index 00000000000..f1952bc7e4c --- /dev/null +++ b/doc/gitlab-geo/faq.md @@ -0,0 +1,5 @@ +--- +redirect_to: '../administration/geo/replication/faq.md' +--- + +This document was moved to [another location](../administration/geo/replication/faq.md). diff --git a/doc/gitlab-geo/ha.md b/doc/gitlab-geo/ha.md new file mode 100644 index 00000000000..23ed11eaf09 --- /dev/null +++ b/doc/gitlab-geo/ha.md @@ -0,0 +1,5 @@ +--- +redirect_to: '../administration/geo/replication/high_availability.md' +--- + +This document was moved to [another location](../administration/geo/replication/high_availability.md). diff --git a/doc/gitlab-geo/object_storage.md b/doc/gitlab-geo/object_storage.md new file mode 100644 index 00000000000..1f29b7b7e8c --- /dev/null +++ b/doc/gitlab-geo/object_storage.md @@ -0,0 +1,5 @@ +--- +redirect_to: '../administration/geo/replication/object_storage.md' +--- + +This document was moved to [another location](../administration/geo/replication/object_storage.md). diff --git a/doc/gitlab-geo/planned-failover.md b/doc/gitlab-geo/planned-failover.md new file mode 100644 index 00000000000..720b6bc9424 --- /dev/null +++ b/doc/gitlab-geo/planned-failover.md @@ -0,0 +1,5 @@ +--- +redirect_to: '../administration/geo/disaster_recovery/planned_failover.md' +--- + +This document was moved to [another location](../administration/geo/disaster_recovery/planned_failover.md). diff --git a/doc/gitlab-geo/security-review.md b/doc/gitlab-geo/security-review.md new file mode 100644 index 00000000000..a0a5b0e536c --- /dev/null +++ b/doc/gitlab-geo/security-review.md @@ -0,0 +1,5 @@ +--- +redirect_to: '../administration/geo/replication/security_review.md' +--- + +This document was moved to [another location](../administration/geo/replication/security_review.md). diff --git a/doc/gitlab-geo/ssh.md b/doc/gitlab-geo/ssh.md new file mode 100644 index 00000000000..4f8db687850 --- /dev/null +++ b/doc/gitlab-geo/ssh.md @@ -0,0 +1,5 @@ +--- +redirect_to: '../administration/operations/fast_ssh_key_lookup.md' +--- + +This document was moved to [another location](../administration/operations/fast_ssh_key_lookup.md). diff --git a/doc/gitlab-geo/troubleshooting.md b/doc/gitlab-geo/troubleshooting.md new file mode 100644 index 00000000000..25fe1372c69 --- /dev/null +++ b/doc/gitlab-geo/troubleshooting.md @@ -0,0 +1,5 @@ +--- +redirect_to: '../administration/geo/replication/troubleshooting.md' +--- + +This document was moved to [another location](../administration/geo/replication/troubleshooting.md). diff --git a/doc/gitlab-geo/tuning.md b/doc/gitlab-geo/tuning.md new file mode 100644 index 00000000000..84ac40f99db --- /dev/null +++ b/doc/gitlab-geo/tuning.md @@ -0,0 +1,5 @@ +--- +redirect_to: '../administration/geo/replication/tuning.md' +--- + +This document was moved to [another location](../administration/geo/replication/tuning.md). diff --git a/doc/gitlab-geo/updating_the_geo_nodes.md b/doc/gitlab-geo/updating_the_geo_nodes.md new file mode 100644 index 00000000000..28234ec02ed --- /dev/null +++ b/doc/gitlab-geo/updating_the_geo_nodes.md @@ -0,0 +1,5 @@ +--- +redirect_to: '../administration/geo/replication/updating_the_geo_nodes.md' +--- + +This document was moved to [another location](../administration/geo/replication/updating_the_geo_nodes.md). diff --git a/doc/gitlab-geo/using_a_geo_server.md b/doc/gitlab-geo/using_a_geo_server.md new file mode 100644 index 00000000000..c8a7b9d1096 --- /dev/null +++ b/doc/gitlab-geo/using_a_geo_server.md @@ -0,0 +1,5 @@ +--- +redirect_to: '../administration/geo/replication/using_a_geo_server.md' +--- + +This document was moved to [another location](../administration/geo/replication/using_a_geo_server.md). diff --git a/doc/hooks/custom_hooks.md b/doc/hooks/custom_hooks.md index 1d5e5dd6e15..fb939ff8aac 100644 --- a/doc/hooks/custom_hooks.md +++ b/doc/hooks/custom_hooks.md @@ -1,3 +1,7 @@ +--- +redirect_to: '../administration/custom_hooks.md' +--- + # Custom Git Hooks This document was moved to [administration/custom_hooks.md](../administration/custom_hooks.md). diff --git a/doc/img/devops-stages.png b/doc/img/devops-stages.png Binary files differindex 424bce95607..2fa5357062c 100644 --- a/doc/img/devops-stages.png +++ b/doc/img/devops-stages.png diff --git a/doc/incoming_email/README.md b/doc/incoming_email/README.md index db0f03f2c98..9544983974f 100644 --- a/doc/incoming_email/README.md +++ b/doc/incoming_email/README.md @@ -1 +1,5 @@ +--- +redirect_to: '../administration/reply_by_email.md' +--- + This document was moved to [administration/reply_by_email](../administration/reply_by_email.md). diff --git a/doc/incoming_email/postfix.md b/doc/incoming_email/postfix.md index 90833238ac5..a7192325229 100644 --- a/doc/incoming_email/postfix.md +++ b/doc/incoming_email/postfix.md @@ -1 +1,5 @@ +--- +redirect_to: '../administration/reply_by_email_postfix_setup.md' +--- + This document was moved to [administration/reply_by_email_postfix_setup](../administration/reply_by_email_postfix_setup.md). diff --git a/doc/install/README.md b/doc/install/README.md index 52011526768..9cc21412898 100644 --- a/doc/install/README.md +++ b/doc/install/README.md @@ -1,6 +1,7 @@ --- comments: false description: Read through the GitLab installation methods. +type: index --- # Installation **[CORE ONLY]** @@ -55,9 +56,9 @@ need to be aware of: - It can be more expensive for smaller installations. The default installation requires more resources than a single node Omnibus deployment, as most services are deployed in a redundant fashion. -- There are some feature [limitations to be aware of](kubernetes/gitlab_chart.md#limitations). +- There are some feature [limitations to be aware of](https://docs.gitlab.com/charts/#limitations). -[**> Install GitLab on Kubernetes using the GitLab Helm charts.**](kubernetes/index.md) +[**> Install GitLab on Kubernetes using the GitLab Helm charts.**](https://docs.gitlab.com/charts/) ## Installing GitLab with Docker @@ -81,7 +82,7 @@ the above methods, provided the cloud provider supports it. - [Install on AWS](aws/index.md): Install Omnibus GitLab on AWS using the community AMIs that GitLab provides. - [Install GitLab on Google Cloud Platform](google_cloud_platform/index.md): Install Omnibus GitLab on a VM in GCP. - [Install GitLab on Azure](azure/index.md): Install Omnibus GitLab from Azure Marketplace. -- [Install GitLab on OpenShift](openshift_and_gitlab/index.md): Install GitLab using the Docker image on OpenShift. +- [Install GitLab on OpenShift](https://docs.gitlab.com/charts/installation/cloud/openshift.html): Install GitLab on OpenShift by using GitLab's Helm charts. - [Install GitLab on DC/OS](https://mesosphere.com/blog/gitlab-dcos/): Install GitLab on Mesosphere DC/OS via the [GitLab-Mesosphere integration](https://about.gitlab.com/2016/09/16/announcing-gitlab-and-mesosphere/). - [Install GitLab on DigitalOcean](https://about.gitlab.com/2016/04/27/getting-started-with-gitlab-and-digitalocean/): Install Omnibus GitLab on DigitalOcean. - _Testing only!_ [DigitalOcean and Docker Machine](digitaloceandocker.md): diff --git a/doc/install/aws/img/add_tags.png b/doc/install/aws/img/add_tags.png Binary files differdeleted file mode 100644 index 3572cd5daa1..00000000000 --- a/doc/install/aws/img/add_tags.png +++ /dev/null diff --git a/doc/install/aws/img/create_route_table.png b/doc/install/aws/img/create_route_table.png Binary files differdeleted file mode 100644 index ea72c57257e..00000000000 --- a/doc/install/aws/img/create_route_table.png +++ /dev/null diff --git a/doc/install/aws/index.md b/doc/install/aws/index.md index 51d2a232dc0..73eaf758923 100644 --- a/doc/install/aws/index.md +++ b/doc/install/aws/index.md @@ -1,11 +1,15 @@ -# Installing GitLab on Amazon Web Services (AWS) +--- +type: howto +--- -To install GitLab on AWS, you can use the Amazon Machine Images (AMIs) that GitLab -provides with [each release](https://about.gitlab.com/releases/). +# Installing GitLab HA on Amazon Web Services (AWS) This page offers a walkthrough of a common HA (Highly Available) configuration for GitLab on AWS. You should customize it to accommodate your needs. +NOTE: **Note** +For organizations with 300 users or less, the recommended AWS installation method is to launch an EC2 single box [Omnibus Installation](https://about.gitlab.com/install/) and implement a snapshot strategy for backing up the data. + ## Introduction GitLab on AWS can leverage many of the services that are already @@ -55,6 +59,8 @@ Here's a list of the AWS services we will use, with links to pricing information - **ElastiCache**: An in-memory cache environment will be used to provide a High Availability Redis configuration. See the [Amazon ElastiCache pricing](https://aws.amazon.com/elasticache/pricing/). + +NOTE: **Note:** Please note that while we will be using EBS for storage, we do not recommend using EFS as it may negatively impact GitLab's performance. You can review the [relevant documentation](../../administration/high_availability/nfs.md#avoid-using-awss-elastic-file-system-efs) for more details. ## Creating an IAM EC2 instance role and profile To minimize the permissions of the user, we'll create a new [IAM](https://docs.aws.amazon.com/IAM/latest/UserGuide/introduction.html) @@ -379,6 +385,10 @@ size depends on your needs and you can always migrate to a bigger volume later. You will be able to [set up that volume](#setting-up-the-ebs-volume) after the instance is created. +CAUTION: **Caution:** +We **do not** recommend using the AWS Elastic File System (EFS), as it can result +in [significantly degraded performance](../../administration/high_availability/nfs.md#avoid-using-awss-elastic-file-system-efs). + ### Configure security group As a last step, configure the security group: @@ -606,7 +616,7 @@ To back up GitLab: To restore GitLab, first review the [restore documentation](../../raketasks/backup_restore.md#restore), and primarily the restore prerequisites. Then, follow the steps under the -[Omnibus installations section](../../raketasks/backup_restore.md#restore-for-omnibus-installations). +[Omnibus installations section](../../raketasks/backup_restore.md#restore-for-omnibus-gitlab-installations). ## Updating GitLab @@ -643,12 +653,24 @@ Have a read through these other resources and feel free to [open an issue](https://gitlab.com/gitlab-org/gitlab-ce/issues/new) to request additional material: -- [GitLab High Availability](https://docs.gitlab.com/ee/administration/high_availability/): +- [GitLab High Availability](../../administration/high_availability/README.md): GitLab supports several different types of clustering and high-availability. -- [Geo replication](https://docs.gitlab.com/ee/administration/geo/replication/): +- [Geo replication](../../administration/geo/replication/index.md): Geo is the solution for widely distributed development teams. - [Omnibus GitLab](https://docs.gitlab.com/omnibus/) - Everything you need to know about administering your GitLab instance. -- [Upload a license](https://docs.gitlab.com/ee/user/admin_area/license.html): +- [Upload a license](../../user/admin_area/license.md): Activate all GitLab Enterprise Edition functionality with a license. - [Pricing](https://about.gitlab.com/pricing): Pricing for the different tiers. + +<!-- ## Troubleshooting + +Include any troubleshooting steps that you can foresee. If you know beforehand what issues +one might have when setting this up, or when something is changed, or on upgrading, it's +important to describe those, too. Think of things that may go wrong and include them here. +This is important to minimize requests for support and to avoid doc comments with +questions that you know someone might ask. + +Each scenario can be a third-level heading, e.g. `### Getting error message X`. +If you have none to add when creating a doc, leave this section in place +but commented out to help encourage others to add to it in the future. --> diff --git a/doc/install/azure/img/azure-vm-management-settings-network-interfaces.png b/doc/install/azure/img/azure-vm-management-settings-network-interfaces.png Binary files differdeleted file mode 100644 index 4ff10718059..00000000000 --- a/doc/install/azure/img/azure-vm-management-settings-network-interfaces.png +++ /dev/null diff --git a/doc/install/azure/img/azure-vm-management.png b/doc/install/azure/img/azure-vm-management.png Binary files differdeleted file mode 100644 index a0e0067258c..00000000000 --- a/doc/install/azure/img/azure-vm-management.png +++ /dev/null diff --git a/doc/install/azure/index.md b/doc/install/azure/index.md index fa5be1d30f9..b1f79893baf 100644 --- a/doc/install/azure/index.md +++ b/doc/install/azure/index.md @@ -1,14 +1,10 @@ --- -description: 'Learn how to spin up a -pre-configured GitLab VM on Microsoft Azure and have your very own private GitLab instance up and running in around 30 minutes.' +description: 'Learn how to spin up a pre-configured GitLab VM on Microsoft Azure.' +type: howto --- # Install GitLab on Microsoft Azure -> _This article was originally written by Dave Wentzel and [published on the GitLab Blog][Original-Blog-Post]._ -> -> _Ported to the GitLab documentation and updated on 2017-08-24 by [Ian Scorer](https://gitlab.com/iscorer)._ - Azure is Microsoft's business cloud and GitLab is a pre-configured offering on the Azure Marketplace. Hopefully, you aren't surprised to hear that Microsoft and Azure have embraced open source software like Ubuntu, Red Hat Enterprise Linux, and of course - GitLab! This means that you can spin up a @@ -444,3 +440,15 @@ Check out our other [Technical Articles][GitLab-Technical-Articles] or browse th [SSH]: https://en.wikipedia.org/wiki/Secure_Shell [PuTTY]: http://www.putty.org/ [Using-SSH-In-Putty]: https://mediatemple.net/community/products/dv/204404604/using-ssh-in-putty- + +<!-- ## Troubleshooting + +Include any troubleshooting steps that you can foresee. If you know beforehand what issues +one might have when setting this up, or when something is changed, or on upgrading, it's +important to describe those, too. Think of things that may go wrong and include them here. +This is important to minimize requests for support and to avoid doc comments with +questions that you know someone might ask. + +Each scenario can be a third-level heading, e.g. `### Getting error message X`. +If you have none to add when creating a doc, leave this section in place +but commented out to help encourage others to add to it in the future. --> diff --git a/doc/install/database_mysql.md b/doc/install/database_mysql.md index e89846107b6..cbb3b766b4e 100644 --- a/doc/install/database_mysql.md +++ b/doc/install/database_mysql.md @@ -1,3 +1,7 @@ +--- +type: reference +--- + # Database MySQL NOTE: **Note:** @@ -301,3 +305,15 @@ Details can be found in the [PostgreSQL][postgres-text-type] and [postgres-text-type]: http://www.postgresql.org/docs/9.2/static/datatype-character.html [ce-38152]: https://gitlab.com/gitlab-org/gitlab-ce/issues/38152 + +<!-- ## Troubleshooting + +Include any troubleshooting steps that you can foresee. If you know beforehand what issues +one might have when setting this up, or when something is changed, or on upgrading, it's +important to describe those, too. Think of things that may go wrong and include them here. +This is important to minimize requests for support and to avoid doc comments with +questions that you know someone might ask. + +Each scenario can be a third-level heading, e.g. `### Getting error message X`. +If you have none to add when creating a doc, leave this section in place +but commented out to help encourage others to add to it in the future. --> diff --git a/doc/install/digitaloceandocker.md b/doc/install/digitaloceandocker.md index d67695d75b4..63bb941ad47 100644 --- a/doc/install/digitaloceandocker.md +++ b/doc/install/digitaloceandocker.md @@ -1,8 +1,11 @@ +--- +type: howto +--- + # Digital Ocean and Docker Machine test environment -CAUTION: **Caution:** -This guide is for quickly testing different versions of GitLab and not recommended for ease of -future upgrades or keeping the data you create. +This guide is for quickly testing different versions of GitLab and not +recommended for ease of future upgrades or keeping the data you create. ## Initial setup @@ -28,7 +31,7 @@ locally on either macOS or Linux. NOTE: **Note:** The rest of the steps are identical for macOS and Linux. -### Create new docker host +## Create new docker host 1. Login to Digital Ocean. 1. Generate a new API token at <https://cloud.digitalocean.com/settings/api/tokens>. @@ -60,9 +63,9 @@ The rest of the steps are identical for macOS and Linux. Resource: <https://docs.docker.com/machine/drivers/digital-ocean/>. -### Creating GitLab test instance +## Creating GitLab test instance -#### Connect your shell to the new machine +### Connect your shell to the new machine In this example we'll create a GitLab EE 8.10.8 instance. @@ -74,7 +77,7 @@ eval "$(docker-machine env gitlab-test-env-do)" You can add this to your `~/.bash_profile` file to ensure the `docker` client uses the `gitlab-test-env-do` docker host -#### Create new GitLab container +### Create new GitLab container - HTTP port: `8888` - SSH port: `2222` @@ -83,7 +86,7 @@ You can add this to your `~/.bash_profile` file to ensure the `docker` client us - Container name: `gitlab-test-8.10` - GitLab version: **EE** `8.10.8-ee.0` -##### Set up container settings +#### Set up container settings ```sh export SSH_PORT=2222 @@ -92,7 +95,7 @@ export VERSION=8.10.8-ee.0 export NAME=gitlab-test-8.10 ``` -##### Create container +#### Create container ```sh docker run --detach \ @@ -103,9 +106,9 @@ docker run --detach \ gitlab/gitlab-ee:$VERSION ``` -#### Connect to the GitLab container +### Connect to the GitLab container -##### Retrieve the docker host IP +#### Retrieve the docker host IP ```sh docker-machine ip gitlab-test-env-do @@ -114,7 +117,7 @@ docker-machine ip gitlab-test-env-do Browse to: <http://192.168.151.134:8888/>. -##### Execute interactive shell/edit configuration +#### Execute interactive shell/edit configuration ```sh docker exec -it $NAME /bin/bash @@ -126,8 +129,20 @@ root@192:/# vi /etc/gitlab/gitlab.rb root@192:/# gitlab-ctl reconfigure ``` -#### Resources +### Resources - <https://docs.gitlab.com/omnibus/docker/>. - <https://docs.docker.com/machine/get-started/>. - <https://docs.docker.com/machine/reference/ip/>. + +<!-- ## Troubleshooting + +Include any troubleshooting steps that you can foresee. If you know beforehand what issues +one might have when setting this up, or when something is changed, or on upgrading, it's +important to describe those, too. Think of things that may go wrong and include them here. +This is important to minimize requests for support, and to avoid doc comments with +questions that you know someone might ask. + +Each scenario can be a third-level heading, e.g. `### Getting error message X`. +If you have none to add when creating a doc, leave this section in place +but commented out to help encourage others to add to it in the future. --> diff --git a/doc/install/docker.md b/doc/install/docker.md index d0129f0f5c4..06da65189ba 100644 --- a/doc/install/docker.md +++ b/doc/install/docker.md @@ -1,8 +1,12 @@ +--- +type: index +--- + # Install GitLab with Docker [Docker](https://www.docker.com) and container technology have been revolutionizing the software world for the past few years. They combine the performance and efficiency of native execution with the abstraction, security, and immutability of virtualization. -GitLab provides official Docker images to allowing you to easily take advantage of the benefits of containerization while operating your GitLab instance. +GitLab provides official Docker images allowing you to easily take advantage of the benefits of containerization while operating your GitLab instance. ## Omnibus GitLab based images @@ -16,4 +20,4 @@ A [complete usage guide](https://docs.gitlab.com/omnibus/docker/) to these image ## Cloud native images -GitLab is also working towards a [cloud native set of containers](https://gitlab.com/charts/helm.gitlab.io#docker-container-images), with a single image for each component service. We intend for these images to eventually replace the [Omnibus GitLab based images](#omnibus-gitlab-based-images). +GitLab is also working towards a [cloud native set of containers](https://docs.gitlab.com/charts/), with a single image for each component service. We intend for these images to eventually replace the [Omnibus GitLab based images](#omnibus-gitlab-based-images). diff --git a/doc/install/google-protobuf.md b/doc/install/google-protobuf.md index a531b4519b3..434817c48cb 100644 --- a/doc/install/google-protobuf.md +++ b/doc/install/google-protobuf.md @@ -1,26 +1,5 @@ -# Installing a locally compiled google-protobuf gem +--- +redirect_to: 'installation.md#google-protobuf-loaderror-libx86_64-linux-gnulibcso6-version-glibc_214-not-found' +--- -First we must find the exact version of google-protobuf that your -GitLab installation requires. - - cd /home/git/gitlab - - # Only one of the following two commands will print something. It - # will look like: * google-protobuf (3.2.0) - bundle list | grep google-protobuf - bundle check | grep google-protobuf - -Below we use `3.2.0` as an example. Replace it with the version number -you found above. - - cd /home/git/gitlab - sudo -u git -H gem install google-protobuf --version 3.2.0 --platform ruby - -Finally, you can test whether google-protobuf loads correctly. The -following should print 'OK'. - - sudo -u git -H bundle exec ruby -rgoogle/protobuf -e 'puts :OK' - -If the `gem install` command fails you may need to install developer -tools. On Debian: `apt-get install build-essential libgmp-dev`, on -Centos/RedHat `yum groupinstall 'Development Tools'`. +This document was moved to [another location](installation.md#google-protobuf-loaderror-libx86_64-linux-gnulibcso6-version-glibc_214-not-found). diff --git a/doc/install/google_cloud_platform/img/gcp_landing.png b/doc/install/google_cloud_platform/img/gcp_landing.png Binary files differdeleted file mode 100644 index 92a9873728c..00000000000 --- a/doc/install/google_cloud_platform/img/gcp_landing.png +++ /dev/null diff --git a/doc/install/google_cloud_platform/index.md b/doc/install/google_cloud_platform/index.md index aa4b3dccf7d..77c61acbfd4 100644 --- a/doc/install/google_cloud_platform/index.md +++ b/doc/install/google_cloud_platform/index.md @@ -1,12 +1,17 @@ --- description: 'Learn how to install a GitLab instance on Google Cloud Platform.' +type: howto --- # Installing GitLab on Google Cloud Platform - +This guide will help you install GitLab on a [Google Cloud Platform (GCP)][gcp] instance. -Getting started with GitLab on a [Google Cloud Platform (GCP)][gcp] instance is quick and easy. +NOTE: **Alternative installation method:** +Google provides a whitepaper for [deploying production-ready GitLab on +Google Kubernetes Engine](https://cloud.google.com/solutions/deploying-production-ready-gitlab-on-gke), +including all steps and external resource configuration. These are an alternative to using a GCP VM, and use +the [Cloud native GitLab Helm chart](https://docs.gitlab.com/charts/). ## Prerequisites @@ -20,10 +25,9 @@ Once you have performed those two steps, you can [create a VM](#creating-the-vm) ## Creating the VM -To deploy GitLab on GCP you need to follow five simple steps: - -1. Go to <https://console.cloud.google.com/compute/instances> and login with your Google credentials. +To deploy GitLab on GCP you first need to create a virtual machine: +1. Go to <https://console.cloud.google.com/compute/instances> and log in with your Google credentials. 1. Click on **Create**  @@ -136,3 +140,15 @@ Kerberos, etc. Here are some documents you might be interested in reading: [ssh]: https://cloud.google.com/compute/docs/instances/connecting-to-instance "Connecting to Linux Instances" [omni-smtp]: https://docs.gitlab.com/omnibus/settings/smtp.html#smtp-settings "Omnibus GitLab SMTP settings" [omni-ssl]: https://docs.gitlab.com/omnibus/settings/nginx.html#enable-https "Omnibus GitLab enable HTTPS" + +<!-- ## Troubleshooting + +Include any troubleshooting steps that you can foresee. If you know beforehand what issues +one might have when setting this up, or when something is changed, or on upgrading, it's +important to describe those, too. Think of things that may go wrong and include them here. +This is important to minimize requests for support, and to avoid doc comments with +questions that you know someone might ask. + +Each scenario can be a third-level heading, e.g. `### Getting error message X`. +If you have none to add when creating a doc, leave this section in place +but commented out to help encourage others to add to it in the future. --> diff --git a/doc/install/installation.md b/doc/install/installation.md index fb24d4fa0ef..10436a15a9e 100644 --- a/doc/install/installation.md +++ b/doc/install/installation.md @@ -1,5 +1,28 @@ +--- +type: howto +--- + # Installation from source +This is the official installation guide to set up a production GitLab server +using the source files. To set up a **development installation** or for many +other installation options, see the [main installation page](index.md). +It was created for and tested on **Debian/Ubuntu** operating systems. +Read [requirements.md](requirements.md) for hardware and operating system requirements. +If you want to install on RHEL/CentOS, we recommend using the +[Omnibus packages](https://about.gitlab.com/downloads/). + +This guide is long because it covers many cases and includes all commands you +need, this is [one of the few installation scripts that actually works out of the box](https://twitter.com/robinvdvleuten/status/424163226532986880). +The following steps have been known to work. **Use caution when you deviate** +from this guide. Make sure you don't violate any assumptions GitLab makes about +its environment. For example, many people run into permission problems because +they changed the location of directories or run services as the wrong user. + +If you find a bug/error in this guide, **submit a merge request** +following the +[contributing guide](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/CONTRIBUTING.md). + ## Consider the Omnibus package installation Since an installation from source is a lot of work and error prone we strongly recommend the fast and reliable [Omnibus package installation](https://about.gitlab.com/downloads/) (deb/rpm). @@ -9,28 +32,45 @@ On heavily used GitLab instances the memory usage of the Sidekiq background work Omnibus packages solve this by [letting the Sidekiq terminate gracefully](../administration/operations/sidekiq_memory_killer.md) if it uses too much memory. After this termination Runit will detect Sidekiq is not running and will start it. -Since installations from source don't have Runit, Sidekiq can't be terminated and its memory usage will grow over time. +Since installations from source don't use Runit for process supervision, Sidekiq +can't be terminated and its memory usage will grow over time. -## Select Version to Install +## Select version to install Make sure you view [this installation guide](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/doc/install/installation.md) from the branch (version) of GitLab you would like to install (e.g., `11-7-stable`). You can select the branch in the version dropdown in the top left corner of GitLab (below the menu bar). If the highest number stable branch is unclear, check the [GitLab blog](https://about.gitlab.com/blog/) for installation guide links by version. -## Important Notes +## GitLab directory structure -This guide is long because it covers many cases and includes all commands you need, this is [one of the few installation scripts that actually works out of the box](https://twitter.com/robinvdvleuten/status/424163226532986880). +This is the main directory structure you will end up with following the instructions +of this page: -This installation guide was created for and tested on **Debian/Ubuntu** operating systems. Read [requirements.md](requirements.md) for hardware and operating system requirements. If you want to install on RHEL/CentOS, we recommend using the [Omnibus packages](https://about.gitlab.com/downloads/). +``` +|-- home +| |-- git +| |-- .ssh +| |-- gitlab +| |-- gitlab-shell +| |-- repositories +``` -This is the official installation guide to set up a production server. To set up a **development installation** or for many other installation options, see [the installation section of the README](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/README.md#installation). +- `/home/git/.ssh` - Contains OpenSSH settings. Specifically the `authorized_keys` + file managed by gitlab-shell. +- `/home/git/gitlab` - GitLab core software. +- `/home/git/gitlab-shell` - Core add-on component of GitLab. Maintains SSH + cloning and other functionality. +- `/home/git/repositories` - Bare repositories for all projects organized by + namespace. This is where the git repositories which are pushed/pulled are + maintained for all projects. **This area contains critical data for projects. + [Keep a backup](../raketasks/backup_restore.md).** -The following steps have been known to work. **Use caution when you deviate** from this guide. Make sure you don't violate any assumptions GitLab makes about its environment. For example, many people run into permission problems because they changed the location of directories or run services as the wrong user. +NOTE: **Note:** +The default locations for repositories can be configured in `config/gitlab.yml` +of GitLab and `config.yml` of gitlab-shell. -If you find a bug/error in this guide, **submit a merge request** -following the -[contributing guide](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/CONTRIBUTING.md). +For a more in-depth overview, see the [GitLab architecture doc](../development/architecture.md). ## Overview @@ -72,7 +112,8 @@ Install the required packages (needed to compile Ruby and native extensions to R ```sh sudo apt-get install -y build-essential zlib1g-dev libyaml-dev libssl-dev libgdbm-dev libre2-dev \ libreadline-dev libncurses5-dev libffi-dev curl openssh-server checkinstall libxml2-dev \ - libxslt-dev libcurl4-openssl-dev libicu-dev logrotate rsync python-docutils pkg-config cmake + libxslt-dev libcurl4-openssl-dev libicu-dev logrotate rsync python-docutils pkg-config cmake \ + runit ``` Ubuntu 14.04 (Trusty Tahr) doesn't have the `libre2-dev` package available, but @@ -93,11 +134,24 @@ Make sure you have the right version of Git installed: # Install Git sudo apt-get install -y git-core -# Make sure Git is version 2.18.0 or higher +# Make sure Git is version 2.21.0 or higher git --version ``` -Is the system packaged Git too old? Remove it and compile from source. +Starting with GitLab 12.0, Git is required to be compiled with `libpcre2`. +Find out if that's the case: + +```sh +ldd /usr/local/bin/git | grep pcre2 +``` + +The output should be similar to: + +``` +libpcre2-8.so.0 => /usr/lib/libpcre2-8.so.0 (0x00007f08461c3000) +``` + +Is the system packaged Git too old, or not compiled with pcre2? Remove it and compile from source: ```sh # Remove packaged Git @@ -106,12 +160,21 @@ sudo apt-get remove git-core # Install dependencies sudo apt-get install -y libcurl4-openssl-dev libexpat1-dev gettext libz-dev libssl-dev build-essential +# Download and compile pcre2 from source +curl --silent --show-error --location https://ftp.pcre.org/pub/pcre/pcre2-10.33.tar.gz --output pcre2.tar.gz +tar -xzf pcre2.tar.gz +cd pcre2-10.33 +chmod +x configure +./configure --prefix=/usr --enable-jit +make +make install + # Download and compile from source cd /tmp -curl --remote-name --location --progress https://www.kernel.org/pub/software/scm/git/git-2.18.0.tar.gz -echo '94faf2c0b02a7920b0b46f4961d8e9cad08e81418614102898a55f980fa3e7e4 git-2.18.0.tar.gz' | shasum -a256 -c - && tar -xzf git-2.18.0.tar.gz -cd git-2.18.0/ -./configure +curl --remote-name --location --progress https://www.kernel.org/pub/software/scm/git/git-2.21.0.tar.gz +echo '85eca51c7404da75e353eba587f87fea9481ba41e162206a6f70ad8118147bee git-2.21.0.tar.gz' | shasum -a256 -c - && tar -xzf git-2.21.0.tar.gz +cd git-2.21.0/ +./configure --with-libpcre make prefix=/usr/local all # Install into /usr/local/bin @@ -161,9 +224,9 @@ Download Ruby and compile it: ```sh mkdir /tmp/ruby && cd /tmp/ruby -curl --remote-name --progress https://cache.ruby-lang.org/pub/ruby/2.5/ruby-2.5.3.tar.gz -echo 'f919a9fbcdb7abecd887157b49833663c5c15fda ruby-2.5.3.tar.gz' | shasum -c - && tar xzf ruby-2.5.3.tar.gz -cd ruby-2.5.3 +curl --remote-name --progress https://cache.ruby-lang.org/pub/ruby/2.6/ruby-2.6.3.tar.gz +echo '2347ed6ca5490a104ebd5684d2b9b5eefa6cd33c ruby-2.6.3.tar.gz' | shasum -c - && tar xzf ruby-2.6.3.tar.gz +cd ruby-2.6.3 ./configure --disable-install-rdoc make @@ -300,16 +363,14 @@ Redis 2.8 with: sudo apt-get install redis-server ``` -If you are using Debian 7 or Ubuntu 12.04, follow the special documentation -on [an alternate Redis installation](redis.md). Once done, follow the rest of -the guide here. +Once done, you can configure Redis: ```sh # Configure redis to use sockets sudo cp /etc/redis/redis.conf /etc/redis/redis.conf.orig # Disable Redis listening on TCP by setting 'port' to 0 -sed 's/^port .*/port 0/' /etc/redis/redis.conf.orig | sudo tee /etc/redis/redis.conf +sudo sed 's/^port .*/port 0/' /etc/redis/redis.conf.orig | sudo tee /etc/redis/redis.conf # Enable Redis socket for default Debian / Ubuntu path echo 'unixsocket /var/run/redis/redis.sock' | sudo tee -a /etc/redis/redis.conf @@ -318,9 +379,9 @@ echo 'unixsocket /var/run/redis/redis.sock' | sudo tee -a /etc/redis/redis.conf echo 'unixsocketperm 770' | sudo tee -a /etc/redis/redis.conf # Create the directory which contains the socket -mkdir /var/run/redis -chown redis:redis /var/run/redis -chmod 755 /var/run/redis +sudo mkdir -p /var/run/redis +sudo chown redis:redis /var/run/redis +sudo chmod 755 /var/run/redis # Persist the directory which contains the socket, if applicable if [ -d /etc/tmpfiles.d ]; then @@ -382,7 +443,7 @@ sudo chmod -R u+rwX tmp/pids/ sudo chmod -R u+rwX tmp/sockets/ # Create the public/uploads/ directory -sudo -u git -H mkdir public/uploads/ +sudo -u git -H mkdir -p public/uploads/ # Make sure only the GitLab user has access to the public/uploads/ directory # now that files in public/uploads are served by gitlab-workhorse @@ -432,7 +493,8 @@ sudo -u git -H editor config/resque.yml ``` CAUTION: **Caution:** -Make sure to edit both `gitlab.yml` and `unicorn.rb` to match your setup. +Make sure to edit both `gitlab.yml` and `unicorn.rb` to match your setup. +If you want to use Puma web server, see [Using Puma](#using-puma) for the additional steps. NOTE: **Note:** If you want to use HTTPS, see [Using HTTPS](#using-https) for the additional steps. @@ -446,6 +508,18 @@ sudo -u git cp config/database.yml.postgresql config/database.yml # MySQL only: sudo -u git cp config/database.yml.mysql config/database.yml +# PostgreSQL only: +# Remove host, username, and password lines from config/database.yml. +# Once modified, the `production` settings will be as follows: +# +# production: +# adapter: postgresql +# encoding: unicode +# database: gitlabhq_production +# pool: 10 +# +sudo -u git -H editor config/database.yml + # MySQL and remote PostgreSQL only: # Update username/password in config/database.yml. # You only need to adapt the production settings (first part). @@ -538,6 +612,7 @@ sudo -u git -H make ```sh # Fetch Gitaly source with Git and compile with Go +cd /home/git/gitlab sudo -u git -H bundle exec rake "gitlab:gitaly:install[/home/git/gitaly,/home/git/repositories]" RAILS_ENV=production ``` @@ -562,9 +637,22 @@ sudo -u git -H editor config.toml For more information about configuring Gitaly see [doc/administration/gitaly](../administration/gitaly). +### Start Gitaly + +Gitaly must be running for the next section. + +```sh +gitlab_path=/home/git/gitlab +gitaly_path=/home/git/gitaly + +sudo -u git -H $gitlab_path/bin/daemon_with_pidfile $gitlab_path/tmp/pids/gitaly.pid \ + $gitaly_path/gitaly $gitaly_path/config.toml >> $gitlab_path/log/gitaly.log 2>&1 & +``` + ### Initialize Database and Activate Advanced Features ```sh +cd /home/git/gitlab sudo -u git -H bundle exec rake gitlab:setup RAILS_ENV=production # Type 'yes' to create the database tables. @@ -575,10 +663,10 @@ sudo -u git -H bundle exec rake gitlab:setup RAILS_ENV=production force=yes ``` NOTE: **Note:** -You can set the Administrator/root password and e-mail by supplying them in environmental variables, `GITLAB_ROOT_PASSWORD` and `GITLAB_ROOT_EMAIL` respectively, as seen below. If you don't set the password (and it is set to the default one), wait to expose GitLab to the public internet until the installation is done and you've logged into the server the first time. During the first login, you'll be forced to change the default password. +You can set the Administrator/root password and e-mail by supplying them in environmental variables, `GITLAB_ROOT_PASSWORD` and `GITLAB_ROOT_EMAIL` respectively, as seen below. If you don't set the password (and it is set to the default one), wait to expose GitLab to the public internet until the installation is done and you've logged into the server the first time. During the first login, you'll be forced to change the default password. An Enterprise Edition license may also be installed at this time by supplying a full path in the `GITLAB_LICENSE_FILE` environment variable. ```sh -sudo -u git -H bundle exec rake gitlab:setup RAILS_ENV=production GITLAB_ROOT_PASSWORD=yourpassword GITLAB_ROOT_EMAIL=youremail +sudo -u git -H bundle exec rake gitlab:setup RAILS_ENV=production GITLAB_ROOT_PASSWORD=yourpassword GITLAB_ROOT_EMAIL=youremail GITLAB_LICENSE_FILE="/path/to/license" ``` ### Secure secrets.yml @@ -636,6 +724,12 @@ sudo -u git -H yarn install --production --pure-lockfile sudo -u git -H bundle exec rake gitlab:assets:compile RAILS_ENV=production NODE_ENV=production ``` +If `rake` fails with `JavaScript heap out of memory` error, try to run it with `NODE_OPTIONS` set as follows. + +```sh +sudo -u git -H bundle exec rake gitlab:assets:compile RAILS_ENV=production NODE_ENV=production NODE_OPTIONS="--max_old_space_size=4096" +``` + ### Start Your GitLab Instance ```sh @@ -841,6 +935,25 @@ You also need to change the corresponding options (e.g. `ssh_user`, `ssh_host`, Apart from the always supported markdown style, there are other rich text files that GitLab can display. But you might have to install a dependency to do so. See the [github-markup gem README](https://github.com/gitlabhq/markup#markups) for more information. +### Using Puma + +Puma is a multi-threaded HTTP 1.1 server for Ruby applications. + +To use GitLab with Puma: + +1. Finish GitLab setup so you have it up and running. +1. Copy the supplied example Puma config file into place: + + ```sh + cd /home/git/gitlab + + # Copy config file for the web server + sudo -u git -H config/puma.rb.example config/puma.rb + ``` + +1. Edit the system `init.d` script to use `EXPERIMENTAL_PUMA=1` flag. If you have `/etc/default/gitlab`, then you should edit it instead. +1. Restart GitLab. + ## Troubleshooting ### "You appear to have cloned an empty repository." @@ -854,8 +967,50 @@ and correctly [configured Nginx](#site-configuration). ### google-protobuf "LoadError: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.14' not found" This can happen on some platforms for some versions of the -google-protobuf gem. The workaround is to [install a source-only -version of this gem](google-protobuf.md). +`google-protobuf` gem. The workaround is to install a source-only +version of this gem. + +First, you must find the exact version of `google-protobuf` that your +GitLab installation requires: + +```sh +cd /home/git/gitlab + +# Only one of the following two commands will print something. It +# will look like: * google-protobuf (3.2.0) +bundle list | grep google-protobuf +bundle check | grep google-protobuf +``` + +Below, `3.2.0` is used as an example. Replace it with the version number +you found above: + +```sh +cd /home/git/gitlab +sudo -u git -H gem install google-protobuf --version 3.2.0 --platform ruby +``` + +Finally, you can test whether `google-protobuf` loads correctly. The +following should print 'OK'. + +```sh +sudo -u git -H bundle exec ruby -rgoogle/protobuf -e 'puts :OK' +``` + +If the `gem install` command fails, you may need to install the developer +tools of your OS. + +On Debian/Ubuntu: + +```sh +sudo apt-get install build-essential libgmp-dev +``` + +On RedHat/CentOS: + +```sh +sudo yum groupinstall 'Development Tools' +``` [RVM]: https://rvm.io/ "RVM Homepage" [rbenv]: https://github.com/sstephenson/rbenv "rbenv on GitHub" diff --git a/doc/install/kubernetes/gitlab_chart.md b/doc/install/kubernetes/gitlab_chart.md index 9db246b3eb3..43655767002 100644 --- a/doc/install/kubernetes/gitlab_chart.md +++ b/doc/install/kubernetes/gitlab_chart.md @@ -1,156 +1,5 @@ -# GitLab Helm Chart +--- +redirect_to: https://docs.gitlab.com/charts/ +--- -This is the official way to install GitLab on a cloud native environment. - -NOTE: **Kubernetes experience required:** -Our Helm charts are recommended for those who are familiar with Kubernetes. -If you're not sure if Kubernetes is for you, our -[Omnibus GitLab packages](../README.md#installing-gitlab-using-the-omnibus-gitlab-package-recommended) -are mature, scalable, support [high availability](../../administration/high_availability/README.md) -and are used today on GitLab.com. -It is not necessary to have GitLab installed on Kubernetes in order to use [GitLab Kubernetes integration](https://docs.gitlab.com/ee/user/project/clusters/index.html). - -## Introduction - -The `gitlab` chart is the best way to operate GitLab on Kubernetes. This chart -contains all the required components to get started, and can scale to large deployments. - -The default deployment includes: - -- Core GitLab components: Unicorn, Shell, Workhorse, Registry, Sidekiq, and Gitaly -- Optional dependencies: Postgres, Redis, Minio -- An auto-scaling, unprivileged [GitLab Runner](https://docs.gitlab.com/runner/) using the Kubernetes executor -- Automatically provisioned SSL via [Let's Encrypt](https://letsencrypt.org/). - -## Limitations - -Some features of GitLab are not currently available: - -- [GitLab Pages](https://gitlab.com/charts/gitlab/issues/37) -- [GitLab Geo](https://gitlab.com/charts/gitlab/issues/8) -- [No in-cluster HA database](https://gitlab.com/charts/gitlab/issues/48) -- MySQL will not be supported, as support is [deprecated within GitLab](https://docs.gitlab.com/omnibus/settings/database.html#using-a-mysql-database-management-server-enterprise-edition-only) - -## Installing GitLab using the Helm Chart - -The `gitlab` chart includes all required dependencies, and takes a few minutes -to deploy. - -TIP: **Tip:** -For production deployments, we strongly recommend using the -[detailed installation instructions](https://gitlab.com/charts/gitlab/blob/master/doc/installation/index.md) -utilizing [external Postgres, Redis, and object storage](https://gitlab.com/charts/gitlab/tree/master/doc/advanced) services. - -### Requirements - -In order to deploy GitLab on Kubernetes, the following are required: - -1. `helm` and `kubectl` [installed on your computer](preparation/tools_installation.md). -1. A Kubernetes cluster, version 1.8 or higher. 6vCPU and 16GB of RAM is recommended. - - [Amazon EKS](https://docs.aws.amazon.com/eks/latest/userguide/getting-started.html) - - [Google GKE](https://cloud.google.com/kubernetes-engine/docs/how-to/creating-a-container-cluster) - - [IBM IKS](https://console.bluemix.net/docs/tutorials/scalable-webapp-kubernetes.html#create_kube_cluster) - - [Microsoft AKS](https://docs.microsoft.com/en-us/azure/aks/kubernetes-walkthrough-portal) -1. A [wildcard DNS entry and external IP address](preparation/networking.md) -1. [Authenticate and connect](preparation/connect.md) to the cluster -1. Configure and initialize [Helm Tiller](preparation/tiller.md). - -### Deployment of GitLab to Kubernetes - -To deploy GitLab, the following three parameters are required: - -- `global.hosts.domain`: the [base domain](preparation/networking.md) of the - wildcard host entry. For example, `example.com` if the wild card entry is - `*.example.com`. -- `global.hosts.externalIP`: the [external IP](preparation/networking.md) which - the wildcard DNS resolves to. -- `certmanager-issuer.email`: the email address to use when requesting new SSL - certificates from Let's Encrypt. - -NOTE: **Note:** -For deployments to Amazon EKS, there are -[additional configuration requirements](preparation/eks.md). A full list of -configuration options is [also available](https://gitlab.com/charts/gitlab/blob/master/doc/installation/command-line-options.md). - -Once you have all of your configuration options collected, you can get any -dependencies and run helm. In this example, the helm release is named "gitlab": - -```sh -helm repo add gitlab https://charts.gitlab.io/ -helm repo update -helm upgrade --install gitlab gitlab/gitlab \ - --timeout 600 \ - --set global.hosts.domain=example.com \ - --set global.hosts.externalIP=10.10.10.10 \ - --set certmanager-issuer.email=email@example.com -``` - -### Monitoring the Deployment - -This will output the list of resources installed once the deployment finishes, -which may take 5-10 minutes. - -The status of the deployment can be checked by running `helm status gitlab` -which can also be done while the deployment is taking place if you run the -command in another terminal. - -### Initial login - -You can access the GitLab instance by visiting the domain name beginning with -`gitlab.` followed by the domain specified during installation. From the example -above, the URL would be `https://gitlab.example.com`. - -If you manually created the secret for initial root password, you -can use that to sign in as `root` user. If not, GitLab automatically -created a random password for `root` user. This can be extracted by the -following command (replace `<name>` by name of the release - which is `gitlab` -if you used the command above): - -```sh -kubectl get secret <name>-gitlab-initial-root-password -ojsonpath={.data.password} | base64 --decode ; echo -``` - -### Outgoing email - -By default outgoing email is disabled. To enable it, provide details for your SMTP server -using the `global.smtp` and `global.email` settings. You can find details for these settings in the -[command line options](https://gitlab.com/charts/gitlab/blob/master/doc/installation/command-line-options.md#email-configuration). - -If your SMTP server requires authentication make sure to read the section on providing -your password in the [secrets documentation](https://gitlab.com/charts/gitlab/blob/master/doc/installation/secrets.md#smtp-password). -You can disable authentication settings with `--set global.smtp.authentication=""`. - -If your Kubernetes cluster is on GKE, be aware that SMTP port [25 is blocked](https://cloud.google.com/compute/docs/tutorials/sending-mail/#using_standard_email_ports). - -### Deploying the Community Edition - -To deploy the Community Edition, include these options in your `helm install` command: - -```sh ---set gitlab.migrations.image.repository=registry.gitlab.com/gitlab-org/build/cng/gitlab-rails-ce ---set gitlab.sidekiq.image.repository=registry.gitlab.com/gitlab-org/build/cng/gitlab-sidekiq-ce ---set gitlab.unicorn.image.repository=registry.gitlab.com/gitlab-org/build/cng/gitlab-unicorn-ce ---set gitlab.unicorn.workhorse.image=registry.gitlab.com/gitlab-org/build/cng/gitlab-workhorse-ce ---set gitlab.task-runner.image.repository=registry.gitlab.com/gitlab-org/build/cng/gitlab-task-runner-ce -``` - -## Updating GitLab using the Helm Chart - -Once your GitLab Chart is installed, configuration changes and chart updates -should be done using `helm upgrade`: - -```sh -helm repo update -helm upgrade --reuse-values gitlab gitlab/gitlab -``` - -## Uninstalling GitLab using the Helm Chart - -To uninstall the GitLab Chart, run the following: - -```sh -helm delete gitlab -``` - -[kube-srv]: https://kubernetes.io/docs/concepts/services-networking/service/#publishing-services---service-types -[storageclass]: https://kubernetes.io/docs/concepts/storage/persistent-volumes/#storageclasses +This document was moved to [another location](https://docs.gitlab.com/charts/). diff --git a/doc/install/kubernetes/gitlab_omnibus.md b/doc/install/kubernetes/gitlab_omnibus.md index c0cb7694e91..43655767002 100644 --- a/doc/install/kubernetes/gitlab_omnibus.md +++ b/doc/install/kubernetes/gitlab_omnibus.md @@ -1,246 +1,5 @@ -# GitLab-Omnibus Helm Chart +--- +redirect_to: https://docs.gitlab.com/charts/ +--- -CAUTION: **Caution:** -This chart is **deprecated**. We recommend using the [`gitlab` chart](gitlab_chart.md) -instead. A comparison of the two charts is available in [this video](https://youtu.be/Z6jWR8Z8dv8). - -For more information on available GitLab Helm Charts, see [Installing GitLab on Kubernetes](index.md). - -- This GitLab-Omnibus chart has been tested on Google Kubernetes Engine and Azure Container Service. -- This work is based partially on: <https://github.com/lwolf/kubernetes-gitlab/>. GitLab would like to thank Sergey Nuzhdin for his work. - -## Introduction - -This chart provides an easy way to get started with GitLab, provisioning an -installation with nearly all functionality enabled. SSL is automatically -provisioned via [Let's Encrypt](https://letsencrypt.org/). - -This Helm chart is suited for small to medium deployments and is **deprecated** -and replaced by the [cloud native GitLab chart](https://gitlab.com/charts/helm.gitlab.io/blob/master/README.md). -Due to the significant architectural changes, migrating will require backing up -data out of this instance and importing it into the new deployment. - -The deployment includes: - -- A [GitLab Omnibus](https://docs.gitlab.com/omnibus/) Pod, including Mattermost, Container Registry, and Prometheus -- An auto-scaling [GitLab Runner](https://docs.gitlab.com/runner/) using the Kubernetes executor -- [Redis](https://github.com/kubernetes/charts/tree/master/stable/redis) -- [PostgreSQL](https://github.com/kubernetes/charts/tree/master/stable/postgresql) -- [NGINX Ingress](https://github.com/kubernetes/charts/tree/master/stable/nginx-ingress) -- Persistent Volume Claims for Data, Registry, Postgres, and Redis - -## Limitations - -[High Availability](../../administration/high_availability/README.md) and -[Geo](https://docs.gitlab.com/ee/administration/geo/replication/index.html) are not supported. - -## Requirements - -- _At least_ 4 GB of RAM available on your cluster. 41GB of storage and 2 CPU are also required. -- Kubernetes 1.4+ with Beta APIs enabled -- [Persistent Volume](https://kubernetes.io/docs/concepts/storage/persistent-volumes/) provisioner support in the underlying infrastructure -- A [wildcard DNS entry](#networking-requirements), which resolves to the external IP address -- The `kubectl` CLI installed locally and authenticated for the cluster -- The [Helm client](https://github.com/kubernetes/helm/blob/master/docs/quickstart.md) installed locally on your machine - -### Networking requirements - -This chart configures a GitLab server and Kubernetes cluster which can support -dynamic [Review Apps](../../ci/review_apps/index.md), as well as services like -the integrated [Container Registry](../../user/project/container_registry.md) -and [Mattermost](https://docs.gitlab.com/omnibus/gitlab-mattermost/). - -To support the GitLab services and dynamic environments, a wildcard DNS entry -is required which resolves to the [load balancer](#load-balancer-ip) or -[external IP](#external-ip-recommended). Configuration of the DNS entry will depend upon -the DNS service being used. - -#### External IP (recommended) - -To provision an external IP on GCP and Azure, simply request a new address from -the Networking section. Ensure that the region matches the region your container -cluster is created in. It is important that the IP is not assigned at this point -in time. It will be automatically assigned once the Helm chart is installed, -and assigned to the Load Balancer. - -Now that an external IP address has been allocated, ensure that the wildcard -DNS entry you would like to use resolves to this IP. Please consult the -documentation for your DNS service for more information on creating DNS records. - -Finally, set the `baseIP` setting to this IP address when -[deploying GitLab](#configuring-and-installing-gitlab). - -#### Load Balancer IP - -If you do not specify a `baseIP`, an IP will be assigned to the Load Balancer or -Ingress. You can retrieve this IP by running the following command *after* deploying GitLab: - -```sh -kubectl get svc -w --namespace nginx-ingress nginx -``` - -The IP address will be displayed in the `EXTERNAL-IP` field, and should be used -to configure the Wildcard DNS entry. For more information on creating a wildcard -DNS entry, consult the documentation for the DNS server you are using. - -For production deployments of GitLab, we strongly recommend using a -[external IP](#external-ip-recommended). - -## Configuring and Installing GitLab - -For most installations, two parameters are required: - -- `baseDomain`: the [base domain](#networking-requirements) of the wildcard host entry. For example, `mycompany.io` if the wild card entry is `*.mycompany.io`. -- `legoEmail`: Email address to use when requesting new SSL certificates from Let's Encrypt. - -Other common configuration options: - -- `baseIP`: the desired [external IP address](#external-ip-recommended) -- `gitlab`: Choose the [desired edition](https://about.gitlab.com/pricing), either `ee` or `ce`. `ce` is the default. -- `gitlabEELicense`: For Enterprise Edition, the [license](https://docs.gitlab.com/ee/user/admin_area/license.html) can be installed directly via the Chart -- `provider`: Optimizes the deployment for a cloud provider. The default is `gke` for [Google Kubernetes Engine](https://cloud.google.com/kubernetes-engine/), with `acs` also supported for the [Azure Container Service](https://azure.microsoft.com/en-us/services/container-service/). - -For additional configuration options, consult the -[`values.yaml`](https://gitlab.com/charts/gitlab-omnibus/blob/master/values.yaml). - -### Choosing a different GitLab release version - -The version of GitLab installed is based on the `gitlab` setting (see [section](#configuring-and-installing-gitLab) above), and -the value of the corresponding helm setting: `gitlabCEImage` or `gitabEEImage`. - -```yaml -gitlab: CE -gitlabCEImage: gitlab/gitlab-ce:9.5.2-ce.0 -gitlabEEImage: gitlab/gitlab-ee:9.5.2-ee.0 -``` - -The different images can be found in the [gitlab-ce](https://hub.docker.com/r/gitlab/gitlab-ce/tags/) and [gitlab-ee](https://hub.docker.com/r/gitlab/gitlab-ee/tags/) -repositories on Docker Hub. - -### Persistent storage - -NOTE: **Note:** -If you are using a machine type with support for less than 4 attached disks, -like an Azure trial, you should disable dedicated storage for Postgres and Redis. - -By default, persistent storage is enabled for GitLab and the charts it depends -on (Redis and PostgreSQL). Components can have their claim size set from your -`values.yaml`, along with whether to provision separate storage for Postgres and Redis. - -Basic configuration: - -```yaml -redisImage: redis:3.2.10 -redisDedicatedStorage: true -redisStorageSize: 5Gi -postgresImage: postgres:9.6.3 -# If you disable postgresDedicatedStorage, you should consider bumping up gitlabRailsStorageSize -postgresDedicatedStorage: true -postgresStorageSize: 30Gi -gitlabRailsStorageSize: 30Gi -gitlabRegistryStorageSize: 30Gi -gitlabConfigStorageSize: 1Gi -``` - -### Routing and SSL - -Ingress routing and SSL are automatically configured within this Chart. An NGINX -ingress is provisioned and configured, and will route traffic to any service. -SSL certificates are automatically created and configured by -[kube-lego](https://github.com/kubernetes/charts/tree/master/stable/kube-lego). - -NOTE: **Note:** -Let's Encrypt limits a single TLD to five certificate requests within a single -week. This means that common DNS wildcard services like [nip.io](http://nip.io) -and [xip.io](http://xip.io) are unlikely to work. - -## Installing GitLab using the Helm Chart - -NOTE: **Note:** -You may see a temporary error message `SchedulerPredicates failed due to PersistentVolumeClaim is not bound` -while storage provisions. Once the storage provisions, the pods will automatically start. -This may take a couple minutes depending on your cloud provider. If the error persists, -please review the [requirements sections](#requirements) to ensure you have enough RAM, CPU, and storage. - -Add the GitLab Helm repository and initialize Helm: - -```bash -helm init -helm repo add gitlab https://charts.gitlab.io -``` - -Once you have reviewed the [configuration settings](#configuring-and-installing-gitlab), -you can install the chart. We recommending saving your configuration options in a -`values.yaml` file for easier upgrades in the future: - -```bash -helm install --name gitlab -f values.yaml gitlab/gitlab-omnibus -``` - -Or you can pass them on the command line: - -```bash -helm install --name gitlab --set baseDomain=gitlab.io,baseIP=192.0.2.1,gitlab=ee,gitlabEELicense=$LICENSE,legoEmail=email@gitlab.com gitlab/gitlab-omnibus -``` - -## Updating GitLab using the Helm Chart - -If you are upgrading from a previous version to 0.1.35 or above, you will need to -change the access mode values for GitLab's storage. To do this, set the following -in `values.yaml` or on the CLI: - -```sh -gitlabDataAccessMode=ReadWriteMany -gitlabRegistryAccessMode=ReadWriteMany -gitlabConfigAccessMode=ReadWriteMany -``` - -Once your GitLab Chart is installed, configuration changes and chart updates -should be done using `helm upgrade`: - -```sh -helm upgrade -f values.yaml gitlab gitlab/gitlab-omnibus -``` - -## Upgrading from CE to EE using the Helm Chart - -If you have installed the Community Edition using this chart, upgrading to -Enterprise Edition is easy. - -If you are using a `values.yaml` file to specify the configuration options, edit -the file and set `gitlab=ee`. If you would like to run a specific version of -GitLab EE, set `gitlabEEImage` to be the desired GitLab -[docker image](https://hub.docker.com/r/gitlab/gitlab-ee/tags/). Then you can -use `helm upgrade` to update your GitLab instance to EE: - -```bash -helm upgrade -f values.yaml gitlab gitlab/gitlab-omnibus -``` - -You can also upgrade and specify these options via the command line: - -```bash -helm upgrade gitlab --set gitlab=ee,gitlabEEImage=gitlab/gitlab-ee:9.5.5-ee.0 gitlab/gitlab-omnibus -``` - -## Uninstalling GitLab using the Helm Chart - -To uninstall the GitLab Chart, run the following: - -```bash -helm delete --purge gitlab -``` - -## Troubleshooting - -### Storage errors when updating `gitlab-omnibus` versions prior to 0.1.35 - -Users upgrading `gitlab-omnibus` from a version prior to 0.1.35, may see an error -like: `Error: UPGRADE FAILED: PersistentVolumeClaim "gitlab-gitlab-config-storage" is invalid: spec: Forbidden: field is immutable after creation`. - -This is due to a change in the access mode for GitLab storage in version 0.1.35. -To successfully upgrade, the access mode flags must be set to `ReadWriteMany` -as detailed in the [update section](#updating-gitlab-using-the-helm-chart). - -[kube-srv]: https://kubernetes.io/docs/concepts/services-networking/service/#publishing-services---service-types -[storageclass]: https://kubernetes.io/docs/concepts/storage/persistent-volumes/#storageclasses +This document was moved to [another location](https://docs.gitlab.com/charts/). diff --git a/doc/install/kubernetes/gitlab_runner_chart.md b/doc/install/kubernetes/gitlab_runner_chart.md index 68b2a146115..08ccf2cf9ad 100644 --- a/doc/install/kubernetes/gitlab_runner_chart.md +++ b/doc/install/kubernetes/gitlab_runner_chart.md @@ -1,269 +1,5 @@ -# GitLab Runner Helm Chart -> **Note:** -These charts have been tested on Google Kubernetes Engine and Azure Container Service. Other Kubernetes installations may work as well, if not please [open an issue](https://gitlab.com/gitlab-org/gitlab-runner/issues). +--- +redirect_to: https://docs.gitlab.com/runner/install/kubernetes.html +--- -The `gitlab-runner` Helm chart deploys a GitLab Runner instance into your -Kubernetes cluster. - -This chart configures the Runner to: - -- Run using the GitLab Runner [Kubernetes executor](https://docs.gitlab.com/runner/install/kubernetes.html) -- For each new job it receives from [GitLab CI](https://about.gitlab.com/features/gitlab-ci-cd/), it will provision a - new pod within the specified namespace to run it. - -For more information on available GitLab Helm Charts, please see our [overview](index.md). - -## Prerequisites - -- Your GitLab Server's API is reachable from the cluster -- Kubernetes 1.4+ with Beta APIs enabled -- The `kubectl` CLI installed locally and authenticated for the cluster -- The [Helm client](https://github.com/kubernetes/helm/blob/master/docs/quickstart.md) installed locally on your machine - -## Configuring GitLab Runner using the Helm Chart - -Create a `values.yaml` file for your GitLab Runner configuration. See [Helm docs](https://github.com/kubernetes/helm/blob/master/docs/chart_template_guide/values_files.md) -for information on how your values file will override the defaults. - -The default configuration can always be found in the [values.yaml](https://gitlab.com/charts/gitlab-runner/blob/master/values.yaml) in the chart repository. - -### Required configuration - -In order for GitLab Runner to function, your config file **must** specify the following: - - - `gitlabUrl` - the GitLab Server URL (with protocol) to register the runner against - - `runnerRegistrationToken` - The Registration Token for adding new Runners to the GitLab Server. This must be - retrieved from your GitLab Instance. See the [GitLab Runner Documentation](../../ci/runners/README.md) for more information. - -Unless you need to specify additional configuration, you are [ready to install](#installing-gitlab-runner-using-the-helm-chart). - -### Other configuration - -The rest of the configuration is [documented in the `values.yaml`](https://gitlab.com/charts/gitlab-runner/blob/master/values.yaml) in the chart repository. - -Here is a snippet of the important settings: - -```yaml -## The GitLab Server URL (with protocol) that want to register the runner against -## ref: https://docs.gitlab.com/runner/commands/README.html#gitlab-runner-register -## -gitlabUrl: http://gitlab.your-domain.com/ - -## The Registration Token for adding new Runners to the GitLab Server. This must -## be retrieved from your GitLab Instance. -## ref: https://docs.gitlab.com/ee/ci/runners/README.html -## -runnerRegistrationToken: "" - -## Set the certsSecretName in order to pass custom certificates for GitLab Runner to use -## Provide resource name for a Kubernetes Secret Object in the same namespace, -## this is used to populate the /etc/gitlab-runner/certs directory -## ref: https://docs.gitlab.com/runner/configuration/tls-self-signed.html#supported-options-for-self-signed-certificates -## -#certsSecretName: - -## Configure the maximum number of concurrent jobs -## ref: https://docs.gitlab.com/runner/configuration/advanced-configuration.html#the-global-section -## -concurrent: 10 - -## Defines in seconds how often to check GitLab for a new builds -## ref: https://docs.gitlab.com/runner/configuration/advanced-configuration.html#the-global-section -## -checkInterval: 30 - -## For RBAC support: -rbac: - create: false - - ## Run the gitlab-bastion container with the ability to deploy/manage containers of jobs - ## cluster-wide or only within namespace - clusterWideAccess: false - - ## Use the following Kubernetes Service Account name if RBAC is disabled in this Helm chart (see rbac.create) - ## - # serviceAccountName: default - -## Configuration for the Pods that the runner launches for each new job -## -runners: - ## Default container image to use for builds when none is specified - ## - image: ubuntu:16.04 - - ## Run all containers with the privileged flag enabled - ## This will allow the docker:stable-dind image to run if you need to run Docker - ## commands. Please read the docs before turning this on: - ## ref: https://docs.gitlab.com/runner/executors/kubernetes.html#using-docker-dind - ## - privileged: false - - ## Namespace to run Kubernetes jobs in (defaults to 'default') - ## - # namespace: - - ## Build Container specific configuration - ## - builds: - # cpuLimit: 200m - # memoryLimit: 256Mi - cpuRequests: 100m - memoryRequests: 128Mi - - ## Service Container specific configuration - ## - services: - # cpuLimit: 200m - # memoryLimit: 256Mi - cpuRequests: 100m - memoryRequests: 128Mi - - ## Helper Container specific configuration - ## - helpers: - # cpuLimit: 200m - # memoryLimit: 256Mi - cpuRequests: 100m - memoryRequests: 128Mi - -``` - -### Enabling RBAC support - -If your cluster has RBAC enabled, you can choose to either have the chart create its own service account or provide one. - -To have the chart create the service account for you, set `rbac.create` to true. - -### Controlling maximum Runner concurrency - -A single GitLab Runner deployed on Kubernetes is able to execute multiple jobs in parallel by automatically starting additional Runner pods. The [`concurrent` setting](https://docs.gitlab.com/runner/configuration/advanced-configuration.html#the-global-section) controls the maximum number of pods allowed at a single time, and defaults to `10`. - -```yaml -## Configure the maximum number of concurrent jobs -## ref: https://docs.gitlab.com/runner/configuration/advanced-configuration.html#the-global-section -## -concurrent: 10 -``` - -### Running Docker-in-Docker containers with GitLab Runners - -See [Running Privileged Containers for the Runners](#running-privileged-containers-for-the-runners) for how to enable it, -and the [GitLab CI Runner documentation](https://docs.gitlab.com/runner/executors/kubernetes.html#using-docker-in-your-builds) on running dind. - -### Running privileged containers for the Runners - -You can tell the GitLab Runner to run using privileged containers. You may need -this enabled if you need to use the Docker executable within your GitLab CI jobs. - -This comes with several risks that you can read about in the -[GitLab CI Runner documentation](https://docs.gitlab.com/runner/executors/kubernetes.html#using-docker-in-your-builds). - -If you are okay with the risks, and your GitLab CI Runner instance is registered -against a specific project in GitLab that you trust the CI jobs of, you can -enable privileged mode in `values.yaml`: - -```yaml -runners: - ## Run all containers with the privileged flag enabled - ## This will allow the docker:stable-dind image to run if you need to run Docker - ## commands. Please read the docs before turning this on: - ## ref: https://docs.gitlab.com/runner/executors/kubernetes.html#using-docker-dind - ## - privileged: true -``` - -### Providing a custom certificate for accessing GitLab - -You can provide a [Kubernetes Secret](https://kubernetes.io/docs/concepts/configuration/secret/) -to the GitLab Runner Helm Chart, which will be used to populate the container's -`/etc/gitlab-runner/certs` directory. - -Each key name in the Secret will be used as a filename in the directory, with the -file content being the value associated with the key. - -More information on how GitLab Runner uses these certificates can be found in the -[Runner Documentation](https://docs.gitlab.com/runner/configuration/tls-self-signed.html#supported-options-for-self-signed-certificates). - - - The key/file name used should be in the format `<gitlab-hostname>.crt`. For example: `gitlab.your-domain.com.crt`. - - Any intermediate certificates need to be concatenated to your server certificate in the same file. - - The hostname used should be the one the certificate is registered for. - -The GitLab Runner Helm Chart does not create a secret for you. In order to create -the secret, you can prepare your certificate on you local machine, and then run -the `kubectl create secret` command from the directory with the certificate - -```bash -kubectl - --namespace <NAMESPACE> - create secret generic <SECRET_NAME> - --from-file=<CERTFICATE_FILENAME> -``` - -- `<NAMESPACE>` is the Kubernetes namespace where you want to install the GitLab Runner. -- `<SECRET_NAME>` is the Kubernetes Secret resource name. For example: `gitlab-domain-cert` -- `<CERTFICATE_FILENAME>` is the filename for the certificate in your current directory that will be imported into the secret - -You then need to provide the secret's name to the GitLab Runner chart. - -Add the following to your `values.yaml` - -```yaml -## Set the certsSecretName in order to pass custom certificates for GitLab Runner to use -## Provide resource name for a Kubernetes Secret Object in the same namespace, -## this is used to populate the /etc/gitlab-runner/certs directory -## ref: https://docs.gitlab.com/runner/configuration/tls-self-signed.html#supported-options-for-self-signed-certificates -## -certsSecretName: <SECRET NAME> -``` - -- `<SECRET_NAME>` is the Kubernetes Secret resource name. For example: `gitlab-domain-cert` - -## Installing GitLab Runner using the Helm Chart - -Add the GitLab Helm repository and initialize Helm: - -```bash -helm repo add gitlab https://charts.gitlab.io -helm init -``` - -Once you [have configured](#configuring-gitlab-runner-using-the-helm-chart) GitLab Runner in your `values.yml` file, -run the following: - -```bash -helm install --namespace <NAMESPACE> --name gitlab-runner -f <CONFIG_VALUES_FILE> gitlab/gitlab-runner -``` - -- `<NAMESPACE>` is the Kubernetes namespace where you want to install the GitLab Runner. -- `<CONFIG_VALUES_FILE>` is the path to values file containing your custom configuration. See the - [Configuring GitLab Runner using the Helm Chart](#configuring-gitlab-runner-using-the-helm-chart) section to create it. - -## Updating GitLab Runner using the Helm Chart - -Once your GitLab Runner Chart is installed, configuration changes and chart updates should we done using `helm upgrade` - -```bash -helm upgrade --namespace <NAMESPACE> -f <CONFIG_VALUES_FILE> <RELEASE-NAME> gitlab/gitlab-runner -``` - -Where: - -- `<NAMESPACE>` is the Kubernetes namespace where GitLab Runner is installed -- `<CONFIG_VALUES_FILE>` is the path to values file containing your custom configuration. See the - [Configuring GitLab Runner using the Helm Chart](#configuring-gitlab-runner-using-the-helm-chart) section to create it. -- `<RELEASE-NAME>` is the name you gave the chart when installing it. - In the [Installing GitLab Runner using the Helm Chart](#installing-gitlab-runner-using-the-helm-chart) section, we called it `gitlab-runner`. - -## Uninstalling GitLab Runner using the Helm Chart - -To uninstall the GitLab Runner Chart, run the following: - -```bash -helm delete --namespace <NAMESPACE> <RELEASE-NAME> -``` - -where: - -- `<NAMESPACE>` is the Kubernetes namespace where GitLab Runner is installed -- `<RELEASE-NAME>` is the name you gave the chart when installing it. - In the [Installing GitLab Runner using the Helm Chart](#installing-gitlab-runner-using-the-helm-chart) section, we called it `gitlab-runner`. +This document was moved to [another location](https://docs.gitlab.com/runner/install/kubernetes.html). diff --git a/doc/install/kubernetes/index.md b/doc/install/kubernetes/index.md index ecc956d04e9..43655767002 100644 --- a/doc/install/kubernetes/index.md +++ b/doc/install/kubernetes/index.md @@ -1,42 +1,5 @@ --- -description: 'Read through the different methods to deploy GitLab on Kubernetes.' +redirect_to: https://docs.gitlab.com/charts/ --- -# Installing GitLab on Kubernetes - -NOTE: **Kubernetes experience required:** -Our Helm charts are recommended for those who are familiar with Kubernetes. -If you're not sure if Kubernetes is for you, our -[Omnibus GitLab packages](../README.md#installing-gitlab-using-the-omnibus-gitlab-package-recommended) -are mature, scalable, support [high availability](../../administration/high_availability/README.md) -and are used today on GitLab.com. -It is not necessary to have GitLab installed on Kubernetes in order to use [GitLab Kubernetes integration](https://docs.gitlab.com/ee/user/project/clusters/index.html). - -The easiest method to deploy GitLab on [Kubernetes](https://kubernetes.io/) is -to take advantage of GitLab's Helm charts. [Helm](https://github.com/kubernetes/helm/blob/master/README.md) -is a package management tool for Kubernetes, allowing apps to be easily managed via their -Charts. A [Chart](https://github.com/kubernetes/charts) is a detailed description -of the application including how it should be deployed, upgraded, and configured. - -## GitLab Chart - -This chart contains all the required components to get started, and can scale to -large deployments. It offers a number of benefits: - -- Horizontal scaling of individual components -- No requirement for shared storage to scale -- Containers do not need `root` permissions -- Automatic SSL with Let's Encrypt -- An unprivileged GitLab Runner -- and plenty more. - -Learn more about the [GitLab chart](gitlab_chart.md). - -## GitLab Runner Chart - -If you already have a GitLab instance running, inside or outside of Kubernetes, -and you'd like to leverage the Runner's -[Kubernetes capabilities](https://docs.gitlab.com/runner/executors/kubernetes.html), -it can be deployed with the GitLab Runner chart. - -Learn more about [gitlab-runner chart](gitlab_runner_chart.md). +This document was moved to [another location](https://docs.gitlab.com/charts/). diff --git a/doc/install/kubernetes/preparation/connect.md b/doc/install/kubernetes/preparation/connect.md index a3a0cba4bf2..db55e03d3d4 100644 --- a/doc/install/kubernetes/preparation/connect.md +++ b/doc/install/kubernetes/preparation/connect.md @@ -1,27 +1,5 @@ -# Connecting your computer to a cluster +--- +redirect_to: https://docs.gitlab.com/charts/installation/cloud/ +--- -In order to deploy software and settings to a cluster, you must connect and authenticate to it. - -## Connect to GKE cluster - -The command for connection to the cluster can be obtained from the -[Google Cloud Platform Console](https://console.cloud.google.com/kubernetes/list) -by the individual cluster. - -Look for the **Connect** button in the clusters list page or use the command below, -filling in your cluster's information: - -``` -gcloud container clusters get-credentials <cluster-name> --zone <zone> --project <project-id> -``` - -## Connect to EKS cluster - -For the most up to date instructions, follow the Amazon EKS documentation on -[connecting to a cluster](https://docs.aws.amazon.com/eks/latest/userguide/getting-started.html#eks-configure-kubectl). - -## Connect to local minikube cluster - -If you are doing local development, you can use `minikube` as your -local cluster. If `kubectl cluster-info` is not showing `minikube` as the current -cluster, use `kubectl config set-cluster minikube` to set the active cluster. +This document was moved to [another location](https://docs.gitlab.com/charts/installation/cloud/). diff --git a/doc/install/kubernetes/preparation/eks.md b/doc/install/kubernetes/preparation/eks.md index ea3b075dd82..975d35c11c6 100644 --- a/doc/install/kubernetes/preparation/eks.md +++ b/doc/install/kubernetes/preparation/eks.md @@ -1,45 +1,5 @@ -# Running GitLab on EKS +--- +redirect_to: https://docs.gitlab.com/charts/installation/cloud/eks.html +--- -There are a few nuances to Amazon EKS which are important to be aware of, when deploying GitLab. - -## Persistent volume management - -There are two methods to manage volume claims on Kubernetes: - -1. Manually creating each persistent volume (recommended on EKS) -1. Utilizing dynamic provisioning to automatically create the persistent volumes - -### Manual provisioning of volumes (Recommended) - -Manually creating the volumes allows you to control the zone of each volume, as well as all other details supported by the underlying storage. - -Follow our documentation on [manually creating persistent volumes](https://gitlab.com/charts/gitlab/blob/master/doc/installation/storage.md#manually-creating-static-volumes). - -### Dynamic provisioning of volumes - -Dynamic provisioning utilizes a Kubernetes provisioner, like `aws-ebs`, to automatically create persistent volumes to fulfill each claim. - -With EKS, there are a few important details to keep in mind: - -1. Clusters are required to span multiple AZ's -1. Kubernetes volume provisioners create volumes across zones without regard to which pod they belong to. This leads to scenarios where a pod with multiple volumes being unable to start due to the volumes being in different zones. -1. There is no default Storage Class. - -The easiest way to solve this and still utilize dynamic provisioning is to utilize, or create, a Storage Class that is locked to a specific zone. - -> **Note**: Restricting volumes to specific zone will cause GitLab and any other application using this Storage Class to only reside in that zone. For multiple zone support, utilize [manually provisioned volumes](#manual-provisioning-of-volumes-recommended). - -To create the storage class, download and edit Amazon EKS's [sample Storage Class](https://docs.aws.amazon.com/eks/latest/userguide/storage-classes.html) and add the following parameter: - -```yaml -parameters: - zone: <desired-zone> -``` - -Then [specify the Storage Class](https://gitlab.com/charts/gitlab/blob/master/doc/installation/storage.md#using-a-custom-storage-class) name when deploying GitLab. - -## External access to GitLab - -By default, GitLab will an deploy an ingress which will create an associated Elastic Load Balancer. Since the DNS names of ELB's cannot be known ahead of time, it is difficult to utilize Let's Encrypt to automatically provision HTTPS certificates. - -We recommend [using your own certificates](https://gitlab.com/charts/gitlab/blob/master/doc/installation/tls.md#option-2-use-your-own-wildcard-certificate), and then mapping your desired DNS name to the created ELB using a CNAME record. +This document was moved to [another location](https://docs.gitlab.com/charts/installation/cloud/eks.html). diff --git a/doc/install/kubernetes/preparation/networking.md b/doc/install/kubernetes/preparation/networking.md index b9fb9a7399f..2af16a752dc 100644 --- a/doc/install/kubernetes/preparation/networking.md +++ b/doc/install/kubernetes/preparation/networking.md @@ -1,38 +1,5 @@ -# Networking Prerequisites +--- +redirect_to: https://docs.gitlab.com/charts/installation/deployment.html#networking-and-dns +--- -NOTE: **Note:** -Amazon EKS utilizes Elastic Load Balancers, which are addressed by DNS name and -cannot be known ahead of time. If you're using EKS, you can skip this section. - -The `gitlab` chart configures a GitLab server and Kubernetes cluster which can support dynamic [Review Apps](https://docs.gitlab.com/ee/ci/review_apps/index.html), as well as services like the integrated [Container Registry](https://docs.gitlab.com/ee/user/project/container_registry.html). - -To support the GitLab services and dynamic environments, a wildcard DNS entry is required which resolves to the external IP. - -## External IP - -To provision an external IP on GCP and Azure, simply request a new address from the Networking section. Ensure that the region matches the region your container cluster is created in. Note, it is important that the IP is not assigned at this point in time. It will be automatically assigned once the Helm chart is installed, to the Load Balancer. - -Set `global.hosts.externalIP` to this IP address when [deploying GitLab](../gitlab_chart.md#installing-gitlab-using-the-helm-chart). - -Then, create a [wildcard DNS record](#wildcard-dns-entry) which resolves to this IP address. - -### Creating an external IP on GCP - -When creating the external IP, it is critical to create it in the same region as your cluster. Otherwise, the IP address will fail to bind to the Load Balancer. - -1. Open the [web console](https://console.cloud.google.com) -1. In the sidebar, browse to `VPC Network > External IP addresses` -1. Click `Reserve static address` -1. Choose `Regional` and select the region of your cluster -1. Leave `Attached to` blank, as it will be automatically assigned during deployment - -## Wildcard DNS entry - -Now that an external IP address has been allocated, ensure that the wildcard DNS entry you would like to use resolves to this IP. Typically this would be an `A record` for `*`, resolving to the external IP above. - -Please consult the documentation for your DNS service for more information on creating DNS records: - -- [Google Domains](https://support.google.com/domains/answer/3290350?hl=en) -- [GoDaddy](https://www.godaddy.com/help/add-an-a-record-19238) - -Set `global.hosts.domain` to this DNS name when [deploying GitLab](../gitlab_chart.md#installing-gitlab-using-the-helm-chart). +This document was moved to [another location](https://docs.gitlab.com/charts/installation/deployment.html#networking-and-dns). diff --git a/doc/install/kubernetes/preparation/rbac.md b/doc/install/kubernetes/preparation/rbac.md index c5f8d7a7e9e..f94e7c24cdc 100644 --- a/doc/install/kubernetes/preparation/rbac.md +++ b/doc/install/kubernetes/preparation/rbac.md @@ -1,20 +1,5 @@ -# Role Based Access Control +--- +redirect_to: https://docs.gitlab.com/charts/installation/deployment.html#rbac +--- -Until Kubernetes 1.7, there were no permissions within a cluster. With the launch -of 1.7, there is now a [role based access control system (RBAC)](https://kubernetes.io/docs/admin/authorization/rbac/) -which determines what services can perform actions within a cluster. - -RBAC affects a few different aspects of GitLab: - -- [Installation of GitLab using Helm](tiller.md#preparing-for-helm-with-rbac) -- Prometheus monitoring -- GitLab Runner - -## Checking that RBAC is enabled - -Try listing the current cluster roles, if it fails then `RBAC` is disabled. -The following command will output `false` if `RBAC` is disabled and `true` otherwise: - -```sh -kubectl get clusterroles > /dev/null 2>&1 && echo true || echo false -``` +This document was moved to [another location](https://docs.gitlab.com/charts/installation/deployment.html#rbac). diff --git a/doc/install/kubernetes/preparation/tiller.md b/doc/install/kubernetes/preparation/tiller.md index 00b0737402b..66d6c8faece 100644 --- a/doc/install/kubernetes/preparation/tiller.md +++ b/doc/install/kubernetes/preparation/tiller.md @@ -1,109 +1,5 @@ -# Configuring and initializing Helm Tiller - -To make use of Helm, you must have a [Kubernetes][k8s-io] cluster. Ensure you can -access your cluster using `kubectl`. - -Helm consists of two parts, the `helm` client and a `tiller` server inside Kubernetes. - -NOTE: **Note:** -If you are not able to run Tiller in your cluster, for example on OpenShift, it -is possible to use [Tiller locally](https://gitlab.com/charts/gitlab/tree/master/doc/helm#local-tiller) -and avoid deploying it into the cluster. This should only be used when Tiller -cannot be normally deployed. - -## Initialize Helm and Tiller - -Tiller is deployed into the cluster and interacts with the Kubernetes API to deploy your applications. If role based access control (RBAC) is enabled, Tiller will need to be [granted permissions](#preparing-for-helm-with-rbac) to allow it to talk to the Kubernetes API. - -If RBAC is not enabled, skip to [initializing Helm](#initialize-helm). - -If you are not sure whether RBAC is enabled in your cluster, or to learn more, read through our [RBAC documentation](rbac.md). - -## Preparing for Helm with RBAC - -Helm's Tiller will need to be granted permissions to perform operations. These instructions grant cluster wide permissions, however for more advanced deployments [permissions can be restricted to a single namespace](https://docs.helm.sh/using_helm/#example-deploy-tiller-in-a-namespace-restricted-to-deploying-resources-only-in-that-namespace). To grant access to the cluster, we will create a new `tiller` service account and bind it to the `cluster-admin` role. - -Create a file `rbac-config.yaml` with the following contents: - -```yaml -apiVersion: v1 -kind: ServiceAccount -metadata: - name: tiller - namespace: kube-system --- -apiVersion: rbac.authorization.k8s.io/v1beta1 -kind: ClusterRoleBinding -metadata: - name: tiller -roleRef: - apiGroup: rbac.authorization.k8s.io - kind: ClusterRole - name: cluster-admin -subjects: - - kind: ServiceAccount - name: tiller - namespace: kube-system -``` - -Next we need to connect to the cluster and upload the RBAC config. - -### Upload the RBAC config - -Some clusters require authentication to use `kubectl` to create the Tiller roles. - -#### Upload the RBAC config as an admin user (GKE) - -For GKE, you need to obtain the admin credentials. This command will output the admin password: - -``` -gcloud container clusters describe <cluster-name> --zone <zone> --project <project-id> --format='value(masterAuth.password)' -``` - -Use the admin password to set the admin credentials. Replace the password value below with the output value from the above step: - -``` -kubectl config set-credentials admin --username=admin --password=xxxxxxxxxxxxxx -``` - -Once credentials have been set, create the role: - -``` -kubectl --user=admin create -f rbac-config.yaml -``` - -#### Upload the RBAC config (Non-GKE clusters) - -For other clusters like Amazon EKS, you can directly upload the RBAC configuration. - -``` -kubectl create -f rbac-config.yaml -``` - -## Initialize Helm - -Deploy Helm Tiller with a service account: - -``` -helm init --service-account tiller -``` - -If your cluster previously had Helm/Tiller installed, -run the following to ensure that the deployed version of Tiller matches the local Helm version: - -``` -helm init --upgrade --service-account tiller -``` - -### Patching Helm Tiller for Amazon EKS - -Helm Tiller requires a flag to be enabled to work properly on Amazon EKS: - -``` -kubectl -n kube-system patch deployment tiller-deploy -p '{"spec": {"template": {"spec": {"automountServiceAccountToken": true}}}}' -``` +redirect_to: https://docs.gitlab.com/charts/installation/tools.html +--- -[helm]: https://helm.sh -[helm-using]: https://docs.helm.sh/using_helm -[k8s-io]: https://kubernetes.io/ -[gcp-k8s]: https://console.cloud.google.com/kubernetes/list +This document was moved to [another location](https://docs.gitlab.com/charts/installation/tools.html). diff --git a/doc/install/kubernetes/preparation/tools_installation.md b/doc/install/kubernetes/preparation/tools_installation.md index d2f7a69a0af..66d6c8faece 100644 --- a/doc/install/kubernetes/preparation/tools_installation.md +++ b/doc/install/kubernetes/preparation/tools_installation.md @@ -1,19 +1,5 @@ -# Installing kubectl and Helm on your computer +--- +redirect_to: https://docs.gitlab.com/charts/installation/tools.html +--- -In order to work with the GitLab Helm charts, `kubectl` and `helm` must be installed and configured on your computer. - -## Installing `kubectl` - -`kubectl` is the Kubernetes command line tool, which can be used to deploy settings to the cluster. - -Follow the [official documentation](https://kubernetes.io/docs/tasks/tools/install-kubectl/) for the most up to date instructions. - -## Installing `helm` - -Helm is a package management tool for Kubernetes, and is used to deploy charts. - -You can get Helm from the project's [releases page](https://github.com/kubernetes/helm/releases), or follow other options under the official documentation of [Installing Helm](https://docs.helm.sh/using_helm/#installing-helm). - -# Next steps - -Once installed, proceed to the next [installation step](../gitlab_chart.md#installing-gitlab-using-the-helm-chart). +This document was moved to [another location](https://docs.gitlab.com/charts/installation/tools.html). diff --git a/doc/install/ldap.md b/doc/install/ldap.md new file mode 100644 index 00000000000..d8d54864586 --- /dev/null +++ b/doc/install/ldap.md @@ -0,0 +1,5 @@ +--- +redirect_to: '../administration/auth/ldap.md' +--- + +This document was moved to [another location](../administration/auth/ldap.md). diff --git a/doc/install/openshift_and_gitlab/img/pods-overview.png b/doc/install/openshift_and_gitlab/img/pods-overview.png Binary files differdeleted file mode 100644 index 65927f65f4f..00000000000 --- a/doc/install/openshift_and_gitlab/img/pods-overview.png +++ /dev/null diff --git a/doc/install/openshift_and_gitlab/img/storage-volumes.png b/doc/install/openshift_and_gitlab/img/storage-volumes.png Binary files differdeleted file mode 100644 index 3fd092919bb..00000000000 --- a/doc/install/openshift_and_gitlab/img/storage-volumes.png +++ /dev/null diff --git a/doc/install/openshift_and_gitlab/index.md b/doc/install/openshift_and_gitlab/index.md index 509020d1975..18981c43464 100644 --- a/doc/install/openshift_and_gitlab/index.md +++ b/doc/install/openshift_and_gitlab/index.md @@ -1,8 +1,5 @@ --- -author: Achilleas Pipinellis -author_gitlab: axil -level: intermediary -article_type: tutorial +type: howto date: 2016-06-28 --- @@ -11,7 +8,7 @@ date: 2016-06-28 CAUTION: **Deprecated:** This article is deprecated. Use the official Kubernetes Helm charts for installing GitLab to OpenShift. Check out the -[official installation docs](https://gitlab.com/charts/gitlab/blob/master/doc/cloud/openshift.md) +[official installation docs](https://docs.gitlab.com/charts/installation/cloud/openshift.html) for details. ## Introduction diff --git a/doc/install/pivotal/index.md b/doc/install/pivotal/index.md new file mode 100644 index 00000000000..f068572f1e9 --- /dev/null +++ b/doc/install/pivotal/index.md @@ -0,0 +1,12 @@ +# GitLab Pivotal Tile **[PREMIUM ONLY]** + +CAUTION: **Discontinued:** +As of September 13, 2017, the GitLab Enterprise Plus for Pivotal Cloud Foundry +tile on Pivotal Network has reached its End of Availability (“EoA”) and is no +longer available for download or sale through Pivotal. Current customers with +active subscriptions will continue to receive support from GitLab through their +subscription term. Pivotal and GitLab are collaborating on creating a new +Kubernetes-based tile for the Pivotal Container Service. Please contact GitLab +support with any questions regarding GitLab Enterprise Plus for Pivotal Cloud Foundry. + +Original article: <https://docs.pivotal.io/partners/gitlab/index.html>. diff --git a/doc/install/redis.md b/doc/install/redis.md index 4075e6283d0..cff5c2f2611 100644 --- a/doc/install/redis.md +++ b/doc/install/redis.md @@ -1,60 +1,5 @@ -# Install Redis on old distributions +--- +redirect_to: installation.md#7-redis +--- -GitLab requires at least Redis 2.8. The following guide is for Debian 7 and -Ubuntu 12.04. If you are using Debian 8 or Ubuntu 14.04 and up, follow the -[installation guide](installation.md). - -## Install Redis 2.8 in Debian 7 - -Redis 2.8 is included in the Debian Wheezy [backports] repository. - -1. Edit `/etc/apt/sources.list` and add the following line: - - ``` - deb http://http.debian.net/debian wheezy-backports main - ``` - -1. Update the repositories: - - ``` - sudo apt-get update - ``` - -1. Install `redis-server`: - - ``` - sudo apt-get -t wheezy-backports install redis-server - ``` - -1. Follow the rest of the [installation guide](installation.md). - -## Install Redis 2.8 in Ubuntu 12.04 - -We will [use a PPA](https://launchpad.net/~chris-lea/+archive/ubuntu/redis-server) -to install a recent version of Redis. - -1. Install the PPA repository: - - ``` - sudo add-apt-repository ppa:chris-lea/redis-server - ``` - - Your system will now fetch the PPA's key. This enables your Ubuntu system to - verify that the packages in the PPA have not been interfered with since they - were built. - -1. Update the repositories: - - ``` - sudo apt-get update - ``` - -1. Install `redis-server`: - - ``` - sudo apt-get install redis-server - ``` - -1. Follow the rest of the [installation guide](installation.md). - -[backports]: http://backports.debian.org/Instructions/ "Debian backports website" +This document was moved to [another location](installation.md#7-redis). diff --git a/doc/install/relative_url.md b/doc/install/relative_url.md index 5f129fd3bd1..96b7d0f3648 100644 --- a/doc/install/relative_url.md +++ b/doc/install/relative_url.md @@ -1,18 +1,19 @@ +--- +type: reference +--- + # Install GitLab under a relative URL -NOTE: **Note:** +While it is recommended to install GitLab on its own (sub)domain, sometimes +this is not possible due to a variety of reasons. In that case, GitLab can also +be installed under a relative URL, for example `https://example.com/gitlab`. + This document describes how to run GitLab under a relative URL for installations from source. If you are using an Omnibus package, [the steps are different][omnibus-rel]. Use this guide along with the [installation guide](installation.md) if you are installing GitLab for the first time. ---- - -While it is recommended to install GitLab on its own (sub)domain, sometimes -this is not possible due to a variety of reasons. In that case, GitLab can also -be installed under a relative URL, for example `https://example.com/gitlab`. - There is no limit to how deeply nested the relative URL can be. For example you could serve GitLab under `/foo/bar/gitlab/git` without any issues. @@ -20,8 +21,6 @@ Note that by changing the URL on an existing GitLab installation, all remote URLs will change, so you'll have to manually edit them in any local repository that points to your GitLab instance. ---- - The TL;DR list of configuration files that you need to change in order to serve GitLab under a relative URL is: @@ -126,3 +125,15 @@ To disable the relative URL: [omnibus-rel]: http://docs.gitlab.com/omnibus/settings/configuration.html#configuring-a-relative-url-for-gitlab "How to set up relative URL in Omnibus GitLab" [restart gitlab]: ../administration/restart_gitlab.md#installations-from-source "How to restart GitLab" + +<!-- ## Troubleshooting + +Include any troubleshooting steps that you can foresee. If you know beforehand what issues +one might have when setting this up, or when something is changed, or on upgrading, it's +important to describe those, too. Think of things that may go wrong and include them here. +This is important to minimize requests for support, and to avoid doc comments with +questions that you know someone might ask. + +Each scenario can be a third-level heading, e.g. `### Getting error message X`. +If you have none to add when creating a doc, leave this section in place +but commented out to help encourage others to add to it in the future. --> diff --git a/doc/install/requirements.md b/doc/install/requirements.md index c1f2297f3be..107d48fb90c 100644 --- a/doc/install/requirements.md +++ b/doc/install/requirements.md @@ -1,5 +1,12 @@ +--- +type: reference +--- + # Requirements +This page includes useful information on the supported Operating Systems as well +as the hardware requirements that are needed to install and use GitLab. + ## Operating Systems ### Supported Unix distributions @@ -12,7 +19,7 @@ - Scientific Linux (please use the CentOS packages and instructions) - Oracle Linux (please use the CentOS packages and instructions) -For the installations options please see [the installation page on the GitLab website](https://about.gitlab.com/installation/). +For the installations options, see [the main installation page](README.md). ### Unsupported Unix distributions @@ -51,6 +58,8 @@ Apart from a local hard drive you can also mount a volume that supports the netw If you have enough RAM memory and a recent CPU the speed of GitLab is mainly limited by hard drive seek times. Having a fast drive (7200 RPM and up) or a solid state drive (SSD) will improve the responsiveness of GitLab. +NOTE: **Note:** Since file system performance may affect GitLab's overall performance, we do not recommend using EFS for storage. See the [relevant documentation](../administration/high_availability/nfs.md#avoid-using-awss-elastic-file-system-efs) for more details. + ### CPU - 1 core supports up to 100 users but the application can be a bit slower due to having all workers and background jobs running on the same core @@ -85,7 +94,7 @@ if your available memory changes. We also recommend [configuring the kernel's sw to a low value like `10` to make the most of your RAM while still having the swap available when needed. -Notice: The 25 workers of Sidekiq will show up as separate processes in your process overview (such as `top` or `htop`) but they share the same RAM allocation since Sidekiq is a multithreaded application. Please see the section below about Unicorn workers for information about how many you need of those. +NOTE: **Note:** The 25 workers of Sidekiq will show up as separate processes in your process overview (such as `top` or `htop`) but they share the same RAM allocation since Sidekiq is a multithreaded application. Please see the section below about Unicorn workers for information about how many you need of those. ## Database @@ -103,8 +112,10 @@ features of GitLab work with MySQL/MariaDB: 1. MySQL support for subgroups was [dropped with GitLab 9.3][post]. See [issue #30472][30472] for more information. -1. Geo does [not support MySQL](https://docs.gitlab.com/ee/administration/geo/replication/database.html#mysql-replication). This means no supported Disaster Recovery solution if using MySQL. **[PREMIUM ONLY]** +1. Geo does [not support MySQL](../administration/geo/replication/database.md). This means no supported Disaster Recovery solution if using MySQL. **[PREMIUM ONLY]** 1. [Zero downtime migrations](../update/README.md#upgrading-without-downtime) do not work with MySQL. +1. [Database load balancing](../administration/database_load_balancing.md) is + supported only for PostgreSQL. **[PREMIUM ONLY]** 1. GitLab [optimizes the loading of dashboard events](https://gitlab.com/gitlab-org/gitlab-ce/issues/31806) using [PostgreSQL LATERAL JOINs](https://blog.heapanalytics.com/postgresqls-powerful-new-join-type-lateral/). 1. In general, SQL optimized for PostgreSQL may run much slower in MySQL due to differences in query planners. For example, subqueries that work well in PostgreSQL @@ -139,7 +150,17 @@ On some systems you may need to install an additional package (e.g. #### Additional requirements for GitLab Geo -If you are using [GitLab Geo](https://docs.gitlab.com/ee/development/geo.html), the [tracking database](https://docs.gitlab.com/ee/development/geo.html#geo-tracking-database) also requires the `postgres_fdw` extension. +If you are using [GitLab Geo](../development/geo.md): + +- We strongly recommend running Omnibus-managed instances as they are actively + developed and tested. We aim to be compatible with most external (not managed + by Omnibus) databases (for example, AWS RDS) but we do not guarantee + compatibility. +- The + [tracking database](../development/geo.md#using-the-tracking-database) + requires the + [postgres_fdw](https://www.postgresql.org/docs/9.6/static/postgres-fdw.html) + extension. ``` CREATE EXTENSION postgres_fdw; @@ -155,7 +176,7 @@ So for a machine with 2 cores, 3 unicorn workers is ideal. For all machines that have 2GB and up we recommend a minimum of three unicorn workers. If you have a 1GB machine we recommend to configure only two Unicorn workers to prevent excessive swapping. -To change the Unicorn workers when you have the Omnibus package (which defaults to the recommendation above) please see [the Unicorn settings in the Omnibus GitLab documentation](https://gitlab.com/gitlab-org/omnibus-gitlab/blob/master/doc/settings/unicorn.md#unicorn-settings). +To change the Unicorn workers when you have the Omnibus package (which defaults to the recommendation above) please see [the Unicorn settings in the Omnibus GitLab documentation](https://docs.gitlab.com/omnibus/settings/unicorn.html). ## Redis and Sidekiq @@ -187,13 +208,23 @@ you decide to run GitLab Runner and the GitLab Rails application on the same machine. It is also not safe to install everything on a single machine, because of the -[security reasons] - especially when you plan to use shell executor with GitLab +[security reasons](https://docs.gitlab.com/runner/security/), especially when you plan to use shell executor with GitLab Runner. We recommend using a separate machine for each GitLab Runner, if you plan to use the CI features. +The GitLab Runner server requirements depend on: + +- The type of [executor](https://docs.gitlab.com/runner/executors/) you configured on GitLab Runner. +- Resources required to run build jobs. +- Job concurrency settings. + +Since the nature of the jobs varies for each use case, you will need to experiment by adjusting the job concurrency to get the optimum setting. -[security reasons]: https://gitlab.com/gitlab-org/gitlab-runner/blob/master/docs/security/index.md +For reference, GitLab.com's [auto-scaling shared runner](../user/gitlab_com/index.md#shared-runners) is configured so that a **single job** will run in a **single instance** with: + +- 1vCPU. +- 3.75GB of RAM. ## Supported web browsers @@ -205,7 +236,22 @@ We support the current and the previous major release of: - Microsoft Edge - Internet Explorer 11 +The browser vendors release regular minor version updates with important bug fixes and security updates. +Support is only provided for the current minor version of the major version you are running. + Each time a new browser version is released, we begin supporting that version and stop supporting the third most recent version. -Note: We do not support running GitLab with JavaScript disabled in the browser and have no plans of supporting that +NOTE: **Note:** We do not support running GitLab with JavaScript disabled in the browser and have no plans of supporting that in the future because we have features such as Issue Boards which require JavaScript extensively. + +<!-- ## Troubleshooting + +Include any troubleshooting steps that you can foresee. If you know beforehand what issues +one might have when setting this up, or when something is changed, or on upgrading, it's +important to describe those, too. Think of things that may go wrong and include them here. +This is important to minimize requests for support, and to avoid doc comments with +questions that you know someone might ask. + +Each scenario can be a third-level heading, e.g. `### Getting error message X`. +If you have none to add when creating a doc, leave this section in place +but commented out to help encourage others to add to it in the future. --> diff --git a/doc/install/structure.md b/doc/install/structure.md index 8fc6ab4ab2f..87ef11c60fe 100644 --- a/doc/install/structure.md +++ b/doc/install/structure.md @@ -1,19 +1,5 @@ -# GitLab directory structure +--- +redirect_to: installation.md#gitlab-directory-structure +--- -This is the directory structure you will end up with following the instructions in the Installation Guide. - - |-- home - | |-- git - | |-- .ssh - | |-- gitlab - | |-- gitlab-shell - | |-- repositories - -- `/home/git/.ssh` - contains openssh settings. Specifically the `authorized_keys` file managed by gitlab-shell. -- `/home/git/gitlab` - GitLab core software. -- `/home/git/gitlab-shell` - Core add-on component of GitLab. Maintains SSH cloning and other functionality. -- `/home/git/repositories` - bare repositories for all projects organized by namespace. This is where the git repositories which are pushed/pulled are maintained for all projects. **This area is critical data for projects. [Keep a backup](../raketasks/backup_restore.md).** - -*Note: the default locations for repositories can be configured in `config/gitlab.yml` of GitLab and `config.yml` of gitlab-shell.* - -To see a more in-depth overview see the [GitLab architecture doc](../development/architecture.md). +This page was moved to [another location](#installation.md#gitlab-directory-structure). diff --git a/doc/integration/README.md b/doc/integration/README.md index f5bc0693b84..1fea6a32c28 100644 --- a/doc/integration/README.md +++ b/doc/integration/README.md @@ -15,7 +15,9 @@ See the documentation below for details on how to configure these services. - [CAS](cas.md) Configure GitLab to sign in using CAS - [External issue tracker](external-issue-tracker.md) Redmine, JIRA, etc. - [Gmail actions buttons](gmail_action_buttons_for_gitlab.md) Adds GitLab actions to messages +- [Jenkins](jenkins.md) Integrate with the Jenkins CI - [JIRA](../user/project/integrations/jira.md) Integrate with the JIRA issue tracker +- [Kerberos](kerberos.md) Integrate with Kerberos - [LDAP](ldap.md) Set up sign in via LDAP - [OAuth2 provider](oauth_provider.md) OAuth2 application creation - [OmniAuth](omniauth.md) Sign in via Twitter, GitHub, GitLab.com, Google, Bitbucket, Facebook, Shibboleth, SAML, Crowd, Azure and Authentiq ID @@ -29,8 +31,8 @@ See the documentation below for details on how to configure these services. ## Project services -Integration with services such as Campfire, Flowdock, Pivotal Tracker, and Slack -are available in the form of a [Project Service][]. +Integration with services such as Campfire, Flowdock, HipChat, +Pivotal Tracker, and Slack are available in the form of a [Project Service][]. [Project Service]: ../user/project/integrations/project_services.md diff --git a/doc/integration/chat_commands.md b/doc/integration/chat_commands.md index 2856992ee25..1a4fb46046d 100644 --- a/doc/integration/chat_commands.md +++ b/doc/integration/chat_commands.md @@ -1 +1,5 @@ +--- +redirect_to: 'slash_commands.md' +--- + This document was moved to [integration/slash_commands.md](slash_commands.md). diff --git a/doc/integration/crowd.md b/doc/integration/crowd.md index 2bc526dc3db..30e3e888b29 100644 --- a/doc/integration/crowd.md +++ b/doc/integration/crowd.md @@ -1 +1,5 @@ +--- +redirect_to: '../administration/auth/crowd.md' +--- + This document was moved to [`administration/auth/crowd`](../administration/auth/crowd.md). diff --git a/doc/integration/elasticsearch.md b/doc/integration/elasticsearch.md new file mode 100644 index 00000000000..57a5a42fbed --- /dev/null +++ b/doc/integration/elasticsearch.md @@ -0,0 +1,520 @@ +# Elasticsearch integration **[STARTER ONLY]** + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/109 "Elasticsearch Merge Request") in GitLab [Starter](https://about.gitlab.com/pricing/) 8.4. Support +> for [Amazon Elasticsearch](http://docs.aws.amazon.com/elasticsearch-service/latest/developerguide/es-gsg.html) was [introduced](https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/1305) in GitLab +> [Starter](https://about.gitlab.com/pricing/) 9.0. + +This document describes how to set up Elasticsearch with GitLab. Once enabled, +you'll have the benefit of fast search response times and the advantage of two +special searches: + +- [Advanced Global Search](../user/search/advanced_global_search.md) +- [Advanced Syntax Search](../user/search/advanced_search_syntax.md) + +## Version Requirements +<!-- Please remember to update ee/lib/system_check/app/elasticsearch_check.rb if this changes --> + +| GitLab version | Elasticsearch version | +| -------------- | --------------------- | +| GitLab Enterprise Edition 8.4 - 8.17 | Elasticsearch 2.4 with [Delete By Query Plugin](https://www.elastic.co/guide/en/elasticsearch/plugins/2.4/plugins-delete-by-query.html) installed | +| GitLab Enterprise Edition 9.0 - 11.4 | Elasticsearch 5.1 - 5.5 | +| GitLab Enterprise Edition 11.5+ | Elasticsearch 5.6 - 6.x | + +## Installing Elasticsearch + +Elasticsearch is _not_ included in the Omnibus packages. You will have to +install it yourself whether you are using the Omnibus package or installed +GitLab from source. Providing detailed information on installing Elasticsearch +is out of the scope of this document. + +Once the data is added to the database or repository and [Elasticsearch is +enabled in the admin area](#enabling-elasticsearch) the search index will be +updated automatically. Elasticsearch can be installed on the same machine as +GitLab or on a separate server, or you can use the [Amazon Elasticsearch](http://docs.aws.amazon.com/elasticsearch-service/latest/developerguide/es-gsg.html) +service. + +You can follow the steps as described in the [official web site](https://www.elastic.co/guide/en/elasticsearch/reference/current/install-elasticsearch.html "Elasticsearch installation documentation") or +use the packages that are available for your OS. + +## Elasticsearch repository indexer (beta) + +In order to improve elasticsearch indexing performance, GitLab has made available a [new indexer written in Go](https://gitlab.com/gitlab-org/gitlab-elasticsearch-indexer). +This will replace the included Ruby indexer in the future but should be considered beta software for now, so there may be some bugs. + +If you would like to use it, please follow the instructions below. + +### Installation + +First, we need to install some dependencies, then we'll build and install +the indexer itself. + +#### Dependencies + +This project relies on [ICU](http://site.icu-project.org/) for text encoding, +therefore we need to ensure the development packages for your platform are +installed before running `make`. + +##### Debian / Ubuntu + +To install on Debian or Ubuntu, run: + +```sh +sudo apt install libicu-dev +``` + +##### CentOS / RHEL + +To install on CentOS or RHEL, run: + +```sh +sudo yum install libicu-devel +``` + +##### Mac OSX + +To install on macOS, run: + +```sh +brew install icu4c +export PKG_CONFIG_PATH="/usr/local/opt/icu4c/lib/pkgconfig:$PKG_CONFIG_PATH" +``` + +#### Building and installing + +To build and install the indexer, run: + +```sh +git clone https://gitlab.com/gitlab-org/gitlab-elasticsearch-indexer.git +cd gitlab-elasticsearch-indexer +make +sudo make install +``` + +The `gitlab-elasticsearch-indexer` will be installed to `/usr/local/bin`. + +You can change the installation path with the `PREFIX` env variable. +Please remember to pass the `-E` flag to `sudo` if you do so. + +Example: + +```sh +PREFIX=/usr sudo -E make install +``` + +Once installed, enable it under your instance's elasticsearch settings explained [below](#enabling-elasticsearch). + +## System Requirements + +Elasticsearch requires additional resources in excess of those documented in the +[GitLab system requirements](../install/requirements.md). These will vary by +installation size, but you should ensure **at least** an additional **8 GiB of RAM** +for each Elasticsearch node, per the [official guidelines](https://www.elastic.co/guide/en/elasticsearch/guide/current/hardware.html). + +Keep in mind, this is the **minimum requirements** as per Elasticsearch. For +production instances, they recommend considerably more resources. + +Storage requirements also vary based on the installation side, but as a rule of +thumb, you should allocate the total size of your production database, **plus** +two-thirds of the total size of your git repositories. Efforts to reduce this +total are being tracked in this epic: [gitlab-org&153](https://gitlab.com/groups/gitlab-org/-/epics/153). + +## Enabling Elasticsearch + +In order to enable Elasticsearch, you need to have admin access. Go to +**Admin > Settings > Integrations** and find the "Elasticsearch" section. + +The following Elasticsearch settings are available: + +| Parameter | Description | +| --------- | ----------- | +| `Elasticsearch indexing` | Enables/disables Elasticsearch indexing. You may want to enable indexing but disable search in order to give the index time to be fully completed, for example. Also, keep in mind that this option doesn't have any impact on existing data, this only enables/disables background indexer which tracks data changes. So by enabling this you will not get your existing data indexed, use special rake task for that as explained in [Adding GitLab's data to the Elasticsearch index](#adding-gitlabs-data-to-the-elasticsearch-index). | +| `Use the new repository indexer (beta)` | Perform repository indexing using [GitLab Elasticsearch Indexer](https://gitlab.com/gitlab-org/gitlab-elasticsearch-indexer). | +| `Search with Elasticsearch enabled` | Enables/disables using Elasticsearch in search. | +| `URL` | The URL to use for connecting to Elasticsearch. Use a comma-separated list to support clustering (e.g., "http://host1, https://host2:9200"). If your Elasticsearch instance is password protected, pass the `username:password` in the URL (e.g., `http://<username>:<password>@<elastic_host>:9200/`). | +| `Number of Elasticsearch shards` | Elasticsearch indexes are split into multiple shards for performance reasons. In general, larger indexes need to have more shards. Changes to this value do not take effect until the index is recreated. You can read more about tradeoffs in the [Elasticsearch documentation](https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-create-index.html#create-index-settings) | +| `Number of Elasticsearch replicas` | Each Elasticsearch shard can have a number of replicas. These are a complete copy of the shard, and can provide increased query performance or resilience against hardware failure. Increasing this value will greatly increase total disk space required by the index. | +| `Limit namespaces and projects that can be indexed` | Enabling this will allow you to select namespaces and projects to index. All other namespaces and projects will use database search instead. Please note that if you enable this option but do not select any namespaces or projects, none will be indexed. [Read more below](#limiting-namespaces-and-projects). +| `Using AWS hosted Elasticsearch with IAM credentials` | Sign your Elasticsearch requests using [AWS IAM authorization](http://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_access-keys.html) or [AWS EC2 Instance Profile Credentials](http://docs.aws.amazon.com/codedeploy/latest/userguide/getting-started-create-iam-instance-profile.html#getting-started-create-iam-instance-profile-cli). The policies must be configured to allow `es:*` actions. | +| `AWS Region` | The AWS region your Elasticsearch service is located in. | +| `AWS Access Key` | The AWS access key. | +| `AWS Secret Access Key` | The AWS secret access key. | + +### Limiting namespaces and projects + +If you select `Limit namespaces and projects that can be indexed`, more options will become available + + +You can select namespaces and projects to index exclusively. Please note that if the namespace is a group it will include +any sub-groups and projects belonging to those sub-groups to be indexed as well. + +You can filter the selection dropdown by writing part of the namespace or project name you're interested in. + + +NOTE: **Note**: +If no namespaces or projects are selected, no Elasticsearch indexing will take place. + +CAUTION: **Warning**: +If you have already indexed your instance, you will have to regenerate the index in order to delete all existing data +for filtering to work correctly. To do this run the rake tasks `gitlab:elastic:create_empty_index` and +`gitlab:elastic:clear_index_status` Afterwards, removing a namespace or a projeect from the list will delete the data +from the Elasticsearch index as expected. + +## Disabling Elasticsearch + +To disable the Elasticsearch integration: + +1. Navigate to the **Admin > Settings > Integrations** +1. Find the 'Elasticsearch' section and uncheck 'Search with Elasticsearch enabled' + and 'Elasticsearch indexing' +1. Click **Save** for the changes to take effect +1. [Optional] Delete the existing index by running the command `sudo gitlab-rake gitlab:elastic:delete_index` + +## Adding GitLab's data to the Elasticsearch index + +### Indexing small instances (database size less than 500 MiB, size of repos less than 5 GiB) + +Configure Elasticsearch's host and port in **Admin > Settings**. Then index the data using one of the following commands: + +```sh +# Omnibus installations +sudo gitlab-rake gitlab:elastic:index + +# Installations from source +bundle exec rake gitlab:elastic:index RAILS_ENV=production +``` + +After it completes the indexing process, [enable Elasticsearch searching](elasticsearch.md#enabling-elasticsearch). + +### Indexing large instances + +WARNING: **Warning**: +Performing asynchronous indexing, as this will describe, will generate a lot of sidekiq jobs. +Make sure to prepare for this task by either [Horizontally Scaling](../administration/high_availability/README.md#basic-scaling) +or creating [extra sidekiq processes](../administration/operations/extra_sidekiq_processes.md) + +Configure Elasticsearch's host and port in **Admin > Settings > Integrations**. Then create empty indexes using one of the following commands: + +```sh +# Omnibus installations +sudo gitlab-rake gitlab:elastic:create_empty_index + +# Installations from source +bundle exec rake gitlab:elastic:create_empty_index RAILS_ENV=production +``` + +Indexing large Git repositories can take a while. To speed up the process, you +can temporarily disable auto-refreshing and replicating. In our experience, you can expect a 20% +decrease in indexing time. We'll enable them when indexing is done. This step is optional! + +```bash +curl --request PUT localhost:9200/gitlab-production/_settings --data '{ + "index" : { + "refresh_interval" : "-1", + "number_of_replicas" : 0 + } }' +``` + +Then enable Elasticsearch indexing and run project indexing tasks: + +```sh +# Omnibus installations +sudo gitlab-rake gitlab:elastic:index_projects + +# Installations from source +bundle exec rake gitlab:elastic:index_projects RAILS_ENV=production +``` + +This enqueues a Sidekiq job for each project that needs to be indexed. +You can view the jobs in the admin panel (they are placed in the `elastic_indexer` +queue), or you can query indexing status using a rake task: + +```sh +# Omnibus installations +sudo gitlab-rake gitlab:elastic:index_projects_status + +# Installations from source +bundle exec rake gitlab:elastic:index_projects_status RAILS_ENV=production + +Indexing is 65.55% complete (6555/10000 projects) +``` + +If you want to limit the index to a range of projects you can provide the +`ID_FROM` and `ID_TO` parameters: + +```sh +# Omnibus installations +sudo gitlab-rake gitlab:elastic:index_projects ID_FROM=1001 ID_TO=2000 + +# Installations from source +bundle exec rake gitlab:elastic:index_projects ID_FROM=1001 ID_TO=2000 RAILS_ENV=production +``` + +Where `ID_FROM` and `ID_TO` are project IDs. Both parameters are optional. +The above examples will index all projects starting with ID `1001` up to (and including) ID `2000`. + +TIP: **Troubleshooting:** +Sometimes the project indexing jobs queued by `gitlab:elastic:index_projects` +can get interrupted. This may happen for many reasons, but it's always safe +to run the indexing task again - it will skip those repositories that have +already been indexed. + +As the indexer stores the last commit SHA of every indexed repository in the +database, you can run the indexer with the special parameter `UPDATE_INDEX` and +it will check every project repository again to make sure that every commit in +that repository is indexed, it can be useful in case if your index is outdated: + +```sh +# Omnibus installations +sudo gitlab-rake gitlab:elastic:index_projects UPDATE_INDEX=true ID_TO=1000 + +# Installations from source +bundle exec rake gitlab:elastic:index_projects UPDATE_INDEX=true ID_TO=1000 RAILS_ENV=production +``` + +You can also use the `gitlab:elastic:clear_index_status` Rake task to force the +indexer to "forget" all progress, so retrying the indexing process from the +start. + +To index all wikis: + +```sh +# Omnibus installations +sudo gitlab-rake gitlab:elastic:index_wikis + +# Installations from source +bundle exec rake gitlab:elastic:index_wikis RAILS_ENV=production +``` + +The wiki indexer also supports the `ID_FROM` and `ID_TO` parameters if you want +to limit a project set. + +Enable replication and refreshing again after indexing (only if you previously disabled it): + +```bash +curl --request PUT localhost:9200/gitlab-production/_settings --data '{ + "index" : { + "number_of_replicas" : 1, + "refresh_interval" : "1s" + } }' +``` + +A force merge should be called after enabling the refreshing above. + +For Elasticsearch 6.x, before proceeding with the force merge, the index should be in read-only mode: + +```bash +curl --request PUT localhost:9200/gitlab-production/_settings --data '{ + "settings": { + "index.blocks.write": true + } }' +``` + +Then, initiate the force merge: + +```bash +curl --request POST 'http://localhost:9200/gitlab-production/_forcemerge?max_num_segments=5' +``` + +After this, if your index is in read-only, switch back to read-write: + +```bash +curl --request PUT localhost:9200/gitlab-production/_settings --data '{ + "settings": { + "index.blocks.write": false + } }' +``` + +Enable Elasticsearch search in **Admin > Settings > Integrations**. That's it. Enjoy it! + +## GitLab Elasticsearch Rake Tasks + +There are several rake tasks available to you via the command line: + +- [sudo gitlab-rake gitlab:elastic:index](https://gitlab.com/gitlab-org/gitlab-ee/blob/master/ee/lib/tasks/gitlab/elastic.rake) + - This is a wrapper task. It does the following: + - `sudo gitlab-rake gitlab:elastic:create_empty_index` + - `sudo gitlab-rake gitlab:elastic:clear_index_status` + - `sudo gitlab-rake gitlab:elastic:index_projects` + - `sudo gitlab-rake gitlab:elastic:index_wikis` + - `sudo gitlab-rake gitlab:elastic:index_snippets` +- [sudo gitlab-rake gitlab:elastic:index_projects](https://gitlab.com/gitlab-org/gitlab-ee/blob/master/ee/lib/tasks/gitlab/elastic.rake) + - This iterates over all projects and queues sidekiq jobs to index them in the background. +- [sudo gitlab-rake gitlab:elastic:index_projects_status](https://gitlab.com/gitlab-org/gitlab-ee/blob/master/ee/lib/tasks/gitlab/elastic.rake) + - This determines the overall status of the indexing. It is done by counting the total number of indexed projects, dividing by a count of the total number of projects, then multiplying by 100. +- [sudo gitlab-rake gitlab:elastic:index_wikis](https://gitlab.com/gitlab-org/gitlab-ee/blob/master/ee/lib/tasks/gitlab/elastic.rake) + - Iterates over every project, determines if said project contains wiki data, and then indexes the blobs (content) of said wiki data. +- [sudo gitlab-rake gitlab:elastic:create_empty_index](https://gitlab.com/gitlab-org/gitlab-ee/blob/master/ee/lib/tasks/gitlab/elastic.rake) + - This generates an empty index on the Elasticsearch side. +- [sudo gitlab-rake gitlab:elastic:clear_index_status](https://gitlab.com/gitlab-org/gitlab-ee/blob/master/ee/lib/tasks/gitlab/elastic.rake) + - This deletes all instances of IndexStatus for all projects. +- [sudo gitlab-rake gitlab:elastic:delete_index](https://gitlab.com/gitlab-org/gitlab-ee/blob/master/ee/lib/tasks/gitlab/elastic.rake) + - This removes the GitLab index on the Elasticsearch instance. +- [sudo gitlab-rake gitlab:elastic:recreate_index](https://gitlab.com/gitlab-org/gitlab-ee/blob/master/ee/lib/tasks/gitlab/elastic.rake) + - Does the same thing as `sudo gitlab-rake gitlab:elastic:create_empty_index` +- [sudo gitlab-rake gitlab:elastic:add_feature_visibility_levels_to_project](https://gitlab.com/gitlab-org/gitlab-ee/blob/master/ee/lib/tasks/gitlab/elastic.rake) + - Adds visibility information to the indices for projects. +- [sudo gitlab-rake gitlab:elastic:index_snippets](https://gitlab.com/gitlab-org/gitlab-ee/blob/master/ee/lib/tasks/gitlab/elastic.rake) + - Performs an Elasticsearch import that indexes the snippets data. + +### Environment Variables + +In addition to the rake tasks, there are some environment variables that can be used to modify the process: + +| Environment Variable | Data Type | What it does | +| -------------------- |:---------:| ---------------------------------------------------------------------------- | +| `UPDATE_INDEX` | Boolean | Tells the indexer to overwrite any existing index data (true/false). | +| `ID_TO` | Integer | Tells the indexer to only index projects less than or equal to the value. | +| `ID_FROM` | Integer | Tells the indexer to only index projects greater than or equal to the value. | + +### Indexing a specific project + +Because the `ID_TO` and `ID_FROM` environment variables use the `or equal to` comparison, you can index only one project by using both these variables with the same project ID number: + +```sh +root@git:~# sudo gitlab-rake gitlab:elastic:index_projects ID_TO=5 ID_FROM=5 +Indexing project repositories...I, [2019-03-04T21:27:03.083410 #3384] INFO -- : Indexing GitLab User / test (ID=33)... +I, [2019-03-04T21:27:05.215266 #3384] INFO -- : Indexing GitLab User / test (ID=33) is done! +``` + +## Elasticsearch Index Scopes + +When performing a search, the GitLab index will use the following scopes: + +| Scope Name | What it searches | +| ---------------- | ---------------------- | +| `commits` | Commit data | +| `projects` | Project data (default) | +| `blobs` | Code | +| `issues` | Issue data | +| `merge_requests` | Merge Request data | +| `milestones` | Milestone data | +| `notes` | Note data | +| `snippets` | Snippet data | +| `wiki_blobs` | Wiki contents | + +## Tuning + +### Deleted documents + +Whenever a change or deletion is made to an indexed GitLab object (a merge request description is changed, a file is deleted from the master branch in a repository, a project is deleted, etc), a document in the index is deleted. However, since these are "soft" deletes, the overall number of "deleted documents", and therefore wasted space, increases. Elasticsearch does intelligent merging of segments in order to remove these deleted documents. However, depending on the amount and type of activity in your GitLab installation, it's possible to see as much as 50% wasted space in the index. + +In general, we recommend simply letting Elasticseach merge and reclaim space automatically, with the default settings. From [Lucene's Handling of Deleted Documents](https://www.elastic.co/blog/lucenes-handling-of-deleted-documents "Lucene's Handling of Deleted Documents"), _"Overall, besides perhaps decreasing the maximum segment size, it is best to leave Lucene's defaults as-is and not fret too much about when deletes are reclaimed."_ + +However, some larger installations may wish to tune the merge policy settings: + +- Consider reducing the `index.merge.policy.max_merged_segment` size from the default 5 GB to maybe 2 GB or 3 GB. Merging only happens when a segment has at least 50% deletions. Smaller segment sizes will allow merging to happen more frequently. + + ```bash + curl --request PUT http://localhost:9200/gitlab-production/_settings --data '{ + "index" : { + "merge.policy.max_merged_segment": "2gb" + } + }' + ``` + +- You can also adjust `index.merge.policy.reclaim_deletes_weight`, which controls how aggressively deletions are targeted. But this can lead to costly merge decisions, so we recommend not changing this unless you understand the tradeoffs. + + ```bash + curl --request PUT http://localhost:9200/gitlab-production/_settings --data '{ + "index" : { + "merge.policy.reclaim_deletes_weight": "3.0" + } + }' + ``` + +- Do not do a [force merge](https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-forcemerge.html "Force Merge") to remove deleted documents. A warning in the [documentation](https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-forcemerge.html "Force Merge") states that this can lead to very large segments that may never get reclaimed, and can also cause significant performance or availability issues. + +## Troubleshooting + +Here are some common pitfalls and how to overcome them: + +- **How can I verify my GitLab instance is using Elasticsearch?** + + The easiest method is via the rails console (`sudo gitlab-rails console`) by running the following: + + ```ruby + u = User.find_by_username('your-username') + s = SearchService.new(u, {:search => 'search_term'}) + pp s.search_objects.class.name + ``` + + If you see `Elasticsearch::Model::Response::Records`, you are using Elasticsearch. + +- **I updated GitLab and now I can't find anything** + + We continuously make updates to our indexing strategies and aim to support + newer versions of Elasticsearch. When indexing changes are made, it may + be necessary for you to [reindex](#adding-gitlabs-data-to-the-elasticsearch-index) after updating GitLab. + +- **I indexed all the repositories but I can't find anything** + + Make sure you indexed all the database data [as stated above](#adding-gitlabs-data-to-the-elasticsearch-index). + + Beyond that, check via the [Elasticsearch Search API](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-search.html) to see if the data shows up on the Elasticsearch side. + + If it shows up via the [Elasticsearch Search API](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-search.html), check that it shows up via the rails console (`sudo gitlab-rails console`): + + ```ruby + u = User.find_by_username('your-username') + s = SearchService.new(u, {:search => 'search_term', :scope => ‘blobs’}) + pp s.search_objects.to_a + ``` + + See [Elasticsearch Index Scopes](elasticsearch.md#elasticsearch-index-scopes) for more information on searching for specific types of data. + +- **I indexed all the repositories but then switched elastic search servers and now I can't find anything** + + You will need to re-run all the rake tasks to re-index the database, repositories, and wikis. + +- **The indexing process is taking a very long time** + + The more data present in your GitLab instance, the longer the indexing process takes. + +- **No new data is added to the Elasticsearch index when I push code** + + When performing the initial indexing of blobs, we lock all projects until the project finishes indexing. It could + happen that an error during the process causes one or multiple projects to remain locked. In order to unlock them, + run the `gitlab:elastic:clear_locked_projects` rake task. + +- **"Can't specify parent if no parent field has been configured"** + + If you enabled Elasticsearch before GitLab 8.12 and have not rebuilt indexes you will get + exception in lots of different cases: + + ```text + Elasticsearch::Transport::Transport::Errors::BadRequest([400] { + "error": { + "root_cause": [{ + "type": "illegal_argument_exception", + "reason": "Can't specify parent if no parent field has been configured" + }], + "type": "illegal_argument_exception", + "reason": "Can't specify parent if no parent field has been configured" + }, + "status": 400 + }): + ``` + + This is because we changed the index mapping in GitLab 8.12 and the old indexes should be removed and built from scratch again, + see details in the [8-11-to-8-12 update guide](https://gitlab.com/gitlab-org/gitlab-ee/blob/master/doc/update/8.11-to-8.12.md#11-elasticsearch-index-update-if-you-currently-use-elasticsearch). + +- Exception `Elasticsearch::Transport::Transport::Errors::BadRequest` + + If you have this exception (just like in the case above but the actual message is different) please check if you have the correct Elasticsearch version and you met the other [requirements](#system-requirements). + There is also an easy way to check it automatically with `sudo gitlab-rake gitlab:check` command. + +- Exception `Elasticsearch::Transport::Transport::Errors::RequestEntityTooLarge` + + ```text + [413] {"Message":"Request size exceeded 10485760 bytes"} + ``` + + This exception is seen when your Elasticsearch cluster is configured to reject + requests above a certain size (10MiB in this case). This corresponds to the + `http.max_content_length` setting in `elasticsearch.yml`. Increase it to a + larger size and restart your Elasticsearch cluster. + + AWS has [fixed limits](http://docs.aws.amazon.com/elasticsearch-service/latest/developerguide/aes-limits.html) + for this setting ("Maximum Size of HTTP Request Payloads"), based on the size of + the underlying instance. + diff --git a/doc/integration/external-issue-tracker.md b/doc/integration/external-issue-tracker.md index 075feaeead9..c3328e01081 100644 --- a/doc/integration/external-issue-tracker.md +++ b/doc/integration/external-issue-tracker.md @@ -1,14 +1,17 @@ # External issue tracker -GitLab has a great issue tracker but you can also use an external one such as -Jira, Redmine, or Bugzilla. Issue trackers are configurable per GitLab project and allow -you to do the following: +GitLab has a great [issue tracker](../user/project/issues/index.md) but you can also use an external one +such as Jira, Redmine, YouTrack, or Bugzilla. External issue trackers are configurable per GitLab project. -- you can reference these external issues inside GitLab interface - (merge requests, commits, comments) and they will be automatically converted - into links +Once configured, you can reference external issues using the format `CODE-123`, where: -You can have enabled both external and internal GitLab issue trackers in parallel. The **Issues** link always opens the internal issue tracker and in case the internal issue tracker is disabled the link is not visible in the menu. +- `CODE` is a unique code for the tracker. +- `123` is the issue number in the tracker. + +These references in GitLab merge requests, commits, or comments are automatically converted to links to the issues. + +You can keep GitLab's issue tracker enabled in parallel or disable it. When enabled, the **Issues** link in the +GitLab menu always opens the internal issue tracker. When disabled, the link is not visible in the menu. ## Configuration @@ -20,6 +23,7 @@ To enable an external issue tracker you must configure the appropriate **Service Visit the links below for details: - [Redmine](../user/project/integrations/redmine.md) +- [YouTrack](../user/project/integrations/youtrack.md) - [Jira](../user/project/integrations/jira.md) - [Bugzilla](../user/project/integrations/bugzilla.md) - [Custom Issue Tracker](../user/project/integrations/custom_issue_tracker.md) diff --git a/doc/integration/github.md b/doc/integration/github.md index bee68688ace..5b01dd9feb7 100644 --- a/doc/integration/github.md +++ b/doc/integration/github.md @@ -19,12 +19,12 @@ To get the credentials (a pair of Client ID and Client Secret), you must registe 1. Provide the required details. - Application name: This can be anything. Consider something like `<Organization>'s GitLab` or `<Your Name>'s GitLab` or something else descriptive. - - Homepage URL: the URL to your GitLab installation. e.g., `https://gitlab.company.com` + - Homepage URL: The URL of your GitLab installation. For example, `https://gitlab.example.com`. - Application description: Fill this in if you wish. - - Authorization callback URL: `http(s)://${YOUR_DOMAIN}/users/auth/github/callback`. Please make sure the port is included if your GitLab instance is not configured on default port. + - Authorization callback URL: `http(s)://${YOUR_DOMAIN}/users/auth`. Please make sure the port is included if your GitLab instance is not configured on default port.  - NOTE: Be sure to append `/users/auth/github/callback` to the end of the callback URL + NOTE: Be sure to append `/users/auth` to the end of the callback URL to prevent a [OAuth2 convert redirect](http://tetraph.com/covert_redirect/) vulnerability. diff --git a/doc/integration/img/github_app_entry.png b/doc/integration/img/github_app_entry.png Binary files differindex 0a1fe0ca65a..c736bed3dd6 100644 --- a/doc/integration/img/github_app_entry.png +++ b/doc/integration/img/github_app_entry.png diff --git a/doc/integration/img/github_register_app.png b/doc/integration/img/github_register_app.png Binary files differindex 5786b822f53..d03ca5cc2aa 100644 --- a/doc/integration/img/github_register_app.png +++ b/doc/integration/img/github_register_app.png diff --git a/doc/integration/img/google_app.png b/doc/integration/img/google_app.png Binary files differdeleted file mode 100644 index 08f230452b4..00000000000 --- a/doc/integration/img/google_app.png +++ /dev/null diff --git a/doc/integration/img/jenkins_gitlab_plugin_config.png b/doc/integration/img/jenkins_gitlab_plugin_config.png Binary files differnew file mode 100644 index 00000000000..3e1f83e96dc --- /dev/null +++ b/doc/integration/img/jenkins_gitlab_plugin_config.png diff --git a/doc/integration/img/jenkins_gitlab_service.png b/doc/integration/img/jenkins_gitlab_service.png Binary files differnew file mode 100644 index 00000000000..682a5ae8ee2 --- /dev/null +++ b/doc/integration/img/jenkins_gitlab_service.png diff --git a/doc/integration/img/jenkins_gitlab_service_settings.png b/doc/integration/img/jenkins_gitlab_service_settings.png Binary files differnew file mode 100644 index 00000000000..5a12e9cb39a --- /dev/null +++ b/doc/integration/img/jenkins_gitlab_service_settings.png diff --git a/doc/integration/img/jenkins_project.png b/doc/integration/img/jenkins_project.png Binary files differnew file mode 100644 index 00000000000..126b05c8879 --- /dev/null +++ b/doc/integration/img/jenkins_project.png diff --git a/doc/integration/img/jira_dev_panel_gl_setup_1.png b/doc/integration/img/jira_dev_panel_gl_setup_1.png Binary files differnew file mode 100644 index 00000000000..75279877c93 --- /dev/null +++ b/doc/integration/img/jira_dev_panel_gl_setup_1.png diff --git a/doc/integration/img/jira_dev_panel_jira_setup_1.png b/doc/integration/img/jira_dev_panel_jira_setup_1.png Binary files differnew file mode 100644 index 00000000000..5c0f594cc1d --- /dev/null +++ b/doc/integration/img/jira_dev_panel_jira_setup_1.png diff --git a/doc/integration/img/jira_dev_panel_jira_setup_2.png b/doc/integration/img/jira_dev_panel_jira_setup_2.png Binary files differnew file mode 100644 index 00000000000..a4778a00dd5 --- /dev/null +++ b/doc/integration/img/jira_dev_panel_jira_setup_2.png diff --git a/doc/integration/img/jira_dev_panel_jira_setup_3.png b/doc/integration/img/jira_dev_panel_jira_setup_3.png Binary files differnew file mode 100644 index 00000000000..4049a65f56b --- /dev/null +++ b/doc/integration/img/jira_dev_panel_jira_setup_3.png diff --git a/doc/integration/img/jira_dev_panel_jira_setup_4.png b/doc/integration/img/jira_dev_panel_jira_setup_4.png Binary files differnew file mode 100644 index 00000000000..81d84cb173d --- /dev/null +++ b/doc/integration/img/jira_dev_panel_jira_setup_4.png diff --git a/doc/integration/img/jira_dev_panel_jira_setup_5.png b/doc/integration/img/jira_dev_panel_jira_setup_5.png Binary files differnew file mode 100644 index 00000000000..73dc867d301 --- /dev/null +++ b/doc/integration/img/jira_dev_panel_jira_setup_5.png diff --git a/doc/integration/img/jira_dev_panel_manual_refresh.png b/doc/integration/img/jira_dev_panel_manual_refresh.png Binary files differnew file mode 100644 index 00000000000..dc92d533bde --- /dev/null +++ b/doc/integration/img/jira_dev_panel_manual_refresh.png diff --git a/doc/integration/img/limit_namespace_filter.png b/doc/integration/img/limit_namespace_filter.png Binary files differnew file mode 100644 index 00000000000..88f5caa41db --- /dev/null +++ b/doc/integration/img/limit_namespace_filter.png diff --git a/doc/integration/img/limit_namespaces_projects_options.png b/doc/integration/img/limit_namespaces_projects_options.png Binary files differnew file mode 100644 index 00000000000..488341f7e92 --- /dev/null +++ b/doc/integration/img/limit_namespaces_projects_options.png diff --git a/doc/integration/img/salesforce_app_details.png b/doc/integration/img/salesforce_app_details.png Binary files differnew file mode 100644 index 00000000000..00e66f07282 --- /dev/null +++ b/doc/integration/img/salesforce_app_details.png diff --git a/doc/integration/img/salesforce_app_secret_details.png b/doc/integration/img/salesforce_app_secret_details.png Binary files differnew file mode 100644 index 00000000000..fad2a4a1f97 --- /dev/null +++ b/doc/integration/img/salesforce_app_secret_details.png diff --git a/doc/integration/img/salesforce_oauth_app_details.png b/doc/integration/img/salesforce_oauth_app_details.png Binary files differnew file mode 100644 index 00000000000..a5fb680cca6 --- /dev/null +++ b/doc/integration/img/salesforce_oauth_app_details.png diff --git a/doc/integration/img/ultra_auth_credentials.png b/doc/integration/img/ultra_auth_credentials.png Binary files differnew file mode 100644 index 00000000000..cff98a4b056 --- /dev/null +++ b/doc/integration/img/ultra_auth_credentials.png diff --git a/doc/integration/img/ultra_auth_edit_callback_url.png b/doc/integration/img/ultra_auth_edit_callback_url.png Binary files differnew file mode 100644 index 00000000000..b7548122c5e --- /dev/null +++ b/doc/integration/img/ultra_auth_edit_callback_url.png diff --git a/doc/integration/img/ultra_auth_edit_callback_url_highlighted.png b/doc/integration/img/ultra_auth_edit_callback_url_highlighted.png Binary files differnew file mode 100644 index 00000000000..4abf224756c --- /dev/null +++ b/doc/integration/img/ultra_auth_edit_callback_url_highlighted.png diff --git a/doc/integration/jenkins.md b/doc/integration/jenkins.md new file mode 100644 index 00000000000..f60333aa07c --- /dev/null +++ b/doc/integration/jenkins.md @@ -0,0 +1,129 @@ +# Jenkins CI service **[STARTER]** + +>**Note:** +In GitLab 8.3, Jenkins integration using the +[GitLab Hook Plugin](https://wiki.jenkins.io/display/JENKINS/GitLab+Hook+Plugin) +was deprecated in favor of the +[GitLab Plugin](https://wiki.jenkins.io/display/JENKINS/GitLab+Plugin). +The deprecated integration has been renamed to [Jenkins CI (Deprecated)](jenkins_deprecated.md) in the +project service settings. We may remove this in a future release and recommend +using the new 'Jenkins CI' project service instead which is described in this +document. + +## Overview + +[Jenkins](https://jenkins.io/) is a great Continuous Integration tool, similar to our built-in +[GitLab CI](../ci/README.md). + +GitLab's Jenkins integration allows you to trigger a Jenkins build when you +push code to a repository, or when a merge request is created. Additionally, +it shows the pipeline status on merge requests widgets and on the project's home page. + +## Use cases + +- Suppose you are new to GitLab, and want to keep using Jenkins until you prepare +your projects to build with [GitLab CI/CD](../ci/README.md). You set up the +integration between GitLab and Jenkins, then you migrate to GitLab CI later. While +you organize yourself and your team to onboard GitLab, you keep your pipelines +running with Jenkins, but view the results in your project's repository in GitLab. +- Your team uses [Jenkins Plugins](https://plugins.jenkins.io/) for other proceedings, +therefore, you opt for keep using Jenkins to build your apps. Show the results of your +pipelines directly in GitLab. + +## Requirements + +* [Jenkins GitLab Plugin](https://wiki.jenkins.io/display/JENKINS/GitLab+Plugin) +* [Jenkins Git Plugin](https://wiki.jenkins.io/display/JENKINS/Git+Plugin) +* Git clone access for Jenkins from the GitLab repository +* GitLab API access to report build status + +## Configure GitLab users + +Create a user or choose an existing user that Jenkins will use to interact +through the GitLab API. This user will need to be a global Admin or added +as a member to each Group/Project. Developer permission is required for reporting +build status. This is because a successful build status can trigger a merge +when 'Merge when pipeline succeeds' feature is used. Some features of the GitLab +Plugin may require additional privileges. For example, there is an option to +accept a merge request if the build is successful. Using this feature would +require developer, maintainer or owner-level permission. + +Copy the private API token from **Profile Settings -> Account**. You will need this +when configuring the Jenkins server later. + +## Configure the Jenkins server + +Install [Jenkins GitLab Plugin](https://wiki.jenkins.io/display/JENKINS/GitLab+Plugin) +and [Jenkins Git Plugin](https://wiki.jenkins.io/display/JENKINS/Git+Plugin). + +Go to Manage Jenkins -> Configure System and scroll down to the 'GitLab' section. +Enter the GitLab server URL in the 'GitLab host URL' field and paste the API token +copied earlier in the 'API Token' field. + +For more information, see GitLab Plugin documentation about +[Jenkins-to-GitLab authentication](https://github.com/jenkinsci/gitlab-plugin#jenkins-to-gitlab-authentication) + + + +## Configure a Jenkins project + +Follow the GitLab Plugin documentation about [Jenkins Job Configuration](https://github.com/jenkinsci/gitlab-plugin#jenkins-job-configuration). + +NOTE: **Note:** +Be sure to include the steps about [Build status configuration](https://github.com/jenkinsci/gitlab-plugin#build-status-configuration). +The 'Publish build status to GitLab' post-build step is required to view +Jenkins build status in GitLab Merge Requests. + +## Configure a GitLab project + +Create a new GitLab project or choose an existing one. Then, go to **Integrations -> +Jenkins CI**. + +Check the 'Active' box. Select whether you want GitLab to trigger a build +on push, Merge Request creation, tag push, or any combination of these. We +recommend unchecking 'Merge Request events' unless you have a specific use-case +that requires re-building a commit when a merge request is created. With 'Push +events' selected, GitLab will build the latest commit on each push and the build +status will be displayed in the merge request. + +Enter the Jenkins URL and Project name. The project name should be URL-friendly +where spaces are replaced with underscores. To be safe, copy the project name +from the URL bar of your browser while viewing the Jenkins project. + +Optionally, enter a username and password if your Jenkins server requires +authentication. + + + +## Plugin functional overview + +GitLab does not contain a database table listing commits. Commits are always +read from the repository directly. Therefore, it is not possible to retain the +build status of a commit in GitLab. This is overcome by requesting build +information from the integrated CI tool. The CI tool is responsible for creating +and storing build status for Commits and Merge Requests. + +### Steps required to implement a similar integration + +>**Note:** +All steps are implemented using AJAX requests on the merge request page. + +1. In order to display the build status in a merge request you must create a project service in GitLab. +2. Your project service will do a (JSON) query to a URL of the CI tool with the SHA1 of the commit. +3. The project service builds this URL and payload based on project service settings and knowledge of the CI tool. +4. The response is parsed to give a response in GitLab (success/failed/pending). + +## Troubleshooting + +### Error in merge requests - "Could not connect to the CI server" + +This integration relies on Jenkins reporting the build status back to GitLab via +the [Commit Status API](../api/commits.md#commit-status). + +The error 'Could not connect to the CI server' usually means that GitLab did not +receive a build status update via the API. Either Jenkins was not properly +configured or there was an error reporting the status via the API. + +1. [Configure the Jenkins server](#configure-the-jenkins-server) for GitLab API access +2. [Configure a Jenkins project](#configure-a-jenkins-project), including the + 'Publish build status to GitLab' post-build action. diff --git a/doc/integration/jenkins_deprecated.md b/doc/integration/jenkins_deprecated.md new file mode 100644 index 00000000000..8001c5dbd83 --- /dev/null +++ b/doc/integration/jenkins_deprecated.md @@ -0,0 +1,53 @@ +# Jenkins CI (deprecated) service + +>**Note:** In GitLab 8.3, Jenkins integration using the +[GitLab Hook Plugin](https://wiki.jenkins.io/display/JENKINS/GitLab+Hook+Plugin) +was deprecated in favor of the +[GitLab Plugin](https://wiki.jenkins.io/display/JENKINS/GitLab+Plugin). +Please use documentation for the new [Jenkins CI service](jenkins.md). + +Integration includes: + +* Trigger Jenkins build after push to repo +* Show build status on Merge Request page + +Requirements: + +* [Jenkins GitLab Hook plugin](https://wiki.jenkins.io/display/JENKINS/GitLab+Hook+Plugin) +* git clone access for Jenkins from GitLab repo (via ssh key) + +## Jenkins + +1. Install [GitLab Hook plugin](https://wiki.jenkins.io/display/JENKINS/GitLab+Hook+Plugin) +2. Set up jenkins project + + + +## GitLab + +In GitLab, perform the following steps. + +### Read access to repository + +Jenkins needs read access to the GitLab repository. We already specified a +private key to use in Jenkins, now we need to add a public one to the GitLab +project. For that case we will need a Deploy key. Read the documentation on +[how to set up a Deploy key](../ssh/README.md#deploy-keys). + +### Jenkins service + +Now navigate to GitLab services page and activate Jenkins + + + +Done! Now when you push to GitLab - it will create a build for Jenkins. +And also you will be able to see merge request build status with a link to the Jenkins build. + +### Multi-project Configuration + +The GitLab Hook plugin in Jenkins supports the automatic creation of a project +for each feature branch. After configuration GitLab will trigger feature branch +builds and a corresponding project will be created in Jenkins. + +Configure the GitLab Hook plugin in Jenkins. Go to 'Manage Jenkins' and then +'Configure System'. Find the 'GitLab Web Hook' section and configure as shown below. diff --git a/doc/integration/jira.md b/doc/integration/jira.md index b6923f74e28..c09fde08326 100644 --- a/doc/integration/jira.md +++ b/doc/integration/jira.md @@ -1 +1,5 @@ +--- +redirect_to: '../user/project/integrations/jira.md' +--- + This document was moved to [integrations/jira](../user/project/integrations/jira.md). diff --git a/doc/integration/jira_development_panel.md b/doc/integration/jira_development_panel.md new file mode 100644 index 00000000000..703736eeb3c --- /dev/null +++ b/doc/integration/jira_development_panel.md @@ -0,0 +1,141 @@ +# GitLab Jira development panel integration **[PREMIUM]** + +> [Introduced][ee-2381] in [GitLab Premium][eep] 10.0. + +Complementary to our [existing Jira][existing-jira] project integration, you're now able to integrate +GitLab projects with [Jira Development Panel][jira-development-panel]. Both can be used +simultaneously. This works with self-hosted GitLab or GitLab.com integrated with self-hosted Jira +or cloud Jira. + +By doing this you can easily access related GitLab merge requests, branches, and commits directly from a Jira issue. + +This integration connects all GitLab projects within a top-level group or a personal namespace to projects in the Jira instance. +A top-level GitLab group is one that does not have any parent group itself. All the projects of that top-level group, +as well as projects of the top-level group's subgroups nesting down, are connected. Alternatively, you can specify +a GitLab personal namespace in the Jira configuration, which will then connect the projects in that personal namespace to Jira. + +NOTE: **Note**: +Note this is different from the [existing Jira][existing-jira] project integration, where the mapping +is one GitLab project to the entire Jira instance. + +We recommend that a GitLab group admin +or instance admin (in the case of self-hosted GitLab) set up the integration, +in order to simplify administration. + +TIP: **Tip:** +Create and use a single-purpose "jira" user in GitLab, so that removing +regular users won't impact your integration. + +## Requirements + +### Self-hosted GitLab + +If you are using self-hosted GitLab, make sure your GitLab instance is accessible by Jira. + +- If you are connecting to Jira Cloud, make sure your instance is accessible via the internet. +- If you are using Jira Server, make sure your instance is accessible however your network is set up. + +### GitLab.com + +There are no special requirements if you are using GitLab.com. + +## GitLab Configuration + +1. In GitLab, create a new application in order to allow Jira to connect with your GitLab account + + While logged-in, go to `Settings -> Applications`. (Click your profile avatar at + the top right, choose `Settings`, and then navigate to `Applications` from the left + navigation menu.) Use the form to create a new application. + + Enter a useful name for the `Name` field. + + For the `Redirect URI` field, enter `https://<your-gitlab-instance-domain>/login/oauth/callback`, + replacing `<your-gitlab-instance-domain>` appropriately. So for example, if you are using GitLab.com, + this would be `https://gitlab.com/login/oauth/callback`. + + NOTE: **Note**: + If using a GitLab version earlier than 11.3 the `Redirect URI` value should be `https://<your-gitlab-instance-domain>/-/jira/login/oauth/callback`. + +  + - Check `api` in the Scopes section. + +2. Click `Save application`. You will see the generated 'Application Id' and 'Secret' values. + Copy these values that you will use on the Jira configuration side. + +## Jira Configuration + +1. In Jira, from the gear menu at the top right, go to `Applications`. Navigate to `DVCS accounts` + from the left navigation menu. Click `Link GitHub account` to start creating a new integration. + (We are pretending to be GitHub in this integration until there is further platform support from Jira.) + +  + +2. Complete the form + + Select GitHub Enterprise for the `Host` field. + + For the `Team or User Account` field, enter the relative path of a top-level GitLab group that you have access to, + or the relative path of your personal namespace. + +  + + For the `Host URL` field, enter `https://<your-gitlab-instance-domain>/`, + replacing `<your-gitlab-instance-domain>` appropriately. So for example, if you are using GitLab.com, + this would be `https://gitlab.com/`. + + NOTE: **Note**: + If using a GitLab version earlier than 11.3 the `Host URL` value should be `https://<your-gitlab-instance-domain>/-/jira` + + For the `Client ID` field, use the `Application ID` value from the previous section. + + For the `Client Secret` field, use the `Secret` value from the previous section. + + Ensure that the rest of the checkboxes are checked. + +3. Click `Add` to complete and create the integration. + + Jira takes up to a few minutes to know about (import behind the scenes) all the commits and branches + for all the projects in the GitLab group you specified in the previous step. These are refreshed + every 60 minutes. + + > **Note:** + > In the future, we plan on implementating real-time integration. If you need + > to refresh the data manually, you can do this from the `Applications -> DVCS + > accounts` screen where you initially set up the integration: + > + >  + +To connect additional GitLab projects from other GitLab top-level groups (or personal namespaces), repeat the above +steps with additional Jira DVCS accounts. + +You may now refer any Jira issue by its ID in branch names, commit messages and merge request names on GitLab's side, +and you will be able to see the linked `branches`, `commits`, and `merge requests` when entering a Jira issue +(inside the Jira issue, merge requests will be called "pull requests"). + + + +Click the links to see your GitLab repository data. + + + + + +## Limitations + +- This integration is currently not supported on GitLab instances under a [relative url][relative-url] (e.g. `http://example.com/gitlab`). + +## Changelog + +### 11.10 + +- [Instance admins can now setup integration for all namespaces](https://gitlab.com/gitlab-org/gitlab-ee/issues/8902) + +### 11.1 + +- [Support GitLab subgroups in Jira development panel](https://gitlab.com/gitlab-org/gitlab-ee/issues/3561) + +[existing-jira]: ../user/project/integrations/jira.md +[jira-development-panel]: https://confluence.atlassian.com/adminjiraserver070/integrating-with-development-tools-776637096.html#Integratingwithdevelopmenttools-Developmentpanelonissues +[eep]: https://about.gitlab.com/pricing/ +[ee-2381]: https://gitlab.com/gitlab-org/gitlab-ee/issues/2381 +[relative-url]: https://docs.gitlab.com/omnibus/settings/configuration.html#configuring-a-relative-url-for-gitlab diff --git a/doc/integration/kerberos.md b/doc/integration/kerberos.md new file mode 100644 index 00000000000..44117755b83 --- /dev/null +++ b/doc/integration/kerberos.md @@ -0,0 +1,315 @@ +# Kerberos integration **[STARTER ONLY]** + +GitLab can integrate with [Kerberos][kerb] as an authentication mechanism. + +## Overview + +[Kerberos][kerb] is a secure method for authenticating a request for a service in a +computer network. Kerberos was developed in the Athena Project at the +[Massachusetts Institute of Technology (MIT)][mit]. The name is taken from Greek +mythology; Kerberos was a three-headed dog who guarded the gates of Hades. + +## Use-cases + +- GitLab can be configured to allow your users to sign with their Kerberos credentials. +- You can use Kerberos to [prevent][why-kerb] anyone from intercepting or eavesdropping on the transmitted password. + +## Configuration + +For GitLab to offer Kerberos token-based authentication, perform the +following prerequisites. You still need to configure your system for +Kerberos usage, such as specifying realms. GitLab will make use of the +system's Kerberos settings. + +### GitLab keytab + +1. Create a Kerberos Service Principal for the HTTP service on your GitLab server. + If your GitLab server is `gitlab.example.com` and your Kerberos realm + `EXAMPLE.COM`, create a Service Principal `HTTP/gitlab.example.com@EXAMPLE.COM` + in your Kerberos database. +1. Create a keytab on the GitLab server for the above Service Principal, e.g. + `/etc/http.keytab`. + +The keytab is a sensitive file and must be readable by the GitLab user. Set +ownership and protect the file appropriately: + +``` +sudo chown git /etc/http.keytab +sudo chmod 0600 /etc/http.keytab +``` + +### Configure GitLab + +**Installations from source** + +>**Note:** +For source installations, make sure the `kerberos` gem group +[has been installed](../install/installation.md#install-gems). + +1. Edit the kerberos section of [gitlab.yml] to enable Kerberos ticket-based + authentication. In most cases, you only need to enable Kerberos and specify + the location of the keytab: + + ```yaml + omniauth: + enabled: true + allow_single_sign_on: ['kerberos'] + + kerberos: + # Allow the HTTP Negotiate authentication method for Git clients + enabled: true + + # Kerberos 5 keytab file. The keytab file must be readable by the GitLab user, + # and should be different from other keytabs in the system. + # (default: use default keytab from Krb5 config) + keytab: /etc/http.keytab + ``` + +1. [Restart GitLab] for the changes to take effect. + +--- + +**Omnibus package installations** + +1. Edit `/etc/gitlab/gitlab.rb`: + + ```ruby + gitlab_rails['omniauth_enabled'] = true + gitlab_rails['omniauth_allow_single_sign_on'] = ['kerberos'] + + gitlab_rails['kerberos_enabled'] = true + gitlab_rails['kerberos_keytab'] = "/etc/http.keytab" + ``` + +1. [Reconfigure GitLab] for the changes to take effect. + +--- + +GitLab will now offer the `negotiate` authentication method for signing in and +HTTP Git access, enabling Git clients that support this authentication protocol +to authenticate with Kerberos tokens. + +## Creating and linking Kerberos accounts + +The Administrative user can navigate to **Admin > Users > Example User > Identities** +and attach a Kerberos account. Existing GitLab users can go to **Profile > Account** +and attach a Kerberos account. If you want to allow users without a GitLab +account to login, you should enable the option `allow_single_sign_on` as +described in the [Configure GitLab](#configure-gitlab) section. Then, the first +time a user signs in with Kerberos credentials, GitLab will create a new GitLab +user associated with the email, which is built from the Kerberos username and +realm. User accounts will be created automatically when authentication was +successful. + +## Linking Kerberos and LDAP accounts together + +If your users log in with Kerberos, but you also have [LDAP integration](../administration/auth/ldap.md) +enabled, then your users will be automatically linked to their LDAP accounts on +first login. For this to work, some prerequisites must be met: + +The Kerberos username must match the LDAP user's UID. You can choose which LDAP +attribute is used as the UID in GitLab's [LDAP configuration](../administration/auth/ldap.md#configuration) +but for Active Directory, this should be `sAMAccountName`. + +The Kerberos realm must match the domain part of the LDAP user's Distinguished +Name. For instance, if the Kerberos realm is `AD.EXAMPLE.COM`, then the LDAP +user's Distinguished Name should end in `dc=ad,dc=example,dc=com`. + +Taken together, these rules mean that linking will only work if your users' +Kerberos usernames are of the form `foo@AD.EXAMPLE.COM` and their +LDAP Distinguished Names look like `sAMAccountName=foo,dc=ad,dc=example,dc=com`. + +## HTTP Git access + +A linked Kerberos account enables you to `git pull` and `git push` using your +Kerberos account, as well as your standard GitLab credentials. + +GitLab users with a linked Kerberos account can also `git pull` and `git push` +using Kerberos tokens, i.e., without having to send their password with each +operation. + +### HTTP Git access with Kerberos token (passwordless authentication) + +#### Support for Git before 2.4 + +Until Git version 2.4, the `git` command uses only the `negotiate` authentication +method if the HTTP server offers it, even if this method fails (such as when +the client does not have a Kerberos token). It is thus not possible to fall back +to username/password (also known as `basic`) authentication if Kerberos +authentication fails. + +For GitLab users to be able to use either `basic` or `negotiate` authentication +with older Git versions, it is possible to offer Kerberos ticket-based +authentication on a different port (e.g. 8443) while the standard port will +keep offering only `basic` authentication. + +**For source installations with HTTPS** + +1. Edit the NGINX configuration file for GitLab + (e.g., `/etc/nginx/sites-available/gitlab-ssl`) and configure NGINX to + listen to port `8443` in addition to the standard HTTPS port: + + ```conf + server { + listen 0.0.0.0:443 ssl; + listen [::]:443 ipv6only=on ssl default_server; + listen 0.0.0.0:8443 ssl; + listen [::]:8443 ipv6only=on ssl; + ``` + +1. Update the Kerberos section of [gitlab.yml]: + + ```yaml + kerberos: + # Dedicated port: Git before 2.4 does not fall back to Basic authentication if Negotiate fails. + # To support both Basic and Negotiate methods with older versions of Git, configure + # nginx to proxy GitLab on an extra port (e.g. 8443) and uncomment the following lines + # to dedicate this port to Kerberos authentication. (default: false) + use_dedicated_port: true + port: 8443 + https: true + ``` + +1. [Restart GitLab] and NGINX for the changes to take effect. + +--- + +**For Omnibus package installations** + +1. Edit `/etc/gitlab/gitlab.rb`: + + ```ruby + gitlab_rails['kerberos_use_dedicated_port'] = true + gitlab_rails['kerberos_port'] = 8443 + gitlab_rails['kerberos_https'] = true + ``` + +1. [Reconfigure GitLab] for the changes to take effect. + +--- + +After this change, all Git remote URLs will have to be updated to +`https://gitlab.example.com:8443/mygroup/myproject.git` in order to use +Kerberos ticket-based authentication. + +## Upgrading from password-based to ticket-based Kerberos sign-ins + +Prior to GitLab 8.10 Enterprise Edition, users had to submit their +Kerberos username and password to GitLab when signing in. We will +remove support for password-based Kerberos sign-ins in a future +release, so we recommend that you upgrade to ticket-based sign-ins. + +Depending on your existing GitLab configuration, the 'Sign in with: +Kerberos Spnego' button may already be visible on your GitLab sign-in +page. If not, then add the settings [described above](#configuration). + +Once you have verified that the 'Kerberos Spnego' button works +without entering any passwords, you can proceed to disable +password-based Kerberos sign-ins. To do this you need only need to +remove the OmniAuth provider named `kerberos` from your `gitlab.yml` / +`gitlab.rb` file. + +**For installations from source** + +1. Edit [gitlab.yml] and remove the `- { name: 'kerberos' }` line under omniauth + providers: + + ```yaml + omniauth: + # ... + providers: + - { name: 'kerberos' } # <-- remove this line + ``` + +1. [Restart GitLab] for the changes to take effect. + +--- + +**For Omnibus installations** + +1. Edit `/etc/gitlab/gitlab.rb` and remove the `{ "name" => "kerberos" }` line + under `gitlab_rails['omniauth_providers']`: + + ```ruby + gitlab_rails['omniauth_providers'] = [ + { "name" => "kerberos" } # <-- remove this entry + ] + ``` + +1. [Reconfigure GitLab] for the changes to take effect. + +## Support for Active Directory Kerberos environments + +When using Kerberos ticket-based authentication in an Active Directory domain, +it may be necessary to increase the maximum header size allowed by NGINX, +as extensions to the Kerberos protocol may result in HTTP authentication headers +larger than the default size of 8kB. Configure `large_client_header_buffers` +to a larger value in [the NGINX configuration][nginx]. + +## Troubleshooting + +### Unsupported GSSAPI mechanism + +With Kerberos SPNEGO authentication, the browser is expected to send a list of +mechanisms it supports to GitLab. If it doesn't support any of the mechanisms +GitLab supports, authentication will fail with a message like this in the log: + +``` +OmniauthKerberosSpnegoController: failed to process Negotiate/Kerberos authentication: gss_accept_sec_context did not return GSS_S_COMPLETE: An unsupported mechanism was requested Unknown error +``` + +This is usually seen when the browser is unable to contact the Kerberos server +directly. It will fall back to an unsupported mechanism known as +[`IAKERB`](https://k5wiki.kerberos.org/wiki/Projects/IAKERB), which tries to use +the GitLab server as an intermediary to the Kerberos server. + +If you're experiencing this error, ensure there is connectivity between the +client machine and the Kerberos server - this is a prerequisite! Traffic may be +blocked by a firewall, or the DNS records may be incorrect. + +Another failure mode occurs when the forward and reverse DNS records for the +GitLab server do not match. Often, Windows clients will work in this case, while +Linux clients will fail. They use reverse DNS while detecting the Kerberos +realm. If they get the wrong realm, then ordinary Kerberos mechanisms will fail, +so the client will fall back to attempting to negotiate `IAKERB`, leading to the +above error message. + +To fix this, ensure that the forward and reverse DNS for your GitLab server +match. So for instance, if you acces GitLab as `gitlab.example.com`, resolving +to IP address `1.2.3.4`, then `4.3.2.1.in-addr.arpa` must be a PTR record for +`gitlab.example.com`. + +Finally, it's possible that the browser or client machine lack Kerberos support +completely. Ensure that the Kerberos libraries are installed and that you can +authenticate to other Kerberos services. + +### HTTP Basic: Access denied when cloning + +```sh +remote: HTTP Basic: Access denied +fatal: Authentication failed for '<KRB5 path>' +``` + +If you are using Git v2.11 or newer and see the above error when cloning, you can +set the `http.emptyAuth` Git option to `true` to fix this: + +``` +git config --global http.emptyAuth true +``` + +See also: [Git v2.11 release notes](https://github.com/git/git/blob/master/Documentation/RelNotes/2.11.0.txt#L482-L486) + +## Helpful links + +- <https://help.ubuntu.com/community/Kerberos> +- <http://blog.manula.org/2012/04/setting-up-kerberos-server-with-debian.html> +- <http://www.roguelynn.com/words/explain-like-im-5-kerberos/> + +[gitlab.yml]: https://gitlab.com/gitlab-org/gitlab-ce/blob/master/config/gitlab.yml.example +[restart gitlab]: ../administration/restart_gitlab.md#installations-from-source +[reconfigure gitlab]: ../administration/restart_gitlab.md#omnibus-gitlab-reconfigure +[nginx]: http://nginx.org/en/docs/http/ngx_http_core_module.html#large_client_header_buffers +[kerb]: https://web.mit.edu/kerberos/ +[mit]: http://web.mit.edu/ +[why-kerb]: http://web.mit.edu/sipb/doc/working/guide/guide/node20.html +[ee]: https://about.gitlab.com/pricing/ diff --git a/doc/integration/ldap.md b/doc/integration/ldap.md index 242890af981..76c124d2ce9 100644 --- a/doc/integration/ldap.md +++ b/doc/integration/ldap.md @@ -1 +1,5 @@ +--- +redirect_to: '../administration/auth/ldap.md' +--- + This document was moved to [`administration/auth/ldap`](../administration/auth/ldap.md). diff --git a/doc/integration/omniauth.md b/doc/integration/omniauth.md index 4db986197f3..a13e9f73f48 100644 --- a/doc/integration/omniauth.md +++ b/doc/integration/omniauth.md @@ -33,6 +33,9 @@ contains some settings that are common for all providers. - [Authentiq](../administration/auth/authentiq.md) - [OAuth2Generic](oauth2_generic.md) - [JWT](../administration/auth/jwt.md) +- [OpenID Connect](../administration/auth/oidc.md) +- [UltraAuth](ultra_auth.md) +- [SalesForce](salesforce.md) ## Initial OmniAuth Configuration @@ -199,7 +202,7 @@ from the Omniauth provider's documentation. sudo -u git -H bundle install --without development test mysql --path vendor/bundle --no-deployment - > These are the same commands you used in the [Install Gems section](#install-gems) with `--path vendor/bundle --no-deployment` instead of `--deployment`. + > These are the same commands you used during initial installation in the [Install Gems section](../install/installation.md#install-gems) with `--path vendor/bundle --no-deployment` instead of `--deployment`. - Start GitLab: @@ -256,7 +259,7 @@ gitlab_rails['omniauth_enabled'] = false You can enable profile syncing from selected OmniAuth providers and for all or for specific user information. -When authenticating using LDAP, the user's email is always synced. +When authenticating using LDAP, the user's name and email are always synced. ```ruby gitlab_rails['sync_profile_from_provider'] = ['twitter', 'google_oauth2'] diff --git a/doc/integration/salesforce.md b/doc/integration/salesforce.md new file mode 100644 index 00000000000..1ef43cfcece --- /dev/null +++ b/doc/integration/salesforce.md @@ -0,0 +1,79 @@ +# Salesforce OmniAuth Provider + +You can integrate your GitLab instance with [Salesforce](https://www.salesforce.com/) to enable users to log in to your GitLab instance with their Salesforce account. + +## Create a Salesforce Connected App + +To enable Salesforce OmniAuth provider, you must use Salesforce's credentials for your GitLab instance. +To get the credentials (a pair of Client ID and Client Secret), you must [create a Connected App](https://help.salesforce.com/articleView?id=connected_app_create.htm&type=5) on Salesforce. + +1. Sign in to [Salesforce](https://login.salesforce.com/). + +1. In Setup, enter `App Manager` in the Quick Find box, click **App Manager**, then click **New Connected App**. + +1. Fill in the application details into the following fields: + - **Connected App Name** and **API Name**: Set to any value but consider something like `<Organization>'s GitLab`, `<Your Name>'s GitLab`, or something else that is descriptive. + - **Contact Email**: Enter the contact email for Salesforce to use when contacting you or your support team. + - **Description**: Description for the application. + +  +1. Select **API (Enable OAuth Settings)** and click on **Enable OAuth Settings**. +1. Fill in the application details into the following fields: + - **Callback URL**: The callback URL of your GitLab installation. For example, `https://gitlab.example.com/users/auth/salesforce/callback`. + - **Selected OAuth Scopes**: Move **Access your basic information (id, profile, email, address, phone)** and **Allow access to your unique identifier (openid)** to the right column. + +  +1. Click **Save**. + +1. On your GitLab server, open the configuration file. + + For omnibus package: + + ```sh + sudo editor /etc/gitlab/gitlab.rb + ``` + + For installations from source: + + ```sh + cd /home/git/gitlab + sudo -u git -H editor config/gitlab.yml + ``` + +1. See [Initial OmniAuth Configuration](omniauth.md#initial-omniauth-configuration) for initial settings. + +1. Add the provider configuration: + + For omnibus package: + + ```ruby + gitlab_rails['omniauth_providers'] = [ + { + "name" => "salesforce", + "app_id" => "SALESFORCE_CLIENT_ID", + "app_secret" => "SALESFORCE_CLIENT_SECRET" + } + ] + ``` + + For installation from source: + + ``` + - { name: 'salesforce', + app_id: 'SALESFORCE_CLIENT_ID', + app_secret: 'SALESFORCE_CLIENT_SECRET' + } + ``` +1. Change `SALESFORCE_CLIENT_ID` to the Consumer Key from the Salesforce connected application page. +1. Change `SALESFORCE_CLIENT_SECRET` to the Consumer Secret from the Salesforce connected application page. +  + +1. Save the configuration file. +1. [Reconfigure GitLab]( ../administration/restart_gitlab.md#omnibus-gitlab-reconfigure ) or [restart GitLab]( ../administration/restart_gitlab.md#installations-from-source ) for the changes to take effect if you installed GitLab via Omnibus or from source respectively. + +On the sign in page, there should now be a Salesforce icon below the regular sign in form. +Click the icon to begin the authentication process. Salesforce will ask the user to sign in and authorize the GitLab application. +If everything goes well, the user will be returned to GitLab and will be signed in. + +NOTE: **Note:** +GitLab requires the email address of each new user. Once the user is logged in using Salesforce, GitLab will redirect the user to the profile page where they will have to provide the email and verify the email.
\ No newline at end of file diff --git a/doc/integration/saml.md b/doc/integration/saml.md index 8ee07a7fcdc..22e07594d6f 100644 --- a/doc/integration/saml.md +++ b/doc/integration/saml.md @@ -1,5 +1,7 @@ # SAML OmniAuth Provider +> This topic is for SAML on self-managed GitLab instances. For SAML on GitLab.com, see [SAML SSO for GitLab.com Groups](../user/group/saml_sso/index.md). + NOTE: **Note:** You need to [enable OmniAuth](omniauth.md) in order to use this. @@ -10,115 +12,115 @@ Microsoft ADFS to authenticate users. First configure SAML 2.0 support in GitLab, then register the GitLab application in your SAML IdP: -1. Make sure GitLab is configured with HTTPS. - See [Using HTTPS](../install/installation.md#using-https) for instructions. - -1. On your GitLab server, open the configuration file. +1. Make sure GitLab is configured with HTTPS. + See [Using HTTPS](../install/installation.md#using-https) for instructions. + +1. On your GitLab server, open the configuration file. - For omnibus package: + For omnibus package: - ```sh - sudo editor /etc/gitlab/gitlab.rb - ``` + ```sh + sudo editor /etc/gitlab/gitlab.rb + ``` - For installations from source: + For installations from source: - ```sh - cd /home/git/gitlab + ```sh + cd /home/git/gitlab - sudo -u git -H editor config/gitlab.yml - ``` + sudo -u git -H editor config/gitlab.yml + ``` -1. To allow your users to use SAML to sign up without having to manually create - an account first, don't forget to add the following values to your configuration: +1. To allow your users to use SAML to sign up without having to manually create + an account first, don't forget to add the following values to your configuration: - For omnibus package: + For omnibus package: - ```ruby - gitlab_rails['omniauth_enabled'] = true - gitlab_rails['omniauth_allow_single_sign_on'] = ['saml'] - gitlab_rails['omniauth_block_auto_created_users'] = false - ``` + ```ruby + gitlab_rails['omniauth_enabled'] = true + gitlab_rails['omniauth_allow_single_sign_on'] = ['saml'] + gitlab_rails['omniauth_block_auto_created_users'] = false + ``` - For installations from source: + For installations from source: - ```yaml - omniauth: - enabled: true - allow_single_sign_on: ["saml"] - block_auto_created_users: false - ``` + ```yaml + omniauth: + enabled: true + allow_single_sign_on: ["saml"] + block_auto_created_users: false + ``` -1. You can also automatically link SAML users with existing GitLab users if their - email addresses match by adding the following setting: +1. You can also automatically link SAML users with existing GitLab users if their + email addresses match by adding the following setting: - For omnibus package: + For omnibus package: - ```ruby - gitlab_rails['omniauth_auto_link_saml_user'] = true - ``` + ```ruby + gitlab_rails['omniauth_auto_link_saml_user'] = true + ``` - For installations from source: + For installations from source: - ```yaml - auto_link_saml_user: true - ``` + ```yaml + auto_link_saml_user: true + ``` -1. Add the provider configuration: +1. Add the provider configuration: - For omnibus package: + For omnibus package: - ```ruby - gitlab_rails['omniauth_providers'] = [ - { - name: 'saml', - args: { - assertion_consumer_service_url: 'https://gitlab.example.com/users/auth/saml/callback', - idp_cert_fingerprint: '43:51:43:a1:b5:fc:8b:b7:0a:3a:a9:b1:0f:66:73:a8', - idp_sso_target_url: 'https://login.example.com/idp', - issuer: 'https://gitlab.example.com', - name_identifier_format: 'urn:oasis:names:tc:SAML:2.0:nameid-format:persistent' - }, - label: 'Company Login' # optional label for SAML login button, defaults to "Saml" - } - ] - ``` + ```ruby + gitlab_rails['omniauth_providers'] = [ + { + name: 'saml', + args: { + assertion_consumer_service_url: 'https://gitlab.example.com/users/auth/saml/callback', + idp_cert_fingerprint: '43:51:43:a1:b5:fc:8b:b7:0a:3a:a9:b1:0f:66:73:a8', + idp_sso_target_url: 'https://login.example.com/idp', + issuer: 'https://gitlab.example.com', + name_identifier_format: 'urn:oasis:names:tc:SAML:2.0:nameid-format:persistent' + }, + label: 'Company Login' # optional label for SAML login button, defaults to "Saml" + } + ] + ``` - For installations from source: - - ```yaml - omniauth: - providers: - - { - name: 'saml', - args: { - assertion_consumer_service_url: 'https://gitlab.example.com/users/auth/saml/callback', - idp_cert_fingerprint: '43:51:43:a1:b5:fc:8b:b7:0a:3a:a9:b1:0f:66:73:a8', - idp_sso_target_url: 'https://login.example.com/idp', - issuer: 'https://gitlab.example.com', - name_identifier_format: 'urn:oasis:names:tc:SAML:2.0:nameid-format:persistent' - }, - label: 'Company Login' # optional label for SAML login button, defaults to "Saml" - } - ``` + For installations from source: -1. Change the value for `assertion_consumer_service_url` to match the HTTPS endpoint - of GitLab (append `users/auth/saml/callback` to the HTTPS URL of your GitLab - installation to generate the correct value). + ```yaml + omniauth: + providers: + - { + name: 'saml', + args: { + assertion_consumer_service_url: 'https://gitlab.example.com/users/auth/saml/callback', + idp_cert_fingerprint: '43:51:43:a1:b5:fc:8b:b7:0a:3a:a9:b1:0f:66:73:a8', + idp_sso_target_url: 'https://login.example.com/idp', + issuer: 'https://gitlab.example.com', + name_identifier_format: 'urn:oasis:names:tc:SAML:2.0:nameid-format:persistent' + }, + label: 'Company Login' # optional label for SAML login button, defaults to "Saml" + } + ``` -1. Change the values of `idp_cert_fingerprint`, `idp_sso_target_url`, - `name_identifier_format` to match your IdP. If a fingerprint is used it must - be a SHA1 fingerprint; check - [the omniauth-saml documentation](https://github.com/omniauth/omniauth-saml) - for more details on these options. +1. Change the value for `assertion_consumer_service_url` to match the HTTPS endpoint + of GitLab (append `users/auth/saml/callback` to the HTTPS URL of your GitLab + installation to generate the correct value). -1. Change the value of `issuer` to a unique name, which will identify the application - to the IdP. +1. Change the values of `idp_cert_fingerprint`, `idp_sso_target_url`, + `name_identifier_format` to match your IdP. If a fingerprint is used it must + be a SHA1 fingerprint; check + [the omniauth-saml documentation](https://github.com/omniauth/omniauth-saml) + for more details on these options. -1. For the changes to take effect, you must [reconfigure][] GitLab if you installed via Omnibus or [restart GitLab][] if you installed from source. +1. Change the value of `issuer` to a unique name, which will identify the application + to the IdP. -1. Register the GitLab SP in your SAML 2.0 IdP, using the application name specified - in `issuer`. +1. For the changes to take effect, you must [reconfigure][] GitLab if you installed via Omnibus or [restart GitLab][] if you installed from source. + +1. Register the GitLab SP in your SAML 2.0 IdP, using the application name specified + in `issuer`. To ease configuration, most IdP accept a metadata URL for the application to provide configuration information to the IdP. To build the metadata URL for GitLab, append @@ -185,6 +187,78 @@ tell GitLab which groups are external via the `external_groups:` element: } } ``` +## Required groups + +>**Note:** +This setting is only available on GitLab 10.2 EE and above. + +This setting works like `External Groups` setting. Just like there, your IdP has to +pass Group Information to GitLab, you have to tell GitLab where to look or the +groups SAML response, and which group membership should be requisite for logging in. +When `required_groups` is not set or it is empty, anyone with proper authentication +will be able to use the service. + +Example: + +```yaml +{ name: 'saml', + label: 'Our SAML Provider', + groups_attribute: 'Groups', + required_groups: ['Developers', 'Managers', 'Admins'], + args: { + assertion_consumer_service_url: 'https://gitlab.example.com/users/auth/saml/callback', + idp_cert_fingerprint: '43:51:43:a1:b5:fc:8b:b7:0a:3a:a9:b1:0f:66:73:a8', + idp_sso_target_url: 'https://login.example.com/idp', + issuer: 'https://gitlab.example.com', + name_identifier_format: 'urn:oasis:names:tc:SAML:2.0:nameid-format:transient' + } } +``` + +## Admin Groups + +>**Note:** +This setting is only available on GitLab 8.8 EE and above. + +This setting works very similarly to the `External Groups` setting. The requirements +are the same, your IdP needs to pass Group information to GitLab, you need to tell +GitLab where to look for the groups in the SAML response, and which group should be +considered `admin groups`. + +```yaml +{ name: 'saml', + label: 'Our SAML Provider', + groups_attribute: 'Groups', + admin_groups: ['Managers', 'Admins'], + args: { + assertion_consumer_service_url: 'https://gitlab.example.com/users/auth/saml/callback', + idp_cert_fingerprint: '43:51:43:a1:b5:fc:8b:b7:0a:3a:a9:b1:0f:66:73:a8', + idp_sso_target_url: 'https://login.example.com/idp', + issuer: 'https://gitlab.example.com', + name_identifier_format: 'urn:oasis:names:tc:SAML:2.0:nameid-format:transient' + } } +``` + +## Auditor Groups + +>**Note:** +This setting is only available on GitLab 11.4 EE and above. + +This setting also follows the requirements documented for the `External Groups` setting. GitLab uses the Group information provided by your IdP to determine if a user should be assigned the `auditor` role. + +```yaml +{ name: 'saml', + label: 'Our SAML Provider', + groups_attribute: 'Groups', + auditor_groups: ['Auditors', 'Security'], + args: { + assertion_consumer_service_url: 'https://gitlab.example.com/users/auth/saml/callback', + idp_cert_fingerprint: '43:51:43:a1:b5:fc:8b:b7:0a:3a:a9:b1:0f:66:73:a8', + idp_sso_target_url: 'https://login.example.com/idp', + issuer: 'https://gitlab.example.com', + name_identifier_format: 'urn:oasis:names:tc:SAML:2.0:nameid-format:transient' + } } +``` + ## Bypass two factor authentication If you want some SAML authentication methods to count as 2FA on a per session basis, you can register them in the @@ -194,28 +268,28 @@ If you want some SAML authentication methods to count as 2FA on a per session ba 1. Edit `/etc/gitlab/gitlab.rb`: - ```ruby - gitlab_rails['omniauth_providers'] = [ - { - name: 'saml', - args: { - assertion_consumer_service_url: 'https://gitlab.example.com/users/auth/saml/callback', - idp_cert_fingerprint: '43:51:43:a1:b5:fc:8b:b7:0a:3a:a9:b1:0f:66:73:a8', - idp_sso_target_url: 'https://login.example.com/idp', - issuer: 'https://gitlab.example.com', - name_identifier_format: 'urn:oasis:names:tc:SAML:2.0:nameid-format:persistent', - upstream_two_factor_authn_contexts: - %w( - urn:oasis:names:tc:SAML:2.0:ac:classes:CertificateProtectedTransport - urn:oasis:names:tc:SAML:2.0:ac:classes:SecondFactorOTPSMS - urn:oasis:names:tc:SAML:2.0:ac:classes:SecondFactorIGTOKEN - ) - - }, - label: 'Company Login' # optional label for SAML login button, defaults to "Saml" - } - ] - ``` + ```ruby + gitlab_rails['omniauth_providers'] = [ + { + name: 'saml', + args: { + assertion_consumer_service_url: 'https://gitlab.example.com/users/auth/saml/callback', + idp_cert_fingerprint: '43:51:43:a1:b5:fc:8b:b7:0a:3a:a9:b1:0f:66:73:a8', + idp_sso_target_url: 'https://login.example.com/idp', + issuer: 'https://gitlab.example.com', + name_identifier_format: 'urn:oasis:names:tc:SAML:2.0:nameid-format:persistent', + upstream_two_factor_authn_contexts: + %w( + urn:oasis:names:tc:SAML:2.0:ac:classes:CertificateProtectedTransport + urn:oasis:names:tc:SAML:2.0:ac:classes:SecondFactorOTPSMS + urn:oasis:names:tc:SAML:2.0:ac:classes:SecondFactorIGTOKEN + ) + + }, + label: 'Company Login' # optional label for SAML login button, defaults to "Saml" + } + ] + ``` 1. Save the file and [reconfigure][] GitLab for the changes to take effect. @@ -225,27 +299,27 @@ If you want some SAML authentication methods to count as 2FA on a per session ba 1. Edit `config/gitlab.yml`: - ```yaml - omniauth: - providers: - - { - name: 'saml', - args: { - assertion_consumer_service_url: 'https://gitlab.example.com/users/auth/saml/callback', - idp_cert_fingerprint: '43:51:43:a1:b5:fc:8b:b7:0a:3a:a9:b1:0f:66:73:a8', - idp_sso_target_url: 'https://login.example.com/idp', - issuer: 'https://gitlab.example.com', - name_identifier_format: 'urn:oasis:names:tc:SAML:2.0:nameid-format:persistent', - upstream_two_factor_authn_contexts: - [ - 'urn:oasis:names:tc:SAML:2.0:ac:classes:CertificateProtectedTransport', - 'urn:oasis:names:tc:SAML:2.0:ac:classes:SecondFactorOTPSMS', - 'urn:oasis:names:tc:SAML:2.0:ac:classes:SecondFactorIGTOKEN' - ] - }, - label: 'Company Login' # optional label for SAML login button, defaults to "Saml" - } - ``` + ```yaml + omniauth: + providers: + - { + name: 'saml', + args: { + assertion_consumer_service_url: 'https://gitlab.example.com/users/auth/saml/callback', + idp_cert_fingerprint: '43:51:43:a1:b5:fc:8b:b7:0a:3a:a9:b1:0f:66:73:a8', + idp_sso_target_url: 'https://login.example.com/idp', + issuer: 'https://gitlab.example.com', + name_identifier_format: 'urn:oasis:names:tc:SAML:2.0:nameid-format:persistent', + upstream_two_factor_authn_contexts: + [ + 'urn:oasis:names:tc:SAML:2.0:ac:classes:CertificateProtectedTransport', + 'urn:oasis:names:tc:SAML:2.0:ac:classes:SecondFactorOTPSMS', + 'urn:oasis:names:tc:SAML:2.0:ac:classes:SecondFactorIGTOKEN' + ] + }, + label: 'Company Login' # optional label for SAML login button, defaults to "Saml" + } + ``` 1. Save the file and [restart GitLab][] for the changes ot take effect diff --git a/doc/integration/slack.md b/doc/integration/slack.md index 8cd151fbf95..f84ab769218 100644 --- a/doc/integration/slack.md +++ b/doc/integration/slack.md @@ -1 +1,5 @@ +--- +redirect_to: '../project_services/slack.md' +--- + This document was moved to [project_services/slack.md](../project_services/slack.md). diff --git a/doc/integration/slash_commands.md b/doc/integration/slash_commands.md index cd755089be8..71ea2e25533 100644 --- a/doc/integration/slash_commands.md +++ b/doc/integration/slash_commands.md @@ -20,8 +20,8 @@ Taking the trigger term as `project-name`, the commands are: | `/project-name deploy <from> to <to>` | Deploy from the `<from>` environment to the `<to>` environment | | `/project-name run <job name> <arguments>` | Execute [ChatOps](../ci/chatops/README.md) job `<job name>` on `master` | -Note that if you are using the [GitLab Slack application](https://docs.gitlab.com/ee/user/project/integrations/gitlab_slack_application.html) for -your GitLab.com projects, you need to [add the `gitlab` keyword at the beginning of the command](https://docs.gitlab.com/ee/user/project/integrations/gitlab_slack_application.html#usage). +Note that if you are using the [GitLab Slack application](../user/project/integrations/gitlab_slack_application.md) for +your GitLab.com projects, you need to [add the `gitlab` keyword at the beginning of the command](../user/project/integrations/gitlab_slack_application.md#usage). ## Issue commands diff --git a/doc/integration/ultra_auth.md b/doc/integration/ultra_auth.md new file mode 100644 index 00000000000..139cca456aa --- /dev/null +++ b/doc/integration/ultra_auth.md @@ -0,0 +1,78 @@ +# UltraAuth OmniAuth Provider + +You can integrate your GitLab instance with [UltraAuth](https://ultraauth.com) to enable users to perform secure biometric authentication to your GitLab instance with your UltraAuth account. Users have to perform the biometric authentication using their mobile device with fingerprint sensor. + +## Create UltraAuth Application + +To enable UltraAuth OmniAuth provider, you must use UltraAuth's credentials for your GitLab instance. +To get the credentials (a pair of Client ID and Client Secret), you must register an application on UltraAuth. + +1. Sign in to [UltraAuth](https://ultraauth.com). +1. Navigate to [Create an App](https://ultraauth.com/select-strategy) and click on "Ruby on Rails". +1. Scroll down the page that is displayed to locate the **Client ID** and **Client Secret**. + Keep this page open as you continue configuration. +  +1. Click on "Edit Callback URL" link. +  +1. The callback URL will be `http(s)://<your_domain>/users/auth/ultraauth/callback` +  +1. Select **Register application**. +1. On your GitLab server, open the configuration file. + + For omnibus package: + + ```sh + sudo editor /etc/gitlab/gitlab.rb + ``` + + For installations from source: + + ```sh + cd /home/git/gitlab + sudo -u git -H editor config/gitlab.yml + ``` +1. See [Initial OmniAuth Configuration](omniauth.md#initial-omniauth-configuration) for initial settings. +1. Add the provider configuration: + + For omnibus package: + + ```ruby + gitlab_rails['omniauth_providers'] = [ + { + "name" => "ultraauth", + "app_id" => "OPENID_CLIENT_ID", + "app_secret" => "OPENID_CLIENT_SECRET", + "args" => { + "client_options" => { + "redirect_uri" => "https://example.com/users/auth/ultraauth/callback" + } + } + } + ] + ``` + + For installation from source: + + ``` + - { name: 'ultraauth', + app_id: 'OPENID_CLIENT_ID', + app_secret: 'OPENID_CLIENT_SECRET', + args: { + client_options: { + redirect_uri: 'https://example.com/users/auth/ultraauth/callback' + } + } + } + ``` + __Replace `https://example.com/users/auth/ultraauth/callback` with your application's Callback URL.__ +1. Change `OPENID_CLIENT_ID` to the Client ID from the UltraAuth application page. +1. Change `OPENID_CLIENT_SECRET` to the Client Secret from the UltraAuth application page. +1. Save the configuration file. +1. [Reconfigure GitLab]( ../administration/restart_gitlab.md#omnibus-gitlab-reconfigure ) or [restart GitLab]( ../administration/restart_gitlab.md#installations-from-source ) for the changes to take effect if you + installed GitLab via Omnibus or from source respectively. + +On the sign in page, there should now be a UltraAuth icon below the regular sign in form. +Click the icon to begin the authentication process. UltraAuth will ask the user to sign in and authorize the GitLab application. +If everything goes well, the user will be returned to GitLab and will be signed in. + +**Note:** GitLab requires the email address of each new user. Once the user is logged in using UltraAuth, GitLab will redirect the user to the profile page where they will have to provide the email and verify the email. diff --git a/doc/legal/corporate_contributor_license_agreement.md b/doc/legal/corporate_contributor_license_agreement.md index e5fc7a3c85f..7f08188bd65 100644 --- a/doc/legal/corporate_contributor_license_agreement.md +++ b/doc/legal/corporate_contributor_license_agreement.md @@ -1,3 +1,29 @@ ---- -redirect_to: 'README.md' ---- +# Corporate contributor license agreement + +You accept and agree to the following terms and conditions for Your present and future Contributions submitted to GitLab B.V.. Except for the license granted herein to GitLab B.V. and recipients of software distributed by GitLab B.V., You reserve all right, title, and interest in and to Your Contributions. + +1. Definitions. + + "You" (or "Your") shall mean the copyright owner or legal entity authorized by the copyright owner that is making this Agreement with GitLab B.V.. For legal entities, the entity making a Contribution and all other entities that control, are controlled by, or are under common control with that entity are considered to be a single Contributor. For the purposes of this definition, "control" means (i) the power, direct or indirect, to cause the direction or management of such entity, whether by contract or otherwise, or (ii) ownership of fifty percent (50%) or more of the outstanding shares, or (iii) beneficial ownership of such entity. + + "Contribution" shall mean the code, documentation or other original works of authorship, including any modifications or additions to an existing work, that is submitted by You to GitLab B.V. for inclusion in, or documentation of, any of the products owned or managed by GitLab B.V. (the "Work"). For the purposes of this definition, "submitted" means any form of electronic, verbal, or written communication sent to GitLab B.V. or its representatives, including but not limited to communication on electronic mailing lists, source code control systems, and issue tracking systems that are managed by, or on behalf of, GitLab B.V. for the purpose of discussing and improving the Work, but excluding communication that is conspicuously marked or otherwise designated in writing by You as "Not a Contribution." + +2. Grant of Copyright License. + +Subject to the terms and conditions of this Agreement, You hereby grant to GitLab B.V. and to recipients of software distributed by GitLab B.V. a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable copyright license to reproduce, prepare derivative works of, publicly display, publicly perform, sublicense, and distribute Your Contributions and such derivative works. + +3. Grant of Patent License. + +Subject to the terms and conditions of this Agreement, You hereby grant to GitLab B.V. and to recipients of software distributed by GitLab B.V. a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable (except as stated in this section) patent license to make, have made, use, offer to sell, sell, import, and otherwise transfer the Work, where such license applies only to those patent claims licensable by You that are necessarily infringed by Your Contribution(s) alone or by combination of Your Contribution(s) with the Work to which such Contribution(s) was submitted. If any entity institutes patent litigation against You or any other entity (including a cross-claim or counterclaim in a lawsuit) alleging that your Contribution, or the Work to which you have contributed, constitutes direct or contributory patent infringement, then any patent licenses granted to that entity under this Agreement for that Contribution or Work shall terminate as of the date such litigation is filed. + +4. You represent that You are legally entitled to grant the above license. You represent further that each of Your employees is authorized to submit Contributions on Your behalf, but excluding employees that are designated in writing by You as "Not authorized to submit Contributions on behalf of [name of Your corporation here]." Such designations of exclusion for unauthorized employees are to be submitted via email to legal@gitlab.com. + +5. You represent that each of Your Contributions is Your original creation (see section 7 for submissions on behalf of others). + +6. You are not expected to provide support for Your Contributions, except to the extent You desire to provide support. You may provide support for free, for a fee, or not at all. Unless required by applicable law or agreed to in writing, You provide Your Contributions on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied, including, without limitation, any warranties or conditions of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A PARTICULAR PURPOSE. + +7. Should You wish to submit work that is not Your original creation, You may submit it to GitLab B.V. separately from any Contribution, identifying the complete details of its source and of any license or other restriction (including, but not limited to, related patents, trademarks, and license agreements) of which you are personally aware, and conspicuously marking the work as "Submitted on behalf of a third-party: [named here]". + +8. It is Your responsibility to notify GitLab.com when any change is required to the list of designated employees excluded from submitting Contributions on Your behalf per Section 4. Such notification should be sent via email to legal@gitlab.com. + +This text is licensed under the [Creative Commons Attribution 3.0 License](https://creativecommons.org/licenses/by/3.0/) and the original source is the Google Open Source Programs Office. diff --git a/doc/legal/individual_contributor_license_agreement.md b/doc/legal/individual_contributor_license_agreement.md index e5fc7a3c85f..59803aea080 100644 --- a/doc/legal/individual_contributor_license_agreement.md +++ b/doc/legal/individual_contributor_license_agreement.md @@ -1,3 +1,25 @@ ---- -redirect_to: 'README.md' ---- +# Individual contributor license agreement + +You accept and agree to the following terms and conditions for Your present and future Contributions submitted to GitLab B.V.. Except for the license granted herein to GitLab B.V. and recipients of software distributed by GitLab B.V., You reserve all right, title, and interest in and to Your Contributions. + +1. Definitions. + + "You" (or "Your") shall mean the copyright owner or legal entity authorized by the copyright owner that is making this Agreement with GitLab B.V.. For legal entities, the entity making a Contribution and all other entities that control, are controlled by, or are under common control with that entity are considered to be a single Contributor. For the purposes of this definition, "control" means (i) the power, direct or indirect, to cause the direction or management of such entity, whether by contract or otherwise, or (ii) ownership of fifty percent (50%) or more of the outstanding shares, or (iii) beneficial ownership of such entity. + + "Contribution" shall mean any original work of authorship, including any modifications or additions to an existing work, that is intentionally submitted by You to GitLab B.V. for inclusion in, or documentation of, any of the products owned or managed by GitLab B.V. (the "Work"). For the purposes of this definition, "submitted" means any form of electronic, verbal, or written communication sent to GitLab B.V. or its representatives, including but not limited to communication on electronic mailing lists, source code control systems, and issue tracking systems that are managed by, or on behalf of, GitLab B.V. for the purpose of discussing and improving the Work, but excluding communication that is conspicuously marked or otherwise designated in writing by You as "Not a Contribution." + +2. Grant of Copyright License. Subject to the terms and conditions of this Agreement, You hereby grant to GitLab B.V. and to recipients of software distributed by GitLab B.V. a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable copyright license to reproduce, prepare derivative works of, publicly display, publicly perform, sublicense, and distribute Your Contributions and such derivative works. + +3. Grant of Patent License. Subject to the terms and conditions of this Agreement, You hereby grant to GitLab B.V. and to recipients of software distributed by GitLab B.V. a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable (except as stated in this section) patent license to make, have made, use, offer to sell, sell, import, and otherwise transfer the Work, where such license applies only to those patent claims licensable by You that are necessarily infringed by Your Contribution(s) alone or by combination of Your Contribution(s) with the Work to which such Contribution(s) was submitted. If any entity institutes patent litigation against You or any other entity (including a cross-claim or counterclaim in a lawsuit) alleging that your Contribution, or the Work to which you have contributed, constitutes direct or contributory patent infringement, then any patent licenses granted to that entity under this Agreement for that Contribution or Work shall terminate as of the date such litigation is filed. + +4. You represent that you are legally entitled to grant the above license. If your employer(s) has rights to intellectual property that you create that includes your Contributions, you represent that you have received permission to make Contributions on behalf of that employer, that your employer has waived such rights for your Contributions to GitLab B.V., or that your employer has executed a separate Corporate CLA with GitLab B.V.. + +5. You represent that each of Your Contributions is Your original creation (see section 7 for submissions on behalf of others). You represent that Your Contribution submissions include complete details of any third-party license or other restriction (including, but not limited to, related patents and trademarks) of which you are personally aware and which are associated with any part of Your Contributions. + +6. You are not expected to provide support for Your Contributions, except to the extent You desire to provide support. You may provide support for free, for a fee, or not at all. Unless required by applicable law or agreed to in writing, You provide Your Contributions on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied, including, without limitation, any warranties or conditions of TITLE, NON- INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A PARTICULAR PURPOSE. + +7. Should You wish to submit work that is not Your original creation, You may submit it to GitLab B.V. separately from any Contribution, identifying the complete details of its source and of any license or other restriction (including, but not limited to, related patents, trademarks, and license agreements) of which you are personally aware, and conspicuously marking the work as "Submitted on behalf of a third-party: [insert_name_here]". + +8. You agree to notify GitLab B.V. of any facts or circumstances of which you become aware that would make these representations inaccurate in any respect. + +This text is licensed under the [Creative Commons Attribution 3.0 License](https://creativecommons.org/licenses/by/3.0/) and the original source is the Google Open Source Programs Office. diff --git a/doc/license/README.md b/doc/license/README.md new file mode 100644 index 00000000000..fd110a39b61 --- /dev/null +++ b/doc/license/README.md @@ -0,0 +1,5 @@ +--- +redirect_to: '../user/admin_area/license.md' +--- + +This document was moved to [another location](../user/admin_area/license.md). diff --git a/doc/logs/logs.md b/doc/logs/logs.md index a2eca62d691..0cb092c85fd 100644 --- a/doc/logs/logs.md +++ b/doc/logs/logs.md @@ -1 +1,5 @@ +--- +redirect_to: '../administration/logs.md' +--- + This document was moved to [administration/logs.md](../administration/logs.md). diff --git a/doc/markdown/markdown.md b/doc/markdown/markdown.md index 4ac81ab3ee7..29cb6ac9164 100644 --- a/doc/markdown/markdown.md +++ b/doc/markdown/markdown.md @@ -1 +1,5 @@ -This document was moved to [user/markdown.md](../user/markdown.md). +--- +redirect_to: '../user/markdown.md' +--- + +This document was moved to [another location](../user/markdown.md). diff --git a/doc/monitoring/health_check.md b/doc/monitoring/health_check.md index 6cf93c33ec2..b611fa388b4 100644 --- a/doc/monitoring/health_check.md +++ b/doc/monitoring/health_check.md @@ -1 +1,5 @@ +--- +redirect_to: '../user/admin_area/monitoring/health_check.md' +--- + This document was moved to [user/admin_area/monitoring/health_check](../user/admin_area/monitoring/health_check.md). diff --git a/doc/monitoring/performance/gitlab_configuration.md b/doc/monitoring/performance/gitlab_configuration.md index 19d46135930..233a12ebd6f 100644 --- a/doc/monitoring/performance/gitlab_configuration.md +++ b/doc/monitoring/performance/gitlab_configuration.md @@ -1 +1,5 @@ +--- +redirect_to: '../../administration/monitoring/performance/gitlab_configuration.md' +--- + This document was moved to [administration/monitoring/performance/gitlab_configuration](../../administration/monitoring/performance/gitlab_configuration.md). diff --git a/doc/monitoring/performance/grafana_configuration.md b/doc/monitoring/performance/grafana_configuration.md index 0d4be02ff5f..f4e3561a19f 100644 --- a/doc/monitoring/performance/grafana_configuration.md +++ b/doc/monitoring/performance/grafana_configuration.md @@ -1 +1,5 @@ +--- +redirect_to: '../../administration/monitoring/performance/grafana_configuration.md' +--- + This document was moved to [administration/monitoring/performance/grafana_configuration](../../administration/monitoring/performance/grafana_configuration.md). diff --git a/doc/monitoring/performance/img/grafana_dashboard_dropdown.png b/doc/monitoring/performance/img/grafana_dashboard_dropdown.png Binary files differdeleted file mode 100644 index 51eef90068d..00000000000 --- a/doc/monitoring/performance/img/grafana_dashboard_dropdown.png +++ /dev/null diff --git a/doc/monitoring/performance/img/grafana_dashboard_import.png b/doc/monitoring/performance/img/grafana_dashboard_import.png Binary files differdeleted file mode 100644 index fd639ee0eb8..00000000000 --- a/doc/monitoring/performance/img/grafana_dashboard_import.png +++ /dev/null diff --git a/doc/monitoring/performance/img/grafana_data_source_configuration.png b/doc/monitoring/performance/img/grafana_data_source_configuration.png Binary files differdeleted file mode 100644 index a98e0ed1e7d..00000000000 --- a/doc/monitoring/performance/img/grafana_data_source_configuration.png +++ /dev/null diff --git a/doc/monitoring/performance/img/grafana_data_source_empty.png b/doc/monitoring/performance/img/grafana_data_source_empty.png Binary files differdeleted file mode 100644 index 549ada8343e..00000000000 --- a/doc/monitoring/performance/img/grafana_data_source_empty.png +++ /dev/null diff --git a/doc/monitoring/performance/img/grafana_save_icon.png b/doc/monitoring/performance/img/grafana_save_icon.png Binary files differdeleted file mode 100644 index 68a071f5ae2..00000000000 --- a/doc/monitoring/performance/img/grafana_save_icon.png +++ /dev/null diff --git a/doc/monitoring/performance/img/metrics_gitlab_configuration_settings.png b/doc/monitoring/performance/img/metrics_gitlab_configuration_settings.png Binary files differdeleted file mode 100644 index b9563a00e97..00000000000 --- a/doc/monitoring/performance/img/metrics_gitlab_configuration_settings.png +++ /dev/null diff --git a/doc/monitoring/performance/influxdb_configuration.md b/doc/monitoring/performance/influxdb_configuration.md index 15fd275e916..ae5f4c7e9df 100644 --- a/doc/monitoring/performance/influxdb_configuration.md +++ b/doc/monitoring/performance/influxdb_configuration.md @@ -1 +1,5 @@ +--- +redirect_to: '../../administration/monitoring/performance/influxdb_configuration.md' +--- + This document was moved to [administration/monitoring/performance/influxdb_configuration](../../administration/monitoring/performance/influxdb_configuration.md). diff --git a/doc/monitoring/performance/influxdb_schema.md b/doc/monitoring/performance/influxdb_schema.md index e53f9701dc3..57fb74cb6cd 100644 --- a/doc/monitoring/performance/influxdb_schema.md +++ b/doc/monitoring/performance/influxdb_schema.md @@ -1 +1,5 @@ +--- +redirect_to: '../../administration/monitoring/performance/influxdb_schema.md' +--- + This document was moved to [administration/monitoring/performance/influxdb_schema](../../administration/monitoring/performance/influxdb_schema.md). diff --git a/doc/monitoring/performance/introduction.md b/doc/monitoring/performance/introduction.md index 4d6f02b6547..e23eabd5f40 100644 --- a/doc/monitoring/performance/introduction.md +++ b/doc/monitoring/performance/introduction.md @@ -1 +1,5 @@ +--- +redirect_to: '../../administration/monitoring/performance/index.md' +--- + This document was moved to [administration/monitoring/performance/introduction](../../administration/monitoring/performance/index.md). diff --git a/doc/operations/README.md b/doc/operations/README.md index d7a83948b87..319e5f48d38 100644 --- a/doc/operations/README.md +++ b/doc/operations/README.md @@ -1 +1,5 @@ +--- +redirect_to: '../administration/operations/index.md' +--- + This document was moved to [another location](../administration/operations/index.md). diff --git a/doc/operations/cleaning_up_redis_sessions.md b/doc/operations/cleaning_up_redis_sessions.md index 2a1d0a8c8eb..bde7fffd090 100644 --- a/doc/operations/cleaning_up_redis_sessions.md +++ b/doc/operations/cleaning_up_redis_sessions.md @@ -1 +1,5 @@ -This document was moved to [administration/operations/cleaning_up_redis_sessions](../administration/operations/cleaning_up_redis_sessions.md). +--- +redirect_to: '../administration/operations/cleaning_up_redis_sessions.md' +--- + +This document was moved to [another location](../administration/operations/cleaning_up_redis_sessions.md). diff --git a/doc/operations/moving_repositories.md b/doc/operations/moving_repositories.md index c54bca324a5..57d47e3e9ea 100644 --- a/doc/operations/moving_repositories.md +++ b/doc/operations/moving_repositories.md @@ -1 +1,5 @@ -This document was moved to [administration/operations/moving_repositories](../administration/operations/moving_repositories.md). +--- +redirect_to: '../administration/operations/moving_repositories.md' +--- + +This document was moved to [another location](../administration/operations/moving_repositories.md). diff --git a/doc/operations/sidekiq_memory_killer.md b/doc/operations/sidekiq_memory_killer.md index cf7c3b2e2ed..2df4a6e5648 100644 --- a/doc/operations/sidekiq_memory_killer.md +++ b/doc/operations/sidekiq_memory_killer.md @@ -1 +1,5 @@ -This document was moved to [administration/operations/sidekiq_memory_killer](../administration/operations/sidekiq_memory_killer.md). +--- +redirect_to: '../administration/operations/sidekiq_memory_killer.md' +--- + +This document was moved to [another location](../administration/operations/sidekiq_memory_killer.md). diff --git a/doc/operations/unicorn.md b/doc/operations/unicorn.md index fbc9697b755..949f4a66c9d 100644 --- a/doc/operations/unicorn.md +++ b/doc/operations/unicorn.md @@ -1 +1,5 @@ -This document was moved to [administration/operations/unicorn](../administration/operations/unicorn.md). +--- +redirect_to: '../administration/operations/unicorn.md' +--- + +This document was moved to [another location](../administration/operations/unicorn.md). diff --git a/doc/pages/README.md b/doc/pages/README.md index 7878bce3f10..c67847f1a83 100644 --- a/doc/pages/README.md +++ b/doc/pages/README.md @@ -1 +1,5 @@ -This document was moved to [pages/index.md](../user/project/pages/index.md). +--- +redirect_to: '../user/project/pages/index.md' +--- + +This document was moved to [another location](../user/project/pages/index.md). diff --git a/doc/pages/administration.md b/doc/pages/administration.md index 4eb3bb32c77..015dd54ec7f 100644 --- a/doc/pages/administration.md +++ b/doc/pages/administration.md @@ -1 +1,5 @@ -This document was moved to [administration/pages](../administration/pages/index.md). +--- +redirect_to: '../administration/pages/index.md' +--- + +This document was moved to [another location](../administration/pages/index.md). diff --git a/doc/pages/getting_started_part_one.md b/doc/pages/getting_started_part_one.md index 1d63ccb4d2f..a0feed0b477 100644 --- a/doc/pages/getting_started_part_one.md +++ b/doc/pages/getting_started_part_one.md @@ -1 +1,5 @@ +--- +redirect_to: '../user/project/pages/getting_started_part_one.md' +--- + This document was moved to [another location](../user/project/pages/getting_started_part_one.md). diff --git a/doc/pages/getting_started_part_three.md b/doc/pages/getting_started_part_three.md index 1697b5cd6b4..b65247ff7b7 100644 --- a/doc/pages/getting_started_part_three.md +++ b/doc/pages/getting_started_part_three.md @@ -1 +1,5 @@ +--- +redirect_to: '../user/project/pages/getting_started_part_three.md' +--- + This document was moved to [another location](../user/project/pages/getting_started_part_three.md). diff --git a/doc/pages/getting_started_part_two.md b/doc/pages/getting_started_part_two.md index a58affec73d..05353c171fc 100644 --- a/doc/pages/getting_started_part_two.md +++ b/doc/pages/getting_started_part_two.md @@ -1 +1,5 @@ +--- +redirect_to: '../user/project/pages/getting_started_part_two.md' +--- + This document was moved to [another location](../user/project/pages/getting_started_part_two.md). diff --git a/doc/permissions/permissions.md b/doc/permissions/permissions.md index 78d67aeec78..85923a40b2c 100644 --- a/doc/permissions/permissions.md +++ b/doc/permissions/permissions.md @@ -1,3 +1,7 @@ +--- +redirect_to: '../user/permissions.md' +--- + # Permissions This document was moved to [user/permissions.md](../user/permissions.md). diff --git a/doc/profile/README.md b/doc/profile/README.md index fda6d85a84c..4932cf33b87 100644 --- a/doc/profile/README.md +++ b/doc/profile/README.md @@ -1 +1,5 @@ +--- +redirect_to: '../user/profile/index.md' +--- + This document was moved to [user/profile/account](../user/profile/index.md). diff --git a/doc/profile/preferences.md b/doc/profile/preferences.md index cc16f3afe41..cf99bd61f5d 100644 --- a/doc/profile/preferences.md +++ b/doc/profile/preferences.md @@ -1 +1,5 @@ +--- +redirect_to: '../user/profile/preferences.md' +--- + This document was moved to [another location](../user/profile/preferences.md). diff --git a/doc/profile/two_factor_authentication.md b/doc/profile/two_factor_authentication.md index 60918a0339c..453ac833f59 100644 --- a/doc/profile/two_factor_authentication.md +++ b/doc/profile/two_factor_authentication.md @@ -1 +1,5 @@ +--- +redirect_to: '../user/profile/account/two_factor_authentication.md' +--- + This document was moved to [user/profile/account](../user/profile/account/two_factor_authentication.md). diff --git a/doc/project_services/bamboo.md b/doc/project_services/bamboo.md index 5b171080c72..a1ff8909f87 100644 --- a/doc/project_services/bamboo.md +++ b/doc/project_services/bamboo.md @@ -1 +1,5 @@ +--- +redirect_to: '../user/project/integrations/bamboo.md' +--- + This document was moved to [user/project/integrations/bamboo.md](../user/project/integrations/bamboo.md). diff --git a/doc/project_services/bugzilla.md b/doc/project_services/bugzilla.md index e67055d5616..1abf1b28939 100644 --- a/doc/project_services/bugzilla.md +++ b/doc/project_services/bugzilla.md @@ -1 +1,5 @@ +--- +redirect_to: '../user/project/integrations/bugzilla.md' +--- + This document was moved to [user/project/integrations/bugzilla.md](../user/project/integrations/bugzilla.md). diff --git a/doc/project_services/emails_on_push.md b/doc/project_services/emails_on_push.md index a2e831ada34..c5ab8aa5c70 100644 --- a/doc/project_services/emails_on_push.md +++ b/doc/project_services/emails_on_push.md @@ -1 +1,5 @@ +--- +redirect_to: '../user/project/integrations/emails_on_push.md' +--- + This document was moved to [user/project/integrations/emails_on_push.md](../user/project/integrations/emails_on_push.md). diff --git a/doc/project_services/hipchat.md b/doc/project_services/hipchat.md new file mode 100644 index 00000000000..4ae9f6c6b2e --- /dev/null +++ b/doc/project_services/hipchat.md @@ -0,0 +1 @@ +This document was moved to [user/project/integrations/hipchat.md](../user/project/integrations/hipchat.md). diff --git a/doc/project_services/irker.md b/doc/project_services/irker.md index 7f0850dcc24..af47abab117 100644 --- a/doc/project_services/irker.md +++ b/doc/project_services/irker.md @@ -1 +1,5 @@ +--- +redirect_to: '../user/project/integrations/irker.md' +--- + This document was moved to [user/project/integrations/irker.md](../user/project/integrations/irker.md). diff --git a/doc/project_services/jira.md b/doc/project_services/jira.md index 63614feba82..6a0108f400f 100644 --- a/doc/project_services/jira.md +++ b/doc/project_services/jira.md @@ -1 +1,5 @@ +--- +redirect_to: '../user/project/integrations/jira.md' +--- + This document was moved to [user/project/integrations/jira.md](../user/project/integrations/jira.md). diff --git a/doc/project_services/kubernetes.md b/doc/project_services/kubernetes.md index 0497a13c2b7..cfe36fcd1b2 100644 --- a/doc/project_services/kubernetes.md +++ b/doc/project_services/kubernetes.md @@ -1 +1,5 @@ +--- +redirect_to: '../user/project/integrations/kubernetes.md' +--- + This document was moved to [user/project/integrations/kubernetes.md](../user/project/integrations/kubernetes.md). diff --git a/doc/project_services/mattermost.md b/doc/project_services/mattermost.md index 554a028853e..de9f4d14cf7 100644 --- a/doc/project_services/mattermost.md +++ b/doc/project_services/mattermost.md @@ -1 +1,5 @@ +--- +redirect_to: '../user/project/integrations/mattermost.md' +--- + This document was moved to [user/project/integrations/mattermost.md](../user/project/integrations/mattermost.md). diff --git a/doc/project_services/mattermost_slash_commands.md b/doc/project_services/mattermost_slash_commands.md index 7c238b5dc37..82ec34739c1 100644 --- a/doc/project_services/mattermost_slash_commands.md +++ b/doc/project_services/mattermost_slash_commands.md @@ -1 +1,5 @@ +--- +redirect_to: '../user/project/integrations/mattermost_slash_commands.md' +--- + This document was moved to [user/project/integrations/mattermost_slash_commands.md](../user/project/integrations/mattermost_slash_commands.md). diff --git a/doc/project_services/project_services.md b/doc/project_services/project_services.md index 2c555c4edae..a355851a273 100644 --- a/doc/project_services/project_services.md +++ b/doc/project_services/project_services.md @@ -1 +1,5 @@ +--- +redirect_to: '../user/project/integrations/project_services.md' +--- + This document was moved to [user/project/integrations/project_services.md](../user/project/integrations/project_services.md). diff --git a/doc/project_services/redmine.md b/doc/project_services/redmine.md index 6010aa4dc75..05f28f00adc 100644 --- a/doc/project_services/redmine.md +++ b/doc/project_services/redmine.md @@ -1 +1,5 @@ +--- +redirect_to: '../user/project/integrations/redmine.md' +--- + This document was moved to [user/project/integrations/redmine.md](../user/project/integrations/redmine.md). diff --git a/doc/project_services/services_templates.md b/doc/project_services/services_templates.md index 8905d667c5a..ac6b85cc801 100644 --- a/doc/project_services/services_templates.md +++ b/doc/project_services/services_templates.md @@ -1 +1,5 @@ +--- +redirect_to: '../user/project/integrations/services_templates.md' +--- + This document was moved to [user/project/integrations/services_templates.md](../user/project/integrations/services_templates.md). diff --git a/doc/project_services/slack.md b/doc/project_services/slack.md index 1d3f98705e3..4c89ce92002 100644 --- a/doc/project_services/slack.md +++ b/doc/project_services/slack.md @@ -1 +1,5 @@ +--- +redirect_to: '../user/project/integrations/slack.md' +--- + This document was moved to [user/project/integrations/slack.md](../user/project/integrations/slack.md). diff --git a/doc/project_services/slack_slash_commands.md b/doc/project_services/slack_slash_commands.md index 9554c8decc8..ca0034256f1 100644 --- a/doc/project_services/slack_slash_commands.md +++ b/doc/project_services/slack_slash_commands.md @@ -1 +1,5 @@ +--- +redirect_to: '../user/project/integrations/slack_slash_commands.md' +--- + This document was moved to [user/project/integrations/slack_slash_commands.md](../user/project/integrations/slack_slash_commands.md). diff --git a/doc/push_rules/push_rules.md b/doc/push_rules/push_rules.md new file mode 100644 index 00000000000..e44eab2556e --- /dev/null +++ b/doc/push_rules/push_rules.md @@ -0,0 +1,156 @@ +# Push Rules **[STARTER]** + +Gain additional control over pushes to your repository. + +## Overview + +GitLab already offers [protected branches][protected-branches], but there are +cases when you need some specific rules like preventing git tag removal or +enforcing a special format for commit messages. + +Push rules are essentially [pre-receive Git hooks][hooks] that are easy to +enable in a user-friendly interface. They are defined globally if you are an +admin or per project so you can have different rules applied to different +projects depending on your needs. + +## Use cases + +Every push rule could have its own use case, but let's consider some examples. + +### Commit messages with a specific reference + +Let's assume you have the following requirements for your workflow: + +- every commit should reference a JIRA issue, for example: `Refactored css. Fixes JIRA-123.` +- users should not be able to remove git tags with `git push` + +All you need to do is write a simple regular expression that requires the mention +of a JIRA issue in the commit message, like `JIRA\-\d+`. + +Now when a user tries to push a commit with a message `Bugfix`, their push will +be declined. Only pushing commits with messages like `Bugfix according to JIRA-123` +will be accepted. + +### Restrict branch names + +Let's assume there's a strictly policy for branch names in your company, and +you want the branches to start with a certain name because you have different +GitLab CI jobs (`feature`, `hotfix`, `docker`, `android`, etc.) that rely on the +branch name. + +Your developers however, don't always remember that policy, so they push +various branches and CI pipelines do not work as expected. By restricting the +branch names globally in Push Rules, you can now sleep without the anxiety +of your developers' mistakes. Every branch that doesn't match your push rule +will get rejected. + +## Enabling push rules + +>**Note:** +GitLab administrators can set push rules globally under +**Admin area > Push Rules** that all new projects will inherit. You can later +override them in a project's settings. + +1. Navigate to your project's **Settings > Repository** and expand **Push Rules** +1. Set the rule you want +1. Click **Save Push Rules** for the changes to take effect + +The following options are available. + +| Push rule | GitLab version | Description | +| --------- | :------------: | ----------- | +| Removal of tags with `git push` | **Starter** 7.10 | Forbid users to remove git tags with `git push`. Tags will still be able to be deleted through the web UI. | +| Check whether author is a GitLab user | **Starter** 7.10 | Restrict commits by author (email) to existing GitLab users. | +| Committer restriction | **Premium** 10.2 | GitLab will reject any commit that was not committed by the current authenticated user | +| Check whether commit is signed through GPG | **Premium** 10.1 | Reject commit when it is not signed through GPG. Read [signing commits with GPG][signing-commits]. | +| Prevent committing secrets to Git | **Starter** 8.12 | GitLab will reject any files that are likely to contain secrets. Read [what files are forbidden](#prevent-pushing-secrets-to-the-repository). | +| Restrict by commit message | **Starter** 7.10 | Only commit messages that match this regular expression are allowed to be pushed. Leave empty to allow any commit message. Uses multiline mode, which can be disabled using `(?-m)`. | +| Restrict by commit message (negative match)| **Starter** 11.1 | Only commit messages that do not match this regular expression are allowed to be pushed. Leave empty to allow any commit message. Uses multiline mode, which can be disabled using `(?-m)`. | +| Restrict by branch name | **Starter** 9.3 | Only branch names that match this regular expression are allowed to be pushed. Leave empty to allow any branch name. | +| Restrict by commit author's email | **Starter** 7.10 | Only commit author's email that match this regular expression are allowed to be pushed. Leave empty to allow any email. | +| Prohibited file names | **Starter** 7.10 | Any committed filenames that match this regular expression are not allowed to be pushed. Leave empty to allow any filenames. | +| Maximum file size | **Starter** 7.12 | Pushes that contain added or updated files that exceed this file size (in MB) are rejected. Set to 0 to allow files of any size. | + +>**Tip:** +GitLab uses [RE2 syntax](https://github.com/google/re2/wiki/Syntax) for regular expressions in push rules. You can check your regular expressions at <https://regex-golang.appspot.com>. + +## Prevent pushing secrets to the repository + +> [Introduced][ee-385] in [GitLab Starter][ee] 8.12. + +You can turn on a predefined blacklist of files which won't be allowed to be +pushed to a repository. + +By selecting the checkbox *Prevent committing secrets to Git*, GitLab prevents +pushes to the repository when a file matches a regular expression as read from +[`files_blacklist.yml`][list] (make sure you are at the right branch +as your GitLab version when viewing this file). + +NOTE: **Note**: +Files already committed won't get restricted by this push rule. + +Below is an example list of what will be rejected by these regular expressions: + +```shell +##################### +# AWS CLI credential blobs +##################### +.aws/credentials +aws/credentials +homefolder/aws/credentials + +##################### +# Private RSA SSH keys +##################### +/ssh/id_rsa +/.ssh/personal_rsa +/config/server_rsa +id_rsa +.id_rsa + +##################### +# Private DSA SSH keys +##################### +/ssh/id_dsa +/.ssh/personal_dsa +/config/server_dsa +id_dsa +.id_dsa + +##################### +# Private ed25519 SSH keys +##################### +/ssh/id_ed25519 +/.ssh/personal_ed25519 +/config/server_ed25519 +id_ed25519 +.id_ed25519 + +##################### +# Private ECDSA SSH keys +##################### +/ssh/id_ecdsa +/.ssh/personal_ecdsa +/config/server_ecdsa +id_ecdsa +.id_ecdsa + +##################### +# Any file with .pem or .key extensions +##################### +*.pem +*.key + +##################### +# Any file ending with _history or .history extension +##################### +pry.history +bash_history +``` + +[protected-branches]: ../user/project/protected_branches.md +[signing-commits]: ../user/project/repository/gpg_signed_commits/index.md +[ee-385]: https://gitlab.com/gitlab-org/gitlab-ee/issues/385 +[list]: https://gitlab.com/gitlab-org/gitlab-ee/blob/master/ee/lib/gitlab/checks/files_blacklist.yml +[hooks]: https://git-scm.com/book/en/v2/Customizing-Git-Git-Hooks +[ee]: https://about.gitlab.com/pricing/ diff --git a/doc/raketasks/README.md b/doc/raketasks/README.md index 90187617c41..0729875daf8 100644 --- a/doc/raketasks/README.md +++ b/doc/raketasks/README.md @@ -15,3 +15,4 @@ comments: false - [Import](import.md) of git repositories in bulk - [Rebuild authorized_keys file](http://docs.gitlab.com/ce/raketasks/maintenance.html#rebuild-authorized_keys-file) task for administrators - [Migrate Uploads](../administration/raketasks/uploads/migrate.md) +- [Sanitize Uploads](../administration/raketasks/uploads/sanitize.md) diff --git a/doc/raketasks/backup_restore.md b/doc/raketasks/backup_restore.md index 069c87f921a..764916ca82d 100644 --- a/doc/raketasks/backup_restore.md +++ b/doc/raketasks/backup_restore.md @@ -100,6 +100,13 @@ the gitlab task runner pod via `kubectl`. Refer to [backing up a GitLab installa kubectl exec -it <gitlab task-runner pod> backup-utility ``` +Similarly to the Kubernetes case, if you have scaled out your GitLab +cluster to use multiple application servers, you should pick a +designated node (that won't be auto-scaled away) for running the +backup rake task. Because the backup rake task is tightly coupled to +the main Rails application, this is typically a node on which you're +also running Unicorn/Puma and/or Sidekiq. + Example output: ``` @@ -195,6 +202,26 @@ To use the `copy` strategy instead of the default streaming strategy, specify sudo gitlab-rake gitlab:backup:create STRATEGY=copy ``` +### Backup filename + +By default a backup file is created according to the specification in [the Backup timestamp](#backup-timestamp) section above. You can however override the `[TIMESTAMP]` part of the filename by setting the `BACKUP` environment variable. For example: + +```sh +sudo gitlab-rake gitlab:backup:create BACKUP=dump +``` + +The resulting file will then be `dump_gitlab_backup.tar`. This is useful for systems that make use of rsync and incremental backups, and will result in considerably faster transfer speeds. + +### Rsyncable + +To make sure the generated archive is intelligently transferable by rsync, the `GZIP_RSYNCABLE=yes` option can be set. This will set the `--rsyncable` option to `gzip`. This is only useful in combination with setting [the Backup filename option](#backup-filename). + +Note that the `--rsyncable` option in `gzip` is not guaranteed to be available on all distributions. To verify that it is available in your distribution you can run `gzip --help` or consult the man pages. + +```sh +sudo gitlab-rake gitlab:backup:create BACKUP=dump GZIP_RSYNCABLE=yes +``` + ### Excluding specific directories from the backup You can choose what should be exempt from the backup up by adding the environment variable `SKIP`. @@ -382,6 +409,8 @@ an access key from the Google console first: 1. Select "Interoperability" and create an access key 1. Make note of the "Access Key" and "Secret" and replace them in the configurations below +1. In the buckets advanced settings ensure the Access Control option "Set object-level + and bucket-level permissions" is selected 1. Make sure you already have a bucket created For Omnibus GitLab packages: @@ -603,7 +632,7 @@ directory (repositories, uploads). To restore a backup, you will also need to restore `/etc/gitlab/gitlab-secrets.json` (for Omnibus packages) or `/home/git/gitlab/.secret` (for installations from source). This file contains the database encryption key, -[CI/CD variables](../ci/variables/README.md#variables), and +[CI/CD variables](../ci/variables/README.md#gitlab-cicd-environment-variables), and variables used for [two-factor authentication](../user/profile/account/two_factor_authentication.md). If you fail to restore this encryption key file along with the application data backup, users with two-factor authentication enabled and GitLab Runners will @@ -819,15 +848,24 @@ including (but not restricted to): * [Project mirroring](../workflow/repository_mirroring.md) * [Web hooks](../user/project/integrations/webhooks.md) -In the case of CI/CD, variables, you might experience some weird behavior, like -stuck jobs or 500 errors. In that case, you can try removing contents of the -`ci_group_variables` and `ci_project_variables` tables from the database. +In cases like CI/CD variables and Runner authentication, you might +experience some unexpected behavior such as: + +- Stuck jobs. +- 500 errors. + +In this case, you are required to reset all the tokens for CI/CD variables +and Runner Authentication, which is described in more detail below. After +resetting the tokens, you should be able to visit your project and the jobs +will have started running again. CAUTION: **Warning:** Use the following commands at your own risk, and make sure you've taken a backup beforehand. -1. Enter the Rails console: +#### Reset CI/CD variables + +1. Enter the DB console: For Omnibus GitLab packages: @@ -860,8 +898,39 @@ backup beforehand. 1. You may need to reconfigure or restart GitLab for the changes to take effect. -You should now be able to visit your project, and the jobs will start -running again. + +#### Reset Runner registration tokens + +1. Enter the DB console: + + For Omnibus GitLab packages: + + ```sh + sudo gitlab-rails dbconsole + ``` + + For installations from source: + + ```sh + sudo -u git -H bundle exec rails dbconsole RAILS_ENV=production + ``` + +1. Clear all the tokens for projects, groups, and the whole instance: + + CAUTION: **Caution:** + The last UPDATE operation will stop the runners being able to pick up + new jobs. You must register new runners. + + ```sql + -- Clear project tokens + UPDATE projects SET runners_token = null, runners_token_encrypted = null; + -- Clear group tokens + UPDATE namespaces SET runners_token = null, runners_token_encrypted = null; + -- Clear instance tokens + UPDATE application_settings SET runners_registration_token_encrypted = null; + -- Clear runner tokens + UPDATE ci_runners SET token = null, token_encrypted = null; + ``` A similar strategy can be employed for the remaining features - by removing the data that cannot be decrypted, GitLab can be brought back into working order, diff --git a/doc/raketasks/check.md b/doc/raketasks/check.md index f7f6a40cd04..ceb089e80c0 100644 --- a/doc/raketasks/check.md +++ b/doc/raketasks/check.md @@ -1,3 +1,5 @@ -# Check Rake Tasks +--- +redirect_to: '../administration/raketasks/check.md' +--- -This document was moved to [administration/raketasks/check](../administration/raketasks/check.md). +This document was moved to [another location](../administration/raketasks/check.md). diff --git a/doc/raketasks/cleanup.md b/doc/raketasks/cleanup.md index e70a009323e..f5c788af578 100644 --- a/doc/raketasks/cleanup.md +++ b/doc/raketasks/cleanup.md @@ -23,6 +23,16 @@ sudo gitlab-rake gitlab:cleanup:repos bundle exec rake gitlab:cleanup:repos RAILS_ENV=production ``` +Remove old repository copies from repositories moved to another storage. + +``` +# omnibus-gitlab +sudo gitlab-rake gitlab:cleanup:moved + +# installation from source +bundle exec rake gitlab:cleanup:moved RAILS_ENV=production +``` + Clean up local project upload files if they don't exist in GitLab database. The task attempts to fix the file if it can find its project, otherwise it moves the file to a lost and found directory. diff --git a/doc/raketasks/import.md b/doc/raketasks/import.md index bb316df5b9a..b59c06a24ea 100644 --- a/doc/raketasks/import.md +++ b/doc/raketasks/import.md @@ -16,7 +16,7 @@ The new folder needs to have git user ownership and read/write/execute access for git user and its group: ``` -sudo -u git mkdir /var/opt/gitlab/git-data/repository-import-<date>/new_group +sudo -u git mkdir -p /var/opt/gitlab/git-data/repository-import-<date>/new_group ``` ### Copy your bare repositories inside this newly created folder: diff --git a/doc/raketasks/maintenance.md b/doc/raketasks/maintenance.md index 266aeb7d60e..f554a09d94d 100644 --- a/doc/raketasks/maintenance.md +++ b/doc/raketasks/maintenance.md @@ -1,3 +1,5 @@ -# Maintenance Rake Tasks +--- +redirect_to: '../administration/raketasks/maintenance.md' +--- -This document was moved to [administration/raketasks/maintenance](../administration/raketasks/maintenance.md). +This document was moved to [another location](../administration/raketasks/maintenance.md). diff --git a/doc/security/img/two_factor_authentication_group_settings.png b/doc/security/img/two_factor_authentication_group_settings.png Binary files differdeleted file mode 100644 index 05d95554fd9..00000000000 --- a/doc/security/img/two_factor_authentication_group_settings.png +++ /dev/null diff --git a/doc/security/img/two_factor_authentication_settings.png b/doc/security/img/two_factor_authentication_settings.png Binary files differdeleted file mode 100644 index 2a2208f98bd..00000000000 --- a/doc/security/img/two_factor_authentication_settings.png +++ /dev/null diff --git a/doc/security/rack_attack.md b/doc/security/rack_attack.md index ad83dc05a93..66081d7e376 100644 --- a/doc/security/rack_attack.md +++ b/doc/security/rack_attack.md @@ -94,7 +94,7 @@ In case you want to remove a blocked IP, follow these steps: 1. Find the IPs that have been blocked in the production log: ```sh - grep "Rack_Attack" /var/log/gitlab/gitlab-rails/production.log + grep "Rack_Attack" /var/log/gitlab/gitlab-rails/auth.log ``` 1. Since the blacklist is stored in Redis, you need to open up `redis-cli`: diff --git a/doc/security/two_factor_authentication.md b/doc/security/two_factor_authentication.md index 4b65b901487..2ece4ed3fc9 100644 --- a/doc/security/two_factor_authentication.md +++ b/doc/security/two_factor_authentication.md @@ -16,39 +16,35 @@ enforce everyone to set up 2FA, you can choose from two different ways: - Enforce on next login. - Suggest on next login, but allow a grace period before enforcing. -In the Admin area under **Settings** (`/admin/application_settings`), look for -the "Sign-in Restrictions" area, where you can configure both. +After the configured grace period has elapsed, users will be able to log in but +won't be able to leave the 2FA configuration area at `/profile/two_factor_auth`. + +To enable 2FA for all users: + +1. Navigate to **Admin area > Settings > General** (`/admin/application_settings`). +1. Expand the **Sign-in restrictions** section, where you can configure both. If you want 2FA enforcement to take effect on next login, change the grace period to `0`. ---- - - +## Enforcing 2FA for all users in a group ---- +If you want to enforce 2FA only for certain groups, you can: -## Enforcing 2FA for all users in a group +1. Enable it in the group's **Settings > General** page. +1. Optionally specify a grace period as above. -If you want to enforce 2FA only for certain groups, you can enable it in the -group settings and specify a grace period as above. To change this setting you -need to be administrator or owner of the group. +To change this setting, you need to be administrator or owner of the group. If there are multiple 2FA requirements (i.e. group + all users, or multiple groups) the shortest grace period will be used. ---- - - - ---- - ## Disabling 2FA for everyone There may be some special situations where you want to disable 2FA for everyone even when forced 2FA is disabled. There is a rake task for that: -``` +```sh # Omnibus installations sudo gitlab-rake gitlab:two_factor:disable_for_all_users @@ -56,5 +52,6 @@ sudo gitlab-rake gitlab:two_factor:disable_for_all_users sudo -u git -H bundle exec rake gitlab:two_factor:disable_for_all_users RAILS_ENV=production ``` -**IMPORTANT: this is a permanent and irreversible action. Users will have to - reactivate 2FA from scratch if they want to use it again.** +CAUTION: **Caution:** +This is a permanent and irreversible action. Users will have to +reactivate 2FA from scratch if they want to use it again. diff --git a/doc/ssh/README.md b/doc/ssh/README.md index 9c4a391e8da..3bfebfc5d9b 100644 --- a/doc/ssh/README.md +++ b/doc/ssh/README.md @@ -55,7 +55,7 @@ As an admin, you can restrict By default, all keys are permitted, which is also the case for [GitLab.com](../user/gitlab_com/index.md#ssh-host-keys-fingerprints). -## ED25519 SSH keys +### ED25519 SSH keys Following [best practices](https://linux-audit.com/using-ed25519-openssh-keys-instead-of-dsa-rsa-ecdsa/), you should always favor [ED25519](https://ed25519.cr.yp.to/) SSH keys, since they @@ -65,7 +65,7 @@ They were introduced in OpenSSH 6.5, so any modern OS should include the option to create them. If for any reason your OS or the GitLab instance you interact with doesn't support this, you can fallback to RSA. -## RSA SSH keys +### RSA SSH keys RSA keys are the most common ones and therefore the most compatible with servers that may have an old OpenSSH version. Use them if the GitLab server @@ -166,12 +166,13 @@ Now, it's time to add the newly created public key to your GitLab account. NOTE: **Note:** If you opted to create an RSA key, the name might differ. -1. Add your public SSH key to your GitLab account by clicking your avatar - in the upper right corner and selecting **Settings**. From there on, - navigate to **SSH Keys** and paste your public key in the "Key" section. - If you created the key with a comment, this will appear under "Title". - If not, give your key an identifiable title like _Work Laptop_ or - _Home Workstation_, and click **Add key**. +1. Add your **public** SSH key to your GitLab account by: + 1. Clicking your avatar in the upper right corner and selecting **Settings**. + 1. Navigating to **SSH Keys** and pasting your **public** key in the **Key** field. If you: + + - Created the key with a comment, this will appear in the **Title** field. + - Created the key without a comment, give your key an identifiable title like _Work Laptop_ or _Home Workstation_. + 1. Click the **Add key** button. NOTE: **Note:** If you manually copied your public SSH key make sure you copied the entire @@ -305,7 +306,7 @@ who needs to know and configure the private key. GitLab administrators set up Global Deploy keys in the Admin area under the section **Deploy Keys**. Ensure keys have a meaningful title as that will be the primary way for project maintainers and owners to identify the correct Global -Deploy key to add. For instance, if the key gives access to a SaaS CI instance, +Deploy key to add. For instance, if the key gives access to a SaaS CI instance, use the name of that service in the key name if that is all it is used for. When creating Global Shared Deploy keys, give some thought to the granularity of keys - they could be of very narrow usage such as just a specific service or diff --git a/doc/subscriptions/billing_table.png b/doc/subscriptions/billing_table.png Binary files differnew file mode 100644 index 00000000000..acd1b6193ec --- /dev/null +++ b/doc/subscriptions/billing_table.png diff --git a/doc/subscriptions/index.md b/doc/subscriptions/index.md new file mode 100644 index 00000000000..dfd80f8882e --- /dev/null +++ b/doc/subscriptions/index.md @@ -0,0 +1,103 @@ +# Subscription setup and management + +This page will help get you started with your new subscription or manage an existing one, whether you have subscribed to GitLab.com or self-managed GitLab. + +To subscribe, upgrade, or read more about the types of subscriptions, please see [Subscribe to GitLab](../README.md#subscribe-to-gitlab) on the GitLab Documentation landing page. + +## Set up GitLab + +Learn how GitLab helps you in the stages of the DevOps lifecycle by learning more [about the GitLab product](https://about.gitlab.com/product/), [GitLab features](https://about.gitlab.com/features/), and [GitLab Documentation](../README.md). + +### Self-managed: Install GitLab + +Take a look at [installing GitLab](https://about.gitlab.com/install/) and our [administrator documentation](../administration/index.md). Then, follow the instructions below under [Your subscription](#your-subscription) to apply your license file. + +### GitLab.com: Create a user and group + +Start with creating a user account for yourself using our [sign up page](https://gitlab.com/users/sign_in#register-pane). + +[GitLab groups](../user/group/index.md) help assemble related projects together allowing you to grant members access to several projects at once. A group is not required if you plan on having [projects](../user/project/) inside a personal namespace. + +## Your subscription + +You can view and manage subscriptions through our [Customers portal](https://customers.gitlab.com/). Information on applying your subscription is below. + +Please also see our [subscription FAQ](https://about.gitlab.com/pricing/licensing-faq/) + +### View subscription and seats + +To view and manage the subscriptions you have purchased and the number of seats associated with the subscription, please visit and log into the [Customers’ Portal](https://customers.gitlab.com/subscriptions). For more information, please see our [subscription FAQ](https://about.gitlab.com/pricing/licensing-faq/) and [pricing page](https://about.gitlab.com/pricing/), which includes information on our [true-up pricing policy](https://about.gitlab.com/handbook/product/pricing/#true-up-pricing) when adding more users than at the time of purchase. + +Please note that this account may have the same email, but is a _separate_ login from your GitLab.com account. If the two accounts are linked together, then you can use the "sign in with GitLab.com account" link underneath the `Sign In` button. + +### Change billing information + +In the customers portal, go to the `My Account` page, then revise the `Account Details` information and click on the `Update Account` button. + +Future purchases will use the information in this section. The email listed in this section is used for the Customers Portal login and for license related email communication. + +### Self-managed: Apply your license file + +After purchase, the license file is sent to the email address tied to the Customers portal account, which needs to be [uploaded to the GitLab instance](../user/admin_area/license.md#uploading-your-license). + +### Link your GitLab.com account with your Customers Portal account + +NOTE: **Note:** This is *required* for GitLab.com subscriptions. + +Once signed into the customers portal, if your account is not already linked, you should be prompted to link your account with a "Link my GitLab Account" button. + +You can also go to the [My Account](https://customers.gitlab.com/customers/edit) page to add or change the GitLab.com account link. + +### Change the linked GitLab.com account for your Customers Portal account + +To change which GitLab.com account is associated with a Customers Portal account, please follow these steps: + +1. Log into the [Customers Portal](https://customers.gitlab.com/customers/sign_in). +1. In a separate browser tab, visit [GitLab.com](https://gitlab.com) to ensure you are not logged in, or if you are, log out. +1. Back on the Customers Portal page, click [My Account](https://customers.gitlab.com/customers/edit) in the top menu. +1. Under `Your GitLab.com account`, click the `Change linked account` button. +1. Have the user you want associated log in to their [GitLab.com](https://gitlab.com) account. + +### GitLab.com: Associate your namespace with your subscription + +Once your GitLab.com account is linked, you can go to your [Subscriptions](https://customers.gitlab.com/subscriptions) page to choose or change the namespace your subscription applies to. + +Please note that you need to be a group owner to associate a group to your subscription. + +### Confirm or upgrade your GitLab.com subscription details within GitLab + +To see the status of your GitLab.com subscription, you can click on the Billings +section of the relevant namespace: + +* For individuals, this is located at https://gitlab.com/profile/billings under +in your Settings, +* For groups, this is located under the group's Settings dropdown, under Billing. + +For groups, you can see details of your subscription - including your current +plan - in the included table: + + + +| Field | Description | +| ------ | ------ | +| Seats in subscription | If this is a paid plan, this represents the number of seats you've paid to support in your group. | +| Seats currently in use | The number of active seats currently in use. | +| Max seats used | The highest number of seats you've used. If this exceeds the seats in subscription, you may owe an additional fee for the additional users. | +| Seats owed | If your max seats used exceeds the seats in your subscription, you'll owe an additional fee for the users you've added. | +| Subscription start date | The date your subscription started. If this is for a Free plan, this is the date you transitioned off your group's paid plan. | +| Subscription end date | The date your current subscription will end. This does not apply to Free plans. | + +## Need help? + +[GitLab's Documentation](https://docs.gitlab.com/) offers a wide range of topics covering the use and administration of GitLab. + +We also encourage all users to search our project trackers for known issues and existing feature requests in: + +- [GitLab CE](https://gitlab.com/gitlab-org/gitlab-ce/issues/) for features included in all tiers, and +- [GitLab EE](https://gitlab.com/gitlab-org/gitlab-ee/issues/) for paid-tier features. + +These issues are the best avenue for getting updates on specific product plans and for communicating directly with the relevant GitLab team members. + +### Contacting Support + +Learn more about the tiers of [GitLab Support](https://about.gitlab.com/support/) or [submit a request via the Support Portal](https://support.gitlab.com/hc/en-us/requests/new). diff --git a/doc/tools/email.md b/doc/tools/email.md new file mode 100644 index 00000000000..ab39206ffa4 --- /dev/null +++ b/doc/tools/email.md @@ -0,0 +1,38 @@ +# Email from GitLab **[STARTER ONLY]** + +As a GitLab administrator you can email GitLab users from within GitLab. + +## Overview + +GitLab provides a simple tool to email all users or users of a chosen group or +project right from the admin area. Users will receive the email to their primary +email address. + +## Use-cases + +- Notify your users about a new project, a new feature, or a new product launch. +- Notify your users about a new deployment, or that will be downtime expected + for a particular reason. + +## Sending emails to users from within GitLab + +1. Go to the admin area using the wrench icon in the top right corner and + navigate to **Overview > Users > Send email to users**. + +  + +1. Compose an email and choose where it will be sent (all users or users of a + chosen group or project): + +  + +## Unsubscribing from emails + +User can choose to unsubscribe from receiving emails from GitLab by following +the unsubscribe link from the email. Unsubscribing is unauthenticated in order +to keep the simplicity of this feature. + +On unsubscribe, user will receive an email notifying that unsubscribe happened. +The endpoint that provides the unsubscribe option is rate-limited. + +[ee]: https://about.gitlab.com/pricing/ diff --git a/doc/tools/email1.png b/doc/tools/email1.png Binary files differnew file mode 100644 index 00000000000..e79ccc3e9a9 --- /dev/null +++ b/doc/tools/email1.png diff --git a/doc/tools/email2.png b/doc/tools/email2.png Binary files differnew file mode 100644 index 00000000000..d073c0e42da --- /dev/null +++ b/doc/tools/email2.png diff --git a/doc/topics/authentication/index.md b/doc/topics/authentication/index.md index df6897002c9..228da2d1f57 100644 --- a/doc/topics/authentication/index.md +++ b/doc/topics/authentication/index.md @@ -17,11 +17,11 @@ This page gathers all the resources for the topic **Authentication** within GitL ## GitLab administrators - [LDAP (Community Edition)](../../administration/auth/ldap.md) -- [LDAP (Enterprise Edition)](https://docs.gitlab.com/ee/administration/auth/ldap-ee.html) +- [LDAP (Enterprise Edition)](../../administration/auth/ldap-ee.md) **[STARTER]** - [Enforce Two-factor Authentication (2FA)](../../security/two_factor_authentication.md#enforce-two-factor-authentication-2fa) - **Articles:** - [How to Configure LDAP with GitLab CE](../../administration/auth/how_to_configure_ldap_gitlab_ce/index.md) - - [How to Configure LDAP with GitLab EE](https://docs.gitlab.com/ee/administration/auth/how_to_configure_ldap_gitlab_ee/index.md) + - [How to Configure LDAP with GitLab EE](../../administration/auth/how_to_configure_ldap_gitlab_ee/index.md) **[STARTER]** - [Feature Highlight: LDAP Integration](https://about.gitlab.com/2014/07/10/feature-highlight-ldap-sync/) - [Debugging LDAP](https://about.gitlab.com/handbook/support/workflows/support-engineering/ldap/debugging_ldap.html) - **Integrations:** @@ -30,12 +30,14 @@ This page gathers all the resources for the topic **Authentication** within GitL - [Atlassian Crowd OmniAuth Provider](../../administration/auth/crowd.md) - [CAS OmniAuth Provider](../../integration/cas.md) - [SAML OmniAuth Provider](../../integration/saml.md) + - [SAML for GitLab.com Groups](../../user/group/saml_sso/index.md) **[SILVER ONLY]** + - [SCIM user provisioning for GitLab.com Groups](../../user/group/saml_sso/scim_setup.md) **[SILVER ONLY]** - [Okta SSO provider](../../administration/auth/okta.md) - - [Kerberos integration (GitLab EE)](https://docs.gitlab.com/ee/integration/kerberos.html) + - [Kerberos integration (GitLab EE)](../../integration/kerberos.md) **[STARTER]** ## API -- [OAuth 2 Tokens](../../api/README.md#oauth-2-tokens) +- [OAuth 2 Tokens](../../api/README.md#oauth2-tokens) - [Personal access tokens](../../api/README.md#personal-access-tokens) - [Impersonation tokens](../../api/README.md#impersonation-tokens) - [GitLab as an OAuth2 provider](../../api/oauth2.md#gitlab-as-an-oauth2-provider) diff --git a/doc/topics/autodevops/img/autodevops_domain_variables.png b/doc/topics/autodevops/img/autodevops_domain_variables.png Binary files differdeleted file mode 100644 index b6f8864796f..00000000000 --- a/doc/topics/autodevops/img/autodevops_domain_variables.png +++ /dev/null diff --git a/doc/topics/autodevops/img/guide_connect_cluster.png b/doc/topics/autodevops/img/guide_connect_cluster.png Binary files differdeleted file mode 100644 index 703d536f37a..00000000000 --- a/doc/topics/autodevops/img/guide_connect_cluster.png +++ /dev/null diff --git a/doc/topics/autodevops/img/guide_create_cluster.png b/doc/topics/autodevops/img/guide_create_cluster.png Binary files differdeleted file mode 100644 index cd1d0fdd8da..00000000000 --- a/doc/topics/autodevops/img/guide_create_cluster.png +++ /dev/null diff --git a/doc/topics/autodevops/img/guide_gke_apis_after.png b/doc/topics/autodevops/img/guide_gke_apis_after.png Binary files differdeleted file mode 100644 index 380de958867..00000000000 --- a/doc/topics/autodevops/img/guide_gke_apis_after.png +++ /dev/null diff --git a/doc/topics/autodevops/img/guide_gke_apis_before.png b/doc/topics/autodevops/img/guide_gke_apis_before.png Binary files differdeleted file mode 100644 index d06fc707887..00000000000 --- a/doc/topics/autodevops/img/guide_gke_apis_before.png +++ /dev/null diff --git a/doc/topics/autodevops/img/guide_merge_request_ide.png b/doc/topics/autodevops/img/guide_merge_request_ide.png Binary files differdeleted file mode 100644 index c825b0849e1..00000000000 --- a/doc/topics/autodevops/img/guide_merge_request_ide.png +++ /dev/null diff --git a/doc/topics/autodevops/index.md b/doc/topics/autodevops/index.md index 7a18354bf66..b00a8afa386 100644 --- a/doc/topics/autodevops/index.md +++ b/doc/topics/autodevops/index.md @@ -13,7 +13,7 @@ Starting with GitLab 11.3, the Auto DevOps pipeline is enabled by default for al projects. If it has not been explicitly enabled for the project, Auto DevOps will be automatically disabled on the first pipeline failure. Your project will continue to use an alternative [CI/CD configuration file](../../ci/yaml/README.md) if one is found. A GitLab -administrator can [change this setting](../../user/admin_area/settings/continuous_integration.html#auto-devops-core-only) +administrator can [change this setting](../../user/admin_area/settings/continuous_integration.md#auto-devops-core-only) in the admin area. With Auto DevOps, the software development process becomes easier to set up @@ -126,10 +126,6 @@ Auto Deploy, and Auto Monitoring will be silently skipped. ## Auto DevOps base domain -NOTE: **Note** -`AUTO_DEVOPS_DOMAIN` environment variable is deprecated and -[is scheduled to be removed](https://gitlab.com/gitlab-org/gitlab-ce/issues/56959). - The Auto DevOps base domain is required if you want to make use of [Auto Review Apps](#auto-review-apps) and [Auto Deploy](#auto-deploy). It can be defined in any of the following places: @@ -141,7 +137,7 @@ in any of the following places: NOTE: **Note** The Auto DevOps base domain variable (`KUBE_INGRESS_BASE_DOMAIN`) follows the same order of precedence -as other environment [variables](../../ci/variables/README.md#priority-of-variables). +as other environment [variables](../../ci/variables/README.md#priority-of-environment-variables). A wildcard DNS A record matching the base domain(s) is required, for example, given a base domain of `example.com`, you'd need a DNS entry like: @@ -162,6 +158,12 @@ Auto DevOps base domain to `1.2.3.4.nip.io`. Once set up, all requests will hit the load balancer, which in turn will route them to the Kubernetes pods that run your application(s). +NOTE: **Note:** +From GitLab 11.8, `KUBE_INGRESS_BASE_DOMAIN` replaces `AUTO_DEVOPS_DOMAIN`. +Support for `AUTO_DEVOPS_DOMAIN` was [removed in GitLab +12.0](https://gitlab.com/gitlab-org/gitlab-ce/issues/56959). + + ## Using multiple Kubernetes clusters **[PREMIUM]** When using Auto DevOps, you may want to deploy different environments to @@ -179,7 +181,7 @@ Those environments are tied to jobs that use [Auto Deploy](#auto-deploy), so except for the environment scope, they would also need to have a different domain they would be deployed to. This is why you need to define a separate `KUBE_INGRESS_BASE_DOMAIN` variable for all the above -[based on the environment](https://docs.gitlab.com/ee/ci/variables/index.html#limiting-environment-scopes-of-variables-premium). +[based on the environment](../../ci/variables/README.md#limiting-environment-scopes-of-environment-variables-premium). The following table is an example of how the three different clusters would be configured. @@ -209,10 +211,6 @@ and verifying that your app is deployed as a review app in the Kubernetes cluster with the `review/*` environment scope. Similarly, you can check the other environments. -NOTE: **Note:** -From GitLab 11.8, `KUBE_INGRESS_BASE_DOMAIN` replaces `AUTO_DEVOPS_DOMAIN`. -`AUTO_DEVOPS_DOMAIN` [is scheduled to be removed](https://gitlab.com/gitlab-org/gitlab-ce/issues/56959). - ## Enabling/Disabling Auto DevOps When first using Auto Devops, review the [requirements](#requirements) to ensure all necessary components to make @@ -230,8 +228,25 @@ can enable/disable Auto DevOps at either the project-level or instance-level. 1. Click **Save changes** for the changes to take effect. NOTE: **Note:** -Even when disabled at the instance level, project maintainers are still able to enable -Auto DevOps at the project level. +Even when disabled at the instance level, group owners and project maintainers are still able to enable +Auto DevOps at group-level and project-level, respectively. + +### Enabling/disabling Auto DevOps at the group-level + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/issues/52447) in GitLab 11.10. + +To enable or disable Auto DevOps at the group-level: + +1. Go to group's **Settings > CI/CD > Auto DevOps** page. +1. Toggle the **Default to Auto DevOps pipeline** checkbox (checked to enable, unchecked to disable). +1. Click **Save changes** button for the changes to take effect. + +When enabling or disabling Auto DevOps at group-level, group configuration will be implicitly used for +the subgroups and projects inside that group, unless Auto DevOps is specifically enabled or disabled on +the subgroup or project. + +NOTE: **Note** +Only administrators and group owners are allowed to enable or disable Auto DevOps at group-level. ### Enabling/disabling Auto DevOps at the project-level @@ -291,7 +306,7 @@ Auto Build creates a build of the application using an existing `Dockerfile` or Heroku buildpacks. Either way, the resulting Docker image is automatically pushed to the -[Container Registry][container-registry] and tagged with the commit SHA. +[Container Registry][container-registry] and tagged with the commit SHA or tag. #### Auto Build using a Dockerfile @@ -348,7 +363,7 @@ created, and is uploaded as an artifact which you can later download and check out. Any differences between the source and target branches are also -[shown in the merge request widget](https://docs.gitlab.com/ee/user/project/merge_requests/code_quality.html). +[shown in the merge request widget](../../user/project/merge_requests/code_quality.md). ### Auto SAST **[ULTIMATE]** @@ -360,8 +375,8 @@ analysis on the current code and checks for potential security issues. Once the report is created, it's uploaded as an artifact which you can later download and check out. -Any security warnings are also -[shown in the merge request widget](https://docs.gitlab.com/ee//user/project/merge_requests/sast.html). +Any security warnings are also shown in the merge request widget. Read more how +[SAST works](../../user/application_security/sast/index.md). NOTE: **Note:** The Auto SAST stage will be skipped on licenses other than Ultimate. @@ -379,8 +394,8 @@ to run analysis on the project dependencies and checks for potential security is report is created, it's uploaded as an artifact which you can later download and check out. -Any security warnings are also -[shown in the merge request widget](https://docs.gitlab.com/ee//user/project/merge_requests/dependency_scanning.html). +Any security warnings are also shown in the merge request widget. Read more about +[Dependency Scanning](../../user/application_security/dependency_scanning/index.md). NOTE: **Note:** The Auto Dependency Scanning stage will be skipped on licenses other than Ultimate. @@ -398,8 +413,8 @@ to search the project dependencies for their license. Once the report is created, it's uploaded as an artifact which you can later download and check out. -Any licenses are also -[shown in the merge request widget](https://docs.gitlab.com/ee//user/project/merge_requests/license_management.html). +Any licenses are also shown in the merge request widget. Read more how +[License Management works](../../user/application_security/license_management/index.md). NOTE: **Note:** The Auto License Management stage will be skipped on licenses other than Ultimate. @@ -414,8 +429,8 @@ Docker image and checks for potential security issues. Once the report is created, it's uploaded as an artifact which you can later download and check out. -Any security warnings are also -[shown in the merge request widget](https://docs.gitlab.com/ee//user/project/merge_requests/container_scanning.html). +Any security warnings are also shown in the merge request widget. Read more how +[Container Scanning works](../../user/application_security/container_scanning/index.md). NOTE: **Note:** The Auto Container Scanning stage will be skipped on licenses other than Ultimate. @@ -427,13 +442,6 @@ This is an optional step, since many projects do not have a Kubernetes cluster available. If the [requirements](#requirements) are not met, the job will silently be skipped. -CAUTION: **Caution:** -Your apps should *not* be manipulated outside of Helm (using Kubernetes directly.) -This can cause confusion with Helm not detecting the change, and subsequent -deploys with Auto DevOps can undo your changes. Also, if you change something -and want to undo it by deploying again, Helm may not detect that anything changed -in the first place, and thus not realize that it needs to re-apply the old config. - [Review Apps][review-app] are temporary application environments based on the branch's code so developers, designers, QA, product managers, and other reviewers can actually see and interact with code changes as part of the review @@ -442,13 +450,31 @@ process. Auto Review Apps create a Review App for each branch. Auto Review Apps will deploy your app to your Kubernetes cluster only. When no cluster is available, no deployment will occur. -The Review App will have a unique URL based on the project name, the branch +The Review App will have a unique URL based on the project ID, the branch or tag name, and a unique number, combined with the Auto DevOps base domain. For -example, `user-project-branch-1234.example.com`. A link to the Review App shows -up in the merge request widget for easy discovery. When the branch is deleted, +example, `13083-review-project-branch-123456.example.com`. A link to the Review App shows +up in the merge request widget for easy discovery. When the branch or tag is deleted, for example after the merge request is merged, the Review App will automatically be deleted. +Review apps are deployed using the +[auto-deploy-app](https://gitlab.com/gitlab-org/charts/auto-deploy-app) chart with +Helm. The app will be deployed into the [Kubernetes +namespace](../../user/project/clusters/index.md#deployment-variables) +for the environment. + +Since GitLab 11.4, a [local +Tiller](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/22036) is +used. Previous versions of GitLab had a Tiller installed in the project +namespace. + +CAUTION: **Caution:** +Your apps should *not* be manipulated outside of Helm (using Kubernetes directly). +This can cause confusion with Helm not detecting the change and subsequent +deploys with Auto DevOps can undo your changes. Also, if you change something +and want to undo it by deploying again, Helm may not detect that anything changed +in the first place, and thus not realize that it needs to re-apply the old config. + ### Auto DAST **[ULTIMATE]** > Introduced in [GitLab Ultimate][ee] 10.4. @@ -459,8 +485,8 @@ to perform an analysis on the current code and checks for potential security issues. Once the report is created, it's uploaded as an artifact which you can later download and check out. -Any security warnings are also -[shown in the merge request widget](https://docs.gitlab.com/ee//user/project/merge_requests/dast.html). +Any security warnings are also shown in the merge request widget. Read how +[DAST works](../../user/application_security/dast/index.md). NOTE: **Note:** The Auto DAST stage will be skipped on licenses other than Ultimate. @@ -478,7 +504,7 @@ Auto Browser Performance Testing utilizes the [Sitespeed.io container](https://h ``` Any performance differences between the source and target branches are also -[shown in the merge request widget](https://docs.gitlab.com/ee//user/project/merge_requests/browser_performance_testing.html). +[shown in the merge request widget](../../user/project/merge_requests/browser_performance_testing.md). ### Auto Deploy @@ -487,13 +513,6 @@ This is an optional step, since many projects do not have a Kubernetes cluster available. If the [requirements](#requirements) are not met, the job will silently be skipped. -CAUTION: **Caution:** -Your apps should *not* be manipulated outside of Helm (using Kubernetes directly.) -This can cause confusion with Helm not detecting the change, and subsequent -deploys with Auto DevOps can undo your changes. Also, if you change something -and want to undo it by deploying again, Helm may not detect that anything changed -in the first place, and thus not realize that it needs to re-apply the old config. - After a branch or merge request is merged into the project's default branch (usually `master`), Auto Deploy deploys the application to a `production` environment in the Kubernetes cluster, with a namespace based on the project name and unique @@ -506,17 +525,40 @@ enable them. You can make use of [environment variables](#environment-variables) to automatically scale your pod replicas. -It's important to note that when a project is deployed to a Kubernetes cluster, -it relies on a Docker image that has been pushed to the -[GitLab Container Registry](../../user/project/container_registry.md). Kubernetes -fetches this image and uses it to run the application. If the project is public, -the image can be accessed by Kubernetes without any authentication, allowing us -to have deployments more usable. If the project is private/internal, the -Registry requires credentials to pull the image. Currently, this is addressed -by providing `CI_JOB_TOKEN` as the password that can be used, but this token will -no longer be valid as soon as the deployment job finishes. This means that -Kubernetes can run the application, but in case it should be restarted or -executed somewhere else, it cannot be accessed again. +Apps are deployed using the +[auto-deploy-app](https://gitlab.com/gitlab-org/charts/auto-deploy-app) chart with +Helm. The app will be deployed into the [Kubernetes +namespace](../../user/project/clusters/index.md#deployment-variables) +for the environment. + +Since GitLab 11.4, a [local +Tiller](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/22036) is +used. Previous versions of GitLab had a Tiller installed in the project +namespace. + +CAUTION: **Caution:** +Your apps should *not* be manipulated outside of Helm (using Kubernetes directly). +This can cause confusion with Helm not detecting the change and subsequent +deploys with Auto DevOps can undo your changes. Also, if you change something +and want to undo it by deploying again, Helm may not detect that anything changed +in the first place, and thus not realize that it needs to re-apply the old config. + +> [Introduced][ce-19507] in GitLab 11.0. + +For internal and private projects a [GitLab Deploy Token](../../user/project/deploy_tokens/index.md#gitlab-deploy-token) +will be automatically created, when Auto DevOps is enabled and the Auto DevOps settings are saved. This Deploy Token +can be used for permanent access to the registry. + +If the GitLab Deploy Token cannot be found, `CI_REGISTRY_PASSWORD` is +used. Note that `CI_REGISTRY_PASSWORD` is only valid during deployment. +This means that Kubernetes will be able to successfully pull the +container image during deployment but in cases where the image needs to +be pulled again, e.g. after pod eviction, Kubernetes will fail to do so +as it will be attempting to fetch the image using +`CI_REGISTRY_PASSWORD`. + +NOTE: **Note:** +When the GitLab Deploy Token has been manually revoked, it won't be automatically created. #### Migrations @@ -540,25 +582,17 @@ Note that a post-install hook means that if any deploy succeeds, If present, `DB_MIGRATE` will be run as a shell command within an application pod as a helm pre-upgrade hook. -For example, in a Rails application: +For example, in a Rails application in an image built with +[Herokuish](https://github.com/gliderlabs/herokuish): -- `DB_INITIALIZE` can be set to `cd /app && RAILS_ENV=production - bin/setup` -- `DB_MIGRATE` can be set to `cd /app && RAILS_ENV=production bin/update` +- `DB_INITIALIZE` can be set to `RAILS_ENV=production /bin/herokuish procfile exec bin/rails db:setup` +- `DB_MIGRATE` can be set to `RAILS_ENV=production /bin/herokuish procfile exec bin/rails db:migrate` NOTE: **Note:** -The `/app` path is the directory of your project inside the docker image -as [configured by -Herokuish](https://github.com/gliderlabs/herokuish#paths) - -> [Introduced][ce-19507] in GitLab 11.0. - -For internal and private projects a [GitLab Deploy Token](../../user/project/deploy_tokens/index.md#gitlab-deploy-token) -will be automatically created, when Auto DevOps is enabled and the Auto DevOps settings are saved. This Deploy Token -can be used for permanent access to the registry. - -Note: **Note** -When the GitLab Deploy Token has been manually revoked, it won't be automatically created. +Unless you have a `Dockerfile` in your repo, your image is built with +Herokuish. You must prefix commands run in these images with `/bin/herokuish +procfile exec` in order to replicate the the environment your application is +run in. ### Auto Monitoring @@ -632,15 +666,15 @@ repo or by specifying a project variable: - **Bundled chart** - If your project has a `./chart` directory with a `Chart.yaml` file in it, Auto DevOps will detect the chart and use it instead of the [default - one](https://gitlab.com/charts/auto-deploy-app). + one](https://gitlab.com/gitlab-org/charts/auto-deploy-app). This can be a great way to control exactly how your application is deployed. -- **Project variable** - Create a [project variable](../../ci/variables/README.md#variables) +- **Project variable** - Create a [project variable](../../ci/variables/README.md#gitlab-cicd-environment-variables) `AUTO_DEVOPS_CHART` with the URL of a custom chart to use or create two project variables `AUTO_DEVOPS_CHART_REPOSITORY` with the URL of a custom chart repository and `AUTO_DEVOPS_CHART` with the path to the chart. ### Custom Helm chart per environment **[PREMIUM]** -You can specify the use of a custom Helm chart per environment by scoping the environment variable -to the desired environment. See [Limiting environment scopes of variables](https://docs.gitlab.com/ee/ci/variables/#limiting-environment-scopes-of-variables-premium). +You can specify the use of a custom Helm chart per environment by scoping the environment variable +to the desired environment. See [Limiting environment scopes of variables](../../ci/variables/README.md#limiting-environment-scopes-of-environment-variables-premium). ### Customizing `.gitlab-ci.yml` @@ -664,6 +698,21 @@ renaming `.staging` to `staging`. Then make sure to uncomment the `when` key of the `production` job to turn it into a manual action instead of deploying automatically. +### Using components of Auto-DevOps + +If you only require a subset of the features offered by Auto-DevOps, you can include +individual Auto-DevOps jobs into your own `.gitlab-ci.yml`. + +For example, to make use of [Auto Build](#auto-build), you can add the following to +your `.gitlab-ci.yml`: + +```yaml +include: + - template: Jobs/Build.gitlab-ci.yml +``` + +Consult the [Auto DevOps template] for information on available jobs. + ### PostgreSQL database support In order to support applications that require a database, @@ -684,20 +733,22 @@ also be customized, and you can easily use a [custom buildpack](#custom-buildpac | **Variable** | **Description** | | ------------ | --------------- | -| `AUTO_DEVOPS_DOMAIN` | The [Auto DevOps domain](#auto-devops-base-domain). By default, set automatically by the [Auto DevOps setting](#enablingdisabling-auto-devops). This variable is deprecated and [is scheduled to be removed](https://gitlab.com/gitlab-org/gitlab-ce/issues/56959). Use `KUBE_INGRESS_BASE_DOMAIN` instead. | -| `AUTO_DEVOPS_CHART` | The Helm Chart used to deploy your apps; defaults to the one [provided by GitLab](https://gitlab.com/charts/auto-deploy-app). | +| `AUTO_DEVOPS_CHART` | The Helm Chart used to deploy your apps; defaults to the one [provided by GitLab](https://gitlab.com/gitlab-org/charts/auto-deploy-app). | | `AUTO_DEVOPS_CHART_REPOSITORY` | The Helm Chart repository used to search for charts; defaults to `https://charts.gitlab.io`. | +| `AUTO_DEVOPS_CHART_REPOSITORY_NAME` | From Gitlab 11.11, this variable can be used to set the name of the helm repository; defaults to "gitlab" | +| `AUTO_DEVOPS_CHART_REPOSITORY_USERNAME` | From Gitlab 11.11, this variable can be used to set a username to connect to the helm repository. Defaults to no credentials. (Also set AUTO_DEVOPS_CHART_REPOSITORY_PASSWORD) | +| `AUTO_DEVOPS_CHART_REPOSITORY_PASSWORD` | From Gitlab 11.11, this variable can be used to set a password to connect to the helm repository. Defaults to no credentials. (Also set AUTO_DEVOPS_CHART_REPOSITORY_USERNAME) | | `REPLICAS` | The number of replicas to deploy; defaults to 1. | | `PRODUCTION_REPLICAS` | The number of replicas to deploy in the production environment. This takes precedence over `REPLICAS`; defaults to 1. | -| `CANARY_REPLICAS` | The number of canary replicas to deploy for [Canary Deployments](https://docs.gitlab.com/ee/user/project/canary_deployments.html); defaults to 1 | -| `CANARY_PRODUCTION_REPLICAS` | The number of canary replicas to deploy for [Canary Deployments](https://docs.gitlab.com/ee/user/project/canary_deployments.html) in the production environment. This takes precedence over `CANARY_REPLICAS`; defaults to 1 | +| `CANARY_REPLICAS` | The number of canary replicas to deploy for [Canary Deployments](../../user/project/canary_deployments.md); defaults to 1 | +| `CANARY_PRODUCTION_REPLICAS` | The number of canary replicas to deploy for [Canary Deployments](../../user/project/canary_deployments.md) in the production environment. This takes precedence over `CANARY_REPLICAS`; defaults to 1 | | `ADDITIONAL_HOSTS` | Fully qualified domain names specified as a comma-separated list that are added to the ingress hosts. | | `<ENVIRONMENT>_ADDITIONAL_HOSTS` | For a specific environment, the fully qualified domain names specified as a comma-separated list that are added to the ingress hosts. This takes precedence over `ADDITIONAL_HOSTS`. | | `POSTGRES_ENABLED` | Whether PostgreSQL is enabled; defaults to `"true"`. Set to `false` to disable the automatic deployment of PostgreSQL. | | `POSTGRES_USER` | The PostgreSQL user; defaults to `user`. Set it to use a custom username. | | `POSTGRES_PASSWORD` | The PostgreSQL password; defaults to `testing-password`. Set it to use a custom password. | | `POSTGRES_DB` | The PostgreSQL database name; defaults to the value of [`$CI_ENVIRONMENT_SLUG`](../../ci/variables/README.md#predefined-environment-variables). Set it to use a custom database name. | -| `POSTGRES_VERSION` | The PostgreSQL version; defaults to `9.6.2` | +| `POSTGRES_VERSION` | Tag for the [`postgres` Docker image](https://hub.docker.com/_/postgres) to use. Defaults to `9.6.2`. | | `BUILDPACK_URL` | The buildpack's full URL. It can point to either Git repositories or a tarball URL. For Git repositories, it is possible to point to a specific `ref`, for example `https://github.com/heroku/heroku-buildpack-ruby.git#v142` | | `SAST_CONFIDENCE_LEVEL` | The minimum confidence level of security issues you want to be reported; `1` for Low, `2` for Medium, `3` for High; defaults to `3`.| | `DEP_SCAN_DISABLE_REMOTE_CHECKS` | Whether remote Dependency Scanning checks are disabled; defaults to `"false"`. Set to `"true"` to disable checks that send data to GitLab central servers. [Read more about remote checks](https://gitlab.com/gitlab-org/security-products/dependency-scanning#remote-checks).| @@ -708,6 +759,7 @@ also be customized, and you can easily use a [custom buildpack](#custom-buildpac | `INCREMENTAL_ROLLOUT_MODE`| From GitLab 11.4, this variable, if present, can be used to enable an [incremental rollout](#incremental-rollout-to-production-premium) of your application for the production environment.<br/>Set to: <ul><li>`manual`, for manual deployment jobs.</li><li>`timed`, for automatic rollout deployments with a 5 minute delay each one.</li></ul> | | `TEST_DISABLED` | From GitLab 11.0, this variable can be used to disable the `test` job. If the variable is present, the job will not be created. | | `CODE_QUALITY_DISABLED` | From GitLab 11.0, this variable can be used to disable the `codequality` job. If the variable is present, the job will not be created. | +| `LICENSE_MANAGEMENT_DISABLED` | From GitLab 11.0, this variable can be used to disable the `license_management` job. If the variable is present, the job will not be created. | | `SAST_DISABLED` | From GitLab 11.0, this variable can be used to disable the `sast` job. If the variable is present, the job will not be created. | | `DEPENDENCY_SCANNING_DISABLED` | From GitLab 11.0, this variable can be used to disable the `dependency_scanning` job. If the variable is present, the job will not be created. | | `CONTAINER_SCANNING_DISABLED` | From GitLab 11.0, this variable can be used to disable the `sast:container` job. If the variable is present, the job will not be created. | @@ -716,10 +768,13 @@ also be customized, and you can easily use a [custom buildpack](#custom-buildpac | `PERFORMANCE_DISABLED` | From GitLab 11.0, this variable can be used to disable the `performance` job. If the variable is present, the job will not be created. | | `K8S_SECRET_*` | From GitLab 11.7, any variable prefixed with [`K8S_SECRET_`](#application-secret-variables) will be made available by Auto DevOps as environment variables to the deployed application. | | `KUBE_INGRESS_BASE_DOMAIN` | From GitLab 11.8, this variable can be used to set a domain per cluster. See [cluster domains](../../user/project/clusters/index.md#base-domain) for more information. | +| `ROLLOUT_RESOURCE_TYPE` | From GitLab 11.9, this variable allows specification of the resource type being deployed when using a custom helm chart. Default value is `deployment`. | +| `ROLLOUT_STATUS_DISABLED` | From GitLab 12.0, this variable allows to disable rollout status check because it doesn't support all resource types, for example, `cronjob`. | +| `HELM_UPGRADE_EXTRA_ARGS` | From GitLab 11.11, this variable allows extra arguments in `helm` commands when deploying the application. Note that using quotes will not prevent word splitting. | TIP: **Tip:** Set up the replica variables using a -[project variable](../../ci/variables/README.md#variables) +[project variable](../../ci/variables/README.md#gitlab-cicd-environment-variables) and scale your application by just redeploying it! CAUTION: **Caution:** @@ -863,7 +918,7 @@ you when you're ready to manually deploy to production. > [Introduced](https://gitlab.com/gitlab-org/gitlab-ci-yml/merge_requests/171) in GitLab 11.0. -A [canary environment](https://docs.gitlab.com/ee/user/project/canary_deployments.html) can be used +A [canary environment](../../user/project/canary_deployments.md) can be used before any changes are deployed to production. If `CANARY_ENABLED` is defined in your project (e.g., set `CANARY_ENABLED` to @@ -887,7 +942,7 @@ increasing the rollout up to 100%. If `INCREMENTAL_ROLLOUT_MODE` is set to `manual` in your project, then instead of the standard `production` job, 4 different -[manual jobs](../../ci/pipelines.md#manual-actions-from-the-pipeline-graph) +[manual jobs](../../ci/pipelines.md#manual-actions-from-pipeline-graphs) will be created: 1. `rollout 10%` @@ -904,7 +959,7 @@ required to go from `10%` to `100%`, you can jump to whatever job you want. You can also scale down by running a lower percentage job, just before hitting `100%`. Once you get to `100%`, you cannot scale down, and you'd have to roll back by redeploying the old version using the -[rollback button](../../ci/environments.md#rolling-back-changes) in the +[rollback button](../../ci/environments.md#retrying-and-rolling-back) in the environment page. Below, you can see how the pipeline will look if the rollout or staging @@ -1008,6 +1063,9 @@ planned for a subsequent release. buildpack](#custom-buildpacks). - Auto Test may fail because of a mismatch between testing frameworks. In this case, you may need to customize your `.gitlab-ci.yml` with your test commands. +- Auto Deploy will fail if GitLab can not create a Kubernetes namespace and + service account for your project. For help debugging this issue, see + [Troubleshooting failed deployment jobs](../../user/project/clusters/index.md#troubleshooting-failed-deployment-jobs). ### Disable the banner instance wide @@ -1040,3 +1098,7 @@ curl --data "value=true" --header "PRIVATE-TOKEN: personal_access_token" https:/ [ee]: https://about.gitlab.com/pricing/ [ce-21955]: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/21955 [ce-19507]: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/19507 + +## Development guides + +Configuring [GDK for Auto DevOps](https://gitlab.com/gitlab-org/gitlab-development-kit/blob/master/doc/howto/auto_devops.md). diff --git a/doc/topics/autodevops/quick_start_guide.md b/doc/topics/autodevops/quick_start_guide.md index 367e192b85a..cc83d20d65a 100644 --- a/doc/topics/autodevops/quick_start_guide.md +++ b/doc/topics/autodevops/quick_start_guide.md @@ -208,7 +208,7 @@ applications. In the rightmost column for the production environment, you can ma application is running. Right below, there is the -[Deploy Board](https://docs.gitlab.com/ee/user/project/deploy_boards.html). +[Deploy Board](../../user/project/deploy_boards.md). The squares represent pods in your Kubernetes cluster that are associated with the given environment. Hovering above each square you can see the state of a deployment and clicking a square will take you to the pod's logs page. diff --git a/doc/topics/git/how_to_install_git/index.md b/doc/topics/git/how_to_install_git/index.md index d7e1979217e..c7bede2d269 100644 --- a/doc/topics/git/how_to_install_git/index.md +++ b/doc/topics/git/how_to_install_git/index.md @@ -5,14 +5,20 @@ level: beginner article_type: user guide date: 2017-05-15 description: 'This article describes how to install Git on macOS, Ubuntu Linux and Windows.' +type: howto +last_updated: 2019-05-31 --- # Installing Git -To begin contributing to GitLab projects +To begin contributing to GitLab projects, you will need to install the Git client on your computer. + This article will show you how to install Git on macOS, Ubuntu Linux and Windows. +Information on [installing Git](https://git-scm.com/book/en/v2/Getting-Started-Installing-Git) +is also available at the official Git website. + ## Install Git on macOS using the Homebrew package manager Although it is easy to use the version of Git shipped with macOS @@ -21,7 +27,7 @@ we recommend installing it via Homebrew to get access to an extensive selection of dependency managed libraries and applications. If you are sure you don't need access to any additional development libraries -or don't have approximately 15gb of available disk space for Xcode and Homebrew +or don't have approximately 15gb of available disk space for Xcode and Homebrew, use one of the aforementioned methods. ### Installing Xcode @@ -40,11 +46,12 @@ for the official Homebrew installation instructions. With Homebrew installed you are now ready to install Git. Open a Terminal and enter in the following command: -```bash +```sh brew install git ``` Congratulations you should now have Git installed via Homebrew. + Next read our article on [adding an SSH key to GitLab](../../../ssh/README.md). ## Install Git on Ubuntu Linux @@ -55,16 +62,30 @@ it is recommended to use the built in package manager to install Git. Open a Terminal and enter in the following commands to install the latest Git from the official Git maintained package archives: -```bash +```sh sudo apt-add-repository ppa:git-core/ppa sudo apt-get update sudo apt-get install git ``` Congratulations you should now have Git installed via the Ubuntu package manager. + Next read our article on [adding an SSH key to GitLab](../../../ssh/README.md). ## Installing Git on Windows from the Git website Browse to the [Git website](https://git-scm.com/) and download and install Git for Windows. + Next read our article on [adding an SSH key to GitLab](../../../ssh/README.md). + +<!-- ## Troubleshooting + +Include any troubleshooting steps that you can foresee. If you know beforehand what issues +one might have when setting this up, or when something is changed, or on upgrading, it's +important to describe those, too. Think of things that may go wrong and include them here. +This is important to minimize requests for support, and to avoid doc comments with +questions that you know someone might ask. + +Each scenario can be a third-level heading, e.g. `### Getting error message X`. +If you have none to add when creating a doc, leave this section in place +but commented out to help encourage others to add to it in the future. --> diff --git a/doc/topics/git/index.md b/doc/topics/git/index.md index 7707d56764e..841746cc5de 100644 --- a/doc/topics/git/index.md +++ b/doc/topics/git/index.md @@ -1,8 +1,12 @@ -# Git documentation +--- +type: index +--- + +# Git Git is a [free and open source](https://git-scm.com/about/free-and-open-source) distributed version control system designed to handle everything from small to -very large projects with speed and efficiency. +large projects with speed and efficiency. [GitLab](https://about.gitlab.com) is a Git-based fully integrated platform for software development. Besides Git's functionalities, GitLab has a lot of @@ -11,64 +15,71 @@ powerful [features](https://about.gitlab.com/features/) to enhance your We've gathered some resources to help you to get the best from Git with GitLab. +More information is also available on the [Git website](https://git-scm.com). + ## Getting started -- [Git concepts](../../university/training/user_training.md#git-concepts) +The following resources will help you get started with Git: + +- [Git Basics](https://git-scm.com/book/en/v2/Getting-Started-Git-Basics) +- [Git on the Server - GitLab](https://git-scm.com/book/en/v2/Git-on-the-Server-GitLab) - [How to install Git](how_to_install_git/index.md) - [Start using Git on the command line](../../gitlab-basics/start-using-git.md) - [Command Line basic commands](../../gitlab-basics/command-line-commands.md) - [GitLab Git Cheat Sheet (download)](https://about.gitlab.com/images/press/git-cheat-sheet.pdf) -- Commits - - [Revert a commit](../../user/project/merge_requests/revert_changes.md#reverting-a-commit) +- Commits: + - [Revert a commit](../../user/project/merge_requests/revert_changes.md#reverting-a-commit) - [Cherry-picking a commit](../../user/project/merge_requests/cherry_pick_changes.md#cherry-picking-a-commit) - [Squashing commits](../../workflow/gitlab_flow.md#squashing-commits-with-rebase) -**Third-party references:** +### Concepts -- [Getting Started - Git website](https://git-scm.com) -- [Getting Started - Version control](https://git-scm.com/book/en/v2/Getting-Started-About-Version-Control) -- [Getting Started - Git Basics](https://git-scm.com/book/en/v2/Getting-Started-Git-Basics) -- [Getting Started - Installing Git](https://git-scm.com/book/en/v2/Getting-Started-Installing-Git) -- [Git on the Server - GitLab](https://git-scm.com/book/en/v2/Git-on-the-Server-GitLab) +The following are resources about version control concepts: -### Concepts +- [Git concepts](../../university/training/user_training.md#git-concepts) +- [Why Git is Worth the Learning Curve](https://about.gitlab.com/2017/05/17/learning-curve-is-the-biggest-challenge-developers-face-with-git/) +- [The future of SaaS hosted Git repository pricing](https://about.gitlab.com/2016/05/11/git-repository-pricing/) +- [Git website topic about version control](https://git-scm.com/book/en/v2/Getting-Started-About-Version-Control) +- [GitLab University presentation about Version Control](https://docs.google.com/presentation/d/16sX7hUrCZyOFbpvnrAFrg6tVO5_yT98IgdAqOmXwBho/edit?usp=sharing) -- Article (2017-05-17): [Why Git is Worth the Learning Curve](https://about.gitlab.com/2017/05/17/learning-curve-is-the-biggest-challenge-developers-face-with-git/) -- Article (2016-05-11): [The future of SaaS hosted Git repository pricing](https://about.gitlab.com/2016/05/11/git-repository-pricing/) -- GLU Course (Presentation): [About Version Control](https://docs.google.com/presentation/d/16sX7hUrCZyOFbpvnrAFrg6tVO5_yT98IgdAqOmXwBho/edit?usp=sharing) +## Git tips -## Exploring Git +The following resources may help you become more efficient at using Git: - [Git Tips & Tricks](https://about.gitlab.com/2016/12/08/git-tips-and-tricks/) - [Eight Tips to help you work better with Git](https://about.gitlab.com/2015/02/19/8-tips-to-help-you-work-better-with-git/) ## Troubleshooting Git +If you have problems with Git, the following may help: + - [Numerous _undo_ possibilities in Git](numerous_undo_possibilities_in_git/index.md) -- Learn a few [Git troubleshooting](troubleshooting_git.md) techniques to help you out. +- Learn a few [Git troubleshooting](troubleshooting_git.md) techniques ## Branching strategies -- [GitLab Flow](https://about.gitlab.com/2014/09/29/gitlab-flow/) - -**Third-party references:** - - [Git Branching - Branches in a Nutshell](https://git-scm.com/book/en/v2/Git-Branching-Branches-in-a-Nutshell) - [Git Branching - Branching Workflows](https://git-scm.com/book/en/v2/Git-Branching-Branching-Workflows) +- [GitLab Flow](https://about.gitlab.com/2014/09/29/gitlab-flow/) ## Advanced use +The following are advanced topics for those who want to get the most out of Git: + - [Custom Git Hooks](../../administration/custom_hooks.md) - [Git Attributes](../../user/project/git_attributes.md) - Git Submodules: [Using Git submodules with GitLab CI](../../ci/git_submodules.md#using-git-submodules-with-gitlab-ci) ## API -- [Gitignore templates](../../api/templates/gitignores.md) +[Gitignore templates](../../api/templates/gitignores.md) API allow for +Git-related queries from GitLab. ## Git LFS +The following relate to Git Large File Storage: + - [Getting Started with Git LFS](https://about.gitlab.com/2017/01/30/getting-started-with-git-lfs-tutorial/) - [GitLab Git LFS documentation](../../workflow/lfs/manage_large_binaries_with_git_lfs.md) -- [Git-Annex to Git-LFS migration guide](https://docs.gitlab.com/ee/workflow/lfs/migrate_from_git_annex_to_git_lfs.html) -- Article (2015-08-13): [Towards a production quality open source Git LFS server](https://about.gitlab.com/2015/08/13/towards-a-production-quality-open-source-git-lfs-server/) +- [Git-Annex to Git-LFS migration guide](../../workflow/lfs/migrate_from_git_annex_to_git_lfs.md) +- [Towards a production quality open source Git LFS server](https://about.gitlab.com/2015/08/13/towards-a-production-quality-open-source-git-lfs-server/) diff --git a/doc/topics/git/numerous_undo_possibilities_in_git/index.md b/doc/topics/git/numerous_undo_possibilities_in_git/index.md index 8a8021dc36d..84201e11831 100644 --- a/doc/topics/git/numerous_undo_possibilities_in_git/index.md +++ b/doc/topics/git/numerous_undo_possibilities_in_git/index.md @@ -4,25 +4,30 @@ author_gitlab: Letme level: intermediary article_type: tutorial date: 2017-05-15 +type: howto +last_updated: 2019-05-31 --- # Numerous undo possibilities in Git -## Introduction - In this tutorial, we will show you different ways of undoing your work in Git, for which we will assume you have a basic working knowledge of. Check GitLab's -[Git documentation](../index.md#git-documentation) for reference. +[Git documentation](../index.md) for reference. + Also, we will only provide some general info of the commands, which is enough -to get you started for the easy cases/examples, but for anything more advanced please refer to the [Git book](https://git-scm.com/book/en/v2). +to get you started for the easy cases/examples, but for anything more advanced +please refer to the [Git book](https://git-scm.com/book/en/v2). We will explain a few different techniques to undo your changes based on the stage of the change in your current development. Also, keep in mind that [nothing in -Git is really deleted.][git-autoclean-ref] +Git is really deleted][git-autoclean-ref]. + This means that until Git automatically cleans detached commits (which cannot be accessed by branch or tag) it will be possible to view them with `git reflog` command and access them with direct commit-id. Read more about _[redoing the undo](#redoing-the-undo)_ on the section below. +## Introduction + This guide is organized depending on the [stage of development][git-basics] where you want to undo your changes from and if they were shared with other developers or not. Because Git is tracking changes a created or edited file is in the unstaged state @@ -31,35 +36,41 @@ a file into the **staged** state, which is then committed (`git commit`) to your local repository. After that, file can be shared with other developers (`git push`). Here's what we'll cover in this tutorial: - - [Undo local changes](#undo-local-changes) which were not pushed to remote repository +- [Undo local changes](#undo-local-changes) which were not pushed to remote repository: - - Before you commit, in both unstaged and staged state - - After you committed + - Before you commit, in both unstaged and staged state. + - After you committed. - - Undo changes after they are pushed to remote repository +- Undo changes after they are pushed to remote repository: - - [Without history modification](#undo-remote-changes-without-changing-history) (preferred way) - - [With history modification](#undo-remote-changes-with-modifying-history) (requires - coordination with team and force pushes). - - [Usecases when modifying history is generally acceptable](#where-modifying-history-is-generally-acceptable) - - [How to modify history](#how-modifying-history-is-done) - - [How to remove sensitive information from repository](#deleting-sensitive-information-from-commits) + - [Without history modification](#undo-remote-changes-without-changing-history) (preferred way). + - [With history modification](#undo-remote-changes-with-modifying-history) (requires + coordination with team and force pushes). + - [Use cases when modifying history is generally acceptable](#where-modifying-history-is-generally-acceptable). + - [How to modify history](#how-modifying-history-is-done). + - [How to remove sensitive information from repository](#deleting-sensitive-information-from-commits). ### Branching strategy [Git][git-official] is a de-centralized version control system, which means that beside regular versioning of the whole repository, it has possibilities to exchange changes -with other repositories. To avoid chaos with +with other repositories. + +To avoid chaos with [multiple sources of truth][git-distributed], various development workflows have to be followed, and it depends on your internal workflow how certain changes or commits can be undone or changed. + [GitLab Flow][gitlab-flow] provides a good balance between developers clashing with each other while developing the same feature and cooperating seamlessly, but it does not enable joined development of the same feature by multiple developers by default. + When multiple developers develop the same feature on the same branch, clashing with every synchronization is unavoidable, but a proper or chosen Git Workflow will -prevent that anything is lost or out of sync when feature is complete. You can also +prevent that anything is lost or out of sync when feature is complete. + +You can also read through this blog post on [Git Tips & Tricks][gitlab-git-tips-n-tricks] to learn how to easily **do** things in Git. @@ -97,19 +108,19 @@ no changes added to commit (use "git add" and/or "git commit -a") At this point there are 3 options to undo the local changes you have: -- Discard all local changes, but save them for possible re-use [later](#quickly-save-local-changes) +- Discard all local changes, but save them for possible re-use [later](#quickly-save-local-changes): ```shell git stash ``` -- Discarding local changes (permanently) to a file +- Discarding local changes (permanently) to a file: ```shell git checkout -- <file> ``` -- Discard all local changes to all files permanently +- Discard all local changes to all files permanently: ```shell git reset --hard @@ -150,7 +161,7 @@ of the staging tree. You also have an option to discard all changes with Lets start the example by editing a file, with your favorite editor, to change the content and add it to staging -``` +```sh vim <file> git add <file> ``` @@ -164,30 +175,30 @@ Your branch is up-to-date with 'origin/master'. Changes to be committed: (use "git reset HEAD <file>..." to unstage) - new file: <file> + new file: <file> ``` Now you have 4 options to undo your changes: -- Unstage the file to current commit (HEAD) +- Unstage the file to current commit (HEAD): ```shell git reset HEAD <file> ``` -- Unstage everything - retain changes +- Unstage everything - retain changes: ```shell git reset ``` -- Discard all local changes, but save them for [later](#quickly-save-local-changes) +- Discard all local changes, but save them for [later](#quickly-save-local-changes): ```shell git stash ``` -- Discard everything permanently +- Discard everything permanently: ```shell git reset --hard @@ -206,7 +217,9 @@ your code, you'll have less options to troubleshoot your work. Through the development process some of the previously committed changes do not fit anymore in the end solution, or are source of the bugs. Once you find the commit which triggered bug, or once you have a faulty commit, you can simply -revert it with `git revert commit-id`. This command inverts (swaps) the additions and +revert it with `git revert commit-id`. + +This command inverts (swaps) the additions and deletions in that commit, so that it does not modify history. Retaining history can be helpful in future to notice that some changes have been tried unsuccessfully in the past. @@ -225,19 +238,19 @@ through simple bisection process. You can read more about it [in official Git To In our example we will end up with commit `B`, that introduced bug/error. We have 4 options on how to remove it (or part of it) from our repository. -- Undo (swap additions and deletions) changes introduced by commit `B`. +- Undo (swap additions and deletions) changes introduced by commit `B`: ```shell git revert commit-B-id ``` -- Undo changes on a single file or directory from commit `B`, but retain them in the staged state +- Undo changes on a single file or directory from commit `B`, but retain them in the staged state: ```shell git checkout commit-B-id <file> ``` -- Undo changes on a single file or directory from commit `B`, but retain them in the unstaged state +- Undo changes on a single file or directory from commit `B`, but retain them in the unstaged state: ```shell git reset commit-B-id <file> @@ -246,7 +259,9 @@ In our example we will end up with commit `B`, that introduced bug/error. We hav - There is one command we also must not forget: **creating a new branch** from the point where changes are not applicable or where the development has hit a dead end. For example you have done commits `A-B-C-D` on your feature-branch - and then you figure `C` and `D` are wrong. At this point you either reset to `B` + and then you figure `C` and `D` are wrong. + + At this point you either reset to `B` and do commit `F` (which will cause problems with pushing and if forced pushed also with other developers) since branch now looks `A-B-F`, which clashes with what other developers have locally (you will [change history](#with-history-modification)), or you simply checkout commit `B` create @@ -269,13 +284,13 @@ In our example we will end up with commit `B`, that introduced bug/error. We hav There is one command for history modification and that is `git rebase`. Command provides interactive mode (`-i` flag) which enables you to: - - **reword** commit messages (there is also `git commit --amend` for editing - last commit message) - - **edit** the commit content (changes introduced by commit) and message - - **squash** multiple commits into a single one, and have a custom or aggregated - commit message - - **drop** commits - simply delete them - - and few more options +- **reword** commit messages (there is also `git commit --amend` for editing + last commit message). +- **edit** the commit content (changes introduced by commit) and message. +- **squash** multiple commits into a single one, and have a custom or aggregated + commit message. +- **drop** commits - simply delete them. +- and few more options. Let us check few examples. Again there are commits `A-B-C-D` where you want to delete commit `B`. @@ -301,7 +316,7 @@ In case you want to modify something introduced in commit `B`. - Command opens your favorite text editor where you write `edit` in front of commit `B`, but leave default `pick` with all other commits. Save and exit the editor to - perform a rebase + perform a rebase. - Now do your edits and commit changes: @@ -348,7 +363,9 @@ and then on end description of that action. This topic is roughly same as modifying committed local changes without modifying history. **It should be the preferred way of undoing changes on any remote repository or public branch.** Keep in mind that branching is the best solution when you want -to retain the history of faulty development, yet start anew from certain point. Branching +to retain the history of faulty development, yet start anew from certain point. + +Branching enables you to include the existing changes in new development (by merging) and it also provides a clear timeline and development structure. @@ -386,12 +403,14 @@ the cleanup of detached commits (happens automatically). Modified history breaks the development chain of other developers, as changed history does not have matching commits'ids. For that reason it should not be used on any public branch or on branch that *might* be used by other -developers. When contributing to big open source repositories (e.g. [GitLab CE][gitlab-ce]), +developers. When contributing to big open source repositories (for example, [GitLab CE][gitlab-ce]), it is acceptable to *squash* commits into a single one, to present a nicer history of your contribution. + Keep in mind that this also removes the comments attached to certain commits in merge requests, so if you need to retain traceability in GitLab, then modifying history is not acceptable. + A feature-branch of a merge request is a public branch and might be used by other developers, but project process and rules might allow or require you to use `git rebase` (command that changes history) to reduce number of @@ -400,8 +419,8 @@ GitLab). There is a `git merge --squash` command which does exactly that (squashes commits on feature-branch to a single commit on target branch at merge). ->**Note:** -Never modify the commit history of `master` or shared branch +NOTE: **Note:** +Never modify the commit history of `master` or shared branch. ### How modifying history is done @@ -436,7 +455,7 @@ pick <commit3-id> <commit3-commit-message> # Note that empty commits are commented out ``` ->**Note:** +NOTE: **Note:** It is important to notice that comment from the output clearly states that, if you decide to abort, then do not just close your editor (as that will in-fact modify history), but remove all uncommented lines and save. @@ -470,7 +489,7 @@ tools that can use some of Git specifics to enable faster execution of common tasks (which is exactly what removing sensitive information file is about). An alternative is [BFG Repo-cleaner][bfg-repo-cleaner]. Keep in mind that these tools are faster because they do not provide a same fully feature set as `git filter-branch` -does, but focus on specific usecases. +does, but focus on specific use cases. ## Conclusion @@ -480,6 +499,18 @@ depending on the stage of your process. Git also enables rewriting history, but should be avoided as it might cause problems when multiple developers are contributing to the same codebase. +<!-- ## Troubleshooting + +Include any troubleshooting steps that you can foresee. If you know beforehand what issues +one might have when setting this up, or when something is changed, or on upgrading, it's +important to describe those, too. Think of things that may go wrong and include them here. +This is important to minimize requests for support, and to avoid doc comments with +questions that you know someone might ask. + +Each scenario can be a third-level heading, e.g. `### Getting error message X`. +If you have none to add when creating a doc, leave this section in place +but commented out to help encourage others to add to it in the future. --> + <!-- Identifiers, in alphabetical order --> [bfg-repo-cleaner]: https://rtyley.github.io/bfg-repo-cleaner/ diff --git a/doc/topics/git/troubleshooting_git.md b/doc/topics/git/troubleshooting_git.md index d1729d70158..417d91bf834 100644 --- a/doc/topics/git/troubleshooting_git.md +++ b/doc/topics/git/troubleshooting_git.md @@ -1,3 +1,7 @@ +--- +type: howto +--- + # Troubleshooting Git Sometimes things don't work the way they should or as you might expect when @@ -9,7 +13,7 @@ with Git. 'Broken pipe' errors can occur when attempting to push to a remote repository. When pushing you will usually see: -``` +```text Write failed: Broken pipe fatal: The remote end hung up unexpectedly ``` @@ -39,14 +43,15 @@ There's another option where you can prevent session timeouts by configuring SSH 'keep alive' either on the client or on the server (if you are a GitLab admin and have access to the server). -NOTE: **Note:** configuring *both* the client and the server is unnecessary. +NOTE: **Note:** +Configuring *both* the client and the server is unnecessary. **To configure SSH on the client side**: -- On UNIX, edit `~/.ssh/config` (create the file if it doesn’t exist) and - add or edit: +- On UNIX, edit `~/.ssh/config` (create the file if it doesn’t exist) and + add or edit: - ``` + ```text Host your-gitlab-instance-url.com ServerAliveInterval 60 ServerAliveCountMax 5 @@ -58,7 +63,7 @@ NOTE: **Note:** configuring *both* the client and the server is unnecessary. **To configure SSH on the server side**, edit `/etc/ssh/sshd_config` and add: -``` +```text ClientAliveInterval 60 ClientAliveCountMax 5 ``` @@ -78,13 +83,40 @@ git push In case you're running an older version of Git (< 2.9), consider upgrading to >= 2.9 (see [Broken pipe when pushing to Git repository][Broken-Pipe]). +## `ssh_exchange_identification` error + +Users may experience the following error when attempting to push or pull +using Git over SSH: + +```text +Please make sure you have the correct access rights +and the repository exists. +... +ssh_exchange_identification: read: Connection reset by peer +fatal: Could not read from remote repository. +``` + +This error usually indicates that SSH daemon's `MaxStartups` value is throttling +SSH connections. This setting specifies the maximum number of unauthenticated +connections to the SSH daemon. This affects users with proper authentication +credentials (SSH keys) because every connection is 'unauthenticated' in the +beginning. The default value is `10`. + +Increase `MaxStartups` by adding or modifying the value in `/etc/ssh/sshd_config`: + +```text +MaxStartups 100 +``` + +Restart SSHD for the change to take effect. + ## Timeout during git push/pull If pulling/pushing from/to your repository ends up taking more than 50 seconds, -a timeout will be issued with a log of the number of operations performed +a timeout will be issued with a log of the number of operations performed and their respective timings, like the example below: -``` +```text remote: Running checks for branch: master remote: Scanning for LFS objects... (153ms) remote: Calculating new repository size... (cancelled after 729ms) diff --git a/doc/university/README.md b/doc/university/README.md index 3e7d02770e4..61ed72d25fb 100644 --- a/doc/university/README.md +++ b/doc/university/README.md @@ -4,12 +4,9 @@ comments: false # GitLab University -GitLab University is the best place to learn about **Version Control with Git and GitLab**. +GitLab University is a great place to start when learning about version control with Git and GitLab, as well as other GitLab features. -It doesn't replace, but accompanies our great [Documentation](https://docs.gitlab.com) -and [Blog Articles](https://about.gitlab.com/blog/). - -Would you like to contribute to GitLab University? Then please take a look at our contribution [process](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/PROCESS.md) for more information. +If you're looking for a GitLab subscription for _your university_, see our [Education](https://about.gitlab.com/solutions/education/) page. ## GitLab University Curriculum @@ -54,10 +51,10 @@ The curriculum is composed of GitLab videos, screencasts, presentations, project #### 1.5. Migrating from other Source Control -1. [Migrating from BitBucket/Stash](https://docs.gitlab.com/ee/user/project/import/bitbucket.html) -1. [Migrating from GitHub](https://docs.gitlab.com/ee/user/project/import/github.html) -1. [Migrating from SVN](https://docs.gitlab.com/ee/user/project/import/svn.html) -1. [Migrating from Fogbugz](https://docs.gitlab.com/ee/user/project/import/fogbugz.html) +1. [Migrating from BitBucket/Stash](../user/project/import/bitbucket.md) +1. [Migrating from GitHub](../user/project/import/github.md) +1. [Migrating from SVN](../user/project/import/svn.md) +1. [Migrating from Fogbugz](../user/project/import/fogbugz.md) #### 1.6. GitLab Inc. @@ -95,7 +92,7 @@ The curriculum is composed of GitLab videos, screencasts, presentations, project 1. [Using any Static Site Generator with GitLab Pages](https://about.gitlab.com/2016/06/17/ssg-overview-gitlab-pages-part-3-examples-ci/) 1. [Securing GitLab Pages with SSL](https://about.gitlab.com/2016/06/24/secure-gitlab-pages-with-startssl/) -1. [GitLab Pages Documentation](https://docs.gitlab.com/ce/user/project/pages/) +1. [GitLab Pages Documentation](../user/project/pages/index.md) #### 2.2. GitLab Issues @@ -134,7 +131,7 @@ The curriculum is composed of GitLab videos, screencasts, presentations, project 1. [GitLab Flow vs Forking in GitLab - Video](https://www.youtube.com/watch?v=UGotqAUACZA) 1. [GitLab Flow Overview](https://about.gitlab.com/2014/09/29/gitlab-flow/) 1. [Always Start with an Issue](https://about.gitlab.com/2016/03/03/start-with-an-issue/) -1. [GitLab Flow Documentation](https://docs.gitlab.com/ee/workflow/gitlab_flow.html) +1. [GitLab Flow Documentation](../workflow/gitlab_flow.md) #### 2.5. GitLab Comparisons @@ -194,10 +191,10 @@ The curriculum is composed of GitLab videos, screencasts, presentations, project #### 3.9. Integrations 1. [How to Integrate JIRA and Jenkins with GitLab - Video](https://gitlabmeetings.webex.com/gitlabmeetings/ldr.php?RCID=44b548147a67ab4d8a62274047146415) -1. [How to Integrate Jira with GitLab](https://docs.gitlab.com/ce/user/project/integrations/jira.html) -1. [How to Integrate Jenkins with GitLab](https://docs.gitlab.com/ee/integration/jenkins.html) -1. [How to Integrate Bamboo with GitLab](https://docs.gitlab.com/ce/user/project/integrations/bamboo.html) -1. [How to Integrate Slack with GitLab](https://docs.gitlab.com/ce/user/project/integrations/slack.html) +1. [How to Integrate Jira with GitLab](../user/project/integrations/jira.md) +1. [How to Integrate Jenkins with GitLab](../integration/jenkins.md) +1. [How to Integrate Bamboo with GitLab](../user/project/integrations/bamboo.md) +1. [How to Integrate Slack with GitLab](../user/project/integrations/slack.md) 1. [How to Integrate Convox with GitLab](https://about.gitlab.com/2016/06/09/continuous-delivery-with-gitlab-and-convox/) 1. [Getting Started with GitLab and Shippable CI](https://about.gitlab.com/2016/05/05/getting-started-gitlab-and-shippable/) diff --git a/doc/university/glossary/README.md b/doc/university/glossary/README.md index 254e234a22c..f15b0107de5 100644 --- a/doc/university/glossary/README.md +++ b/doc/university/glossary/README.md @@ -41,7 +41,7 @@ Objects (usually binary and large) created by a build process. These can include ### Atlassian -A [company](https://www.atlassian.com) that develops software products for developers and project managers including Bitbucket, Jira, Confluence, Bamboo. +A [company](https://www.atlassian.com) that develops software products for developers and project managers including Bitbucket, Jira, Hipchat, Confluence, Bamboo. ### Audit Log @@ -62,7 +62,7 @@ Entry level [subscription](https://about.gitlab.com/pricing/) for GitLab EE curr ### Bitbucket -Atlassian's web hosting service for Git and Mercurial Projects. Read about [migrating](https://docs.gitlab.com/ce/workflow/importing/import_projects_from_bitbucket.html) from BitBucket to a GitLab instance. +Atlassian's web hosting service for Git and Mercurial Projects. Read about [migrating](../../user/project/import/bitbucket.md) from BitBucket to a GitLab instance. ### Branch @@ -70,10 +70,10 @@ A branch is a parallel version of a repository. This allows you to work on the r ### Branded Login -Having your own logo on [your GitLab instance login page](https://docs.gitlab.com/ee/customization/branded_login_page.html) instead of the GitLab logo. +Having your own logo on [your GitLab instance login page](../../customization/branded_login_page.md) instead of the GitLab logo. ### Job triggers (Build Triggers) -These protect your code base against breaks, for instance when a team is working on the same project. Learn about [setting up](https://docs.gitlab.com/ce/ci/triggers/README.html) job triggers. +These protect your code base against breaks, for instance when a team is working on the same project. Learn about [setting up](../../ci/triggers/README.md) job triggers. ### CEPH @@ -149,7 +149,7 @@ As in "specify [dependencies](https://gitlab.com/gitlab-org/gitlab-ce/issues/147 ### Deploy Keys -A [SSH key](https://docs.gitlab.com/ce/gitlab-basics/create-your-ssh-keys.html)stored on your server that grants access to a single GitLab repository. This is used by a GitLab runner to clone a project's code so that tests can be run against the checked out code. +A [SSH key](../../gitlab-basics/create-your-ssh-keys.md)stored on your server that grants access to a single GitLab repository. This is used by a GitLab runner to clone a project's code so that tests can be run against the checked out code. ### Developer @@ -169,7 +169,7 @@ A folder used for storing multiple files. ### Docker Container Registry -A [feature](https://docs.gitlab.com/ce/user/project/container_registry.html) of [GitLab projects](https://about.gitlab.com/2016/05/23/gitlab-container-registry/). Containers wrap up a piece of software in a complete filesystem that contains everything it needs to run: code, runtime, system tools, system libraries – anything you can install on a server. This guarantees that it will always run the same, regardless of the environment it is running in. +A [feature](../../user/project/container_registry.md) of [GitLab projects](https://about.gitlab.com/2016/05/23/gitlab-container-registry/). Containers wrap up a piece of software in a complete filesystem that contains everything it needs to run: code, runtime, system tools, system libraries – anything you can install on a server. This guarantees that it will always run the same, regardless of the environment it is running in. ### Dynamic Environment (review apps) @@ -191,7 +191,7 @@ First Byte (sometimes referred to as time to first byte or [TTFB](https://en.wik ### Fork -Your [own copy](https://docs.gitlab.com/ce/workflow/forking_workflow.html) of a repository that allows you to make changes to the repository without affecting the original. +Your [own copy](../../workflow/forking_workflow.md) of a repository that allows you to make changes to the repository without affecting the original. ### Funnel, or: TOFU, MOFU, BOFU @@ -221,7 +221,7 @@ A single-tenant solution that provides GitLab CE or EE as a managed service. Git ### GitHub -A web-based Git repository hosting service with an enterprise offering. Its main features are: issue tracking, pull request with code review, abundancy of integrations and wiki. It offers free public repos, private repos and enterprise services are paid. Read about [importing a project](https://docs.gitlab.com/ce/workflow/importing/import_projects_from_github.html) from GitHub to GitLab. +A web-based Git repository hosting service with an enterprise offering. Its main features are: issue tracking, pull request with code review, abundancy of integrations and wiki. It offers free public repos, private repos and enterprise services are paid. Read about [importing a project](../../workflow/importing/import_projects_from_github.md) from GitHub to GitLab. ### GitLab CE @@ -241,7 +241,7 @@ Our free SaaS for public and private repositories. ### GitLab Geo -Allows you to replicate your GitLab instance to other geographical locations as a read-only fully operational version. It [can be used](https://docs.gitlab.com/ee/administration/geo/replication/index.html) for cloning and fetching projects, in addition to reading any data. This will make working with large repositories over large distances much faster. +Allows you to replicate your GitLab instance to other geographical locations as a read-only fully operational version. It [can be used](../../administration/geo/replication/index.md) for cloning and fetching projects, in addition to reading any data. This will make working with large repositories over large distances much faster. ### GitLab High Availability @@ -299,15 +299,15 @@ An [application layer protocol](http://www.irchelp.org/) that facilitates commun ### Issue Tracker -A [tool](https://docs.gitlab.com/ee/integration/external-issue-tracker.html) used to manage, organize, and maintain a list of issues, making it easier for an organization to manage. +A [tool](../../integration/external-issue-tracker.md) used to manage, organize, and maintain a list of issues, making it easier for an organization to manage. ### Jenkins -An Open Source CI tool written using the Java programming language. [Jenkins](https://jenkins.io/) does the same job as GitLab CI, Bamboo, and Travis CI. It is extremely popular. Related [documentation](https://docs.gitlab.com/ee/integration/jenkins.html). +An Open Source CI tool written using the Java programming language. [Jenkins](https://jenkins.io/) does the same job as GitLab CI, Bamboo, and Travis CI. It is extremely popular. Related [documentation](../../integration/jenkins.md). ### Jira -Atlassian's [project management software](https://www.atlassian.com/software/jira), i.e. a complex issue tracker. GitLab [can be configured](https://docs.gitlab.com/ee/project_services/jira.html) to interact with JIRA Core either using an on-premise instance or the SaaS solution that Atlassian offers. +Atlassian's [project management software](https://www.atlassian.com/software/jira), i.e. a complex issue tracker. GitLab [can be configured](../../project_services/jira.md) to interact with JIRA Core either using an on-premise instance or the SaaS solution that Atlassian offers. ### JUnit @@ -323,7 +323,7 @@ An open source container cluster manager originally designed by Google. It's bas ### Labels -An [identifier](https://docs.gitlab.com/ce/user/project/labels.html) to describe a group of one or more specific file revisions. +An [identifier](../../user/project/labels.md) to describe a group of one or more specific file revisions. ### Lightweight Directory Access Protocol (LDAP) @@ -331,7 +331,7 @@ An [identifier](https://docs.gitlab.com/ce/user/project/labels.html) to describe ### LDAP User Authentication -GitLab [integrates](https://docs.gitlab.com/ce/administration/auth/ldap.html) with LDAP to support user authentication. This enables GitLab to sign in people from an LDAP server (i.e., allowing people whose names are on the electronic user directory server to be able to use their LDAP accounts to login.) +GitLab [integrates](../../administration/auth/ldap.md) with LDAP to support user authentication. This enables GitLab to sign in people from an LDAP server (i.e., allowing people whose names are on the electronic user directory server to be able to use their LDAP accounts to login.) ### LDAP Group Sync @@ -381,9 +381,9 @@ Takes changes from one branch, and [applies them](https://git-scm.com/docs/git-m [Arises](https://about.gitlab.com/2016/09/06/resolving-merge-conflicts-from-the-gitlab-ui/) when a merge can't be performed cleanly between two versions of the same file. -#### Merge Request +#### Merge Request (MR) -[Takes changes](https://docs.gitlab.com/ce/gitlab-basics/add-merge-request.html) from one branch, and applies them into another branch. +[Takes changes](../../gitlab-basics/add-merge-request.md) from one branch, and applies them into another branch. ### Meteor @@ -395,11 +395,11 @@ Allow you to [organize issues](../../user/project/milestones/index.md) and merge ### Mirror Repositories -A project that is set up to automatically have its branches, tags, and commits [updated from an upstream repository](https://docs.gitlab.com/ee/workflow/repository_mirroring.html). This is useful when a repository you're interested in is located on a different server, and you want to be able to browse its content and activity using the familiar GitLab interface. +A project that is set up to automatically have its branches, tags, and commits [updated from an upstream repository](../../workflow/repository_mirroring.md). This is useful when a repository you're interested in is located on a different server, and you want to be able to browse its content and activity using the familiar GitLab interface. ### MIT License -A type of software license. It lets people do anything with your code with proper attribution and without warranty. It is the most common license for open source applications written in Ruby on Rails. GitLab CE is issued under this [license](https://docs.gitlab.com/ce/development/licensing.html). This means you can download the code, modify it as you want, and even build a new commercial product using the underlying code and it's not illegal. The only condition is that there is no form of warranty provided by GitLab so whatever happens when you use the code is your own problem. +A type of software license. It lets people do anything with your code with proper attribution and without warranty. It is the most common license for open source applications written in Ruby on Rails. GitLab CE is issued under this [license](../../development/licensing.md). This means you can download the code, modify it as you want, and even build a new commercial product using the underlying code and it's not illegal. The only condition is that there is no form of warranty provided by GitLab so whatever happens when you use the code is your own problem. ### Mondo Rescue @@ -427,7 +427,7 @@ A web [server](https://www.nginx.com/resources/wiki/) (pronounced "engine x"). [ ### OAuth -An open standard for authorization, commonly used as a way for internet users to log into third party websites using their Microsoft, Google, Facebook or Twitter accounts without exposing their password. GitLab [is](https://docs.gitlab.com/ce/integration/oauth_provider.html) an OAuth2 authentication service provider. +An open standard for authorization, commonly used as a way for internet users to log into third party websites using their Microsoft, Google, Facebook or Twitter accounts without exposing their password. GitLab [is](../../integration/oauth_provider.md) an OAuth2 authentication service provider. ### Omnibus Packages @@ -479,11 +479,11 @@ An [object-relational](https://en.wikipedia.org/wiki/PostgreSQL) database. Toute ### Protected Branches -A [feature](https://docs.gitlab.com/ce/user/project/protected_branches.html) that protects branches from unauthorized pushes, force pushing or deletion. +A [feature](../../user/project/protected_branches.md) that protects branches from unauthorized pushes, force pushing or deletion. ### Protected Tags -A [feature](https://docs.gitlab.com/ce/user/project/protected_tags.html) that protects tags from unauthorized creation, update or deletion +A [feature](../../user/project/protected_tags.md) that protects tags from unauthorized creation, update or deletion ### Pull @@ -547,7 +547,7 @@ Actual build machines/containers that [run and execute tests](https://gitlab.com ### Sidekiq -The background job processor GitLab [uses](https://docs.gitlab.com/ce/administration/troubleshooting/sidekiq.html) to asynchronously run tasks. +The background job processor GitLab [uses](../../administration/troubleshooting/sidekiq.md) to asynchronously run tasks. ### Software as a service (SaaS) @@ -567,7 +567,7 @@ The board used to track the status and progress of each of the sprint backlog it ### Shell -Terminal on Mac OSX, GitBash on Windows, or Linux Terminal on Linux. You [use git](https://docs.gitlab.com/ce/gitlab-basics/start-using-git.html) and make changes to GitLab projects in your shell. You [use git](https://docs.gitlab.com/ce/gitlab-basics/start-using-git.html) and make changes to GitLab projects in your shell. +Terminal on Mac OSX, GitBash on Windows, or Linux Terminal on Linux. You [use git](../../gitlab-basics/start-using-git.md) and make changes to GitLab projects in your shell. You [use git](../../gitlab-basics/start-using-git.md) and make changes to GitLab projects in your shell. ### Shell command runner @@ -577,7 +577,7 @@ The tenant purchases their own copy of the software and the software can be cust ### Slack -Real time messaging app for teams that is used internally by GitLab team members. GitLab users can enable [Slack integration](https://docs.gitlab.com/ce/project_services/slack.html) to trigger push, issue, and merge request events among others. +Real time messaging app for teams that is used internally by GitLab team members. GitLab users can enable [Slack integration](../../project_services/slack.md) to trigger push, issue, and merge request events among others. ### Slash commands @@ -595,7 +595,7 @@ Program code as typed by a computer programmer (i.e. it has not yet been compile ### SSH Key -A unique identifier of a computer. It is used to identify computers without the need for a password (e.g., On GitLab I have [added the ssh key](https://docs.gitlab.com/ce/gitlab-basics/create-your-ssh-keys.html) of all my work machines so that the GitLab instance knows that it can accept code pushes and pulls from this trusted machines whose keys are I have added.) +A unique identifier of a computer. It is used to identify computers without the need for a password (e.g., On GitLab I have [added the ssh key](../../gitlab-basics/create-your-ssh-keys.md) of all my work machines so that the GitLab instance knows that it can accept code pushes and pulls from this trusted machines whose keys are I have added.) ### Single Sign On (SSO) @@ -627,11 +627,11 @@ A program that allows you to perform superuser/administrator actions on Unix Ope ### Subversion (SVN) -An open source version control system. Read about [migrating from SVN](https://docs.gitlab.com/ce/workflow/importing/migrating_from_svn.html) to GitLab using SubGit. +An open source version control system. Read about [migrating from SVN](../../workflow/importing/migrating_from_svn.md) to GitLab using SubGit. ### Tag -[Represents](https://docs.gitlab.com/ce/api/tags.html) a version of a particular branch at a moment in time. +[Represents](../../api/tags.md) a version of a particular branch at a moment in time. ### Tenancy @@ -673,7 +673,7 @@ Version control is a system that records changes to a file or set of files over ### Virtual Private Cloud (VPC) -A [VPC](https://docs.gitlab.com/ce/university/glossary/README.html#virtual-private-cloud-vpc) is an on demand configurable pool of shared computing resources allocated within a public cloud environment, providing some isolation between the different users using the resources. GitLab users need to create a new Amazon VPC in order to [set up High Availability](https://docs.gitlab.com/ce/university/high-availability/aws/). +A [VPC](#virtual-private-cloud-vpc) is an on demand configurable pool of shared computing resources allocated within a public cloud environment, providing some isolation between the different users using the resources. GitLab users need to create a new Amazon VPC in order to [set up High Availability](../../install/aws/index.md). ### Virtual private server (VPS) @@ -689,7 +689,7 @@ A [model](http://www.umsl.edu/~hugheyd/is6840/waterfall.html) of building softwa ### Webhooks -A way for an app to [provide](https://docs.gitlab.com/ce/user/project/integrations/webhooks.html) other applications with real-time information (e.g., send a message to a slack channel when a commit is pushed.) Read about setting up [custom git hooks](https://gitlab.com/help/administration/custom_hooks.md) for when webhooks are insufficient. +A way for an app to [provide](../../user/project/integrations/webhooks.md) other applications with real-time information (e.g., send a message to a slack channel when a commit is pushed.) Read about setting up [custom git hooks](../../administration/custom_hooks.md) for when webhooks are insufficient. ### Wiki diff --git a/doc/university/high-availability/aws/README.md b/doc/university/high-availability/aws/README.md index 4095f71e501..fa04e988042 100644 --- a/doc/university/high-availability/aws/README.md +++ b/doc/university/high-availability/aws/README.md @@ -3,7 +3,7 @@ comments: false --- > **Note**: We **do not** recommend using the AWS Elastic File System (EFS), as it can result -in [significantly degraded performance](https://gitlab.com/gitlab-org/gitlab-ee/blob/master/doc/administration/high_availability/nfs.md#aws-elastic-file-system). +in [significantly degraded performance](../../../administration/high_availability/nfs.md#avoid-using-awss-elastic-file-system-efs). # High Availability on AWS @@ -83,7 +83,7 @@ our newly created VPC. ### Internet Gateway Now still on the same dashboard head over to Internet Gateways and -create a new one. After its created pres on the `Attach to VPC` button and +create a new one. After its created press on the `Attach to VPC` button and select our VPC.  @@ -182,7 +182,7 @@ Another option is to build a simple NFS server using a vanilla Linux server back by AWS Elastic Block Storage (EBS). > **Note:** GitLab does not recommend using AWS Elastic File System (EFS). See - details in [High Availability NFS documentation](../../../administration/high_availability/nfs.md#aws-elastic-file-system) + details in [High Availability NFS documentation](../../../administration/high_availability/nfs.md#avoid-using-awss-elastic-file-system-efs) *** @@ -390,5 +390,5 @@ some redundancy options but it might also imply Geographic replication. There is a lot of ground yet to cover so have a read through these other resources and feel free to open an issue to request additional material. -- [GitLab High Availability](http://docs.gitlab.com/ce/administration/high_availability/README.html#sts=High%20Availability) -- [GitLab Geo](https://docs.gitlab.com/ee/administration/geo/replication/index.html) +- [GitLab High Availability](../../../administration/high_availability/README.md) +- [GitLab Geo](../../../administration/geo/replication/index.md) diff --git a/doc/university/high-availability/aws/img/elastic-file-system.png b/doc/university/high-availability/aws/img/elastic-file-system.png Binary files differdeleted file mode 100644 index 5bcfb8d0588..00000000000 --- a/doc/university/high-availability/aws/img/elastic-file-system.png +++ /dev/null diff --git a/doc/university/high-availability/aws/img/reference-arch2.png b/doc/university/high-availability/aws/img/reference-arch2.png Binary files differindex a9cb6663103..0f8790d0f74 100644 --- a/doc/university/high-availability/aws/img/reference-arch2.png +++ b/doc/university/high-availability/aws/img/reference-arch2.png diff --git a/doc/university/support/README.md b/doc/university/support/README.md index feb90ae9bad..c8ade54a77c 100644 --- a/doc/university/support/README.md +++ b/doc/university/support/README.md @@ -61,28 +61,27 @@ Sometimes we need to upgrade customers from old versions of GitLab to latest, so - Users - Groups - Projects - - [Backup using our Backup rake task](https://docs.gitlab.com/ce/raketasks/backup_restore.html#create-a-backup-of-the-gitlab-system) + - [Backup using our Backup rake task](../../raketasks/backup_restore.md#creating-a-backup-of-the-gitlab-system) - [Upgrade to 5.0 source using our Upgrade documentation](https://gitlab.com/gitlab-org/gitlab-ee/blob/master/doc/update/4.2-to-5.0.md) - [Upgrade to 5.1 source](https://gitlab.com/gitlab-org/gitlab-ee/blob/master/doc/update/5.0-to-5.1.md) - [Upgrade to 6.0 source](https://gitlab.com/gitlab-org/gitlab-ee/blob/master/doc/update/5.1-to-6.0.md) - [Upgrade to 7.14 source](https://gitlab.com/gitlab-org/gitlab-ee/blob/master/doc/update/6.x-or-7.x-to-7.14.md) - - [Backup using our Backup rake task](https://docs.gitlab.com/ce/raketasks/backup_restore.html#create-a-backup-of-the-gitlab-system) - - [Perform the MySQL to PostgreSQL migration to convert your backup](https://docs.gitlab.com/ce/update/mysql_to_postgresql.html#converting-a-gitlab-backup-file-from-mysql-to-postgres) + - [Perform the MySQL to PostgreSQL migration to convert your backup](../../update/mysql_to_postgresql.md) - [Upgrade to Omnibus 7.14](https://docs.gitlab.com/omnibus/update/README.html#upgrading-from-a-non-omnibus-installation-to-an-omnibus-installation) - - [Restore backup using our Restore rake task](https://docs.gitlab.com/ce/raketasks/backup_restore.html#restore-a-previously-created-backup) + - [Restore backup using our Restore rake task](../../raketasks/backup_restore.md#restore) - [Upgrade to latest EE](https://about.gitlab.com/downloads-ee) - (GitLab inc. only) Acquire and apply a license for the Enterprise Edition product, ask in #support -- Perform a downgrade from [EE to CE](https://docs.gitlab.com/ee/downgrade_ee_to_ce/README.html) +- Perform a downgrade from [EE to CE](../../downgrade_ee_to_ce/README.md) #### Start to learn about some of the integrations that we support Our integrations add great value to GitLab. User questions often relate to integrating GitLab with existing external services and the configuration involved - Learn about our Integrations (specially, not only): - - [LDAP](https://docs.gitlab.com/ee/integration/ldap.html) - - [JIRA](https://docs.gitlab.com/ee/project_services/jira.html) - - [Jenkins](https://docs.gitlab.com/ee/integration/jenkins.html) - - [SAML](https://docs.gitlab.com/ce/integration/saml.html) + - [LDAP](../../integration/ldap.md) + - [JIRA](../../project_services/jira.md) + - [Jenkins](../../integration/jenkins.md) + - [SAML](../../integration/saml.md) #### Goals @@ -94,8 +93,8 @@ Our integrations add great value to GitLab. User questions often relate to integ #### Understand the gathering of diagnostics for GitLab instances - Learn about the GitLab checks that are available - - [Environment Information and maintenance checks](https://docs.gitlab.com/ce/raketasks/maintenance.html) - - [GitLab check](https://docs.gitlab.com/ce/raketasks/check.html) + - [Environment Information and maintenance checks](../../raketasks/maintenance.md) + - [GitLab check](../../raketasks/check.md) - Omnibus commands - [Status](https://gitlab.com/gitlab-org/omnibus-gitlab/blob/master/doc/maintenance/README.md#get-service-status) - [Starting and stopping services](https://gitlab.com/gitlab-org/omnibus-gitlab/blob/master/doc/maintenance/README.md#starting-and-stopping) @@ -170,11 +169,11 @@ Some tickets need specific knowledge or a deep understanding of a particular com Move on to understanding some of GitLab's more advanced features. You can make use of GitLab.com to understand the features from an end-user perspective and then use your own instance to understand setup and configuration of the feature from an Administrative perspective -- Set up and try [Git LFS](https://docs.gitlab.com/ee/workflow/lfs/manage_large_binaries_with_git_lfs.html) -- Get to know the [GitLab API](https://docs.gitlab.com/ee/api/README.html), its capabilities and shortcomings -- Learn how to [migrate from SVN to Git](https://docs.gitlab.com/ee/workflow/importing/migrating_from_svn.html) -- Set up [GitLab CI](https://docs.gitlab.com/ee/ci/quick_start/README.html) -- Create your first [GitLab Page](https://docs.gitlab.com/ce/administration/pages/) +- Set up and try [Git LFS](../../workflow/lfs/manage_large_binaries_with_git_lfs.md) +- Get to know the [GitLab API](../../api/README.md), its capabilities and shortcomings +- Learn how to [migrate from SVN to Git](../../user/project/import/svn.md) +- Set up [GitLab CI](../../ci/quick_start/README.md) +- Create your first [GitLab Page](../../administration/pages/index.md) - Get to know the GitLab Codebase by reading through the source code: - Find the differences between the [EE codebase](https://gitlab.com/gitlab-org/gitlab-ce) and the [CE codebase](https://gitlab.com/gitlab-org/gitlab-ce) diff --git a/doc/university/training/topics/unstage.md b/doc/university/training/topics/unstage.md index da36a3218e5..c926f0b4888 100644 --- a/doc/university/training/topics/unstage.md +++ b/doc/university/training/topics/unstage.md @@ -8,13 +8,13 @@ comments: false ## Unstage -- To remove files from stage use reset HEAD. Where HEAD is the last commit of the current branch. +- To remove files from stage use reset HEAD where HEAD is the last commit of the current branch. This will unstage the file but maintain the modifications. ```bash git reset HEAD <file> ``` -- This will unstage the file but maintain the modifications. To revert the file back to the state it was in before the changes we can use: +- To revert the file back to the state it was in before the changes we can use: ```bash git checkout -- <file> diff --git a/doc/update/10.0-ce-to-ee.md b/doc/update/10.0-ce-to-ee.md new file mode 100644 index 00000000000..10c9e21fa81 --- /dev/null +++ b/doc/update/10.0-ce-to-ee.md @@ -0,0 +1,5 @@ +--- +redirect_to: upgrading_from_ce_to_ee.md +--- + +This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/10.1-ce-to-ee.md b/doc/update/10.1-ce-to-ee.md new file mode 100644 index 00000000000..10c9e21fa81 --- /dev/null +++ b/doc/update/10.1-ce-to-ee.md @@ -0,0 +1,5 @@ +--- +redirect_to: upgrading_from_ce_to_ee.md +--- + +This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/10.2-ce-to-ee.md b/doc/update/10.2-ce-to-ee.md new file mode 100644 index 00000000000..10c9e21fa81 --- /dev/null +++ b/doc/update/10.2-ce-to-ee.md @@ -0,0 +1,5 @@ +--- +redirect_to: upgrading_from_ce_to_ee.md +--- + +This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/10.3-ce-to-ee.md b/doc/update/10.3-ce-to-ee.md new file mode 100644 index 00000000000..10c9e21fa81 --- /dev/null +++ b/doc/update/10.3-ce-to-ee.md @@ -0,0 +1,5 @@ +--- +redirect_to: upgrading_from_ce_to_ee.md +--- + +This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/10.4-ce-to-ee.md b/doc/update/10.4-ce-to-ee.md new file mode 100644 index 00000000000..10c9e21fa81 --- /dev/null +++ b/doc/update/10.4-ce-to-ee.md @@ -0,0 +1,5 @@ +--- +redirect_to: upgrading_from_ce_to_ee.md +--- + +This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/10.5-ce-to-ee.md b/doc/update/10.5-ce-to-ee.md new file mode 100644 index 00000000000..10c9e21fa81 --- /dev/null +++ b/doc/update/10.5-ce-to-ee.md @@ -0,0 +1,5 @@ +--- +redirect_to: upgrading_from_ce_to_ee.md +--- + +This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/10.6-ce-to-ee.md b/doc/update/10.6-ce-to-ee.md new file mode 100644 index 00000000000..10c9e21fa81 --- /dev/null +++ b/doc/update/10.6-ce-to-ee.md @@ -0,0 +1,5 @@ +--- +redirect_to: upgrading_from_ce_to_ee.md +--- + +This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/10.7-ce-to-ee.md b/doc/update/10.7-ce-to-ee.md new file mode 100644 index 00000000000..10c9e21fa81 --- /dev/null +++ b/doc/update/10.7-ce-to-ee.md @@ -0,0 +1,5 @@ +--- +redirect_to: upgrading_from_ce_to_ee.md +--- + +This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/10.8-ce-to-ee.md b/doc/update/10.8-ce-to-ee.md new file mode 100644 index 00000000000..10c9e21fa81 --- /dev/null +++ b/doc/update/10.8-ce-to-ee.md @@ -0,0 +1,5 @@ +--- +redirect_to: upgrading_from_ce_to_ee.md +--- + +This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/11.0-ce-to-ee.md b/doc/update/11.0-ce-to-ee.md new file mode 100644 index 00000000000..10c9e21fa81 --- /dev/null +++ b/doc/update/11.0-ce-to-ee.md @@ -0,0 +1,5 @@ +--- +redirect_to: upgrading_from_ce_to_ee.md +--- + +This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/11.1-ce-to-ee.md b/doc/update/11.1-ce-to-ee.md new file mode 100644 index 00000000000..10c9e21fa81 --- /dev/null +++ b/doc/update/11.1-ce-to-ee.md @@ -0,0 +1,5 @@ +--- +redirect_to: upgrading_from_ce_to_ee.md +--- + +This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/11.2-ce-to-ee.md b/doc/update/11.2-ce-to-ee.md new file mode 100644 index 00000000000..10c9e21fa81 --- /dev/null +++ b/doc/update/11.2-ce-to-ee.md @@ -0,0 +1,5 @@ +--- +redirect_to: upgrading_from_ce_to_ee.md +--- + +This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/11.3-ce-to-ee.md b/doc/update/11.3-ce-to-ee.md new file mode 100644 index 00000000000..10c9e21fa81 --- /dev/null +++ b/doc/update/11.3-ce-to-ee.md @@ -0,0 +1,5 @@ +--- +redirect_to: upgrading_from_ce_to_ee.md +--- + +This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/11.4-ce-to-ee.md b/doc/update/11.4-ce-to-ee.md new file mode 100644 index 00000000000..10c9e21fa81 --- /dev/null +++ b/doc/update/11.4-ce-to-ee.md @@ -0,0 +1,5 @@ +--- +redirect_to: upgrading_from_ce_to_ee.md +--- + +This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/11.5-ce-to-ee.md b/doc/update/11.5-ce-to-ee.md new file mode 100644 index 00000000000..10c9e21fa81 --- /dev/null +++ b/doc/update/11.5-ce-to-ee.md @@ -0,0 +1,5 @@ +--- +redirect_to: upgrading_from_ce_to_ee.md +--- + +This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/11.6-ce-to-ee.md b/doc/update/11.6-ce-to-ee.md new file mode 100644 index 00000000000..10c9e21fa81 --- /dev/null +++ b/doc/update/11.6-ce-to-ee.md @@ -0,0 +1,5 @@ +--- +redirect_to: upgrading_from_ce_to_ee.md +--- + +This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/11.7-ce-to-ee.md b/doc/update/11.7-ce-to-ee.md new file mode 100644 index 00000000000..10c9e21fa81 --- /dev/null +++ b/doc/update/11.7-ce-to-ee.md @@ -0,0 +1,5 @@ +--- +redirect_to: upgrading_from_ce_to_ee.md +--- + +This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/11.8-ce-to-ee.md b/doc/update/11.8-ce-to-ee.md new file mode 100644 index 00000000000..10c9e21fa81 --- /dev/null +++ b/doc/update/11.8-ce-to-ee.md @@ -0,0 +1,5 @@ +--- +redirect_to: upgrading_from_ce_to_ee.md +--- + +This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/6.0-ce-to-ee.md b/doc/update/6.0-ce-to-ee.md new file mode 100644 index 00000000000..10c9e21fa81 --- /dev/null +++ b/doc/update/6.0-ce-to-ee.md @@ -0,0 +1,5 @@ +--- +redirect_to: upgrading_from_ce_to_ee.md +--- + +This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/6.1-ce-to-ee.md b/doc/update/6.1-ce-to-ee.md new file mode 100644 index 00000000000..10c9e21fa81 --- /dev/null +++ b/doc/update/6.1-ce-to-ee.md @@ -0,0 +1,5 @@ +--- +redirect_to: upgrading_from_ce_to_ee.md +--- + +This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/6.2-ce-to-ee.md b/doc/update/6.2-ce-to-ee.md new file mode 100644 index 00000000000..10c9e21fa81 --- /dev/null +++ b/doc/update/6.2-ce-to-ee.md @@ -0,0 +1,5 @@ +--- +redirect_to: upgrading_from_ce_to_ee.md +--- + +This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/6.3-ce-to-ee.md b/doc/update/6.3-ce-to-ee.md new file mode 100644 index 00000000000..10c9e21fa81 --- /dev/null +++ b/doc/update/6.3-ce-to-ee.md @@ -0,0 +1,5 @@ +--- +redirect_to: upgrading_from_ce_to_ee.md +--- + +This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/6.4-ce-to-ee.md b/doc/update/6.4-ce-to-ee.md new file mode 100644 index 00000000000..10c9e21fa81 --- /dev/null +++ b/doc/update/6.4-ce-to-ee.md @@ -0,0 +1,5 @@ +--- +redirect_to: upgrading_from_ce_to_ee.md +--- + +This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/6.5-ce-to-ee.md b/doc/update/6.5-ce-to-ee.md new file mode 100644 index 00000000000..10c9e21fa81 --- /dev/null +++ b/doc/update/6.5-ce-to-ee.md @@ -0,0 +1,5 @@ +--- +redirect_to: upgrading_from_ce_to_ee.md +--- + +This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/6.6-ce-to-ee.md b/doc/update/6.6-ce-to-ee.md new file mode 100644 index 00000000000..10c9e21fa81 --- /dev/null +++ b/doc/update/6.6-ce-to-ee.md @@ -0,0 +1,5 @@ +--- +redirect_to: upgrading_from_ce_to_ee.md +--- + +This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/6.7-ce-to-ee.md b/doc/update/6.7-ce-to-ee.md new file mode 100644 index 00000000000..10c9e21fa81 --- /dev/null +++ b/doc/update/6.7-ce-to-ee.md @@ -0,0 +1,5 @@ +--- +redirect_to: upgrading_from_ce_to_ee.md +--- + +This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/6.8-ce-to-ee.md b/doc/update/6.8-ce-to-ee.md new file mode 100644 index 00000000000..10c9e21fa81 --- /dev/null +++ b/doc/update/6.8-ce-to-ee.md @@ -0,0 +1,5 @@ +--- +redirect_to: upgrading_from_ce_to_ee.md +--- + +This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/6.9-ce-to-ee.md b/doc/update/6.9-ce-to-ee.md new file mode 100644 index 00000000000..10c9e21fa81 --- /dev/null +++ b/doc/update/6.9-ce-to-ee.md @@ -0,0 +1,5 @@ +--- +redirect_to: upgrading_from_ce_to_ee.md +--- + +This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/7.0-ce-to-ee.md b/doc/update/7.0-ce-to-ee.md new file mode 100644 index 00000000000..10c9e21fa81 --- /dev/null +++ b/doc/update/7.0-ce-to-ee.md @@ -0,0 +1,5 @@ +--- +redirect_to: upgrading_from_ce_to_ee.md +--- + +This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/7.1-ce-to-ee.md b/doc/update/7.1-ce-to-ee.md new file mode 100644 index 00000000000..10c9e21fa81 --- /dev/null +++ b/doc/update/7.1-ce-to-ee.md @@ -0,0 +1,5 @@ +--- +redirect_to: upgrading_from_ce_to_ee.md +--- + +This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/7.10-ce-to-ee.md b/doc/update/7.10-ce-to-ee.md new file mode 100644 index 00000000000..10c9e21fa81 --- /dev/null +++ b/doc/update/7.10-ce-to-ee.md @@ -0,0 +1,5 @@ +--- +redirect_to: upgrading_from_ce_to_ee.md +--- + +This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/7.11-ce-to-ee.md b/doc/update/7.11-ce-to-ee.md new file mode 100644 index 00000000000..10c9e21fa81 --- /dev/null +++ b/doc/update/7.11-ce-to-ee.md @@ -0,0 +1,5 @@ +--- +redirect_to: upgrading_from_ce_to_ee.md +--- + +This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/7.12-ce-to-ee.md b/doc/update/7.12-ce-to-ee.md new file mode 100644 index 00000000000..10c9e21fa81 --- /dev/null +++ b/doc/update/7.12-ce-to-ee.md @@ -0,0 +1,5 @@ +--- +redirect_to: upgrading_from_ce_to_ee.md +--- + +This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/7.13-ce-to-ee.md b/doc/update/7.13-ce-to-ee.md new file mode 100644 index 00000000000..10c9e21fa81 --- /dev/null +++ b/doc/update/7.13-ce-to-ee.md @@ -0,0 +1,5 @@ +--- +redirect_to: upgrading_from_ce_to_ee.md +--- + +This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/7.14-ce-to-ee.md b/doc/update/7.14-ce-to-ee.md new file mode 100644 index 00000000000..10c9e21fa81 --- /dev/null +++ b/doc/update/7.14-ce-to-ee.md @@ -0,0 +1,5 @@ +--- +redirect_to: upgrading_from_ce_to_ee.md +--- + +This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/7.3-ce-to-ee.md b/doc/update/7.3-ce-to-ee.md new file mode 100644 index 00000000000..10c9e21fa81 --- /dev/null +++ b/doc/update/7.3-ce-to-ee.md @@ -0,0 +1,5 @@ +--- +redirect_to: upgrading_from_ce_to_ee.md +--- + +This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/7.4-ce-to-ee.md b/doc/update/7.4-ce-to-ee.md new file mode 100644 index 00000000000..10c9e21fa81 --- /dev/null +++ b/doc/update/7.4-ce-to-ee.md @@ -0,0 +1,5 @@ +--- +redirect_to: upgrading_from_ce_to_ee.md +--- + +This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/7.5-ce-to-ee.md b/doc/update/7.5-ce-to-ee.md new file mode 100644 index 00000000000..10c9e21fa81 --- /dev/null +++ b/doc/update/7.5-ce-to-ee.md @@ -0,0 +1,5 @@ +--- +redirect_to: upgrading_from_ce_to_ee.md +--- + +This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/7.6-ce-to-ee.md b/doc/update/7.6-ce-to-ee.md new file mode 100644 index 00000000000..10c9e21fa81 --- /dev/null +++ b/doc/update/7.6-ce-to-ee.md @@ -0,0 +1,5 @@ +--- +redirect_to: upgrading_from_ce_to_ee.md +--- + +This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/7.7-ce-to-ee.md b/doc/update/7.7-ce-to-ee.md new file mode 100644 index 00000000000..10c9e21fa81 --- /dev/null +++ b/doc/update/7.7-ce-to-ee.md @@ -0,0 +1,5 @@ +--- +redirect_to: upgrading_from_ce_to_ee.md +--- + +This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/7.8-ce-to-ee.md b/doc/update/7.8-ce-to-ee.md new file mode 100644 index 00000000000..10c9e21fa81 --- /dev/null +++ b/doc/update/7.8-ce-to-ee.md @@ -0,0 +1,5 @@ +--- +redirect_to: upgrading_from_ce_to_ee.md +--- + +This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/7.9-ce-to-ee.md b/doc/update/7.9-ce-to-ee.md new file mode 100644 index 00000000000..10c9e21fa81 --- /dev/null +++ b/doc/update/7.9-ce-to-ee.md @@ -0,0 +1,5 @@ +--- +redirect_to: upgrading_from_ce_to_ee.md +--- + +This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/8.0-ce-to-ee.md b/doc/update/8.0-ce-to-ee.md new file mode 100644 index 00000000000..10c9e21fa81 --- /dev/null +++ b/doc/update/8.0-ce-to-ee.md @@ -0,0 +1,5 @@ +--- +redirect_to: upgrading_from_ce_to_ee.md +--- + +This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/8.1-ce-to-ee.md b/doc/update/8.1-ce-to-ee.md new file mode 100644 index 00000000000..10c9e21fa81 --- /dev/null +++ b/doc/update/8.1-ce-to-ee.md @@ -0,0 +1,5 @@ +--- +redirect_to: upgrading_from_ce_to_ee.md +--- + +This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/8.10-ce-to-ee.md b/doc/update/8.10-ce-to-ee.md new file mode 100644 index 00000000000..10c9e21fa81 --- /dev/null +++ b/doc/update/8.10-ce-to-ee.md @@ -0,0 +1,5 @@ +--- +redirect_to: upgrading_from_ce_to_ee.md +--- + +This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/8.11-ce-to-ee.md b/doc/update/8.11-ce-to-ee.md new file mode 100644 index 00000000000..10c9e21fa81 --- /dev/null +++ b/doc/update/8.11-ce-to-ee.md @@ -0,0 +1,5 @@ +--- +redirect_to: upgrading_from_ce_to_ee.md +--- + +This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/8.12-ce-to-ee.md b/doc/update/8.12-ce-to-ee.md new file mode 100644 index 00000000000..10c9e21fa81 --- /dev/null +++ b/doc/update/8.12-ce-to-ee.md @@ -0,0 +1,5 @@ +--- +redirect_to: upgrading_from_ce_to_ee.md +--- + +This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/8.13-ce-to-ee.md b/doc/update/8.13-ce-to-ee.md new file mode 100644 index 00000000000..10c9e21fa81 --- /dev/null +++ b/doc/update/8.13-ce-to-ee.md @@ -0,0 +1,5 @@ +--- +redirect_to: upgrading_from_ce_to_ee.md +--- + +This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/8.14-ce-to-ee.md b/doc/update/8.14-ce-to-ee.md new file mode 100644 index 00000000000..10c9e21fa81 --- /dev/null +++ b/doc/update/8.14-ce-to-ee.md @@ -0,0 +1,5 @@ +--- +redirect_to: upgrading_from_ce_to_ee.md +--- + +This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/8.15-ce-to-ee.md b/doc/update/8.15-ce-to-ee.md new file mode 100644 index 00000000000..10c9e21fa81 --- /dev/null +++ b/doc/update/8.15-ce-to-ee.md @@ -0,0 +1,5 @@ +--- +redirect_to: upgrading_from_ce_to_ee.md +--- + +This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/8.16-ce-to-ee.md b/doc/update/8.16-ce-to-ee.md new file mode 100644 index 00000000000..10c9e21fa81 --- /dev/null +++ b/doc/update/8.16-ce-to-ee.md @@ -0,0 +1,5 @@ +--- +redirect_to: upgrading_from_ce_to_ee.md +--- + +This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/8.17-ce-to-ee.md b/doc/update/8.17-ce-to-ee.md new file mode 100644 index 00000000000..10c9e21fa81 --- /dev/null +++ b/doc/update/8.17-ce-to-ee.md @@ -0,0 +1,5 @@ +--- +redirect_to: upgrading_from_ce_to_ee.md +--- + +This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/8.2-ce-to-ee.md b/doc/update/8.2-ce-to-ee.md new file mode 100644 index 00000000000..10c9e21fa81 --- /dev/null +++ b/doc/update/8.2-ce-to-ee.md @@ -0,0 +1,5 @@ +--- +redirect_to: upgrading_from_ce_to_ee.md +--- + +This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/8.3-ce-to-ee.md b/doc/update/8.3-ce-to-ee.md new file mode 100644 index 00000000000..10c9e21fa81 --- /dev/null +++ b/doc/update/8.3-ce-to-ee.md @@ -0,0 +1,5 @@ +--- +redirect_to: upgrading_from_ce_to_ee.md +--- + +This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/8.4-ce-to-ee.md b/doc/update/8.4-ce-to-ee.md new file mode 100644 index 00000000000..10c9e21fa81 --- /dev/null +++ b/doc/update/8.4-ce-to-ee.md @@ -0,0 +1,5 @@ +--- +redirect_to: upgrading_from_ce_to_ee.md +--- + +This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/8.5-ce-to-ee.md b/doc/update/8.5-ce-to-ee.md new file mode 100644 index 00000000000..10c9e21fa81 --- /dev/null +++ b/doc/update/8.5-ce-to-ee.md @@ -0,0 +1,5 @@ +--- +redirect_to: upgrading_from_ce_to_ee.md +--- + +This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/8.6-ce-to-ee.md b/doc/update/8.6-ce-to-ee.md new file mode 100644 index 00000000000..10c9e21fa81 --- /dev/null +++ b/doc/update/8.6-ce-to-ee.md @@ -0,0 +1,5 @@ +--- +redirect_to: upgrading_from_ce_to_ee.md +--- + +This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/8.7-ce-to-ee.md b/doc/update/8.7-ce-to-ee.md new file mode 100644 index 00000000000..10c9e21fa81 --- /dev/null +++ b/doc/update/8.7-ce-to-ee.md @@ -0,0 +1,5 @@ +--- +redirect_to: upgrading_from_ce_to_ee.md +--- + +This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/8.8-ce-to-ee.md b/doc/update/8.8-ce-to-ee.md new file mode 100644 index 00000000000..10c9e21fa81 --- /dev/null +++ b/doc/update/8.8-ce-to-ee.md @@ -0,0 +1,5 @@ +--- +redirect_to: upgrading_from_ce_to_ee.md +--- + +This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/8.9-ce-to-ee.md b/doc/update/8.9-ce-to-ee.md new file mode 100644 index 00000000000..10c9e21fa81 --- /dev/null +++ b/doc/update/8.9-ce-to-ee.md @@ -0,0 +1,5 @@ +--- +redirect_to: upgrading_from_ce_to_ee.md +--- + +This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/9.0-ce-to-ee.md b/doc/update/9.0-ce-to-ee.md new file mode 100644 index 00000000000..10c9e21fa81 --- /dev/null +++ b/doc/update/9.0-ce-to-ee.md @@ -0,0 +1,5 @@ +--- +redirect_to: upgrading_from_ce_to_ee.md +--- + +This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/9.1-ce-to-ee.md b/doc/update/9.1-ce-to-ee.md new file mode 100644 index 00000000000..10c9e21fa81 --- /dev/null +++ b/doc/update/9.1-ce-to-ee.md @@ -0,0 +1,5 @@ +--- +redirect_to: upgrading_from_ce_to_ee.md +--- + +This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/9.2-ce-to-ee.md b/doc/update/9.2-ce-to-ee.md new file mode 100644 index 00000000000..10c9e21fa81 --- /dev/null +++ b/doc/update/9.2-ce-to-ee.md @@ -0,0 +1,5 @@ +--- +redirect_to: upgrading_from_ce_to_ee.md +--- + +This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/9.3-ce-to-ee.md b/doc/update/9.3-ce-to-ee.md new file mode 100644 index 00000000000..10c9e21fa81 --- /dev/null +++ b/doc/update/9.3-ce-to-ee.md @@ -0,0 +1,5 @@ +--- +redirect_to: upgrading_from_ce_to_ee.md +--- + +This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/9.4-ce-to-ee.md b/doc/update/9.4-ce-to-ee.md new file mode 100644 index 00000000000..10c9e21fa81 --- /dev/null +++ b/doc/update/9.4-ce-to-ee.md @@ -0,0 +1,5 @@ +--- +redirect_to: upgrading_from_ce_to_ee.md +--- + +This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/9.5-ce-to-ee.md b/doc/update/9.5-ce-to-ee.md new file mode 100644 index 00000000000..10c9e21fa81 --- /dev/null +++ b/doc/update/9.5-ce-to-ee.md @@ -0,0 +1,5 @@ +--- +redirect_to: upgrading_from_ce_to_ee.md +--- + +This document was moved to [another location](upgrading_from_ce_to_ee.md). diff --git a/doc/update/mysql_to_postgresql.md b/doc/update/mysql_to_postgresql.md index 350072186ee..b83abcd36f7 100644 --- a/doc/update/mysql_to_postgresql.md +++ b/doc/update/mysql_to_postgresql.md @@ -1,31 +1,74 @@ --- -last_updated: 2018-02-07 +last_updated: 2019-03-27 --- # Migrating from MySQL to PostgreSQL -> **Note:** This guide assumes you have a working GitLab instance with -> MySQL and want to migrate to bundled PostgreSQL database. +This guide documents how to take a working GitLab instance that uses MySQL and +migrate it to a PostgreSQL database. -## Omnibus installation +## Requirements -### Prerequisites +[pgloader](http://pgloader.io) 3.4.1+ is required. -First, we'll need to enable the bundled PostgreSQL database with up-to-date -schema. Next, we'll use [pgloader](http://pgloader.io) to migrate the data -from the old MySQL database to the new PostgreSQL one. +You can install it directly from your distribution, for example in +Debian/Ubuntu: -Here's what you'll need to have installed: +1. Search for the version: -- pgloader 3.4.1+ -- Omnibus GitLab -- MySQL + ```bash + apt-cache madison pgloader + ``` -### Enable bundled PostgreSQL database +1. If the version is 3.4.1+, install it with: + + ```bash + sudo apt-get install pgloader + ``` + + If your distribution's version is too old, use PostgreSQL's repository: + + ```bash + # Add repository + sudo sh -c 'echo "deb http://apt.postgresql.org/pub/repos/apt/ $(lsb_release -cs)-pgdg main" > /etc/apt/sources.list.d/pgdg.list' + + # Add key + sudo apt-get install wget ca-certificates + wget --quiet -O - https://www.postgresql.org/media/keys/ACCC4CF8.asc | sudo apt-key add - + + # Install package + sudo apt-get update + sudo apt-get install pgloader + ``` + +For other distributions, follow the instructions in PostgreSQL's +[download page](https://www.postgresql.org/download/) to add their repository +and then install `pgloader`. + +If you are migrating to a Docker based installation, you will need to install +pgloader within the container as it is not included in the container image. + +1. Start a shell session in the context of the running container: + + ``` bash + docker exec -it gitlab bash + ``` + +1. Install pgloader: + + ``` bash + apt-get update + apt-get -y install pgloader + ``` + +## Omnibus GitLab installations + +For [Omnibus GitLab packages](https://about.gitlab.com/install/), you'll first +need to enable the bundled PostgreSQL: 1. Stop GitLab: - ``` bash + ```bash sudo gitlab-ctl stop ``` @@ -40,39 +83,34 @@ Here's what you'll need to have installed: and alike. You could just comment all of them out so that we'll just use the defaults. -1. [Reconfigure GitLab] for the changes to take effect: - - ``` bash - sudo gitlab-ctl reconfigure - ``` - +1. [Reconfigure GitLab](../administration/restart_gitlab.md#omnibus-gitlab-reconfigure) + for the changes to take effect. 1. Start Unicorn and PostgreSQL so that we can prepare the schema: - ``` bash + ```bash sudo gitlab-ctl start unicorn sudo gitlab-ctl start postgresql ``` 1. Run the following commands to prepare the schema: - ``` bash + ```bash sudo gitlab-rake db:create db:migrate ``` 1. Stop Unicorn to prevent other database access from interfering with the loading of data: - ``` bash + ```bash sudo gitlab-ctl stop unicorn ``` After these steps, you'll have a fresh PostgreSQL database with up-to-date schema. -### Migrate data from MySQL to PostgreSQL - -Now, you can use pgloader to migrate the data from MySQL to PostgreSQL: +Next, we'll use `pgloader` to migrate the data from the old MySQL database to the +new PostgreSQL one: 1. Save the following snippet in a `commands.load` file, and edit with your - database `username`, `password` and `host`: + MySQL database `username`, `password` and `host`: ``` LOAD DATABASE @@ -90,7 +128,7 @@ Now, you can use pgloader to migrate the data from MySQL to PostgreSQL: 1. Start the migration: - ``` bash + ```bash sudo -u gitlab-psql pgloader commands.load ``` @@ -117,170 +155,140 @@ Now, you can use pgloader to migrate the data from MySQL to PostgreSQL: Total import time 1894 1894 0 12.497s ``` - If there is no output for more than 30 minutes, it's possible pgloader encountered an error. See - the [troubleshooting guide](#Troubleshooting) for more details. + If there is no output for more than 30 minutes, it's possible `pgloader` encountered an error. See + the [troubleshooting guide](#troubleshooting) for more details. 1. Start GitLab: - ``` bash + ```bash sudo gitlab-ctl start ``` -Now, you can verify that everything worked by visiting GitLab. - -### Troubleshooting - -#### Permissions - -Note that the PostgreSQL user that you use for the above MUST have **superuser** privileges. Otherwise, you may see -a similar message to the following: - -``` -debugger invoked on a CL-POSTGRES-ERROR:INSUFFICIENT-PRIVILEGE in thread - #<THREAD "lparallel" RUNNING {10078A3513}>: - Database error 42501: permission denied: "RI_ConstraintTrigger_a_20937" is a system trigger - QUERY: ALTER TABLE ci_builds DISABLE TRIGGER ALL; - 2017-08-23T00:36:56.782000Z ERROR Database error 42501: permission denied: "RI_ConstraintTrigger_c_20864" is a system trigger - QUERY: ALTER TABLE approver_groups DISABLE TRIGGER ALL; -``` - -#### Experiencing 500 errors after the migration - -If you experience 500 errors after the migration, try to clear the cache: - -``` bash -sudo gitlab-rake cache:clear -``` - -[reconfigure GitLab]: ../administration/restart_gitlab.md#omnibus-gitlab-reconfigure - -## Source installation +You can now verify that everything works as expected by visiting GitLab. -### Prerequisites +## Source installations -#### Install PostgreSQL and create database +For installations from source that use MySQL, you'll first need to +[install PostgreSQL and create a database](../install/installation.md#6-database). -See [installation guide](../install/installation.md#6-database). - -#### Install [pgloader](http://pgloader.io) 3.4.1+ - -Install directly from your distro: -``` bash -sudo apt-get install pgloader -``` - -If this version is too old, use PostgreSQL's repository: -``` bash -# add repository -sudo sh -c 'echo "deb http://apt.postgresql.org/pub/repos/apt/ $(lsb_release -cs)-pgdg main" > /etc/apt/sources.list.d/pgdg.list' - -# add key -sudo apt-get install wget ca-certificates -wget --quiet -O - https://www.postgresql.org/media/keys/ACCC4CF8.asc | sudo apt-key add - - -# install package -sudo apt-get update -sudo apt-get install pgloader -``` - -### Enable bundled PostgreSQL database +After the database is created, go on with the following steps: 1. Stop GitLab: - ``` bash - sudo service gitlab stop - ``` + ```bash + sudo service gitlab stop + ``` 1. Switch database from MySQL to PostgreSQL - ``` bash - cd /home/git/gitlab - sudo -u git mv config/database.yml config/database.yml.bak - sudo -u git cp config/database.yml.postgresql config/database.yml - sudo -u git -H chmod o-rwx config/database.yml - ``` + ```bash + cd /home/git/gitlab + sudo -u git mv config/database.yml config/database.yml.bak + sudo -u git cp config/database.yml.postgresql config/database.yml + sudo -u git -H chmod o-rwx config/database.yml + ``` + 1. Install Gems related to Postgresql - ``` bash - sudo -u git -H rm .bundle/config - sudo -u git -H bundle install --deployment --without development test mysql aws kerberos - ``` + ```bash + sudo -u git -H rm .bundle/config + sudo -u git -H bundle install --deployment --without development test mysql aws kerberos + ``` 1. Run the following commands to prepare the schema: - ``` bash - sudo -u git -H bundle exec rake db:create db:migrate RAILS_ENV=production - ``` + ```bash + sudo -u git -H bundle exec rake db:create db:migrate RAILS_ENV=production + ``` After these steps, you'll have a fresh PostgreSQL database with up-to-date schema. -### Migrate data from MySQL to PostgreSQL - -Now, you can use pgloader to migrate the data from MySQL to PostgreSQL: +Next, we'll use `pgloader` to migrate the data from the old MySQL database to the +new PostgreSQL one: 1. Save the following snippet in a `commands.load` file, and edit with your MySQL `username`, `password` and `host`: - ``` - LOAD DATABASE - FROM mysql://username:password@host/gitlabhq_production - INTO postgresql://postgres@unix://var/run/postgresql:/gitlabhq_production + ``` + LOAD DATABASE + FROM mysql://username:password@host/gitlabhq_production + INTO postgresql://postgres@unix://var/run/postgresql:/gitlabhq_production - WITH include no drop, truncate, disable triggers, create no tables, - create no indexes, preserve index names, no foreign keys, - data only + WITH include no drop, truncate, disable triggers, create no tables, + create no indexes, preserve index names, no foreign keys, + data only - ALTER SCHEMA 'gitlabhq_production' RENAME TO 'public' + ALTER SCHEMA 'gitlabhq_production' RENAME TO 'public' - ; - ``` + ; + ``` 1. Start the migration: - ``` bash - sudo -u postgres pgloader commands.load - ``` + ```bash + sudo -u postgres pgloader commands.load + ``` 1. Once the migration finishes, you should see a summary table that looks like the following: - ``` - table name read imported errors total time - ----------------------------------------------- --------- --------- --------- -------------- - fetch meta data 119 119 0 0.388s - Truncate 119 119 0 1.134s - ----------------------------------------------- --------- --------- --------- -------------- - public.abuse_reports 0 0 0 0.490s - public.appearances 0 0 0 0.488s - . - . - . - public.web_hook_logs 0 0 0 1.080s - ----------------------------------------------- --------- --------- --------- -------------- - COPY Threads Completion 4 4 0 2.008s - Reset Sequences 113 113 0 0.304s - Install Comments 0 0 0 0.000s - ----------------------------------------------- --------- --------- --------- -------------- - Total import time 1894 1894 0 12.497s - ``` - - If there is no output for more than 30 minutes, it's possible pgloader encountered an error. See - the [troubleshooting guide](#Troubleshooting) for more details. + ``` + table name read imported errors total time + ----------------------------------------------- --------- --------- --------- -------------- + fetch meta data 119 119 0 0.388s + Truncate 119 119 0 1.134s + ----------------------------------------------- --------- --------- --------- -------------- + public.abuse_reports 0 0 0 0.490s + public.appearances 0 0 0 0.488s + . + . + . + public.web_hook_logs 0 0 0 1.080s + ----------------------------------------------- --------- --------- --------- -------------- + COPY Threads Completion 4 4 0 2.008s + Reset Sequences 113 113 0 0.304s + Install Comments 0 0 0 0.000s + ----------------------------------------------- --------- --------- --------- -------------- + Total import time 1894 1894 0 12.497s + ``` + + If there is no output for more than 30 minutes, it's possible `pgloader` encountered an error. See + the [troubleshooting guide](#troubleshooting) for more details. 1. Start GitLab: - ``` bash - sudo service gitlab start - ``` + ```bash + sudo service gitlab start + ``` -Now, you can verify that everything worked by visiting GitLab. +You can now verify that everything works as expected by visiting GitLab. -### Troubleshooting +## Troubleshooting -#### Experiencing 500 errors after the migration +Sometimes, you might encounter some errors during or after the migration. + +### Database error permission denied + +The PostgreSQL user that you use for the migration MUST have **superuser** privileges. +Otherwise, you may see a similar message to the following: + +``` +debugger invoked on a CL-POSTGRES-ERROR:INSUFFICIENT-PRIVILEGE in thread + #<THREAD "lparallel" RUNNING {10078A3513}>: + Database error 42501: permission denied: "RI_ConstraintTrigger_a_20937" is a system trigger + QUERY: ALTER TABLE ci_builds DISABLE TRIGGER ALL; + 2017-08-23T00:36:56.782000Z ERROR Database error 42501: permission denied: "RI_ConstraintTrigger_c_20864" is a system trigger + QUERY: ALTER TABLE approver_groups DISABLE TRIGGER ALL; +``` + +### Experiencing 500 errors after the migration If you experience 500 errors after the migration, try to clear the cache: -``` bash +```bash +# Omnibus GitLab +sudo gitlab-rake cache:clear + +# Installations from source sudo -u git -H bundle exec rake cache:clear RAILS_ENV=production ``` diff --git a/doc/update/patch_versions.md b/doc/update/patch_versions.md index 2e8380aa5d8..f2df4277ca8 100644 --- a/doc/update/patch_versions.md +++ b/doc/update/patch_versions.md @@ -67,7 +67,7 @@ sudo -u git -H bundle exec rake gettext:pack RAILS_ENV=production sudo -u git -H bundle exec rake gettext:po_to_json RAILS_ENV=production # Clean up assets and cache -sudo -u git -H bundle exec rake yarn:install gitlab:assets:clean gitlab:assets:compile cache:clear RAILS_ENV=production NODE_ENV=production +sudo -u git -H bundle exec rake yarn:install gitlab:assets:clean gitlab:assets:compile cache:clear RAILS_ENV=production NODE_ENV=production NODE_OPTIONS="--max_old_space_size=4096" ``` ### 4. Update gitlab-workhorse to the corresponding version @@ -106,14 +106,20 @@ sudo -u git -H git checkout v$(</home/git/gitlab/GITLAB_PAGES_VERSION) sudo -u git -H make ``` -### 8. Start application +### 8. Install/Update `gitlab-elasticsearch-indexer` (optional) **[STARTER ONLY]** + +If you're interested in using GitLab's new [elasticsearch repository indexer](../integration/elasticsearch.md#elasticsearch-repository-indexer-beta) (currently in beta) +please follow the instructions on the document linked above and enable the +indexer usage in the GitLab admin settings. + +### 9. Start application ```bash sudo service gitlab start sudo service nginx restart ``` -### 9. Check application status +### 10. Check application status Check if GitLab and its environment are configured correctly: diff --git a/doc/update/restore_after_failure.md b/doc/update/restore_after_failure.md index faa3a6e65cb..964a3c76c04 100644 --- a/doc/update/restore_after_failure.md +++ b/doc/update/restore_after_failure.md @@ -12,7 +12,7 @@ First, roll back the code or package. For source installations this involves checking out the older version (branch or tag). For Omnibus installations this means installing the older .deb or .rpm package. Then, restore from a backup. Follow the instructions in the -[Backup and Restore](../raketasks/backup_restore.md#restore-a-previously-created-backup) +[Backup and Restore](../raketasks/backup_restore.md#restore) documentation. ## Potential problems on the next upgrade diff --git a/doc/update/upgrading_from_ce_to_ee.md b/doc/update/upgrading_from_ce_to_ee.md index 0d1ecab5f8e..428377adb19 100644 --- a/doc/update/upgrading_from_ce_to_ee.md +++ b/doc/update/upgrading_from_ce_to_ee.md @@ -9,6 +9,10 @@ Community Edition to Enterprise Edition. These documents can be found in the [`doc/update` directory of Enterprise Edition's source code][old-ee-upgrade-docs]. +If you want to upgrade the version only, for example 11.8 to 11.9, *without* changing the +GitLab edition you are using (Community or Enterprise), see the +[Upgrading from source](upgrading_from_source.md) documentation. + ## General upgrading steps This guide assumes you have a correctly configured and tested installation of @@ -71,7 +75,7 @@ sudo -u git -H bundle exec rake assets:clean assets:precompile cache:clear RAILS ### 4. Install `gitlab-elasticsearch-indexer` (optional) **[STARTER ONLY]** If you're interested in using GitLab's new [elasticsearch repository -indexer][indexer-beta] (currently in beta) please follow the instructions on the +indexer](../integration/elasticsearch.md) (currently in beta) please follow the instructions on the document linked above and enable the indexer usage in the GitLab admin settings. ### 5. Start application @@ -129,4 +133,3 @@ Additional instructions here. [support@gitlab.com]: mailto:support@gitlab.com [old-ee-upgrade-docs]: https://gitlab.com/gitlab-org/gitlab-ee/tree/11-8-stable-ee/doc/update -[indexer-beta]: https://docs.gitlab.com/ee/integration/elasticsearch.html diff --git a/doc/update/upgrading_from_source.md b/doc/update/upgrading_from_source.md index 20d8ebecc0a..f82d666c7be 100644 --- a/doc/update/upgrading_from_source.md +++ b/doc/update/upgrading_from_source.md @@ -16,19 +16,16 @@ If the highest number stable branch is unclear please check the [GitLab Blog](https://about.gitlab.com/blog/archives.html) for installation guide links by version. +If you are changing from GitLab Community Edition to GitLab Enterprise Edition, see +the [Upgrading from CE to EE](upgrading_from_ce_to_ee.md) documentation. + ## Guidelines for all versions This section contains all the steps necessary to upgrade Community Edition or Enterprise Edition, regardless of the version you are upgrading to. Version specific guidelines (should there be any) are covered separately. -### 1. Stop server - -```bash -sudo service gitlab stop -``` - -### 2. Backup +### 1. Backup NOTE: If you installed GitLab from source, make sure `rsync` is installed. @@ -38,10 +35,16 @@ cd /home/git/gitlab sudo -u git -H bundle exec rake gitlab:backup:create RAILS_ENV=production ``` +### 2. Stop server + +```bash +sudo service gitlab stop +``` + ### 3. Update Ruby -NOTE: Beginning in GitLab 11.0, we only support Ruby 2.4 or higher, and dropped -support for Ruby 2.3. Be sure to upgrade if necessary. +NOTE: Beginning in GitLab 11.6, we only support Ruby 2.5 or higher, and dropped +support for Ruby 2.4. Be sure to upgrade if necessary. You can check which version you are running with `ruby -v`. @@ -49,9 +52,9 @@ Download Ruby and compile it: ```bash mkdir /tmp/ruby && cd /tmp/ruby -curl --remote-name --progress https://cache.ruby-lang.org/pub/ruby/2.5/ruby-2.5.3.tar.gz -echo 'f919a9fbcdb7abecd887157b49833663c5c15fda ruby-2.5.3.tar.gz' | shasum -c - && tar xzf ruby-2.5.3.tar.gz -cd ruby-2.5.3 +curl --remote-name --progress https://cache.ruby-lang.org/pub/ruby/2.6/ruby-2.6.3.tar.gz +echo '2347ed6ca5490a104ebd5684d2b9b5eefa6cd33c ruby-2.6.3.tar.gz' | shasum -c - && tar xzf ruby-2.6.3.tar.gz +cd ruby-2.6.3 ./configure --disable-install-rdoc make @@ -111,7 +114,47 @@ sudo ln -sf /usr/local/go/bin/{go,godoc,gofmt} /usr/local/bin/ rm go1.10.5.linux-amd64.tar.gz ``` -### 6. Get latest code +### 6. Update git + +NOTE: **Note:** +GitLab 11.11 and higher only supports Git 2.21.x and newer, and +[dropped support for older versions](https://gitlab.com/gitlab-org/gitlab-ce/issues/54255). +Be sure to upgrade your installation if necessary. + +```bash +# Make sure Git is version 2.21.0 or higher +git --version + +# Remove packaged Git +sudo apt-get remove git-core + +# Install dependencies +sudo apt-get install -y libcurl4-openssl-dev libexpat1-dev gettext libz-dev libssl-dev build-essential + +# Download and compile pcre2 from source +curl --silent --show-error --location https://ftp.pcre.org/pub/pcre/pcre2-10.33.tar.gz --output pcre2.tar.gz +tar -xzf pcre2.tar.gz +cd pcre2-10.33 +chmod +x configure +./configure --prefix=/usr --enable-jit +make +make install + +# Download and compile from source +cd /tmp +curl --remote-name --location --progress https://www.kernel.org/pub/software/scm/git/git-2.21.0.tar.gz +echo '85eca51c7404da75e353eba587f87fea9481ba41e162206a6f70ad8118147bee git-2.21.0.tar.gz' | shasum -a256 -c - && tar -xzf git-2.21.0.tar.gz +cd git-2.21.0/ +./configure --with-libpcre +make prefix=/usr/local all + +# Install into /usr/local/bin +sudo make prefix=/usr/local install + +# You should edit config/gitlab.yml, change the git -> bin_path to /usr/local/bin/git +``` + +### 7. Get latest code ```bash cd /home/git/gitlab @@ -139,7 +182,7 @@ cd /home/git/gitlab sudo -u git -H git checkout BRANCH-ee ``` -### 7. Update gitlab-shell +### 8. Update gitlab-shell ```bash cd /home/git/gitlab-shell @@ -149,7 +192,7 @@ sudo -u git -H git checkout v$(</home/git/gitlab/GITLAB_SHELL_VERSION) sudo -u git -H bin/compile ``` -### 8. Update gitlab-workhorse +### 9. Update gitlab-workhorse Install and compile gitlab-workhorse. GitLab-Workhorse uses [GNU Make](https://www.gnu.org/software/make/). @@ -164,7 +207,7 @@ sudo -u git -H git checkout v$(</home/git/gitlab/GITLAB_WORKHORSE_VERSION) sudo -u git -H make ``` -### 9. Update Gitaly +### 10. Update Gitaly #### Compile Gitaly @@ -175,7 +218,7 @@ sudo -u git -H git checkout v$(</home/git/gitlab/GITALY_SERVER_VERSION) sudo -u git -H make ``` -### 10. Update gitlab-pages +### 11. Update gitlab-pages #### Only needed if you use GitLab Pages @@ -192,7 +235,7 @@ sudo -u git -H git checkout v$(</home/git/gitlab/GITLAB_PAGES_VERSION) sudo -u git -H make ``` -### 11. Update MySQL permissions +### 12. Update MySQL permissions If you are using MySQL you need to grant the GitLab user the necessary permissions on the database: @@ -214,7 +257,7 @@ You can make this setting permanent by adding it to your `my.cnf`: log_bin_trust_function_creators=1 ``` -### 12. Update configuration files +### 13. Update configuration files #### New configuration options for `gitlab.yml` @@ -288,7 +331,7 @@ For Ubuntu 16.04.1 LTS: sudo systemctl daemon-reload ``` -### 13. Install libs, migrations, etc. +### 14. Install libs, migrations, etc. ```bash cd /home/git/gitlab @@ -311,7 +354,7 @@ sudo -u git -H bundle exec rake db:migrate RAILS_ENV=production sudo -u git -H bundle exec rake gettext:compile RAILS_ENV=production # Update node dependencies and recompile assets -sudo -u git -H bundle exec rake yarn:install gitlab:assets:clean gitlab:assets:compile RAILS_ENV=production NODE_ENV=production +sudo -u git -H bundle exec rake yarn:install gitlab:assets:clean gitlab:assets:compile RAILS_ENV=production NODE_ENV=production NODE_OPTIONS="--max_old_space_size=4096" # Clean up cache sudo -u git -H bundle exec rake cache:clear RAILS_ENV=production @@ -320,14 +363,14 @@ sudo -u git -H bundle exec rake cache:clear RAILS_ENV=production **MySQL installations**: Run through the `MySQL strings limits` and `Tables and data conversion to utf8mb4` [tasks](../install/database_mysql.md). -### 14. Start application +### 15. Start application ```bash sudo service gitlab start sudo service nginx restart ``` -### 15. Check application status +### 16. Check application status Check if GitLab and its environment are configured correctly: diff --git a/doc/user/account/security.md b/doc/user/account/security.md index f4078876fab..8a8edc23529 100644 --- a/doc/user/account/security.md +++ b/doc/user/account/security.md @@ -1 +1,5 @@ +--- +redirect_to: '../profile/account/index.md' +--- + This document was moved to [profile](../profile/account/index.md). diff --git a/doc/user/account/two_factor_authentication.md b/doc/user/account/two_factor_authentication.md index ea2c8307860..42a66becc50 100644 --- a/doc/user/account/two_factor_authentication.md +++ b/doc/user/account/two_factor_authentication.md @@ -1 +1,5 @@ +--- +redirect_to: '../profile/account/two_factor_authentication.md' +--- + This document was moved to [profile/account/two_factor_authentication](../profile/account/two_factor_authentication.md). diff --git a/doc/user/admin_area/diff_limits.md b/doc/user/admin_area/diff_limits.md index 9205860ef1f..4063c40a751 100644 --- a/doc/user/admin_area/diff_limits.md +++ b/doc/user/admin_area/diff_limits.md @@ -1,21 +1,40 @@ +--- +type: reference +--- + # Diff limits administration +You can set a maximum size for display of diff files (patches). + +## Maximum diff patch size + +Diff files which exceed this value will be presented as 'too large' and won't +be expandable. Instead of an expandable view, a link to the blob view will be +shown. + +Patches greater than 10% of this size will be automatically collapsed, and a +link to expand the diff will be presented. + NOTE: **Note:** Merge requests and branch comparison views will be affected. CAUTION: **Caution:** -These settings are currently under experimental state. They'll -increase the resource consumption of your instance and should -be edited mindfully. +This setting is experimental. An increased maximum will increase resource +consumption of your instance. Keep this in mind when adjusting the maximum. -1. Access **Admin area > Settings > General** -1. Expand **Diff limits** +1. Go to **Admin area > Settings > General**. +1. Expand **Diff limits**. +1. Enter a value for **Maximum diff patch size**, measured in bytes. +1. Click on **Save changes**. -### Maximum diff patch size +<!-- ## Troubleshooting -This is the content size each diff file (patch) is allowed to reach before -it's collapsed, without the possibility of being expanded. A link redirecting -to the blob view will be presented for the patches that surpass this limit. +Include any troubleshooting steps that you can foresee. If you know beforehand what issues +one might have when setting this up, or when something is changed, or on upgrading, it's +important to describe those, too. Think of things that may go wrong and include them here. +This is important to minimize requests for support, and to avoid doc comments with +questions that you know someone might ask. -Patches surpassing 10% of this content size will be automatically collapsed, -but expandable (a link to expand the diff will be presented). +Each scenario can be a third-level heading, e.g. `### Getting error message X`. +If you have none to add when creating a doc, leave this section in place +but commented out to help encourage others to add to it in the future. --> diff --git a/doc/user/admin_area/geo_nodes.md b/doc/user/admin_area/geo_nodes.md new file mode 100644 index 00000000000..d99b87cbc5c --- /dev/null +++ b/doc/user/admin_area/geo_nodes.md @@ -0,0 +1,86 @@ +--- +type: howto +--- + +# Geo nodes admin area **[PREMIUM ONLY]** + +You can configure various settings for GitLab Geo nodes. For more information, see +[Geo documentation](../../administration/geo/replication/index.md). + +On the primary node, go to **Admin area > Geo**. On secondary nodes, go to **Admin area > Geo > Nodes**. + +## Common settings + +All Geo nodes have the following settings: + +| Setting | Description | +| --------| ----------- | +| Primary | This marks a Geo Node as **primary** node. There can be only one **primary** node; make sure that you first add the **primary** node and then all the others. | +| Name | The unique identifier for the Geo node. Must match the setting `gitlab_rails[geo_node_name]` in `/etc/gitlab/gitlab.rb`. The setting defaults to `external_url` with a trailing slash. | +| URL | The instance's user-facing URL. | + +The node you're reading from is indicated with a green `Current node` label, and +the **primary** node is given a blue `Primary` label. Remember that you can only make +changes on the **primary** node! + +## **Secondary** node settings + +**Secondary** nodes have a number of additional settings available: + +| Setting | Description | +|---------------------------|-------------| +| Selective synchronization | Enable Geo [selective sync](../../administration/geo/replication/configuration.md#selective-synchronization) for this **secondary** node. | +| Repository sync capacity | Number of concurrent requests this **secondary** node will make to the **primary** node when backfilling repositories. | +| File sync capacity | Number of concurrent requests this **secondary** node will make to the **primary** node when backfilling files. | + +## Geo backfill + +**Secondary** nodes are notified of changes to repositories and files by the **primary** node, +and will always attempt to synchronize those changes as quickly as possible. + +Backfill is the act of populating the **secondary** node with repositories and files that +existed *before* the **secondary** node was added to the database. Since there may be +extremely large numbers of repositories and files, it's infeasible to attempt to +download them all at once, so GitLab places an upper limit on the concurrency of +these operations. + +How long the backfill takes is a function of the maximum concurrency, but higher +values place more strain on the **primary** node. From [GitLab 10.2](https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/3107), +the limits are configurable. If your **primary** node has lots of surplus capacity, +you can increase the values to complete backfill in a shorter time. If it's +under heavy load and backfill is reducing its availability for normal requests, +you can decrease them. + +## Using a different URL for synchronization + +The **primary** node's Internal URL is used by **secondary** nodes to contact it +(to sync repositories, for example). The name Internal URL distinguishes it from +[External URL](https://docs.gitlab.com/omnibus/settings/configuration.html#configuring-the-external-url-for-gitlab) +which is used by users. Internal URL does not need to be a private address. + +Internal URL defaults to External URL, but you can customize it under +**Admin area > Geo Nodes**. + +## Multiple secondary nodes behind a load balancer + +In GitLab 11.11, **secondary** nodes can use identical external URLs as long as +a unique `name` is set for each Geo node. The `gitlab.rb` setting +`gitlab_rails[geo_node_name]` must: + +- Be set for each GitLab instance that runs `unicorn`, `sidekiq`, or `geo_logcursor`. +- Match a Geo node name. + +The load balancer must use sticky sessions in order to avoid authentication +failures and cross site request errors. + +<!-- ## Troubleshooting + +Include any troubleshooting steps that you can foresee. If you know beforehand what issues +one might have when setting this up, or when something is changed, or on upgrading, it's +important to describe those, too. Think of things that may go wrong and include them here. +This is important to minimize requests for support, and to avoid doc comments with +questions that you know someone might ask. + +Each scenario can be a third-level heading, e.g. `### Getting error message X`. +If you have none to add when creating a doc, leave this section in place +but commented out to help encourage others to add to it in the future. -->
\ No newline at end of file diff --git a/doc/user/admin_area/img/admin_wrench.png b/doc/user/admin_area/img/admin_wrench.png Binary files differnew file mode 100644 index 00000000000..17eee143e87 --- /dev/null +++ b/doc/user/admin_area/img/admin_wrench.png diff --git a/doc/user/admin_area/img/index_runners_search_or_filter.png b/doc/user/admin_area/img/index_runners_search_or_filter.png Binary files differnew file mode 100644 index 00000000000..5176a1a39bf --- /dev/null +++ b/doc/user/admin_area/img/index_runners_search_or_filter.png diff --git a/doc/user/admin_area/img/license_admin_area.png b/doc/user/admin_area/img/license_admin_area.png Binary files differnew file mode 100644 index 00000000000..b5662b81c5e --- /dev/null +++ b/doc/user/admin_area/img/license_admin_area.png diff --git a/doc/user/admin_area/img/license_details.png b/doc/user/admin_area/img/license_details.png Binary files differnew file mode 100644 index 00000000000..2085bb437ad --- /dev/null +++ b/doc/user/admin_area/img/license_details.png diff --git a/doc/user/admin_area/img/license_upload.png b/doc/user/admin_area/img/license_upload.png Binary files differnew file mode 100644 index 00000000000..29d55175a2d --- /dev/null +++ b/doc/user/admin_area/img/license_upload.png diff --git a/doc/user/admin_area/index.md b/doc/user/admin_area/index.md index 00cea22e4e1..527110d53df 100644 --- a/doc/user/admin_area/index.md +++ b/doc/user/admin_area/index.md @@ -1,3 +1,7 @@ +--- +type: reference +--- + # GitLab Admin Area **[CORE ONLY]** The Admin Area provides a web UI for administering some features of GitLab self-managed instances. @@ -14,16 +18,191 @@ Only admin users can access the Admin Area. The Admin Area is made up of the following sections: -| Section | Description | -|:------------------|:---------------------------------------------------------------------------------------------------------------------------------------------------------| -| Overview | View your GitLab Dashboard, and maintain projects, users, groups, jobs, runners, and Gitaly servers. | -| Monitoring | View GitLab system information, and information on background jobs, logs, [health checks](monitoring/health_check.md), request profiles, and audit logs. | -| Messages | Send and manage [broadcast messages](broadcast_messages.md) for your users. | -| System Hooks | Configure [system hooks](../../system_hooks/system_hooks.md) for many events. | -| Applications | Create system [OAuth applications](../../integration/oauth_provider.md) for integrations with other services. | -| Abuse Reports | Manage [abuse reports](abuse_reports.md) submitted by your users. | -| Deploy Keys | Create instance-wide [SSH deploy keys](../../ssh/README.md#deploy-keys). | -| Service Templates | Create [service templates](../project/integrations/services_templates.md) for projects. | -| Labels | Create and maintain [labels](labels.md) for your GitLab instance. | -| Appearance | Customize [GitLab's appearance](../../customization/index.md). | -| Settings | Modify the [settings](settings/index.md) for your GitLab instance. | +| Section | Description | +|:---------------------------|:---------------------------------------------------------------------------------------------------------------------------------------------------------| +| Overview | View your GitLab [Dashboard](#admin-dashboard), and administer [projects](#administer-projects), [users](#administer-users), groups, [jobs](#administer-jobs), [Runners](#administer-runners), and [Gitaly servers](#administer-gitaly-servers). | +| Monitoring | View GitLab system information, and information on background jobs, logs, [health checks](monitoring/health_check.md), request profiles, and audit logs. | +| Messages | Send and manage [broadcast messages](broadcast_messages.md) for your users. | +| System Hooks | Configure [system hooks](../../system_hooks/system_hooks.md) for many events. | +| Applications | Create system [OAuth applications](../../integration/oauth_provider.md) for integrations with other services. | +| Abuse Reports | Manage [abuse reports](abuse_reports.md) submitted by your users. | +| License **[STARTER ONLY]** | Upload, display, and remove [licenses](license.md). | +| Push Rules **[STARTER]** | Configure pre-defined git [push rules](../../push_rules/push_rules.md) for projects. | +| Geo **[PREMIUM ONLY]** | Configure and maintain [Geo nodes](geo_nodes.md). | +| Deploy Keys | Create instance-wide [SSH deploy keys](../../ssh/README.md#deploy-keys). | +| Service Templates | Create [service templates](../project/integrations/services_templates.md) for projects. | +| Labels | Create and maintain [labels](labels.md) for your GitLab instance. | +| Appearance | Customize [GitLab's appearance](../../customization/index.md). | +| Settings | Modify the [settings](settings/index.md) for your GitLab instance. | + +## Admin Dashboard + +The Dashboard provides statistics and system information about the GitLab instance. + +To access the Dashboard, either: + +- Click the Admin Area icon (the wrench icon). +- Visit `/admin` on your self-managed instance. + +The Dashboard is the default view of the Admin Area, and is made up of the following sections: + +| Section | Description | +|------------|---------------| +| Projects | The total number of projects, up to 10 of the latest projects, and the option of creating a new project. | +| Users | The total number of users, up to 10 of the latest users, and the option of creating a new user. | +| Groups | The total number of groups, up to 10 of the latest groups, and the option of creating a new group. | +| Statistics | Totals of all elements of the GitLab instance. | +| Features | All features available on the GitLab instance. Enabled features are marked with a green circle icon, and disabled features are marked with a power icon. | +| Components | The major components of GitLab and the version number of each. A link to the Gitaly Servers is also included. | + +## Administer Projects + +You can administer all projects in the GitLab instance from the Admin Area's Projects page. + +To access the Projects page, go to **Admin Area > Overview > Projects**. + +Click the **All**, **Private**, **Internal**, or **Public** tab to list only projects of that +criteria. + +By default, all projects are listed, in reverse order of when they were last updated. For each +project, the following information is listed: + +- Name. +- Namespace. +- Description. +- Size, updated every 15 minutes at most. + +Projects can be edited or deleted. + +The list of projects can be sorted by: + +- Name. +- Last created. +- Oldest created. +- Last updated. +- Oldest updated. +- Owner. + +A user can choose to hide or show archived projects in the list. + +In the **Filter by name** field, type the project name you want to find, and GitLab will filter +them as you type. + +Select from the **Namespace** dropdown to filter only projects in that namespace. + +You can combine the filter options. For example, to list only public projects with `score` in their name: + +1. Click the **Public** tab. +1. Enter `score` in the **Filter by name...** input box. + +## Administer Users + +You can administer all users in the GitLab instance from the Admin Area's Users page. + +To access the Users page, go to **Admin Area > Overview > Users**. + +Click the **Active**, **Admins**, **2FA Enabled**, or **2FA Disabled**, **External**, or +**Without projects** tab to list only users of that criteria. + +For each user, their username, email address, are listed, also the date their account was +created and the date of last activity. To edit a user, click the **Edit** button in that user's +row. To delete the user, or delete the user and their contributions, click the cog dropdown in +that user's row, and select the desired option. + +To change the sort order: + +1. Click the sort dropdown. +1. Select the desired order. + +By default the sort dropdown shows **Name**. + +To search for users, enter your criteria in the search field. The user search is case +insensitive, and applies partial matching to name and username. To search for an email address, +you must provide the complete email address. + +## Administer Jobs + +You can administer all jobs in the GitLab instance from the Admin Area's Jobs page. + +To access the Jobs page, go to **Admin Area > Overview > Jobs**. + +All jobs are listed, in reverse order of their job ID. + +Click the **All** tab to list all jobs. Click the **Pending**, **Running**, or **Finished** tab to list only jobs of that status. + +For each job, the following details are listed: + +| Field | Description | +|--------- | ----------- | +| Status | Job status, either **passed**, **skipped**, or **failed**. | +| Job | Includes links to the job, branch, and the commit that started the job. | +| Pipeline | Includes a link to the specific pipeline. | +| Project | Name of the project, and organization, to which the job belongs. | +| Runner | Name of the CI runner assigned to execute the job. | +| Stage | Stage that the job is declared in a `.gitlab-ci.yml` file. | +| Name | Name of the job specified in a `.gitlab-ci.yml` file. | +| Timing | Duration of the job, and how long ago the job completed. | +| Coverage | Percentage of tests coverage. | + +## Administer Runners + +You can adminster all Runners in the GitLab instance from the Admin Area's **Runners** page. See +[GitLab Runner](https://docs.gitlab.com/runner/) for more information on Runner itself. + +To access the **Runners** page, go to **Admin Area > Overview > Runners**. + +The **Runners** page features: + +- A description of Runners, and their possible states. +- Instructions on installing a Runner. +- A list of all registered Runners. + +Runners are listed in descending order by the date they were created, by default. You can change +the sort order to *Last Contacted* from the dropdown beside the search field. + +To search Runners' descriptions: + +1. In the **Search or filter results...** field, type the description of the Runner you want to +find. +1. Press Enter. + +You can also filter Runners by status, type, and tag. To filter: + +1. Click in the **Search or filter results...** field. +1. Select **status:**, **type:**, or **tag:** +1. Select or enter your search criteria. + + + +For each Runner, the following attributes are listed: + +| Attribute | Description | +| ------------ | ----------- | +| Type | One or more of the following states: shared, group, specific, locked, or paused | +| Runner token | Token used to identify the Runner, and which the Runner uses to communicate with the GitLab instance | +| Description | Description given to the Runner when it was created | +| Version | GitLab Runner version | +| IP address | IP address of the host on which the Runner is registered | +| Projects | Projects to which the Runner is assigned | +| Jobs | Total of jobs run by the Runner | +| Tags | Tags associated with the Runner | +| Last contact | Timestamp indicating when the GitLab instance last contacted the Runner | + +You can also edit, pause, or remove each Runner. + +## Administer Gitaly servers + +You can list all Gitaly servers in the GitLab instance from the Admin Area's **Gitaly Servers** +page. For more details, see [Gitaly](../../administration/gitaly/index.md). + +To access the **Gitaly Servers** page, go to **Admin Area > Overview > Gitaly Servers**. + +For each Gitaly server, the following details are listed: + +| Field | Description | +| -------------- | ----------- | +| Storage | Repository storage | +| Address | Network address on which the Gitaly server is listening | +| Server version | Gitaly version | +| Git version | Version of Git installed on the Gitaly server | +| Up to date | Indicates if the Gitaly server version is the latest version available. A green dot indicates the server is up to date. | diff --git a/doc/user/admin_area/labels.md b/doc/user/admin_area/labels.md index e383142c33e..eba27548f86 100644 --- a/doc/user/admin_area/labels.md +++ b/doc/user/admin_area/labels.md @@ -1,9 +1,25 @@ +--- +type: reference +--- + # Labels administration **[CORE ONLY]** -## Default Labels +In the Admin Area, you can manage labels for the GitLab instance. For more details, see [Labels](../project/labels.md). -### Define your own default Label Set +## Default Labels -Labels that are created within the Labels view on the Admin Dashboard will be automatically added to each new project. +Labels created in the Admin Area become available to each _new_ project.  + +<!-- ## Troubleshooting + +Include any troubleshooting steps that you can foresee. If you know beforehand what issues +one might have when setting this up, or when something is changed, or on upgrading, it's +important to describe those, too. Think of things that may go wrong and include them here. +This is important to minimize requests for support, and to avoid doc comments with +questions that you know someone might ask. + +Each scenario can be a third-level heading, e.g. `### Getting error message X`. +If you have none to add when creating a doc, leave this section in place +but commented out to help encourage others to add to it in the future. --> diff --git a/doc/user/admin_area/license.md b/doc/user/admin_area/license.md new file mode 100644 index 00000000000..1e8ce04da92 --- /dev/null +++ b/doc/user/admin_area/license.md @@ -0,0 +1,126 @@ +--- +type: howto +--- + +# Activate all GitLab Enterprise Edition functionality with a license **[STARTER ONLY]** + +To activate all GitLab Enterprise Edition (EE) functionality, you need to upload +a license. Once you've received your license from GitLab Inc., you can upload it +by **signing into your GitLab instance as an admin** or add it at +installation time. + +The license has the form of a base64 encoded ASCII text with a `.gitlab-license` +extension and can be obtained when you [purchase one][pricing] or when you sign +up for a [free trial]. + +NOTE: **Note:** +As of GitLab Enterprise Edition 9.4.0, a newly-installed instance without an +uploaded license will only have the Core features active. A trial license will +activate all Ultimate features, but after +[the trial expires](#what-happens-when-your-license-expires), some functionality +will be locked. + +## Uploading your license + +The very first time you visit your GitLab EE installation signed in as an admin, +you should see a note urging you to upload a license with a link that takes you +straight to the License admin area. + +Otherwise, you can: + +1. Navigate manually to the **Admin Area** by clicking the wrench icon in the menu bar. + +  + +1. And then going to the **License** tab and click on **Upload New License**. + +  + +1. If you've received a `.gitlab-license` file, you should have already downloaded + it in your local machine. You can then upload it directly by choosing the + license file and clicking the **Upload license** button. In the image below, + you can see that the selected license file is named `GitLab.gitlab-license`. + +  + + If you've received your license as plain text, you need to select the + "Enter license key" option, copy the license, paste it into the "License key" + field and click **Upload license**. + +## Add your license at install time + +The license may be automatically injected during installation using one of +two methods. + +The first requires a license file named `Gitlab.gitlab-release`. + +Place it in the `config/` directory if installing from source or in the +`/etc/gitlab/` directory if installing Omnibus. + +The second allows the administrator to configure the location and +filename of the license. + +Source installations should set the `GITLAB_LICENSE_FILE` environment +variable with the path to a valid GitLab Enterprise Edition license. + +```sh +export GITLAB_LICENSE_FILE="/path/to/license/file" +``` + +Omnibus installations should add this entry to `gitlab.rb`: + +```ruby +gitlab_rails['license_file'] = "/path/to/license/file" +``` + +CAUTION:: **Caution:** +These methods will only add a license at the time of installation. Use the +admin area in the web ui to renew or upgrade licenses. + +--- + +Once the license is uploaded, all GitLab Enterprise Edition functionality +will be active until the end of the license period. When that period ends, the +instance will [fall back](#what-happens-when-your-license-expires) to Core-only +functionality. + +You can review the license details at any time in the License section of the +Admin Area. + + + +## Notification before the license expires + +One month before the license expires, a message informing when the expiration +is due to, will start appearing to GitLab admins. Make sure that you update your +license, otherwise you will miss all the paid features if it expires. + +## What happens when your license expires + +In case your license expires, GitLab will lock down some features like Git pushes, +issue creation, etc., and a message to inform of the expired license will be +presented to all admins. + +In order to get back all the previous functionality, a new license must be uploaded. +To fall back to having only the Core features active, you'll need to delete the +expired license(s). + +## License history + +It's possible to upload and view more than one license, +but only the latest license will be used as the active license. + +[free trial]: https://about.gitlab.com/free-trial/ +[pricing]: https://about.gitlab.com/pricing/ + +<!-- ## Troubleshooting + +Include any troubleshooting steps that you can foresee. If you know beforehand what issues +one might have when setting this up, or when something is changed, or on upgrading, it's +important to describe those, too. Think of things that may go wrong and include them here. +This is important to minimize requests for support, and to avoid doc comments with +questions that you know someone might ask. + +Each scenario can be a third-level heading, e.g. `### Getting error message X`. +If you have none to add when creating a doc, leave this section in place +but commented out to help encourage others to add to it in the future. -->
\ No newline at end of file diff --git a/doc/user/admin_area/monitoring/health_check.md b/doc/user/admin_area/monitoring/health_check.md index c22982ac190..ff056490653 100644 --- a/doc/user/admin_area/monitoring/health_check.md +++ b/doc/user/admin_area/monitoring/health_check.md @@ -1,12 +1,14 @@ +--- +type: concepts, howto +--- + # Health Check -> **Notes:** -> - Liveness and readiness probes were [introduced][ce-10416] in GitLab 9.1. -> - The `health_check` endpoint was [introduced][ce-3888] in GitLab 8.8 and will -> be deprecated in GitLab 9.1. Read more in the [old behavior](#old-behavior) -> section. -> - [Access token](#access-token) has been deprecated in GitLab 9.4 -> in favor of [IP whitelist](#ip-whitelist) +> - Liveness and readiness probes were [introduced][ce-10416] in GitLab 9.1. +> - The `health_check` endpoint was [introduced][ce-3888] in GitLab 8.8 and was +> be deprecated in GitLab 9.1. +> - [Access token](#access-token-deprecated) has been deprecated in GitLab 9.4 +> in favor of [IP whitelist](#ip-whitelist). GitLab provides liveness and readiness probes to indicate service health and reachability to required services. These probes report on the status of the @@ -16,31 +18,28 @@ traffic until the system is ready or restart the container as needed. ## IP whitelist -To access monitoring resources, the client IP needs to be included in a whitelist. - -[Read how to add IPs to a whitelist for the monitoring endpoints][admin]. +To access monitoring resources, the requesting client IP needs to be included in a whitelist. +For details, see [how to add IPs to a whitelist for the monitoring endpoints](../../../administration/monitoring/ip_whitelist.md). ## Using the endpoints -With default whitelist settings, the probes can be accessed from localhost: +With default whitelist settings, the probes can be accessed from localhost using the following URLs: - `http://localhost/-/health` - `http://localhost/-/readiness` - `http://localhost/-/liveness` -The first endpoint, `/-/health/`, only checks whether the application server is running. It does --not verify the database or other services are running. A successful response will return -a 200 status code with the following message: +The first endpoint, `health`, only checks whether the application server is running. It does not verify the database or other services are running. A successful response will return a 200 status code with the following message: -``` +```text GitLab OK ``` The readiness and liveness probes will provide a report of system health in JSON format. -Readiness example output: +`readiness` probe example output: -``` +```json { "queues_check" : { "status" : "ok" @@ -60,9 +59,9 @@ Readiness example output: } ``` -Liveness example output: +`liveness` probe example output: -``` +```json { "cache_check" : { "status" : "ok" @@ -89,9 +88,8 @@ will return a valid successful HTTP status code, and a `success` message. ## Access token (Deprecated) ->**Note:** -Access token has been deprecated in GitLab 9.4 -in favor of [IP whitelist](#ip-whitelist) +> NOTE: **Note:** +> Access token has been deprecated in GitLab 9.4 in favor of [IP whitelist](#ip-whitelist). An access token needs to be provided while accessing the probe endpoints. The current accepted token can be found under the **Admin area ➔ Monitoring ➔ Health check** @@ -101,14 +99,25 @@ accepted token can be found under the **Admin area ➔ Monitoring ➔ Health che The access token can be passed as a URL parameter: -``` +```text https://gitlab.example.com/-/readiness?token=ACCESS_TOKEN ``` +<!-- ## Troubleshooting + +Include any troubleshooting steps that you can foresee. If you know beforehand what issues +one might have when setting this up, or when something is changed, or on upgrading, it's +important to describe those, too. Think of things that may go wrong and include them here. +This is important to minimize requests for support, and to avoid doc comments with +questions that you know someone might ask. + +Each scenario can be a third-level heading, e.g. `### Getting error message X`. +If you have none to add when creating a doc, leave this section in place +but commented out to help encourage others to add to it in the future. --> + [ce-10416]: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/10416 [ce-3888]: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/3888 [pingdom]: https://www.pingdom.com [nagios-health]: https://nagios-plugins.org/doc/man/check_http.html [newrelic-health]: https://docs.newrelic.com/docs/alerts/alert-policies/downtime-alerts/availability-monitoring [kubernetes]: https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-probes/ -[admin]: ../../../administration/monitoring/ip_whitelist.md diff --git a/doc/user/admin_area/settings/account_and_limit_settings.md b/doc/user/admin_area/settings/account_and_limit_settings.md new file mode 100644 index 00000000000..001e4b6bf48 --- /dev/null +++ b/doc/user/admin_area/settings/account_and_limit_settings.md @@ -0,0 +1,62 @@ +--- +type: reference +--- + +# Account and limit settings + +## Repository size limit **[STARTER]** + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/740) in [GitLab Enterprise Edition 8.12](https://about.gitlab.com/2016/09/22/gitlab-8-12-released/#limit-project-size-ee). +> Available in [GitLab Starter](https://about.gitlab.com/pricing/). + +Repositories within your GitLab instance can grow quickly, especially if you are +using LFS. Their size can grow exponentially, rapidly consuming available storage. + +To avoid this from happening, you can set a hard limit for your repositories' size. +This limit can be set globally, per group, or per project, with per project limits +taking the highest priority. + +There are numerous use cases where you might set up a limit for repository size. +For instance, consider the following workflow: + +1. Your team develops apps which require large files to be stored in + the application repository. +1. Although you have enabled [Git LFS](../../../workflow/lfs/manage_large_binaries_with_git_lfs.md#git-lfs) + to your project, your storage has grown significantly. +1. Before you exceed available storage, you set up a limit of 10 GB + per repository. + +### How it works + +Only a GitLab administrator can set those limits. Setting the limit to `0` means +there are no restrictions. + +These settings can be found within: + +- Each project's settings. +- A group's settings. +- The **Size limit per repository (MB)** field in the **Account and limit** section of a GitLab instance's + settings by navigating to either: + - **Admin Area > Settings > General**. + - The path `/admin/application_settings`. + +The first push of a new project, including LFS objects, will be checked for size +and **will** be rejected if the sum of their sizes exceeds the maximum allowed +repository size. + +For details on manually purging files, see [reducing the repository size using Git](../../project/repository/reducing_the_repo_size_using_git.md). + +NOTE: **Note:** +For GitLab.com, the repository size limit is 10 GB. + +<!-- ## Troubleshooting + +Include any troubleshooting steps that you can foresee. If you know beforehand what issues +one might have when setting this up, or when something is changed, or on upgrading, it's +important to describe those, too. Think of things that may go wrong and include them here. +This is important to minimize requests for support, and to avoid doc comments with +questions that you know someone might ask. + +Each scenario can be a third-level heading, e.g. `### Getting error message X`. +If you have none to add when creating a doc, leave this section in place +but commented out to help encourage others to add to it in the future. --> diff --git a/doc/user/admin_area/settings/continuous_integration.md b/doc/user/admin_area/settings/continuous_integration.md index 01979f12a01..6c4abce83c2 100644 --- a/doc/user/admin_area/settings/continuous_integration.md +++ b/doc/user/admin_area/settings/continuous_integration.md @@ -1,3 +1,7 @@ +--- +type: reference +--- + # Continuous Integration and Deployment Admin settings **[CORE ONLY]** In this area, you will find settings for Auto DevOps, Runners and job artifacts. @@ -10,8 +14,8 @@ You can find it in the admin area, under **Settings > Continuous Integration and To enable (or disable) [Auto DevOps](../../../topics/autodevops/index.md) for all projects: -1. Go to **Admin area > Settings > Continuous Integration and Deployment**. -1. Check (or uncheck to disable) the box that says "Default to Auto DevOps pipeline for all projects". +1. Go to **Admin area > Settings > Continuous Integration and Deployment** +1. Check (or uncheck to disable) the box that says "Default to Auto DevOps pipeline for all projects" 1. Optionally, set up the [Auto DevOps base domain](../../../topics/autodevops/index.md#auto-devops-base-domain) which is going to be used for Auto Deploy and Auto Review Apps. 1. Hit **Save changes** for the changes to take effect. @@ -20,13 +24,13 @@ From now on, every existing project and newly created ones that don't have a `.gitlab-ci.yml`, will use the Auto DevOps pipelines. If you want to disable it for a specific project, you can do so in -[its settings](../../../topics/autodevops/index.md##enablingdisabling-auto-devops). +[its settings](../../../topics/autodevops/index.md#enablingdisabling-auto-devops). ## Maximum artifacts size **[CORE ONLY]** -The maximum size of the [job artifacts][art-yml] can be set in the Admin area -of your GitLab instance. The value is in *MB* and the default is 100MB per job; -on GitLab.com it's [set to 1G](../../gitlab_com/index.md#gitlab-ci-cd). +The maximum size of the [job artifacts](../../../administration/job_artifacts.md) +can be set in the Admin area of your GitLab instance. The value is in *MB* and +the default is 100MB per job; on GitLab.com it's [set to 1G](../../gitlab_com/index.md#gitlab-cicd). To change it: @@ -40,7 +44,7 @@ The default expiration time of the [job artifacts](../../../administration/job_a can be set in the Admin area of your GitLab instance. The syntax of duration is described in [`artifacts:expire_in`](../../../ci/yaml/README.md#artifactsexpire_in) and the default value is `30 days`. On GitLab.com they -[never expire](../../gitlab_com/index.md#gitlab-ci-cd). +[never expire](../../gitlab_com/index.md#gitlab-cicd). 1. Go to **Admin area > Settings > Continuous Integration and Deployment**. 1. Change the value of default expiration time. @@ -50,6 +54,86 @@ This setting is set per job and can be overridden in [`.gitlab-ci.yml`](../../../ci/yaml/README.md#artifactsexpire_in). To disable the expiration, set it to `0`. The default unit is in seconds. +## Shared Runners pipeline minutes quota **[STARTER ONLY]** + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/1078) +in GitLab Starter 8.16. + +If you have enabled shared Runners for your GitLab instance, you can limit their +usage by setting a maximum number of pipeline minutes that a group can use on +shared Runners per month. Setting this to `0` (default value) will grant +unlimited pipeline minutes. While build limits are stored as minutes, the +counting is done in seconds. Usage resets on the first day of each month. +On GitLab.com, the quota is calculated based on your +[subscription plan](https://about.gitlab.com/pricing/#gitlab-com). + +To change the pipelines minutes quota: + +1. Go to **Admin area > Settings > Continuous Integration and Deployment** +1. Set the pipeline minutes quota limit. +1. Hit **Save changes** for the changes to take effect + +--- + +While the setting in the Admin area has a global effect, as an admin you can +also change each group's pipeline minutes quota to override the global value. + +1. Navigate to the **Groups** admin area and hit the **Edit** button for the + group you wish to change the pipeline minutes quota. +1. Set the pipeline minutes quota to the desired value +1. Hit **Save changes** for the changes to take effect. + +Once saved, you can see the build quota in the group admin view. +The quota can also be viewed in the project admin view if shared Runners +are enabled. + + + +When the pipeline minutes quota for a group is set to a value different than 0, +the **Pipelines quota** page is available to the group page settings list. +You can see there an overview of the pipeline minutes quota of all projects of +the group. + + + + +## Extra Shared Runners pipeline minutes quota + +NOTE: **Note:** +Only available on GitLab.com. + +You can purchase additional CI minutes so your pipelines will not be blocked after you have +used all your CI minutes from your main quota. + +In order to purchase additional minutes, you should follow these steps: + +1. Go to **Group > Settings > Pipelines quota**. Once you are on that page, click on **Buy additional minutes**. + +  + +1. Locate the subscription card that is linked to your group on GitLab.com, +click on **Buy more CI minutes**, and complete the details about the transaction. + +  + +1. Once we have processed your payment, the extra CI minutes +will be synced to your Group and you can visualize it from the +**Group > Settings > Pipelines quota** page: + +  + +Be aware that: + +1. If you have purchased extra CI minutes before the purchase of a paid plan, +we will calculate a pro-rated charge for your paid plan. That means you may +be charged for less than one year since your subscription was previously +created with the extra CI minutes. +1. Once the extra CI minutes has been assigned to a Group they cannot be transferred +to a different Group. +1. If you have some minutes used over your default quota, these minutes will +be deducted from your Additional Minutes quota immediately after your purchase of additional +minutes. + ## Archive jobs **[CORE ONLY]** Archiving jobs is useful for reducing the CI/CD footprint on the system by @@ -65,3 +149,15 @@ To set the duration for which the jobs will be considered as old and expired: Once that time passes, the jobs will be archived and no longer able to be retried. Make it empty to never expire jobs. It has to be no less than 1 day, for example: <code>15 days</code>, <code>1 month</code>, <code>2 years</code>. + +<!-- ## Troubleshooting + +Include any troubleshooting steps that you can foresee. If you know beforehand what issues +one might have when setting this up, or when something is changed, or on upgrading, it's +important to describe those, too. Think of things that may go wrong and include them here. +This is important to minimize requests for support, and to avoid doc comments with +questions that you know someone might ask. + +Each scenario can be a third-level heading, e.g. `### Getting error message X`. +If you have none to add when creating a doc, leave this section in place +but commented out to help encourage others to add to it in the future. -->
\ No newline at end of file diff --git a/doc/user/admin_area/settings/email.md b/doc/user/admin_area/settings/email.md index 50c318a4969..9555a695b13 100644 --- a/doc/user/admin_area/settings/email.md +++ b/doc/user/admin_area/settings/email.md @@ -1,15 +1,37 @@ +--- +type: reference +--- + # Email +You can customize some of the content in emails sent from your GitLab instance. + ## Custom logo The logo in the header of some emails can be customized, see the [logo customization section](../../../customization/branded_page_and_email_header.md). +## Custom additional text **[PREMIUM ONLY]** + +> [Introduced][ee-5031] in [GitLab Premium][eep] 10.7. + +The additional text will appear at the bottom of any email and can be used for +legal/auditing/compliance reasons. + +1. Go to **Admin area > Settings** (`/admin/application_settings`). +1. Under the **Email** section, change the **Additional text** field. +1. Hit **Save** for the changes to take effect. + + + +[ee-5031]: https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/5031 +[eep]: https://about.gitlab.com/pricing/ + ## Custom hostname for private commit emails > [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/22560) in GitLab 11.5. -This configuration option sets the email hostname for [private commit emails](../../profile/index.md#private-commit-email), -and it's, by default, set to `users.noreply.YOUR_CONFIGURED_HOSTNAME`. +This configuration option sets the email hostname for [private commit emails](../../profile/index.md#private-commit-email). + By default it is set to `users.noreply.YOUR_CONFIGURED_HOSTNAME`. In order to change this option: @@ -18,5 +40,17 @@ In order to change this option: 1. Hit **Save** for the changes to take effect. NOTE: **Note**: Once the hostname gets configured, every private commit email using the previous hostname, will not get -recognized by GitLab. This can directly conflict with certain [Push rules](https://docs.gitlab.com/ee/push_rules/push_rules.html) such as +recognized by GitLab. This can directly conflict with certain [Push rules](../../../push_rules/push_rules.md) such as `Check whether author is a GitLab user` and `Check whether committer is the current authenticated user`. + +<!-- ## Troubleshooting + +Include any troubleshooting steps that you can foresee. If you know beforehand what issues +one might have when setting this up, or when something is changed, or on upgrading, it's +important to describe those, too. Think of things that may go wrong and include them here. +This is important to minimize requests for support, and to avoid doc comments with +questions that you know someone might ask. + +Each scenario can be a third-level heading, e.g. `### Getting error message X`. +If you have none to add when creating a doc, leave this section in place +but commented out to help encourage others to add to it in the future. -->
\ No newline at end of file diff --git a/doc/user/admin_area/settings/external_authorization.md b/doc/user/admin_area/settings/external_authorization.md new file mode 100644 index 00000000000..11c0867da17 --- /dev/null +++ b/doc/user/admin_area/settings/external_authorization.md @@ -0,0 +1,128 @@ +--- +type: reference +--- + +# External authorization control **[CORE ONLY]** + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/issues/4216) in +> [GitLab Premium](https://about.gitlab.com/pricing) 10.6. +> [Moved](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/27056) to +> [GitLab Core](https://about.gitlab.com/pricing/) in 11.10. + +In highly controlled environments, it may be necessary for access policy to be +controlled by an external service that permits access based on project +classification and user access. GitLab provides a way to check project +authorization with your own defined service. + +## Overview + +Once the external service is configured and enabled, when a project is accessed, +a request is made to the external service with the user information and project +classification label assigned to the project. When the service replies with a +known response, the result is cached for 6 hours. + +If the external authorization is enabled, GitLab will further block pages and +functionality that render cross-project data. That includes: + +- most pages under Dashboard (Activity, Milestones, Snippets, Assigned merge + requests, Assigned issues, Todos) +- under a specific group (Activity, Contribution analytics, Issues, Issue boards, + Labels, Milestones, Merge requests) +- Global and Group search will be disabled + +This is to prevent performing to many requests at once to the external +authorization service. + +Whenever access is granted or denied this is logged in a logfile called +`external-policy-access-control.log`. +Read more about logs GitLab keeps in the [omnibus documentation][omnibus-log-docs]. + +## Configuration + +The external authorization service can be enabled by an admin on the GitLab's +admin area under the settings page: + + + +The available required properties are: + +- **Service URL**: The URL to make authorization requests to. When leaving the + URL blank, cross project features will remain available while still being able + to specify classification labels for projects. +- **External authorization request timeout**: The timeout after which an + authorization request is aborted. When a request times out, access is denied + to the user. +- **Client authentication certificate**: The certificate to use to authenticate + with the external authorization service. +- **Client authentication key**: Private key for the certificate when + authentication is required for the external authorization service, this is + encrypted when stored. +- **Client authentication key password**: Passphrase to use for the private key when authenticating with the external service this is encrypted when stored. +- **Default classification label**: The classification label to use when + requesting authorization if no specific label is defined on the project + +When using TLS Authentication with a self signed certificate, the CA certificate +needs to be trused by the openssl installation. When using GitLab installed using +Omnibus, learn to install a custom CA in the +[omnibus documentation][omnibus-ssl-docs]. Alternatively learn where to install +custom certificates using `openssl version -d`. + +## How it works + +When GitLab requests access, it will send a JSON POST request to the external +service with this body: + +```json +{ + "user_identifier": "jane@acme.org", + "project_classification_label": "project-label", + "user_ldap_dn": "CN=Jane Doe,CN=admin,DC=acme" +} +``` + +The `user_ldap_dn` is optional and is only sent when the user is logged in +through LDAP. + +When the external authorization service responds with a status code 200, the +user is granted access. When the external service responds with a status code +401 or 403, the user is denied access. In any case, the request is cached for 6 hours. + +When denying access, a `reason` can be optionally specified in the JSON body: + +```json +{ + "reason": "You are not allowed access to this project." +} +``` + +Any other status code than 200, 401 or 403 will also deny access to the user, but the +response will not be cached. + +If the service times out (after 500ms), a message "External Policy Server did +not respond" will be displayed. + +## Classification labels + +You can use your own classification label in the project's +**Settings > General > General project settings** page in the "Classification +label" box. When no classification label is specified on a project, the default +label defined in the [global settings](#configuration) will be used. + +The label will be shown on all project pages in the upper right corner. + + + +<!-- ## Troubleshooting + +Include any troubleshooting steps that you can foresee. If you know beforehand what issues +one might have when setting this up, or when something is changed, or on upgrading, it's +important to describe those, too. Think of things that may go wrong and include them here. +This is important to minimize requests for support, and to avoid doc comments with +questions that you know someone might ask. + +Each scenario can be a third-level heading, e.g. `### Getting error message X`. +If you have none to add when creating a doc, leave this section in place +but commented out to help encourage others to add to it in the future. --> + +[omnibus-ssl-docs]: https://docs.gitlab.com/omnibus/settings/ssl.html +[omnibus-log-docs]: https://docs.gitlab.com/omnibus/settings/logs.html diff --git a/doc/user/admin_area/settings/img/additional_minutes.png b/doc/user/admin_area/settings/img/additional_minutes.png Binary files differnew file mode 100644 index 00000000000..d148ed79b92 --- /dev/null +++ b/doc/user/admin_area/settings/img/additional_minutes.png diff --git a/doc/user/admin_area/settings/img/admin_area_default_artifacts_expiration.png b/doc/user/admin_area/settings/img/admin_area_default_artifacts_expiration.png Binary files differdeleted file mode 100644 index 723be23e77b..00000000000 --- a/doc/user/admin_area/settings/img/admin_area_default_artifacts_expiration.png +++ /dev/null diff --git a/doc/user/admin_area/settings/img/admin_area_maximum_artifacts_size.png b/doc/user/admin_area/settings/img/admin_area_maximum_artifacts_size.png Binary files differdeleted file mode 100644 index 3f827f1f7a3..00000000000 --- a/doc/user/admin_area/settings/img/admin_area_maximum_artifacts_size.png +++ /dev/null diff --git a/doc/user/admin_area/settings/img/admin_project_quota_view.png b/doc/user/admin_area/settings/img/admin_project_quota_view.png Binary files differnew file mode 100644 index 00000000000..8320be860da --- /dev/null +++ b/doc/user/admin_area/settings/img/admin_project_quota_view.png diff --git a/doc/user/admin_area/settings/img/buy_btn.png b/doc/user/admin_area/settings/img/buy_btn.png Binary files differnew file mode 100644 index 00000000000..0cc88b8a48f --- /dev/null +++ b/doc/user/admin_area/settings/img/buy_btn.png diff --git a/doc/user/admin_area/settings/img/buy_minutes_card.png b/doc/user/admin_area/settings/img/buy_minutes_card.png Binary files differnew file mode 100644 index 00000000000..cf4ad34ead7 --- /dev/null +++ b/doc/user/admin_area/settings/img/buy_minutes_card.png diff --git a/doc/user/admin_area/settings/img/classification_label_on_project_page.png b/doc/user/admin_area/settings/img/classification_label_on_project_page.png Binary files differnew file mode 100644 index 00000000000..4aedb332cec --- /dev/null +++ b/doc/user/admin_area/settings/img/classification_label_on_project_page.png diff --git a/doc/user/admin_area/settings/img/email_settings.png b/doc/user/admin_area/settings/img/email_settings.png Binary files differnew file mode 100644 index 00000000000..ed0a80d10ce --- /dev/null +++ b/doc/user/admin_area/settings/img/email_settings.png diff --git a/doc/user/admin_area/settings/img/external_authorization_service_settings.png b/doc/user/admin_area/settings/img/external_authorization_service_settings.png Binary files differnew file mode 100644 index 00000000000..9b8658fd1a1 --- /dev/null +++ b/doc/user/admin_area/settings/img/external_authorization_service_settings.png diff --git a/doc/user/admin_area/settings/img/file_template_admin_area.png b/doc/user/admin_area/settings/img/file_template_admin_area.png Binary files differnew file mode 100644 index 00000000000..269d997e1d9 --- /dev/null +++ b/doc/user/admin_area/settings/img/file_template_admin_area.png diff --git a/doc/user/admin_area/settings/img/file_template_user_dropdown.png b/doc/user/admin_area/settings/img/file_template_user_dropdown.png Binary files differnew file mode 100644 index 00000000000..8c9eb49f6c9 --- /dev/null +++ b/doc/user/admin_area/settings/img/file_template_user_dropdown.png diff --git a/doc/user/admin_area/settings/img/group_pipelines_quota.png b/doc/user/admin_area/settings/img/group_pipelines_quota.png Binary files differnew file mode 100644 index 00000000000..d94b609ad6f --- /dev/null +++ b/doc/user/admin_area/settings/img/group_pipelines_quota.png diff --git a/doc/user/admin_area/settings/img/mirror_settings.png b/doc/user/admin_area/settings/img/mirror_settings.png Binary files differnew file mode 100644 index 00000000000..090db6808a7 --- /dev/null +++ b/doc/user/admin_area/settings/img/mirror_settings.png diff --git a/doc/user/admin_area/settings/index.md b/doc/user/admin_area/settings/index.md index 8358fe64f18..0e697b9e445 100644 --- a/doc/user/admin_area/settings/index.md +++ b/doc/user/admin_area/settings/index.md @@ -6,6 +6,7 @@ instance like sign-up restrictions, account limits and quota, metrics, etc. Navigate to it by going to **Admin area > Settings**. Some of the settings include: +- [Account and limit settings](account_and_limit_settings.md) **[STARTER]** - [Continuous Integration and Deployment](continuous_integration.md) - [Email](email.md) - [Sign up restrictions](sign_up_restrictions.md) @@ -13,6 +14,7 @@ include: - [Third party offers](third_party_offers.md) - [Usage statistics](usage_statistics.md) - [Visibility and access controls](visibility_and_access_controls.md) +- [Custom templates repository](instance_template_repository.md) **[PREMIUM]** NOTE: **Note:** You can change the [first day of the week](../../profile/preferences.md) for the entire GitLab instance diff --git a/doc/user/admin_area/settings/instance_template_repository.md b/doc/user/admin_area/settings/instance_template_repository.md new file mode 100644 index 00000000000..91286a67c31 --- /dev/null +++ b/doc/user/admin_area/settings/instance_template_repository.md @@ -0,0 +1,79 @@ +--- +type: reference +--- + +# Instance template repository **[PREMIUM ONLY]** + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/issues/5986) in +> [GitLab Premium](https://about.gitlab.com/pricing) 11.3. + +## Overview + +In hosted systems, enterprises often have a need to share their own templates +across teams. This feature allows an administrator to pick a project to be the +instance-wide collection of file templates. These templates are then exposed to +all users [via the web editor](../../project/repository/web_editor.md#template-dropdowns) +while the project remains secure. + +## Configuration + +As an administrator, navigate to **Admin area > Settings > Templates** and +select the project to serve as the custom template repository. + + + +Once a project has been selected, you can add custom templates to the repository, +and they will appear in the appropriate places in the +[frontend](../../project/repository/web_editor.md#template-dropdowns) and +[API](../../../api/settings.md). + +Templates must be added to a specific subdirectory in the repository, +corresponding to the kind of template. The following types of custom templates +are supported: + +| Type | Directory | Extension | +| :---------------: | :-----------: | :-----------: | +| `Dockerfile` | `Dockerfile` | `.dockerfile` | +| `.gitignore` | `gitignore` | `.gitignore` | +| `.gitlab-ci.yml` | `gitlab-ci` | `.yml` | +| `LICENSE` | `LICENSE` | `.txt` | + +Each template must go in its respective subdirectory, have the correct +extension and not be empty. So, the hierarchy should look like this: + +```text +|-- README.md +|-- Dockerfile + |-- custom_dockerfile.dockerfile + |-- another_dockerfile.dockerfile +|-- gitignore + |-- custom_gitignore.gitignore + |-- another_gitignore.gitignore +|-- gitlab-ci + |-- custom_gitlab-ci.yml + |-- another_gitlab-ci.yml +|-- LICENSE + |-- custom_license.txt + |-- another_license.txt +``` + +Once this is established, the list of custom templates will be included when +creating a new file and the template type is selected. These will appear at the +top of the list. + + + +If this feature is disabled or no templates are present, there will be +no "Custom" section in the selection dropdown. + +<!-- ## Troubleshooting + +Include any troubleshooting steps that you can foresee. If you know beforehand what issues +one might have when setting this up, or when something is changed, or on upgrading, it's +important to describe those, too. Think of things that may go wrong and include them here. +This is important to minimize requests for support, and to avoid doc comments with +questions that you know someone might ask. + +Each scenario can be a third-level heading, e.g. `### Getting error message X`. +If you have none to add when creating a doc, leave this section in place +but commented out to help encourage others to add to it in the future. --> diff --git a/doc/user/admin_area/settings/sign_up_restrictions.md b/doc/user/admin_area/settings/sign_up_restrictions.md index d3ecfd42903..cebf36c7ec1 100644 --- a/doc/user/admin_area/settings/sign_up_restrictions.md +++ b/doc/user/admin_area/settings/sign_up_restrictions.md @@ -1,3 +1,7 @@ +--- +type: reference +--- + # Sign-up restrictions You can block email addresses of specific domains, or whitelist only some @@ -37,5 +41,17 @@ semicolon, comma, or a new line.  +<!-- ## Troubleshooting + +Include any troubleshooting steps that you can foresee. If you know beforehand what issues +one might have when setting this up, or when something is changed, or on upgrading, it's +important to describe those, too. Think of things that may go wrong and include them here. +This is important to minimize requests for support, and to avoid doc comments with +questions that you know someone might ask. + +Each scenario can be a third-level heading, e.g. `### Getting error message X`. +If you have none to add when creating a doc, leave this section in place +but commented out to help encourage others to add to it in the future. --> + [ce-5259]: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/5259 [ce-598]: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/598 diff --git a/doc/user/admin_area/settings/terms.md b/doc/user/admin_area/settings/terms.md index e2290bf0598..a5f8d05f662 100644 --- a/doc/user/admin_area/settings/terms.md +++ b/doc/user/admin_area/settings/terms.md @@ -1,29 +1,35 @@ +--- +type: reference +--- + # Enforce accepting Terms of Service > [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/18570) > in [GitLab Core](https://about.gitlab.com/pricing/) 10.8 +An admin can enforce acceptance of a terms of service and privacy policy. When this option is enabled, new and existing users must accept the terms. + ## Configuration -When it is required for all users of the GitLab instance to accept the -Terms of Service, this can be configured by an admin on the settings -page: +To enforce acceptance of a Terms of Service and Privacy Policy: -. +1. Log in to the GitLab instance as an admin user. +1. Go to **Admin Area > Settings > General**. +1. Expand the **Terms of Service and Privacy Policy** section. +1. Check the **Require all users to accept Terms of Service and Privacy Policy when they access +GitLab.** checkbox. +1. Input the text of the **Terms of Service and Privacy Policy**. Markdown formatting can be used in this input box. +1. Click **Save changes**. +1. When you are presented with the **Terms of Service** statement, click **Accept terms**. -The terms itself can be entered using Markdown. For each update to the -terms, a new version is stored. When a user accepts or declines the -terms, GitLab will keep track of which version they accepted or -declined. +. -When an admin enables this feature, they will automattically be -directed to the page to accept the terms themselves. After they -accept, they will be directed back to the settings page. +For each update to the terms, a new version is stored. When a user accepts or declines the terms, +GitLab will record which version they accepted or declined. -## New registrations +## New users -When this feature is enabled, a checkbox will be available in the -sign-up form. +When this feature is enabled, a checkbox is added to the sign-up form.  @@ -49,3 +55,15 @@ If the user was already logged in when the feature was turned on, they will be asked to accept the terms on their next interaction. If a user declines the terms, they will be signed out. + +<!-- ## Troubleshooting + +Include any troubleshooting steps that you can foresee. If you know beforehand what issues +one might have when setting this up, or when something is changed, or on upgrading, it's +important to describe those, too. Think of things that may go wrong and include them here. +This is important to minimize requests for support, and to avoid doc comments with +questions that you know someone might ask. + +Each scenario can be a third-level heading, e.g. `### Getting error message X`. +If you have none to add when creating a doc, leave this section in place +but commented out to help encourage others to add to it in the future. --> diff --git a/doc/user/admin_area/settings/third_party_offers.md b/doc/user/admin_area/settings/third_party_offers.md index 23311801790..d3c9cf7d8ff 100644 --- a/doc/user/admin_area/settings/third_party_offers.md +++ b/doc/user/admin_area/settings/third_party_offers.md @@ -1,9 +1,26 @@ +--- +type: reference +--- + # Third party offers > [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/20379) > in [GitLab Core](https://about.gitlab.com/pricing/) 11.1 -Within GitLab, we inform users of available third-party offers they might find valuable in order to enhance the development of their projects. -An example is the Google Cloud Platform free credit for using [Google Kubernetes Engine](https://cloud.google.com/kubernetes-engine/). +Within GitLab, we inform users of available third-party offers they might find valuable in order +to enhance the development of their projects. An example is the Google Cloud Platform free credit +for using [Google Kubernetes Engine](https://cloud.google.com/kubernetes-engine/). + +The display of third-party offers can be toggled in the **Admin Area > Settings** page. + +<!-- ## Troubleshooting + +Include any troubleshooting steps that you can foresee. If you know beforehand what issues +one might have when setting this up, or when something is changed, or on upgrading, it's +important to describe those, too. Think of things that may go wrong and include them here. +This is important to minimize requests for support, and to avoid doc comments with +questions that you know someone might ask. -The display of third-party offers can be toggled in the Admin area on the Settings page. +Each scenario can be a third-level heading, e.g. `### Getting error message X`. +If you have none to add when creating a doc, leave this section in place +but commented out to help encourage others to add to it in the future. --> diff --git a/doc/user/admin_area/settings/usage_statistics.md b/doc/user/admin_area/settings/usage_statistics.md index e165a120162..652d6ad2cdd 100644 --- a/doc/user/admin_area/settings/usage_statistics.md +++ b/doc/user/admin_area/settings/usage_statistics.md @@ -1,10 +1,14 @@ +--- +type: reference +--- + # Usage statistics GitLab Inc. will periodically collect information about your instance in order to perform various actions. All statistics are opt-out, you can enable/disable them from the admin panel -under **Admin area > Settings > Usage statistics**. +under **Admin area > Settings > Metrics and profiling > Usage statistics**. ## Version check **[CORE ONLY]** @@ -48,6 +52,8 @@ You can view the exact JSON payload in the administration panel. To view the pay 1. Expand **Settings** in the left sidebar and click on **Metrics and profiling**. 1. Expand **Usage statistics** and click on the **Preview payload** button. +You can see how [the usage ping data maps to different stages of the product](https://gitlab.com/gitlab-data/analytics/blob/master/transform/snowflake-dbt/data/ping_metrics_to_stage_mapping_data.csv). + ### Deactivate the usage ping The usage ping is opt-out. If you want to deactivate this feature, go to @@ -81,6 +87,18 @@ of your instance to your users. This can be restricted to admins by selecting "Only admins" in the Instance Statistics visibility section under **Admin area > Settings > Usage statistics**. +<!-- ## Troubleshooting + +Include any troubleshooting steps that you can foresee. If you know beforehand what issues +one might have when setting this up, or when something is changed, or on upgrading, it's +important to describe those, too. Think of things that may go wrong and include them here. +This is important to minimize requests for support, and to avoid doc comments with +questions that you know someone might ask. + +Each scenario can be a third-level heading, e.g. `### Getting error message X`. +If you have none to add when creating a doc, leave this section in place +but commented out to help encourage others to add to it in the future. --> + [ee-557]: https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/557 [ee-735]: https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/735 [ce-23361]: https://gitlab.com/gitlab-org/gitlab-ce/issues/23361 diff --git a/doc/user/admin_area/settings/visibility_and_access_controls.md b/doc/user/admin_area/settings/visibility_and_access_controls.md index 6a1e8004f87..4d1b7d0f252 100644 --- a/doc/user/admin_area/settings/visibility_and_access_controls.md +++ b/doc/user/admin_area/settings/visibility_and_access_controls.md @@ -49,5 +49,15 @@ block access to the server itself. The ports used for the protocol, be it SSH or HTTP, will still be accessible. What GitLab does is restrict access on the application level. +## Allow mirrors to be set up for projects + +> [Introduced][ee-3586] in GitLab 10.3. + +This option is enabled by default. By disabling it, both pull and push mirroring will no longer +work in every repository and can only be re-enabled on a per-project basis by an admin. + + + [ce-4696]: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/4696 [ce-18021]: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/18021 +[ee-3586]: https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/3586 diff --git a/doc/user/application_security/container_scanning/img/container_scanning.png b/doc/user/application_security/container_scanning/img/container_scanning.png Binary files differnew file mode 100644 index 00000000000..e47f62acd9d --- /dev/null +++ b/doc/user/application_security/container_scanning/img/container_scanning.png diff --git a/doc/user/application_security/container_scanning/index.md b/doc/user/application_security/container_scanning/index.md new file mode 100644 index 00000000000..adb6868516e --- /dev/null +++ b/doc/user/application_security/container_scanning/index.md @@ -0,0 +1,204 @@ +# Container Scanning **[ULTIMATE]** + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/3672) +in [GitLab Ultimate](https://about.gitlab.com/pricing/) 10.4. + +## Overview + +If you are using [GitLab CI/CD](../../../ci/README.md), you can check your Docker +images (or more precisely the containers) for known vulnerabilities by using +[Clair](https://github.com/coreos/clair) and [clair-scanner](https://github.com/arminc/clair-scanner), +two open source tools for Vulnerability Static Analysis for containers. + +You can take advantage of Container Scanning by either [including the CI job](#including-the-provided-template) in +your existing `.gitlab-ci.yml` file or by implicitly using +[Auto Container Scanning](../../../topics/autodevops/index.md#auto-container-scanning) +that is provided by [Auto DevOps](../../../topics/autodevops/index.md). + +GitLab checks the Container Scanning report, compares the found vulnerabilities +between the source and target branches, and shows the information right on the +merge request. + + + +## Use cases + +If you distribute your application with Docker, then there's a great chance +that your image is based on other Docker images that may in turn contain some +known vulnerabilities that could be exploited. + +Having an extra job in your pipeline that checks for those vulnerabilities, +and the fact that they are displayed inside a merge request, makes it very easy +to perform audits for your Docker-based apps. + +## Requirements + +To enable Container Scanning in your pipeline, you need: + +- A GitLab Runner with the + [`docker`](https://docs.gitlab.com/runner/executors/docker.html#use-docker-in-docker-with-privileged-mode) or + [`kubernetes`](https://docs.gitlab.com/runner/install/kubernetes.html#running-privileged-containers-for-the-runners) + executor running in privileged mode. If you're using the shared Runners on GitLab.com, + this is enabled by default. +- To [build and push](../../../ci/docker/using_docker_build.md#container-registry-examples) + your Docker image to your project's [Container Registry](../../project/container_registry.md). + The name of the Docker image should match the following scheme: + + ``` + $CI_REGISTRY_IMAGE/$CI_COMMIT_REF_SLUG:$CI_COMMIT_SHA + ``` + + The variables above can be found in the + [predefined environment variables](../../../ci/variables/predefined_variables.md) + document. + +## Configuring Container Scanning + +To enable Container Scanning in your project, define a job in your +`.gitlab-ci.yml` file that generates the +[Container Scanning report artifact](../../../ci/yaml/README.md#artifactsreportscontainer_scanning-ultimate). + +This can be done in two ways: + +- For GitLab 11.9 and later, including the provided + `Container-Scanning.gitlab-ci.yml` template (recommended). +- Manually specifying the job definition. Not recommended unless using GitLab + 11.8 and earlier. + +### Including the provided template + +NOTE: **Note:** +The CI/CD Container Scanning template is supported on GitLab 11.9 and later versions. +For earlier versions, use the [manual job definition](#manual-job-definition-for-gitlab-115-and-later). + +A CI/CD [Container Scanning template](https://gitlab.com/gitlab-org/gitlab-ee/blob/master/lib/gitlab/ci/templates/Security/Container-Scanning.gitlab-ci.yml) +with the default Container Scanning job definition is provided as a part of your GitLab +installation that you can [include](../../../ci/yaml/README.md#includetemplate) +in your `.gitlab-ci.yml` file. + +To enable Container Scanning using the provided template, add the following to +your `.gitlab-ci.yml` file: + +```yaml +include: + template: Container-Scanning.gitlab-ci.yml +``` + +The included template will: + +- Create a `container_scanning` job in your CI/CD pipeline. +- Pull the already built Docker image from your project's + [Container Registry](../../project/container_registry.md) (see [requirements](#requirements)) + and scan it for possible vulnerabilities. + +The report will be saved as a +[Container Scanning report artifact](../../../ci/yaml/README.md#artifactsreportscontainer_scanning-ultimate) +that you can later download and analyze. +Due to implementation limitations, we always take the latest Container Scanning +artifact available. Behind the scenes, the +[GitLab Container Scanning analyzer](https://gitlab.com/gitlab-org/security-products/container-scanning) +is used and runs the scans. + +If you want to whitelist some specific vulnerabilities, you can do so by defining +them in a YAML file named `clair-whitelist.yml`. Read more in the +[Clair documentation](https://github.com/arminc/clair-scanner/blob/master/README.md#example-whitelist-yaml-file). + +### Manual job definition for GitLab 11.5 and later + +CAUTION: **Caution:** +The job definition shown below is supported on GitLab 11.5 and later versions. +However, if you're using GitLab 11.9+, it's recommended to use +[the provided Container Scanning template](#including-the-provided-template). + +For GitLab 11.5 and GitLab Runner 11.5 and later, the following `container_scanning` +job can be added: + +```yaml +container_scanning: + image: docker:stable + variables: + DOCKER_DRIVER: overlay2 + ## Define two new variables based on GitLab's CI/CD predefined variables + ## https://docs.gitlab.com/ee/ci/variables/README.html#predefined-environment-variables + CI_APPLICATION_REPOSITORY: $CI_REGISTRY_IMAGE/$CI_COMMIT_REF_SLUG + CI_APPLICATION_TAG: $CI_COMMIT_SHA + CLAIR_LOCAL_SCAN_VERSION: v2.0.8_fe9b059d930314b54c78f75afe265955faf4fdc1 + allow_failure: true + services: + - docker:stable-dind + script: + - docker run -d --name db arminc/clair-db:latest + - docker run -p 6060:6060 --link db:postgres -d --name clair --restart on-failure arminc/clair-local-scan:${CLAIR_LOCAL_SCAN_VERSION} + - apk add -U wget ca-certificates + - docker pull ${CI_APPLICATION_REPOSITORY}:${CI_APPLICATION_TAG} + - wget https://github.com/arminc/clair-scanner/releases/download/v8/clair-scanner_linux_amd64 + - mv clair-scanner_linux_amd64 clair-scanner + - chmod +x clair-scanner + - touch clair-whitelist.yml + - while( ! wget -q -O /dev/null http://docker:6060/v1/namespaces ) ; do sleep 1 ; done + - retries=0 + - echo "Waiting for clair daemon to start" + - while( ! wget -T 10 -q -O /dev/null http://docker:6060/v1/namespaces ) ; do sleep 1 ; echo -n "." ; if [ $retries -eq 10 ] ; then echo " Timeout, aborting." ; exit 1 ; fi ; retries=$(($retries+1)) ; done + - ./clair-scanner -c http://docker:6060 --ip $(hostname -i) -r gl-container-scanning-report.json -l clair.log -w clair-whitelist.yml ${CI_APPLICATION_REPOSITORY}:${CI_APPLICATION_TAG} || true + artifacts: + reports: + container_scanning: gl-container-scanning-report.json +``` + +### Manual job definition for GitLab 11.4 and earlier (deprecated) + +CAUTION: **Deprecated:** +Before GitLab 11.5, the Container Scanning job and artifact had to be named specifically +to automatically extract report data and show it in the merge request widget. +While these old job definitions are still maintained, they have been deprecated +and may be removed in the next major release, GitLab 12.0. You are strongly +advised to update your current `.gitlab-ci.yml` configuration to reflect that change. + +For GitLab 11.4 and earlier, the Container Scanning job should look like: + +```yaml +container_scanning: + image: docker:stable + variables: + DOCKER_DRIVER: overlay2 + ## Define two new variables based on GitLab's CI/CD predefined variables + ## https://docs.gitlab.com/ee/ci/variables/README.html#predefined-environment-variables + CI_APPLICATION_REPOSITORY: $CI_REGISTRY_IMAGE/$CI_COMMIT_REF_SLUG + CI_APPLICATION_TAG: $CI_COMMIT_SHA + CLAIR_LOCAL_SCAN_VERSION: v2.0.8_fe9b059d930314b54c78f75afe265955faf4fdc1 + allow_failure: true + services: + - docker:stable-dind + script: + - docker run -d --name db arminc/clair-db:latest + - docker run -p 6060:6060 --link db:postgres -d --name clair --restart on-failure arminc/clair-local-scan:${CLAIR_LOCAL_SCAN_VERSION} + - apk add -U wget ca-certificates + - docker pull ${CI_APPLICATION_REPOSITORY}:${CI_APPLICATION_TAG} + - wget https://github.com/arminc/clair-scanner/releases/download/v8/clair-scanner_linux_amd64 + - mv clair-scanner_linux_amd64 clair-scanner + - chmod +x clair-scanner + - touch clair-whitelist.yml + - while( ! wget -q -O /dev/null http://docker:6060/v1/namespaces ) ; do sleep 1 ; done + - retries=0 + - echo "Waiting for clair daemon to start" + - while( ! wget -T 10 -q -O /dev/null http://docker:6060/v1/namespaces ) ; do sleep 1 ; echo -n "." ; if [ $retries -eq 10 ] ; then echo " Timeout, aborting." ; exit 1 ; fi ; retries=$(($retries+1)) ; done + - ./clair-scanner -c http://docker:6060 --ip $(hostname -i) -r gl-container-scanning-report.json -l clair.log -w clair-whitelist.yml ${CI_APPLICATION_REPOSITORY}:${CI_APPLICATION_TAG} || true + artifacts: + paths: [gl-container-scanning-report.json] +``` + +Alternatively, the job name could be `sast:container` +and the artifact name could be `gl-sast-container-report.json`. +These names have been deprecated with GitLab 11.0 +and may be removed in the next major release, GitLab 12.0. + +## Security Dashboard + +The Security Dashboard is a good place to get an overview of all the security +vulnerabilities in your groups and projects. Read more about the +[Security Dashboard](../security_dashboard/index.md). + +## Interacting with the vulnerabilities + +Once a vulnerability is found, you can interact with it. Read more on how to +[interact with the vulnerabilities](../index.md#interacting-with-the-vulnerabilities). diff --git a/doc/user/application_security/dast/img/dast_all.png b/doc/user/application_security/dast/img/dast_all.png Binary files differnew file mode 100644 index 00000000000..b6edc928dc3 --- /dev/null +++ b/doc/user/application_security/dast/img/dast_all.png diff --git a/doc/user/application_security/dast/img/dast_single.png b/doc/user/application_security/dast/img/dast_single.png Binary files differnew file mode 100644 index 00000000000..26ca4bde786 --- /dev/null +++ b/doc/user/application_security/dast/img/dast_single.png diff --git a/doc/user/application_security/dast/index.md b/doc/user/application_security/dast/index.md new file mode 100644 index 00000000000..028ff72a160 --- /dev/null +++ b/doc/user/application_security/dast/index.md @@ -0,0 +1,261 @@ +# Dynamic Application Security Testing (DAST) **[ULTIMATE]** + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/issues/4348) +in [GitLab Ultimate](https://about.gitlab.com/pricing/) 10.4. + +Running [static checks](../sast/index.md) on your code is the first step to detect +vulnerabilities that can put the security of your code at risk. Yet, once +deployed, your application is exposed to a new category of possible attacks, +such as cross-site scripting or broken authentication flaws. This is where +Dynamic Application Security Testing (DAST) comes into place. + +## Overview + +If you are using [GitLab CI/CD](../../../ci/README.md), you can analyze your running web application(s) +for known vulnerabilities using Dynamic Application Security Testing (DAST). + +You can take advantage of DAST by either [including the CI job](#configuring-dast) in +your existing `.gitlab-ci.yml` file or by implicitly using +[Auto DAST](../../../topics/autodevops/index.md#auto-dast-ultimate) +that is provided by [Auto DevOps](../../../topics/autodevops/index.md). + +GitLab checks the DAST report, compares the found vulnerabilities between the source and target +branches, and shows the information right on the merge request. + + + +By clicking on one of the detected linked vulnerabilities, you will be able to +see the details and the URL(s) affected. + + + +[Dynamic Application Security Testing (DAST)](https://en.wikipedia.org/wiki/Dynamic_Application_Security_Testing) +is using the popular open source tool [OWASP ZAProxy](https://github.com/zaproxy/zaproxy) +to perform an analysis on your running web application. + +By default, DAST executes [ZAP Baseline Scan](https://github.com/zaproxy/zaproxy/wiki/ZAP-Baseline-Scan) and will perform passive scanning only. It will not actively attack your application. + +However, DAST can be [configured](#full-scan) +to also perform a so-called "active scan". That is, attack your application and produce a more extensive security report. +It can be very useful combined with [Review Apps](../../../ci/review_apps/index.md). + +The [`dast`](https://gitlab.com/gitlab-org/security-products/dast/container_registry) Docker image in GitLab container registry is updated on a weekly basis to have all [`owasp2docker-weekly`](https://hub.docker.com/r/owasp/zap2docker-weekly/) updates in it. + +## Use cases + +It helps you automatically find security vulnerabilities in your running web +applications while you are developing and testing your applications. + +## Requirements + +To run a DAST job, you need GitLab Runner with the +[`docker` executor](https://docs.gitlab.com/runner/executors/docker.html). + +## Configuring DAST + +To enable DAST in your project, define a job in your `.gitlab-ci.yml` file that generates the +[DAST report artifact](../../../ci/yaml/README.md#artifactsreportsdast-ultimate). + +This can be done in two ways: + +- For GitLab 11.9 and later, including the provided DAST `.gitlab-ci.yml` template (recommended). +- Manually specifying the job definition. Not recommended unless using GitLab + 11.8 and earlier. + +### Including the provided template + +NOTE: **Note:** +The CI/CD DAST template is supported on GitLab 11.9 and later versions. +For earlier versions, use the [manual job definition](#manual-job-definition-for-gitlab-115-and-later). + +A CI/CD [DAST template](https://gitlab.com/gitlab-org/gitlab-ee/blob/master/lib/gitlab/ci/templates/Security/DAST.gitlab-ci.yml) +with the default DAST job definition is provided as a part of your GitLab +installation which you can [include](../../../ci/yaml/README.md#includetemplate) +in your `.gitlab-ci.yml` file. + +To enable DAST using the provided template, add the following to your `.gitlab-ci.yml` +file: + +```yaml +include: + template: DAST.gitlab-ci.yml + +variables: + DAST_WEBSITE: https://example.com +``` + +The included template will create a `dast` job in your CI/CD pipeline and scan +your project's source code for possible vulnerabilities. + +The report will be saved as a +[DAST report artifact](../../../ci/yaml/README.md#artifactsreportsdast-ultimate) +that you can later download and analyze. Due to implementation limitations we +always take the latest DAST artifact available. Behind the scenes, the +[GitLab DAST Docker image](https://gitlab.com/gitlab-org/security-products/dast) +is used to run the tests on the specified URL and scan it for possible vulnerabilities. + +There are two ways to define the URL to be scanned by DAST: + +- Set the `DAST_WEBSITE` [variable](../../../ci/yaml/README.md#variables). +- Add it in an `environment_url.txt` file at the root of your project. + +#### Authenticated scan + +It's also possible to authenticate the user before performing the DAST checks: + +```yaml +include: + template: DAST.gitlab-ci.yml + +variables: + DAST_WEBSITE: https://example.com + DAST_AUTH_URL: https://example.com/sign-in + DAST_USERNAME: john.doe@example.com + DAST_PASSWORD: john-doe-password + DAST_USERNAME_FIELD: session[user] # the name of username field at the sign-in HTML form + DAST_PASSWORD_FIELD: session[password] # the name of password field at the sign-in HTML form + DAST_AUTH_EXCLUDE_URLS: http://example.com/sign-out,http://example.com/sign-out-2 # optional, URLs to skip during the authenticated scan; comma-separated, no spaces in between +``` + +The report will be saved as a +[DAST report artifact](../../../ci/yaml/README.md#artifactsreportsdast-ultimate) +that you can later download and analyze. +Due to implementation limitations, we always take the latest DAST artifact available. + +#### Full scan + +DAST can be configured to perform [ZAP Full Scan](https://github.com/zaproxy/zaproxy/wiki/ZAP-Full-Scan), which +includes both passive and active scanning against the same target website: + +```yaml +include: + template: DAST.gitlab-ci.yml + +variables: + DAST_FULL_SCAN_ENABLED: "true" +``` + +#### Customizing the DAST settings + +The DAST settings can be changed through environment variables by using the +[`variables`](../../../ci/yaml/README.md#variables) parameter in `.gitlab-ci.yml`. +These variables are documented in the [DAST README](https://gitlab.com/gitlab-org/security-products/dast#settings). + +For example: + +```yaml +include: + template: DAST.gitlab-ci.yml + +variables: + DAST_WEBSITE: https://example.com + DAST_TARGET_AVAILABILITY_TIMEOUT: 120 +``` + +Because the template is [evaluated before](../../../ci/yaml/README.md#include) the pipeline +configuration, the last mention of the variable will take precedence. + +#### Overriding the DAST template + +If you want to override the job definition (for example, change properties like +`variables` or `dependencies`), you need to declare a `dast` job after the +template inclusion and specify any additional keys under it. For example: + +```yaml +include: + template: DAST.gitlab-ci.yml + +dast: + stage: dast # IMPORTANT: don't forget to add this + variables: + DAST_WEBSITE: https://example.com + CI_DEBUG_TRACE: "true" +``` + +As the DAST job belongs to a separate `dast` stage that runs after all +[default stages](../../../ci/yaml/README.md#stages), +don't forget to add `stage: dast` when you override the template job definition. + +### Manual job definition for GitLab 11.5 and later + +For GitLab 11.5 and GitLab Runner 11.5 and later, the following `dast` +job can be added: + +```yaml +dast: + image: registry.gitlab.com/gitlab-org/security-products/zaproxy + variables: + website: "https://example.com" + allow_failure: true + script: + - mkdir /zap/wrk/ + - /zap/zap-baseline.py -J gl-dast-report.json -t $website || true + - cp /zap/wrk/gl-dast-report.json . + artifacts: + reports: + dast: gl-dast-report.json +``` + +Where the `website` variable holds the URL to run the tests against. + +For an authenticated scan, use the following definition: + +```yaml +dast: + image: registry.gitlab.com/gitlab-org/security-products/zaproxy + variables: + website: "https://example.com" + login_url: "https://example.com/sign-in" + username: "john.doe@example.com" + password: "john-doe-password" + allow_failure: true + script: + - mkdir /zap/wrk/ + - /zap/zap-baseline.py -J gl-dast-report.json -t $website + --auth-url $login_url + --auth-username $username + --auth-password $password || true + - cp /zap/wrk/gl-dast-report.json . + artifacts: + reports: + dast: gl-dast-report.json +``` + +See the [zaproxy documentation](https://gitlab.com/gitlab-org/security-products/zaproxy) +to learn more about the authentication settings. + +### Manual job definition for GitLab 11.4 and earlier (deprecated) + +CAUTION: **Caution:** +Before GitLab 11.5, DAST job and artifact had to be named specifically +to automatically extract report data and show it in the merge request widget. +While these old job definitions are still maintained they have been deprecated +and may be removed in next major release, GitLab 12.0. You are strongly advised +to update your current `.gitlab-ci.yml` configuration to reflect that change. + +For GitLab 11.4 and earlier, the job should look like: + +```yaml +dast: + image: registry.gitlab.com/gitlab-org/security-products/zaproxy + variables: + website: "https://example.com" + allow_failure: true + script: + - mkdir /zap/wrk/ + - /zap/zap-baseline.py -J gl-dast-report.json -t $website || true + - cp /zap/wrk/gl-dast-report.json . + artifacts: + paths: [gl-dast-report.json] +``` + +## Security Dashboard + +The Security Dashboard is a good place to get an overview of all the security +vulnerabilities in your groups and projects. Read more about the +[Security Dashboard](../security_dashboard/index.md). + +## Interacting with the vulnerabilities + +Once a vulnerability is found, you can interact with it. Read more on how to +[interact with the vulnerabilities](../index.md#interacting-with-the-vulnerabilities). diff --git a/doc/user/application_security/dependency_scanning/img/dependency_scanning.png b/doc/user/application_security/dependency_scanning/img/dependency_scanning.png Binary files differnew file mode 100644 index 00000000000..18df356f846 --- /dev/null +++ b/doc/user/application_security/dependency_scanning/img/dependency_scanning.png diff --git a/doc/user/application_security/dependency_scanning/index.md b/doc/user/application_security/dependency_scanning/index.md new file mode 100644 index 00000000000..d78cf778110 --- /dev/null +++ b/doc/user/application_security/dependency_scanning/index.md @@ -0,0 +1,243 @@ +# Dependency Scanning **[ULTIMATE]** + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/issues/5105) +in [GitLab Ultimate](https://about.gitlab.com/pricing/) 10.7. + +## Overview + +If you are using [GitLab CI/CD](../../../ci/README.md), you can analyze your dependencies for known +vulnerabilities using Dependency Scanning. + +You can take advantage of Dependency Scanning by either [including the CI job](#including-the-provided-template) +in your existing `.gitlab-ci.yml` file or by implicitly using +[Auto Dependency Scanning](../../../topics/autodevops/index.md#auto-dependency-scanning-ultimate) +that is provided by [Auto DevOps](../../../topics/autodevops/index.md). + +GitLab checks the Dependency Scanning report, compares the found vulnerabilities +between the source and target branches, and shows the information right on the +merge request. + + + +The results are sorted by the severity of the vulnerability: + +1. Critical +1. High +1. Medium +1. Low +1. Unknown +1. Everything else + +## Use cases + +It helps to automatically find security vulnerabilities in your dependencies +while you are developing and testing your applications. For example when your +application is using an external (open source) library which is known to be vulnerable. + +## Requirements + +To run a Dependency Scanning job, you need GitLab Runner with the +[`docker`](https://docs.gitlab.com/runner/executors/docker.html#use-docker-in-docker-with-privileged-mode) or +[`kubernetes`](https://docs.gitlab.com/runner/install/kubernetes.html#running-privileged-containers-for-the-runners) +executor running in privileged mode. If you're using the shared Runners on GitLab.com, +this is enabled by default. + +## Supported languages and package managers + +The following languages and dependency managers are supported. + +| Language (package managers) | Scan tool | +|-----------------------------------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------| +| JavaScript ([npm](https://www.npmjs.com/), [yarn](https://yarnpkg.com/en/)) | [gemnasium](https://gitlab.com/gitlab-org/security-products/gemnasium/general), [Retire.js](https://retirejs.github.io/retire.js) | +| Python ([pip](https://pip.pypa.io/en/stable/)) (only `requirements.txt` supported) | [gemnasium](https://gitlab.com/gitlab-org/security-products/gemnasium/general) | +| Ruby ([gem](https://rubygems.org/)) | [gemnasium](https://gitlab.com/gitlab-org/security-products/gemnasium/general), [bundler-audit](https://github.com/rubysec/bundler-audit) | +| Java ([Maven](https://maven.apache.org/)) | [gemnasium](https://gitlab.com/gitlab-org/security-products/gemnasium/general) | +| PHP ([Composer](https://getcomposer.org/)) | [gemnasium](https://gitlab.com/gitlab-org/security-products/gemnasium/general) | + +Some scanners require to send a list of project dependencies to GitLab's central +servers to check for vulnerabilities. To learn more about this or to disable it, +refer to the [GitLab Dependency Scanning tool documentation](https://gitlab.com/gitlab-org/security-products/dependency-scanning#remote-checks). + +## Configuring Dependency Scanning + +To enable Dependency Scanning in your project, define a job in your `.gitlab-ci.yml` +file that generates the +[Dependency Scanning report artifact](../../../ci/yaml/README.md#artifactsreportsdependency_scanning-ultimate). + +This can be done in two ways: + +- For GitLab 11.9 and later, including the provided Dependency Scanning + `.gitlab-ci.yml` template (recommended). +- Manually specifying the job definition. Not recommended unless using GitLab + 11.8 and earlier. + +### Including the provided template + +NOTE: **Note:** +The CI/CD Dependency Scanning template is supported on GitLab 11.9 and later versions. +For earlier versions, use the [manual job definition](#manual-job-definition-for-gitlab-115-and-later). + +A CI/CD [Dependency Scanning template](https://gitlab.com/gitlab-org/gitlab-ee/blob/master/lib/gitlab/ci/templates/Security/Dependency-Scanning.gitlab-ci.yml) +with the default Dependency Scanning job definition is provided as a part of your GitLab +installation which you can [include](../../../ci/yaml/README.md#includetemplate) +in your `.gitlab-ci.yml` file. + +To enable Dependency Scanning using the provided template, add the following to +your `.gitlab-ci.yml` file: + +```yaml +include: + template: Dependency-Scanning.gitlab-ci.yml +``` + +The included template will create a `dependency_scanning` job in your CI/CD +pipeline and scan your project's source code for possible vulnerabilities. + +The report will be saved as a +[Dependency Scanning report artifact](../../../ci/yaml/README.md#artifactsreportsdependency_scanning-ultimate) +that you can later download and analyze. Due to implementation limitations, we +always take the latest Dependency Scanning artifact available. + +Some security scanners require to send a list of project dependencies to GitLab +central servers to check for vulnerabilities. To learn more about this or to +disable it, check the +[GitLab Dependency Scanning tool documentation](https://gitlab.com/gitlab-org/security-products/dependency-scanning#remote-checks). + +#### Customizing the Dependency Scanning settings + +The Dependency Scanning settings can be changed through environment variables by using the +[`variables`](../../../ci/yaml/README.md#variables) parameter in `.gitlab-ci.yml`. +These variables are documented in the +[Dependency Scanning tool documentation](https://gitlab.com/gitlab-org/security-products/dependency-scanning#settings). + +For example: + +```yaml +include: + template: Dependency-Scanning.gitlab-ci.yml + +variables: + DEP_SCAN_DISABLE_REMOTE_CHECKS: true +``` + +Because template is [evaluated before](../../../ci/yaml/README.md#include) the pipeline +configuration, the last mention of the variable will take precedence. + +#### Overriding the Dependency Scanning template + +If you want to override the job definition (for example, change properties like +`variables` or `dependencies`), you need to declare a `dependency_scanning` job +after the template inclusion and specify any additional keys under it. For example: + +```yaml +include: + template: Dependency-Scanning.gitlab-ci.yml + +dependency_scanning: + variables: + CI_DEBUG_TRACE: "true" +``` + +### Manual job definition for GitLab 11.5 and later + +For GitLab 11.5 and GitLab Runner 11.5 and later, the following `dependency_scanning` +job can be added: + +```yaml +dependency_scanning: + image: docker:stable + variables: + DOCKER_DRIVER: overlay2 + allow_failure: true + services: + - docker:stable-dind + script: + - export DS_VERSION=${SP_VERSION:-$(echo "$CI_SERVER_VERSION" | sed 's/^\([0-9]*\)\.\([0-9]*\).*/\1-\2-stable/')} + - | + docker run \ + --env DS_ANALYZER_IMAGES \ + --env DS_ANALYZER_IMAGE_PREFIX \ + --env DS_ANALYZER_IMAGE_TAG \ + --env DS_DEFAULT_ANALYZERS \ + --env DEP_SCAN_DISABLE_REMOTE_CHECKS \ + --env DS_DOCKER_CLIENT_NEGOTIATION_TIMEOUT \ + --env DS_PULL_ANALYZER_IMAGE_TIMEOUT \ + --env DS_RUN_ANALYZER_TIMEOUT \ + --volume "$PWD:/code" \ + --volume /var/run/docker.sock:/var/run/docker.sock \ + "registry.gitlab.com/gitlab-org/security-products/dependency-scanning:$DS_VERSION" /code + dependencies: [] + artifacts: + reports: + dependency_scanning: gl-dependency-scanning-report.json +``` + +You can supply many other [settings variables](https://gitlab.com/gitlab-org/security-products/dependency-scanning#settings) +via `docker run --env` to customize your job execution. + +### Manual job definition for GitLab 11.4 and earlier (deprecated) + +CAUTION: **Caution:** +Before GitLab 11.5, the Dependency Scanning job and artifact had to be named specifically +to automatically extract the report data and show it in the merge request widget. +While these old job definitions are still maintained, they have been deprecated +and may be removed in the next major release, GitLab 12.0. You are strongly advised +to update your current `.gitlab-ci.yml` configuration to reflect that change. + +For GitLab 11.4 and earlier, the job should look like: + +```yaml +dependency_scanning: + image: docker:stable + variables: + DOCKER_DRIVER: overlay2 + allow_failure: true + services: + - docker:stable-dind + script: + - export DS_VERSION=${SP_VERSION:-$(echo "$CI_SERVER_VERSION" | sed 's/^\([0-9]*\)\.\([0-9]*\).*/\1-\2-stable/')} + - | + docker run \ + --env DS_ANALYZER_IMAGES \ + --env DS_ANALYZER_IMAGE_PREFIX \ + --env DS_ANALYZER_IMAGE_TAG \ + --env DS_DEFAULT_ANALYZERS \ + --env DS_EXCLUDED_PATHS \ + --env DEP_SCAN_DISABLE_REMOTE_CHECKS \ + --env DS_DOCKER_CLIENT_NEGOTIATION_TIMEOUT \ + --env DS_PULL_ANALYZER_IMAGE_TIMEOUT \ + --env DS_RUN_ANALYZER_TIMEOUT \ + --volume "$PWD:/code" \ + --volume /var/run/docker.sock:/var/run/docker.sock \ + "registry.gitlab.com/gitlab-org/security-products/dependency-scanning:$DS_VERSION" /code + artifacts: + paths: [gl-dependency-scanning-report.json] +``` + +## Security Dashboard + +The Security Dashboard is a good place to get an overview of all the security +vulnerabilities in your groups and projects. Read more about the +[Security Dashboard](../security_dashboard/index.md). + +## Interacting with the vulnerabilities + +Once a vulnerability is found, you can interact with it. Read more on how to +[interact with the vulnerabilities](../index.md#interacting-with-the-vulnerabilities). + +## Dependency List + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/issues/10075) +in [GitLab Ultimate](https://about.gitlab.com/pricing/) 12.0. + +An additional benefit of Dependency Scanning is the ability to get a list of your project's dependencies with their versions. + +This list can be generated only for [supported languages and package managers](#supported-languages-and-package-managers). + +To see the generated dependency list, navigate to the Dependency List page under your project's left sidebar menu **Project > Dependency List**. + +## Contributing to the vulnerability database + +You can search the [gemnasium-db](https://gitlab.com/gitlab-org/security-products/gemnasium-db) project +to find a vulnerability in the Gemnasium database. +You can also [submit new vulnerabilities](https://gitlab.com/gitlab-org/security-products/gemnasium-db/blob/master/CONTRIBUTING.md).
\ No newline at end of file diff --git a/doc/user/application_security/img/create_issue_with_list_hover.png b/doc/user/application_security/img/create_issue_with_list_hover.png Binary files differnew file mode 100644 index 00000000000..7d70e8299f5 --- /dev/null +++ b/doc/user/application_security/img/create_issue_with_list_hover.png diff --git a/doc/user/application_security/img/dismissed_info.png b/doc/user/application_security/img/dismissed_info.png Binary files differnew file mode 100644 index 00000000000..64d5cf26ed2 --- /dev/null +++ b/doc/user/application_security/img/dismissed_info.png diff --git a/doc/user/application_security/img/interactive_reports.png b/doc/user/application_security/img/interactive_reports.png Binary files differnew file mode 100644 index 00000000000..373b39104db --- /dev/null +++ b/doc/user/application_security/img/interactive_reports.png diff --git a/doc/user/application_security/img/issue.png b/doc/user/application_security/img/issue.png Binary files differnew file mode 100644 index 00000000000..6467201df3f --- /dev/null +++ b/doc/user/application_security/img/issue.png diff --git a/doc/user/application_security/img/vulnerability_solution.png b/doc/user/application_security/img/vulnerability_solution.png Binary files differnew file mode 100644 index 00000000000..7443b9b6eea --- /dev/null +++ b/doc/user/application_security/img/vulnerability_solution.png diff --git a/doc/user/application_security/index.md b/doc/user/application_security/index.md new file mode 100644 index 00000000000..679847b76d7 --- /dev/null +++ b/doc/user/application_security/index.md @@ -0,0 +1,114 @@ +# GitLab Secure **[ULTIMATE]** + +Check your application for security vulnerabilities that may lead to unauthorized access, +data leaks, and denial of services. GitLab will perform static and dynamic tests on the +code of your application, looking for known flaws and report them in the merge request +so you can fix them before merging. Security teams can use dashboards to get a +high-level view on projects and groups, and start remediation processes when needed. + +## Security scanning tools + +GitLab can scan and report any vulnerabilities found in your project. + +| Secure scanning tools | Description | +|:-----------------------------------------------------------------------------|:-----------------------------------------------------------------------| +| [Container Scanning](container_scanning/index.md) **[ULTIMATE]** | Scan Docker containers for known vulnerabilities. | +| [Dependency Scanning](dependency_scanning/index.md) **[ULTIMATE]** | Analyze your dependencies for known vulnerabilities. | +| [Dynamic Application Security Testing (DAST)](dast/index.md) **[ULTIMATE]** | Analyze running web applications for known vulnerabilities. | +| [License Management](license_management/index.md) **[ULTIMATE]** | Search your project's dependencies for their licenses. | +| [Security Dashboard](security_dashboard/index.md) **[ULTIMATE]** | View vulnerabilities in all your projects and groups. | +| [Static Application Security Testing (SAST)](sast/index.md) **[ULTIMATE]** | Analyze source code for known vulnerabilities. | + +## Interacting with the vulnerabilities + +> Introduced in [GitLab Ultimate](https://about.gitlab.com/pricing) 10.8. + +CAUTION: **Warning:** +This feature is currently [Alpha](https://about.gitlab.com/handbook/product/#alpha-beta-ga) and while you can start using it, it may receive important changes in the future. + +Each security vulnerability in the merge request report or the +[Security Dashboard](security_dashboard/index.md) is actionable. Clicking on an +entry, a detailed information will pop up with different possible options: + +- [Dismiss vulnerability](#dismissing-a-vulnerability): Dismissing a vulnerability + will place a <s>strikethrough</s> styling on it. +- [Create issue](#creating-an-issue-for-a-vulnerability): The new issue will + have the title and description pre-populated with the information from the + vulnerability report and will be created as [confidential](../project/issues/confidential_issues.md) by default. +- [Solution](#solutions-for-vulnerabilities): For some vulnerabilities + ([Dependency Scanning](dependency_scanning/index.md) and [Container Scanning](container_scanning/index.md)) + a solution is provided for how to fix the vulnerability. + + + +### Dismissing a vulnerability + +You can dismiss vulnerabilities by clicking the **Dismiss vulnerability** button. +This will dismiss the vulnerability and re-render it to reflect its dismissed state. +If you wish to undo this dismissal, you can click the **Undo dismiss** button. + +#### Adding a dismissal reason + +> Introduced in [GitLab Ultimate](https://about.gitlab.com/pricing) 12.0. + +When dismissing a vulnerability, it's often helpful to provide a reason for doing so. +If you press the comment button next to **Dismiss vulnerability** in the modal, a text box will appear, allowing you to add a comment with your dismissal. +This comment can not currently be edited or removed, but [future versions](https://gitlab.com/gitlab-org/gitlab-ee/issues/11721) will add this functionality. + + + +### Creating an issue for a vulnerability + +You can create an issue for a vulnerability by selecting the **Create issue** +button from within the vulnerability modal or using the action buttons to the right of +a vulnerability row when in the group security dashboard. + +This will create a [confidential issue](../project/issues/confidential_issues.md) +on the project this vulnerability came from and pre-fill it with some useful +information taken from the vulnerability report. Once the issue is created, you +will be redirected to it so you can edit, assign, or comment on it. + +Upon returning to the group security dashboard, you'll see that +the vulnerability will now have an associated issue next to the name. + + + +### Solutions for vulnerabilities + +> Introduced in [GitLab Ultimate](https://about.gitlab.com/pricing) 11.7. + +CAUTION: **Warning:** +Automatic Patch creation is only available for a subset of +[Dependency Scanning](dependency_scanning/index.md). At the moment only Node.JS +projects managed with yarn are supported. + +Some vulnerabilities can be fixed by applying the solution that GitLab +automatically generates. + +#### Manually applying the suggested patch + +Some vulnerabilities can be fixed by applying a patch that is automatically +generated by GitLab. To apply the fix: + +1. Click on the vulnerability. +1. Download and review the patch file `remediation.patch`. +2. Ensure your local project has the same commit checked out that was used to generate the patch. +3. Run `git apply remediation.patch`. +4. Verify and commit the changes to your branch. + + + +#### Creating a merge request from a vulnerability + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/issues/9224) in + [GitLab Ultimate](https://about.gitlab.com/pricing) 11.9. + +In certain cases, GitLab will allow you to create a merge request that will +automatically remediate the vulnerability. Any vulnerability that has a +[solution](#solutions-for-vulnerabilities) can have a merge request created to +automatically solve the issue. + +If this action is available there will be a **Create merge request** button in the vulnerability modal. +Clicking on this button will create a merge request to apply the solution onto the source branch. + + diff --git a/doc/user/application_security/license_management/img/license_management.png b/doc/user/application_security/license_management/img/license_management.png Binary files differnew file mode 100644 index 00000000000..cdce6b5fe38 --- /dev/null +++ b/doc/user/application_security/license_management/img/license_management.png diff --git a/doc/user/application_security/license_management/img/license_management_decision.png b/doc/user/application_security/license_management/img/license_management_decision.png Binary files differnew file mode 100644 index 00000000000..0763130c375 --- /dev/null +++ b/doc/user/application_security/license_management/img/license_management_decision.png diff --git a/doc/user/application_security/license_management/img/license_management_pipeline_tab.png b/doc/user/application_security/license_management/img/license_management_pipeline_tab.png Binary files differnew file mode 100644 index 00000000000..80ffca815b9 --- /dev/null +++ b/doc/user/application_security/license_management/img/license_management_pipeline_tab.png diff --git a/doc/user/application_security/license_management/img/license_management_settings.png b/doc/user/application_security/license_management/img/license_management_settings.png Binary files differnew file mode 100644 index 00000000000..b5490e59074 --- /dev/null +++ b/doc/user/application_security/license_management/img/license_management_settings.png diff --git a/doc/user/application_security/license_management/index.md b/doc/user/application_security/license_management/index.md new file mode 100644 index 00000000000..8b75995c377 --- /dev/null +++ b/doc/user/application_security/license_management/index.md @@ -0,0 +1,264 @@ +# License Management **[ULTIMATE]** + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/issues/5483) +in [GitLab Ultimate](https://about.gitlab.com/pricing/) 11.0. + +## Overview + +If you are using [GitLab CI/CD](../../../ci/README.md), you can search your project dependencies for their licenses +using License Management. + +You can take advantage of License Management by either [including the job](#configuring-license-management) +in your existing `.gitlab-ci.yml` file or by implicitly using +[Auto License Management](../../../topics/autodevops/index.md#auto-license-management-ultimate) +that is provided by [Auto DevOps](../../../topics/autodevops/index.md). + +GitLab checks the License Management report, compares the licenses between the +source and target branches, and shows the information right on the merge request. +Blacklisted licenses will be clearly visible with an `x` red icon next to them +as well as new licenses which need a decision from you. In addition, you can +[manually approve or blacklist](#project-policies-for-license-management) +licenses in your project's settings. + +NOTE: **Note:** +If the license management report doesn't have anything to compare to, no information +will be displayed in the merge request area. That is the case when you add the +`license_management` job in your `.gitlab-ci.yml` for the first time. +Consecutive merge requests will have something to compare to and the license +management report will be shown properly. + + + +If you are a project or group Maintainer, you can click on a license to be given +the choice to approve it or blacklist it. + + + +## Use cases + +It helps you find what licenses your project uses in its dependencies, and decide for each of then +whether to allow it or forbid it. For example, your application is using an external (open source) +library whose license is incompatible with yours. + +## Supported languages and package managers + +The following languages and package managers are supported. + +| Language | Package managers | +|------------|-------------------------------------------------------------------| +| JavaScript | [Bower](https://bower.io/), [npm](https://www.npmjs.com/) | +| Go | [Godep](https://github.com/tools/godep), go get | +| Java | [Gradle](https://gradle.org/), [Maven](https://maven.apache.org/) | +| .NET | [Nuget](https://www.nuget.org/) | +| Python | [pip](https://pip.pypa.io/en/stable/) | +| Ruby | [gem](https://rubygems.org/) | + +## Requirements + +To run a License Management scanning job, you need GitLab Runner with the +[`docker` executor](https://docs.gitlab.com/runner/executors/docker.html). + +## Configuring License Management + +To enable License Management in your project, define a job in your `.gitlab-ci.yml` +file that generates the [License Management report artifact](../../../ci/yaml/README.md#artifactsreportslicense_management-ultimate). + +This can be done in two ways: + +- For GitLab 11.9 and later, including the provided License Management `.gitlab-ci.yml` template (recommended). +- Manually specifying the job definition. Not recommended unless using GitLab + 11.8 and earlier. + +### Including the provided template + +NOTE: **Note:** +The CI/CD License Management template is supported on GitLab 11.9 and later versions. +For earlier versions, use the [manual job definition](#manual-job-definition-for-gitlab-115-and-later). + +A CI/CD [License Management template](https://gitlab.com/gitlab-org/gitlab-ee/blob/master/lib/gitlab/ci/templates/Security/License-Management.gitlab-ci.yml) +with the default License Management job definition is provided as a part of your GitLab +installation which you can [include](../../../ci/yaml/README.md#includetemplate) +in your `.gitlab-ci.yml` file. + +To enable License Management using the provided template, add the following to +your `.gitlab-ci.yml` file: + +```yaml +include: + template: License-Management.gitlab-ci.yml +``` + +The included template will create a `license_management` job in your CI/CD pipeline +and scan your dependencies to find their licenses. + +The report will be saved as a +[License Management report artifact](../../../ci/yaml/README.md#artifactsreportslicense_management-ultimate) +that you can later download and analyze. Due to implementation limitations, we +always take the latest License Management artifact available. Behind the scenes, the +[GitLab License Management Docker image](https://gitlab.com/gitlab-org/security-products/license-management) +is used to detect the languages/frameworks and in turn analyzes the licenses. + +#### Installing custom dependencies + +> Introduced in [GitLab Ultimate](https://about.gitlab.com/pricing/) 11.4. + +The `license_management` image already embeds many auto-detection scripts, languages, +and packages. Nevertheless, it's almost impossible to cover all cases for all projects. +That's why sometimes it's necessary to install extra packages, or to have extra steps +in the project automated setup, like the download and installation of a certificate. +For that, a `LICENSE_MANAGEMENT_SETUP_CMD` environment variable can be passed to the container, +with the required commands to run before the license detection. + +If present, this variable will override the setup step necessary to install all the packages +of your application (e.g.: for a project with a `Gemfile`, the setup step could be +`bundle install`). + +For example: + +```yaml +include: + template: License-Management.gitlab-ci.yml + +variables: + LICENSE_MANAGEMENT_SETUP_CMD: sh my-custom-install-script.sh +``` + +In this example, `my-custom-install-script.sh` is a shell script at the root +directory of your project. + +#### Overriding the template + +If you want to override the job definition (for example, change properties like +`variables` or `dependencies`), you need to declare a `license_management` job +after the template inclusion and specify any additional keys under it. For example: + +```yaml +include: + template: License-Management.gitlab-ci.yml + +license_management: + variables: + CI_DEBUG_TRACE: "true" +``` + +#### Configuring Maven projects + +The License Management tool provides a `MAVEN_CLI_OPTS` environment variable which can hold +the command line arguments to pass to the `mvn install` command which is executed under the hood. +Feel free to use it for the customization of Maven execution. For example: + +```yaml +include: + template: License-Management.gitlab-ci.yml + +license_management: + variables: + MAVEN_CLI_OPTS: --debug +``` + +`mvn install` runs through all of the [build life cycle](http://maven.apache.org/guides/introduction/introduction-to-the-lifecycle.html) +stages prior to `install`, including `test`. Running unit tests is not directly +necessary for the license scanning purposes and consumes time, so it's skipped +by having the default value of `MAVEN_CLI_OPTS` as `-DskipTests`. If you want +to supply custom `MAVEN_CLI_OPTS` and skip tests at the same time, don't forget +to explicitly add `-DskipTests` to your options. +If you still need to run tests during `mvn install`, add `-DskipTests=false` to +`MAVEN_CLI_OPTS`. + +### Manual job definition for GitLab 11.5 and later + +For GitLab 11.5 and GitLab Runner 11.5 and later, the following `license_management` +job can be added: + +```yaml +license_management: + image: + name: "registry.gitlab.com/gitlab-org/security-products/license-management:$CI_SERVER_VERSION_MAJOR-$CI_SERVER_VERSION_MINOR-stable" + entrypoint: [""] + stage: test + allow_failure: true + script: + - /run.sh analyze . + artifacts: + reports: + license_management: gl-license-management-report.json +``` + +If you want to install custom project dependencies via the `SETUP_CMD` variable: + +```yaml +license_management: + image: + name: "registry.gitlab.com/gitlab-org/security-products/license-management:$CI_SERVER_VERSION_MAJOR-$CI_SERVER_VERSION_MINOR-stable" + entrypoint: [""] + stage: test + variables: + SETUP_CMD: ./my-custom-install-script.sh + allow_failure: true + script: + - /run.sh analyze . + artifacts: + reports: + license_management: gl-license-management-report.json +``` + +### Manual job definition for GitLab 11.4 and earlier (deprecated) + +CAUTION: **Caution:** +Before GitLab 11.5, the License Management job and artifact had to be named specifically +to automatically extract the report data and show it in the merge request widget. +While these old job definitions are still maintained, they have been deprecated +and may be removed in the next major release, GitLab 12.0. You are strongly advised +to update your current `.gitlab-ci.yml` configuration to reflect that change. + +For GitLab 11.4 and earlier, the job should look like: + +```yaml +license_management: + image: + name: "registry.gitlab.com/gitlab-org/security-products/license-management:$CI_SERVER_VERSION_MAJOR-$CI_SERVER_VERSION_MINOR-stable" + entrypoint: [""] + stage: test + allow_failure: true + script: + - /run.sh analyze . + artifacts: + paths: [gl-license-management-report.json] +``` + +## Project policies for License Management + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/issues/5940) +in [GitLab Ultimate](https://about.gitlab.com/pricing/) 11.4. + +From the project's settings: + +- The list of licenses and their status can be managed. +- Licenses can be manually approved or blacklisted. + +To approve or blacklist a license: + +1. Either use the **Manage licenses** button in the merge request widget, or + navigate to the project's **Settings > CI/CD** and expand the + **License Management** section. +1. Click the **Add a license** button. +1. In the **License name** dropdown, either: + - Select one of the available licenses. You can search for licenses in the field + at the top of the list. + - Enter arbitrary text in the field at the top of the list. This will cause the text to be + added as a license name to the list. +1. Select the **Approve** or **Blacklist** radio button to approve or blacklist respectively + the selected license. + +  + +## License Management report under pipelines + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/issues/5491) +in [GitLab Ultimate](https://about.gitlab.com/pricing/) 11.2. + +From your project's left sidebar, navigate to **CI/CD > Pipelines** and click on the +pipeline ID that has a `license_management` job to see the Licenses tab with the listed +licenses (if any). + + diff --git a/doc/user/application_security/sast/img/sast.png b/doc/user/application_security/sast/img/sast.png Binary files differnew file mode 100644 index 00000000000..2c75592c32a --- /dev/null +++ b/doc/user/application_security/sast/img/sast.png diff --git a/doc/user/application_security/sast/img/security_report.png b/doc/user/application_security/sast/img/security_report.png Binary files differnew file mode 100644 index 00000000000..ba41b707238 --- /dev/null +++ b/doc/user/application_security/sast/img/security_report.png diff --git a/doc/user/application_security/sast/index.md b/doc/user/application_security/sast/index.md new file mode 100644 index 00000000000..db328262aba --- /dev/null +++ b/doc/user/application_security/sast/index.md @@ -0,0 +1,254 @@ +# Static Application Security Testing (SAST) **[ULTIMATE]** + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/issues/3775) +in [GitLab Ultimate](https://about.gitlab.com/pricing/) 10.3. + +NOTE: **4 of the top 6 attacks were application based.** +Download our whitepaper, +["A Seismic Shift in Application Security"](https://about.gitlab.com/resources/whitepaper-seismic-shift-application-security/) +to learn how to protect your organization. + +## Overview + +If you are using [GitLab CI/CD](../../../ci/README.md), you can analyze your source code for known +vulnerabilities using Static Application Security Testing (SAST). + +You can take advantage of SAST by either [including the CI job](#configuring-sast) in +your existing `.gitlab-ci.yml` file or by implicitly using +[Auto SAST](../../../topics/autodevops/index.md#auto-sast-ultimate) +that is provided by [Auto DevOps](../../../topics/autodevops/index.md). + +GitLab checks the SAST report, compares the found vulnerabilities between the +source and target branches, and shows the information right on the merge request. + + + +The results are sorted by the priority of the vulnerability: + +1. Critical +1. High +1. Medium +1. Low +1. Unknown +1. Everything else + +## Use cases + +- Your code has a potentially dangerous attribute in a class, or unsafe code + that can lead to unintended code execution. +- Your application is vulnerable to cross-site scripting (XSS) attacks that can + be leveraged to unauthorized access to session data. + +## Requirements + +To run a SAST job, you need GitLab Runner with the +[`docker`](https://docs.gitlab.com/runner/executors/docker.html#use-docker-in-docker-with-privileged-mode) or +[`kubernetes`](https://docs.gitlab.com/runner/install/kubernetes.html#running-privileged-containers-for-the-runners) +executor running in privileged mode. If you're using the shared Runners on GitLab.com, +this is enabled by default. + +## Supported languages and frameworks + +The following table shows which languages, package managers and frameworks are supported and which tools are used. + +| Language (package managers) / framework | Scan tool | Introduced in GitLab Version | +|-----------------------------------------------------------------------------|----------------------------------------------------------------------------------------|------------------------------| +| .NET | [Security Code Scan](https://security-code-scan.github.io) | 11.0 | +| Any | [Gitleaks](https://github.com/zricethezav/gitleaks) and [TruffleHog](https://github.com/dxa4481/truffleHog) | 11.9 | +| C/C++ | [Flawfinder](https://www.dwheeler.com/flawfinder/) | 10.7 | +| Elixir (Phoenix) | [Sobelow](https://github.com/nccgroup/sobelow) | 11.10 | +| Go | [Gosec](https://github.com/securego/gosec) | 10.7 | +| Groovy ([Ant](https://ant.apache.org/), [Gradle](https://gradle.org/), [Maven](https://maven.apache.org/) and [SBT](https://www.scala-sbt.org/)) | [SpotBugs](https://spotbugs.github.io/) with the [find-sec-bugs](https://find-sec-bugs.github.io/) plugin | 11.3 (Gradle) & 11.9 (Ant, Maven, SBT) | +| Java ([Ant](https://ant.apache.org/), [Gradle](https://gradle.org/), [Maven](https://maven.apache.org/) and [SBT](https://www.scala-sbt.org/)) | [SpotBugs](https://spotbugs.github.io/) with the [find-sec-bugs](https://find-sec-bugs.github.io/) plugin | 10.6 (Maven), 10.8 (Gradle) & 11.9 (Ant, SBT) | +| Javascript | [ESLint security plugin](https://github.com/nodesecurity/eslint-plugin-security) | 11.8 | +| Node.js | [NodeJsScan](https://github.com/ajinabraham/NodeJsScan) | 11.1 | +| PHP | [phpcs-security-audit](https://github.com/FloeDesignTechnologies/phpcs-security-audit) | 10.8 | +| Python ([pip](https://pip.pypa.io/en/stable/)) | [bandit](https://github.com/PyCQA/bandit) | 10.3 | +| Ruby on Rails | [brakeman](https://brakemanscanner.org) | 10.3 | +| Scala ([Ant](https://ant.apache.org/), [Gradle](https://gradle.org/), [Maven](https://maven.apache.org/) and [SBT](https://www.scala-sbt.org/)) | [SpotBugs](https://spotbugs.github.io/) with the [find-sec-bugs](https://find-sec-bugs.github.io/) plugin | 11.0 (SBT) & 11.9 (Ant, Gradle, Maven) | +| Typescript | [TSLint config security](https://github.com/webschik/tslint-config-security/) | 11.9 | + +NOTE: **Note:** +The Java analyzers can also be used for variants like the +[Gradle wrapper](https://docs.gradle.org/current/userguide/gradle_wrapper.html), +[Grails](https://grails.org/) and the [Maven wrapper](https://github.com/takari/maven-wrapper). + +## Configuring SAST + +To enable SAST in your project, define a job in your `.gitlab-ci.yml` file that generates the +[SAST report artifact](../../../ci/yaml/README.md#artifactsreportssast-ultimate). + +This can be done in two ways: + +- For GitLab 11.9 and later, including the provided SAST `.gitlab-ci.yml` template (recommended). +- Manually specifying the job definition. Not recommended unless using GitLab + 11.8 and earlier. + +### Including the provided template + +NOTE: **Note:** +The CI/CD SAST template is supported on GitLab 11.9 and later versions. +For earlier versions, use the [manual job definition](#manual-job-definition-for-gitlab-115-and-later). + +A CI/CD [SAST template](https://gitlab.com/gitlab-org/gitlab-ee/blob/master/lib/gitlab/ci/templates/Security/SAST.gitlab-ci.yml) +with the default SAST job definition is provided as a part of your GitLab +installation which you can [include](../../../ci/yaml/README.md#includetemplate) +in your `.gitlab-ci.yml` file. + +To enable SAST using the provided template, add the following to your `.gitlab-ci.yml` +file: + +```yaml +include: + template: SAST.gitlab-ci.yml +``` + +The included template will create a `sast` job in your CI/CD pipeline and scan +your project's source code for possible vulnerabilities. + +The report will be saved as a +[SAST report artifact](../../../ci/yaml/README.md#artifactsreportssast-ultimate) +that you can later download and analyze. Due to implementation limitations, we +always take the latest SAST artifact available. Behind the scenes, the +[GitLab SAST Docker image](https://gitlab.com/gitlab-org/security-products/sast) +is used to detect the languages/frameworks and in turn runs the matching scan tools. + +#### Customizing the SAST settings + +The SAST settings can be changed through environment variables by using the +[`variables`](../../../ci/yaml/README.md#variables) parameter in `.gitlab-ci.yml`. +These variables are documented in the +[SAST tool documentation](https://gitlab.com/gitlab-org/security-products/sast#settings). + +In the following example, we include the SAST template and at the same time we +set the `SAST_GOSEC_LEVEL` variable to `2`: + +```yaml +include: + template: SAST.gitlab-ci.yml + +variables: + SAST_GOSEC_LEVEL: 2 +``` + +Because the template is [evaluated before](../../../ci/yaml/README.md#include) +the pipeline configuration, the last mention of the variable will take precedence. + +#### Overriding the SAST template + +If you want to override the job definition (for example, change properties like +`variables` or `dependencies`), you need to declare a `sast` job after the +template inclusion and specify any additional keys under it. For example: + +```yaml +include: + template: SAST.gitlab-ci.yml + +sast: + variables: + CI_DEBUG_TRACE: "true" +``` + +### Manual job definition for GitLab 11.5 and later + +For GitLab 11.5 and GitLab Runner 11.5 and later, the following `sast` +job can be added: + +```yaml +sast: + stage: test + image: docker:stable + variables: + DOCKER_DRIVER: overlay2 + allow_failure: true + services: + - docker:stable-dind + script: + - export SAST_VERSION=${SP_VERSION:-$(echo "$CI_SERVER_VERSION" | sed 's/^\([0-9]*\)\.\([0-9]*\).*/\1-\2-stable/')} + - | + docker run \ + --env SAST_ANALYZER_IMAGES \ + --env SAST_ANALYZER_IMAGE_PREFIX \ + --env SAST_ANALYZER_IMAGE_TAG \ + --env SAST_DEFAULT_ANALYZERS \ + --env SAST_EXCLUDED_PATHS \ + --env SAST_BANDIT_EXCLUDED_PATHS \ + --env SAST_BRAKEMAN_LEVEL \ + --env SAST_GOSEC_LEVEL \ + --env SAST_FLAWFINDER_LEVEL \ + --env SAST_DOCKER_CLIENT_NEGOTIATION_TIMEOUT \ + --env SAST_PULL_ANALYZER_IMAGE_TIMEOUT \ + --env SAST_RUN_ANALYZER_TIMEOUT \ + --volume "$PWD:/code" \ + --volume /var/run/docker.sock:/var/run/docker.sock \ + "registry.gitlab.com/gitlab-org/security-products/sast:$SAST_VERSION" /app/bin/run /code + dependencies: [] + artifacts: + reports: + sast: gl-sast-report.json +``` + +You can supply many other [settings variables](https://gitlab.com/gitlab-org/security-products/sast#settings) +via `docker run --env` to customize your job execution. + +## Manual job definition for GitLab 11.4 and earlier (deprecated) + +CAUTION: **Deprecated:** +Before GitLab 11.5, the SAST job and artifact had to be named specifically +to automatically extract report data and show it in the merge request widget. +While these old job definitions are still maintained, they have been deprecated +and may be removed in the next major release, GitLab 12.0. You are strongly +advised to update your current `.gitlab-ci.yml` configuration to reflect that change. + +For GitLab 11.4 and earlier, the SAST job should look like: + +```yaml +sast: + image: docker:stable + variables: + DOCKER_DRIVER: overlay2 + allow_failure: true + services: + - docker:stable-dind + script: + - export SAST_VERSION=${SP_VERSION:-$(echo "$CI_SERVER_VERSION" | sed 's/^\([0-9]*\)\.\([0-9]*\).*/\1-\2-stable/')} + - docker run + --env SAST_CONFIDENCE_LEVEL="${SAST_CONFIDENCE_LEVEL:-3}" + --volume "$PWD:/code" + --volume /var/run/docker.sock:/var/run/docker.sock + "registry.gitlab.com/gitlab-org/security-products/sast:$SAST_VERSION" /app/bin/run /code + artifacts: + paths: [gl-sast-report.json] +``` + +## Secret detection + +GitLab is also able to detect secrets and credentials that have been unintentionally pushed to the repository. +For example, an API key that allows write access to third-party deployment environments. + +This check is performed by a specific analyzer during the `sast` job. It runs regardless of the programming +language of your app, and you don't need to change anything to your +CI/CD configuration file to turn it on. Results are available in the SAST report. + +GitLab currently includes [Gitleaks](https://github.com/zricethezav/gitleaks) and [TruffleHog](https://github.com/dxa4481/truffleHog) checks. + +## Security report under pipelines + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/issues/3776) +in [GitLab Ultimate](https://about.gitlab.com/pricing) 10.6. + +Visit any pipeline page which has a `sast` job and you will be able to see +the security report tab with the listed vulnerabilities (if any). + + + +## Security Dashboard + +The Security Dashboard is a good place to get an overview of all the security +vulnerabilities in your groups and projects. Read more about the +[Security Dashboard](../security_dashboard/index.md). + +## Interacting with the vulnerabilities + +Once a vulnerability is found, you can interact with it. Read more on how to +[interact with the vulnerabilities](../index.md#interacting-with-the-vulnerabilities). diff --git a/doc/user/application_security/security_dashboard/img/dashboard.png b/doc/user/application_security/security_dashboard/img/dashboard.png Binary files differnew file mode 100644 index 00000000000..a75168b1ce4 --- /dev/null +++ b/doc/user/application_security/security_dashboard/img/dashboard.png diff --git a/doc/user/application_security/security_dashboard/img/project_security_dashboard.png b/doc/user/application_security/security_dashboard/img/project_security_dashboard.png Binary files differnew file mode 100644 index 00000000000..f0dad6c54d0 --- /dev/null +++ b/doc/user/application_security/security_dashboard/img/project_security_dashboard.png diff --git a/doc/user/application_security/security_dashboard/index.md b/doc/user/application_security/security_dashboard/index.md new file mode 100644 index 00000000000..19eeb06a259 --- /dev/null +++ b/doc/user/application_security/security_dashboard/index.md @@ -0,0 +1,104 @@ +# GitLab Security Dashboard **[ULTIMATE]** + +The Security Dashboard is a good place to get an overview of all the security +vulnerabilities in your groups and projects. + +You can also drill down into a vulnerability and get extra information, see which +project it comes from, the file it's in, and various metadata to help you analyze +the risk. You can also action these vulnerabilities by creating an issue for them, +or by dismissing them. + +To benefit from the Security Dashboard you must first configure one of the +[security reports](../index.md). + +## Supported reports + +The Security Dashboard supports the following reports: + +- [Container Scanning](../container_scanning/index.md) +- [DAST](../dast/index.md) +- [Dependency Scanning](../dependency_scanning/index.md) +- [SAST](../sast/index.md) + +## Requirements + +To use the project or group security dashboard: + +1. At least one project inside a group must be configured with at least one of + the [supported reports](#supported-reports). +2. The configured jobs must use the [new `reports` syntax](../../../ci/yaml/README.md#artifactsreports). +3. [GitLab Runner](https://docs.gitlab.com/runner/) 11.5 or newer must be used. + If you're using the shared Runners on GitLab.com, this is already the case. + +## Project Security Dashboard + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/issues/6165) in [GitLab Ultimate](https://about.gitlab.com/pricing) 11.1. + +At the project level, the Security Dashboard displays the latest security reports +for your project. Use it to find and fix vulnerabilities affecting the +[default branch](../../project/repository/branches/index.md#default-branch). + + + +## Group Security Dashboard + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/issues/6709) in + [GitLab Ultimate](https://about.gitlab.com/pricing) 11.5. + +The group Security Dashboard gives an overview of the vulnerabilities of all the +projects in a group and its subgroups. + +First, navigate to the Security Dashboard found under your group's +**Overview > Security Dashboard**. + +Once you're on the dashboard, at the top you should see a series of filters for: + +- Severity +- Confidence +- Report type +- Project + + + +Selecting one or more filters will filter the results in this page. +The first section is an overview of all the vulnerabilities, grouped by severity. +Underneath this overview is a timeline chart that shows how many open +vulnerabilities your projects had at various points in time. You can filter among 30, 60, and +90 days, with the default being 90. Hover over the chart to get more details about +the open vulnerabilities at a specific time. + +Finally, there is a list of all the vulnerabilities in the group, sorted by severity. +In that list, you can see the severity of the vulnerability, its name, its +confidence (likelihood of the vulnerability to be a positive one), and the project +it's from. + +If you hover over a row, there will appear some actions you can take: + +- "More info" +- "Create issue" +- "Dismiss vulnerability" + +Read more on how to [interact with the vulnerabilities](../index.md#interacting-with-the-vulnerabilities). + +## Keeping the dashboards up to date + +The Security Dashboard displays information from the results of the most recent +security scan on the [default branch](../../project/repository/branches/index.md#default-branch), +which means that security scans are performed every time the branch is updated. + +If the default branch is updated infrequently, scans are run infrequently and the +information on the Security Dashboard can become outdated as new vulnerabilities +are discovered. + +To ensure the information on the Security Dashboard is regularly updated, +[configure a scheduled pipeline](../../project/pipelines/schedules.md) to run a +daily security scan. This will update the information displayed on the Security +Dashboard regardless of how often the default branch is updated. + +That way, reports are created even if no code change happens. + +## Security scans using Auto DevOps + +When using [Auto DevOps](../../../topics/autodevops/index.md), use +[special environment variables](../../../topics/autodevops/index.md#environment-variables) +to configure daily security scans. diff --git a/doc/user/clusters/applications.md b/doc/user/clusters/applications.md new file mode 100644 index 00000000000..b520c4fb579 --- /dev/null +++ b/doc/user/clusters/applications.md @@ -0,0 +1,290 @@ +# GitLab Managed Apps + +GitLab provides **GitLab Managed Apps**, a one-click install for various applications which can +be added directly to your configured cluster. These applications are +needed for [Review Apps](../../ci/review_apps/index.md) and +[deployments](../../ci/environments.md) when using [Auto DevOps](../../topics/autodevops/index.md). +You can install them after you +[create a cluster](../project/clusters/index.md#adding-and-creating-a-new-gke-cluster-via-gitlab). + +## Installing applications + +Applications managed by GitLab will be installed onto the `gitlab-managed-apps` namespace. +This namespace: + +- Is different from the namespace used for project deployments. +- Is created once. +- Has a non-configurable name. + +To see a list of available applications to install: + +1. For a: + - Project-level cluster, navigate to your project's **Operations > Kubernetes**. + - Group-level cluster, navigate to your group's **Kubernetes** page. + +Install Helm first as it's used to install other applications. + +NOTE: **Note:** +As of GitLab 11.6, Helm will be upgraded to the latest version supported +by GitLab before installing any of the applications. + +The following applications can be installed: + +- [Helm](#helm) +- [Ingress](#ingress) +- [Cert-Manager](#cert-manager) +- [Prometheus](#prometheus) +- [GitLab Runner](#gitlab-runner) +- [JupyterHub](#jupyterhub) +- [Knative](#knative) + +With the exception of Knative, the applications will be installed in a dedicated +namespace called `gitlab-managed-apps`. + +NOTE: **Note:** +Some applications are installable only for a project-level cluster. +Support for installing these applications in a group-level cluster is +planned for future releases. +For updates, see [the issue tracking +progress](https://gitlab.com/gitlab-org/gitlab-ce/issues/51989). + +CAUTION: **Caution:** +If you have an existing Kubernetes cluster with Helm already installed, +you should be careful as GitLab cannot detect it. In this case, installing +Helm via the applications will result in the cluster having it twice, which +can lead to confusion during deployments. + +### Helm + +> - Available for project-level clusters since GitLab 10.2. +> - Available for group-level clusters since GitLab 11.6. + +[Helm](https://docs.helm.sh/) is a package manager for Kubernetes and is +required to install all the other applications. It is installed in its +own pod inside the cluster which can run the `helm` CLI in a safe +environment. + +### Cert-Manager + +> - Available for project-level clusters since GitLab 11.6. +> - Available for group-level clusters since GitLab 11.6. + +[Cert-Manager](https://docs.cert-manager.io/en/latest/) is a native +Kubernetes certificate management controller that helps with issuing +certificates. Installing Cert-Manager on your cluster will issue a +certificate by [Let's Encrypt](https://letsencrypt.org/) and ensure that +certificates are valid and up-to-date. + +NOTE: **Note:** +The +[stable/cert-manager](https://github.com/helm/charts/tree/master/stable/cert-manager) +chart is used to install this application with a +[`values.yaml`](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/vendor/cert_manager/values.yaml) +file. + +### GitLab Runner + +> - Available for project-level clusters since GitLab 10.6. +> - Available for group-level clusters since GitLab 11.10. + +[GitLab Runner](https://docs.gitlab.com/runner/) is the open source +project that is used to run your jobs and send the results back to +GitLab. It is used in conjunction with [GitLab +CI/CD](../../ci/README.md), the open-source continuous integration +service included with GitLab that coordinates the jobs. When installing +the GitLab Runner via the applications, it will run in **privileged +mode** by default. Make sure you read the [security +implications](../project/clusters/index.md/#security-implications) before doing so. + +NOTE: **Note:** +The +[runner/gitlab-runner](https://gitlab.com/charts/gitlab-runner) +chart is used to install this application with a +[`values.yaml`](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/vendor/runner/values.yaml) +file. + +### Ingress + +> - Available for project-level clusters since GitLab 10.2. +> - Available for group-level clusters since GitLab 11.6. + +[Ingress](https://kubernetes.github.io/ingress-nginx/) can provide load +balancing, SSL termination, and name-based virtual hosting. It acts as a +web proxy for your applications and is useful if you want to use [Auto +DevOps] or deploy your own web apps. + +NOTE: **Note:** +The +[stable/nginx-ingress](https://github.com/helm/charts/tree/master/stable/nginx-ingress) +chart is used to install this application with a +[`values.yaml`](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/vendor/ingress/values.yaml) +file. + +### JupyterHub + +> Available for project-level clusters since GitLab 11.0. + +[JupyterHub](https://jupyterhub.readthedocs.io/en/stable/) is a +multi-user service for managing notebooks across a team. [Jupyter +Notebooks](https://jupyter-notebook.readthedocs.io/en/latest/) provide a +web-based interactive programming environment used for data analysis, +visualization, and machine learning. + +Authentication will be enabled only for [project +members](../project/members/index.md) with [Developer or +higher](../permissions.md) access to the project. + +We use a [custom Jupyter +image](https://gitlab.com/gitlab-org/jupyterhub-user-image/blob/master/Dockerfile) +that installs additional useful packages on top of the base Jupyter. You +will also see ready-to-use DevOps Runbooks built with Nurtch's [Rubix library](https://github.com/amit1rrr/rubix). + +More information on +creating executable runbooks can be found in [our Runbooks +documentation](../project/clusters/runbooks/index.md#executable-runbooks). Note that +Ingress must be installed and have an IP address assigned before +JupyterHub can be installed. + +NOTE: **Note:** +The +[jupyter/jupyterhub](https://jupyterhub.github.io/helm-chart/) +chart is used to install this application with a +[`values.yaml`](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/vendor/jupyter/values.yaml) +file. + +#### Jupyter Git Integration + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/28783) in GitLab 12 for project-level clusters. + +When installing JupyterHub onto your Kubernetes cluster, [JupyterLab's Git extension](https://github.com/jupyterlab/jupyterlab-git) +is automatically provisioned and configured using the authenticated user's: + +- Name +- Email +- Newly created access token + +JupyterLab's Git extension enables full version control of your notebooks as well as issuance of Git commands within Jupyter. +Git commands can be issued via the **Git** tab on the left panel or via Jupyter's command line prompt. + +NOTE: **Note:** +JupyterLab's Git extension stores the user token in the JupyterHub DB in encrypted format +and in the single user Jupyter instance as plain text. This is because [Git requires storing +credentials as plain text](https://git-scm.com/docs/git-credential-store). Potentially, if +a nefarious user finds a way to read from the file system in the single user Jupyter instance +they could retrieve the token. + + + +You can clone repositories from the files tab in Jupyter: + + + +### Knative + +> Available for project-level clusters since GitLab 11.5. + +[Knative](https://cloud.google.com/knative) provides a platform to +create, deploy, and manage serverless workloads from a Kubernetes +cluster. It is used in conjunction with, and includes +[Istio](https://istio.io) to provide an external IP address for all +programs hosted by Knative. + +You will be prompted to enter a wildcard +domain where your applications will be exposed. Configure your DNS +server to use the external IP address for that domain. For any +application created and installed, they will be accessible as +`<program_name>.<kubernetes_namespace>.<domain_name>`. This will require +your kubernetes cluster to have [RBAC +enabled](../project/clusters/index.md#rbac-cluster-resources). + +NOTE: **Note:** +The +[knative/knative](https://storage.googleapis.com/triggermesh-charts) +chart is used to install this application. + +### Prometheus + +> - Available for project-level clusters since GitLab 10.4. +> - Available for group-level clusters since GitLab 11.11. + +[Prometheus](https://prometheus.io/docs/introduction/overview/) is an +open-source monitoring and alerting system useful to supervise your +deployed applications. + +NOTE: **Note:** +The +[stable/prometheus](https://github.com/helm/charts/tree/master/stable/prometheus) +chart is used to install this application with a +[`values.yaml`](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/vendor/prometheus/values.yaml) +file. + +## Upgrading applications + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/24789) +in GitLab 11.8. + +The applications below can be upgraded. + +| Application | GitLab version | +| ----------- | -------------- | +| Runner | 11.8+ | + +To upgrade an application: + +1. For a: + - Project-level cluster, navigate to your project's **Operations > Kubernetes**. + - Group-level cluster, navigate to your group's **Kubernetes** page. +1. Select your cluster. +1. If an upgrade is available, the **Upgrade** button is displayed. Click the button to upgrade. + +NOTE: **Note:** +Upgrades will reset values back to the values built into the `runner` +chart plus the values set by +[`values.yaml`](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/vendor/runner/values.yaml) + +## Uninstalling applications + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/issues/60665) in +> GitLab 11.11. + +The applications below can be uninstalled. + +| Application | GitLab version | Notes | +| ----------- | -------------- | ----- | +| Prometheus | 11.11+ | All data will be deleted and cannot be restored. | + +To uninstall an application: + +1. For a: + - Project-level cluster, navigate to your project's **Operations > Kubernetes**. + - Group-level cluster, navigate to your group's **Kubernetes** page. +1. Select your cluster. +1. Click the **Uninstall** button for the application. + +Support for uninstalling all applications is planned for progressive rollout. +To follow progress, see [the relevant +epic](https://gitlab.com/groups/gitlab-org/-/epics/1201). + +## Troubleshooting applications + +Applications can fail with the following error: + +```text +Error: remote error: tls: bad certificate +``` + +To avoid installation errors: + +- Before starting the installation of applications, make sure that time is synchronized + between your GitLab server and your Kubernetes cluster. +- Ensure certificates are not out of sync. When installing applications, GitLab expects a new cluster with no previous installation of Helm. + + You can confirm that the certificates match via `kubectl`: + + ```sh + kubectl get configmaps/values-content-configuration-ingress -n gitlab-managed-apps -o \ + "jsonpath={.data['cert\.pem']}" | base64 -d > a.pem + kubectl get secrets/tiller-secret -n gitlab-managed-apps -o "jsonpath={.data['ca\.crt']}" | base64 -d > b.pem + diff a.pem b.pem + ``` + diff --git a/doc/user/clusters/img/jupyter-git-extension.gif b/doc/user/clusters/img/jupyter-git-extension.gif Binary files differnew file mode 100644 index 00000000000..13a88d97425 --- /dev/null +++ b/doc/user/clusters/img/jupyter-git-extension.gif diff --git a/doc/user/clusters/img/jupyter-gitclone.png b/doc/user/clusters/img/jupyter-gitclone.png Binary files differnew file mode 100644 index 00000000000..41d467f806a --- /dev/null +++ b/doc/user/clusters/img/jupyter-gitclone.png diff --git a/doc/user/discussions/img/mr_review_resolve.png b/doc/user/discussions/img/mr_review_resolve.png Binary files differnew file mode 100644 index 00000000000..34176f3fa8e --- /dev/null +++ b/doc/user/discussions/img/mr_review_resolve.png diff --git a/doc/user/discussions/img/mr_review_resolve2.png b/doc/user/discussions/img/mr_review_resolve2.png Binary files differnew file mode 100644 index 00000000000..e4adb5f2c2d --- /dev/null +++ b/doc/user/discussions/img/mr_review_resolve2.png diff --git a/doc/user/discussions/img/mr_review_second_comment.png b/doc/user/discussions/img/mr_review_second_comment.png Binary files differnew file mode 100644 index 00000000000..920ea05ad66 --- /dev/null +++ b/doc/user/discussions/img/mr_review_second_comment.png diff --git a/doc/user/discussions/img/mr_review_second_comment_added.png b/doc/user/discussions/img/mr_review_second_comment_added.png Binary files differnew file mode 100644 index 00000000000..1fb54348547 --- /dev/null +++ b/doc/user/discussions/img/mr_review_second_comment_added.png diff --git a/doc/user/discussions/img/mr_review_start.png b/doc/user/discussions/img/mr_review_start.png Binary files differnew file mode 100644 index 00000000000..38b44bda0d2 --- /dev/null +++ b/doc/user/discussions/img/mr_review_start.png diff --git a/doc/user/discussions/img/mr_review_unresolve.png b/doc/user/discussions/img/mr_review_unresolve.png Binary files differnew file mode 100644 index 00000000000..da895ceb89f --- /dev/null +++ b/doc/user/discussions/img/mr_review_unresolve.png diff --git a/doc/user/discussions/img/mr_review_unresolve2.png b/doc/user/discussions/img/mr_review_unresolve2.png Binary files differnew file mode 100644 index 00000000000..a824b806e4a --- /dev/null +++ b/doc/user/discussions/img/mr_review_unresolve2.png diff --git a/doc/user/discussions/img/multi-line-suggestion-preview.png b/doc/user/discussions/img/multi-line-suggestion-preview.png Binary files differnew file mode 100644 index 00000000000..4288d0ba034 --- /dev/null +++ b/doc/user/discussions/img/multi-line-suggestion-preview.png diff --git a/doc/user/discussions/img/multi-line-suggestion-syntax.png b/doc/user/discussions/img/multi-line-suggestion-syntax.png Binary files differnew file mode 100644 index 00000000000..df0c99b84ef --- /dev/null +++ b/doc/user/discussions/img/multi-line-suggestion-syntax.png diff --git a/doc/user/discussions/img/pending_review_comment.png b/doc/user/discussions/img/pending_review_comment.png Binary files differnew file mode 100644 index 00000000000..916ef5b7452 --- /dev/null +++ b/doc/user/discussions/img/pending_review_comment.png diff --git a/doc/user/discussions/img/reply_to_comment.gif b/doc/user/discussions/img/reply_to_comment.gif Binary files differnew file mode 100644 index 00000000000..c62f7fdb5fe --- /dev/null +++ b/doc/user/discussions/img/reply_to_comment.gif diff --git a/doc/user/discussions/img/reply_to_comment_button.png b/doc/user/discussions/img/reply_to_comment_button.png Binary files differnew file mode 100644 index 00000000000..d362b19785c --- /dev/null +++ b/doc/user/discussions/img/reply_to_comment_button.png diff --git a/doc/user/discussions/img/review_comment_quickactions.png b/doc/user/discussions/img/review_comment_quickactions.png Binary files differnew file mode 100644 index 00000000000..bd9880c329a --- /dev/null +++ b/doc/user/discussions/img/review_comment_quickactions.png diff --git a/doc/user/discussions/img/review_preview.png b/doc/user/discussions/img/review_preview.png Binary files differnew file mode 100644 index 00000000000..4bf53a81b9c --- /dev/null +++ b/doc/user/discussions/img/review_preview.png diff --git a/doc/user/discussions/index.md b/doc/user/discussions/index.md index 1e2fad1efe9..5d69efc3600 100644 --- a/doc/user/discussions/index.md +++ b/doc/user/discussions/index.md @@ -5,36 +5,42 @@ The ability to contribute conversationally is offered throughout GitLab. You can leave a comment in the following places: - issues +- epics **[ULTIMATE]** - merge requests - snippets - commits - commit diffs -The comment area supports [Markdown] and [quick actions]. One can edit their -own comment at any time, and anyone with [Maintainer access level][permissions] or -higher can also edit a comment made by someone else. +There are standard comments, and you also have the option to create a comment +in the form of a threaded discussion. A comment can also be [turned into a discussion](#start-a-discussion-by-replying-to-a-standard-comment) +when it receives a reply. -You could also reply to the notification email in order to reply to a comment, -provided that [Reply by email] is configured by your GitLab admin. This also -supports [Markdown] and [quick actions] as if replied from the web. +The comment area supports [Markdown] and [quick actions]. You can edit your own +comment at any time, and anyone with [Maintainer access level][permissions] or +higher can also edit a comment made by someone else. -Apart from the standard comments, you also have the option to create a comment -in the form of a resolvable or threaded discussion. +You can also reply to a comment notification email to reply to the comment if +[Reply by email] is configured for your GitLab instance. Replying to a standard comment +creates another standard comment. Replying to a discussion comment creates a reply in the +discussion thread. Email replies support [Markdown] and [quick actions], just as if you replied from the web. -## Resolvable discussions +## Resolvable comments and discussions > **Notes:** +> > - The main feature was [introduced][ce-5022] in GitLab 8.11. > - Resolvable discussions can be added only to merge request diffs. Discussion resolution helps keep track of progress during planning or code review. -Resolving comments prevents you from forgetting to address feedback and lets you -hide discussions that are no longer relevant. -!["A discussion between two people on a piece of code"][discussion-view] +Every standard comment or discussion thread in merge requests, commits, commit diffs, and +snippets is initially displayed as unresolved. They can then be individually resolved by anyone +with at least Developer access to the project or by the author of the change being reviewed. -Comments and discussions can be resolved by anyone with at least Developer -access to the project or the author of the merge request. +The need to resolve all standard comments or discussions prevents you from forgetting +to address feedback and lets you hide discussions that are no longer relevant. + +!["A discussion between two people on a piece of code"][discussion-view] ### Commit discussions in the context of a merge request @@ -272,7 +278,76 @@ edit existing comments. Non-team members are restricted from adding or editing c | :-----------: | :----------: | |  |  | -Additionally locked issues can not be reopened. +Additionally, locked issues and merge requests can not be reopened. + +## Merge Request Reviews **[PREMIUM]** + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/4213) in GitLab 11.4. + +When looking at a Merge Request diff, you are able to start a review. +This allows you to create comments inside a Merge Request that are **only visible to you** until published, +in order to allow you to submit them all as a single action. + +### Starting a review + +In order to start a review, simply write a comment on a diff as normal under the **Changes** tab +in an MR and click on the **Start a review** button. + + + +Once a review is started, you will see any comments that are part of this review marked `Pending`. +All comments that are part of a review show two buttons: + +- **Submit review**: Submits all comments that are part of the review, making them visible to other users. +- **Add comment now**: Submits the specific comment as a regular comment instead of as part of the review. + + + +You can use [quick actions] inside review comments. The comment will show the actions that will be performed once published. + + + +To add more comments to a review, start writing a comment as normal and click the **Add to review** button. + + + +This will add the comment to the review. + + + +### Resolving/Unresolving discussions + +Review comments can also resolve/unresolve [resolvable discussions](#resolvable-comments-and-discussions). +When replying to a comment, you will see a checkbox that you can click in order to resolve or unresolve +the discussion once published. + + + + +If a particular pending comment will resolve or unresolve the discussion, this will be shown on the pending +comment itself. + + + + +### Submitting a review + +If you have any comments that have not been submitted, you will see a bar at the +bottom of the screen with two buttons: + +- **Discard**: Discards all comments that have not been submitted. +- **Finish review**: Opens a list of comments ready to be submitted for review. + Clicking **Submit review** will publish all comments. Any quick actions + submitted are performed at this time. + +Alternatively, every pending comment has a button to finish the entire review. + + + +Submitting the review will send a single email to every notifiable user of the +merge request with all the comments associated to it. + +Replying to this email will, consequentially, create a new comment on the associated merge request. ## Filtering notes @@ -310,11 +385,6 @@ the Merge Request authored by the user that applied them.  - > **Note:** - The suggestion will only affect the commented line. Multi-line - suggestions are currently not supported. Will be introduced by - [#53310](https://gitlab.com/gitlab-org/gitlab-ce/issues/53310). - 1. In the comment, add your suggestion to the pre-populated code block:  @@ -326,19 +396,58 @@ the Merge Request authored by the user that applied them.  - > **Note:** - Discussions are _not_ automatically resolved. Will be introduced by - [#54405](https://gitlab.com/gitlab-org/gitlab-ce/issues/54405). - Once the author applies a suggestion, it will be marked with the **Applied** label, -and GitLab will create a new commit with the message `Apply suggestion to <file-name>` -and push the suggested change directly into the codebase in the merge request's branch. +the discussion will be automatically resolved, and GitLab will create a new commit +with the message `Apply suggestion to <file-name>` and push the suggested change +directly into the codebase in the merge request's branch. [Developer permission](../permissions.md) is required to do so. > **Note:** Custom commit messages will be introduced by [#54404](https://gitlab.com/gitlab-org/gitlab-ce/issues/54404). +### Multi-line suggestions + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/issues/53310) in GitLab 11.10. + +Reviewers can also suggest changes to multiple lines with a single suggestion +within Merge Request diff discussions by adjusting the range offsets. The +offsets are relative to the position of the diff discussion, and specify the +range to be replaced by the suggestion when it is applied. + + + +In the example above, the suggestion covers three lines above and four lines +below the commented line. When applied, it would replace from 3 lines _above_ +to 4 lines _below_ the commented line, with the suggested change. + + + +NOTE: **Note:** +Suggestions covering multiple lines are limited to 100 lines _above_ and 100 +lines _below_ the commented diff line, allowing up to 200 changed lines per +suggestion. + +## Start a discussion by replying to a standard comment + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/issues/30299) in GitLab 11.9 + +To reply to a standard (non-discussion) comment, you can use the **Reply to comment** button. + + + +The **Reply to comment** button is only displayed if you have permissions to reply to an existing discussion, or start a discussion from a standard comment. + +Clicking on the **Reply to comment** button will bring the reply area into focus and you can type your reply. + + + +Relying to a non-discussion comment will convert the non-discussion comment to a +threaded discussion once the reply is submitted. This conversion is considered an edit +to the original comment, so a note about when it was last edited will appear underneath it. + +This feature only exists for Issues, Merge requests, and Epics. Commits, Snippets and Merge request diff discussions are not supported yet. + [ce-5022]: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/5022 [ce-7125]: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/7125 [ce-7527]: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/7527 diff --git a/doc/user/gitlab_com/index.md b/doc/user/gitlab_com/index.md index 762cf911fcf..2fb6cec55fa 100644 --- a/doc/user/gitlab_com/index.md +++ b/doc/user/gitlab_com/index.md @@ -51,7 +51,7 @@ Below are the settings for [GitLab Pages]. | TLS certificates support| yes | no | The maximum size of your Pages site is regulated by the artifacts maximum size -which is part of [GitLab CI/CD](#gitlab-ci-cd). +which is part of [GitLab CI/CD](#gitlab-cicd). ## GitLab CI/CD @@ -71,6 +71,14 @@ or over the size limit, you can [reduce your repository size with Git](../projec | ----------- | ----------------- | ------------- | | Repository size including LFS | 10G | Unlimited | +## IP range + +GitLab.com, CI/CD, and related services are deployed into Google Cloud Platform (GCP). Any +IP based firewall can be configured by looking up all +[IP address ranges or CIDR blocks for GCP](https://cloud.google.com/compute/docs/faq#where_can_i_find_product_name_short_ip_ranges). + +[Static endpoints](https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/5071) are being considered. + ## Shared Runners Shared Runners on GitLab.com run in [autoscale mode] and powered by diff --git a/doc/user/group/clusters/index.md b/doc/user/group/clusters/index.md index 8d223d1c5f0..3c5e820c1ca 100644 --- a/doc/user/group/clusters/index.md +++ b/doc/user/group/clusters/index.md @@ -1,3 +1,7 @@ +--- +type: reference +--- + # Group-level Kubernetes clusters > [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/issues/34758) in GitLab 11.6. @@ -5,28 +9,17 @@ ## Overview -Similar to [project Kubernetes -clusters](../../project/clusters/index.md), Group-level Kubernetes -clusters allow you to connect a Kubernetes cluster to your group, -enabling you to use the same cluster across multiple projects. +Similar to [project-level](../../project/clusters/index.md) and +[instance-level](../../instance/clusters/index.md) Kubernetes clusters, +group-level Kubernetes clusters allow you to connect a Kubernetes cluster to +your group, enabling you to use the same cluster across multiple projects. ## Installing applications -GitLab provides a one-click install for various applications that can be -added directly to your cluster. - -NOTE: **Note:** -Applications will be installed in a dedicated namespace called -`gitlab-managed-apps`. If you have added an existing Kubernetes cluster -with Tiller already installed, you should be careful as GitLab cannot -detect it. In this event, installing Tiller via the applications will -result in the cluster having it twice. This can lead to confusion during -deployments. - -| Application | GitLab version | Description | Helm Chart | -| ----------- | -------------- | ----------- | ---------- | -| [Helm Tiller](https://docs.helm.sh) | 10.2+ | Helm is a package manager for Kubernetes and is required to install all the other applications. It is installed in its own pod inside the cluster which can run the `helm` CLI in a safe environment. | n/a | -| [Ingress](https://kubernetes.io/docs/concepts/services-networking/ingress) | 10.2+ | Ingress can provide load balancing, SSL termination, and name-based virtual hosting. It acts as a web proxy for your applications and is useful if you want to use [Auto DevOps](../../../topics/autodevops/index.md) or deploy your own web apps. | [stable/nginx-ingress](https://github.com/helm/charts/tree/master/stable/nginx-ingress) | +GitLab can install and manage some applications in your group-level +cluster. For more information on installing, upgrading, uninstalling, +and troubleshooting applications for your group cluster, see +[Gitlab Managed Apps](../../clusters/applications.md). ## RBAC compatibility @@ -56,9 +49,32 @@ group. That way you can have different clusters for different environments, like dev, staging, production, etc. Add another cluster similar to the first one and make sure to -[set an environment scope](#environment-scopes) that will +[set an environment scope](#environment-scopes-premium) that will differentiate the new cluster from the rest. +## GitLab-managed clusters + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/22011) in GitLab 11.5. +> Became [optional](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/26565) in GitLab 11.11. + +NOTE: **Note:** +Only available when creating clusters. Existing clusters not managed by GitLab +cannot become GitLab-managed later. + +You can choose to allow GitLab to manage your cluster for you. If your cluster is +managed by GitLab, resources for your projects will be automatically created. See the +[Access controls](../../project/clusters/index.md#access-controls) section for details on which resources will +be created. + +If you choose to manage your own cluster, project-specific resources will not be created +automatically. If you are using [Auto DevOps](../../../topics/autodevops/index.md), you will +need to explicitly provide the `KUBE_NAMESPACE` [deployment variable](../../project/clusters/index.md#deployment-variables) +that will be used by your deployment jobs. + +NOTE: **Note:** +If you [install applications](#installing-applications) on your cluster, GitLab will create +the resources required to run these even if you have chosen to manage your own cluster. + ## Base domain > [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/24580) in GitLab 11.8. @@ -72,11 +88,10 @@ The domain should have a wildcard DNS configured to the Ingress IP address. ## Environment scopes **[PREMIUM]** -When adding more than one Kubernetes cluster to your project, you need -to differentiate them with an environment scope. The environment scope -associates clusters with [environments](../../../ci/environments.md) -similar to how the [environment-specific -variables](../../../ci/variables/README.md#limiting-environment-scopes-of-variables) +When adding more than one Kubernetes cluster to your project, you need to differentiate +them with an environment scope. The environment scope associates clusters with +[environments](../../../ci/environments.md) similar to how the +[environment-specific variables](../../../ci/variables/README.md#limiting-environment-scopes-of-environment-variables-premium) work. While evaluating which environment matches the environment scope of a @@ -133,4 +148,16 @@ The following features are not currently available for group-level clusters: 1. Terminals (see [related issue](https://gitlab.com/gitlab-org/gitlab-ce/issues/55487)). 1. Pod logs (see [related issue](https://gitlab.com/gitlab-org/gitlab-ce/issues/55488)). -1. Deployment boards (see [related issue](https://gitlab.com/gitlab-org/gitlab-ce/issues/55488)). +1. Deployment boards (see [related issue](https://gitlab.com/gitlab-org/gitlab-ce/issues/55489)). + +<!-- ## Troubleshooting + +Include any troubleshooting steps that you can foresee. If you know beforehand what issues +one might have when setting this up, or when something is changed, or on upgrading, it's +important to describe those, too. Think of things that may go wrong and include them here. +This is important to minimize requests for support, and to avoid doc comments with +questions that you know someone might ask. + +Each scenario can be a third-level heading, e.g. `### Getting error message X`. +If you have none to add when creating a doc, leave this section in place +but commented out to help encourage others to add to it in the future. --> diff --git a/doc/user/group/contribution_analytics/img/group_stats_cal.png b/doc/user/group/contribution_analytics/img/group_stats_cal.png Binary files differnew file mode 100644 index 00000000000..0238c7bf088 --- /dev/null +++ b/doc/user/group/contribution_analytics/img/group_stats_cal.png diff --git a/doc/user/group/contribution_analytics/img/group_stats_graph.png b/doc/user/group/contribution_analytics/img/group_stats_graph.png Binary files differnew file mode 100644 index 00000000000..ccfd3782c6f --- /dev/null +++ b/doc/user/group/contribution_analytics/img/group_stats_graph.png diff --git a/doc/user/group/contribution_analytics/img/group_stats_table.png b/doc/user/group/contribution_analytics/img/group_stats_table.png Binary files differnew file mode 100644 index 00000000000..f1d1031fa18 --- /dev/null +++ b/doc/user/group/contribution_analytics/img/group_stats_table.png diff --git a/doc/user/group/contribution_analytics/index.md b/doc/user/group/contribution_analytics/index.md new file mode 100644 index 00000000000..7e6cb24a51e --- /dev/null +++ b/doc/user/group/contribution_analytics/index.md @@ -0,0 +1,77 @@ +--- +type: reference +--- + +# Contribution Analytics **[STARTER]** + +> Introduced in [GitLab Starter](https://about.gitlab.com/pricing/) 8.3. + +## Overview + +With Contribution Analytics you can get an overview of the following activity in your +group: + +- Issues +- Merge requests +- Push events + +To view the Contribution Analytics, go to your group's **Overview > Contribution Analytics** +page. + +## Use cases + +- Analyze your team's contributions over a period of time, and offer a bonus for the top +contributors. +- Identify opportunities for improvement with group members who may benefit from additional +support. + +## Using Contribution Analytics + +There are three main bar graphs that illustrate the number of contributions per group +member for the following: + +- Push events +- Merge requests +- Closed issues + +Hover over each bar to display the number of events for a specific group member. + + + +## Changing the period time + +You can choose from the following three periods: + +- Last week (default) +- Last month +- Last three months + +Select the desired period from the calendar dropdown. + + + +## Sorting by different factors + +Contributions per group member are also presented in tabular format. Click a column header to sort the table by that column: + +* Member name +* Number of pushed events +* Number of opened issues +* Number of closed issues +* Number of opened MRs +* Number of accepted MRs +* Number of total contributions + + + +<!-- ## Troubleshooting + +Include any troubleshooting steps that you can foresee. If you know beforehand what issues +one might have when setting this up, or when something is changed, or on upgrading, it's +important to describe those, too. Think of things that may go wrong and include them here. +This is important to minimize requests for support, and to avoid doc comments with +questions that you know someone might ask. + +Each scenario can be a third-level heading, e.g. `### Getting error message X`. +If you have none to add when creating a doc, leave this section in place +but commented out to help encourage others to add to it in the future. --> diff --git a/doc/user/group/custom_project_templates.md b/doc/user/group/custom_project_templates.md index 8e101407ac0..aa088d2fcdb 100644 --- a/doc/user/group/custom_project_templates.md +++ b/doc/user/group/custom_project_templates.md @@ -1,4 +1,8 @@ -# Custom group-level project templates **[PREMIUM ONLY]** +--- +type: reference +--- + +# Custom group-level project templates **[PREMIUM]** > [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/issues/6861) in [GitLab Premium](https://about.gitlab.com/pricing) 11.6. @@ -24,3 +28,15 @@ Projects of nested subgroups of a selected template source cannot be used. Repository and database information that are copied over to each new project are identical to the data exported with [GitLab's Project Import/Export](../project/settings/import_export.md). + +<!-- ## Troubleshooting + +Include any troubleshooting steps that you can foresee. If you know beforehand what issues +one might have when setting this up, or when something is changed, or on upgrading, it's +important to describe those, too. Think of things that may go wrong and include them here. +This is important to minimize requests for support, and to avoid doc comments with +questions that you know someone might ask. + +Each scenario can be a third-level heading, e.g. `### Getting error message X`. +If you have none to add when creating a doc, leave this section in place +but commented out to help encourage others to add to it in the future. --> diff --git a/doc/user/group/dependency_proxy/img/group_dependency_proxy.png b/doc/user/group/dependency_proxy/img/group_dependency_proxy.png Binary files differnew file mode 100644 index 00000000000..035aff0b6c4 --- /dev/null +++ b/doc/user/group/dependency_proxy/img/group_dependency_proxy.png diff --git a/doc/user/group/dependency_proxy/index.md b/doc/user/group/dependency_proxy/index.md new file mode 100644 index 00000000000..4fc2d8e9509 --- /dev/null +++ b/doc/user/group/dependency_proxy/index.md @@ -0,0 +1,74 @@ +# Dependency Proxy **[PREMIUM]** + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/issues/7934) in [GitLab Premium](https://about.gitlab.com/pricing/) 11.11. + +NOTE: **Note:** +This is the user guide. In order to use the dependency proxy, an administrator +must first [configure it](../../../administration/dependency_proxy.md). + +For many organizations, it is desirable to have a local proxy for frequently used +upstream images/packages. In the case of CI/CD, the proxy is responsible for +receiving a request and returning the upstream image from a registry, acting +as a pull-through cache. + +The dependency proxy is available in the group level. To access it, navigate to +a group's **Overview > Dependency Proxy**. + + + +## Supported dependency proxies + +NOTE: **Note:** +For a list of the upcoming additions to the proxies, visit the +[direction page](https://about.gitlab.com/direction/package/dependency_proxy/#top-vision-items). + +The following dependency proxies are supported. + +| Dependency proxy | GitLab version | +| ---------------- | -------------- | +| Docker | 11.11+ | + +## Using the Docker dependency proxy + +With the Docker dependency proxy, you can use GitLab as a source for a Docker image. +To get a Docker image into the dependency proxy: + +1. Find the proxy URL on your group's page under **Overview > Dependency Proxy**, + for example `gitlab.com/groupname/dependency_proxy/containers`. +1. Trigger GitLab to pull the Docker image you want (e.g., `alpine:latest` or + `linuxserver/nextcloud:latest`) and store it in the proxy storage by using + one of the following ways: + + - Manually pulling the Docker image: + + ```bash + docker pull gitlab.com/groupname/dependency_proxy/containers/alpine:latest + ``` + + - From a `Dockerfile`: + + ```bash + FROM gitlab.com/groupname/dependency_proxy/containers/alpine:latest + ``` + + - In [`.gitlab-ci.yml`](../../../ci/yaml/README.md#image): + + ```bash + image: gitlab.com/groupname/dependency_proxy/containers/alpine:latest + ``` + +GitLab will then pull the Docker image from Docker Hub and will cache the blobs +on the GitLab server. The next time you pull the same image, it will get the latest +information about the image from Docker Hub but will serve the existing blobs +from GitLab. + +The blobs are kept forever, and there is no hard limit on how much data can be +stored. + +## Limitations + +The following limitations apply: + +- Only public groups are supported (authentication is not supported yet). +- Only Docker Hub is supported. +- This feature requires Docker Hub being available. diff --git a/doc/user/group/epics/img/button_close_epic.png b/doc/user/group/epics/img/button_close_epic.png Binary files differnew file mode 100644 index 00000000000..aa1a889ea23 --- /dev/null +++ b/doc/user/group/epics/img/button_close_epic.png diff --git a/doc/user/group/epics/img/button_reopen_epic.png b/doc/user/group/epics/img/button_reopen_epic.png Binary files differnew file mode 100644 index 00000000000..7a953189270 --- /dev/null +++ b/doc/user/group/epics/img/button_reopen_epic.png diff --git a/doc/user/group/epics/img/child_epics_roadmap.png b/doc/user/group/epics/img/child_epics_roadmap.png Binary files differnew file mode 100644 index 00000000000..819fed58989 --- /dev/null +++ b/doc/user/group/epics/img/child_epics_roadmap.png diff --git a/doc/user/group/epics/img/containing_epic.png b/doc/user/group/epics/img/containing_epic.png Binary files differnew file mode 100644 index 00000000000..5ed693e6d6a --- /dev/null +++ b/doc/user/group/epics/img/containing_epic.png diff --git a/doc/user/group/epics/img/epic_view.png b/doc/user/group/epics/img/epic_view.png Binary files differnew file mode 100644 index 00000000000..dbda98e4351 --- /dev/null +++ b/doc/user/group/epics/img/epic_view.png diff --git a/doc/user/group/epics/img/epics_list_view.png b/doc/user/group/epics/img/epics_list_view.png Binary files differnew file mode 100644 index 00000000000..b30608d9d31 --- /dev/null +++ b/doc/user/group/epics/img/epics_list_view.png diff --git a/doc/user/group/epics/img/epics_search.png b/doc/user/group/epics/img/epics_search.png Binary files differnew file mode 100644 index 00000000000..96335527468 --- /dev/null +++ b/doc/user/group/epics/img/epics_search.png diff --git a/doc/user/group/epics/img/epics_sort.png b/doc/user/group/epics/img/epics_sort.png Binary files differnew file mode 100644 index 00000000000..b23c65fd0ef --- /dev/null +++ b/doc/user/group/epics/img/epics_sort.png diff --git a/doc/user/group/epics/index.md b/doc/user/group/epics/index.md new file mode 100644 index 00000000000..2e4106f55e5 --- /dev/null +++ b/doc/user/group/epics/index.md @@ -0,0 +1,233 @@ +--- +type: reference, howto +--- + +# Epics **[ULTIMATE]** + +> Introduced in [GitLab Ultimate][ee] 10.2. + +Epics let you manage your portfolio of projects more efficiently and with less +effort by tracking groups of issues that share a theme, across projects and +milestones. + + + +## Use cases + +- Suppose your team is working on a large feature that involves multiple discussions throughout different issues created in distinct projects within a [Group](../index.md). With Epics, you can track all the related activities that together contribute to that single feature. +- Track when the work for the group of issues is targeted to begin, and when it is targeted to end. +- Discuss and collaborate on feature ideas and scope at a high-level. + +## Creating an epic + +A paginated list of epics is available in each group from where you can create +a new epic. The list of epics includes also epics from all subgroups of the +selected group. From your group page: + +1. Go to **Epics** +1. Click the **New epic** button at the top right +1. Enter a descriptive title and hit **Create epic** + +Once created, you will be taken to the view for that newly-created epic where +you can change its title, description, start date, and due date. + + + +## Adding an issue to an epic + +An epic contains a list of issues and an issue can be associated with at most +one epic. When on an epic, you can add its associated issues: + +1. Click the plus icon (<kbd>+</kbd>) under the epic description. +1. Paste the link of the issue (you can hit <kbd>Spacebar</kbd> to add more than + one issues at a time). +1. Click **Add**. + +Any issue belonging to a project in the epic's group or any of the epic's +subgroups are eligible to be added. To remove an issue from an epic, click +on the <kbd>x</kbd> button in the epic's issue list. + +NOTE: **Note:** +When you add an issue or an epic to an epic that's already associated with another epic, +the issue or the epic is automatically removed from the previous epic. + +## Multi-level child epics + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/issues/8333) in GitLab Ultimate 11.7. + +Much like adding issues to an epic, an epic can have multiple child epics with +the maximum depth being 5. To add a child epic: + +1. Click the plus icon (<kbd>+</kbd>) under the epic description. +1. Paste the link of the epic. +1. Click **Add**. + +Any epic that belongs to a group or subgroup of the parent epic's group is +eligible to be added. To remove a child epic from a parent epic, +click on the <kbd>x</kbd> button in the parent epic's epic list. + +## Start date and due date + +For each of the dates in the sidebar of an epic, you can choose to either: + +- Enter a fixed value. +- Inherit a dynamic value called "From milestones". + +If you select "From milestones" for the start date, GitLab will automatically set the +date to be earliest start date across all milestones that are currently assigned +to the issues that are attached to the epic. Similarly, if you select "From milestones" +for the due date, GitLab will set it to be the latest due date across all +milestones that are currently assigned to those issues. + +These are dynamic dates in that if milestones are re-assigned to the issues, if the +milestone dates change, or if issues are added or removed from the epic, then +the re-calculation will happen immediately to set a new dynamic date. + +## Roadmap in epics + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/issues/7327) in [GitLab Ultimate](https://about.gitlab.com/pricing) 11.10. + +If your epic contains one or more [child epics](#multi-level-child-epics) which +have a [start or due date](#start-date-and-due-date), then you can see a +[roadmap](../roadmap/index.md) view of the child epics under the parent epic itself. + + + +## Reordering issues and child epics + +Drag and drop to reorder issues and child epics. New issues and child epics added to an epic appear at the top of the list. + +## Deleting an epic + +NOTE: **Note:** +To delete an epic, you need to be an [Owner][permissions] of a group/subgroup. + +When inside a single epic view, click the **Delete** button to delete the epic. +A modal will pop-up to confirm your action. + +Deleting an epic releases all existing issues from their associated epic in the +system. + +## Closing and reopening epics + +### Using buttons + +Whenever you decide that there is no longer need for that epic, +close the epic using the close button: + + + +You can always reopen it using the reopen button. + + + +### Using quick actions + +You can close or reopen an epic using [Quick actions](../../project/quick_actions.md) + +## Navigating to an epic from an issue + +If an issue belongs to an epic, you can navigate to the containing epic with the +link in the issue sidebar. + + + +## Promoting an issue to an epic + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/issues/3777) in [GitLab Ultimate](https://about.gitlab.com/pricing/) 11.6. + +If you have [permissions](../../permissions.md) to close an issue and create an +epic in the parent group, you can promote an issue to an epic with the `/promote` +[quick action](../../project/quick_actions.md#quick-actions-for-epics-ultimate). +Only issues from projects that are in groups can be promoted. + +When the quick action is executed: + +- An epic is created in the same group as the project of the issue. +- Subscribers of the issue are notified that the epic was created. + +The following issue metadata will be copied to the epic: + +- Title, description, activity/comment thread. +- Upvotes/downvotes. +- Participants. +- Group labels that the issue already has. + +## Searching for an epic from epics list page + +> Introduced in [GitLab Ultimate][ee] 10.5. + +You can search for an epic from the list of epics using filtered search bar (similar to +that of Issues and Merge requests) based on following parameters: + +- Title or description +- Author name / username +- Labels + + + +To search, go to the list of epics and click on the field **Search or filter results...**. +It will display a dropdown menu, from which you can add an author. You can also enter plain +text to search by epic title or description. When done, press <kbd>Enter</kbd> on your +keyboard to filter the list. + +You can also sort epics list by: + +- **Created date** +- **Last updated** +- **Start date** +- **Due date** + +Each option contains a button that can toggle the order between **ascending** and **descending**. The sort option and order will be persisted to be used wherever epics are browsed including the [roadmap](../roadmap/index.md). + + + +## Permissions + +If you have access to view an epic and have access to view an issue already +added to that epic, then you can view the issue in the epic issue list. + +If you have access to edit an epic and have access to edit an issue, then you +can add the issue to or remove it from the epic. + +Note that for a given group, the visibility of all projects must be the same as +the group, or less restrictive. That means if you have access to a group's epic, +then you already have access to its projects' issues. + +You may also consult the [group permissions table][permissions]. + +[ee]: https://about.gitlab.com/pricing/ +[permissions]: ../../permissions.md#group-members-permissions + +## Thread + +- Comments: collaborate on that epic by posting comments in its thread. +These text fields also fully support +[GitLab Flavored Markdown](../../markdown.md#gitlab-flavored-markdown-gfm). + +## Comment, or start a discussion + +Once you wrote your comment, you can either: + +- Click "Comment" and your comment will be published. +- Click "Start discussion": start a thread within that epic's thread to discuss specific points. + +## Award emoji + +- You can [award an emoji](../../award_emojis.md) to that epic or its comments. + +## Notifications + +- [Receive notifications](../../../workflow/notifications.md) for epic events. + +<!-- ## Troubleshooting + +Include any troubleshooting steps that you can foresee. If you know beforehand what issues +one might have when setting this up, or when something is changed, or on upgrading, it's +important to describe those, too. Think of things that may go wrong and include them here. +This is important to minimize requests for support, and to avoid doc comments with +questions that you know someone might ask. + +Each scenario can be a third-level heading, e.g. `### Getting error message X`. +If you have none to add when creating a doc, leave this section in place +but commented out to help encourage others to add to it in the future. --> diff --git a/doc/user/group/img/create_new_group_info.png b/doc/user/group/img/create_new_group_info.png Binary files differindex c2e6ed43c5b..bd724240c37 100644 --- a/doc/user/group/img/create_new_group_info.png +++ b/doc/user/group/img/create_new_group_info.png diff --git a/doc/user/group/img/group_file_template_dropdown.png b/doc/user/group/img/group_file_template_dropdown.png Binary files differnew file mode 100644 index 00000000000..d9caae1a223 --- /dev/null +++ b/doc/user/group/img/group_file_template_dropdown.png diff --git a/doc/user/group/img/group_file_template_settings.png b/doc/user/group/img/group_file_template_settings.png Binary files differnew file mode 100644 index 00000000000..ca42f7726db --- /dev/null +++ b/doc/user/group/img/group_file_template_settings.png diff --git a/doc/user/group/img/member_lock.png b/doc/user/group/img/member_lock.png Binary files differnew file mode 100644 index 00000000000..3f1382e76c6 --- /dev/null +++ b/doc/user/group/img/member_lock.png diff --git a/doc/user/group/img/membership_lock.png b/doc/user/group/img/membership_lock.png Binary files differdeleted file mode 100644 index c9ad82c90f2..00000000000 --- a/doc/user/group/img/membership_lock.png +++ /dev/null diff --git a/doc/user/group/img/new_group_form.png b/doc/user/group/img/new_group_form.png Binary files differdeleted file mode 100644 index 1c4d3ec6ceb..00000000000 --- a/doc/user/group/img/new_group_form.png +++ /dev/null diff --git a/doc/user/group/index.md b/doc/user/group/index.md index 300e0115c60..eb0c7bc998f 100644 --- a/doc/user/group/index.md +++ b/doc/user/group/index.md @@ -1,35 +1,49 @@ +--- +type: reference, howto +--- + # Groups -With GitLab Groups you can assemble related projects together -and grant members access to several projects at once. +With GitLab Groups, you can: + +- Assemble related projects together. +- Grant members access to several projects at once. Groups can also be nested in [subgroups](subgroups/index.md). -Find your groups by expanding the left menu and clicking **Groups**: +Find your groups by clicking **Groups > Your Groups** in the top navigation.  -The Groups page displays all groups you are a member of, how many projects it holds, -how many members it has, the group visibility, and, if you have enough permissions, -a link to the group settings. By clicking the last button you can leave that group. +> The **Groups** dropdown in the top navigation was [introduced](https://gitlab.com/gitlab-org/gitlab-ce/issues/36234) in [GitLab 11.1](https://about.gitlab.com/2018/07/22/gitlab-11-1-released/#groups-dropdown-in-navigation). + +The **Groups** page displays: + +- All groups you are a member of, when **Your groups** is selected. +- A list of public groups, when **Explore public groups** is selected. + +Each group on the **Groups** page is listed with: + +- How many subgroups it has. +- How many projects it contains. +- How many members the group has, not including members inherited from parent groups. +- The group's visibility. +- A link to the group's settings, if you have sufficient permissions. +- A link to leave the group, if you are a member. ## Use cases -You can create groups for numerous reasons. To name a few: - -- Organize related projects under the same [namespace](#namespaces), add members to that - group and grant access to all their projects at once -- Create a group, include members of your team, and make it easier to - `@mention` all the team at once in issues and merge requests - - Create a group for your company members, and create [subgroups](subgroups/index.md) - for each individual team. Let's say you create a group called `company-team`, and among others, - you created subgroups in this group for each individual team `backend-team`, - `frontend-team`, and `production-team`: - 1. When you start a new implementation from an issue, you add a comment: +You can create groups for numerous reasons. To name a couple: + +- Grant access to multiple projects and multiple team members in fewer steps by organizing related projects under the same [namespace](#namespaces) and adding members to the top-level group. +- Make it easier to `@mention` all of your team at once in issues and merge requests by creating a group and including the appropriate members. + +For example, you could create a group for your company members, and create a [subgroup](subgroups/index.md) for each individual team. Let's say you create a group called `company-team`, and you create subgroups in this group for the individual teams `backend-team`, `frontend-team`, and `production-team`. + - When you start a new implementation from an issue, you add a comment: _"`@company-team`, let's do it! `@company-team/backend-team` you're good to go!"_ - 1. When your backend team needs help from frontend, they add a comment: + - When your backend team needs help from frontend, they add a comment: _"`@company-team/frontend-team` could you help us here please?"_ - 1. When the frontend team completes their implementation, they comment: + - When the frontend team completes their implementation, they comment: _"`@company-team/backend-team`, it's done! Let's ship it `@company-team/production-team`!"_ ## Namespaces @@ -44,7 +58,7 @@ For example, consider a user named Alex: 1. Alex creates an account on GitLab.com with the username `alex`; their profile will be accessed under `https://gitlab.example.com/alex` -1. Alex creates a group for their team with the groupname `alex-team`; +1. Alex creates a group for their team with the group name `alex-team`; the group and its projects will be accessed under `https://gitlab.example.com/alex-team` 1. Alex creates a subgroup of `alex-team` with the subgroup name `marketing`; this subgroup and its projects will be accessed under `https://gitlab.example.com/alex-team/marketing` @@ -57,23 +71,22 @@ By doing so: ## Issues and merge requests within a group -Issues and merge requests are part of projects. For a given group, view all the -[issues](../project/issues/index.md#issues-per-group) and [merge requests](../project/merge_requests/index.md#merge-requests-per-group) across all the projects in that group, +Issues and merge requests are part of projects. For a given group, you can view all of the +[issues](../project/issues/index.md#issues-list) and [merge requests](../project/merge_requests/index.md#merge-requests-per-group) across all projects in that group, together in a single list view. ## Create a new group -> **Notes:** -> - For a list of words that are not allowed to be used as group names see the -> [reserved names](../reserved_names.md). +> For a list of words that are not allowed to be used as group names see the +> [reserved names](../reserved_names.md). -You can create a group in GitLab from: +To create a new Group, either: -1. The Groups page: expand the left menu, click **Groups**, and click the green button **New group**: +- In the top menu, click **Groups** and then **Your Groups**, and click the green button **New group**.  -1. Elsewhere: expand the `plus` sign button on the top navbar and choose **New group**: +- Or, in the top menu, expand the `plus` sign and choose **New group**.  @@ -81,57 +94,58 @@ Add the following information:  -1. Set the **Group path** which will be the **namespace** under which your projects - will be hosted (path can contain only letters, digits, underscores, dashes - and dots; it cannot start with dashes or end in dot). -1. The **Group name** will populate with the path. Optionally, you can change - it. This is the name that will display in the group views. -1. Optionally, you can add a description so that others can briefly understand +1. The **Group name** will automatically populate the URL. Optionally, you can change it. + This is the name that displays in group views. + The name can contain only: + - Alphanumeric characters + - Underscores + - Dashes and dots + - Spaces +1. The **Group URL** is the namespace under which your projects will be hosted. + The URL can contain only: + - Alphanumeric characters + - Underscores + - Dashes and dots (it cannot start with dashes or end in a dot) +1. Optionally, you can add a brief description to tell others what this group is about. -1. Optionally, choose an avatar for your project. +1. Optionally, choose an avatar for your group. 1. Choose the [visibility level](../../public_access/public_access.md). ## Add users to a group -Add members to a group by navigating to the group's dashboard, and clicking **Members**: +A benefit of putting multiple projects in one group is that you can +give a user to access to all projects in the group with one action. - +Add members to a group by navigating to the group's dashboard and clicking **Members**. -Select the [permission level](../permissions.md#permissions) and add the new member. You can also set the expiring -date for that user, from which they will no longer have access to your group. + -One of the benefits of putting multiple projects in one group is that you can -give a user to access to all projects in the group with one action. +Select the [permission level](../permissions.md#permissions), and add the new member. You can also set the expiring date for that user; this is the date on which they will no longer have access to your group. -Consider we have a group with two projects: +Consider a group with two projects: -- On the **Group Members** page we can now add a new user to the group. -- Now because this user is a **Developer** member of the group, he automatically +- On the **Group Members** page, you can now add a new user to the group. +- Now, because this user is a **Developer** member of the group, they automatically gets **Developer** access to **all projects** within that group. -If necessary, you can increase the access level of an individual user for a specific project, -by adding them again as a new member to the project with the new permission levels. +To increase the access level of an existing user for a specific project, +add them again as a new member to the project with the desired permission level. ## Request access to a group -As a group owner you can enable or disable non members to request access to -your group. Go to the group settings and click on **Allow users to request access**. +As a group owner, you can enable or disable the ability for non members to request access to +your group. Go to the group settings, and click **Allow users to request access**. -As a user, you can request to be a member of a group. Go to the group you'd -like to be a member of, and click the **Request Access** button on the right +As a user, you can request to be a member of a group, if that setting is enabled. Go to the group for which you'd like to be a member, and click the **Request Access** button on the right side of your screen.  ---- - Group owners and maintainers will be notified of your request and will be able to approve or decline it on the members page.  ---- - If you change your mind before your request is approved, just click the **Withdraw Access Request** button. @@ -141,57 +155,102 @@ If you change your mind before your request is approved, just click the There are two different ways to add a new project to a group: -- Select a group and then click on the **New project** button. +- Select a group, and then click **New project**. You can then continue [creating your project](../../gitlab-basics/create-project.md).  - You can then continue on [creating a project](../../gitlab-basics/create-project.md). - - While you are creating a project, select a group namespace you've already created from the dropdown menu.  +### Default project-creation level + +> [Introduced][ee-2534] in [GitLab Premium][ee] 10.5. +> Brought to [GitLab Starter][ee] in 10.7. +> [Moved](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/25975) to [GitLab Core](https://about.gitlab.com/pricing/) in 11.10. + +Group owners and administrators can allow users with the +Developer role to create projects under groups. + +By default, [Developers and Maintainers](../permissions.md#group-members-permissions) can create projects under a group. You can change this setting for a specific group within the group settings, or +you can set this option globally in the Admin area +at **Settings > General > Visibility and access controls** (you must be a GitLab administrator). + +Available settings are `No one`, `Maintainers`, or `Developers + Maintainers`. + ## Transfer projects into groups -Learn how to [transfer a project into a group](../project/index.md#transfer-an-existing-project-into-a-group). +Learn how to [transfer a project into a group](../project/settings/index.md#transferring-an-existing-project-into-another-namespace). ## Sharing a project with a group You can [share your projects with a group](../project/members/share_project_with_groups.md) -and give your group members access to the project all at once. +and give all group members access to the project at once. Alternatively, you can [lock the sharing with group feature](#share-with-group-lock). ## Manage group memberships via LDAP -In GitLab Enterprise Edition it is possible to manage GitLab group memberships using LDAP groups. +In GitLab Enterprise Edition, it is possible to manage GitLab group memberships using LDAP groups. See [the GitLab Enterprise Edition documentation](../../integration/ldap.md) for more information. -## Transfer groups to another group +## Epics **[ULTIMATE]** + +> Introduced in [GitLab Ultimate][ee] 10.2. + +Epics let you manage your portfolio of projects more efficiently and with less +effort by tracking groups of issues that share a theme, across projects and +milestones. + +[Learn more about Epics.](epics/index.md) -From 10.5 there are two different ways to transfer a group: +## Group Security Dashboard **[ULTIMATE]** -- Either by transferring a group into another group (making it a subgroup of that group). -- Or by converting a subgroup into a root group (a group with no parent). +Get an overview of the vulnerabilities of all the projects in a group and its subgroups. -Please make sure to understand that: +[Learn more about the Group Security Dashboard.](security_dashboard/index.md) -- Changing a group's parent can have unintended side effects. See [Redirects when changing repository paths](https://docs.gitlab.com/ce/user/project/index.html#redirects-when-changing-repository-paths) -- You can only transfer the group to a group you manage. -- You will need to update your local repositories to point to the new location. -- If the parent group's visibility is lower than the group current visibility, visibility levels for subgroups and projects will be changed to match the new parent group's visibility. +## Insights **[ULTIMATE]** + +> Introduced in [GitLab Ultimate][ee] 11.9 behind the `insights` feature flag. + +Configure the Insights that matter for your groups or projects, allowing users to explore data +such as: + +- Triage hygiene +- Issues created/closed per a given period +- Average time for merge requests to be merged +- Much more + +[Learn more about Insights](insights/index.md). + +## Transferring groups + +From GitLab 10.5, you can transfer groups in the following ways: + +- Transfer a subgroup to a new parent group. +- Convert a top-level group into a subgroup by transfering it to the desired group. +- Convert a subgroup into a top-level group by transfering it out of its current group. + +When transferring groups, note: + +- Changing a group's parent can have unintended side effects. See [Redirects when changing repository paths](../project/index.md#redirects-when-changing-repository-paths). +- You can only transfer groups to groups you manage. +- You must update your local repositories to point to the new location. +- If the parent group's visibility is lower than the group's current visibility, visibility levels for subgroups and projects will change to match the new parent group's visibility. +- Only explicit group membership is transferred, not inherited membership. If the group's owners have only inherited membership, this leaves the group without an owner. In this case, the user transferring the group becomes the group's owner. ## Group settings -Once you have created a group, you can manage its settings by navigating to +After creating a group, you can manage its settings by navigating to the group's dashboard, and clicking **Settings**.  ### General settings -Besides giving you the option to edit any settings you've previously +In addition to editing any settings you previously set when [creating the group](#create-a-new-group), you can also access further configurations for your group. @@ -202,14 +261,14 @@ Changing a group's path can have unintended side effects. Read before proceeding. If you are vacating the path so it can be claimed by another group or user, -you may need to rename the group name as well since both names and paths must +you may need to rename the group, too, since both names and paths must be unique. To change your group path: 1. Navigate to your group's **Settings > General**. -1. Enter a new name under "Group path". -1. Hit **Save group**. +1. Enter a new name under **Group path**. +1. Click **Save group**. CAUTION: **Caution:** It is currently not possible to rename a namespace if it contains a @@ -225,20 +284,17 @@ username, you can create a new group and transfer projects to it. Add a security layer to your group by [enforcing two-factor authentication (2FA)](../../security/two_factor_authentication.md#enforcing-2fa-for-all-users-in-a-group) -to all group members. +for all group members. #### Share with group lock Prevent projects in a group from [sharing -a project with another group](../project/members/share_project_with_groups.md). -This allows for tighter control over project access. +a project with another group](../project/members/share_project_with_groups.md) to enable tighter control over project access. -For example, consider you have two distinct teams (Group A and Group B) -working together in a project. -To inherit the group membership, you share the project between the +For example, let's say you have two distinct teams (Group A and Group B) working together in a project, and to inherit the group membership, you share the project between the two groups A and B. **Share with group lock** prevents any project within -the group from being shared with another group. By doing so, you -guarantee only the right group members have access to that projects. +the group from being shared with another group, +guaranteeing that only the right group members have access to those projects. To enable this feature, navigate to the group settings page. Select **Share with group lock** and **Save the group**. @@ -247,29 +303,88 @@ To enable this feature, navigate to the group settings page. Select #### Member Lock **[STARTER]** -With **Member Lock** it is possible to lock membership in project to the -level of members in group. +Member lock lets a group owner prevent any new project membership to all of the +projects within a group, allowing tighter control over project membership. + +For example, if you want to lock the group for an [Audit Event](../../administration/audit_events.md), +enable Member lock to guarantee that project membership cannot be modified during that audit. + +To enable this feature: + +1. Navigate to the group's **Settings > General** page. +1. Expand the **Permissions, LFS, 2FA** section, and select **Member lock**. +1. Click **Save changes**. + + + +This will disable the option for all users who previously had permissions to +operate project memberships, so no new users can be added. Furthermore, any +request to add a new user to a project through API will not be possible. -Learn more about [Member Lock](https://docs.gitlab.com/ee/user/group/index.html#member-lock). +#### Group file templates **[PREMIUM]** -#### Group-level file templates **[PREMIUM]** +Group file templates allow you to share a set of templates for common file +types with every project in a group. It is analogous to the +[instance template repository](../admin_area/settings/instance_template_repository.md) +feature, and the selected project should follow the same naming conventions as +are documented on that page. -Group-level file templates allow you to share a set of templates for common file -types with every project in a group. +You can only choose projects in the group as the template source. +This includes projects shared with the group, but it **excludes** projects in +subgroups or parent groups of the group being configured. -Learn more about [Group-level file templates](https://docs.gitlab.com/ee/user/group/index.html#group-level-file-templates-premium). +You can configure this feature for both subgroups and parent groups. A project +in a subgroup will have access to the templates for that subgroup, as well as +any parent groups. + + + +To enable this feature, navigate to the group settings page, expand the +**Templates** section, choose a project to act as the template repository, and +**Save group**. + + #### Group-level project templates **[PREMIUM]** -Define project templates at a group-level by setting a group as a template source. +Define project templates at a group level by setting a group as the template source. [Learn more about group-level project templates](custom_project_templates.md). ### Advanced settings -- **Projects**: view all projects within that group, add members to each project, - access each project's settings, and remove any project from the same screen. -- **Webhooks**: configure [webhooks](../project/integrations/webhooks.md) to your group. -- **Kubernetes cluster integration**: connect your GitLab group with [Kubernetes clusters](clusters/index.md). -- **Audit Events**: view [Audit Events](https://docs.gitlab.com/ee/administration/audit_events.html#audit-events) +- **Projects**: View all projects within that group, add members to each project, + access each project's settings, and remove any project, all from the same screen. +- **Webhooks**: Configure [webhooks](../project/integrations/webhooks.md) for your group. +- **Kubernetes cluster integration**: Connect your GitLab group with [Kubernetes clusters](clusters/index.md). +- **Audit Events**: View [Audit Events](../../administration/audit_events.md) for the group. **[STARTER ONLY]** -- **Pipelines quota**: keep track of the [pipeline quota](../admin_area/settings/continuous_integration.md) for the group. +- **Pipelines quota**: Keep track of the [pipeline quota](../admin_area/settings/continuous_integration.md) for the group. + +## User contribution analysis **[STARTER]** + +With [GitLab Contribution Analytics](contribution_analytics/index.md), +you have an overview of the contributions (pushes, merge requests, +and issues) performed by your group members. + +## Issues analytics **[PREMIUM]** + +With [GitLab Issues Analytics](issues_analytics/index.md), you can see a bar chart of the number of issues created each month in your groups. + +## Dependency Proxy **[PREMIUM]** + +Use GitLab as a [dependency proxy](dependency_proxy/index.md) for upstream Docker images. + +<!-- ## Troubleshooting + +Include any troubleshooting steps that you can foresee. If you know beforehand what issues +one might have when setting this up, or when something is changed, or on upgrading, it's +important to describe those, too. Think of things that may go wrong and include them here. +This is important to minimize requests for support, and to avoid doc comments with +questions that you know someone might ask. + +Each scenario can be a third-level heading, e.g. `### Getting error message X`. +If you have none to add when creating a doc, leave this section in place +but commented out to help encourage others to add to it in the future. --> + +[ee]: https://about.gitlab.com/pricing/ +[ee-2534]: https://gitlab.com/gitlab-org/gitlab-ee/issues/2534 diff --git a/doc/user/group/insights/img/insights_example_stacked_bar_chart.png b/doc/user/group/insights/img/insights_example_stacked_bar_chart.png Binary files differnew file mode 100644 index 00000000000..791d0e4bcdf --- /dev/null +++ b/doc/user/group/insights/img/insights_example_stacked_bar_chart.png diff --git a/doc/user/group/insights/img/insights_group_configuration.png b/doc/user/group/insights/img/insights_group_configuration.png Binary files differnew file mode 100644 index 00000000000..0af0073e448 --- /dev/null +++ b/doc/user/group/insights/img/insights_group_configuration.png diff --git a/doc/user/group/insights/img/insights_sidebar_link.png b/doc/user/group/insights/img/insights_sidebar_link.png Binary files differnew file mode 100644 index 00000000000..64bbd1fc4d3 --- /dev/null +++ b/doc/user/group/insights/img/insights_sidebar_link.png diff --git a/doc/user/group/insights/index.md b/doc/user/group/insights/index.md new file mode 100644 index 00000000000..a4ea71074ec --- /dev/null +++ b/doc/user/group/insights/index.md @@ -0,0 +1,62 @@ +--- +type: reference, howto +--- + +# Insights **[ULTIMATE]** + +> Introduced in [GitLab Ultimate](https://about.gitlab.com/pricing/) 11.9 behind the `insights` feature flag. + +CAUTION: **Beta:** +Insights is considered beta, and is not ready for production use. +Follow [gitlab-org/quality/team-tasks#137](https://gitlab.com/gitlab-org/quality/team-tasks/issues/137#general-availability) +for updates. + +Configure the Insights that matter for your groups to explore data such as +triage hygiene, issues created/closed per a given period, average time for merge +requests to be merged and much more. + + + +## View your group's Insights + +You can access your group's Insights by clicking the **Overview > Insights** +link in the left sidebar: + + + +## Configure your Insights + +Navigate to your group's **Settings > General**, expand **Insights**, and choose +the project that holds your `.gitlab/insights.yml` configuration file: + + + +If no configuration was set, a [default configuration file]( +https://gitlab.com/gitlab-org/gitlab-ee/blob/master/ee/fixtures/insights/default.yml) +will be used. + +See the [Project's Insights documentation](../../project/insights/index.md) for +more details about the `.gitlab/insights.yml` configuration file. + +## Permissions + +If you have access to view a group, then you have access to view their Insights. + +NOTE: **Note:** +Issues or merge requests that you don't have access to (because you don't have +access to the project they belong to, or because they are confidential) are +filtered out of the Insights charts. + +You may also consult the [group permissions table](../../permissions.md#group-members-permissions). + +<!-- ## Troubleshooting + +Include any troubleshooting steps that you can foresee. If you know beforehand what issues +one might have when setting this up, or when something is changed, or on upgrading, it's +important to describe those, too. Think of things that may go wrong and include them here. +This is important to minimize requests for support, and to avoid doc comments with +questions that you know someone might ask. + +Each scenario can be a third-level heading, e.g. `### Getting error message X`. +If you have none to add when creating a doc, leave this section in place +but commented out to help encourage others to add to it in the future. --> diff --git a/doc/user/group/issues_analytics/img/issues_created_per_month.png b/doc/user/group/issues_analytics/img/issues_created_per_month.png Binary files differnew file mode 100644 index 00000000000..96f0d36c917 --- /dev/null +++ b/doc/user/group/issues_analytics/img/issues_created_per_month.png diff --git a/doc/user/group/issues_analytics/index.md b/doc/user/group/issues_analytics/index.md new file mode 100644 index 00000000000..46d5c1e2e09 --- /dev/null +++ b/doc/user/group/issues_analytics/index.md @@ -0,0 +1,43 @@ +--- +type: reference +--- + +# Issues Analytics **[PREMIUM]** + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/issues/7478) in [GitLab Premium](https://about.gitlab.com/pricing/) 11.5. + +Issues Analytics is a bar graph which illustrates the number of issues created each month. +The default timespan is 13 months, which includes the current month, and the 12 months +prior. + +To access the chart, navigate to a group's sidebar and select **Issues > Analytics**. + +Hover over each bar to see the total number of issues. + +To narrow the scope of issues included in the graph, enter your criteria in the +**Search or filter results...** field. Criteria from the following list can be typed in or selected from a menu: + +- Author +- Assignee +- Milestone +- Label +- My reaction +- Weight + +You can change the total number of months displayed by setting a URL parameter. +For example, `https://gitlab.com/groups/gitlab-org/-/issues_analytics?months_back=15` +shows a total of 15 months for the chart in the GitLab.org group. + + + +<!-- ## Troubleshooting + +Include any troubleshooting steps that you can foresee. If you know beforehand what issues +one might have when setting this up, or when something is changed, or on upgrading, it's +important to describe those, too. Think of things that may go wrong and include them here. +This is important to minimize requests for support, and to avoid doc comments with +questions that you know someone might ask. + +Each scenario can be a third-level heading, e.g. `### Getting error message X`. +If you have none to add when creating a doc, leave this section in place +but commented out to help encourage others to add to it in the future. --> diff --git a/doc/user/group/roadmap/img/epics_state_dropdown.png b/doc/user/group/roadmap/img/epics_state_dropdown.png Binary files differnew file mode 100644 index 00000000000..cbcc3658a54 --- /dev/null +++ b/doc/user/group/roadmap/img/epics_state_dropdown.png diff --git a/doc/user/group/roadmap/img/roadmap_timeline_months.png b/doc/user/group/roadmap/img/roadmap_timeline_months.png Binary files differnew file mode 100644 index 00000000000..5a046b21507 --- /dev/null +++ b/doc/user/group/roadmap/img/roadmap_timeline_months.png diff --git a/doc/user/group/roadmap/img/roadmap_timeline_quarters.png b/doc/user/group/roadmap/img/roadmap_timeline_quarters.png Binary files differnew file mode 100644 index 00000000000..56f428cb471 --- /dev/null +++ b/doc/user/group/roadmap/img/roadmap_timeline_quarters.png diff --git a/doc/user/group/roadmap/img/roadmap_timeline_weeks.png b/doc/user/group/roadmap/img/roadmap_timeline_weeks.png Binary files differnew file mode 100644 index 00000000000..bf4c1dc0284 --- /dev/null +++ b/doc/user/group/roadmap/img/roadmap_timeline_weeks.png diff --git a/doc/user/group/roadmap/img/roadmap_view.png b/doc/user/group/roadmap/img/roadmap_view.png Binary files differnew file mode 100644 index 00000000000..ff41a2e0441 --- /dev/null +++ b/doc/user/group/roadmap/img/roadmap_view.png diff --git a/doc/user/group/roadmap/index.md b/doc/user/group/roadmap/index.md new file mode 100644 index 00000000000..683c715c8d5 --- /dev/null +++ b/doc/user/group/roadmap/index.md @@ -0,0 +1,87 @@ +--- +type: reference +--- + +# Roadmap **[ULTIMATE]** + +> Introduced in [GitLab Ultimate](https://about.gitlab.com/pricing) 10.5. + +An Epic within a group containing **Start date** and/or **Due date** +can be visualized in a form of a timeline (e.g. a Gantt chart). The Epics Roadmap page +shows such a visualization for all the epics which are under a group and/or its subgroups. + + + +A dropdown allows you to show only open or closed epics. By default, all epics are shown. + + + +Epics in the view can be sorted by: + +- **Created date** +- **Last updated** +- **Start date** +- **Due date** + +Each option contains a button that toggles the sort order between **ascending** and **descending**. The sort option and order will be persisted when browsing Epics, +including the [epics list view](../epics/index.md). + +Roadmaps can also be [visualized inside an epic](../epics/index.md#roadmap-in-epics). + +## Timeline duration + +> Introduced in [GitLab Ultimate](https://about.gitlab.com/pricing) 11.0. + +Roadmap supports the following date ranges: + +- Quarters +- Months (Default) +- Weeks + +### Quarters + + + +In _Quarters_ preset, roadmap shows epics which have start or due dates _falling within_ or +_going through_ **past quarter**, **current quarter** and **next 4 quarters**, where _today_ +is shown by the vertical red line in the timeline. The sub-headers underneath the quarter name on +the timeline header represent the month of the quarter. + +### Months + + + +In _Months_ preset, roadmap shows epics which have start or due dates _falling within_ or +_going through_ **past month**, **current month** and **next 5 months**, where _today_ +is shown by the vertical red line in the timeline. The sub-headers underneath the month name on +the timeline header represent the date on starting day (Sunday) of the week. This preset is +selected by default. + +### Weeks + + + +In _Weeks_ preset, roadmap shows epics which have start or due dates _falling within_ or +_going through_ **past week**, **current week** and **next 4 weeks**, where _today_ +is shown by the vertical red line in the timeline. The sub-headers underneath the week name on +the timeline header represent the days of the week. + +## Timeline bar for an epic + +The timeline bar indicates the approximate position of an epic based on its start +and due date. If an epic doesn't have a due date, the timeline bar fades +away towards the future. Similarly, if an epic doesn't have a start date, the +timeline bar becomes more visible as it approaches the epic's due date on the +timeline. + +<!-- ## Troubleshooting + +Include any troubleshooting steps that you can foresee. If you know beforehand what issues +one might have when setting this up, or when something is changed, or on upgrading, it's +important to describe those, too. Think of things that may go wrong and include them here. +This is important to minimize requests for support, and to avoid doc comments with +questions that you know someone might ask. + +Each scenario can be a third-level heading, e.g. `### Getting error message X`. +If you have none to add when creating a doc, leave this section in place +but commented out to help encourage others to add to it in the future. --> diff --git a/doc/user/group/saml_sso/img/group_saml_configuration_information.png b/doc/user/group/saml_sso/img/group_saml_configuration_information.png Binary files differnew file mode 100644 index 00000000000..98b83d0cb0f --- /dev/null +++ b/doc/user/group/saml_sso/img/group_saml_configuration_information.png diff --git a/doc/user/group/saml_sso/img/group_saml_settings.png b/doc/user/group/saml_sso/img/group_saml_settings.png Binary files differnew file mode 100644 index 00000000000..d95acb5075f --- /dev/null +++ b/doc/user/group/saml_sso/img/group_saml_settings.png diff --git a/doc/user/group/saml_sso/img/scim_advanced.png b/doc/user/group/saml_sso/img/scim_advanced.png Binary files differnew file mode 100644 index 00000000000..3b70e3fbe83 --- /dev/null +++ b/doc/user/group/saml_sso/img/scim_advanced.png diff --git a/doc/user/group/saml_sso/img/scim_attribute_mapping.png b/doc/user/group/saml_sso/img/scim_attribute_mapping.png Binary files differnew file mode 100644 index 00000000000..c9f6b71f5b0 --- /dev/null +++ b/doc/user/group/saml_sso/img/scim_attribute_mapping.png diff --git a/doc/user/group/saml_sso/img/scim_token.png b/doc/user/group/saml_sso/img/scim_token.png Binary files differnew file mode 100644 index 00000000000..7eb52bf6ea2 --- /dev/null +++ b/doc/user/group/saml_sso/img/scim_token.png diff --git a/doc/user/group/saml_sso/img/unlink_group_saml.png b/doc/user/group/saml_sso/img/unlink_group_saml.png Binary files differnew file mode 100644 index 00000000000..0561443b5f4 --- /dev/null +++ b/doc/user/group/saml_sso/img/unlink_group_saml.png diff --git a/doc/user/group/saml_sso/index.md b/doc/user/group/saml_sso/index.md new file mode 100644 index 00000000000..fcfd638f185 --- /dev/null +++ b/doc/user/group/saml_sso/index.md @@ -0,0 +1,140 @@ +--- +type: reference, howto +--- + +# SAML SSO for GitLab.com Groups **[SILVER ONLY]** + +> Introduced in [GitLab.com Silver](https://about.gitlab.com/pricing/) 11.0. + +NOTE: **Note:** +This topic is for SAML on GitLab.com Silver tier and above. For SAML on self-managed GitLab instances, see [SAML OmniAuth Provider](../../../integration/saml.md). + +SAML on GitLab.com allows users to be automatically added to a group, and then allows those users to sign into GitLab.com. Users should already have an account on the GitLab instance, or can create one when logging in for the first time. + +User synchronization for GitLab.com is partially supported using [SCIM](scim_setup.md). + +NOTE: **Note:** +SAML SSO for GitLab.com groups does not sync users between providers without using SCIM. If a group is not using SCIM, group Owners will still need to manage user accounts (for example, removing users when necessary). + +## Configuring your Identity Provider + +1. Navigate to the group and click **Settings > SAML SSO**. +1. Configure your SAML server using the **Assertion consumer service URL** and **Identifier**. Alternatively GitLab provides [metadata XML configuration](#metadata-configuration). See [your identity provider's documentation](#providers) for more details. +1. Configure the SAML response to include a NameID that uniquely identifies each user. +1. Configure required assertions using the [table below](#assertions). +1. Once the identity provider is set up, move on to [configuring GitLab](#configuring-gitlab). + + + +### SSO enforcement + +SSO enforcement was: + +- [Introduced in GitLab 11.8](https://gitlab.com/gitlab-org/gitlab-ee/issues/5291). +- [Improved upon in GitLab 11.11 with ongoing enforcement in the GitLab UI](https://gitlab.com/gitlab-org/gitlab-ee/issues/9255). + +With this option enabled, users must use your group's GitLab single sign on URL to be added to the group or be added via SCIM. Users cannot be added manually, and may only access project/group resources via the UI by signing in through the SSO URL. + +We intend to add a similar SSO requirement for [Git and API activity](https://gitlab.com/gitlab-org/gitlab-ee/issues/9152) in the future. + +### NameID + +GitLab.com uses the SAML NameID to identify users. The NameID element: + +- Is a required field in the SAML response. +- Must be unique to each user. +- Must be a persistent value that will never change, such as a unique ID or username. Email could also be used as the NameID, but only if it can be guaranteed to never change. + +### Assertions + +| Field | Supported keys | +|-------|----------------| +| Email (required)| `email`, `mail` | +| Full Name | `name` | +| First Name | `first_name`, `firstname`, `firstName` | +| Last Name | `last_name`, `lastname`, `lastName` | + +## Metadata configuration + +GitLab provides metadata XML that can be used to configure your Identity Provider. + +1. Navigate to the group and click **Settings > SAML SSO**. +1. Copy the provided **GitLab metadata URL** +1. Follow your Identity Provider's documentation and paste the metadata URL when it is requested. + +## Configuring GitLab + +Once you've set up your identity provider to work with GitLab, you'll need to configure GitLab to use it for authentication: + +1. Navigate to the group's **Settings > SAML SSO**. +1. Find the SSO URL from your Identity Provider and enter it the **Identity provider single sign on URL** field. +1. Find and enter the fingerprint for the SAML token signing certificate in the **Certificate** field. +1. Check the **Enable SAML authentication for this group** checkbox. +1. Click the **Save changes** button. + + + +## Providers + +| Provider | Documentation | +|----------|---------------| +| ADFS (Active Directory Federation Services) | [Create a Relying Party Trust](https://docs.microsoft.com/en-us/windows-server/identity/ad-fs/operations/create-a-relying-party-trust) | +| Azure | [Configuring single sign-on to applications](https://docs.microsoft.com/en-us/azure/active-directory/active-directory-saas-custom-apps) | +| Auth0 | [Auth0 as Identity Provider](https://auth0.com/docs/protocols/saml/saml-idp-generic) | +| G Suite | [Set up your own custom SAML application](https://support.google.com/a/answer/6087519?hl=en) | +| JumpCloud | [Single Sign On (SSO) with GitLab](https://support.jumpcloud.com/customer/en/portal/articles/2810701-single-sign-on-sso-with-gitlab) | +| Okta | [Setting up a SAML application in Okta](https://developer.okta.com/standards/SAML/setting_up_a_saml_application_in_okta) | +| OneLogin | [Use the OneLogin SAML Test Connector](https://onelogin.service-now.com/support?id=kb_article&sys_id=93f95543db109700d5505eea4b96198f) | +| Ping Identity | [Add and configure a new SAML application](https://docs.pingidentity.com/bundle/p1_enterpriseConfigSsoSaml_cas/page/enableAppWithoutURL.html) | + +## Linking SAML to your existing GitLab.com account + +To link SAML to your existing GitLab.com account: + +1. Sign in to your GitLab.com account. +1. Locate the SSO URL for the group you are signing in to. A group Admin can find this on the group's **Settings > SAML SSO** page. +1. Visit the SSO URL and click **Authorize**. +1. Enter your credentials on the Identity Provider if prompted. +1. You will be redirected back to GitLab.com and should now have access to the group. In the future, you can use SAML to sign in to GitLab.com. + +## Signing in to GitLab.com with SAML + +1. Locate the SSO URL for the group you are signing in to. A group Admin can find this on a group's **Settings > SAML SSO** page. If configured, it might also be possible to sign in to GitLab starting from your Identity Provider. +1. Visit the SSO URL and click the **Sign in with Single Sign-On** button. +1. Enter your credentials on the Identity Provider if prompted. +1. You will be signed in to GitLab.com and redirected to the group. + +## Unlinking accounts + +Users can unlink SAML for a group from their profile page. This can be helpful if: + +- You no longer want a group to be able to sign you in to GitLab.com. +- Your SAML NameID has changed and so GitLab can no longer find your user. + +For example, to unlink the `MyOrg` account, the following **Disconnect** button will be available under **Profile > Accounts**: + + + +## Glossary + +| Term | Description | +|------|-------------| +| Identity Provider | The service which manages your user identities such as ADFS, Okta, Onelogin or Ping Identity. | +| Service Provider | SAML considers GitLab to be a service provider. | +| Assertion | A piece of information about a user's identity, such as their name or role. Also know as claims or attributes. | +| SSO | Single Sign On. | +| Assertion consumer service URL | The callback on GitLab where users will be redirected after successfully authenticating with the identity provider. | +| Issuer | How GitLab identifies itself to the identity provider. Also known as a "Relying party trust identifier". | +| Certificate fingerprint | Used to confirm that communications over SAML are secure by checking that the server is signing communications with the correct certificate. Also known as a certificate thumbprint. | + +<!-- ## Troubleshooting + +Include any troubleshooting steps that you can foresee. If you know beforehand what issues +one might have when setting this up, or when something is changed, or on upgrading, it's +important to describe those, too. Think of things that may go wrong and include them here. +This is important to minimize requests for support, and to avoid doc comments with +questions that you know someone might ask. + +Each scenario can be a third-level heading, e.g. `### Getting error message X`. +If you have none to add when creating a doc, leave this section in place +but commented out to help encourage others to add to it in the future. --> diff --git a/doc/user/group/saml_sso/scim_setup.md b/doc/user/group/saml_sso/scim_setup.md new file mode 100644 index 00000000000..c00628bf909 --- /dev/null +++ b/doc/user/group/saml_sso/scim_setup.md @@ -0,0 +1,102 @@ +# SCIM provisioning using SAML SSO for Groups **[SILVER ONLY]** + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/9388) in [GitLab.com Silver](https://about.gitlab.com/pricing/) 11.10. + +GitLab's [SCIM API](../../../api/scim.md) implements part of [the RFC7644 protocol](https://tools.ietf.org/html/rfc7644). + +Currently, the following actions are available: + +- CREATE +- UPDATE +- DELETE (deprovisioning) + +The following identity providers are supported: + +- Azure + +## Requirements + +- [Group SSO](index.md) needs to be configured. +- The `scim_group` feature flag must be enabled: + + Run the following commands in a Rails console: + + ```sh + # Omnibus GitLab + gitlab-rails console + + # Installation from source + cd /home/git/gitlab + sudo -u git -H bin/rails console RAILS_ENV=production + ``` + + To enable SCIM for a group named `group_name`: + + ```ruby + group = Group.find_by_full_path('group_name') + Feature.enable(:group_scim, group) + ``` + +### GitLab configuration + +Once [Single sign-on](index.md) has been configured, we can: + +1. Navigate to the group and click **Settings > SAML SSO**. +1. Click on the **Generate a SCIM token** button. +1. Save the token and URL so they can be used in the next step. + + + +## SCIM IdP configuration + +### Configuration on Azure + +In the [Single sign-on](index.md) configuration for the group, make sure +that the **Name identifier value** (NameID) points to a unique identifier, such +as the `user.objectid`. This will match the `extern_uid` used on GitLab. + +The GitLab app in Azure needs to be configured following +[Azure's SCIM setup](https://docs.microsoft.com/en-us/azure/active-directory/manage-apps/use-scim-to-provision-users-and-groups#getting-started). + +Note the following: + +- The `Tenant URL` and `secret token` are the ones retrieved in the +[previous step](#gitlab-configuration). +- Should there be any problems with the availability of GitLab or similar +errors, the notification email set will get those. +- For mappings, we will only leave `Synchronize Azure Active Directory Users to AppName` enabled. + +You can then test the connection clicking on `Test Connection`. + +### Synchronize Azure Active Directory users + +1. Click on `Synchronize Azure Active Directory Users to AppName`, to configure +the attribute mapping. +1. Select the unique identifier (in the example `objectId`) as the `id` and `externalId`, +and enable the `Create`, `Update`, and `Delete` actions. +1. Map the `userPricipalName` to `emails[type eq "work"].value` and `mailNickname` to +`userName`. + + Example configuration: + +  + +1. Click on **Show advanced options > Edit attribute list for AppName**. +1. Leave the `id` as the primary and only required field. + + NOTE: **Note:** + `username` should neither be primary nor required as we don't support + that field on GitLab SCIM yet. + +  + +1. Save all the screens and, in the **Provisioning** step, set +the `Provisioning Status` to `ON`. + + NOTE: **Note:** + You can control what is actually synced by selecting the `Scope`. For example, + `Sync only assigned users and groups` will only sync the users assigned to + the application (`Users and groups`), otherwise it will sync the whole Active Directory. + +Once enabled, the synchronization details and any errors will appear on the +bottom of the **Provisioning** screen, together with a link to the audit logs. diff --git a/doc/user/group/security_dashboard/index.md b/doc/user/group/security_dashboard/index.md new file mode 100644 index 00000000000..c59198df081 --- /dev/null +++ b/doc/user/group/security_dashboard/index.md @@ -0,0 +1,5 @@ +--- +redirect_to: '../../application_security/security_dashboard/index.md' +--- + +This document was moved to [another location](../../application_security/security_dashboard/index.md). diff --git a/doc/user/group/subgroups/index.md b/doc/user/group/subgroups/index.md index b6f8f55978b..4e81e28a45a 100644 --- a/doc/user/group/subgroups/index.md +++ b/doc/user/group/subgroups/index.md @@ -1,8 +1,8 @@ # Subgroups NOTE: **Note:** -[Introduced][ce-2772] in GitLab 9.0. Not available when using MySQL as external -database (support removed in GitLab 9.3 [due to performance reasons][issue]). +[Introduced](https://gitlab.com/gitlab-org/gitlab-ce/issues/2772) in GitLab 9.0. Not available when using MySQL as external +database (support removed in GitLab 9.3 [due to performance reasons](https://gitlab.com/gitlab-org/gitlab-ce/issues/30472#note_27747600)). With subgroups (aka nested groups or hierarchical groups) you can have up to 20 levels of nested groups, which among other things can help you to: @@ -13,7 +13,7 @@ up to 20 levels of nested groups, which among other things can help you to: - **Organize large projects.** For large projects, subgroups makes it potentially easier to separate permissions on parts of the source code. - **Make it easier to manage people and control visibility.** Give people - different [permissions][] depending on their group [membership](#membership). + different [permissions](../../permissions.md#group-members-permissions) depending on their group [membership](#membership). ## Database Requirements @@ -80,9 +80,9 @@ structure. NOTE: **Note:** You need to be an Owner of a group in order to be able to create a subgroup. For -more information check the [permissions table][permissions]. +more information check the [permissions table](../../permissions.md#group-members-permissions). For a list of words that are not allowed to be used as group names see the -[reserved names][reserved]. +[reserved names](../../reserved_names.md). Users can always create subgroups if they are explicitly added as an Owner to a parent group even if group creation is disabled by an administrator in their settings. @@ -167,7 +167,7 @@ Here's a list of what you can't do with subgroups: - [GitLab Pages](../../project/pages/index.md) supports projects hosted under a subgroup, but not subgroup websites. That means that only the highest-level group supports - [group websites](../../project/pages/introduction.html#user-or-group-pages), + [group websites](../../project/pages/getting_started_part_one.md#gitlab-pages-domain-names), although you can have project websites under a subgroup. - It is not possible to share a project with a group that's an ancestor of the group the project is in. That means you can only share as you walk down @@ -176,6 +176,6 @@ Here's a list of what you can't do with subgroups: `group/subgroup01/subgroup03`. [ce-2772]: https://gitlab.com/gitlab-org/gitlab-ce/issues/2772 -[permissions]: ../../permissions.md#group +[permissions]: ../../permissions.md#group-members-permissions [reserved]: ../../reserved_names.md [issue]: https://gitlab.com/gitlab-org/gitlab-ce/issues/30472#note_27747600 diff --git a/doc/user/img/color_inline_colorchip_render_gfm.png b/doc/user/img/color_inline_colorchip_render_gfm.png Binary files differindex fed8ca5c34b..6f93dbeeb10 100644 --- a/doc/user/img/color_inline_colorchip_render_gfm.png +++ b/doc/user/img/color_inline_colorchip_render_gfm.png diff --git a/doc/user/img/mermaid_diagram_render_gfm.png b/doc/user/img/mermaid_diagram_render_gfm.png Binary files differdeleted file mode 100644 index 9d192a30a85..00000000000 --- a/doc/user/img/mermaid_diagram_render_gfm.png +++ /dev/null diff --git a/doc/user/img/task_list_ordered_render_gfm.png b/doc/user/img/task_list_ordered_render_gfm.png Binary files differindex 0905a8378be..98ec791e958 100644 --- a/doc/user/img/task_list_ordered_render_gfm.png +++ b/doc/user/img/task_list_ordered_render_gfm.png diff --git a/doc/user/img/unordered_check_list_render_gfm.png b/doc/user/img/unordered_check_list_render_gfm.png Binary files differindex ccdeab6e62c..2ce0fb95645 100644 --- a/doc/user/img/unordered_check_list_render_gfm.png +++ b/doc/user/img/unordered_check_list_render_gfm.png diff --git a/doc/user/index.md b/doc/user/index.md index 71378920ed4..1fc4e4c43cf 100644 --- a/doc/user/index.md +++ b/doc/user/index.md @@ -8,13 +8,13 @@ Welcome to GitLab! We're glad to have you here! As a GitLab user you'll have access to all the features your [subscription](https://about.gitlab.com/pricing/) -includes, except [GitLab administrator](../README.md#administrator-documentation) +includes, except [GitLab administrator](../administration/index.md) settings, unless you have admin privileges to install, configure, and upgrade your GitLab instance. Admin privileges for [GitLab.com](https://gitlab.com/) are restricted to the GitLab team. -For more information on configuring GitLab self-managed instances, see [Administrator documentation](../README.md#administrator-documentation). +For more information on configuring GitLab self-managed instances, see [Administrator documentation](../administration/index.md). ## Overview @@ -36,38 +36,37 @@ To get familiar with the concepts needed to develop code on GitLab, read the fol GitLab is a Git-based platform that integrates a great number of essential tools for software development and deployment, and project management: -- Hosting code in repositories with version control +- Hosting code in repositories with version control. - Tracking proposals for new implementations, bug reports, and feedback with a - fully featured [Issue Tracker](project/issues/index.md#issue-tracker) -- Organizing and prioritizing with [Issue Boards](project/issues/index.md#issue-boards) + fully featured [Issue Tracker](project/issues/index.md#issues-list). +- Organizing and prioritizing with [Issue Boards](project/issues/index.md#issue-boards). - Reviewing code in [Merge Requests](project/merge_requests/index.md) with live-preview changes per - branch with [Review Apps](../ci/review_apps/index.md) -- Building, testing and deploying with built-in [Continuous Integration](../ci/README.md) -- Deploying personal and professional static websites with [GitLab Pages](project/pages/index.md) -- Integrating with Docker by using [GitLab Container Registry](project/container_registry.md) -- Tracking the development lifecycle by usingn [GitLab Cycle Analytics](project/cycle_analytics.md) + branch with [Review Apps](../ci/review_apps/index.md). +- Building, testing, and deploying with built-in [Continuous Integration](../ci/README.md). +- Deploying personal and professional static websites with [GitLab Pages](project/pages/index.md). +- Integrating with Docker by using [GitLab Container Registry](project/container_registry.md). +- Tracking the development lifecycle by using [GitLab Cycle Analytics](project/cycle_analytics.md). With GitLab Enterprise Edition, you can also: -- Provide support with [Service Desk](https://docs.gitlab.com/ee/user/project/service_desk.html) +- Provide support with [Service Desk](project/service_desk.md). - Improve collaboration with - [Merge Request Approvals](https://docs.gitlab.com/ee/user/project/merge_requests/index.html#merge-request-approvals), - [Multiple Assignees for Issues](https://docs.gitlab.com/ee/user/project/issues/multiple_assignees_for_issues.html), - and [Multiple Issue Boards](https://docs.gitlab.com/ee/user/project/issue_board.html#multiple-issue-boards) -- Create formal relationships between issues with [Related Issues](https://docs.gitlab.com/ee/user/project/issues/related_issues.html) -- Use [Burndown Charts](https://docs.gitlab.com/ee/user/project/milestones/burndown_charts.html) to track progress during a sprint or while working on a new version of their software. -- Leverage [Elasticsearch](https://docs.gitlab.com/ee/integration/elasticsearch.html) with [Advanced Global Search](https://docs.gitlab.com/ee/user/search/advanced_global_search.html) and [Advanced Syntax Search](https://docs.gitlab.com/ee/user/search/advanced_search_syntax.html) for faster, more advanced code search across your entire GitLab instance -- [Authenticate users with Kerberos](https://docs.gitlab.com/ee/integration/kerberos.html) -- [Mirror a repository](https://docs.gitlab.com/ee/workflow/repository_mirroring.html) from elsewhere on your local server. -- [Export issues as CSV](https://docs.gitlab.com/ee/user/project/issues/csv_export.html) -- View your entire CI/CD pipeline involving more than one project with [Multiple-Project Pipeline Graphs](https://docs.gitlab.com/ee/ci/multi_project_pipeline_graphs.html) -- [Lock files](https://docs.gitlab.com/ee/user/project/file_lock.html) to prevent conflicts -- View the current health and status of each CI environment running on Kubernetes with [Deploy Boards](https://docs.gitlab.com/ee/user/project/deploy_boards.html) -- Leverage continuous delivery method with [Canary Deployments](https://docs.gitlab.com/ee/user/project/canary_deployments.html) - -You can also [integrate](project/integrations/project_services.md) GitLab with -numerous third-party applications, such as Mattermost, Microsoft Teams, Trello, -Slack, Bamboo CI, JIRA, and a lot more. + [Merge Request Approvals](project/merge_requests/index.md#merge-request-approvals-starter), + [Multiple Assignees for Issues](project/issues/multiple_assignees_for_issues.md), + and [Multiple Issue Boards](project/issue_board.md#multiple-issue-boards-starter). +- Create formal relationships between issues with [Related Issues](project/issues/related_issues.md). +- Use [Burndown Charts](project/milestones/burndown_charts.md) to track progress during a sprint or while working on a new version of their software. +- Leverage [Elasticsearch](../integration/elasticsearch.md) with [Advanced Global Search](search/advanced_global_search.md) and [Advanced Syntax Search](search/advanced_search_syntax.md) for faster, more advanced code search across your entire GitLab instance. +- [Authenticate users with Kerberos](../integration/kerberos.md). +- [Mirror a repository](../workflow/repository_mirroring.md) from elsewhere on your local server. +- [Export issues as CSV](project/issues/csv_export.md). +- View your entire CI/CD pipeline involving more than one project with [Multiple-Project Pipelines](../ci/multi_project_pipeline_graphs.md). +- [Lock files](project/file_lock.md) to prevent conflicts. +- View the current health and status of each CI environment running on Kubernetes with [Deploy Boards](project/deploy_boards.md). +- Leverage continuous delivery method with [Canary Deployments](project/canary_deployments.md). +- Scan your code for vulnerabilities and [display them in merge requests](application_security/sast/index.md). + +You can also [integrate](project/integrations/project_services.md) GitLab with numerous third-party applications, such as Mattermost, Microsoft Teams, HipChat, Trello, Slack, Bamboo CI, JIRA, and a lot more. ## Projects @@ -108,8 +107,8 @@ to enjoy the best of GitLab. - [Permissions](permissions.md): Learn the different set of permissions levels for each user type (guest, reporter, developer, maintainer, owner). - [Feature highlight](feature_highlight.md): Learn more about the little blue dots - around the app that explain certain features -- [Abuse reports](abuse_reports.md): Report abuse from users to GitLab administrators + around the app that explain certain features. +- [Abuse reports](abuse_reports.md): Report abuse from users to GitLab administrators. ## Groups @@ -125,7 +124,7 @@ merge requests, code snippets, and commits. When performing inline reviews to implementations to your codebase through merge requests you can -gather feedback through [resolvable discussions](discussions/index.md#resolvable-discussions). +gather feedback through [resolvable discussions](discussions/index.md#resolvable-comments-and-discussions). ### GitLab Flavored Markdown (GFM) @@ -173,3 +172,8 @@ Learn what is [Git](../topics/git/index.md) and its best practices. ## Instance statistics See [various statistics](instance_statistics/index.md) of your GitLab instance. + +## Operations Dashboard **[PREMIUM]** + +See [Operations Dashboard](operations_dashboard/index.md) for a summary of each +project's operational health. diff --git a/doc/user/instance/clusters/index.md b/doc/user/instance/clusters/index.md new file mode 100644 index 00000000000..894f83d3c75 --- /dev/null +++ b/doc/user/instance/clusters/index.md @@ -0,0 +1,23 @@ +# Instance-level Kubernetes clusters + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/issues/39840) in GitLab 11.11. +> Instance-level cluster integration is currently in [Beta](https://about.gitlab.com/handbook/product/#alpha-beta-ga). + +## Overview + +Similar to [project-level](../../project/clusters/index.md) +and [group-level](../../group/clusters/index.md) Kubernetes clusters, +instance-level Kubernetes clusters allow you to connect a Kubernetes cluster to +the GitLab instance, which enables you to use the same cluster across multiple +projects. + +## Cluster precedence + +GitLab will try match to clusters in the following order: + +- Project-level clusters +- Group-level clusters +- Instance level + +To be selected, the cluster must be enabled and +match the [environment selector](../../../ci/environments.md#scoping-environments-with-specs-premium). diff --git a/doc/user/instance_statistics/convdev.md b/doc/user/instance_statistics/convdev.md index 52b99b69a02..2c9e0ecbf65 100644 --- a/doc/user/instance_statistics/convdev.md +++ b/doc/user/instance_statistics/convdev.md @@ -1,27 +1,26 @@ # Conversational Development Index -> [Introduced][ce-30469] in GitLab 9.3. +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/issues/30469) in GitLab 9.3. + +NOTE: **Note:** +Your GitLab instance's [usage ping](../admin_area/settings/usage_statistics.md#usage-ping-core-only) must be activated in order to use this feature. The Conversational Development Index (ConvDev Index) gives you an overview of your entire instance's adoption of [Concurrent DevOps](https://about.gitlab.com/concurrent-devops/) -from planning to monitoring. It displays the usage of these GitLab features over +from planning to monitoring. + +This displays the usage of these GitLab features over the last 30 days, averaged over the number of active users in that time period. It also provides a Lead score per feature, which is calculated based on GitLab's analysis -of top-performing instances based on [usage ping data][ping] that GitLab has -collected. Your score is compared to the lead score, expressed as a percentage. -Your overall index score is an average of all your feature score percentages. +of top-performing instances based on [usage ping data](../admin_area/settings/usage_statistics.md#usage-ping-core-only) that GitLab has +collected. Your score is compared to the lead score of each feature and then expressed as a percentage at the bottom of said feature. +Your overall index score is an average of all your feature score percentages - this percentage value is presented above all the of features on the page.  The page also provides helpful links to articles and GitLab docs, to help you improve your scores. -Your GitLab instance's [usage ping][ping] must be activated in order to use this feature. Usage ping data is aggregated on GitLab's servers for analysis. Your usage -information is **not sent** to any other GitLab instances. - -If you have just started using GitLab, it may take a few weeks for data to be +information is **not sent** to any other GitLab instances. If you have just started using GitLab, it may take a few weeks for data to be collected before this feature is available. - -[ce-30469]: https://gitlab.com/gitlab-org/gitlab-ce/issues/30469 -[ping]: ../admin_area/settings/usage_statistics.md#usage-ping diff --git a/doc/user/markdown.md b/doc/user/markdown.md index a7a87773eec..6b6e5ab7634 100644 --- a/doc/user/markdown.md +++ b/doc/user/markdown.md @@ -20,6 +20,7 @@ You can use GFM in the following areas: - Snippets (the snippet must be named with a `.md` extension) - Wiki pages - Markdown documents inside repositories +- Epics **[ULTIMATE]** You can also use other rich text files in GitLab. You might have to install a dependency to do so. Please see the [`github-markup` gem readme](https://github.com/gitlabhq/markup#markups) for more information. @@ -267,10 +268,7 @@ However the wrapping tags cannot be mixed as such: ### Emoji -> If this is not rendered correctly, see -https://gitlab.com/gitlab-org/gitlab-ce/blob/master/doc/user/markdown.md#emoji - -``` +```md Sometimes you want to :monkey: around a bit and add some :star2: to your :speech_balloon:. Well we have a gift for you: :zap: You can use emoji anywhere GFM is supported. :v: @@ -288,15 +286,15 @@ On Linux, you can download [Noto Color Emoji](https://www.google.com/get/noto/he Ubuntu 18.04 (like many modern Linux distros) has this font installed by default. ``` -Sometimes you want to <img src="https://gitlab.com/gitlab-org/gitlab-ce/raw/master/app/assets/images/emoji/monkey.png" width="20px" height="20px"> around a bit and add some <img src="https://gitlab.com/gitlab-org/gitlab-ce/raw/master/app/assets/images/emoji/star2.png" width="20px" height="20px"> to your <img src="https://gitlab.com/gitlab-org/gitlab-ce/raw/master/app/assets/images/emoji/speech_balloon.png" width="20px" height="20px">. Well we have a gift for you: +Sometimes you want to <img src="https://gitlab.com/gitlab-org/gitlab-ce/raw/master/app/assets/images/emoji/monkey.png" width="20px" height="20px" style="display:inline;margin:0"> around a bit and add some <img src="https://gitlab.com/gitlab-org/gitlab-ce/raw/master/app/assets/images/emoji/star2.png" width="20px" height="20px" style="display:inline;margin:0"> to your <img src="https://gitlab.com/gitlab-org/gitlab-ce/raw/master/app/assets/images/emoji/speech_balloon.png" width="20px" height="20px" style="display:inline;margin:0">. Well we have a gift for you: -<img src="https://gitlab.com/gitlab-org/gitlab-ce/raw/master/app/assets/images/emoji/zap.png" width="20px" height="20px">You can use emoji anywhere GFM is supported. <img src="https://gitlab.com/gitlab-org/gitlab-ce/raw/master/app/assets/images/emoji/v.png" width="20px" height="20px"> +<img src="https://gitlab.com/gitlab-org/gitlab-ce/raw/master/app/assets/images/emoji/zap.png" width="20px" height="20px" style="display:inline;margin:0">You can use emoji anywhere GFM is supported. <img src="https://gitlab.com/gitlab-org/gitlab-ce/raw/master/app/assets/images/emoji/v.png" width="20px" height="20px" style="display:inline;margin:0"> -You can use it to point out a <img src="https://gitlab.com/gitlab-org/gitlab-ce/raw/master/app/assets/images/emoji/bug.png" width="20px" height="20px"> or warn about <img src="https://gitlab.com/gitlab-org/gitlab-ce/raw/master/app/assets/images/emoji/speak_no_evil.png" width="20px" height="20px"> patches. And if someone improves your really <img src="https://gitlab.com/gitlab-org/gitlab-ce/raw/master/app/assets/images/emoji/snail.png" width="20px" height="20px"> code, send them some <img src="https://gitlab.com/gitlab-org/gitlab-ce/raw/master/app/assets/images/emoji/birthday.png" width="20px" height="20px">. People will <img src="https://gitlab.com/gitlab-org/gitlab-ce/raw/master/app/assets/images/emoji/heart.png" width="20px" height="20px"> you for that. +You can use it to point out a <img src="https://gitlab.com/gitlab-org/gitlab-ce/raw/master/app/assets/images/emoji/bug.png" width="20px" height="20px" style="display:inline;margin:0"> or warn about <img src="https://gitlab.com/gitlab-org/gitlab-ce/raw/master/app/assets/images/emoji/speak_no_evil.png" width="20px" height="20px" style="display:inline;margin:0"> patches. And if someone improves your really <img src="https://gitlab.com/gitlab-org/gitlab-ce/raw/master/app/assets/images/emoji/snail.png" width="20px" height="20px" style="display:inline;margin:0"> code, send them some <img src="https://gitlab.com/gitlab-org/gitlab-ce/raw/master/app/assets/images/emoji/birthday.png" width="20px" height="20px" style="display:inline;margin:0">. People will <img src="https://gitlab.com/gitlab-org/gitlab-ce/raw/master/app/assets/images/emoji/heart.png" width="20px" height="20px" style="display:inline;margin:0"> you for that. -If you are new to this, don't be <img src="https://gitlab.com/gitlab-org/gitlab-ce/raw/master/app/assets/images/emoji/fearful.png" width="20px" height="20px">. You can easily join the emoji <img src="https://gitlab.com/gitlab-org/gitlab-ce/raw/master/app/assets/images/emoji/family.png" width="20px" height="20px">. All you need to do is to look up one of the supported codes. +If you are new to this, don't be <img src="https://gitlab.com/gitlab-org/gitlab-ce/raw/master/app/assets/images/emoji/fearful.png" width="20px" height="20px" style="display:inline;margin:0">. You can easily join the emoji <img src="https://gitlab.com/gitlab-org/gitlab-ce/raw/master/app/assets/images/emoji/family.png" width="20px" height="20px" style="display:inline;margin:0">. All you need to do is to look up one of the supported codes. -Consult the [Emoji Cheat Sheet](https://www.webfx.com/tools/emoji-cheat-sheet/) for a list of all supported emoji codes. <img src="https://gitlab.com/gitlab-org/gitlab-ce/raw/master/app/assets/images/emoji/thumbsup.png" width="20px" height="20px"> +Consult the [Emoji Cheat Sheet](https://www.webfx.com/tools/emoji-cheat-sheet/) for a list of all supported emoji codes. <img src="https://gitlab.com/gitlab-org/gitlab-ce/raw/master/app/assets/images/emoji/thumbsup.png" width="20px" height="20px" style="display:inline;margin:0"> Most emoji are natively supported on macOS, Windows, iOS, Android and will fallback to image-based emoji where there is lack of support. @@ -323,6 +321,7 @@ GFM will recognize the following: | `#12345` | issue | | `!123` | merge request | | `$123` | snippet | +| `&123` | epic **[ULTIMATE]** | | `~123` | label by ID | | `~bug` | one-word label by name | | `~"feature request"` | multi-word label by name | @@ -343,6 +342,7 @@ GFM also recognizes certain cross-project references: | `namespace/project%123` | project milestone | | `namespace/project$123` | snippet | | `namespace/project@9ba12248` | specific commit | +| `group1/subgroup&123` | epic **[ULTIMATE]** | | `namespace/project@9ba12248...b19a04f5` | commit range comparison | | `namespace/project~"Some label"` | issues with given label | @@ -480,7 +480,7 @@ GitLab 10.3. > If this is not rendered correctly, see https://gitlab.com/gitlab-org/gitlab-ce/blob/master/doc/user/markdown.md#mermaid -It is possible to generate diagrams and flowcharts from text using [Mermaid][mermaid]. +It is possible to generate diagrams and flowcharts from text using [Mermaid](https://mermaidjs.github.io/). In order to generate a diagram or flowchart, you should write your text inside the `mermaid` block. @@ -496,9 +496,15 @@ Example: Becomes: -<img src="./img/mermaid_diagram_render_gfm.png" width="200px" height="400px"> +```mermaid +graph TD; + A-->B; + A-->C; + B-->D; + C-->D; +``` -For details see the [Mermaid official page][mermaid]. +For details see the [Mermaid official page](https://mermaidjs.github.io/). ### Front matter @@ -581,11 +587,11 @@ Alt-H2 ------ ``` -### Header IDs and links +#### Header IDs and links -All Markdown-rendered headers automatically get IDs, except in comments. +All Markdown-rendered headers automatically get IDs, which can be linked to, except in comments. -On hover, a link to those IDs becomes visible to make it easier to copy the link to the header to give it to someone else. +On hover, a link to those IDs becomes visible to make it easier to copy the link to the header to use it somewhere else. The IDs are generated from the content of the header according to the following rules: @@ -612,8 +618,8 @@ Would generate the following link IDs: 1. `this-header-has-spaces-in-it` 1. `this-header-has-a-in-it` 1. `this-header-has-unicode-in-it-한글` -1. `this-header-has-spaces-in-it` 1. `this-header-has-spaces-in-it-1` +1. `this-header-has-spaces-in-it-2` 1. `this-header-has-3-5-in-it-and-parentheses` Note that the Emoji processing happens before the header IDs are generated, so the Emoji is converted to an image which then gets removed from the ID. @@ -718,25 +724,25 @@ Becomes: There are two ways to create links, inline-style and reference-style. - [I'm an inline-style link](https://www.google.com) - - [I'm a reference-style link][Arbitrary case-insensitive reference text] - - [I'm a relative reference to a repository file](LICENSE) - - [I am an absolute reference within the repository](/doc/user/markdown.md) - - [I link to the Milestones page](/../milestones) +```markdown +[I'm an inline-style link](https://www.google.com) +[I'm a link to a repository file in the same directory](index.md) +[I am an absolute reference within the repository](/doc/user/index.md) +[I'm a relative link to the Milestones page](../README.md) - [You can use numbers for reference-style link definitions][1] +[I link to a section on a different markdown page, using a header ID](index.md#overview) +[I link to a different section on the same page, using the header ID](#header-ids-and-links) - Or leave it empty and use the [link text itself][] +[I'm a reference-style link][Arbitrary case-insensitive reference text] +[You can use numbers for reference-style link definitions][1] +Or leave it empty and use the [link text itself][] - Some text to show that the reference links can follow later. +Some text to show that the reference links can follow later. - [arbitrary case-insensitive reference text]: https://www.mozilla.org - [1]: http://slashdot.org - [link text itself]: https://www.reddit.com +[arbitrary case-insensitive reference text]: https://www.mozilla.org +[1]: http://slashdot.org +[link text itself]: https://www.reddit.com +``` >**Note:** Relative links do not allow referencing project files in a wiki page or wiki @@ -1021,7 +1027,7 @@ A link can be constructed relative to the current wiki page using `./<page>`, it would link to `<your_wiki>/documentation/related`: ```markdown - [Link to Related Page](./related) + [Link to Related Page](related) ``` - If this snippet was placed on a page at `<your_wiki>/documentation/related/content`, @@ -1035,7 +1041,7 @@ A link can be constructed relative to the current wiki page using `./<page>`, it would link to `<your_wiki>/documentation/related.md`: ```markdown - [Link to Related Page](./related.md) + [Link to Related Page](related.md) ``` - If this snippet was placed on a page at `<your_wiki>/documentation/related/content`, @@ -1072,7 +1078,6 @@ A link starting with a `/` is relative to the wiki root. [^2]: This is my awesome footnote. [markdown.md]: https://gitlab.com/gitlab-org/gitlab-ce/blob/master/doc/user/markdown.md -[mermaid]: https://mermaidjs.github.io/ "Mermaid website" [rouge]: http://rouge.jneen.net/ "Rouge website" [redcarpet]: https://github.com/vmg/redcarpet "Redcarpet website" [katex]: https://github.com/Khan/KaTeX "KaTeX website" diff --git a/doc/user/operations_dashboard/img/index_operations_dashboard_top_bar_icon.png b/doc/user/operations_dashboard/img/index_operations_dashboard_top_bar_icon.png Binary files differnew file mode 100644 index 00000000000..9d6a509ea72 --- /dev/null +++ b/doc/user/operations_dashboard/img/index_operations_dashboard_top_bar_icon.png diff --git a/doc/user/operations_dashboard/img/index_operations_dashboard_with_projects.png b/doc/user/operations_dashboard/img/index_operations_dashboard_with_projects.png Binary files differnew file mode 100644 index 00000000000..ee33eee8134 --- /dev/null +++ b/doc/user/operations_dashboard/img/index_operations_dashboard_with_projects.png diff --git a/doc/user/operations_dashboard/index.md b/doc/user/operations_dashboard/index.md new file mode 100644 index 00000000000..66362f27299 --- /dev/null +++ b/doc/user/operations_dashboard/index.md @@ -0,0 +1,37 @@ +# Operations Dashboard **[PREMIUM]** + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/issues/5781) +in [GitLab Ultimate](https://about.gitlab.com/pricing/) 11.5. +[Moved](https://gitlab.com/gitlab-org/gitlab-ee/issues/9218) to +[GitLab Premium](https://about.gitlab.com/pricing/) in 11.10. + +The Operations Dashboard provides a summary of each project's operational health, +including pipeline and alert status. + +The dashboard can be accessed via the top bar, by clicking on the new +dashboard icon: + + + +## Adding a project to the dashboard + +NOTE: **Note:** +For GitLab.com, the Operations Dashboard is available for free for public projects. +If your project is private, the group it belongs to must have a +[Gold](https://about.gitlab.com/pricing/) plan. + +To add a project to the dashboard: + +1. Click the **Add projects** button in the homescreen of the dashboard. +1. Search and add one or more projects using the **Search your projects** field. +1. Click the **Add projects** button. + +Once added, the dashboard will display the project's number of active alerts, +last commit, pipeline status, and when it was last deployed. + + + +## Making it the default dashboard when you sign in + +The Operations Dashboard can also be made the default GitLab dashboard shown when +you sign in. To make it the default, visit your [profile preferences](../profile/preferences.md). diff --git a/doc/user/permissions.md b/doc/user/permissions.md index dff77acd71b..a6e2f187090 100644 --- a/doc/user/permissions.md +++ b/doc/user/permissions.md @@ -11,10 +11,10 @@ project itself, the highest permission level is used. On public and internal projects the Guest role is not enforced. All users will be able to create issues, leave comments, and clone or download the project code. -When a member leaves the team all the assigned [Issues](project/issues/index.md) and [Merge Requests](project/merge_requests/index.md) +When a member leaves a team's project, all the assigned [Issues](project/issues/index.md) and [Merge Requests](project/merge_requests/index.md) will be unassigned automatically. -GitLab [administrators](../README.md#administrator-documentation) receive all permissions. +GitLab [administrators](../administration/index.md) receive all permissions. To add or import a user, you can follow the [project members documentation](../user/project/members/index.md). @@ -23,6 +23,12 @@ To add or import a user, you can follow the See our [product handbook on permissions](https://about.gitlab.com/handbook/product#permissions-in-gitlab) +## Instance-wide user permissions + +By default, users can create top-level groups and change their +usernames. A GitLab administrator can configure the GitLab instance to +[modify this behavior](../administration/user_settings.md). + ## Project members permissions NOTE: **Note:** @@ -30,86 +36,96 @@ In GitLab 11.0, the Master role was renamed to Maintainer. The following table depicts the various user permission levels in a project. -| Action | Guest | Reporter | Developer |Maintainer| Owner | -|---------------------------------------|---------|------------|-------------|----------|--------| -| Create new issue | ✓ [^1] | ✓ | ✓ | ✓ | ✓ | -| Create confidential issue | ✓ [^1] | ✓ | ✓ | ✓ | ✓ | -| View confidential issues | (✓) [^2] | ✓ | ✓ | ✓ | ✓ | -| Leave comments | ✓ [^1] | ✓ | ✓ | ✓ | ✓ | -| Lock issue discussions | | ✓ | ✓ | ✓ | ✓ | -| Lock merge request discussions | | | ✓ | ✓ | ✓ | -| See a list of jobs | ✓ [^3] | ✓ | ✓ | ✓ | ✓ | -| See a job log | ✓ [^3] | ✓ | ✓ | ✓ | ✓ | -| Download and browse job artifacts | ✓ [^3] | ✓ | ✓ | ✓ | ✓ | -| View wiki pages | ✓ [^1] | ✓ | ✓ | ✓ | ✓ | -| View license management reports **[ULTIMATE]** | ✓ [^1] | ✓ | ✓ | ✓ | ✓ | -| View Security reports **[ULTIMATE]** | ✓ [^1] | ✓ | ✓ | ✓ | ✓ | -| View project code | [^1] | ✓ | ✓ | ✓ | ✓ | -| Pull project code | [^1] | ✓ | ✓ | ✓ | ✓ | -| Download project | [^1] | ✓ | ✓ | ✓ | ✓ | -| Assign issues | | ✓ | ✓ | ✓ | ✓ | -| Assign merge requests | | | ✓ | ✓ | ✓ | -| Label issues | | ✓ | ✓ | ✓ | ✓ | -| Label merge requests | | | ✓ | ✓ | ✓ | -| Create code snippets | | ✓ | ✓ | ✓ | ✓ | -| Manage issue tracker | | ✓ | ✓ | ✓ | ✓ | -| Manage labels | | ✓ | ✓ | ✓ | ✓ | -| See a commit status | | ✓ | ✓ | ✓ | ✓ | -| See a container registry | | ✓ | ✓ | ✓ | ✓ | -| See environments | | ✓ | ✓ | ✓ | ✓ | -| See a list of merge requests | | ✓ | ✓ | ✓ | ✓ | -| Manage related issues **[STARTER]** | | ✓ | ✓ | ✓ | ✓ | -| Lock issue discussions | | ✓ | ✓ | ✓ | ✓ | -| Create issue from vulnerability **[ULTIMATE]** | | ✓ | ✓ | ✓ | ✓ | -| View Error Tracking list | | ✓ | ✓ | ✓ | ✓ | -| Lock merge request discussions | | | ✓ | ✓ | ✓ | -| Create new environments | | | ✓ | ✓ | ✓ | -| Stop environments | | | ✓ | ✓ | ✓ | -| Manage/Accept merge requests | | | ✓ | ✓ | ✓ | -| Create new merge request | | | ✓ | ✓ | ✓ | -| Create new branches | | | ✓ | ✓ | ✓ | -| Push to non-protected branches | | | ✓ | ✓ | ✓ | -| Force push to non-protected branches | | | ✓ | ✓ | ✓ | -| Remove non-protected branches | | | ✓ | ✓ | ✓ | -| Add tags | | | ✓ | ✓ | ✓ | -| Write a wiki | | | ✓ | ✓ | ✓ | -| Cancel and retry jobs | | | ✓ | ✓ | ✓ | -| Create or update commit status | | | ✓ | ✓ | ✓ | -| Update a container registry | | | ✓ | ✓ | ✓ | -| Remove a container registry image | | | ✓ | ✓ | ✓ | -| Create/edit/delete project milestones | | | ✓ | ✓ | ✓ | -| View approved/blacklisted licenses **[ULTIMATE]** | | | ✓ | ✓ | ✓ | -| Use security dashboard **[ULTIMATE]** | | | ✓ | ✓ | ✓ | -| Dismiss vulnerability **[ULTIMATE]** | | | ✓ | ✓ | ✓ | -| Apply code change suggestions | | | ✓ | ✓ | ✓ | -| Use environment terminals | | | | ✓ | ✓ | -| Add new team members | | | | ✓ | ✓ | -| Push to protected branches | | | | ✓ | ✓ | -| Enable/disable branch protection | | | | ✓ | ✓ | -| Turn on/off protected branch push for devs| | | | ✓ | ✓ | -| Enable/disable tag protections | | | | ✓ | ✓ | -| Rewrite/remove Git tags | | | | ✓ | ✓ | -| Edit project | | | | ✓ | ✓ | -| Add deploy keys to project | | | | ✓ | ✓ | -| Configure project hooks | | | | ✓ | ✓ | -| Manage Runners | | | | ✓ | ✓ | -| Manage job triggers | | | | ✓ | ✓ | -| Manage variables | | | | ✓ | ✓ | -| Manage GitLab Pages | | | | ✓ | ✓ | -| Manage GitLab Pages domains and certificates | | | | ✓ | ✓ | -| Remove GitLab Pages | | | | ✓ | ✓ | +| Action | Guest | Reporter | Developer |Maintainer| Owner | +|---------------------------------------------------|---------|------------|-------------|----------|--------| +| Download project | ✓ (*1*) | ✓ | ✓ | ✓ | ✓ | +| Leave comments | ✓ (*1*) | ✓ | ✓ | ✓ | ✓ | +| View Insights charts **[ULTIMATE]** | ✓ | ✓ | ✓ | ✓ | ✓ | +| View approved/blacklisted licenses **[ULTIMATE]** | ✓ | ✓ | ✓ | ✓ | ✓ | +| View license management reports **[ULTIMATE]** | ✓ (*1*) | ✓ | ✓ | ✓ | ✓ | +| View Security reports **[ULTIMATE]** | ✓ (*1*) | ✓ | ✓ | ✓ | ✓ | +| View project code | ✓ (*1*) | ✓ | ✓ | ✓ | ✓ | +| Pull project code | ✓ (*1*) | ✓ | ✓ | ✓ | ✓ | | View GitLab Pages protected by [access control](project/pages/introduction.md#gitlab-pages-access-control-core-only) | ✓ | ✓ | ✓ | ✓ | ✓ | -| Manage clusters | | | | ✓ | ✓ | -| Manage license policy **[ULTIMATE]** | | | | ✓ | ✓ | -| Edit comments (posted by any user) | | | | ✓ | ✓ | -| Manage Error Tracking | | | | ✓ | ✓ | -| Switch visibility level | | | | | ✓ | -| Transfer project to another namespace | | | | | ✓ | -| Remove project | | | | | ✓ | -| Delete issues | | | | | ✓ | -| Force push to protected branches [^4] | | | | | | -| Remove protected branches [^4] | | | | | | -| View project Audit Events | | | | ✓ | ✓ | +| View wiki pages | ✓ (*1*) | ✓ | ✓ | ✓ | ✓ | +| See a list of jobs | ✓ (*3*) | ✓ | ✓ | ✓ | ✓ | +| See a job log | ✓ (*3*) | ✓ | ✓ | ✓ | ✓ | +| Download and browse job artifacts | ✓ (*3*) | ✓ | ✓ | ✓ | ✓ | +| Create new issue | ✓ (*1*) | ✓ | ✓ | ✓ | ✓ | +| See related issues | ✓ | ✓ | ✓ | ✓ | ✓ | +| Create confidential issue | ✓ (*1*) | ✓ | ✓ | ✓ | ✓ | +| View confidential issues | (*2*) | ✓ | ✓ | ✓ | ✓ | +| Assign issues | | ✓ | ✓ | ✓ | ✓ | +| Label issues | | ✓ | ✓ | ✓ | ✓ | +| Lock issue discussions | | ✓ | ✓ | ✓ | ✓ | +| Manage issue tracker | | ✓ | ✓ | ✓ | ✓ | +| Manage related issues **[STARTER]** | | ✓ | ✓ | ✓ | ✓ | +| Create issue from vulnerability **[ULTIMATE]** | | ✓ | ✓ | ✓ | ✓ | +| Manage labels | | ✓ | ✓ | ✓ | ✓ | +| Create code snippets | | ✓ | ✓ | ✓ | ✓ | +| See a commit status | | ✓ | ✓ | ✓ | ✓ | +| See a container registry | | ✓ | ✓ | ✓ | ✓ | +| See environments | | ✓ | ✓ | ✓ | ✓ | +| See a list of merge requests | | ✓ | ✓ | ✓ | ✓ | +| View project statistics | | ✓ | ✓ | ✓ | ✓ | +| View Error Tracking list | | ✓ | ✓ | ✓ | ✓ | +| Pull from [Maven repository](project/packages/maven_repository.md) or [NPM registry](project/packages/npm_registry.md) **[PREMIUM]** | | ✓ | ✓ | ✓ | ✓ | +| Publish to [Maven repository](project/packages/maven_repository.md) or [NPM registry](project/packages/npm_registry.md) **[PREMIUM]** | | | ✓ | ✓ | ✓ || +| Create new branches | | | ✓ | ✓ | ✓ | +| Push to non-protected branches | | | ✓ | ✓ | ✓ | +| Force push to non-protected branches | | | ✓ | ✓ | ✓ | +| Remove non-protected branches | | | ✓ | ✓ | ✓ | +| Create new merge request | | | ✓ | ✓ | ✓ | +| Assign merge requests | | | ✓ | ✓ | ✓ | +| Label merge requests | | | ✓ | ✓ | ✓ | +| Lock merge request discussions | | | ✓ | ✓ | ✓ | +| Manage/Accept merge requests | | | ✓ | ✓ | ✓ | +| Create new environments | | | ✓ | ✓ | ✓ | +| Stop environments | | | ✓ | ✓ | ✓ | +| Add tags | | | ✓ | ✓ | ✓ | +| Cancel and retry jobs | | | ✓ | ✓ | ✓ | +| Create or update commit status | | | ✓ | ✓ | ✓ | +| Update a container registry | | | ✓ | ✓ | ✓ | +| Remove a container registry image | | | ✓ | ✓ | ✓ | +| Create/edit/delete project milestones | | | ✓ | ✓ | ✓ | +| Use security dashboard **[ULTIMATE]** | | | ✓ | ✓ | ✓ | +| Dismiss vulnerability **[ULTIMATE]** | | | ✓ | ✓ | ✓ | +| Apply code change suggestions | | | ✓ | ✓ | ✓ | +| Create and edit wiki pages | | | ✓ | ✓ | ✓ | +| Use environment terminals | | | | ✓ | ✓ | +| Run Web IDE's Interactive Web Terminals **[ULTIMATE ONLY]** | | | | ✓ | ✓ | +| Add new team members | | | | ✓ | ✓ | +| Enable/disable branch protection | | | | ✓ | ✓ | +| Push to protected branches | | | | ✓ | ✓ | +| Turn on/off protected branch push for devs | | | | ✓ | ✓ | +| Enable/disable tag protections | | | | ✓ | ✓ | +| Rewrite/remove Git tags | | | | ✓ | ✓ | +| Edit project | | | | ✓ | ✓ | +| Add deploy keys to project | | | | ✓ | ✓ | +| Configure project hooks | | | | ✓ | ✓ | +| Manage Runners | | | | ✓ | ✓ | +| Manage job triggers | | | | ✓ | ✓ | +| Manage variables | | | | ✓ | ✓ | +| Manage GitLab Pages | | | | ✓ | ✓ | +| Manage GitLab Pages domains and certificates | | | | ✓ | ✓ | +| Remove GitLab Pages | | | | ✓ | ✓ | +| Manage clusters | | | | ✓ | ✓ | +| Manage license policy **[ULTIMATE]** | | | | ✓ | ✓ | +| Edit comments (posted by any user) | | | | ✓ | ✓ | +| Manage Error Tracking | | | | ✓ | ✓ | +| Delete wiki pages | | | | ✓ | ✓ | +| View project Audit Events | | | | ✓ | ✓ | +| Switch visibility level | | | | | ✓ | +| Transfer project to another namespace | | | | | ✓ | +| Remove project | | | | | ✓ | +| Delete issues | | | | | ✓ | +| Force push to protected branches [^4] | | | | | | +| Remove protected branches [^4] | | | | | | + +- (*1*): All users are able to perform this action on public and internal projects, but not private projects. +- (*2*): Guest users can only view the confidential issues they created themselves +- (*3*): If **Public pipelines** is enabled in **Project Settings > CI/CD** +- (*4*): Not allowed for Guest, Reporter, Developer, Maintainer, or Owner ## Project features permissions @@ -152,7 +168,7 @@ to learn more. The user that locks a file or directory is the only one that can edit and push their changes back to the repository where the locked objects are located. -Read through the documentation on [permissions for File Locking](https://docs.gitlab.com/ee/user/project/file_lock.html#permissions-on-file-locking) to learn more. +Read through the documentation on [permissions for File Locking](project/file_lock.md#permissions-on-file-locking) to learn more. ### Confidential Issues permissions @@ -162,8 +178,12 @@ read through the documentation on [permissions and access to confidential issues ### Releases permissions -[Project Releases](project/releases/index.md) can be read by all project -members (Reporters, Developers, Maintainers, Owners) **except Guests**. +[Project Releases](project/releases/index.md) can be read by project +members with Reporter, Developer, Maintainer, and Owner permissions. +Guest users can access Release pages for downloading assets but +are not allowed to download the source code nor see repository +information such as tags and commits. + Releases can be created, updated, or deleted via [Releases APIs](../api/releases/index.md) by project Developers, Maintainers, and Owners. @@ -176,20 +196,21 @@ Any user can remove themselves from a group, unless they are the last Owner of the group. The following table depicts the various user permission levels in a group. -| Action | Guest | Reporter | Developer | Maintainer | Owner | -|-------------------------|-------|----------|-----------|--------|-------| -| Browse group | ✓ | ✓ | ✓ | ✓ | ✓ | -| Edit group | | | | | ✓ | -| Create subgroup | | | | | ✓ | -| Create project in group | | | ✓ | ✓ | ✓ | -| Manage group members | | | | | ✓ | -| Remove group | | | | | ✓ | -| Manage group labels | | ✓ | ✓ | ✓ | ✓ | -| Create/edit/delete group milestones | | | ✓ | ✓ | ✓ | -| View group epic **[ULTIMATE]** | ✓ | ✓ | ✓ | ✓ | ✓ | -| Create/edit group epic **[ULTIMATE]** | | ✓ | ✓ | ✓ | ✓ | -| Delete group epic **[ULTIMATE]** | | | | | ✓ | -| View group Audit Events | | | | | ✓ | +| Action | Guest | Reporter | Developer | Maintainer | Owner | +|---------------------------------------|-------|----------|-----------|------------|-------| +| Browse group | ✓ | ✓ | ✓ | ✓ | ✓ | +| View Insights charts **[ULTIMATE]** | ✓ | ✓ | ✓ | ✓ | ✓ | +| View group epic **[ULTIMATE]** | ✓ | ✓ | ✓ | ✓ | ✓ | +| Create/edit group epic **[ULTIMATE]** | | ✓ | ✓ | ✓ | ✓ | +| Manage group labels | | ✓ | ✓ | ✓ | ✓ | +| Create project in group | | | ✓ | ✓ | ✓ | +| Create/edit/delete group milestones | | | ✓ | ✓ | ✓ | +| Edit group | | | | | ✓ | +| Create subgroup | | | | | ✓ | +| Manage group members | | | | | ✓ | +| Remove group | | | | | ✓ | +| Delete group epic **[ULTIMATE]** | | | | | ✓ | +| View group Audit Events | | | | | ✓ | ### Subgroup permissions @@ -234,22 +255,22 @@ The regex pattern format is Ruby, but it needs to be convertible to JavaScript, Here are some examples: -- Use `\.internal@domain\.com` to mark email addresses containing ".internal@domain.com" internal. +- Use `\.internal@domain\.com$` to mark email addresses ending with ".internal@domain.com" internal. - Use `^(?:(?!\.ext@domain\.com).)*$\r?` to mark users with email addresses NOT including .ext@domain.com internal. Please be aware that this regex could lead to a DOS attack, [see](https://en.wikipedia.org/wiki/ReDoS?) ReDos on Wikipedia. ## Auditor users **[PREMIUM ONLY]** ->[Introduced][ee-998] in [GitLab Premium][eep] 8.17. +>[Introduced](https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/998) in [GitLab Premium](https://about.gitlab.com/pricing/) 8.17. Auditor users are given read-only access to all projects, groups, and other resources on the GitLab instance. An Auditor user should be able to access all projects and groups of a GitLab instance -with the permissions described on the documentation on [auditor users permissions](https://docs.gitlab.com/ee/administration/auditor_users.html#permissions-and-restrictions-of-an-auditor-user). +with the permissions described on the documentation on [auditor users permissions](../administration/auditor_users.md#permissions-and-restrictions-of-an-auditor-user). -[Read more about Auditor users.](https://docs.gitlab.com/ee/administration/auditor_users.html) +[Read more about Auditor users.](../administration/auditor_users.md) ## Project features @@ -282,7 +303,7 @@ instance and project. In addition, all admins can use the admin interface under |---------------------------------------|-----------------|-------------|----------|--------| | See commits and jobs | ✓ | ✓ | ✓ | ✓ | | Retry or cancel job | | ✓ | ✓ | ✓ | -| Erase job artifacts and trace | | ✓ [^5] | ✓ | ✓ | +| Erase job artifacts and trace | | ✓ (*1*) | ✓ | ✓ | | Remove project | | | ✓ | ✓ | | Create project | | | ✓ | ✓ | | Change project configuration | | | ✓ | ✓ | @@ -291,6 +312,8 @@ instance and project. In addition, all admins can use the admin interface under | See events in the system | | | | ✓ | | Admin interface | | | | ✓ | +- *1*: Only if the job was triggered by the user + ### Job permissions NOTE: **Note:** @@ -298,25 +321,28 @@ In GitLab 11.0, the Master role was renamed to Maintainer. >**Note:** GitLab 8.12 has a completely redesigned job permissions system. -Read all about the [new model and its implications][new-mod]. +Read all about the [new model and its implications](project/new_ci_build_permissions_model.md). This table shows granted privileges for jobs triggered by specific types of users: -| Action | Guest, Reporter | Developer |Maintainer| Admin | -|---------------------------------------------|-----------------|-------------|----------|--------| -| Run CI job | | ✓ | ✓ | ✓ | -| Clone source and LFS from current project | | ✓ | ✓ | ✓ | -| Clone source and LFS from public projects | | ✓ | ✓ | ✓ | -| Clone source and LFS from internal projects | | ✓ [^6] | ✓ [^6] | ✓ | -| Clone source and LFS from private projects | | ✓ [^7] | ✓ [^7] | ✓ [^7] | -| Push source and LFS | | | | | -| Pull container images from current project | | ✓ | ✓ | ✓ | -| Pull container images from public projects | | ✓ | ✓ | ✓ | -| Pull container images from internal projects| | ✓ [^6] | ✓ [^6] | ✓ | -| Pull container images from private projects | | ✓ [^7] | ✓ [^7] | ✓ [^7] | -| Push container images to current project | | ✓ | ✓ | ✓ | -| Push container images to other projects | | | | | +| Action | Guest, Reporter | Developer |Maintainer| Admin | +|---------------------------------------------|-----------------|-------------|----------|---------| +| Run CI job | | ✓ | ✓ | ✓ | +| Clone source and LFS from current project | | ✓ | ✓ | ✓ | +| Clone source and LFS from public projects | | ✓ | ✓ | ✓ | +| Clone source and LFS from internal projects | | ✓ (*1*) | ✓ (*1*) | ✓ | +| Clone source and LFS from private projects | | ✓ (*2*) | ✓ (*2*) | ✓ (*2*) | +| Pull container images from current project | | ✓ | ✓ | ✓ | +| Pull container images from public projects | | ✓ | ✓ | ✓ | +| Pull container images from internal projects| | ✓ (*1*) | ✓ (*1*) | ✓ | +| Pull container images from private projects | | ✓ (*2*) | ✓ (*2*) | ✓ (*2*) | +| Push container images to current project | | ✓ | ✓ | ✓ | +| Push container images to other projects | | | | | +| Push source and LFS | | | | | + +- *1*: Only if the user is not an external one +- *2*: Only if the user is a member of the project ### New CI job permissions model @@ -334,17 +360,4 @@ for details about the pipelines security model. ## LDAP users permissions Since GitLab 8.15, LDAP user permissions can now be manually overridden by an admin user. -Read through the documentation on [LDAP users permissions](https://docs.gitlab.com/ee/articles/how_to_configure_ldap_gitlab_ee/index.html#updating-user-permissions-new-feature) to learn more. - -[^1]: On public and internal projects, all users are able to perform this action -[^2]: Guest users can only view the confidential issues they created themselves -[^3]: If **Public pipelines** is enabled in **Project Settings > CI/CD** -[^4]: Not allowed for Guest, Reporter, Developer, Maintainer, or Owner -[^5]: Only if the job was triggered by the user -[^6]: Only if user is not external one -[^7]: Only if user is a member of the project - -[ce-18994]: https://gitlab.com/gitlab-org/gitlab-ce/issues/18994 -[new-mod]: project/new_ci_build_permissions_model.md -[ee-998]: https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/998 -[eep]: https://about.gitlab.com/pricing/ +Read through the documentation on [LDAP users permissions](../administration/auth/how_to_configure_ldap_gitlab_ee/index.html) to learn more. diff --git a/doc/user/profile/account/delete_account.md b/doc/user/profile/account/delete_account.md index b497cc414af..0d7bbb0af79 100644 --- a/doc/user/profile/account/delete_account.md +++ b/doc/user/profile/account/delete_account.md @@ -14,7 +14,7 @@ Deleting a user will delete all projects in that user namespace. [GitLab 9.1][ce-10273], and from the API in [GitLab 9.3][ce-11853]. When a user account is deleted, not all associated records are deleted with it. -Here's a list of things that will not be deleted: +Here's a list of things that will **not** be deleted: - Issues that the user created - Merge requests that the user created diff --git a/doc/user/profile/account/index.md b/doc/user/profile/account/index.md index 06667bfc5f1..56b6498e16c 100644 --- a/doc/user/profile/account/index.md +++ b/doc/user/profile/account/index.md @@ -1,2 +1,5 @@ +--- +redirect_to: '../index.md#profile-settings' +--- This document was moved to [../index.md#profile-settings](../index.md#profile-settings). diff --git a/doc/user/profile/account/two_factor_authentication.md b/doc/user/profile/account/two_factor_authentication.md index 8e5fe4b0fb9..df413a11af0 100644 --- a/doc/user/profile/account/two_factor_authentication.md +++ b/doc/user/profile/account/two_factor_authentication.md @@ -18,9 +18,10 @@ the second factor of authentication. Once enabled, in addition to supplying your password to login, you'll be prompted to activate your U2F device (usually by pressing a button on it), and it will perform secure authentication on your behalf. -The U2F workflow is only [supported by](https://caniuse.com/#search=U2F) Google Chrome, Opera and Firefox at this point, so we _strongly_ recommend -that you set up both methods of two-factor authentication, so you can still access your account -from other browsers. +The U2F workflow is [supported by](https://caniuse.com/#search=U2F) Google Chrome, Opera, and Firefox. + +We recommend that you set up 2FA with both a [one-time password authenticator](#enable-2fa-via-one-time-password-authenticator) and a [U2F device](#enable-2fa-via-u2f-device), so you can still access your account +if you lose your U2F device. ## Enabling 2FA @@ -59,6 +60,7 @@ of recovery codes. ### Enable 2FA via U2F device > **Notes:** +> > - GitLab officially only supports [Yubikey] U2F devices. > - Support for U2F devices was added in GitLab 8.8. @@ -159,8 +161,8 @@ a new set of recovery codes with SSH. 1. Run `ssh git@gitlab.example.com 2fa_recovery_codes`. 1. You are prompted to confirm that you want to generate new codes. Continuing this process invalidates previously saved codes. - ``` - bash + + ```sh $ ssh git@gitlab.example.com 2fa_recovery_codes Are you sure you want to generate new two-factor recovery codes? Any existing recovery codes you saved will be invalidated. (yes/no) @@ -205,17 +207,17 @@ Sign in and re-enable two-factor authentication as soon as possible. - You need to take special care to that 2FA keeps working after [restoring a GitLab backup](../../../raketasks/backup_restore.md). - To ensure 2FA authorizes correctly with TOTP server, you may want to ensure - your GitLab server's time is synchronized via a service like NTP. Otherwise, + your GitLab server's time is synchronized via a service like NTP. Otherwise, you may have cases where authorization always fails because of time differences. - The GitLab U2F implementation does _not_ work when the GitLab instance is accessed from multiple hostnames, or FQDNs. Each U2F registration is linked to the _current hostname_ at the time of registration, and cannot be used for other hostnames/FQDNs. - For example, if a user is trying to access a GitLab instance from `first.host.xyz` and `second.host.xyz`: + For example, if a user is trying to access a GitLab instance from `first.host.xyz` and `second.host.xyz`: - - The user logs in via `first.host.xyz` and registers their U2F key. - - The user logs out and attempts to log in via `first.host.xyz` - U2F authentication succeeds. - - The user logs out and attempts to log in via `second.host.xyz` - U2F authentication fails, because + - The user logs in via `first.host.xyz` and registers their U2F key. + - The user logs out and attempts to log in via `first.host.xyz` - U2F authentication succeeds. + - The user logs out and attempts to log in via `second.host.xyz` - U2F authentication fails, because the U2F key has only been registered on `first.host.xyz`. [Google Authenticator]: https://support.google.com/accounts/answer/1066447?hl=en diff --git a/doc/user/profile/active_sessions.md b/doc/user/profile/active_sessions.md index 5119c0e30d0..28e3f4904a9 100644 --- a/doc/user/profile/active_sessions.md +++ b/doc/user/profile/active_sessions.md @@ -4,7 +4,7 @@ > in GitLab 10.8. GitLab lists all devices that have logged into your account. This allows you to -review the sessions and revoke any of it that you don't recognize. +review the sessions. ## Listing all active sessions @@ -12,9 +12,3 @@ review the sessions and revoke any of it that you don't recognize. 1. Navigate to the **Active Sessions** tab.  - -## Revoking a session - -1. Navigate to your [profile's](#profile-settings) **Settings > Active Sessions**. -1. Click on **Revoke** besides a session. The current session cannot be - revoked, as this would sign you out of GitLab. diff --git a/doc/user/profile/img/active_sessions_list.png b/doc/user/profile/img/active_sessions_list.png Binary files differindex 5d94dca69cc..1e242ac4710 100644 --- a/doc/user/profile/img/active_sessions_list.png +++ b/doc/user/profile/img/active_sessions_list.png diff --git a/doc/user/profile/img/personal_access_tokens.png b/doc/user/profile/img/personal_access_tokens.png Binary files differdeleted file mode 100644 index d29f4cb0a20..00000000000 --- a/doc/user/profile/img/personal_access_tokens.png +++ /dev/null diff --git a/doc/user/profile/index.md b/doc/user/profile/index.md index a2b15d058d7..61a30a775b0 100644 --- a/doc/user/profile/index.md +++ b/doc/user/profile/index.md @@ -33,13 +33,14 @@ From there, you can: - Set a [custom status](#current-status) for your profile - Manage your [commit email](#commit-email) for your profile - Manage [2FA](account/two_factor_authentication.md) +- Add details of [external accounts](#add-details-of-external-accounts). - Change your username and [delete your account](account/delete_account.md) - Manage applications that can [use GitLab as an OAuth provider](../../integration/oauth_provider.md#introduction-to-oauth) - Manage [personal access tokens](personal_access_tokens.md) to access your account via API and authorized applications - Add and delete emails linked to your account - Choose which email to use for notifications, web-based commits, and display on your public profile -- Manage [SSH keys](../../ssh/README.md#ssh) to access your account via SSH +- Manage [SSH keys](../../ssh/README.md) to access your account via SSH - Manage your [preferences](preferences.md#syntax-highlighting-theme) to customize your own GitLab experience - [View your active sessions](active_sessions.md) and revoke any of them if necessary @@ -92,6 +93,16 @@ To enable private profile: NOTE: **Note:** You and GitLab admins can see your the abovementioned information on your profile even if it is private. +## Add details of external accounts + +GitLab allows you to add links to certain other external accounts you might have, like Skype and Twitter. They can help other users connect with you on other platforms. + +To add links to other accounts: + +1. Navigate to your **User Settings > Profile**. +1. In the **Main settings** section, locate and fill out fields for links to external accounts like Skype and Twitter. +1. Click the **Update profile settings** button. + ## Private contributions > [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/issues/14078) in GitLab 11.3. diff --git a/doc/user/profile/personal_access_tokens.md b/doc/user/profile/personal_access_tokens.md index 7d55048c994..0b224fc7e01 100644 --- a/doc/user/profile/personal_access_tokens.md +++ b/doc/user/profile/personal_access_tokens.md @@ -5,12 +5,9 @@ Personal access tokens are the preferred way for third party applications and scripts to authenticate with the [GitLab API][api], if using [OAuth2](../../api/oauth2.md) is not practical. -You can also use them to authenticate against Git over HTTP. They are the only -accepted method of authentication when you have -[Two-Factor Authentication (2FA)][2fa] enabled. +You can also use personal access tokens to authenticate against Git over HTTP or SSH. They must be used when you have [Two-Factor Authentication (2FA)][2fa] enabled. Authenticate with a token in place of your password. -Once you have your token, [pass it to the API][usage] using either the -`private_token` parameter or the `Private-Token` header. +To make [authenticated requests to the API][usage], use either the `private_token` parameter or the `Private-Token` header. The expiration of personal access tokens happens on the date you define, at midnight UTC. @@ -20,21 +17,19 @@ at midnight UTC. You can create as many personal access tokens as you like from your GitLab profile. -1. Log in to your GitLab account. -1. Go to your **Profile settings**. -1. Go to **Access tokens**. -1. Choose a name and optionally an expiry date for the token. +1. Log in to GitLab. +1. In the upper-right corner, click your avatar and select **Settings**. +1. On the **User Settings** menu, select **Access Tokens**. +1. Choose a name and optional expiry date for the token. 1. Choose the [desired scopes](#limiting-scopes-of-a-personal-access-token). -1. Click on **Create personal access token**. +1. Click the **Create personal access token** button. 1. Save the personal access token somewhere safe. Once you leave or refresh the page, you won't be able to access it again. - +### Revoking a personal access token -## Revoking a personal access token - -At any time, you can revoke any personal access token by just clicking the -respective **Revoke** button under the 'Active personal access tokens' area. +At any time, you can revoke any personal access token by clicking the +respective **Revoke** button under the **Active Personal Access Token** area. ## Limiting scopes of a personal access token @@ -42,18 +37,18 @@ Personal access tokens can be created with one or more scopes that allow various actions that a given token can perform. The available scopes are depicted in the following table. -| Scope | Description | -| ----- | ----------- | -|`read_user` | Allows access to the read-only endpoints under `/users`. Essentially, any of the `GET` requests in the [Users API][users] are allowed ([introduced][ce-5951] in GitLab 8.15). | -| `api` | Grants complete access to the API and Container Registry (read/write) ([introduced](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/5951) in GitLab 8.15). Required for accessing Git repositories over HTTP when 2FA is enabled. | -| `read_registry` | Allows to read (pull) [container registry] images if a project is private and authorization is required ([introduced](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/11845) in GitLab 9.3). | -| `sudo` | Allows performing API actions as any user in the system (if the authenticated user is an admin) ([introduced][ce-14838] in GitLab 10.2). | -| `read_repository` | Allows read-access (pull) to the repository through git clone. | +| Scope | Introduced in | Description | +| ------------------ | ------------- | ----------- | +| `read_user` | [GitLab 8.15](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/5951) | Allows access to the read-only endpoints under `/users`. Essentially, any of the `GET` requests in the [Users API][users] are allowed. | +| `api` | [GitLab 8.15](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/5951) | Grants complete access to the API and Container Registry (read/write). | +| `read_registry` | [GitLab 9.3](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/11845) | Allows to read (pull) [container registry] images if a project is private and authorization is required. | +| `sudo` | [GitLab 10.2](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/14838) | Allows performing API actions as any user in the system (if the authenticated user is an admin). | +| `read_repository` | [GitLab 10.7](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/17894) | Allows read-only access (pull) to the repository through git clone. | +| `write_repository` | [GitLab 11.11](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/26021) | Allows read-write access (pull, push) to the repository through git clone. Required for accessing Git repositories over HTTP when 2FA is enabled. | [2fa]: ../account/two_factor_authentication.md [api]: ../../api/README.md [ce-3749]: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/3749 -[ce-14838]: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/14838 [container registry]: ../project/container_registry.md [users]: ../../api/users.md [usage]: ../../api/README.md#personal-access-tokens diff --git a/doc/user/profile/preferences.md b/doc/user/profile/preferences.md index db68510c46d..b61216b7b67 100644 --- a/doc/user/profile/preferences.md +++ b/doc/user/profile/preferences.md @@ -79,6 +79,7 @@ You have 8 options here that you can use for your default dashboard view: - Your [Todos](../../workflow/todos.md) - Assigned Issues - Assigned Merge Requests +- Operations Dashboard **[PREMIUM]** ### Project overview content @@ -103,4 +104,10 @@ Select your preferred language from a list of supported languages. The first day of the week can be customised for calendar views and date pickers. -You can choose **Sunday** or **Monday** as the first day of the week. If you select **System Default**, the system-wide default setting will be used. +You can choose one of the following options as the first day of the week: + +- Saturday +- Sunday +- Monday + +If you select **System Default**, the system-wide default setting will be used. diff --git a/doc/user/project/badges.md b/doc/user/project/badges.md index 19eb95099ce..8849dd2d684 100644 --- a/doc/user/project/badges.md +++ b/doc/user/project/badges.md @@ -63,6 +63,12 @@ are available: - `%{commit_sha}`: ID of the most recent commit to the default branch of a project's repository +NOTE: **Note:** +Placeholders allow badges to expose otherwise-private information, such as the +default branch or commit SHA when the project is configured to have a private +repository. This is by design, as badges are intended to be used publicly. Avoid +using these placeholders if the information is sensitive. + ## API You can also configure badges via the GitLab API. As in the settings, there is diff --git a/doc/user/project/builds/artifacts.md b/doc/user/project/builds/artifacts.md index 514c729b37d..1b0f3f61394 100644 --- a/doc/user/project/builds/artifacts.md +++ b/doc/user/project/builds/artifacts.md @@ -1 +1,5 @@ +--- +redirect_to: '../pipelines/job_artifacts.md' +--- + This document was moved to [pipelines/job_artifacts](../pipelines/job_artifacts.md). diff --git a/doc/user/project/bulk_editing.md b/doc/user/project/bulk_editing.md index fead99c5e88..d0c7daf4692 100644 --- a/doc/user/project/bulk_editing.md +++ b/doc/user/project/bulk_editing.md @@ -1,6 +1,7 @@ # Bulk editing issues and merge requests > **Notes:** +> > - A permission level of `Reporter` or higher is required in order to manage > issues. > - A permission level of `Developer` or higher is required in order to manage diff --git a/doc/user/project/canary_deployments.md b/doc/user/project/canary_deployments.md new file mode 100644 index 00000000000..cce862b0911 --- /dev/null +++ b/doc/user/project/canary_deployments.md @@ -0,0 +1,71 @@ +# Canary Deployments **[PREMIUM]** + +> [Introduced][ee-1659] in [GitLab Premium][eep] 9.1. + +A popular [Continuous Integration](https://en.wikipedia.org/wiki/Continuous_integration) +strategy, where a small portion of the fleet is updated to the new version of +your application. + +## Overview + +When embracing [Continuous Delivery][cd-blog], a company needs to decide what +type of deployment strategy to use. One of the most popular strategies is canary +deployments, where a small portion of the fleet is updated to the new version +first. This subset, the canaries, then serve as the proverbial +[canary in the coal mine](https://en.wiktionary.org/wiki/canary_in_a_coal_mine). + +If there is a problem with the new version of the application, only a small +percentage of users are affected and the change can either be fixed or quickly +reverted. + +Leveraging [Kubernetes' Canary deployments][kube-canary], visualize your canary +deployments right inside the [Deploy Board], without the need to leave GitLab. + +## Use cases + +Canary deployments can be used when you want to ship features to only a portion of +your pods fleet and watch their behavior as a percentage of your user base +visits the temporarily deployed feature. If all works well, you can deploy the +feature to production knowing that it won't cause any problems. + +Canary deployments are also especially useful for backend refactors, performance +improvements, or other changes where the user interface doesn't change, but you +want to make sure the performance stays the same, or improves. Developers need +to be careful when using canaries with user-facing changes, because by default, +requests from the same user will be randomly distributed between canary and +non-canary pods, which could result in confusion or even errors. If needed, you +may want to consider [setting `service.spec.sessionAffinity` to `ClientIP` in +your Kubernetes service definitions][kube-net], but that is beyond the scope of +this document. + +## Enabling Canary Deployments + +Canary deployments require that you properly configure Deploy Boards: + +1. Follow the steps to [enable Deploy Boards](deploy_boards.md#enabling-deploy-boards). +1. To track canary deployments you need to label your Kubernetes deployments and + pods with `track: canary`. To get started quickly, you can use the [Auto Deploy] + template for canary deployments that GitLab provides. + +Depending on the deploy, the label should be either `stable` or `canary`. +Usually, `stable` and blank or missing label means the same thing, and `canary` +or any other track means canary/temporary. +This allows GitLab to discover whether deployment is stable or canary (temporary). + +Once all of the above are set up and the pipeline has run at least once, +navigate to the environments page under **Pipelines > Environments**. +As the pipeline executes Deploy Boards will clearly mark canary pods, enabling +quick and easy insight into the status of each environment and deployment. + +Canary deployments are marked with a yellow dot in the Deploy Board so that you +can easily notice them. + + + +[autodeploy]: ../../ci/autodeploy/index.md "GitLab Autodeploy" +[eep]: https://about.gitlab.com/pricing/ +[ee-1659]: https://gitlab.com/gitlab-org/gitlab-ee/issues/1659 +[kube-canary]: https://kubernetes.io/docs/concepts/cluster-administration/manage-deployment/#canary-deployments +[deploy board]: deploy_boards.md +[cd-blog]: https://about.gitlab.com/2016/08/05/continuous-integration-delivery-and-deployment-with-gitlab/ +[kube-net]: https://kubernetes.io/docs/concepts/services-networking/service/#virtual-ips-and-service-proxies diff --git a/doc/user/project/ci_cd_for_external_repo.md b/doc/user/project/ci_cd_for_external_repo.md new file mode 100644 index 00000000000..a92d3a2c308 --- /dev/null +++ b/doc/user/project/ci_cd_for_external_repo.md @@ -0,0 +1,5 @@ +--- +redirect_to: '../../ci/ci_cd_for_external_repos/index.md' +--- + +This document was moved to [another location](../../ci/ci_cd_for_external_repos/index.md). diff --git a/doc/user/project/clusters/img/k8s_cluster_monitoring.png b/doc/user/project/clusters/img/k8s_cluster_monitoring.png Binary files differnew file mode 100644 index 00000000000..e449893a606 --- /dev/null +++ b/doc/user/project/clusters/img/k8s_cluster_monitoring.png diff --git a/doc/user/project/clusters/img/kubernetes_pod_logs.png b/doc/user/project/clusters/img/kubernetes_pod_logs.png Binary files differnew file mode 100644 index 00000000000..e664a47386a --- /dev/null +++ b/doc/user/project/clusters/img/kubernetes_pod_logs.png diff --git a/doc/user/project/clusters/img/pod_logs_deploy_board.png b/doc/user/project/clusters/img/pod_logs_deploy_board.png Binary files differnew file mode 100644 index 00000000000..7f83382968b --- /dev/null +++ b/doc/user/project/clusters/img/pod_logs_deploy_board.png diff --git a/doc/user/project/clusters/index.md b/doc/user/project/clusters/index.md index e601d7b2ccc..dc21db603d6 100644 --- a/doc/user/project/clusters/index.md +++ b/doc/user/project/clusters/index.md @@ -19,8 +19,10 @@ or provide the credentials to an [existing Kubernetes cluster](#adding-an-existi NOTE: **Note:** From [GitLab 11.6](https://gitlab.com/gitlab-org/gitlab-ce/issues/34758) you -can also associate a Kubernetes cluster to your groups. Learn more about -[group Kubernetes clusters](../../group/clusters/index.md). +can also associate a Kubernetes cluster to your groups and from +[GitLab 11.11](https://gitlab.com/gitlab-org/gitlab-ce/issues/39840), +to the GitLab instance. Learn more about [group-level](../../group/clusters/index.md) +and [instance-level](../../instance/clusters/index.md) Kubernetes clusters. ## Adding and creating a new GKE cluster via GitLab @@ -41,7 +43,7 @@ integration, make sure the following requirements are met: - A [billing account](https://cloud.google.com/billing/docs/how-to/manage-billing-account) is set up and you have permissions to access it. -- The Kubernetes Engine API is enabled. Follow the steps as outlined in the +- The Kubernetes Engine API and related service are enabled. It should work immediately but may take up to 10 minutes after you create a project. For more information see the ["Before you begin" section of the Kubernetes Engine docs](https://cloud.google.com/kubernetes-engine/docs/quickstart#before-you-begin). ### Creating the cluster @@ -69,11 +71,21 @@ new Kubernetes cluster to your project: - **Number of nodes** - Enter the number of nodes you wish the cluster to have. - **Machine type** - The [machine type](https://cloud.google.com/compute/docs/machine-types) of the Virtual Machine instance that the cluster will be based on. + - **RBAC-enabled cluster** - Leave this checked if using default GKE creation options, see the [RBAC section](#rbac-cluster-resources) for more information. + - **GitLab-managed cluster** - Leave this checked if you want GitLab to manage namespaces and service accounts for this cluster. See the [Managed clusters section](#gitlab-managed-clusters) for more information. 1. Finally, click the **Create Kubernetes cluster** button. After a couple of minutes, your cluster will be ready to go. You can now proceed to install some [pre-defined applications](#installing-applications). +NOTE: **Note:** +GitLab requires basic authentication enabled and a client certificate issued for +the cluster in order to setup an [initial service +account](#access-controls). Starting from [GitLab +11.10](https://gitlab.com/gitlab-org/gitlab-ce/issues/58208), the cluster +creation process will explicitly request that basic authentication and +client certificate is enabled. + ## Adding an existing Kubernetes cluster To add an existing Kubernetes cluster to your project: @@ -87,19 +99,25 @@ To add an existing Kubernetes cluster to your project: 1. Click **Add an existing Kubernetes cluster** and fill in the details: - **Kubernetes cluster name** (required) - The name you wish to give the cluster. - **Environment scope** (required) - The - [associated environment](#setting-the-environment-scope) to this cluster. + [associated environment](#setting-the-environment-scope-premium) to this cluster. - **API URL** (required) - It's the URL that GitLab uses to access the Kubernetes API. Kubernetes exposes several APIs, we want the "base" URL that is common to all of them, e.g., `https://kubernetes.example.com` rather than `https://kubernetes.example.com/api/v1`. + + Get the API URL by running this command: + + ```sh + kubectl cluster-info | grep 'Kubernetes master' | awk '/http/ {print $NF}' + ``` - **CA certificate** (required) - A valid Kubernetes certificate is needed to authenticate to the EKS cluster. We will use the certificate created by default. - - List the secrets with `kubectl get secrets`, and one should named similar to + - List the secrets with `kubectl get secrets`, and one should named similar to `default-token-xxxxx`. Copy that token name for use below. - - Get the certificate by running this command: + - Get the certificate by running this command: - ```sh - kubectl get secret <secret name> -o jsonpath="{['data']['ca\.crt']}" | base64 --decode - ``` + ```sh + kubectl get secret <secret name> -o jsonpath="{['data']['ca\.crt']}" | base64 --decode + ``` - **Token** - GitLab authenticates against Kubernetes using service tokens, which are scoped to a particular `namespace`. @@ -109,15 +127,28 @@ To add an existing Kubernetes cluster to your project: 1. Create a file called `gitlab-admin-service-account.yaml` with contents: - ```yaml - apiVersion: v1 - kind: ServiceAccount - metadata: - name: gitlab-admin - namespace: kube-system - ``` - - 2. Apply the service account to your cluster: + ```yaml + apiVersion: v1 + kind: ServiceAccount + metadata: + name: gitlab-admin + namespace: kube-system + --- + apiVersion: rbac.authorization.k8s.io/v1beta1 + kind: ClusterRoleBinding + metadata: + name: gitlab-admin + roleRef: + apiGroup: rbac.authorization.k8s.io + kind: ClusterRole + name: cluster-admin + subjects: + - kind: ServiceAccount + name: gitlab-admin + namespace: kube-system + ``` + + 1. Apply the service account and cluster role binding to your cluster: ```bash kubectl apply -f gitlab-admin-service-account.yaml @@ -125,69 +156,44 @@ To add an existing Kubernetes cluster to your project: Output: - ```bash - serviceaccount "gitlab-admin" created - ``` - - 3. Create a file called `gitlab-admin-cluster-role-binding.yaml` with contents: - - ```yaml - apiVersion: rbac.authorization.k8s.io/v1beta1 - kind: ClusterRoleBinding - metadata: - name: gitlab-admin - roleRef: - apiGroup: rbac.authorization.k8s.io - kind: ClusterRole - name: cluster-admin - subjects: - - kind: ServiceAccount - name: gitlab-admin - namespace: kube-system - ``` - - 4. Apply the cluster role binding to your cluster: - - ```bash - kubectl apply -f gitlab-admin-cluster-role-binding.yaml - ``` - - Output: - ```bash + serviceaccount "gitlab-admin" created clusterrolebinding "gitlab-admin" created ``` - 5. Retrieve the token for the `gitlab-admin` service account: + 1. Retrieve the token for the `gitlab-admin` service account: ```bash kubectl -n kube-system describe secret $(kubectl -n kube-system get secret | grep gitlab-admin | awk '{print $1}') ``` - Copy the `<authentication_token>` value from the output: + Copy the `<authentication_token>` value from the output: - ```yaml - Name: gitlab-admin-token-b5zv4 - Namespace: kube-system - Labels: <none> - Annotations: kubernetes.io/service-account.name=gitlab-admin - kubernetes.io/service-account.uid=bcfe66ac-39be-11e8-97e8-026dce96b6e8 + ```yaml + Name: gitlab-admin-token-b5zv4 + Namespace: kube-system + Labels: <none> + Annotations: kubernetes.io/service-account.name=gitlab-admin + kubernetes.io/service-account.uid=bcfe66ac-39be-11e8-97e8-026dce96b6e8 - Type: kubernetes.io/service-account-token + Type: kubernetes.io/service-account-token + + Data + ==== + ca.crt: 1025 bytes + namespace: 11 bytes + token: <authentication_token> + ``` - Data - ==== - ca.crt: 1025 bytes - namespace: 11 bytes - token: <authentication_token> - ``` - NOTE: **Note:** For GKE clusters, you will need the `container.clusterRoleBindings.create` permission to create a cluster role binding. You can follow the [Google Cloud documentation](https://cloud.google.com/iam/docs/granting-changing-revoking-access) to grant access. + + - **GitLab-managed cluster** - Leave this checked if you want GitLab to manage namespaces and service accounts for this cluster. See the [Managed clusters section](#gitlab-managed-clusters) for more information. + - **Project namespace** (optional) - You don't have to fill it in; by leaving it blank, GitLab will create one for you. Also: - Each project should have a unique namespace. @@ -203,14 +209,6 @@ To add an existing Kubernetes cluster to your project: After a couple of minutes, your cluster will be ready to go. You can now proceed to install some [pre-defined applications](#installing-applications). -To determine the: - -- API URL, run `kubectl cluster-info | grep 'Kubernetes master' | awk '/http/ {print $NF}'`. -- Token: - 1. List the secrets by running: `kubectl get secrets`. Note the name of the secret you need the token for. - 1. Get the token for the appropriate secret by running: `kubectl get secret <SECRET_NAME> -o jsonpath="{['data']['token']}" | base64 --decode`. -- CA certificate, run `kubectl get secret <secret name> -o jsonpath="{['data']['ca\.crt']}" | base64 --decode`. - ## Security implications CAUTION: **Important:** @@ -222,78 +220,112 @@ functionalities needed to successfully build and deploy a containerized application. Bear in mind that the same credentials are used for all the applications running on the cluster. +## Gitlab-managed clusters + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/22011) in GitLab 11.5. +> Became [optional](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/26565) in GitLab 11.11. + +NOTE: **Note:** +Only available when creating clusters. Existing clusters not managed by GitLab +cannot become GitLab-managed later. + +You can choose to allow GitLab to manage your cluster for you. If your cluster is +managed by GitLab, resources for your projects will be automatically created. See the +[Access controls](#access-controls) section for details on which resources will +be created. + +If you choose to manage your own cluster, project-specific resources will not be created +automatically. If you are using [Auto DevOps](../../../topics/autodevops/index.md), you will +need to explicitly provide the `KUBE_NAMESPACE` [deployment variable](#deployment-variables) +that will be used by your deployment jobs, otherwise a namespace will be created for you. + +NOTE: **Note:** +If you [install applications](#installing-applications) on your cluster, GitLab will create +the resources required to run these even if you have chosen to manage your own cluster. + ## Base domain > [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/24580) in GitLab 11.8. -Domains at the cluster level permit support for multiple domains -per [multiple Kubernetes clusters](#multiple-kubernetes-clusters-premium). When specifying a domain, -this will be automatically set as an environment variable (`KUBE_INGRESS_BASE_DOMAIN`) during -the [Auto DevOps](../../../topics/autodevops/index.md) stages. +NOTE: **Note:** +You do not need to specify a base domain on cluster settings when using GitLab Serverless. The domain in that case +will be specified as part of the Knative installation. See [Installing Applications](#installing-applications). + +Specifying a base domain will automatically set `KUBE_INGRESS_BASE_DOMAIN` as an environment variable. +If you are using [Auto DevOps](../../../topics/autodevops/index.md), this domain will be used for the different +stages. For example, Auto Review Apps and Auto Deploy. + +The domain should have a wildcard DNS configured to the Ingress IP address. After ingress has been installed (see [Installing Applications](#installing-applications)), +you can either: -The domain should have a wildcard DNS configured to the Ingress IP address. +- Create an `A` record that points to the Ingress IP address with your domain provider. +- Enter a wildcard DNS address using a service such as nip.io or xip.io. For example, `192.168.1.1.xip.io`. ## Access controls -When creating a cluster in GitLab, you will be asked if you would like to create an -[Attribute-based access control (ABAC)](https://kubernetes.io/docs/admin/authorization/abac/) cluster, or -a [Role-based access control (RBAC)](https://kubernetes.io/docs/admin/authorization/rbac/) one. +When creating a cluster in GitLab, you will be asked if you would like to create either: -NOTE: **Note:** -[RBAC](#role-based-access-control-rbac) is recommended and the GitLab default. +- An [Attribute-based access control (ABAC)](https://kubernetes.io/docs/admin/authorization/abac/) cluster. +- A [Role-based access control (RBAC)](https://kubernetes.io/docs/admin/authorization/rbac/) cluster. -Whether [ABAC](#attribute-based-access-control-abac) or [RBAC](#role-based-access-control-rbac) is enabled, -GitLab will create the necessary service accounts and privileges in order to install and run -[GitLab managed applications](#installing-applications): +NOTE: **Note:** +[RBAC](#rbac-cluster-resources) is recommended and the GitLab default. -- If GitLab is creating the cluster, a `gitlab` service account with - `cluster-admin` privileges will be created in the `default` namespace, - which will be used by GitLab to manage the newly created cluster. +GitLab creates the necessary service accounts and privileges to install and run +[GitLab managed applications](#installing-applications). When GitLab creates the cluster: +- A `gitlab` service account with `cluster-admin` privileges is created in the `default` namespace + to manage the newly created cluster. - A project service account with [`edit` privileges](https://kubernetes.io/docs/reference/access-authn-authz/rbac/#user-facing-roles) - will be created in the project namespace (also created by GitLab), which will - be used in [deployment jobs](#deployment-variables). + is created in the GitLab-created project namespace for [deployment jobs](#deployment-variables). NOTE: **Note:** Restricted service account for deployment was [introduced](https://gitlab.com/gitlab-org/gitlab-ce/issues/51716) in GitLab 11.5. -- When you install Helm Tiller into your cluster, the `tiller` service account - will be created with `cluster-admin` privileges in the `gitlab-managed-apps` - namespace. This service account will be added to the installed Helm Tiller and will - be used by Helm to install and run [GitLab managed applications](#installing-applications). - Helm Tiller will also create additional service accounts and other resources for each - installed application. Consult the documentation of the Helm charts for each application - for details. +When you install Helm into your cluster, the `tiller` service account +is created with `cluster-admin` privileges in the `gitlab-managed-apps` +namespace. This service account will be added to the installed Helm Tiller and will +be used by Helm to install and run [GitLab managed applications](#installing-applications). +Helm will also create additional service accounts and other resources for each +installed application. Consult the documentation of the Helm charts for each application +for details. If you are [adding an existing Kubernetes cluster](#adding-an-existing-kubernetes-cluster), ensure the token of the account has administrator privileges for the cluster. -The following sections summarize which resources will be created on ABAC/RBAC clusters. +The resources created by GitLab differ depending on the type of cluster. + +### ABAC cluster resources + +GitLab creates the following resources for ABAC clusters. -### Attribute-based access control (ABAC) +| Name | Type | Details | Created when | +|:------------------|:---------------------|:----------------------------------|:---------------------------| +| `gitlab` | `ServiceAccount` | `default` namespace | Creating a new GKE Cluster | +| `gitlab-token` | `Secret` | Token for `gitlab` ServiceAccount | Creating a new GKE Cluster | +| `tiller` | `ServiceAccount` | `gitlab-managed-apps` namespace | Installing Helm Tiller | +| `tiller-admin` | `ClusterRoleBinding` | `cluster-admin` roleRef | Installing Helm Tiller | +| Project namespace | `ServiceAccount` | Uses namespace of Project | Deploying to a cluster | +| Project namespace | `Secret` | Token for project ServiceAccount | Deploying to a cluster | -| Name | Kind | Details | Created when | -| --- | --- | --- | --- | -| `gitlab` | `ServiceAccount` | `default` namespace | Creating a new GKE Cluster | -| `gitlab-token` | `Secret` | Token for `gitlab` ServiceAccount | Creating a new GKE Cluster | -| `tiller` | `ServiceAccount` | `gitlab-managed-apps` namespace | Installing Helm Tiller | -| `tiller-admin` | `ClusterRoleBinding` | `cluster-admin` roleRef | Installing Helm Tiller | -| Project namespace | `ServiceAccount` | Uses namespace of Project | Creating/Adding a new GKE Cluster | -| Project namespace | `Secret` | Token for project ServiceAccount | Creating/Adding a new GKE Cluster | +### RBAC cluster resources -### Role-based access control (RBAC) +GitLab creates the following resources for RBAC clusters. -| Name | Kind | Details | Created when | -| --- | --- | --- | --- | -| `gitlab` | `ServiceAccount` | `default` namespace | Creating a new GKE Cluster | -| `gitlab-admin` | `ClusterRoleBinding` | [`cluster-admin`](https://kubernetes.io/docs/reference/access-authn-authz/rbac/#user-facing-roles) roleRef | Creating a new GKE Cluster | -| `gitlab-token` | `Secret` | Token for `gitlab` ServiceAccount | Creating a new GKE Cluster | -| `tiller` | `ServiceAccount` | `gitlab-managed-apps` namespace | Installing Helm Tiller | -| `tiller-admin` | `ClusterRoleBinding` | `cluster-admin` roleRef | Installing Helm Tiller | -| Project namespace | `ServiceAccount` | Uses namespace of Project | Creating/Adding a new GKE Cluster | -| Project namespace | `Secret` | Token for project ServiceAccount | Creating/Adding a new GKE Cluster | -| Project namespace | `RoleBinding` | [`edit`](https://kubernetes.io/docs/reference/access-authn-authz/rbac/#user-facing-roles) roleRef | Creating/Adding a new GKE Cluster | +| Name | Type | Details | Created when | +|:------------------|:---------------------|:-----------------------------------------------------------------------------------------------------------|:---------------------------| +| `gitlab` | `ServiceAccount` | `default` namespace | Creating a new GKE Cluster | +| `gitlab-admin` | `ClusterRoleBinding` | [`cluster-admin`](https://kubernetes.io/docs/reference/access-authn-authz/rbac/#user-facing-roles) roleRef | Creating a new GKE Cluster | +| `gitlab-token` | `Secret` | Token for `gitlab` ServiceAccount | Creating a new GKE Cluster | +| `tiller` | `ServiceAccount` | `gitlab-managed-apps` namespace | Installing Helm Tiller | +| `tiller-admin` | `ClusterRoleBinding` | `cluster-admin` roleRef | Installing Helm Tiller | +| Project namespace | `ServiceAccount` | Uses namespace of Project | Deploying to a cluster | +| Project namespace | `Secret` | Token for project ServiceAccount | Deploying to a cluster | +| Project namespace | `RoleBinding` | [`edit`](https://kubernetes.io/docs/reference/access-authn-authz/rbac/#user-facing-roles) roleRef | Deploying to a cluster | + +NOTE: **Note:** +Project-specific resources are only created if your cluster is [managed by GitLab](#gitlab-managed-clusters). ### Security of GitLab Runners @@ -315,91 +347,38 @@ install it manually. ## Installing applications -NOTE: **Note:** -Before starting the installation of applications, make sure that time is synchronized -between your GitLab server and your Kubernetes cluster. Otherwise, installation could fail -and you may get errors like `Error: remote error: tls: bad certificate` -in the `stdout` of pods created by GitLab in your Kubernetes cluster. - -GitLab provides a one-click install for various applications which can -be added directly to your configured cluster. Those applications are -needed for [Review Apps](../../../ci/review_apps/index.md) and -[deployments](../../../ci/environments.md). You can install them after you -[create a cluster](#adding-and-creating-a-new-gke-cluster-via-gitlab). - -To see a list of available applications to install: - -1. Navigate to your project's **Operations > Kubernetes**. -1. Select your cluster. - -Install Helm Tiller first because it's used to install other applications. - -NOTE: **Note:** -As of GitLab 11.6, Helm Tiller will be upgraded to the latest version supported -by GitLab before installing any of the applications. - -| Application | GitLab version | Description | Helm Chart | -| ----------- | :------------: | ----------- | --------------- | -| [Helm Tiller](https://docs.helm.sh/) | 10.2+ | Helm is a package manager for Kubernetes and is required to install all the other applications. It is installed in its own pod inside the cluster which can run the `helm` CLI in a safe environment. | n/a | -| [Ingress](https://kubernetes.io/docs/concepts/services-networking/ingress/) | 10.2+ | Ingress can provide load balancing, SSL termination, and name-based virtual hosting. It acts as a web proxy for your applications and is useful if you want to use [Auto DevOps] or deploy your own web apps. | [stable/nginx-ingress](https://github.com/helm/charts/tree/master/stable/nginx-ingress) | -| [Cert Manager](http://docs.cert-manager.io/en/latest/) | 11.6+ | Cert Manager is a native Kubernetes certificate management controller that helps with issuing certificates. Installing Cert Manager on your cluster will issue a certificate by [Let's Encrypt](https://letsencrypt.org/) and ensure that certificates are valid and up-to-date. | [stable/cert-manager](https://github.com/helm/charts/tree/master/stable/cert-manager) | -| [Prometheus](https://prometheus.io/docs/introduction/overview/) | 10.4+ | Prometheus is an open-source monitoring and alerting system useful to supervise your deployed applications. | [stable/prometheus](https://github.com/helm/charts/tree/master/stable/prometheus) | -| [GitLab Runner](https://docs.gitlab.com/runner/) | 10.6+ | GitLab Runner is the open source project that is used to run your jobs and send the results back to GitLab. It is used in conjunction with [GitLab CI/CD](https://about.gitlab.com/features/gitlab-ci-cd/), the open-source continuous integration service included with GitLab that coordinates the jobs. When installing the GitLab Runner via the applications, it will run in **privileged mode** by default. Make sure you read the [security implications](#security-implications) before doing so. | [runner/gitlab-runner](https://gitlab.com/charts/gitlab-runner) | -| [JupyterHub](http://jupyter.org/) | 11.0+ | [JupyterHub](https://jupyterhub.readthedocs.io/en/stable/) is a multi-user service for managing notebooks across a team. [Jupyter Notebooks](https://jupyter-notebook.readthedocs.io/en/latest/) provide a web-based interactive programming environment used for data analysis, visualization, and machine learning. We use a [custom Jupyter image](https://gitlab.com/gitlab-org/jupyterhub-user-image/blob/master/Dockerfile) that installs additional useful packages on top of the base Jupyter. You will also see ready-to-use DevOps Runbooks built with Nurtch's [Rubix library](https://github.com/amit1rrr/rubix). More information on creating executable runbooks can be found in [our Nurtch documentation](runbooks/index.md#nurtch-executable-runbooks). **Note**: Authentication will be enabled for any user of the GitLab server via OAuth2. HTTPS will be supported in a future release. | [jupyter/jupyterhub](https://jupyterhub.github.io/helm-chart/) | -| [Knative](https://cloud.google.com/knative) | 11.5+ | Knative provides a platform to create, deploy, and manage serverless workloads from a Kubernetes cluster. It is used in conjunction with, and includes [Istio](https://istio.io) to provide an external IP address for all programs hosted by Knative. You will be prompted to enter a wildcard domain where your applications will be exposed. Configure your DNS server to use the external IP address for that domain. For any application created and installed, they will be accessible as `<program_name>.<kubernetes_namespace>.<domain_name>`. This will require your kubernetes cluster to have [RBAC enabled](#role-based-access-control-rbac). | [knative/knative](https://storage.googleapis.com/triggermesh-charts) - -With the exception of Knative, the applications will be installed in a dedicated -namespace called `gitlab-managed-apps`. - -CAUTION: **Caution:** -If you have an existing Kubernetes cluster with Tiller already installed, -you should be careful as GitLab cannot detect it. In this case, installing -Tiller via the applications will result in the cluster having it twice, which -can lead to confusion during deployments. - -### Upgrading applications - -> [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/24789) -in GitLab 11.8. - -Users can perform a one-click upgrade for the GitLab Runner application, -when there is an upgrade available. - -To upgrade the GitLab Runner application: - -1. Navigate to your project's **Operations > Kubernetes**. -1. Select your cluster. -1. Click the **Upgrade** button for the Runnner application. - -The **Upgrade** button will not be shown if there is no upgrade -available. +GitLab can install and manage some applications in your project-level +cluster. For more information on installing, upgrading, uninstalling, +and troubleshooting applications for your project cluster, see +[Gitlab Managed Apps](../../clusters/applications.md). -NOTE: **Note:** -Upgrades will reset values back to the values built into the `runner` -chart plus the values set by -[`values.yaml`](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/vendor/runner/values.yaml) - -## Getting the external IP address +## Getting the external endpoint NOTE: **Note:** With the following procedure, a load balancer must be installed in your cluster -to obtain the external IP address. You can use either +to obtain the endpoint. You can use either [Ingress](#installing-applications), or Knative's own load balancer ([Istio](https://istio.io)) if using [Knative](#installing-applications). -In order to publish your web application, you first need to find the external IP -address associated to your load balancer. +In order to publish your web application, you first need to find the endpoint which will be either an IP +address or a hostname associated with your load balancer. -### Let GitLab fetch the IP address +### Automatically determining the external endpoint > [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/17052) in GitLab 10.6. -If you [installed Ingress or Knative](#installing-applications), -you should see the Ingress IP address on this same page within a few minutes. -If you don't see this, GitLab might not be able to determine the IP address of -your ingress application in which case you should manually determine it. +After you install [Ingress or Knative](#installing-applications), Gitlab attempts to determine the external endpoint +and it should be available within a few minutes. If the endpoint doesn't appear +and your cluster runs on Google Kubernetes Engine: + +1. Check your [Kubernetes cluster on Google Kubernetes Engine](https://console.cloud.google.com/kubernetes) to ensure there are no errors on its nodes. +1. Ensure you have enough [Quotas](https://console.cloud.google.com/iam-admin/quotas) on Google Kubernetes Engine. For more information, see [Resource Quotas](https://cloud.google.com/compute/quotas). +1. Check [Google Cloud's Status](https://status.cloud.google.com/) to ensure they are not having any disruptions. -### Manually determining the IP address +If GitLab is still unable to determine the endpoint of your Ingress or Knative application, you can +manually determine it by following the steps below. + +### Manually determining the external endpoint If the cluster is on GKE, click the **Google Kubernetes Engine** link in the **Advanced settings**, or go directly to the @@ -409,7 +388,7 @@ the `gcloud` command in a local terminal or using the **Cloud Shell**. If the cluster is not on GKE, follow the specific instructions for your Kubernetes provider to configure `kubectl` with the right credentials. -The output of the following examples will show the external IP address of your +The output of the following examples will show the external endpoint of your cluster. This information can then be used to set up DNS entries and forwarding rules that allow external access to your deployed applications. @@ -417,19 +396,19 @@ If you installed the Ingress [via the **Applications**](#installing-applications run the following command: ```bash -kubectl get svc --namespace=gitlab-managed-apps ingress-nginx-ingress-controller -o jsonpath='{.status.loadBalancer.ingress[0].ip} ' +kubectl get service --namespace=gitlab-managed-apps ingress-nginx-ingress-controller -o jsonpath='{.status.loadBalancer.ingress[0].ip}' ``` -For Istio/Knative, the command will be different: +Some Kubernetes clusters return a hostname instead, like [Amazon EKS](https://aws.amazon.com/eks/). For these platforms, run: ```bash -kubectl get svc --namespace=istio-system knative-ingressgateway -o jsonpath='{.status.loadBalancer.ingress[0].ip} ' +kubectl get service --namespace=gitlab-managed-apps ingress-nginx-ingress-controller -o jsonpath='{.status.loadBalancer.ingress[0].hostname}' ``` -Some Kubernetes clusters return a hostname instead, like [Amazon EKS](https://aws.amazon.com/eks/). For these platforms, run: +For Istio/Knative, the command will be different: ```bash -kubectl get service ingress-nginx-ingress-controller -n gitlab-managed-apps -o jsonpath="{.status.loadBalancer.ingress[0].hostname}". +kubectl get svc --namespace=istio-system knative-ingressgateway -o jsonpath='{.status.loadBalancer.ingress[0].ip} ' ``` Otherwise, you can list the IP addresses of all load balancers: @@ -448,13 +427,12 @@ reserved IP. Read how to [promote an ephemeral external IP address in GKE](https://cloud.google.com/compute/docs/ip-addresses/reserve-static-external-ip-address#promote_ephemeral_ip). -### Pointing your DNS at the cluster IP +### Pointing your DNS at the external endpoint -Once you've set up the static IP, you should associate it to a [wildcard DNS -record](https://en.wikipedia.org/wiki/Wildcard_DNS_record), in order to be able -to reach your apps. This heavily depends on your domain provider, but in case -you aren't sure, just create an A record with a wildcard host like -`*.example.com.`. +Once you've set up the external endpoint, you should associate it with a [wildcard DNS +record](https://en.wikipedia.org/wiki/Wildcard_DNS_record) such as `*.example.com.` +in order to be able to reach your apps. If your external endpoint is an IP address, +use an A record. If your external endpoint is a hostname, use a CNAME record. ## Multiple Kubernetes clusters **[PREMIUM]** @@ -465,17 +443,14 @@ project. That way you can have different clusters for different environments, like dev, staging, production, etc. Simply add another cluster, like you did the first time, and make sure to -[set an environment scope](#setting-the-environment-scope) that will +[set an environment scope](#setting-the-environment-scope-premium) that will differentiate the new cluster with the rest. ## Setting the environment scope **[PREMIUM]** -When adding more than one Kubernetes clusters to your project, you need -to differentiate them with an environment scope. The environment scope -associates clusters with [environments](../../../ci/environments.md) -similar to how the [environment-specific -variables](../../../ci/variables/README.md#limiting-environment-scopes-of-variables) -work. +When adding more than one Kubernetes cluster to your project, you need to differentiate +them with an environment scope. The environment scope associates clusters with [environments](../../../ci/environments.md) similar to how the +[environment-specific variables](../../../ci/variables/README.md#limiting-environment-scopes-of-environment-variables-premium) work. The default environment scope is `*`, which means all jobs, regardless of their environment, will use that cluster. Each scope can only be used by a single @@ -527,7 +502,7 @@ The result will then be: ## Deployment variables The Kubernetes cluster integration exposes the following -[deployment variables](../../../ci/variables/README.md#deployment-variables) in the +[deployment variables](../../../ci/variables/README.md#deployment-environment-variables) in the GitLab CI/CD build environment. | Variable | Description | @@ -538,33 +513,34 @@ GitLab CI/CD build environment. | `KUBE_CA_PEM_FILE` | Path to a file containing PEM data. Only present if a custom CA bundle was specified. | | `KUBE_CA_PEM` | (**deprecated**) Raw PEM data. Only if a custom CA bundle was specified. | | `KUBECONFIG` | Path to a file containing `kubeconfig` for this deployment. CA bundle would be embedded if specified. This config also embeds the same token defined in `KUBE_TOKEN` so you likely will only need this variable. This variable name is also automatically picked up by `kubectl` so you won't actually need to reference it explicitly if using `kubectl`. | -| `KUBE_INGRESS_BASE_DOMAIN` | From GitLab 11.8, this variable can be used to set a domain per cluster. See [cluster domains](#base-domain) for more information. | +| `KUBE_INGRESS_BASE_DOMAIN` | From GitLab 11.8, this variable can be used to set a domain per cluster. See [cluster domains](#base-domain) for more information. | NOTE: **NOTE:** Prior to GitLab 11.5, `KUBE_TOKEN` was the Kubernetes token of the main service account of the cluster integration. -### Troubleshooting missing `KUBECONFIG` or `KUBE_TOKEN` +### Troubleshooting failed deployment jobs + +GitLab will create a namespace and service account specifically for your +deployment jobs. On project level clusters, this happens when the cluster +is created. On group level clusters, resources are created immediately +before the deployment job starts. -GitLab will create a new service account specifically for your CI builds. The -new service account is created when the cluster is added to the project. -Sometimes there may be errors that cause the service account creation to fail. +However, sometimes GitLab can not create them. In such instances, your job will fail with the message: -In such instances, your build will not be passed the `KUBECONFIG` or -`KUBE_TOKEN` variables and, if you are using Auto DevOps, your Auto DevOps -pipelines will no longer trigger a `production` deploy build. You will need to -check the [logs](../../../administration/logs.md) to debug why the service -account creation failed. +```text +This job failed because the necessary resources were not successfully created. +``` -A common reason for failure is that the token you gave GitLab did not have -[`cluster-admin`](https://kubernetes.io/docs/reference/access-authn-authz/rbac/#user-facing-roles) -privileges as GitLab expects. +To find the cause of this error when creating a namespace and service account, check the [logs](../../../administration/logs.md#kuberneteslog). -Another common problem for why these variables are not being passed to your -builds is that they must have a matching -[`environment:name`](../../../ci/environments.md#defining-environments). If -your build has no `environment:name` set, it will not be passed the Kubernetes -credentials. +Common reasons for failure include: + +- The token you gave GitLab did not have [`cluster-admin`](https://kubernetes.io/docs/reference/access-authn-authz/rbac/#user-facing-roles) + privileges required by GitLab. +- Missing `KUBECONFIG` or `KUBE_TOKEN` variables. To be passed to your job, they must have a matching + [`environment:name`](../../../ci/environments.md#defining-environments). If your job has no + `environment:name` set, it will not be passed the Kubernetes credentials. ## Monitoring your Kubernetes cluster **[ULTIMATE]** @@ -572,7 +548,7 @@ credentials. When [Prometheus is deployed](#installing-applications), GitLab will automatically monitor the cluster's health. At the top of the cluster settings page, CPU and Memory utilization is displayed, along with the total amount available. Keeping an eye on cluster resources can be important, if the cluster runs out of memory pods may be shutdown or fail to start. - + ## Enabling or disabling the Kubernetes cluster integration @@ -600,11 +576,6 @@ To remove the Kubernetes cluster integration from your project, simply click the **Remove integration** button. You will then be able to follow the procedure and add a Kubernetes cluster again. -## View Kubernetes pod logs from GitLab **[ULTIMATE]** - -Learn how to easily -[view the logs of running pods in connected Kubernetes clusters](https://docs.gitlab.com/ee/user/project/clusters/kubernetes_pod_logs.html). - ## What you can get with the Kubernetes integration Here's what you can do with GitLab if you enable the Kubernetes integration. @@ -617,7 +588,7 @@ displaying the status of the pods in the deployment. Developers and other teammates can view the progress and status of a rollout, pod by pod, in the workflow they already use without any need to access Kubernetes. -[Read more about Deploy Boards](https://docs.gitlab.com/ee/user/project/deploy_boards.html) +[Read more about Deploy Boards](../deploy_boards.md) ### Canary Deployments **[PREMIUM]** @@ -625,7 +596,13 @@ Leverage [Kubernetes' Canary deployments](https://kubernetes.io/docs/concepts/cl and visualize your canary deployments right inside the Deploy Board, without the need to leave GitLab. -[Read more about Canary Deployments](https://docs.gitlab.com/ee/user/project/canary_deployments.html) +[Read more about Canary Deployments](../canary_deployments.md) + +### Pod logs **[ULTIMATE]** + +GitLab makes it easy to view the logs of running pods in connected Kubernetes clusters. By displaying the logs directly in GitLab, developers can avoid having to manage console tools or jump to a different interface. + +[Read more about Kubernetes pod logs](kubernetes_pod_logs.md) ### Kubernetes monitoring diff --git a/doc/user/project/clusters/kubernetes_pod_logs.md b/doc/user/project/clusters/kubernetes_pod_logs.md new file mode 100644 index 00000000000..368031070c1 --- /dev/null +++ b/doc/user/project/clusters/kubernetes_pod_logs.md @@ -0,0 +1,21 @@ +# Kubernetes Pod Logs **[ULTIMATE]** + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/issues/4752) in [GitLab Ultimate](https://about.gitlab.com/pricing/) 11.0. + +GitLab makes it easy to view the logs of running pods in [connected Kubernetes clusters](index.md). +By displaying the logs directly in GitLab, developers can avoid having to manage console tools or jump to a different interface. + +## Overview + +[Kubernetes](https://kubernetes.io) pod logs can be viewed directly within GitLab. Logs can be displayed by clicking on a specific pod from [Deploy Boards](../deploy_boards.md): + +1. Go to **Operations > Environments** and find the environment which contains the desired pod, like `production`. +1. On the **Environments** page, you should see the status of the environment's pods with [Deploy Boards](../deploy_boards.md). +1. When mousing over the list of pods, a tooltip will appear with the exact pod name and status. + +1. Click on the desired pod to bring up the logs view, which will contain the last 500 lines for that pod. Support for pods with multiple containers is coming [in a future release](https://gitlab.com/gitlab-org/gitlab-ee/issues/6502). + + +## Requirements + +[Enabling Deploy Boards](../deploy_boards.md#enabling-deploy-boards) is required in order to be able to use Pod Logs. diff --git a/doc/user/project/clusters/runbooks/img/authorize-jupyter.png b/doc/user/project/clusters/runbooks/img/authorize-jupyter.png Binary files differindex 84cce311483..f3e868de802 100644 --- a/doc/user/project/clusters/runbooks/img/authorize-jupyter.png +++ b/doc/user/project/clusters/runbooks/img/authorize-jupyter.png diff --git a/doc/user/project/clusters/runbooks/img/gitlab-variables.png b/doc/user/project/clusters/runbooks/img/gitlab-variables.png Binary files differindex 1d338f063a9..7c9c9099851 100644 --- a/doc/user/project/clusters/runbooks/img/gitlab-variables.png +++ b/doc/user/project/clusters/runbooks/img/gitlab-variables.png diff --git a/doc/user/project/clusters/runbooks/img/helm-install.png b/doc/user/project/clusters/runbooks/img/helm-install.png Binary files differindex 003e482e756..e67cf317ec1 100644 --- a/doc/user/project/clusters/runbooks/img/helm-install.png +++ b/doc/user/project/clusters/runbooks/img/helm-install.png diff --git a/doc/user/project/clusters/runbooks/img/ingress-install.png b/doc/user/project/clusters/runbooks/img/ingress-install.png Binary files differindex 7edc11d5b45..08256a65138 100644 --- a/doc/user/project/clusters/runbooks/img/ingress-install.png +++ b/doc/user/project/clusters/runbooks/img/ingress-install.png diff --git a/doc/user/project/clusters/runbooks/img/jupyterhub-install.png b/doc/user/project/clusters/runbooks/img/jupyterhub-install.png Binary files differindex 75c6028a763..784e508ff25 100644 --- a/doc/user/project/clusters/runbooks/img/jupyterhub-install.png +++ b/doc/user/project/clusters/runbooks/img/jupyterhub-install.png diff --git a/doc/user/project/clusters/runbooks/img/postgres-query.png b/doc/user/project/clusters/runbooks/img/postgres-query.png Binary files differindex 04315d54d5e..dbb5a62ebba 100644 --- a/doc/user/project/clusters/runbooks/img/postgres-query.png +++ b/doc/user/project/clusters/runbooks/img/postgres-query.png diff --git a/doc/user/project/clusters/runbooks/img/sample-runbook.png b/doc/user/project/clusters/runbooks/img/sample-runbook.png Binary files differindex 70011202bf0..545d34fd4ad 100644 --- a/doc/user/project/clusters/runbooks/img/sample-runbook.png +++ b/doc/user/project/clusters/runbooks/img/sample-runbook.png diff --git a/doc/user/project/clusters/runbooks/index.md b/doc/user/project/clusters/runbooks/index.md index 512a4cb32f4..c67b12fb91a 100644 --- a/doc/user/project/clusters/runbooks/index.md +++ b/doc/user/project/clusters/runbooks/index.md @@ -4,6 +4,10 @@ Runbooks are a collection of documented procedures that explain how to carry out a particular process, be it starting, stopping, debugging, or troubleshooting a particular system. +Using [Jupyter Notebooks](https://jupyter.org/) and the [Rubix library](https://github.com/Nurtch/rubix), +users can get started writing their own executable runbooks. + + ## Overview Historically, runbooks took the form of a decision tree or a detailed @@ -13,13 +17,15 @@ Modern implementations have introduced the concept of an "executable runbooks", where, along with a well-defined process, operators can execute pre-written code blocks or database queries against a given environment. -## Nurtch Executable Runbooks +## Executable Runbooks > [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/issues/45912) in GitLab 11.4. The JupyterHub app offered via GitLab’s Kubernetes integration now ships with Nurtch’s Rubix library, providing a simple way to create DevOps -runbooks. A sample runbook is provided, showcasing common operations. +runbooks. A sample runbook is provided, showcasing common operations. While Rubix makes it +simple to create common Kubernetes and AWS workflows, you can also create them manually without +Rubix. **<i class="fa fa-youtube-play youtube" aria-hidden="true"></i> Watch this [video](https://www.youtube.com/watch?v=Q_OqHIIUPjE) @@ -30,13 +36,13 @@ for an overview of how this is accomplished in GitLab!** To create an executable runbook, you will need: 1. **Kubernetes** - A Kubernetes cluster is required to deploy the rest of the applications. - The simplest way to get started is to add a cluster using [GitLab's GKE integration](https://docs.gitlab.com/ee/user/project/clusters/#adding-and-creating-a-new-gke-cluster-via-gitlab). + The simplest way to get started is to add a cluster using [GitLab's GKE integration](../index.md#adding-and-creating-a-new-gke-cluster-via-gitlab). 1. **Helm Tiller** - Helm is a package manager for Kubernetes and is required to install all the other applications. It is installed in its own pod inside the cluster which can run the helm CLI in a safe environment. 1. **Ingress** - Ingress can provide load balancing, SSL termination, and name-based virtual hosting. It acts as a web proxy for your applications. -1. **JupyterHub** - JupyterHub is a multi-user service for managing notebooks across +1. **JupyterHub** - [JupyterHub](https://jupyterhub.readthedocs.io/) is a multi-user service for managing notebooks across a team. Jupyter Notebooks provide a web-based interactive programming environment used for data analysis, visualization, and machine learning. @@ -55,7 +61,7 @@ the components outlined above and the preloaded demo runbook. ### 1. Add a Kubernetes cluster -Follow the steps outlined in [Adding and creating a new GKE cluster via GitLab](https://docs.gitlab.com/ee/user/project/clusters/#adding-and-creating-a-new-gke-cluster-via-gitlab) +Follow the steps outlined in [Adding and creating a new GKE cluster via GitLab](../index.md#adding-and-creating-a-new-gke-cluster-via-gitlab) to add a Kubernetes cluster to your project. ### 2. Install Helm Tiller, Ingress, and JupyterHub @@ -86,7 +92,7 @@ The server will take a couple of seconds to start. ### 4. Configure access In order for the runbook to access your GitLab project, you will need to enter a -[GitLab Access Token](https://docs.gitlab.com/ee/user/profile/personal_access_tokens.html) +[GitLab Access Token](../../../profile/personal_access_tokens.md) as well as your Project ID in the **Setup** section of the demo runbook. Double-click the **DevOps-Runbook-Demo** folder located on the left panel. diff --git a/doc/user/project/clusters/serverless/img/function-details-loaded.png b/doc/user/project/clusters/serverless/img/function-details-loaded.png Binary files differnew file mode 100644 index 00000000000..34465c5c087 --- /dev/null +++ b/doc/user/project/clusters/serverless/img/function-details-loaded.png diff --git a/doc/user/project/clusters/serverless/index.md b/doc/user/project/clusters/serverless/index.md index 46d50f8894a..2cda850f24e 100644 --- a/doc/user/project/clusters/serverless/index.md +++ b/doc/user/project/clusters/serverless/index.md @@ -11,18 +11,22 @@ Run serverless workloads on Kubernetes using [Knative](https://cloud.google.com/ Knative extends Kubernetes to provide a set of middleware components that are useful to build modern, source-centric, container-based applications. Knative brings some significant benefits out of the box through its main components: -- [Build](https://github.com/knative/build): Source-to-container build orchestration. -- [Eventing](https://github.com/knative/eventing): Management and delivery of events. - [Serving](https://github.com/knative/serving): Request-driven compute that can scale to zero. +- [Eventing](https://github.com/knative/eventing): Management and delivery of events. For more information on Knative, visit the [Knative docs repo](https://github.com/knative/docs). -With GitLab serverless, you can deploy both functions-as-a-service (FaaS) and serverless applications. +With GitLab Serverless, you can deploy both functions-as-a-service (FaaS) and serverless applications. -## Requirements +## Prerequisites To run Knative on Gitlab, you will need: +1. **Existing GitLab project:** You will need a GitLab project to associate all resources. The simplest way to get started: + + - If you are planning on deploying functions, clone the [functions example project](https://gitlab.com/knative-examples/functions) to get started. + - If you are planning on deploying a serverless application, clone the sample [Knative Ruby App](https://gitlab.com/knative-examples/knative-ruby-app) to get started. + 1. **Kubernetes Cluster:** An RBAC-enabled Kubernetes cluster is required to deploy Knative. The simplest way to get started is to add a cluster using [GitLab's GKE integration](../index.md#adding-and-creating-a-new-gke-cluster-via-gitlab). The set of minimum recommended cluster specifications to run Knative is 3 nodes, 6 vCPUs, and 22.50 GB memory. @@ -32,17 +36,22 @@ To run Knative on Gitlab, you will need: applications or functions onto your cluster. You can install the GitLab Runner onto the existing Kubernetes cluster. See [Installing Applications](../index.md#installing-applications) for more information. 1. **Domain Name:** Knative will provide its own load balancer using Istio. It will provide an - external IP address for all the applications served by Knative. You will be prompted to enter a + external IP address or hostname for all the applications served by Knative. You will be prompted to enter a wildcard domain where your applications will be served. Configure your DNS server to use the - external IP address for that domain. + external IP address or hostname for that domain. 1. **`.gitlab-ci.yml`:** GitLab uses [Kaniko](https://github.com/GoogleContainerTools/kaniko) - to build the application and the [TriggerMesh CLI](https://github.com/triggermesh/tm) to simplify the - deployment of knative services and functions. + to build the application. We also use [gitlabktl](https://gitlab.com/gitlab-org/gitlabktl) + and [TriggerMesh CLI](https://github.com/triggermesh/tm) CLIs to simplify the + deployment of services and functions to Knative. 1. **`serverless.yml`** (for [functions only](#deploying-functions)): When using serverless to deploy functions, the `serverless.yml` file will contain the information for all the functions being hosted in the repository as well as a reference to the runtime being used. -1. **`Dockerfile`** (for [applications only](#deploying-serverless-applications): Knative requires a `Dockerfile` in order to build your application. It should be included - at the root of your project's repo and expose port `8080`. +1. **`Dockerfile`** (for [applications only](#deploying-serverless-applications): Knative requires a + `Dockerfile` in order to build your applications. It should be included at the root of your + project's repo and expose port `8080`. `Dockerfile` is not require if you plan to build serverless functions + using our [runtimes](https://gitlab.com/gitlab-org/serverless/runtimes). +1. **Prometheus** (optional): Installing Prometheus allows you to monitor the scale and traffic of your serverless function/application. + See [Installing Applications](../index.md#installing-applications) for more information. ## Installing Knative via GitLab's Kubernetes integration @@ -55,23 +64,18 @@ The minimum recommended cluster size to run Knative is 3-nodes, 6 vCPUs, and 22.  -1. After the Knative installation has finished, you can wait for the IP address to be displayed in the - **Knative IP Address** field or retrieve the Istio Ingress IP address by running the following command: - - ```bash - kubectl get svc --namespace=istio-system knative-ingressgateway -o jsonpath='{.status.loadBalancer.ingress[0].ip} ' - ``` - - Output: +1. After the Knative installation has finished, you can wait for the IP address or hostname to be displayed in the + **Knative Endpoint** field or [retrieve the Istio Ingress Endpoint manually](../#manually-determining-the-external-endpoint). - ```bash - 35.161.143.124 my-machine-name:~ my-user$ - ``` + NOTE: **Note:** + Running `kubectl` commands on your cluster requires setting up access to the cluster first. + For clusters created on GKE, see [GKE Cluster Access](https://cloud.google.com/kubernetes-engine/docs/how-to/cluster-access-for-kubectl), + for other platforms [Install kubectl](https://kubernetes.io/docs/tasks/tools/install-kubectl/). 1. The ingress is now available at this address and will route incoming requests to the proper service based on the DNS name in the request. To support this, a wildcard DNS A record should be created for the desired domain name. For example, - if your Knative base domain is `example.com` then you need to create an A record with domain `*.example.com` - pointing the ip address of the ingress. + if your Knative base domain is `knative.info` then you need to create an A record or CNAME record with domain `*.knative.info` + pointing the ip address or hostname of the ingress.  @@ -87,14 +91,15 @@ Using functions is useful for dealing with independent events without needing to maintain a complex unified infrastructure. This allows you to focus on a single task that can be executed/scaled automatically and independently. -Currently the following [runtimes](https://gitlab.com/triggermesh/runtimes) are offered: +Currently the following [runtimes](https://gitlab.com/gitlab-org/serverless/runtimes) are offered: +- ruby - node.js -- kaniko +- Dockerfile You can find and import all the files referenced in this doc in the **[functions example project](https://gitlab.com/knative-examples/functions)**. -Follow these steps to deploy a function using the Node.js runtime to your Knative instance: +Follow these steps to deploy a function using the Node.js runtime to your Knative instance (you can skip these steps if you've cloned the example project): 1. Create a directory that will house the function. In this example we will create a directory called `echo` at the root of the project. @@ -102,53 +107,60 @@ Follow these steps to deploy a function using the Node.js runtime to your Knativ - Public, continue to the next step. - Private, you will need to [create a GitLab deploy token](../../deploy_tokens/index.md#creating-a-deploy-token) with `gitlab-deploy-token` as the name and the `read_registry` scope. -1. `.gitlab-ci.yml`: This template allows to define the stage, environment, and - image to be used for your functions. It must be included at the root of your repository: +1. `.gitlab-ci.yml`: this defines a pipeline used to deploy your functions. + It must be included at the root of your repository: ```yaml - stages: - - deploy + include: + template: Serverless.gitlab-ci.yml - functions: - stage: deploy - environment: test - image: gcr.io/triggermesh/tm:v0.0.9 - script: - - tm -n "$KUBE_NAMESPACE" set registry-auth gitlab-registry --registry "$CI_REGISTRY" --username "$CI_REGISTRY_USER" --password "$CI_JOB_TOKEN" --push - - tm -n "$KUBE_NAMESPACE" set registry-auth gitlab-registry --registry "$CI_REGISTRY" --username "$CI_DEPLOY_USER" --password "$CI_DEPLOY_PASSWORD" --pull - - tm -n "$KUBE_NAMESPACE" deploy --wait + functions:build: + extends: .serverless:build:functions + environment: production + functions:deploy: + extends: .serverless:deploy:functions + environment: production ``` - The `gitlab-ci.yml` template creates a `Deploy` stage with a `functions` job that invokes the `tm` CLI with the required parameters. + This `.gitlab-ci.yml` creates jobs that invoke some predefined commands to + build and deploy your functions to your cluster. + + `Serverless.gitlab-ci.yml` is a template that allows customization. + You can either import it with `include` parameter and use `extends` to + customize your jobs, or you can inline the entire template by choosing it + from **Apply a template** dropdown when editing the `.gitlab-ci.yml` file through + the user interface. -2. `serverless.yml`: This file contains the metadata for your functions, - such as name, runtime, and environment. It must be included at the root of your repository. The following is a sample `echo` function which shows the required structure for the file. You can find the relevant files for this project in the [functions example project](https://gitlab.com/knative-examples/functions). +2. `serverless.yml`: this file contains the metadata for your functions, + such as name, runtime, and environment. + + It must be included at the root of your repository. + The following is a sample `echo` function which shows the required structure + for the file. + + You can find the relevant files for this project in the [functions example project](https://gitlab.com/knative-examples/functions). ```yaml - service: my-functions - description: "Deploying functions from GitLab using Knative" + service: functions + description: "GitLab Serverless functions using Knative" provider: name: triggermesh - registry-secret: gitlab-registry environment: - FOO: BAR + FOO: value functions: - echo: - handler: echo - runtime: https://gitlab.com/triggermesh/runtimes/raw/master/nodejs.yaml - description: "echo function using node.js runtime" - buildargs: - - DIRECTORY=echo + echo-js: + handler: echo-js + source: ./echo-js + runtime: https://gitlab.com/gitlab-org/serverless/runtimes/nodejs + description: "node.js runtime function" environment: - FUNCTION: echo + MY_FUNCTION: echo-js ``` -The `serverless.yml` file references both an `echo` directory (under `buildargs`) and an `echo` file (under `handler`), -which is a reference to `echo.js` in the [repository](https://gitlab.com/knative-examples/functions). Additionally, it -contains three sections with distinct parameters: +Explanation of the fields used above: ### `service` @@ -162,7 +174,6 @@ contains three sections with distinct parameters: | Parameter | Description | |-----------|-------------| | `name` | Indicates which provider is used to execute the `serverless.yml` file. In this case, the TriggerMesh `tm` CLI. | -| `registry-secret` | Indicates which registry will be used to store docker images. The sample function is using the GitLab Registry (`gitlab-registry`). A different registry host may be specified using `registry` key in the `provider` object. If changing the default, update the permission and the secret value on the `gitlab-ci.yml` file | | `environment` | Includes the environment variables to be passed as part of function execution for **all** functions in the file, where `FOO` is the variable name and `BAR` are he variable contents. You may replace this with you own variables. | ### `functions` @@ -171,10 +182,10 @@ In the `serverless.yml` example above, the function name is `echo` and the subse | Parameter | Description | |-----------|-------------| -| `handler` | The function's file name. In the example above, both the function name and the handler are the same. | +| `handler` | The function's name. | +| `source` | Directory with sources of a functions. | | `runtime` | The runtime to be used to execute the function. | | `description` | A short description of the function. | -| `buildargs` | Pointer to the function file in the repo. In the sample the function is located in the `echo` directory. | | `environment` | Sets an environment variable for the specific function only. | After the `gitlab-ci.yml` template has been added and the `serverless.yml` file has been @@ -188,7 +199,7 @@ appear under **Operations > Serverless**. This page contains all functions available for the project, the description for accessing the function, and, if available, the function's runtime information. The details are derived from the Knative installation inside each of the project's -Kubernetes cluster. +Kubernetes cluster. Click on each function to obtain detailed scale and invocation data. The function details can be retrieved directly from Knative on the cluster: @@ -198,7 +209,7 @@ kubectl -n "$KUBE_NAMESPACE" get services.serving.knative.dev The sample function can now be triggered from any HTTP client using a simple `POST` call: - 1. Using curl + 1. Using curl (replace the URL on the last line with the URL of your application): ```bash curl \ @@ -219,35 +230,25 @@ NOTE: **Note:** You can reference and import the sample [Knative Ruby App](https://gitlab.com/knative-examples/knative-ruby-app) to get started. Add the following `.gitlab-ci.yml` to the root of your repository -(you may skip this step if using the sample [Knative Ruby App](https://gitlab.com/knative-examples/knative-ruby-app) mentioned above): +(you may skip this step if you've previously cloned the sample [Knative Ruby App](https://gitlab.com/knative-examples/knative-ruby-app) mentioned above): ```yaml -stages: - - build - - deploy +include: + template: Serverless.gitlab-ci.yml build: - stage: build - image: - name: gcr.io/kaniko-project/executor:debug - entrypoint: [""] - only: - - master - script: - - echo "{\"auths\":{\"$CI_REGISTRY\":{\"username\":\"$CI_REGISTRY_USER\",\"password\":\"$CI_REGISTRY_PASSWORD\"}}}" > /kaniko/.docker/config.json - - /kaniko/executor --context $CI_PROJECT_DIR --dockerfile $CI_PROJECT_DIR/Dockerfile --destination $CI_REGISTRY_IMAGE + extends: .serverless:build:image deploy: - stage: deploy - image: gcr.io/triggermesh/tm@sha256:e3ee74db94d215bd297738d93577481f3e4db38013326c90d57f873df7ab41d5 - only: - - master - environment: production - script: - - echo "$CI_REGISTRY_IMAGE" - - tm -n "$KUBE_NAMESPACE" --config "$KUBECONFIG" deploy service "$CI_PROJECT_NAME" --from-image "$CI_REGISTRY_IMAGE" --wait + extends: .serverless:deploy:image ``` +`Serverless.gitlab-ci.yml` is a template that allows customization. +You can either import it with `include` parameter and use `extends` to +customize your jobs, or you can inline the entire template by choosing it +from **Apply a template** dropdown when editing the `.gitlab-ci.yml` file through +the user interface. + ### Deploy the application with Knative With all the pieces in place, the next time a CI pipeline runs, the Knative application will be deployed. Navigate to @@ -285,3 +286,23 @@ The second to last line, labeled **Service domain** contains the URL for the dep browser to see the app live.  + +## Function details + +Go to the **Operations > Serverless** page and click on one of the function +rows to bring up the function details page. + + + +The pod count will give you the number of pods running the serverless function instances on a given cluster. + +### Prometheus support + +For the Knative function invocations to appear, +[Prometheus must be installed](../index.md#installing-applications). + +Once Prometheus is installed, a message may appear indicating that the metrics data _is +loading or is not available at this time._ It will appear upon the first access of the +page, but should go away after a few seconds. If the message does not disappear, then it +is possible that GitLab is unable to connect to the Prometheus instance running on the +cluster. diff --git a/doc/user/project/code_owners.md b/doc/user/project/code_owners.md new file mode 100644 index 00000000000..ae04616943f --- /dev/null +++ b/doc/user/project/code_owners.md @@ -0,0 +1,85 @@ +# Code Owners **[STARTER]** + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/6916) +in [GitLab Starter](https://about.gitlab.com/pricing/) 11.3. + +You can use a `CODEOWNERS` file to specify users that are responsible +for certain files in a repository. + +You can choose and add the `CODEOWNERS` file in three places: + +- to the root directory of the repository +- inside the `.gitlab/` directory +- inside the `docs/` directory + +The `CODEOWNERS` file is scoped to a branch, which means that with the +introduction of new files, the person adding the new content can +specify themselves as a code owner, all before the new changes +get merged to the default branch. + +When a file matches multiple entries in the `CODEOWNERS` file, +the users from all entries are displayed on the blob page of +the given file. + +## The syntax of Code Owners files + +Files can be specified using the same kind of patterns you would use +in the `.gitignore` file followed by the `@username` or email of one +or more users that should be owners of the file. + +The order in which the paths are defined is significant: the last +pattern that matches a given path will be used to find the code +owners. + +Starting a line with a `#` indicates a comment. This needs to be +escaped using `\#` to address files for which the name starts with a +`#`. + +Example `CODEOWNERS` file: + +``` +# This is an example code owners file, lines starting with a `#` will +# be ignored. + +# app/ @commented-rule + +# We can specify a default match using wildcards: +* @default-codeowner + +# Rules defined later in the file take precedence over the rules +# defined before. +# This will match all files for which the file name ends in `.rb` +*.rb @ruby-owner + +# Files with a `#` can still be accesssed by escaping the pound sign +\#file_with_pound.rb @owner-file-with-pound + +# Multiple codeowners can be specified, separated by whitespace +CODEOWNERS @multiple @owners @tab-separated + +# Both usernames or email addresses can be used to match +# users. Everything else will be ignored. For example this will +# specify `@legal` and a user with email `janedoe@gitlab.com` as the +# owner for the LICENSE file +LICENSE @legal this_does_not_match janedoe@gitlab.com + +# Ending a path in a `/` will specify the code owners for every file +# nested in that directory, on any level +/docs/ @all-docs + +# Ending a path in `/*` will specify code owners for every file in +# that directory, but not nested deeper. This will match +# `docs/index.md` but not `docs/projects/index.md` +/docs/* @root-docs + +# This will make a `lib` directory nested anywhere in the repository +# match +lib/ @lib-owner + +# This will only match a `config` directory in the root of the +# repository +/config/ @config-owner + +# If the path contains spaces, these need to be escaped like this: +path\ with\ spaces/ @space-owner +``` diff --git a/doc/user/project/container_registry.md b/doc/user/project/container_registry.md index dec6eac2508..58b7fe33906 100644 --- a/doc/user/project/container_registry.md +++ b/doc/user/project/container_registry.md @@ -1,7 +1,8 @@ # GitLab Container Registry > **Notes:** -> [Introduced][ce-4040] in GitLab 8.8. +> +> - [Introduced][ce-4040] in GitLab 8.8. > - Docker Registry manifest `v1` support was added in GitLab 8.9 to support Docker > versions earlier than 1.10. > - This document is about the user guide. To learn how to enable GitLab Container @@ -10,7 +11,7 @@ > - Starting from GitLab 8.12, if you have 2FA enabled in your account, you need > to pass a [personal access token][pat] instead of your password in order to > login to GitLab's Container Registry. -> - Multiple level image names support was added in GitLab 9.1 +> - Multiple level image names support was added in GitLab 9.1. With the Docker Container Registry integrated into GitLab, every project can have its own space to store its Docker images. @@ -41,6 +42,7 @@ to enable it. ## Build and push images > **Notes:** +> > - Moving or renaming existing container registry repositories is not supported > once you have pushed images because the images are signed, and the > signature includes the repository name. @@ -51,7 +53,7 @@ If you visit the **Registry** link under your project's menu, you can see the explicit instructions to login to the Container Registry using your GitLab credentials. -For example if the Registry's URL is `registry.example.com`, the you should be +For example if the Registry's URL is `registry.example.com`, then you should be able to login with: ``` @@ -202,7 +204,7 @@ at the communication between the client and the Registry. The REST API between the Docker client and Registry is [described here](https://docs.docker.com/registry/spec/api/). Normally, one would just use Wireshark or tcpdump to capture the traffic and see where things went -wrong. However, since all communication between Docker clients and servers +wrong. However, since all communications between Docker clients and servers are done over HTTPS, it's a bit difficult to decrypt the traffic quickly even if you know the private key. What can we do instead? diff --git a/doc/user/project/cycle_analytics.md b/doc/user/project/cycle_analytics.md index 05127b274e0..19ab911e39f 100644 --- a/doc/user/project/cycle_analytics.md +++ b/doc/user/project/cycle_analytics.md @@ -3,45 +3,40 @@ > [Introduced][ce-5986] in GitLab 8.12. Further features were added in GitLab 8.14. -Cycle Analytics measures the time it takes to go from an [idea to production] for -each project you have. This is achieved by not only indicating the total time it -takes to reach that point, but the total time is broken down into the -multiple stages an idea has to pass through to be shipped. +Cycle Analytics measures the time spent to go from an [idea to production] for +each of your projects. Cycle Analytics displays the median time for an idea to +reach production, along with the time typically spent in each DevOps stage along the way. + +Cycle Analytics is useful in order to quickly determine the velocity of a given +project. It points to bottlenecks in the development process, enabling management +to uncover, triage, and root-cause slowdowns in the software development life cycle. Cycle Analytics is tightly coupled with the [GitLab flow] and calculates a separate median for each stage. ## Overview -You can find the Cycle Analytics page under your project's **Pipelines ➔ Cycle +You can find the Cycle Analytics page under your project's **Project ➔ Cycle Analytics** tab.  -You can see that there are seven stages in total: +There are seven stages that are tracked as part of the Cycle Analytics calculations. - **Issue** (Tracker) - - Median time from issue creation until given a milestone or list label - (first assignment, any milestone, milestone date or assignee is not required) + - Time to schedule an issue (by milestone or by adding it to an issue board) - **Plan** (Board) - - Median time from giving an issue a milestone or label until pushing the - first commit to the branch + - Time to first commit - **Code** (IDE) - - Median time from the first commit to the branch until the merge request is - created + - Time to create a merge request - **Test** (CI) - - Median total test time for all commits/merges + - Time it takes GitLab CI/CD to test your code - **Review** (Merge Request/MR) - - Median time from merge request creation until the merge request is merged - (closed merge requests won't be taken into account) + - Time spent on code review - **Staging** (Continuous Deployment) - - Median time from when the merge request got merged until the deploy to - production (production is last stage/environment) + - Time between merging and deploying to production - **Production** (Total) - - Sum of all the above stages' times excluding the Test (CI) time. To clarify, - it's not so much that CI time is "excluded", but rather CI time is already - counted in the review stage since CI is done automatically. Most of the - other stages are purely sequential, but **Test** is not. + - Total lifecycle time; i.e. the velocity of the project or team ## How the data is measured diff --git a/doc/user/project/deploy_boards.md b/doc/user/project/deploy_boards.md new file mode 100644 index 00000000000..2f2a9c5eec3 --- /dev/null +++ b/doc/user/project/deploy_boards.md @@ -0,0 +1,132 @@ +# Deploy Boards **[PREMIUM]** + +> [Introduced][ee-1589] in [GitLab Premium][ee] 9.0. + +GitLab's Deploy Boards offer a consolidated view of the current health and +status of each CI [environment] running on [Kubernetes], displaying the status +of the pods in the deployment. Developers and other teammates can view the +progress and status of a rollout, pod by pod, in the workflow they already use +without any need to access Kubernetes. + +## Overview + +With Deploy Boards you can gain more insight into deploys with benefits such as: + +- Following a deploy from the start, not just when it's done +- Watching the rollout of a build across multiple servers +- Finer state detail (Waiting, Deploying, Finished, Unknown) +- See [Canary Deployments](canary_deployments.md) + +Here's an example of a Deploy Board of the production environment. + + + +The squares represent pods in your Kubernetes cluster that are associated with +the given environment. Hovering above each square you can see the state of a +deploy rolling out. The percentage is the percent of the pods that are updated +to the latest release. + +Since Deploy Boards are tightly coupled with Kubernetes, there is some required +knowledge. In particular you should be familiar with: + +- [Kubernetes pods](https://kubernetes.io/docs/user-guide/pods) +- [Kubernetes labels](https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/) +- [Kubernetes namespaces](https://kubernetes.io/docs/user-guide/namespaces/) +- [Kubernetes canary deployments](https://kubernetes.io/docs/concepts/cluster-administration/manage-deployment/#canary-deployments) + +## Use cases + +Since the Deploy Board is a visual representation of the Kubernetes pods for a +specific environment, there are lot of uses cases. To name a few: + +- You want to promote what's running in staging, to production. You go to the + environments list, verify that what's running in staging is what you think is + running, then click on the [manual action](../../ci/yaml/README.md#whenmanual) to deploy to production. +- You trigger a deploy, and you've got lots of containers to upgrade so you know + it'll take a while (you've also throttled your deploy to only take down X + containers at a time). But you need to tell someone when it's deployed, so you + go to the environments list, look at the production environment to see what + the progress is in real-time as each pod is rolled. +- You get a report that something is weird in production, so you look at the + production environment to see what is running, and if a deploy is ongoing or + stuck or failed. +- You've got an MR that looks good, but you want to run it on staging because + staging is set up in some way closer to production. You go to the environment + list, find the [Review App][review apps] you're interested in, and click the + manual action to deploy it to staging. + +## Enabling Deploy Boards + +To display the Deploy Boards for a specific [environment] you should: + +1. Have a Kubernetes cluster up and running. + + NOTE: **Running on OpenShift:** + If you are using OpenShift, ensure that you're using the `Deployment` resource + instead of `DeploymentConfiguration`, otherwise the Deploy Boards won't render + correctly. For more information, read the + [OpenShift docs](https://docs.openshift.com/container-platform/3.7/dev_guide/deployments/kubernetes_deployments.html#kubernetes-deployments-vs-deployment-configurations) + and [GitLab issue #4584](https://gitlab.com/gitlab-org/gitlab-ee/issues/4584). + +1. [Configure GitLab Runner][runners] with the [Docker][docker-exec] or + [Kubernetes][kube-exec] executor. +1. Configure the [Kubernetes service][kube-service] in your project for the + cluster. The Kubernetes namespace is of particular note as you will need it + for your deployment scripts (exposed by the `KUBE_NAMESPACE` env variable). +1. Ensure Kubernetes annotations of `app.gitlab.com/env: $CI_ENVIRONMENT_SLUG` + and `app.gitlab.com/app: $CI_PROJECT_PATH_SLUG` are applied to the + deployments, replica sets, and pods, where `$CI_ENVIRONMENT_SLUG` and + `$CI_PROJECT_PATH_SLUG` are the values of the CI variables. This is so we can + lookup the proper environment in a cluster/namespace which may have more + than one. These resources should be contained in the namespace defined in + the Kubernetes service setting. You can use an [Autodeploy] `.gitlab-ci.yml` + template which has predefined stages and commands to use, and automatically + applies the annotations. Each project will need to have a unique namespace in + Kubernetes as well. The image below demonstrates how this is shown inside + Kubernetes. + + NOTE: **Note:** + The Kubernetes label of `app` is deprecated and may be removed in next major + release, GitLab 12.0. + +  + +Once all of the above are set up and the pipeline has run at least once, +navigate to the environments page under **Operations > Environments**. + +Deploy Boards are visible by default. You can explicitly click +the triangle next to their respective environment name in order to hide them. +GitLab will then query Kubernetes for the state of each pod (e.g., waiting, +deploying, finished, unknown), and the Deploy Board status will finally appear. + +GitLab will only display a Deploy Board for top-level environments. Foldered +environments like `review/*` (usually used for [Review Apps]) won't have a +Deploy Board attached to them. + +## Canary Deployments + +A popular CI strategy, where a small portion of the fleet is updated to the new +version of your application. + +[Read more about Canary Deployments.](canary_deployments.md) + +## Further reading + +- [GitLab Autodeploy][autodeploy] +- [GitLab CI environment variables][variables] +- [Environments and deployments][environment] +- [Kubernetes deploy example][kube-deploy] + +[ee-1589]: https://gitlab.com/gitlab-org/gitlab-ee/issues/1589 "Deploy Boards initial issue" +[ee]: https://about.gitlab.com/pricing/ "GitLab Enterprise Edition landing page" +[kube-deploy]: https://gitlab.com/gitlab-examples/kubernetes-deploy "Kubernetes deploy example project" +[kubernetes]: https://kubernetes.io "Kubernetes website" +[environment]: ../../ci/environments.md "Environments and deployments documentation" +[docker-exec]: https://docs.gitlab.com/runner/executors/docker.html "GitLab Runner Docker executor" +[kube-exec]: https://docs.gitlab.com/runner/executors/kubernetes.html "GitLab Runner Kubernetes executor" +[kube-service]: integrations/kubernetes.md "Kubernetes project service" +[review apps]: ../../ci/review_apps/index.md "Review Apps documentation" +[variables]: ../../ci/variables/README.md "GitLab CI variables" +[autodeploy]: ../../ci/autodeploy/index.md "GitLab Autodeploy" +[kube-image]: https://gitlab.com/gitlab-examples/kubernetes-deploy/container_registry "Kubernetes deploy Container Registry" +[runners]: ../../ci/runners/README.md diff --git a/doc/user/project/deploy_tokens/index.md b/doc/user/project/deploy_tokens/index.md index 7688508c6ac..92a29b68a22 100644 --- a/doc/user/project/deploy_tokens/index.md +++ b/doc/user/project/deploy_tokens/index.md @@ -5,7 +5,7 @@ Deploy tokens allow to download (through `git clone`), or read the container registry images of a project without the need of having a user and a password. Please note, that the expiration of deploy tokens happens on the date you define, -at midnight UTC and that they can be only managed by [maintainers](https://docs.gitlab.com/ee/user/permissions.html). +at midnight UTC and that they can be only managed by [maintainers](../../permissions.md). ## Creating a Deploy Token diff --git a/doc/user/project/description_templates.md b/doc/user/project/description_templates.md index 08647caaf07..05ad15476ab 100644 --- a/doc/user/project/description_templates.md +++ b/doc/user/project/description_templates.md @@ -2,8 +2,12 @@ >[Introduced][ce-4981] in GitLab 8.11. +We all know that a properly submitted issue is more likely to be addressed in +a timely manner by the developers of a project. + Description templates allow you to define context-specific templates for issue -and merge request description fields for your project. +and merge request description fields for your project, as well as help filter +out a lot of unnecessary noise from issues. ## Overview @@ -18,6 +22,18 @@ Description templates must be written in [Markdown](../markdown.md) and stored in your project's repository under a directory named `.gitlab`. Only the templates of the default branch will be taken into account. +## Use-cases + +- Add a template to be used in every issue for a specific project, +giving instructions and guidelines, requiring for information specific to that subject. +For example, if you have a project for tracking new blog posts, you can require the +title, outlines, author name, author social media information, etc. +- Following the previous example, you can make a template for every MR submitted +with a new blog post, requiring information about the post date, frontmatter data, +images guidelines, link to the related issue, reviewer name, etc. +- You can also create issues and merge request templates for different +stages of your workflow, e.g., feature proposal, feature improvement, bug report, etc. + ## Creating issue templates Create a new Markdown (`.md`) file inside the `.gitlab/issue_templates/` @@ -39,12 +55,38 @@ changes you made after picking the template and return it to its initial status.  +## Setting a default template for issues and merge requests **[STARTER]** + +> **Notes:** +> - This feature was introduced before [description templates](#overview) and is +> available in [GitLab Starter][products]. It can be enabled +> in the project's settings. +> - Templates for issues were [introduced][ee-28] in GitLab EE 8.1. +> - Templates for merge requests were [introduced][ee-7478ece] in GitLab EE 6.9. + +The visibility of issues and/or merge requests should be set to either "Everyone +with access" or "Only team members" in your project's **Settings** otherwise the +template text areas won't show. This is the default behavior so in most cases +you should be fine. + +Go to your project's **Settings** and fill in the "Default description template +for issues" and "Default description template for merge requests" text areas +for issues and merge requests respectively. Since GitLab issues and merge +request support [Markdown](../markdown.md), you can use special markup like +headings, lists, etc. + + + +After you add the description, hit **Save changes** for the settings to take +effect. Now, every time a new issue or merge request is created, it will be +pre-filled with the text you entered in the template(s). + ## Description template example We make use of Description Templates for Issues and Merge Requests within the GitLab Community Edition project. Please refer to the [`.gitlab` folder][gitlab-ce-templates] for some examples. > **Tip:** -It is possible to use [quick actions](./quick_actions.md) within description templates to quickly add labels, assignees, and milestones. The quick actions will only be executed if the user submitting the Issue or Merge Request has the permissions perform the relevant actions. +It is possible to use [quick actions](quick_actions.md) within description templates to quickly add labels, assignees, and milestones. The quick actions will only be executed if the user submitting the Issue or Merge Request has the permissions perform the relevant actions. Here is an example for a Bug report template: @@ -93,3 +135,6 @@ Possible fixes [ce-4981]: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/4981 [gitlab-ce-templates]: https://gitlab.com/gitlab-org/gitlab-ce/tree/master/.gitlab +[ee-28]: https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/28 "Merge Request for adding issues template" +[ee-7478ece]: https://gitlab.com/gitlab-org/gitlab-ee/commit/7478ece8b48e80782b5465b96c79f85cc91d391b "Commit that introduced merge requests templates" +[products]: https://about.gitlab.com/pricing/ diff --git a/doc/user/project/file_lock.md b/doc/user/project/file_lock.md new file mode 100644 index 00000000000..3386eb9d0d4 --- /dev/null +++ b/doc/user/project/file_lock.md @@ -0,0 +1,101 @@ +# File Locking **[PREMIUM]** + +> - [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/440) in [GitLab Premium](https://about.gitlab.com/pricing/) 8.9. +> - This feature needs to have a license with the "File Lock" option enabled. +> - If you are using Premium but you don't see the "Lock" button, ask your GitLab administrator. + +File Locking helps you avoid merge conflicts and better manage your binary files. +Lock any file or directory, make your changes, and then unlock it so another +member of the team can edit it. + +## Overview + +Working with multiple people on the same file can be a risk. Conflicts +when merging a non-text file are hard to overcome and will require a lot +of manual work to resolve. With GitLab Premium, File +Locking helps you avoid merge conflicts and better manage your binary +files by preventing everyone, except you, from modifying a specific file +or entire directory. + +## Use-cases + +The file locking feature is useful in situations when: + +- Multiple people are working on the same file and you want to avoid merge + conflicts. +- Your repository contains binary files in which situation there is no easy + way to tell the diff between yours and your colleagues' changes. +- Prevent design assets from being overwritten. + +Locked directories are locked recursively, which means that everything that +lies under them is also locked. + +## Permissions on file locking + +The user that locks a file or directory **is the only one** that can edit and +push their changes back to the repository where the locked objects are located. + +Locks can be created by any person who has [push access](../../user/permissions.md) to the repository; i.e., +Developer and higher level, and can be removed solely by their author and any +user with Maintainer permissions and above. + +If a file is locked and you are not the author of its locked state, a +pre-receive hook will reject your changes when you try to push. In the +following example, a user who has no permissions on the locked `.gitignore` +file will see the message below: + +```bash +Counting objects: 3, done. +Delta compression using up to 4 threads. +Compressing objects: 100% (3/3), done. +Writing objects: 100% (3/3), 320 bytes | 0 bytes/s, done. +Total 3 (delta 1), reused 0 (delta 0) +remote: GitLab: The path '.gitignore' is locked by Administrator +To https://example.com/gitlab-org/gitlab-ce.git + ! [remote rejected] master -> master (pre-receive hook declined) + error: failed to push some refs to 'https://example.com/gitlab-org/gitlab-ce.git' +``` + +Similarly, when a user that is not the author of the locked state of a file +accepts a merge request, an error message will appear stating that the file +is locked. + + + +## Locking a file or a directory + +>**Note:** +Locking only works for the default branch you have set in the project's settings +(usually `master`). + +To lock a file, navigate to the repository tree under the **Repository > Files** tab, +pick the file you want to lock and hit the "Lock" button. + + + +--- + +To lock an entire directory, look for the "Lock" link next to "History". + + + +--- + +After you lock a file or directory, it will appear as locked in the repository +view. + + + +## Unlocking a file or a directory + +To unlock a file or a directory, follow the same procedure as when you locked +them. For a detailed view of every existing lock, see the next section on +"Viewing and managing existing locks". + +## Viewing and managing existing locks + +To view or manage every existing lock, navigate to the +**Project > Repository > Locked Files** area. There, you can view all existing +locks and [remove the ones you have permission for](#permissions-on-file-locking). + + diff --git a/doc/user/project/gpg_signed_commits/index.md b/doc/user/project/gpg_signed_commits/index.md index 261eedeb412..bd9a5313489 100644 --- a/doc/user/project/gpg_signed_commits/index.md +++ b/doc/user/project/gpg_signed_commits/index.md @@ -1 +1,5 @@ +--- +redirect_to: '../repository/gpg_signed_commits/index.md' +--- + This document was moved to [another location](../repository/gpg_signed_commits/index.md). diff --git a/doc/user/project/img/cycle_analytics_landing_page.png b/doc/user/project/img/cycle_analytics_landing_page.png Binary files differindex 8b17fae5e05..cf46098b9a4 100644 --- a/doc/user/project/img/cycle_analytics_landing_page.png +++ b/doc/user/project/img/cycle_analytics_landing_page.png diff --git a/doc/user/project/img/deploy_boards_canary_deployments.png b/doc/user/project/img/deploy_boards_canary_deployments.png Binary files differnew file mode 100644 index 00000000000..037b4a908ab --- /dev/null +++ b/doc/user/project/img/deploy_boards_canary_deployments.png diff --git a/doc/user/project/img/deploy_boards_kubernetes_label.png b/doc/user/project/img/deploy_boards_kubernetes_label.png Binary files differnew file mode 100644 index 00000000000..19785c74d07 --- /dev/null +++ b/doc/user/project/img/deploy_boards_kubernetes_label.png diff --git a/doc/user/project/img/deploy_boards_landing_page.png b/doc/user/project/img/deploy_boards_landing_page.png Binary files differnew file mode 100644 index 00000000000..c9621a06860 --- /dev/null +++ b/doc/user/project/img/deploy_boards_landing_page.png diff --git a/doc/user/project/img/description_templates_default_settings.png b/doc/user/project/img/description_templates_default_settings.png Binary files differnew file mode 100644 index 00000000000..ab314e83d06 --- /dev/null +++ b/doc/user/project/img/description_templates_default_settings.png diff --git a/doc/user/project/img/file_lock.png b/doc/user/project/img/file_lock.png Binary files differnew file mode 100644 index 00000000000..33f96f20e91 --- /dev/null +++ b/doc/user/project/img/file_lock.png diff --git a/doc/user/project/img/file_lock_folders.png b/doc/user/project/img/file_lock_folders.png Binary files differnew file mode 100644 index 00000000000..5ff3d4d9dbd --- /dev/null +++ b/doc/user/project/img/file_lock_folders.png diff --git a/doc/user/project/img/file_lock_list.png b/doc/user/project/img/file_lock_list.png Binary files differnew file mode 100644 index 00000000000..2c276335c83 --- /dev/null +++ b/doc/user/project/img/file_lock_list.png diff --git a/doc/user/project/img/file_lock_merge_request_error_message.png b/doc/user/project/img/file_lock_merge_request_error_message.png Binary files differnew file mode 100644 index 00000000000..65bc3692da0 --- /dev/null +++ b/doc/user/project/img/file_lock_merge_request_error_message.png diff --git a/doc/user/project/img/file_lock_repository_view.png b/doc/user/project/img/file_lock_repository_view.png Binary files differnew file mode 100644 index 00000000000..8f4ef52aacd --- /dev/null +++ b/doc/user/project/img/file_lock_repository_view.png diff --git a/doc/user/project/img/issue_board.png b/doc/user/project/img/issue_board.png Binary files differdeleted file mode 100644 index b753593d212..00000000000 --- a/doc/user/project/img/issue_board.png +++ /dev/null diff --git a/doc/user/project/img/issue_boards_multiple.png b/doc/user/project/img/issue_boards_multiple.png Binary files differindex 242460925f7..4b1a8356dc9 100644 --- a/doc/user/project/img/issue_boards_multiple.png +++ b/doc/user/project/img/issue_boards_multiple.png diff --git a/doc/user/project/img/key_value_labels.png b/doc/user/project/img/key_value_labels.png Binary files differnew file mode 100644 index 00000000000..bec901f127f --- /dev/null +++ b/doc/user/project/img/key_value_labels.png diff --git a/doc/user/project/img/labels_default.png b/doc/user/project/img/labels_default.png Binary files differindex 7a7fab611a4..221c5cf55a2 100644 --- a/doc/user/project/img/labels_default.png +++ b/doc/user/project/img/labels_default.png diff --git a/doc/user/project/img/labels_epic_sidebar.png b/doc/user/project/img/labels_epic_sidebar.png Binary files differnew file mode 100644 index 00000000000..f8162d89e9d --- /dev/null +++ b/doc/user/project/img/labels_epic_sidebar.png diff --git a/doc/user/project/img/labels_list.png b/doc/user/project/img/labels_list.png Binary files differindex 6878349fc0c..6940dde68bf 100644 --- a/doc/user/project/img/labels_list.png +++ b/doc/user/project/img/labels_list.png diff --git a/doc/user/project/img/labels_sidebar_inline.png b/doc/user/project/img/labels_sidebar_inline.png Binary files differdeleted file mode 100644 index 2186f14ea92..00000000000 --- a/doc/user/project/img/labels_sidebar_inline.png +++ /dev/null diff --git a/doc/user/project/img/priority_sort_order.png b/doc/user/project/img/priority_sort_order.png Binary files differdeleted file mode 100644 index cd1dd8237c0..00000000000 --- a/doc/user/project/img/priority_sort_order.png +++ /dev/null diff --git a/doc/user/project/img/protected_branches_choose_branch.png b/doc/user/project/img/protected_branches_choose_branch.png Binary files differdeleted file mode 100644 index c2848db9c96..00000000000 --- a/doc/user/project/img/protected_branches_choose_branch.png +++ /dev/null diff --git a/doc/user/project/img/protected_branches_error_ui.png b/doc/user/project/img/protected_branches_error_ui.png Binary files differdeleted file mode 100644 index 62839e49d89..00000000000 --- a/doc/user/project/img/protected_branches_error_ui.png +++ /dev/null diff --git a/doc/user/project/img/protected_branches_select_roles_and_users.png b/doc/user/project/img/protected_branches_select_roles_and_users.png Binary files differnew file mode 100644 index 00000000000..4f62b057cc7 --- /dev/null +++ b/doc/user/project/img/protected_branches_select_roles_and_users.png diff --git a/doc/user/project/img/protected_branches_select_roles_and_users_list.png b/doc/user/project/img/protected_branches_select_roles_and_users_list.png Binary files differnew file mode 100644 index 00000000000..519f2267496 --- /dev/null +++ b/doc/user/project/img/protected_branches_select_roles_and_users_list.png diff --git a/doc/user/project/img/service_desk_confirmation_email.png b/doc/user/project/img/service_desk_confirmation_email.png Binary files differnew file mode 100644 index 00000000000..e08dced1e38 --- /dev/null +++ b/doc/user/project/img/service_desk_confirmation_email.png diff --git a/doc/user/project/img/service_desk_disabled.png b/doc/user/project/img/service_desk_disabled.png Binary files differnew file mode 100644 index 00000000000..3ae64dcbe96 --- /dev/null +++ b/doc/user/project/img/service_desk_disabled.png diff --git a/doc/user/project/img/service_desk_enabled.png b/doc/user/project/img/service_desk_enabled.png Binary files differnew file mode 100644 index 00000000000..329348e4b52 --- /dev/null +++ b/doc/user/project/img/service_desk_enabled.png diff --git a/doc/user/project/img/service_desk_issue_tracker.png b/doc/user/project/img/service_desk_issue_tracker.png Binary files differnew file mode 100644 index 00000000000..485751fab0a --- /dev/null +++ b/doc/user/project/img/service_desk_issue_tracker.png diff --git a/doc/user/project/img/service_desk_nav_item.png b/doc/user/project/img/service_desk_nav_item.png Binary files differnew file mode 100644 index 00000000000..3420e355f67 --- /dev/null +++ b/doc/user/project/img/service_desk_nav_item.png diff --git a/doc/user/project/img/service_desk_reply.png b/doc/user/project/img/service_desk_reply.png Binary files differnew file mode 100644 index 00000000000..ae6ce31713b --- /dev/null +++ b/doc/user/project/img/service_desk_reply.png diff --git a/doc/user/project/img/service_desk_thread.png b/doc/user/project/img/service_desk_thread.png Binary files differnew file mode 100644 index 00000000000..8d8ac39cc5b --- /dev/null +++ b/doc/user/project/img/service_desk_thread.png diff --git a/doc/user/project/import/gemnasium.md b/doc/user/project/import/gemnasium.md new file mode 100644 index 00000000000..7f79ebf6353 --- /dev/null +++ b/doc/user/project/import/gemnasium.md @@ -0,0 +1,102 @@ +# Gemnasium **[ULTIMATE]** + +This guide describes how to migrate from Gemnasium.com to your own GitLab +instance or GitLab.com. + +## Why is Gemnasium.com closed? + +Gemnasium has been [acquired by GitLab](https://about.gitlab.com/press/releases/2018-01-30-gemnasium-acquisition.html) +in January 2018. Since May 15, 2018, the services provided by Gemnasium are no longer available. +The team behind Gemnasium has joined GitLab as the new Security Products team +and is working on a wider range of tools than just Dependency Scanning: +[SAST](../../application_security/sast/index.md), +[DAST](../../application_security/dast/index.md), +[Container Scanning](../../application_security/container_scanning/index.md) and more. +If you want to continue monitoring your dependencies, see the +[Migrating to GitLab](#migrating-to-gitlab) section below. + +## What happened to my account? + +Your account has been automatically closed on May 15th, 2018. If you had a paid +subscription at that time, your card will be refunded on a pro rata temporis basis. +You may contact `gemnasium@gitlab.com` regarding your closed account. + +## Will my account/data be transferred to GitLab Inc.? + +All accounts and data have been deleted on May 15th, 2018. GitLab Inc. +doesn't know anything about your private data, nor your projects, and therefore +if they were vulnerable or not. GitLab Inc. takes personal information very seriously. + +## What happened to my badge? + +To avoid broken 404 images, all badges pointing to Gemnasium.com will be a +placeholder, inviting you to migrate to GitLab (and pointing to this page). + +## Migrating to GitLab + +Gemnasium has been ported and integrated directly into GitLab CI/CD. +You can still benefit from our dependency monitoring features, and it requires +some steps to migrate your projects. There is no automatic import since GitLab +doesn't know anything about any projects which existed on Gemnasium.com. +Security features are free for public (open-source) projects hosted on GitLab.com. + +### If your project is hosted on GitLab (https://gitlab.com / self-hosted) + +You're almost set! If you're already using +[Auto DevOps](../../../topics/autodevops/), you are already covered. +Otherwise, you must configure your `.gitlab-ci.yml` according to the +[dependency scanning page](../../application_security/dependency_scanning/index.md). + +### If your project is hosted on GitHub (https://github.com / GitHub Enterprise) + +Since [GitLab 10.6 comes with GitHub integration](https://about.gitlab.com/features/github/), +GitLab users can now create a CI/CD project in GitLab connected to an external +GitHub.com or GitHub Enterprise repository. This will automatically prompt +GitLab CI/CD to run whenever code is pushed to GitHub and post CI/CD results +back to both GitLab and GitHub when completed. + +1. Create a new project, and select the "CI/CD for external repo" tab: + +  + +1. Use the "GitHub" button to connect your repositories. + +  + +1. Select the project(s) to be set up with GitLab CI/CD and chose "Connect". + +  + + Once the configuration is done, you may click on your new + project on GitLab. + +  + + Your project is now mirrored on GitLab, where the Runners will be able to access + your source code and run your tests. + + Optional step: If you set this up on GitLab.com, make sure the project is + public (in the project settings) if your GitHub project is public, since + the security feature is available only for [GitLab Ultimate](https://about.gitlab.com/pricing). + +1. To set up the dependency scanning job, corresponding to what Gemnasium was + doing, you must create a `.gitlab-ci.yml` file, or update it according to + the [dependency scanning docs](../../application_security/dependency_scanning/index.md). + The mirroring is pull-only by default, so you may create or update the file on + GitHub: + +  + +1. Once your file has been committed, a new pipeline will be automatically + triggered if your file is valid: + +  + +1. The result of the job will be visible directly from the pipeline view: + +  + +NOTE: **Note:** +If you don't commit very often to your project, you may want to use +[scheduled pipelines](../pipelines/schedules.md) to run the job on a regular +basis. diff --git a/doc/user/project/import/github.md b/doc/user/project/import/github.md index 63b90dd76fd..8fba892594b 100644 --- a/doc/user/project/import/github.md +++ b/doc/user/project/import/github.md @@ -119,9 +119,9 @@ Depending your GitLab tier, [project mirroring](../../../workflow/repository_mir your imported project in sync with its GitHub copy. Additionally, you can configure GitLab to send pipeline status updates back GitHub with the -[GitHub Project Integration](https://docs.gitlab.com/ee/user/project/integrations/github.html). **[PREMIUM]** +[GitHub Project Integration](../integrations/github.md). **[PREMIUM]** -If you import your project using [CI/CD for external repo](https://docs.gitlab.com/ee/ci/ci_cd_for_external_repos/), then both +If you import your project using [CI/CD for external repo](../../../ci/ci_cd_for_external_repos/index.md), then both of the above are automatically configured. **[PREMIUM]** ## Improving the speed of imports on self-hosted instances diff --git a/doc/user/project/import/gitlab_com.md b/doc/user/project/import/gitlab_com.md index 3b37da67a5b..f48a158e2d3 100644 --- a/doc/user/project/import/gitlab_com.md +++ b/doc/user/project/import/gitlab_com.md @@ -1,8 +1,9 @@ # Project importing from GitLab.com to your private GitLab instance You can import your existing GitLab.com projects to your GitLab instance. But keep in mind that it is possible only if -GitLab support is enabled on your GitLab instance. -You can read more about GitLab support [here](http://docs.gitlab.com/ce/integration/gitlab.html) +GitLab.com integration is enabled on your GitLab instance. +[Read more about GitLab.com integration for self-managed GitLab instances](../../../integration/gitlab.md). + To get to the importer page you need to go to "New project" page. >**Note:** diff --git a/doc/user/project/import/img/gemnasium/connect_github.png b/doc/user/project/import/img/gemnasium/connect_github.png Binary files differnew file mode 100644 index 00000000000..5933f4d5d0a --- /dev/null +++ b/doc/user/project/import/img/gemnasium/connect_github.png diff --git a/doc/user/project/import/img/gemnasium/create_project.png b/doc/user/project/import/img/gemnasium/create_project.png Binary files differnew file mode 100644 index 00000000000..6e00bf63405 --- /dev/null +++ b/doc/user/project/import/img/gemnasium/create_project.png diff --git a/doc/user/project/import/img/gemnasium/edit_gitlab-ci.png b/doc/user/project/import/img/gemnasium/edit_gitlab-ci.png Binary files differnew file mode 100644 index 00000000000..8336c803b83 --- /dev/null +++ b/doc/user/project/import/img/gemnasium/edit_gitlab-ci.png diff --git a/doc/user/project/import/img/gemnasium/pipeline.png b/doc/user/project/import/img/gemnasium/pipeline.png Binary files differnew file mode 100644 index 00000000000..ae4d5295ffa --- /dev/null +++ b/doc/user/project/import/img/gemnasium/pipeline.png diff --git a/doc/user/project/import/img/gemnasium/project_connected.png b/doc/user/project/import/img/gemnasium/project_connected.png Binary files differnew file mode 100644 index 00000000000..8e7216c015b --- /dev/null +++ b/doc/user/project/import/img/gemnasium/project_connected.png diff --git a/doc/user/project/import/img/gemnasium/report.png b/doc/user/project/import/img/gemnasium/report.png Binary files differnew file mode 100644 index 00000000000..5c4d58662c0 --- /dev/null +++ b/doc/user/project/import/img/gemnasium/report.png diff --git a/doc/user/project/import/img/gemnasium/select_project.png b/doc/user/project/import/img/gemnasium/select_project.png Binary files differnew file mode 100644 index 00000000000..588c5ca7ce1 --- /dev/null +++ b/doc/user/project/import/img/gemnasium/select_project.png diff --git a/doc/user/project/import/img/import_projects_from_github_select_auth_method.png b/doc/user/project/import/img/import_projects_from_github_select_auth_method.png Binary files differdeleted file mode 100644 index 90e6243aec0..00000000000 --- a/doc/user/project/import/img/import_projects_from_github_select_auth_method.png +++ /dev/null diff --git a/doc/user/project/import/index.md b/doc/user/project/import/index.md index ca02e4e9e96..2b6927bd780 100644 --- a/doc/user/project/import/index.md +++ b/doc/user/project/import/index.md @@ -13,18 +13,39 @@ 1. [From TFS](tfs.md) 1. [From repo by URL](repo_by_url.md) 1. [By uploading a manifest file (AOSP)](manifest.md) +1. [From Gemnasium](gemnasium.md) +1. [From Phabricator](phabricator.md) In addition to the specific migration documentation above, you can import any Git repository via HTTP from the New Project page. Be aware that if the repository is too large the import can timeout. +There is also the option of [connecting your external repository to get CI/CD benefits](../../../ci/ci_cd_for_external_repos/index.md). **[PREMIUM]** + ## Migrating from self-hosted GitLab to GitLab.com -You can copy your repos by changing the remote and pushing to the new server, -but issues and merge requests can't be imported. +If you only need to migrate git repos, you can [import each project by URL](repo_by_url.md), but issues and merge requests can't be imported. If you want to retain all metadata like issues and merge requests, you can use -the [import/export feature](../settings/import_export.md). +the [import/export feature](../settings/import_export.md) to export projects from self-hosted GitLab and import those projects into GitLab.com. + +NOTE: **Note:** +This approach assumes all users from the self-hosted instance have already been migrated. +If the users haven't been migrated yet, the user conducting the import +will take the place of all references to the missing user(s). + +If you need to migrate all data over, you can leverage our [api](../../../api/README.md) to migrate from self-hosted to GitLab.com. +The order of assets to migrate from a self-hosted instance to GitLab is the following: + +1. [Users](../../../api/users.md) +1. [Groups](../../../api/groups.md) +1. [Projects](../../../api/projects.md) +1. [Project variables](../../../api/project_level_variables.md) + +Keep in mind the limitations of the [import/export feature](../settings/import_export.md#exported-contents). + +You will still need to migrate your Container Registry over a series of +Docker pulls and pushes and re-run any CI pipelines to retrieve any build artifacts. ## Migrating between two self-hosted GitLab instances @@ -32,3 +53,6 @@ The best method for migrating a project from one GitLab instance to another, perhaps from an old server to a new server for example, is to [back up the project](../../../raketasks/backup_restore.md), then restore it on the new server. + +In the event of merging two GitLab instances together (for example, both instances have existing data on them and one can't be wiped), +refer to the instructions in [Migrating from self-hosted GitLab to GitLab.com](#migrating-from-self-hosted-gitlab-to-gitlabcom). diff --git a/doc/user/project/import/phabricator.md b/doc/user/project/import/phabricator.md new file mode 100644 index 00000000000..5c624e3aff6 --- /dev/null +++ b/doc/user/project/import/phabricator.md @@ -0,0 +1,29 @@ +# Import Phabricator tasks into a GitLab project + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/issues/60562) in +GitLab 12.0. + +GitLab allows you to import all tasks from a Phabricator instance into +GitLab issues. The import creates a single project with the +repository disabled. + +Currently, only the following basic fields are imported: + +- Title +- Description +- State (open or closed) +- Created at +- Closed at + +## Enabling this feature + +While this feature is incomplete, a feature flag is required to enable it so that +we can gain early feedback before releasing it for everyone. To enable it: + +1. Run the following command in a Rails console: + + ```ruby + Feature.enable(:phabricator_import) + ``` + +1. Enable Phabricator as an [import source](../../admin_area/settings/visibility_and_access_controls.md#import-sources) in the Admin area. diff --git a/doc/user/project/import/svn.md b/doc/user/project/import/svn.md index a5923986292..4825b005a85 100644 --- a/doc/user/project/import/svn.md +++ b/doc/user/project/import/svn.md @@ -8,7 +8,7 @@ between the two, for more information consult your favorite search engine. There are two approaches to SVN to Git migration: -1. [Git/SVN Mirror](#smooth-migration-with-a-git-svn-mirror-using-subgit) which: +1. [Git/SVN Mirror](#smooth-migration-with-a-gitsvn-mirror-using-subgit) which: - Makes the GitLab repository to mirror the SVN project. - Git and SVN repositories are kept in sync; you can use either one. - Smoothens the migration process and allows to manage migration risks. diff --git a/doc/user/project/index.md b/doc/user/project/index.md index c62de41c539..a24f525253d 100644 --- a/doc/user/project/index.md +++ b/doc/user/project/index.md @@ -8,16 +8,16 @@ Your projects can be [available](../../public_access/public_access.md) publicly, internally, or privately, at your choice. GitLab does not limit the number of private projects you create. -## Project's features +## Project features When you create a project in GitLab, you'll have access to a large number of [features](https://about.gitlab.com/features/): -**Issues and merge requests:** +**Repositories:** - [Issue tracker](issues/index.md): Discuss implementations with your team within issues - [Issue Boards](issue_board.md): Organize and prioritize your workflow - - [Multiple Issue Boards](https://docs.gitlab.com/ee/user/project/issue_board.html#multiple-issue-boards): Allow your teams to create their own workflows (Issue Boards) for the same project **[STARTER]** + - [Multiple Issue Boards](issue_board.md#multiple-issue-boards-starter): Allow your teams to create their own workflows (Issue Boards) for the same project **[STARTER]** - [Repositories](repository/index.md): Host your code in a fully integrated platform - [Branches](repository/branches/index.md): use Git branching strategies to @@ -28,9 +28,16 @@ When you create a project in GitLab, you'll have access to a large number of permission to create tags, and prevent accidental update or deletion - [Signing commits](gpg_signed_commits/index.md): use GPG to sign your commits - [Deploy tokens](deploy_tokens/index.md): Manage project-based deploy tokens that allow permanent access to the repository and Container Registry. +- [Web IDE](web_ide/index.md) + +**Issues and merge requests:** + +- [Issue tracker](issues/index.md): Discuss implementations with your team within issues + - [Issue Boards](issue_board.md): Organize and prioritize your workflow + - [Multiple Issue Boards](issue_board.md#multiple-issue-boards-starter): Allow your teams to create their own workflows (Issue Boards) for the same project **[STARTER]** - [Merge Requests](merge_requests/index.md): Apply your branching strategy and get reviewed by your team - - [Merge Request Approvals](https://docs.gitlab.com/ee/user/project/merge_requests/merge_request_approvals.html): Ask for approval before + - [Merge Request Approvals](merge_requests/merge_request_approvals.md): Ask for approval before implementing a change **[STARTER]** - [Fix merge conflicts from the UI](merge_requests/resolve_conflicts.md): Your Git diff tool right from GitLab's UI @@ -55,11 +62,11 @@ When you create a project in GitLab, you'll have access to a large number of - [Auto Deploy](../../ci/autodeploy/index.md): Configure GitLab CI/CD to automatically set up your app's deployment - [Enable and disable GitLab CI](../../ci/enable_or_disable_ci.md) - - [Pipelines](../../ci/pipelines.md#pipelines): Configure and visualize + - [Pipelines](../../ci/pipelines.md): Configure and visualize your GitLab CI/CD pipelines from the UI - [Scheduled Pipelines](pipelines/schedules.md): Schedule a pipeline to start at a chosen time - - [Pipeline Graphs](../../ci/pipelines.md#pipeline-graphs): View your + - [Pipeline Graphs](../../ci/pipelines.md#visualizing-pipelines): View your entire pipeline from the UI - [Job artifacts](pipelines/job_artifacts.md): Define, browse, and download job artifacts @@ -67,6 +74,8 @@ When you create a project in GitLab, you'll have access to a large number of timeout (defines the maximum amount of time in minutes that a job is able run), custom path for `.gitlab-ci.yml`, test coverage parsing, pipeline's visibility, and much more - [Kubernetes cluster integration](clusters/index.md): Connecting your GitLab project with a Kubernetes cluster + - [Feature Flags](operations/feature_flags.md): Feature flags allow you to ship a project in + different flavors by dynamically toggling certain functionality **[PREMIUM]** - [GitLab Pages](pages/index.md): Build, test, and deploy your static website with GitLab Pages @@ -75,14 +84,19 @@ When you create a project in GitLab, you'll have access to a large number of - [Wiki](wiki/index.md): document your GitLab project in an integrated Wiki. - [Snippets](../snippets.md): store, share and collaborate on code snippets. - [Cycle Analytics](cycle_analytics.md): review your development lifecycle. +- [Security Dashboard](security_dashboard.md): Security Dashboard. **[ULTIMATE]** - [Syntax highlighting](highlighting.md): an alternative to customize your code blocks, overriding GitLab's default choice of language. - [Badges](badges.md): badges for the project overview. - [Releases](releases/index.md): a way to track deliverables in your project as snapshot in time of the source, build output, and other metadata or artifacts associated with a released version of your code. +- [Maven packages](packages/maven_repository.md): your private Maven repository in GitLab. **[PREMIUM]** +- [NPM packages](packages/npm_registry.md): your private NPM package registry in GitLab. **[PREMIUM]** +- [Code owners](code_owners.md): specify code owners for certain files **[STARTER]** +- [License Management](../application_security/license_management/index.md): approve and blacklist licenses for projects. **[ULTIMATE]** -### Project's integrations +### Project integrations [Integrate your project](integrations/index.md) with Jira, Mattermost, Kubernetes, Slack, and a lot more. @@ -116,7 +130,14 @@ Read through the documentation on [project settings](settings/index.md). - [Export a project from GitLab](settings/import_export.md#exporting-a-project-and-its-data) - [Importing and exporting projects between GitLab instances](settings/import_export.md) -## Project's members +## CI/CD for external repositories **[PREMIUM]** + +Instead of importing a repository directly to GitLab, you can connect your repository +as a CI/CD project. + +Read through the documentation on [CI/CD for external repositories](../../ci/ci_cd_for_external_repos/index.md). + +## Project members Learn how to [add members to your projects](members/index.md). @@ -137,7 +158,7 @@ and Git push/pull redirects. Depending on the situation, different things apply. When [renaming a user](../profile/index.md#changing-your-username), -[changing a group path](../group/index.md#changing-a-group-s-path) or [renaming a repository](settings/index.md#renaming-a-repository): +[changing a group path](../group/index.md#changing-a-groups-path) or [renaming a repository](settings/index.md#renaming-a-repository): - Existing web URLs for the namespace and anything under it (e.g., projects) will redirect to the new URLs. @@ -170,3 +191,23 @@ password <personal_access_token> To quickly access a project from the GitLab UI using the project ID, visit the `/projects/:id` URL in your browser or other tool accessing the project. + +## Project APIs + +There are numerous [APIs](../../api/README.md) to use with your projects: + +- [Badges](../../api/project_badges.md) +- [Clusters](../../api/project_clusters.md) +- [Discussions](../../api/discussions.md) +- [General](../../api/projects.md) +- [Import/export](../../api/project_import_export.md) +- [Issue Board](../../api/boards.md) +- [Labels](../../api/labels.md) +- [Markdown](../../api/markdown.md) +- [Merge Requests](../../api/merge_requests.md) +- [Milestones](../../api/milestones.md) +- [Services](../../api/services.md) +- [Snippets](../../api/project_snippets.md) +- [Templates](../../api/project_templates.md) +- [Traffic](../../api/project_statistics.md) +- [Variables](../../api/project_level_variables.md) diff --git a/doc/user/project/insights/img/insights_example_bar_chart.png b/doc/user/project/insights/img/insights_example_bar_chart.png Binary files differnew file mode 100644 index 00000000000..eb96eb4a90f --- /dev/null +++ b/doc/user/project/insights/img/insights_example_bar_chart.png diff --git a/doc/user/project/insights/img/insights_example_bar_time_series_chart.png b/doc/user/project/insights/img/insights_example_bar_time_series_chart.png Binary files differnew file mode 100644 index 00000000000..28aa81939d8 --- /dev/null +++ b/doc/user/project/insights/img/insights_example_bar_time_series_chart.png diff --git a/doc/user/project/insights/img/insights_example_line_chart.png b/doc/user/project/insights/img/insights_example_line_chart.png Binary files differnew file mode 100644 index 00000000000..131dbedd35e --- /dev/null +++ b/doc/user/project/insights/img/insights_example_line_chart.png diff --git a/doc/user/project/insights/img/insights_example_pie_chart.png b/doc/user/project/insights/img/insights_example_pie_chart.png Binary files differnew file mode 100644 index 00000000000..518ed7338f9 --- /dev/null +++ b/doc/user/project/insights/img/insights_example_pie_chart.png diff --git a/doc/user/project/insights/img/insights_example_stacked_bar_chart.png b/doc/user/project/insights/img/insights_example_stacked_bar_chart.png Binary files differnew file mode 100644 index 00000000000..aafec4b394e --- /dev/null +++ b/doc/user/project/insights/img/insights_example_stacked_bar_chart.png diff --git a/doc/user/project/insights/img/insights_sidebar_link.png b/doc/user/project/insights/img/insights_sidebar_link.png Binary files differnew file mode 100644 index 00000000000..aadb5745992 --- /dev/null +++ b/doc/user/project/insights/img/insights_sidebar_link.png diff --git a/doc/user/project/insights/img/project_insights.png b/doc/user/project/insights/img/project_insights.png Binary files differnew file mode 100644 index 00000000000..2d0292dda54 --- /dev/null +++ b/doc/user/project/insights/img/project_insights.png diff --git a/doc/user/project/insights/index.md b/doc/user/project/insights/index.md new file mode 100644 index 00000000000..3344e560870 --- /dev/null +++ b/doc/user/project/insights/index.md @@ -0,0 +1,307 @@ +# Insights **[ULTIMATE]** + +> Introduced in [GitLab Ultimate](https://about.gitlab.com/pricing/) 11.9 behind the `insights` feature flag. + +CAUTION: **Beta:** +Insights is considered beta, and is not ready for production use. +Follow [gitlab-org/quality/team-tasks#137](https://gitlab.com/gitlab-org/quality/team-tasks/issues/137#general-availability) +for updates. + +Configure the Insights that matter for your projects to explore data such as +triage hygiene, issues created/closed per a given period, average time for merge +requests to be merged and much more. + + + +NOTE: **Note:** +This feature is [also available at the group level](../../group/insights/index.md). + +## View your project's Insights + +You can access your project's Insights by clicking the **Project > Insights** +link in the left sidebar: + + + +## Configure your Insights + +Insights are configured using a YAML file called `.gitlab/insights.yml` within +a project. That file will then be used in the project's Insights page. + +See [Writing your `.gitlab/insights.yml`](#writing-your-gitlabinsightsyml) below +for details about the content of this file. + +NOTE: **Note:** +Once the configuration file is created, you can also +[use it for your project's group](../../group/insights/index.md#configure-your-insights). + +NOTE: **Note:** +If the project doesn't have any configuration file, it'll try to use +the group configuration if possible. If the group doesn't have any +configuration, the default configuration will be used. + +## Permissions + +If you have access to view a project, then you have access to view their +Insights. + +NOTE: **Note:** +Issues or merge requests that you don't have access to (because you don't have +access to the project they belong to, or because they are confidential) are +filtered out of the Insights charts. + +You may also consult the [group permissions table](../../permissions.md#group-members-permissions). + +## Writing your `.gitlab/insights.yml` + +The `.gitlab/insights.yml` file defines the structure and order of the Insights +charts that will be displayed in each Insights page of your project or group. + +Each page has a unique key and a collection of charts to fetch and display. + +For example, here's a single definition for Insights that will display one page with one chart: + +```yaml +bugsCharts: + title: 'Charts for Bugs' + charts: + - title: Monthly Bugs Created (bar) + type: bar + query: + issuable_type: issue + issuable_state: opened + filter_labels: + - bug + group_by: month + period_limit: 24 +``` + +Each chart definition is made up of a hash composed of key-value pairs. + +For example, here's single chart definition: + +```yaml +- title: Monthly Bugs Created (bar) + type: bar + query: + issuable_type: issue + issuable_state: opened + filter_labels: + - bug + group_by: month + period_limit: 24 +``` + +## Configuration parameters + +A chart is defined as a list of parameters that define the chart's behavior. + +The following table lists available parameters for charts: + +| Keyword | Description | +|:---------------------------------------------------|:------------| +| [`title`](#title) | The title of the chart. This will displayed on the Insights page. | +| [`type`](#type) | The type of chart: `bar`, `line`, `stacked-bar`, `pie` etc. | +| [`query`](#query) | A hash that defines the conditions for issues / merge requests to be part of the chart. | + +## Parameter details + +The following are detailed explanations for parameters used to configure +Insights charts. + +### `title` + +`title` is the title of the chart as it will be displayed on the Insights page. +For example: + +```yaml +monthlyBugsCreated: + title: Monthly Bugs Created (bar) +``` + +### `type` + +`type` is the chart type. + +For example: + +```yaml +monthlyBugsCreated: + title: Monthly Bugs Created (bar) + type: bar +``` + +Supported values are: + +| Name | Example | +| ----- | ------- | +| `bar` |  | +| `bar` (time series, i.e. when `group_by` is used) |  | +| `pie` |  | +| `line` |  | +| `stacked-bar` |  | + +### `query` + +`query` allows to define the conditions for issues / merge requests to be part +of the chart. + +Example: + +```yaml +monthlyBugsCreated: + title: Monthly Bugs Created (bar) + type: bar + query: + issuable_type: issue + issuable_state: opened + filter_labels: + - bug + collection_labels: + - S1 + - S2 + - S3 + - S4 + group_by: week + period_limit: 104 +``` + +#### `query.issuable_type` + +Defines the type of "issuable" you want to create a chart for. + +Supported values are: + +- `issue`: The chart will display issues' data. +- `merge_request`: The chart will display merge requests' data. + +#### `query.issuable_state` + +Filter by the state of the queried "issuable". + +If you omit it, the `opened` state filter will be applied. + +Supported values are: + +- `opened`: Open issues / merge requests. +- `closed`: Closed Open issues / merge requests. +- `locked`: Issues / merge requests that have their discussion locked. +- `merged`: Merged merge requests. +- `all`: Issues / merge requests in all states + +#### `query.filter_labels` + +Filter by labels applied to the queried "issuable". + +If you omit it, no labels filter will be applied. All the defined labels must be +applied to the "issuable" in order for it to be selected. + +Example: + +```yaml +monthlyBugsCreated: + title: Monthly regressions Created (bar) + type: bar + query: + issuable_type: issue + issuable_state: opened + filter_labels: + - bug + - regression +``` + +#### `query.collection_labels` + +Group "issuable" by the configured labels. + +If you omit it, no grouping will be done. When using this keyword, you need to +set `type` to either `line` or `stacked-bar`. + +Example: + +```yaml +weeklyBugsBySeverity: + title: Weekly Bugs By Severity (stacked bar) + type: stacked-bar + query: + issuable_type: issue + issuable_state: opened + filter_labels: + - bug + collection_labels: + - S1 + - S2 + - S3 + - S4 +``` + +#### `query.group_by` + +Define the X-axis of your chart. + +Supported values are: + +- `day`: Group data per day. +- `week`: Group data per week. +- `month`: Group data per month. + +#### `query.period_limit` + +Define how far "issuables" are queried in the past. + +The unit is related to the `query.group_by` you defined. For instance if you +defined `query.group_by: 'day'` then `query.period_limit: 365` would mean +"Gather and display data for the last 365 days". + +If you omit it, default values will be applied depending on the `query.group_by` +you defined. + +| `query.group_by` | Default value | +| ---------------- | ------------- | +| `day` | 30 | +| `week` | 4 | +| `month` | 12 | + +## Complete example + +```yaml +bugsCharts: + title: 'Charts for Bugs' + charts: + - title: Monthly Bugs Created (bar) + type: bar + query: + issuable_type: issue + issuable_state: opened + filter_labels: + - bug + group_by: month + period_limit: 24 + - title: Weekly Bugs By Severity (stacked bar) + type: stacked-bar + query: + issuable_type: issue + issuable_state: opened + filter_labels: + - bug + collection_labels: + - S1 + - S2 + - S3 + - S4 + group_by: week + period_limit: 104 + - title: Monthly Bugs By Team (line) + type: line + query: + issuable_type: merge_request + issuable_state: opened + filter_labels: + - bug + collection_labels: + - Manage + - Plan + - Create + group_by: month + period_limit: 24 +``` diff --git a/doc/user/project/integrations/custom_issue_tracker.md b/doc/user/project/integrations/custom_issue_tracker.md index 6fc083170b6..23f1ce7a15a 100644 --- a/doc/user/project/integrations/custom_issue_tracker.md +++ b/doc/user/project/integrations/custom_issue_tracker.md @@ -7,9 +7,9 @@ in the table below. | Field | Description | | ----- | ----------- | -| `title` | A title for the issue tracker (to differentiate between instances, for example) | +| `title` | A title for the issue tracker (to differentiate between instances, for example). | | `description` | A name for the issue tracker (to differentiate between instances, for example) | -| `project_url` | Currently unused. Will be changed in a future release. | +| `project_url` | The URL to the project in the custom issue tracker. | | `issues_url` | The URL to the issue in the issue tracker project that is linked to this GitLab project. Note that the `issues_url` requires `:id` in the URL. This ID is used by GitLab as a placeholder to replace the issue number. For example, `https://customissuetracker.com/project-name/:id`. | | `new_issue_url` | Currently unused. Will be changed in a future release. | diff --git a/doc/user/project/integrations/github.md b/doc/user/project/integrations/github.md new file mode 100644 index 00000000000..cdb0e34fdf6 --- /dev/null +++ b/doc/user/project/integrations/github.md @@ -0,0 +1,48 @@ +# GitHub project integration **[PREMIUM]** + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/issues/3836) in GitLab Premium 10.6. + +GitLab provides an integration for updating the pipeline statuses on GitHub. +This is especially useful if using GitLab for CI/CD only. + +This project integration is separate from the [instance wide GitHub integration](../import/github.md#mirroring-and-pipeline-status-sharing) +and is automatically configured on [GitHub import](../../../integration/github.md). + + + +## Configuration + +### Complete these steps on GitHub + +This integration requires a [GitHub API token](https://help.github.com/articles/creating-a-personal-access-token-for-the-command-line/) +with `repo:status` access granted: + +1. Go to your "Personal access tokens" page at https://github.com/settings/tokens +1. Click "Generate New Token" +1. Ensure that `repo:status` is checked and click "Generate token" +1. Copy the generated token to use on GitLab + +### Complete these steps on GitLab + +1. Navigate to the project you want to configure. +1. Navigate to the [Integrations page](project_services.md#accessing-the-project-services) +1. Click "GitHub". +1. Select the "Active" checkbox. +1. Paste the token you've generated on GitHub +1. Enter the path to your project on GitHub, such as `https://github.com/username/repository` +1. Optionally check "Static status check names" checkbox to enable static status check names. +1. Save or optionally click "Test Settings". + +#### Static / dynamic status check names + +Since GitLab 11.5 it is possible to opt-in to using static status check names. + +This makes it possible to mark these status checks as _Required_ on GitHub. +If you check "Static status check names" checkbox on the integration page, your +GitLab instance host name is going to be appended to a status check name, +whereas in case of dynamic status check names, a branch name is going to be +appended. + +Dynamic status check name is a default behavior. + + diff --git a/doc/user/project/integrations/gitlab_slack_application.md b/doc/user/project/integrations/gitlab_slack_application.md new file mode 100644 index 00000000000..8e062ca627b --- /dev/null +++ b/doc/user/project/integrations/gitlab_slack_application.md @@ -0,0 +1,65 @@ +# GitLab Slack application **[FREE ONLY]** + +NOTE: **Note:** +The GitLab Slack application is only configurable for GitLab.com. It will **not** +work for on-premises installations where you can configure the +[Slack slash commands](slack_slash_commands.md) service instead. We're working +with Slack on making this configurable for all GitLab installations, but there's +no ETA. +It was first introduced in GitLab 9.4 and distributed to Slack App Directory in +GitLab 10.2. + +Slack provides a native application which you can enable via your project's +integrations on GitLab.com. + +## Slack App Directory + +The simplest way to enable the GitLab Slack application for your workspace is to +install the [GitLab application](https://slack-platform.slack.com/apps/A676ADMV5-gitlab) from +the [Slack App Directory](https://slack.com/apps). + +Clicking install will take you to the +[GitLab Slack application landing page](https://gitlab.com/profile/slack/edit) +where you can select a project to enable the GitLab Slack application for. + + + +## Configuration + +Alternatively, you can configure the Slack application with a project's +integration settings. + +Keep in mind that you need to have the appropriate permissions for your Slack +team in order to be able to install a new application, read more in Slack's +docs on [Adding an app to your team][slack-docs]. + +To enable GitLab's service for your Slack team: + +1. Go to your project's **Settings > Integration > Slack application** (only + visible on GitLab.com) +1. Click the "Add to Slack" button + +That's all! You can now start using the Slack slash commands. + +## Usage + +After confirming the installation, you, and everyone else in your Slack team, +can use all the [slash commands]. + +When you perform your first slash command you will be asked to authorize your +Slack user on GitLab.com. + +The only difference with the [manually configurable Slack slash commands][slack-manual] +is that all the commands should be prefixed with the `/gitlab` keyword. +We are working on making this configurable in the future. + +For example, to show the issue number `1001` under the `gitlab-org/gitlab-ce` +project, you would do: + +``` +/gitlab gitlab-org/gitlab-ce issue show 1001 +``` + +[slack-docs]: https://get.slack.help/hc/en-us/articles/202035138-Adding-apps-to-your-team +[slash commands]: ../../../integration/slash_commands.md +[slack-manual]: slack_slash_commands.md diff --git a/doc/user/project/integrations/hipchat.md b/doc/user/project/integrations/hipchat.md new file mode 100644 index 00000000000..0fd847d415f --- /dev/null +++ b/doc/user/project/integrations/hipchat.md @@ -0,0 +1,53 @@ +# Atlassian HipChat + +GitLab provides a way to send HipChat notifications upon a number of events, +such as when a user pushes code, creates a branch or tag, adds a comment, and +creates a merge request. + +## Setup + +GitLab requires the use of a HipChat v2 API token to work. v1 tokens are +not supported at this time. Note the differences between v1 and v2 tokens: + +HipChat v1 API (legacy) supports "API Auth Tokens" in the Group API menu. A v1 +token is allowed to send messages to *any* room. + +HipChat v2 API has tokens that are can be created using the Integrations tab +in the Group or Room admin page. By design, these are lightweight tokens that +allow GitLab to send messages only to *one* room. + +### Complete these steps in HipChat + +1. Go to: <https://admin.hipchat.com/admin> +1. Click on "Group Admin" -> "Integrations". +1. Find "Build Your Own!" and click "Create". +1. Select the desired room, name the integration "GitLab", and click "Create". +1. In the "Send messages to this room by posting this URL" column, you should +see a URL in the format: + +``` +https://api.hipchat.com/v2/room/<room>/notification?auth_token=<token> +``` + +HipChat is now ready to accept messages from GitLab. Next, set up the HipChat +service in GitLab. + +### Complete these steps in GitLab + +1. Navigate to the project you want to configure for notifications. +1. Navigate to the [Integrations page](project_services.md#accessing-the-project-services) +1. Click "HipChat". +1. Select the "Active" checkbox. +1. Insert the `token` field from the URL into the `Token` field on the Web page. +1. Insert the `room` field from the URL into the `Room` field on the Web page. +1. Save or optionally click "Test Settings". + +## Troubleshooting + +If you do not see notifications, make sure you are using a HipChat v2 API +token, not a v1 token. + +Note that the v2 token is tied to a specific room. If you want to be able to +specify arbitrary rooms, you can create an API token for a specific user in +HipChat under "Account settings" and "API access". Use the `XXX` value under +`auth_token=XXX`. diff --git a/doc/user/project/integrations/img/github_configuration.png b/doc/user/project/integrations/img/github_configuration.png Binary files differnew file mode 100644 index 00000000000..9f2d47921c7 --- /dev/null +++ b/doc/user/project/integrations/img/github_configuration.png diff --git a/doc/user/project/integrations/img/github_status_check_pipeline_update.png b/doc/user/project/integrations/img/github_status_check_pipeline_update.png Binary files differnew file mode 100644 index 00000000000..9d02ca18497 --- /dev/null +++ b/doc/user/project/integrations/img/github_status_check_pipeline_update.png diff --git a/doc/user/project/integrations/img/gitlab_slack_app_landing_page.png b/doc/user/project/integrations/img/gitlab_slack_app_landing_page.png Binary files differnew file mode 100644 index 00000000000..57cd35c9f5d --- /dev/null +++ b/doc/user/project/integrations/img/gitlab_slack_app_landing_page.png diff --git a/doc/user/project/integrations/img/issue_configuration.png b/doc/user/project/integrations/img/issue_configuration.png Binary files differdeleted file mode 100644 index 5dfd85974d8..00000000000 --- a/doc/user/project/integrations/img/issue_configuration.png +++ /dev/null diff --git a/doc/user/project/integrations/img/jira_add_user_to_group.png b/doc/user/project/integrations/img/jira_add_user_to_group.png Binary files differindex 27dac49260c..d8cf541a81e 100644 --- a/doc/user/project/integrations/img/jira_add_user_to_group.png +++ b/doc/user/project/integrations/img/jira_add_user_to_group.png diff --git a/doc/user/project/integrations/img/jira_added_user_to_group.png b/doc/user/project/integrations/img/jira_added_user_to_group.png Binary files differnew file mode 100644 index 00000000000..b3e29a65d6e --- /dev/null +++ b/doc/user/project/integrations/img/jira_added_user_to_group.png diff --git a/doc/user/project/integrations/img/jira_api_token_menu.png b/doc/user/project/integrations/img/jira_api_token_menu.png Binary files differindex 55c8fb1bdb9..14037bd0b47 100644 --- a/doc/user/project/integrations/img/jira_api_token_menu.png +++ b/doc/user/project/integrations/img/jira_api_token_menu.png diff --git a/doc/user/project/integrations/img/jira_create_new_group.png b/doc/user/project/integrations/img/jira_create_new_group.png Binary files differindex 06c4e84fc61..84be3a94a45 100644 --- a/doc/user/project/integrations/img/jira_create_new_group.png +++ b/doc/user/project/integrations/img/jira_create_new_group.png diff --git a/doc/user/project/integrations/img/jira_create_new_user.png b/doc/user/project/integrations/img/jira_create_new_user.png Binary files differindex e9c03ed770d..8460dc98ef9 100644 --- a/doc/user/project/integrations/img/jira_create_new_user.png +++ b/doc/user/project/integrations/img/jira_create_new_user.png diff --git a/doc/user/project/integrations/img/jira_group_access.png b/doc/user/project/integrations/img/jira_group_access.png Binary files differindex 448cc55504d..58cf114bd55 100644 --- a/doc/user/project/integrations/img/jira_group_access.png +++ b/doc/user/project/integrations/img/jira_group_access.png diff --git a/doc/user/project/integrations/img/jira_project_name.png b/doc/user/project/integrations/img/jira_project_name.png Binary files differdeleted file mode 100644 index 981c7f7ca18..00000000000 --- a/doc/user/project/integrations/img/jira_project_name.png +++ /dev/null diff --git a/doc/user/project/integrations/img/jira_service.png b/doc/user/project/integrations/img/jira_service.png Binary files differdeleted file mode 100644 index 0ae2fa28756..00000000000 --- a/doc/user/project/integrations/img/jira_service.png +++ /dev/null diff --git a/doc/user/project/integrations/img/jira_service_page.png b/doc/user/project/integrations/img/jira_service_page.png Binary files differindex 3a27b4df841..377b69d9d06 100644 --- a/doc/user/project/integrations/img/jira_service_page.png +++ b/doc/user/project/integrations/img/jira_service_page.png diff --git a/doc/user/project/integrations/img/jira_user_management_link.png b/doc/user/project/integrations/img/jira_user_management_link.png Binary files differindex 5eb9d031c3e..43ef18da6c8 100644 --- a/doc/user/project/integrations/img/jira_user_management_link.png +++ b/doc/user/project/integrations/img/jira_user_management_link.png diff --git a/doc/user/project/integrations/img/mattermost_configuration.png b/doc/user/project/integrations/img/mattermost_configuration.png Binary files differindex e0b55b23520..75ef0310f2d 100644 --- a/doc/user/project/integrations/img/mattermost_configuration.png +++ b/doc/user/project/integrations/img/mattermost_configuration.png diff --git a/doc/user/project/integrations/img/prometheus_add_metric.png b/doc/user/project/integrations/img/prometheus_add_metric.png Binary files differnew file mode 100644 index 00000000000..e85670e1a13 --- /dev/null +++ b/doc/user/project/integrations/img/prometheus_add_metric.png diff --git a/doc/user/project/integrations/img/prometheus_alert.png b/doc/user/project/integrations/img/prometheus_alert.png Binary files differnew file mode 100644 index 00000000000..a37f0477fd9 --- /dev/null +++ b/doc/user/project/integrations/img/prometheus_alert.png diff --git a/doc/user/project/integrations/img/prometheus_service_alerts.png b/doc/user/project/integrations/img/prometheus_service_alerts.png Binary files differnew file mode 100644 index 00000000000..a81dfe7da14 --- /dev/null +++ b/doc/user/project/integrations/img/prometheus_service_alerts.png diff --git a/doc/user/project/integrations/img/prometheus_yaml_deploy.png b/doc/user/project/integrations/img/prometheus_yaml_deploy.png Binary files differdeleted file mode 100644 index 78dd178a077..00000000000 --- a/doc/user/project/integrations/img/prometheus_yaml_deploy.png +++ /dev/null diff --git a/doc/user/project/integrations/img/slack_configuration.png b/doc/user/project/integrations/img/slack_configuration.png Binary files differindex 53b30e0e8cd..a14d2969488 100644 --- a/doc/user/project/integrations/img/slack_configuration.png +++ b/doc/user/project/integrations/img/slack_configuration.png diff --git a/doc/user/project/integrations/jira.md b/doc/user/project/integrations/jira.md index 754711f5919..c652149052e 100644 --- a/doc/user/project/integrations/jira.md +++ b/doc/user/project/integrations/jira.md @@ -47,15 +47,16 @@ project in Jira and then enter the correct values in GitLab. When connecting to **JIRA Server**, which supports basic authentication, a **username and password** are required. Check the link below and proceed to the next step: -- [Setting up an user in JIRA server](jira_server_configuration.md) +- [Setting up a user in JIRA server](jira_server_configuration.md) When connecting to **JIRA Cloud**, which supports authentication via API token, an **email and API token**, are required. Check the link below and proceed to the next step: -- [Setting up an user in JIRA cloud](jira_cloud_configuration.md) +- [Setting up a user in JIRA cloud](jira_cloud_configuration.md) ### Configuring GitLab > **Notes:** +> > - The currently supported Jira versions are `v6.x` and `v7.x.`. GitLab 7.8 or > higher is required. > - GitLab 8.14 introduced a new way to integrate with Jira which greatly simplified @@ -142,6 +143,7 @@ the same goal: where `PROJECT-1` is the issue ID of the Jira project. > **Notes:** +> > - Only commits and merges into the project's default branch (usually **master**) will > close an issue in Jira. You can change your projects default branch under > [project settings](img/jira_project_settings.png). diff --git a/doc/user/project/integrations/jira_server_configuration.md b/doc/user/project/integrations/jira_server_configuration.md index 20036183187..13d65c4d8e4 100644 --- a/doc/user/project/integrations/jira_server_configuration.md +++ b/doc/user/project/integrations/jira_server_configuration.md @@ -1,18 +1,16 @@ # Creating a username and password for JIRA server We need to create a user in Jira which will have access to all projects that -need to integrate with GitLab. Login to your Jira instance as admin and under -*Administration*, go to *User Management* and create a new user. +need to integrate with GitLab. As an example, we'll create a user named `gitlab` and add it to the `Jira-developers` group. NOTE: **Note** -It is important that the user `gitlab` has 'write' access to projects in Jira. +It is important that the Jira user created for the integration is given 'write' +access to your Jira projects. This is covered in the process below. -We have split this stage in steps so it is easier to follow. - -1. Log in to your Jira instance as an administrator and under **Administration** +1. Log in to your Jira instance as an administrator and under **Jira Administration** go to **User Management** to create a new user.  @@ -27,27 +25,34 @@ We have split this stage in steps so it is easier to follow.  -1. Create a `gitlab-developers` group which will have write access - to projects in Jira. Go to the **Groups** tab and select **Create group**. +1. Create a `gitlab-developers` group. (We will give this group write access to Jira + projects in a later step). Go to the **Groups** tab on the left, and select **Add group**.  - Give it an optional description and click **Create group**. + Give it a name and click **Add group**. -  +1. Add the `gitlab` user to the `gitlab-developers` group by clicking **Edit members**. + The `gitlab-developers` group should be listed in the leftmost box as a selected group. + Under **Add members to selected group(s)**, enter `gitlab`. -1. To give the newly-created group 'write' access, go to - **Application access > View configuration** and add the `gitlab-developers` - group to Jira Core. +  + + Click **Add selected users** and `gitlab` should appear in the **Group member(s)** box. + This membership is saved automatically. + +  + +1. To give the newly-created group 'write' access, you need to create a **Permission Scheme**. + To do this, click the gear icon and select **Issues**. Then click **Permission Schemes**. + Click **Add Permission Scheme** and enter a **Name** and, optionally, a **Description**. + +1. Once your permission scheme is created, you'll be taken back to the permissions scheme list. + Locate your new permissions scheme and click **Permissions**. Next to **Administer Projects**, + click **Edit**. In the resulting dialog box, select **Group** and select `gitlab-developers` + from the dropdown.  -1. Add the `gitlab` user to the `gitlab-developers` group by going to - **Users > GitLab user > Add group** and selecting the `gitlab-developers` - group from the dropdown menu. Notice that the group says _Access_, which is - intended as part of this process. - -  - The Jira configuration is complete. Write down the new Jira username and its password as they will be needed when [configuring GitLab in the next section](jira.md#configuring-gitlab). diff --git a/doc/user/project/integrations/kubernetes.md b/doc/user/project/integrations/kubernetes.md index 9342a2cbb00..ab43eb2da7e 100644 --- a/doc/user/project/integrations/kubernetes.md +++ b/doc/user/project/integrations/kubernetes.md @@ -1 +1,5 @@ +--- +redirect_to: '../clusters/index.md' +--- + This document was moved to [another location](../clusters/index.md). diff --git a/doc/user/project/integrations/mattermost.md b/doc/user/project/integrations/mattermost.md index 8c5461de42f..d7fd75fd728 100644 --- a/doc/user/project/integrations/mattermost.md +++ b/doc/user/project/integrations/mattermost.md @@ -27,9 +27,11 @@ There, you will see a checkbox with the following events that can be triggered: - Confidential issue - Merge request - Note +- Confidential note - Tag push - Pipeline - Wiki page +- Deployment Below each of these event checkboxes, you have an input field to enter which Mattermost channel you want to send that event message. Enter your preferred channel handle (the hash sign `#` is optional). diff --git a/doc/user/project/integrations/mattermost_slash_commands.md b/doc/user/project/integrations/mattermost_slash_commands.md index e031dcad2c3..9c69437537a 100644 --- a/doc/user/project/integrations/mattermost_slash_commands.md +++ b/doc/user/project/integrations/mattermost_slash_commands.md @@ -152,7 +152,7 @@ trigger word followed by <kbd>help</kbd>. Example: <samp>/gitlab help</samp> ## Permissions The permissions to run the [available commands](#available-slash-commands) derive from -the [permissions you have on the project](../../permissions.md#project). +the [permissions you have on the project](../../permissions.md#project-members-permissions). ## Further reading diff --git a/doc/user/project/integrations/mock_ci.md b/doc/user/project/integrations/mock_ci.md index 8b1908c46fe..1c64b275d6e 100644 --- a/doc/user/project/integrations/mock_ci.md +++ b/doc/user/project/integrations/mock_ci.md @@ -5,7 +5,7 @@ To set up the mock CI service server, respond to the following endpoints - `commit_status`: `#{project.namespace.path}/#{project.path}/status/#{sha}.json` - - Have your service return `200 { status: ['failed'|'canceled'|'running'|'pending'|'success'|'success_with_warnings'|'skipped'|'not_found'] }` + - Have your service return `200 { status: ['failed'|'canceled'|'running'|'pending'|'success'|'success-with-warnings'|'skipped'|'not_found'] }` - If the service returns a 404, it is interpreted as `pending` - `build_page`: `#{project.namespace.path}/#{project.path}/status/#{sha}` - Just where the build is linked to, doesn't matter if implemented diff --git a/doc/user/project/integrations/project_services.md b/doc/user/project/integrations/project_services.md index cec9018b67f..0bfee3bac99 100644 --- a/doc/user/project/integrations/project_services.md +++ b/doc/user/project/integrations/project_services.md @@ -14,8 +14,7 @@ want to configure.  -Below, you will find a list of the currently supported ones accompanied with -comprehensive documentation. +Below, you will find a list of the currently supported ones accompanied with comprehensive documentation. ## Services @@ -35,9 +34,12 @@ Click on the service links to see further configuration instructions and details | [Emails on push](emails_on_push.md) | Email the commits and diff of each push to a list of recipients | | External Wiki | Replaces the link to the internal wiki with a link to an external wiki | | Flowdock | Flowdock is a collaboration web app for technical teams | +| [GitHub](github.md) **[PREMIUM]** | Sends pipeline notifications to GitHub | | [Hangouts Chat](hangouts_chat.md) | Receive events notifications in Google Hangouts Chat | +| [HipChat](hipchat.md) | Private group chat and IM | | [Irker (IRC gateway)](irker.md) | Send IRC messages, on update, to a list of recipients through an Irker gateway | | [JIRA](jira.md) | JIRA issue tracker | +| [Jenkins](../../../integration/jenkins.md) **[STARTER]** | An extendable open source continuous integration server | | JetBrains TeamCity CI | A continuous integration and build server | | [Mattermost slash commands](mattermost_slash_commands.md) | Mattermost chat and ChatOps slash commands | | [Mattermost Notifications](mattermost.md) | Receive event notifications in Mattermost | @@ -45,11 +47,13 @@ Click on the service links to see further configuration instructions and details | Packagist | Update your project on Packagist, the main Composer repository | | Pipelines emails | Email the pipeline status to a list of recipients | | [Slack Notifications](slack.md) | Send GitLab events (e.g. issue created) to Slack as notifications | -| [Slack slash commands](slack_slash_commands.md) | Use slash commands in Slack to control GitLab | +| [Slack slash commands](slack_slash_commands.md) **[CORE ONLY]** | Use slash commands in Slack to control GitLab | +| [GitLab Slack application](gitlab_slack_application.md) **[FREE ONLY]** | Use Slack's official application | | PivotalTracker | Project Management Software (Source Commits Endpoint) | | [Prometheus](prometheus.md) | Monitor the performance of your deployed apps | | Pushover | Pushover makes it easy to get real-time notifications on your Android device, iPhone, iPad, and Desktop | | [Redmine](redmine.md) | Redmine issue tracker | +| [YouTrack](youtrack.md) | YouTrack issue tracker | ## Services templates diff --git a/doc/user/project/integrations/prometheus.md b/doc/user/project/integrations/prometheus.md index ed289b0c4eb..751e8e44e60 100644 --- a/doc/user/project/integrations/prometheus.md +++ b/doc/user/project/integrations/prometheus.md @@ -13,7 +13,7 @@ There are two ways to set up Prometheus integration, depending on where your app - For deployments on Kubernetes, GitLab can automatically [deploy and manage Prometheus](#managed-prometheus-on-kubernetes). - For other deployment targets, simply [specify the Prometheus server](#manual-configuration-of-prometheus). -Once enabled, GitLab will automatically detect metrics from known services in the [metric library](#monitoring-ci-cd-environments). +Once enabled, GitLab will automatically detect metrics from known services in the [metric library](#monitoring-cicd-environments). You are also able to [add your own metrics](#adding-additional-metrics-premium) as well. ## Enabling Prometheus Integration @@ -93,6 +93,85 @@ GitLab will automatically scan the Prometheus server for metrics from known serv You can view the performance dashboard for an environment by [clicking on the monitoring button](../../../ci/environments.md#monitoring-environments). +### Adding additional metrics **[PREMIUM]** + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/3799) in [GitLab Premium](https://about.gitlab.com/pricing/) 10.6. + +Additional metrics can be monitored by adding them on the Prometheus integration page. Once saved, they will be displayed on the environment performance dashboard. + + + +A few fields are required: + +- **Name**: Chart title +- **Type**: Type of metric. Metrics of the same type will be shown together. +- **Query**: Valid [PromQL query](https://prometheus.io/docs/prometheus/latest/querying/basics/). +- **Y-axis label**: Y axis title to display on the dashboard. +- **Unit label**: Query units, for example `req / sec`. Shown next to the value. + +Multiple metrics can be displayed on the same chart if the fields **Name**, **Type**, and **Y-axis label** match between metrics. For example, a metric with **Name** `Requests Rate`, **Type** `Business`, and **Y-axis label** `rec / sec` would display on the same chart as a second metric with the same values. A **Legend label** is suggested if this feature used. + +#### Query Variables + +GitLab supports a limited set of [CI variables](../../../ci/variables/README.html) in the Prometheus query. This is particularly useful for identifying a specific environment, for example with `CI_ENVIRONMENT_SLUG`. The supported variables are: + +- CI_ENVIRONMENT_SLUG +- KUBE_NAMESPACE + +To specify a variable in a query, enclose it in curly braces with a leading percent. For example: `%{ci_environment_slug}`. + +### Setting up alerts for Prometheus metrics **[ULTIMATE]** + +#### Managed Prometheus instances + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/6590) in [GitLab Ultimate](https://about.gitlab.com/pricing/) 11.2 for [custom metrics](#adding-additional-metrics-premium), and 11.3 for [library metrics](prometheus_library/metrics.md). + +For managed Prometheus instances using auto configuration, alerts for metrics [can be configured](#adding-additional-metrics-premium) directly in the performance dashboard. + +To set an alert, click on the alarm icon in the top right corner of the metric you want to create the alert for. A dropdown +will appear, with options to set the threshold and operator. Click **Add** to save and activate the alert. + + + +To remove the alert, click back on the alert icon for the desired metric, and click **Delete**. + +#### External Prometheus instances + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/issues/9258) in [GitLab Ultimate](https://about.gitlab.com/pricing/) 11.8. + +For manually configured Prometheus servers, a notify endpoint is provided to use with Prometheus webhooks. If you have manual configuration enabled, an **Alerts** section is added to **Settings > Integrations > Prometheus**. This contains the *URL* and *Authorization Key*. The **Reset Key** button will invalidate the key and generate a new one. + + + +To send GitLab alert notifications, copy the *URL* and *Authorization Key* into the [`webhook_configs`](https://prometheus.io/docs/alerting/configuration/#webhook_config) section of your Prometheus Alertmanager configuration: + +```yaml +receivers: + name: gitlab + webhook_configs: + - http_config: + bearer_token: 9e1cbfcd546896a9ea8be557caf13a76 + send_resolved: true + url: http://192.168.178.31:3001/root/manual_prometheus/prometheus/alerts/notify.json + ... +``` + +### Taking action on incidents **[ULTIMATE]** + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/issues/4925) in [GitLab Ultimate](https://about.gitlab.com/pricing/) 11.11. + +Alerts can be used to trigger actions, like open an issue automatically. To configure the actions: + +1. Navigate to your project's **Settings > Operations > Incidents**. +1. Enable the option to create issues. +1. Choose the [issue template](../description_templates.md) to create the issue from. +1. Optionally, select whether to send an email notification to the developers of the project. +1. Click **Save changes**. + +Once enabled, an issue will be opened automatically when an alert is triggered. The author of the issue will be the GitLab Alert Bot. To further customize the issue, you can add labels, mentions, or any other supported [quick action](../quick_actions.md) in the selected issue template. + +If the metric exceeds the threshold of the alert for over 5 minutes, an email will be sent to all [Maintainers and Owners](../../permissions.md#project-members-permissions) of the project. + ## Determining the performance impact of a merge > [Introduced][ce-10408] in GitLab 9.2. @@ -120,7 +199,7 @@ If the "No data found" screen continues to appear, it could be due to: - No successful deployments have occurred to this environment. - Prometheus does not have performance data for this environment, or the metrics are not labeled correctly. To test this, connect to the Prometheus server and - [run a query](#gitlab-prometheus-queries), replacing `$CI_ENVIRONMENT_SLUG` + [run a query](prometheus_library/kubernetes.html#metrics-supported), replacing `$CI_ENVIRONMENT_SLUG` with the name of your environment. [autodeploy]: ../../../ci/autodeploy/index.md diff --git a/doc/user/project/integrations/prometheus_library/kubernetes.md b/doc/user/project/integrations/prometheus_library/kubernetes.md index 7a45c87ada0..8b1cf1a251a 100644 --- a/doc/user/project/integrations/prometheus_library/kubernetes.md +++ b/doc/user/project/integrations/prometheus_library/kubernetes.md @@ -1,6 +1,6 @@ # Monitoring Kubernetes -> [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/8935) in GitLab 9.0 +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/8935) in GitLab 9.0. GitLab has support for automatically detecting and monitoring Kubernetes metrics. @@ -27,7 +27,7 @@ integration services must be enabled. Prometheus needs to be deployed into the cluster and configured properly in order to gather Kubernetes metrics. GitLab supports two methods for doing so: -- GitLab [integrates with Kubernetes](../../clusters/index.md), and can [deploy Prometheus into a connected cluster](../prometheus.html#managed-prometheus-on-kubernetes). It is automatically configured to collect Kubernetes metrics. +- GitLab [integrates with Kubernetes](../../clusters/index.md), and can [deploy Prometheus into a connected cluster](../prometheus.md#managed-prometheus-on-kubernetes). It is automatically configured to collect Kubernetes metrics. - To configure your own Prometheus server, you can follow the [Prometheus documentation](https://prometheus.io/docs/introduction/overview/). ## Specifying the Environment @@ -35,3 +35,25 @@ Prometheus needs to be deployed into the cluster and configured properly in orde In order to isolate and only display relevant CPU and Memory metrics for a given environment, GitLab needs a method to detect which containers it is running. Because these metrics are tracked at the container level, traditional Kubernetes labels are not available. Instead, the [Deployment](https://kubernetes.io/docs/concepts/workloads/controllers/deployment/) or [DaemonSet](https://kubernetes.io/docs/concepts/workloads/controllers/daemonset/) name should begin with [CI_ENVIRONMENT_SLUG](../../../../ci/variables/README.md#predefined-environment-variables). It can be followed by a `-` and additional content if desired. For example, a deployment name of `review-homepage-5620p5` would match the `review/homepage` environment. + +## Displaying Canary metrics **[PREMIUM]** + +> Introduced in [GitLab 10.2](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/15201). + +GitLab also gathers Kubernetes metrics for [canary deployments](../../canary_deployments.md), allowing easy comparison between the current deployed version and the canary. + +These metrics expect the [Deployment](https://kubernetes.io/docs/concepts/workloads/controllers/deployment/) or [DaemonSet](https://kubernetes.io/docs/concepts/workloads/controllers/daemonset/) name to begin with `$CI_ENVIRONMENT_SLUG-canary`, to isolate the canary metrics. + +### Canary metrics supported + +- Average Memory Usage (MB) + + ``` + avg(sum(container_memory_usage_bytes{container_name!="POD",pod_name=~"^%{ci_environment_slug}-canary-(.*)",namespace="%{kube_namespace}"}) by (job)) without (job) / count(avg(container_memory_usage_bytes{container_name!="POD",pod_name=~"^%{ci_environment_slug}-canary-(.*)",namespace="%{kube_namespace}"}) without (job)) /1024/1024 + ``` + +- Average CPU Utilization (%) + + ``` + avg(sum(rate(container_cpu_usage_seconds_total{container_name!="POD",pod_name=~"^%{ci_environment_slug}-canary-(.*)",namespace="%{kube_namespace}"}[15m])) by (job)) without (job) / count(sum(rate(container_cpu_usage_seconds_total{container_name!="POD",pod_name=~"^%{ci_environment_slug}-canary-(.*)",namespace="%{kube_namespace}"}[15m])) by (pod_name)) + ``` diff --git a/doc/user/project/integrations/prometheus_library/metrics.md b/doc/user/project/integrations/prometheus_library/metrics.md index 37a5388d2fc..7ace0ec5a93 100644 --- a/doc/user/project/integrations/prometheus_library/metrics.md +++ b/doc/user/project/integrations/prometheus_library/metrics.md @@ -1 +1,5 @@ +--- +redirect_to: 'index.md' +--- + This document was moved to [another location](index.md). diff --git a/doc/user/project/integrations/prometheus_library/nginx_ingress.md b/doc/user/project/integrations/prometheus_library/nginx_ingress.md index b7601f26802..de7fc93f0a4 100644 --- a/doc/user/project/integrations/prometheus_library/nginx_ingress.md +++ b/doc/user/project/integrations/prometheus_library/nginx_ingress.md @@ -30,7 +30,7 @@ For other deployments, there is [some configuration](#manually-setting-up-nginx- ### About managed NGINX Ingress deployments -NGINX Ingress is deployed into the `gitlab-managed-apps` namespace, using the [official Helm chart](https://github.com/kubernetes/charts/tree/master/stable/nginx-ingress). NGINX Ingress will be [externally reachable via the Load Balancer's IP](../../clusters/index.md#getting-the-external-ip-address). +NGINX Ingress is deployed into the `gitlab-managed-apps` namespace, using the [official Helm chart](https://github.com/kubernetes/charts/tree/master/stable/nginx-ingress). NGINX Ingress will be [externally reachable via the Load Balancer's Endpoint](../../clusters/index.md#getting-the-external-endpoint). NGINX is configured for Prometheus monitoring, by setting: diff --git a/doc/user/project/integrations/prometheus_library/nginx_ingress_vts.md b/doc/user/project/integrations/prometheus_library/nginx_ingress_vts.md index 081eb8732ad..31ac53c0d14 100644 --- a/doc/user/project/integrations/prometheus_library/nginx_ingress_vts.md +++ b/doc/user/project/integrations/prometheus_library/nginx_ingress_vts.md @@ -30,7 +30,7 @@ For other deployments, there is [some configuration](#manually-setting-up-nginx- ### About managed NGINX Ingress deployments -NGINX Ingress is deployed into the `gitlab-managed-apps` namespace, using the [official Helm chart](https://github.com/kubernetes/charts/tree/master/stable/nginx-ingress). NGINX Ingress will be [externally reachable via the Load Balancer's IP](../../clusters/index.md#getting-the-external-ip-address). +NGINX Ingress is deployed into the `gitlab-managed-apps` namespace, using the [official Helm chart](https://github.com/kubernetes/charts/tree/master/stable/nginx-ingress). NGINX Ingress will be [externally reachable via the Load Balancer's Endpoint](../../clusters/index.md#getting-the-external-endpoint). NGINX is configured for Prometheus monitoring, by setting: diff --git a/doc/user/project/integrations/redmine.md b/doc/user/project/integrations/redmine.md index 8112aa21859..bac7eecfce4 100644 --- a/doc/user/project/integrations/redmine.md +++ b/doc/user/project/integrations/redmine.md @@ -18,9 +18,7 @@  -1. To disable the internal issue tracking system in a project, navigate to the General page, expand [Permissions](../settings/index.md#sharing-and-permissions), and slide the Issues switch invalid. - -  +1. To disable the internal issue tracking system in a project, navigate to the General page, expand the [permissions](../settings/index.md#sharing-and-permissions) section and switch the **Issues** toggle to disabled. ## Referencing issues in Redmine diff --git a/doc/user/project/integrations/slack_slash_commands.md b/doc/user/project/integrations/slack_slash_commands.md index c267da69bb3..371e78ca3a4 100644 --- a/doc/user/project/integrations/slack_slash_commands.md +++ b/doc/user/project/integrations/slack_slash_commands.md @@ -1,10 +1,15 @@ -# Slack slash commands +# Slack slash commands **[CORE ONLY]** -> Introduced in GitLab 8.15 +> Introduced in GitLab 8.15. -Slack slash commands allow you to control GitLab and view content right inside Slack, without having to leave it. This requires configurations in both Slack and GitLab. +Slack slash commands allow you to control GitLab and view content right inside +Slack, without having to leave it. This requires configurations in both Slack and GitLab. -> Note: GitLab can also send events (e.g. issue created) to Slack as notifications. This is the separately configured [Slack Notifications Service](slack.md). +GitLab can also send events (e.g., `issue created`) to Slack as notifications. +This is the separately configured [Slack Notifications Service](slack.md). + +NOTE: **Note:** +For GitLab.com, use the [Slack app](gitlab_slack_application.md) instead. ## Configuration diff --git a/doc/user/project/integrations/webhooks.md b/doc/user/project/integrations/webhooks.md index c3fc6d4b859..a0f500a939f 100644 --- a/doc/user/project/integrations/webhooks.md +++ b/doc/user/project/integrations/webhooks.md @@ -48,7 +48,7 @@ Navigate to the webhooks page by going to your project's ## Use-cases - You can set up a webhook in GitLab to send a notification to - [Slack](https://api.slack.com/incoming-webhooks) every time a build fails, for example + [Slack](https://api.slack.com/incoming-webhooks) every time a job fails. - You can [integrate with Twilio to be notified via SMS](https://www.datadoghq.com/blog/send-alerts-sms-customizable-webhooks-twilio/) every time an issue is created for a specific project or group within GitLab - You can use them to [automatically assign labels to merge requests](https://about.gitlab.com/2016/08/19/applying-gitlab-labels-automatically/). @@ -321,8 +321,14 @@ X-Gitlab-Event: Issue Hook "group_id": 41 }], "changes": { - "updated_by_id": [null, 1], - "updated_at": ["2017-09-15 16:50:55 UTC", "2017-09-15 16:52:00 UTC"], + "updated_by_id": { + "previous": null, + "current": 1 + }, + "updated_at": { + "previous": "2017-09-15 16:50:55 UTC", + "current": "2017-09-15 16:52:00 UTC" + }, "labels": { "previous": [{ "id": 206, @@ -851,8 +857,14 @@ X-Gitlab-Event: Merge Request Hook "group_id": 41 }], "changes": { - "updated_by_id": [null, 1], - "updated_at": ["2017-09-15 16:50:55 UTC", "2017-09-15 16:52:00 UTC"], + "updated_by_id": { + "previous": null, + "current": 1 + }, + "updated_at": { + "previous": "2017-09-15 16:50:55 UTC", + "current":"2017-09-15 16:52:00 UTC" + }, "labels": { "previous": [{ "id": 206, @@ -1041,7 +1053,12 @@ X-Gitlab-Event: Pipeline Hook "username": "root", "avatar_url": "http://www.gravatar.com/avatar/e32bd13e2add097461cb96824b7a829c?s=80\u0026d=identicon" }, - "runner": null, + "runner": { + "id":380987, + "description":"shared-runners-manager-6.gitlab.com", + "active":true, + "is_shared":true + }, "artifacts_file":{ "filename": null, "size": null @@ -1062,7 +1079,12 @@ X-Gitlab-Event: Pipeline Hook "username": "root", "avatar_url": "http://www.gravatar.com/avatar/e32bd13e2add097461cb96824b7a829c?s=80\u0026d=identicon" }, - "runner": null, + "runner": { + "id":380987, + "description":"shared-runners-manager-6.gitlab.com", + "active":true, + "is_shared":true + }, "artifacts_file":{ "filename": null, "size": null @@ -1083,7 +1105,12 @@ X-Gitlab-Event: Pipeline Hook "username": "root", "avatar_url": "http://www.gravatar.com/avatar/e32bd13e2add097461cb96824b7a829c?s=80\u0026d=identicon" }, - "runner": null, + "runner": { + "id":380987, + "description":"shared-runners-manager-6.gitlab.com", + "active":true, + "is_shared":true + }, "artifacts_file":{ "filename": null, "size": null @@ -1114,34 +1141,34 @@ X-Gitlab-Event: Pipeline Hook } ``` -### Build events +### Job events -Triggered on status change of a Build. +Triggered on status change of a job. **Request Header**: ``` -X-Gitlab-Event: Build Hook +X-Gitlab-Event: Job Hook ``` **Request Body**: ```json { - "object_kind": "build", + "object_kind": "job", "ref": "gitlab-script-trigger", "tag": false, "before_sha": "2293ada6b400935a1378653304eaf6221e0fdb8f", "sha": "2293ada6b400935a1378653304eaf6221e0fdb8f", - "build_id": 1977, - "build_name": "test", - "build_stage": "test", - "build_status": "created", - "build_started_at": null, - "build_finished_at": null, - "build_duration": null, - "build_allow_failure": false, - "build_failure_reason": "script_failure", + "job_id": 1977, + "job_name": "test", + "job_stage": "test", + "job_status": "created", + "job_started_at": null, + "job_finished_at": null, + "job_duration": null, + "job_allow_failure": false, + "job_failure_reason": "script_failure", "project_id": 380, "project_name": "gitlab-org/gitlab-test", "user": { diff --git a/doc/user/project/integrations/youtrack.md b/doc/user/project/integrations/youtrack.md new file mode 100644 index 00000000000..81c148e41fd --- /dev/null +++ b/doc/user/project/integrations/youtrack.md @@ -0,0 +1,38 @@ +# YouTrack Service + +JetBrains [YouTrack](https://www.jetbrains.com/help/youtrack/standalone/YouTrack-Documentation.html) is a web-based issue tracking and project management platform. + +You can configure YouTrack as an [External Issue Tracker](../../../integration/external-issue-tracker.md) in GitLab. + +## Enable the YouTrack integration + +To enable YouTrack integration in a project: + +1. Navigate to the project's **Settings > [Integrations](project_services.md#accessing-the-project-services)** page. +1. Click the **YouTrack** service, ensure it's active, and enter the required details on the page as described in the table below. + + | Field | Description | + |:----------------|:------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| + | **Description** | Name for the issue tracker (to differentiate between instances, for example). | + | **Project url** | URL to the project in YouTrack which is being linked to this GitLab project. | + | **Issues url** | URL to the issue in YouTrack project that is linked to this GitLab project. Note that the **Issues url** requires `:id` in the URL. This ID is used by GitLab as a placeholder to replace the issue number. | + +1. Click the **Save changes** button. + +Once you have configured and enabled YouTrack, you'll see the YouTrack link on the GitLab project pages that takes you to the appropriate YouTrack project. + +## Disable the internal issue tracker + +To disable the internal issue tracker in a project: + +1. Navigate to the project's **Settings > General** page. +1. Expand the [permissions section](../settings/index.md#sharing-and-permissions) and switch the **Issues** toggle to disabled. + +## Referencing YouTrack issues in GitLab + +Issues in YouTrack can be referenced as `<PROJECT>-<ID>`. `<PROJECT>` +must start with a letter and is followed by letters, numbers, or underscores. +`<ID>` is a number. An example reference is `YT-101`, `Api_32-143` or `gl-030`. + +References to `<PROJECT>-<ID>` in merge requests, commits, or comments are automatically linked to the YouTrack issue URL. +For more information, see the [External Issue Tracker](../../../integration/external-issue-tracker.md) documentation. diff --git a/doc/user/project/issue_board.md b/doc/user/project/issue_board.md index 0bd565547c3..31020de5208 100644 --- a/doc/user/project/issue_board.md +++ b/doc/user/project/issue_board.md @@ -28,11 +28,11 @@ Issue Boards** (version introduced in GitLab 8.11 - August 2016). ### Advanced features of Issue Boards With [GitLab Starter](https://about.gitlab.com/pricing/), you can create -[multiple issue boards](#multiple-issue-boards) for a given project. **[STARTER]** +[multiple issue boards](#multiple-issue-boards-starter) for a given project. **[STARTER]** With [GitLab Premium](https://about.gitlab.com/pricing/), you can also create multiple -issue boards for your groups, and add lists for [assignees](#assignee-lists) and -[milestones](#milestone-lists). **[PREMIUM]** +issue boards for your groups, and add lists for [assignees](#assignee-lists-premium) and +[milestones](#milestone-lists-premium). **[PREMIUM]** Check all the [advanced features of Issue Boards](#gitlab-enterprise-features-for-issue-boards) below. @@ -42,7 +42,7 @@ below. ## How it works The Issue Board builds on GitLab's existing -[issue tracking functionality](issues/index.md#issue-tracker) and +[issue tracking functionality](issues/index.md#issues-list) and leverages the power of [labels](labels.md) by utilizing them as lists of the scrum board. With the Issue Board you can have a different view of your issues while @@ -97,7 +97,7 @@ If we have the labels "**backend**", "**frontend**", "**staging**", and ### Use cases for Multiple Issue Boards -With [Multiple Issue Boards](#multiple-issue-boards), available only in +With [Multiple Issue Boards](#multiple-issue-boards-starter), available only in [GitLab Enterprise Edition](https://about.gitlab.com/pricing/), each team can have their own board to organize their workflow individually. @@ -151,7 +151,7 @@ Create lists for each of your team members and quickly drag-and-drop issues onto ## Permissions -[Developers and up](../permissions.md) can use all the functionality of the +[Reporters and up](../permissions.md) can use all the functionality of the Issue Board, that is, create or delete lists and drag issues from one list to another. ## GitLab Enterprise features for Issue Boards @@ -173,10 +173,14 @@ or in situations where a repository is used to host the code of multiple products. Clicking on the current board name in the upper left corner will reveal a -menu from where you can create another Issue Board and rename or delete the -existing one. +menu from where you can create another Issue Board or delete the existing one. Using the search box at the top of the menu, you can filter the listed boards. +When you have 10 or more boards available, a "Recent" section is also shown in the menu. +These are shortcuts to your last 4 visited boards. + + + When you're revisiting an issue board in a project or group with multiple boards, GitLab will automatically load the last board you visited. @@ -184,8 +188,6 @@ NOTE: **Note:** The Multiple Issue Boards feature is available for **projects in GitLab Starter Edition** and for **groups in GitLab Premium Edition**. - - ### Configurable Issue Boards **[STARTER]** > Introduced in [GitLab Starter Edition 10.2](https://about.gitlab.com/2017/11/22/gitlab-10-2-released/#issue-boards-configuration). @@ -218,7 +220,7 @@ Click the button at the top right to toggle focus mode on and off. In focus mode The top of each list indicates the sum of issue weights for the issues that belong to that list. This is useful when using boards for capacity allocation, -especially in combination with [assignee lists](#assignee-lists). +especially in combination with [assignee lists](#assignee-lists-premium).  diff --git a/doc/user/project/issues/confidential_issues.md b/doc/user/project/issues/confidential_issues.md index 8eada25234f..2c755e0fb4d 100644 --- a/doc/user/project/issues/confidential_issues.md +++ b/doc/user/project/issues/confidential_issues.md @@ -1,6 +1,6 @@ # Confidential issues -> [Introduced][ce-3282] in GitLab 8.6. +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/3282) in GitLab 8.6. Confidential issues are issues visible only to members of a project with [sufficient permissions](#permissions-and-access-to-confidential-issues). @@ -67,7 +67,7 @@ There is also an indicator on the sidebar denoting confidentiality. There are two kinds of level access for confidential issues. The general rule is that confidential issues are visible only to members of a project with at -least [Reporter access][permissions]. However, a guest user can also create +least [Reporter access](../../permissions.md#project-members-permissions). However, a guest user can also create confidential issues, but can only view the ones that they created themselves. Confidential issues are also hidden in search results for unprivileged users. @@ -77,6 +77,3 @@ project's search results respectively. | Maintainer access | Guest access | | :-----------: | :----------: | |  |  | - -[permissions]: ../../permissions.md#project -[ce-3282]: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/3282 diff --git a/doc/user/project/issues/create_new_issue.md b/doc/user/project/issues/create_new_issue.md index 40040e44d64..c2916c79876 100644 --- a/doc/user/project/issues/create_new_issue.md +++ b/doc/user/project/issues/create_new_issue.md @@ -7,7 +7,7 @@ the information illustrated on the image below.  -Read through the [issues functionalities documentation](issues_functionalities.md#issues-functionalities) +Read through the [issue data and actions documentation](issue_data_and_actions.md#parts-of-an-issue) to understand these fields one by one. ## New issue from the Issue Tracker @@ -65,6 +65,26 @@ In GitLab 11.7, we updated the format of the generated email address. However the older format is still supported, allowing existing aliases or contacts to continue working._ +## New issue via Service Desk **[PREMIUM]** + +Enable [Service Desk](../service_desk.md) to your project and offer email support. +By doing so, when your customer sends a new email, a new issue can be created in +the appropriate project and followed up from there. + +## New issue from the group-level Issue Tracker + +Head to the Group dashboard and click "Issues" in the sidebar to visit the Issue Tracker +for all projects in your Group. Select the project you'd like to add an issue for +using the dropdown button at the top-right of the page. + + + +We'll keep track of the project you selected most recently, and use it as the default +for your next visit. This should save you a lot of time and clicks, if you mostly +create issues for the same project. + + + ## New issue via URL with prefilled fields You can link directly to the new issue page for a given project, with prefilled diff --git a/doc/user/project/issues/crosslinking_issues.md b/doc/user/project/issues/crosslinking_issues.md index 786d1c81b1b..ff5b1f2ce50 100644 --- a/doc/user/project/issues/crosslinking_issues.md +++ b/doc/user/project/issues/crosslinking_issues.md @@ -48,13 +48,12 @@ issues in merge requests. ## From Merge Requests -Mentioning issues in merge request comments work exactly the same way +Mentioning issues in merge request comments works exactly the same way as they do for [related issues](#from-related-issues). -When you mention an issue in a merge request description, you can either -[close the issue as soon as the merge request is merged](closing_issues.md#via-merge-request), -or simply link both issue and merge request as described in the -[closing issues documentation](closing_issues.md#from-related-issues). +When you mention an issue in a merge request description, it will simply +[link the issue and merge request together](#from-related-issues). Additionally, +you can also [set an issue to close as soon as the merge request is merged](closing_issues.md#via-merge-request).  diff --git a/doc/user/project/issues/csv_export.md b/doc/user/project/issues/csv_export.md new file mode 100644 index 00000000000..56b94585672 --- /dev/null +++ b/doc/user/project/issues/csv_export.md @@ -0,0 +1,77 @@ +# Export Issues to CSV **[STARTER]** + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/1126) in [GitLab Starter 9.0](https://about.gitlab.com/2017/03/22/gitlab-9-0-released/#export-issues-ees-eep). + +Issues can be exported as CSV from GitLab and are sent to your default notification email as an attachment. + +## Overview + +**Export Issues to CSV** enables you and your team to export all the data collected from issues into +a **[comma-separated values](https://en.wikipedia.org/wiki/Comma-separated_values)** (CSV) file, +which stores tabular data in plain text. + +> _CSVs are a handy way of getting data from one program to another where one program cannot read the other ones normal output._ [Ref](https://www.quora.com/What-is-a-CSV-file-and-its-uses) + +CSV files can be used with any plotter or spreadsheet-based program, such as Microsoft Excel, +Open Office Calc, or Google Spreadsheets. + +## Use cases + +Among numerous use cases for exporting issues for CSV, we can name a few: + +- Make a snapshot of issues for offline analysis or to communicate with other teams who may not be in GitLab +- Create diagrams, graphs, and charts from the CSV data +- Present the data in any other format for auditing or sharing reasons +- Import the issues elsewhere to a system outside of GitLab +- Long-term issues' data analysis with multiple snapshots created along the time +- Use the long-term data to gather relevant feedback given in the issues, and improve your product based on real metrics + +## Choosing which issues to include + +From the issues page you can narrow down which issues to export using the search bar, along with the All/Open/Closed tabs. All issues returned will be exported, including those not shown on the first page. + + + +You will be asked to confirm the number of issues and email address for the export, after which the email will begin being prepared. + + + +## Sorting + +Exported issues are always sorted by `Issue ID`. + +## Format + +> **Time Estimate** and **Time Spent** columns were [introduced](https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/2627) in GitLab Starter 10.0. +> +> **Weight** and **Locked** columns were [introduced](https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/5300) in GitLab Starter 10.8. + +Data will be encoded with a comma as the column delimiter, with `"` used to quote fields if needed, and newlines to separate rows. The first row will be the headers, which are listed in the following table along with a description of the values: + + +| Column | Description | +|---------|-------------| +| Issue ID | Issue `iid` | +| URL | A link to the issue on GitLab | +| Title | Issue `title` | +| State | `Open` or `Closed` | +| Description | Issue `description` | +| Author | Full name of the issue author | +| Author Username | Username of the author, with the `@` symbol omitted | +| Assignee | Full name of the issue assignee | +| Assignee Username | Username of the author, with the `@` symbol omitted | +| Confidential | `Yes` or `No` | +| Locked | `Yes` or `No` | +| Due Date | Formated as `YYYY-MM-DD` | +| Created At (UTC) | Formated as `YYYY-MM-DD HH:MM:SS` | +| Updated At (UTC) | Formated as `YYYY-MM-DD HH:MM:SS` | +| Milestone | Title of the issue milestone | +| Weight | Issue weight | +| Labels | Title of any labels joined with a `,` | +| Time Estimate | [Time estimate](../../../workflow/time_tracking.md#estimates) in seconds | +| Time Spent | [Time spent](../../../workflow/time_tracking.md#time-spent) in seconds | + + +## Limitations + +As the issues will be sent as an email attachment, there is a limit on how much data can be exported. Currently this limit is 20MB to ensure successful delivery across a range of email providers. If this limit is reached we suggest narrowing the search before export, perhaps by exporting open and closed issues separately. diff --git a/doc/user/project/issues/csv_import.md b/doc/user/project/issues/csv_import.md index 032e3a73ad0..b0b1cfcfdf7 100644 --- a/doc/user/project/issues/csv_import.md +++ b/doc/user/project/issues/csv_import.md @@ -1,17 +1,23 @@ -# Importing Issues from CSV +# Importing issues from CSV > [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/23532) in GitLab 11.7. -Issues can be imported to a project by uploading a CSV file. Supported fields are -`title` and `description`. +Issues can be imported to a project by uploading a CSV file with the columns +`title` and `description`, in that order. The user uploading the CSV file will be set as the author of the imported issues. > **Note:** A permission level of `Developer` or higher is required to import issues. +## Prepare for the import + +- Consider importing a test file containing only a few issues. There is no way to undo a large import without using the GitLab API. +- Ensure your CSV file meets the [file format](#csv-file-format) requirements. + +## Import the file + To import issues: -1. Ensure your CSV file meets the [file format](#csv-file-format) requirements. 1. Navigate to a project's Issues list page. 1. If existing issues are present, click the import icon at the top right, next to the **Edit issues** button. 1. For a project without any issues, click the button labeled **Import CSV** in the middle of the page. @@ -20,11 +26,11 @@ To import issues: The file is processed in the background and a notification email is sent to you once the import is completed. -## CSV File Format +## CSV file format ### Header row -CSV files must contain a header row beginning with at least two columns, `title` and `description`, in that order. +CSV files must contain a header row where the first column header is `title` and the second is `description`. If additional columns are present, they will be ignored. ### Column separator @@ -53,7 +59,7 @@ The limit depends on the configuration value of Max Attachment Size for the GitL For GitLab.com, it is set to 10 MB. -## Sample Data +## Sample data ```csv title,description diff --git a/doc/user/project/issues/due_dates.md b/doc/user/project/issues/due_dates.md index 7972c14c1c4..987c16dfab6 100644 --- a/doc/user/project/issues/due_dates.md +++ b/doc/user/project/issues/due_dates.md @@ -1,11 +1,11 @@ # Due dates -> [Introduced][ce-3614] in GitLab 8.7. +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/3614) in GitLab 8.7. Please read through the [GitLab Issue Documentation](index.md) for an overview on GitLab Issues. Due dates can be used in issues to keep track of deadlines and make sure -features are shipped on time. Due dates require at least [Reporter permissions][permissions] +features are shipped on time. Due dates require at least [Reporter permissions](../../permissions.md#project-members-permissions) to be able to edit them. On the contrary, they can be seen by everybody. ## Setting a due date @@ -47,6 +47,3 @@ on the _Subscribe to calendar_ button on the following pages: GitLab header - on the **Project Issues** page - on the **Group Issues** page - -[ce-3614]: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/3614 -[permissions]: ../../permissions.md#project diff --git a/doc/user/project/issues/img/create_issue_from_group_level_issue_tracker.png b/doc/user/project/issues/img/create_issue_from_group_level_issue_tracker.png Binary files differnew file mode 100644 index 00000000000..8996490ae63 --- /dev/null +++ b/doc/user/project/issues/img/create_issue_from_group_level_issue_tracker.png diff --git a/doc/user/project/issues/img/csv_export_button.png b/doc/user/project/issues/img/csv_export_button.png Binary files differnew file mode 100644 index 00000000000..f9fcfd71c2f --- /dev/null +++ b/doc/user/project/issues/img/csv_export_button.png diff --git a/doc/user/project/issues/img/csv_export_modal.png b/doc/user/project/issues/img/csv_export_modal.png Binary files differnew file mode 100644 index 00000000000..f988deec966 --- /dev/null +++ b/doc/user/project/issues/img/csv_export_modal.png diff --git a/doc/user/project/issues/img/group_issues_list_view.png b/doc/user/project/issues/img/group_issues_list_view.png Binary files differdeleted file mode 100644 index c951a9e2dcd..00000000000 --- a/doc/user/project/issues/img/group_issues_list_view.png +++ /dev/null diff --git a/doc/user/project/issues/img/issue_template.png b/doc/user/project/issues/img/issue_template.png Binary files differdeleted file mode 100644 index 6cb2c07d27e..00000000000 --- a/doc/user/project/issues/img/issue_template.png +++ /dev/null diff --git a/doc/user/project/issues/img/multiple_assignees.gif b/doc/user/project/issues/img/multiple_assignees.gif Binary files differnew file mode 100644 index 00000000000..055a0efd9ae --- /dev/null +++ b/doc/user/project/issues/img/multiple_assignees.gif diff --git a/doc/user/project/issues/img/multiple_assignees_for_issues.png b/doc/user/project/issues/img/multiple_assignees_for_issues.png Binary files differnew file mode 100644 index 00000000000..e8ae37d427d --- /dev/null +++ b/doc/user/project/issues/img/multiple_assignees_for_issues.png diff --git a/doc/user/project/issues/img/related_issues_add.png b/doc/user/project/issues/img/related_issues_add.png Binary files differnew file mode 100644 index 00000000000..f59d2335386 --- /dev/null +++ b/doc/user/project/issues/img/related_issues_add.png diff --git a/doc/user/project/issues/img/related_issues_remove.png b/doc/user/project/issues/img/related_issues_remove.png Binary files differnew file mode 100644 index 00000000000..be2ec59e61b --- /dev/null +++ b/doc/user/project/issues/img/related_issues_remove.png diff --git a/doc/user/project/issues/img/select_project_from_group_level_issue_tracker.png b/doc/user/project/issues/img/select_project_from_group_level_issue_tracker.png Binary files differnew file mode 100644 index 00000000000..97d93620b2a --- /dev/null +++ b/doc/user/project/issues/img/select_project_from_group_level_issue_tracker.png diff --git a/doc/user/project/issues/img/similar_issues.png b/doc/user/project/issues/img/similar_issues.png Binary files differindex 0dfb5b00e02..c5b7902b2ac 100644 --- a/doc/user/project/issues/img/similar_issues.png +++ b/doc/user/project/issues/img/similar_issues.png diff --git a/doc/user/project/issues/index.md b/doc/user/project/issues/index.md index 5a3ac9c175b..94865ad46ee 100644 --- a/doc/user/project/issues/index.md +++ b/doc/user/project/issues/index.md @@ -1,170 +1,135 @@ # Issues -The GitLab Issue Tracker is an advanced and complete tool -for tracking the evolution of a new idea or the process -of solving a problem. +Issues are the fundamental medium for collaborating on ideas and planning work in GitLab. -It allows you, your team, and your collaborators to share -and discuss proposals before and while implementing them. +## Overview -GitLab Issues and the GitLab Issue Tracker are available in all -[GitLab Products](https://about.gitlab.com/pricing/) as -part of the [GitLab Workflow](https://about.gitlab.com/2016/10/25/gitlab-workflow-an-overview/). +The GitLab issue tracker is an advanced tool for collaboratively developing ideas, solving problems, and planning work. -## Use cases +Issues can allow you, your team, and your collaborators to share and discuss proposals before and during their implementation. +However, they can be used for a variety of other purposes, customized to your needs and workflow. -Issues can have endless applications. Just to exemplify, these are -some cases for which creating issues are most used: +Issues are always associated with a specific project, but if you have multiple projects in a group, +you can also view all the issues collectively at the group level. + +**Common use cases include:** - Discussing the implementation of a new idea -- Submitting feature proposals -- Asking questions -- Reporting bugs and malfunction -- Obtaining support +- Tracking tasks and work status +- Accepting feature proposals, questions, support requests, or bug reports - Elaborating new code implementations See also the blog post "[Always start a discussion with an issue](https://about.gitlab.com/2016/03/03/start-with-an-issue/)". -### Keep private things private - -For instance, let's assume you have a public project but want to start a discussion on something -you don't want to be public. With [Confidential Issues](#confidential-issues), -you can discuss private matters among the project members, and still keep -your project public, open to collaboration. - -### Streamline collaboration - -With [Multiple Assignees for Issues](https://docs.gitlab.com/ee/user/project/issues/multiple_assignees_for_issues.html), -available in [GitLab Starter](https://about.gitlab.com/pricing/) -you can streamline collaboration and allow shared responsibilities to be clearly displayed. -All assignees are shown across your workflows and receive notifications (as they -would as single assignees), simplifying communication and ownership. - -### Consistent collaboration - -Create [issue templates](#issue-templates) to make collaboration consistent and -containing all information you need. For example, you can create a template -for feature proposals and another one for bug reports. - -## Issue Tracker - -The Issue Tracker is the collection of opened and closed issues created in a project. -It is available for all projects, from the moment the project is created. - -Find the issue tracker by navigating to your **Project's homepage** > **Issues**. - -### Issues per project - -When you access your project's issues, GitLab will present them in a list, -and you can use the tabs available to quickly filter by open and closed issues. - - - -You can also [search and filter](../../search/index.md#issues-and-merge-requests-per-project) the results more deeply with GitLab's search capacities. - -### Issues per group - -View issues in all projects in the group, including all projects of all descendant subgroups of the group. Navigate to **Group > Issues** to view these issues. This view also has the open and closed issues tabs. - - - -## GitLab Issues Functionalities - -The image bellow illustrates how an issue looks like: +## Parts of an issue + +Issues contain a variety of content and metadata, enabling a large range of flexibility in how they are used. Each issue can contain the following attributes, though some items may remain unset. + +<table class="borderless-table fixed-table"> +<tr> + <td> + <ul> + <li>Content</li> + <ul> + <li>Title</li> + <li>Description and tasks</li> + <li>Comments and other activity</li> + </ul> + <li>People</li> + <ul> + <li>Author</li> + <li>Assignee(s)</li> + </ul> + <li>State</li> + <ul> + <li>Status (open/closed)</li> + <li>Confidentiality</li> + <li>Tasks (completed vs. outstanding)</li> + </ul> + </ul> + </td> + <td> + <ul> + <li>Planning and tracking</li> + <ul> + <li>Milestone</li> + <li>Due date</li> + <li>Weight</li> + <li>Time tracking</li> + <li>Labels</li> + <li>Votes</li> + <li>Reaction emoji</li> + <li>Linked issues</li> + <li>Assigned epic</li> + <li>Unique issue number and URL</li> + </ul> + </ul> + </td> +</tr> +</table> + +## Viewing and managing issues + +While you can view and manage the full detail of an issue at its URL, you can also work with multiple issues at a time using the Issues List, Issue Boards, Epics **[ULTIMATE]**, and issue references. + +### Issue page  -Learn more about it on the [GitLab Issues Functionalities documentation](issues_functionalities.md). - -## New issue +On an issue’s page, you can view all aspects of the issue, and you can also modify them if you you have the necessary [permissions](../../permissions.md). -Read through the [documentation on creating issues](create_new_issue.md). +For more information, see the [Issue Data and Actions](issue_data_and_actions.md) page. -## Closing issues +### Issues list -Learn distinct ways to [close issues](closing_issues.md) in GitLab. - -## Moving issues - -Read through the [documentation on moving issues](moving_issues.md). - -## Deleting issues + -Read through the [documentation on deleting issues](deleting_issues.md) +On the Issues List, you can view all issues in the current project, or from multiple projects when opening the Issues List from the higher-level group context. Filter the issue list by [any search query](../../search/index.md#issues-and-merge-requests-per-project) and/or specific metadata, such as label(s), assignees(s), status, and more. From this view, you can also make certain changes [in bulk](../bulk_editing.md) to the displayed issues. -## Create a merge request from an issue +For more information, see the [Issue Data and Actions](issue_data_and_actions.md) page. -Learn more about it on the [GitLab Issues Functionalities documentation](issues_functionalities.md#18-new-merge-request). +### Issue boards -## Search for an issue + -Learn how to [find an issue](../../search/index.md) by searching for and filtering them. +Issue boards are Kanban boards with columns that display issues based on their labels or their assignees**[PREMIUM]**. They offer the flexibility to manage issues using highly customizable workflows. -## Advanced features +You can reorder issues within a column, or drag an issue card to another column; its associated label or assignee will change to match that of the new column. The entire board can also be filtered to only include issues from a certain milestone or an overarching label. -### Confidential Issues +For more information, see the [Issue Boards](../issue_board.md) page. -Whenever you want to keep the discussion presented in a -issue within your team only, you can make that -[issue confidential](confidential_issues.md). Even if your project -is public, that issue will be preserved. The browser will -respond with a 404 error whenever someone who is not a project -member with at least [Reporter level](../../permissions.md#project) tries to -access that issue's URL. +### Epics **[ULTIMATE]** -Learn more about them on the [confidential issues documentation](confidential_issues.md). +Epics let you manage your portfolio of projects more efficiently and with less effort by tracking groups of issues that share a theme, across projects and milestones. -### Issue templates +For more information, see the [Epics](../../group/epics/index.md) page. -Create templates for every new issue. They will be available from -the dropdown menu **Choose a template** when you create a new issue: +### Related issues **[STARTER]** - +You can mark two issues as related, so that when viewing each one, the other is always listed in its Related Issues section. This can help display important context, such as past work, dependencies, or duplicates. -Learn more about them on the [issue templates documentation](../../project/description_templates.md#creating-issue-templates). +For more information, see [Related Issues](related_issues.md). ### Crosslinking issues -Learn more about [crosslinking](crosslinking_issues.md) issues and merge requests. - -### Issue Board - -The [GitLab Issue Board](https://about.gitlab.com/features/issueboard/) is a way to -enhance your workflow by organizing and prioritizing issues in GitLab. - - - -Find GitLab Issue Boards by navigating to your **Project's Dashboard** > **Issues** > **Board**. +When you reference an issue from another issue or merge request by including its URL or ID, the referenced issue displays a message in the Activity stream about the reference, with a link to the other issue or MR. -Read through the documentation for [Issue Boards](../issue_board.md) -to find out more about this feature. +For more information, see [Crosslinking issues](crosslinking_issues.md). -With [GitLab Starter](https://about.gitlab.com/pricing/), you can also -create various boards per project with [Multiple Issue Boards](https://docs.gitlab.com/ee/user/project/issue_board.html#multiple-issue-boards). +## Issue actions -### Import Issues from CSV +- [Create an issue](create_new_issue.md) +- [Create an issue from a template](../../project/description_templates.md#using-the-templates) +- [Close an issue](closing_issues.md) +- [Move an issue](moving_issues.md) +- [Delete an issue](deleting_issues.md) +- [Create a merge request from an issue](issue_data_and_actions.md#18-new-merge-request) -You can import a CSV file containing issue titles and descriptions to create -a batch of issues simultaneously. +## Advanced issue management -When you navigate to the Issues list page, an import button is displayed. - -For further details, see [Importing issues from CSV](csv_import.md) - -### External Issue Tracker - -Alternatively to GitLab's built-in Issue Tracker, you can also use an [external -tracker](../../../integration/external-issue-tracker.md) such as Jira, Redmine, +- [Bulk edit issues](../bulk_editing.md) - From the Issues List, select multiple issues in order to change their status, assignee, milestone, or labels in bulk. +- [Import issues](csv_import.md) +- [Export issues](csv_export.md) **[STARTER]** +- [Issues API](../../../api/issues.md) +- Configure an [external issue tracker](../../../integration/external-issue-tracker.md) such as Jira, Redmine, or Bugzilla. - -### Issue API - -See the [API documentation](../../../api/issues.md). - -### Bulk editing issues - -See the [bulk editing issues](../../project/bulk_editing.md) page. - -### Similar issues - -See the [similar issues](similar_issues.md) page. diff --git a/doc/user/project/issues/issues_functionalities.md b/doc/user/project/issues/issue_data_and_actions.md index 27b9dc51760..fc11c0251e0 100644 --- a/doc/user/project/issues/issues_functionalities.md +++ b/doc/user/project/issues/issue_data_and_actions.md @@ -1,8 +1,8 @@ -# GitLab Issues Functionalities +# Issue Data and Actions Please read through the [GitLab Issue Documentation](index.md) for an overview on GitLab Issues. -## Issues Functionalities +## Parts of an Issue The image below illustrates what an issue looks like: @@ -50,7 +50,7 @@ where there is shared ownership of an issue. In [GitLab Starter](https://about.gitlab.com/pricing/), you can assign multiple people to an issue. -Learn more in the [Multiple Assignees documentation](https://docs.gitlab.com/ee/user/project/issues/multiple_assignees_for_issues.html). +Learn more in the [Multiple Assignees documentation](multiple_assignees_for_issues.md). #### 4. Milestone @@ -77,7 +77,7 @@ can be changed as many times as needed. Categorize issues by giving them [labels](../labels.md). They help to organize workflows, and they enable you to work with the -[GitLab Issue Board](index.md#issue-board). +[GitLab Issue Board](index.md#issue-boards). Group Labels, which allow you to use the same labels for a group of projects, can be also given to issues. They work exactly the same, @@ -90,7 +90,7 @@ If a label doesn't exist yet, you can click **Edit**, and it opens a dropdown me - Assign a weight. Larger values are used to indicate more effort is required to complete the issue. Only positive values or zero are allowed. -Learn more in the [Issue Weight documentation](https://docs.gitlab.com/ee/workflow/issue_weight.html). +Learn more in the [Issue Weight documentation](../../../workflow/issue_weight.md). #### 9. Participants @@ -103,7 +103,7 @@ Learn more in the [Issue Weight documentation](https://docs.gitlab.com/ee/workfl - Unsubscribe: if you are receiving notifications on that issue but no longer want to receive them, unsubscribe from it. -Read more in the [notifications documentation](../../../workflow/notifications.md#issue--merge-request-events). +Read more in the [notifications documentation](../../../workflow/notifications.md#issue--epics--merge-request-events). #### 11. Reference diff --git a/doc/user/project/issues/multiple_assignees_for_issues.md b/doc/user/project/issues/multiple_assignees_for_issues.md new file mode 100644 index 00000000000..d1db0790d69 --- /dev/null +++ b/doc/user/project/issues/multiple_assignees_for_issues.md @@ -0,0 +1,41 @@ +# Multiple Assignees for Issues **[STARTER]** + +> **Note:** +[Introduced](https://gitlab.com/gitlab-org/gitlab-ee/issues/1904) +in [GitLab Starter 9.2](https://about.gitlab.com/2017/05/22/gitlab-9-2-released/#multiple-assignees-for-issues). + +## Overview + +In large teams, where there is shared ownership of an issue, it can be difficult +to track who is working on it, who already completed their contributions, who +didn't even start yet. + +In [GitLab Enterprise Edition](https://about.gitlab.com/pricing/), +you can also select multiple assignees to an issue, making it easier to +track, and making clearer who is accountable for it. + + + +## Use cases + +Consider a team formed by frontend developers, backend developers, +UX designers, QA testers, and a product manager working together to bring an idea to +market. + +Multiple Assignees for Issues makes collaboration smoother, +and allows shared responsibilities to be clearly displayed. +All assignees are shown across your team's workflows and receive notifications (as they +would as single assignees), simplifying communication and ownership. + +Once an assignee had their work completed, they would remove themselves as assignees, making +it clear that their role is complete. + +## How it works + +From an opened issue, expand the right sidebar, locate the assignees entry, +and click on **Edit**. From the dropdown menu, select as many users as you want +to assign the issue to. + + + +An assignee can be easily removed by deselecting them from the same dropdown menu. diff --git a/doc/user/project/issues/related_issues.md b/doc/user/project/issues/related_issues.md new file mode 100644 index 00000000000..e679ebf86e6 --- /dev/null +++ b/doc/user/project/issues/related_issues.md @@ -0,0 +1,40 @@ +# Related issues **[STARTER]** + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/1797) in [GitLab Starter](https://about.gitlab.com/pricing/) 9.4. + +Related issues are a bi-directional relationship between any two issues +and appear in a block below the issue description. Issues can be across groups +and projects. + +The relationship only shows up in the UI if the user can see both issues. + +## Adding a related issue + +You can relate one issue to another by clicking the related issues "+" button +in the header of the related issue block. Then, input the issue reference number +or paste in the full URL of the issue. + +Issues of the same project can be specified just by the reference number. +Issues from a different project require additional information like the +group and the project name. For example: + +- same project: `#44` +- same group: `project#44 ` +- different group: `group/project#44` + +Valid references will be added to a temporary list that you can review. +When ready, click the green "Add related issues" button to submit. + + + +## Removing a related issue + +In the related issues block, click the "x" icon on the right-side of each issue +token that you wish to remove. Due to the bi-directional relationship, it +will no longer appear in either issue. + + + +Please access our [permissions](../../permissions.md) page for more information. + +Additionally, you are also able to manage related issues through [our API](../../../api/issue_links.md). diff --git a/doc/user/project/labels.md b/doc/user/project/labels.md index 9c045dfcce4..e5f62a3bb8d 100644 --- a/doc/user/project/labels.md +++ b/doc/user/project/labels.md @@ -8,13 +8,60 @@ Labels allow you to categorize issues or merge requests using descriptive titles In GitLab, you can create project and group labels: -- **Project labels** can be assigned to issues or merge requests in that project only. +- **Project labels** can be assigned to issues or merge requests in that project only. - **Group labels** can be assigned to any issue or merge request of any project in that group or any subgroups of the group. +## Scoped labels **[PREMIUM]** + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/issues/9175) in [GitLab Premium](https://about.gitlab.com/pricing/) 11.10. + +Scoped labels allow teams to use the simple and familiar feature of labels to +annotate their issues, merge requests, and epics to achieve custom fields and +custom workflow states by leveraging a special label title syntax. + +A scoped label is a kind of label defined only by a special double-colon syntax +in the label’s title, using the format `key::value`. For example: + + + +Two scoped labels with the same key but a different value cannot simultaneously +apply to an issue, epic, or merge request. For example, if an issue already has `priority::3` +and you apply `priority::2` to it, `priority::3` is automatically removed from the issue. + +An issue, epic, or merge request cannot have two scoped labels with the same key. +For example, if an issue is already labeled `priority::3` and you apply the label `priority::2` to it, +`priority::3` is automatically removed. + +### Labels with multiple colon pairs + +If labels have multiple instances of `::`, the longest path from left to right, until the last `::`, is considered the "key" or the "scope". + +For example, `nested::key1::value1` and `nested::key1::value2` cannot both exist on the same issue. Adding the latter label will automatically remove the former due to the shared scope of `nested::key1`. + +`nested::key1::value1` and `nested::key2::value1` can both exist on the same issue, as these are considered to use two different label scopes, `nested::key1` and `nested::key2`. + +### Workflows with scoped labels **[PREMIUM]** + +Suppose you wanted a custom field in issues to track the platform operating system +that your features target, where each issue should only target one platform. You +would then create labels `platform::iOS`, `platform::Android`, `platform::Linux`, +etc., as necessary. Applying any one of these labels on a given issue would +automatically remove any other existing label that starts with `platform::`. + +The same pattern could be applied to represent the workflow states of your teams. +Suppose you have the labels `workflow::development`, `workflow::review`, and +`workflow::deployed`. If an issue already has the label `workflow::development` +applied, and a developer wanted to advance the issue to `workflow::review`, they +would simply apply that label, and the `workflow::development` label would +automatically be removed. This behavior already exists when you move issues +across label lists in an [issue board](issue_board.md#creating-workflows), but +now, team members who may not be working in an issue board directly would still +be able to advance workflow states consistently in issues themselves. + ## Creating labels >**Note:** -A permission level of `Developer` or higher is required to create labels. +A permission level of Reporter or higher is required to create labels. ### New project label @@ -35,6 +82,9 @@ GitLab will add the following default labels to the project: To create a **group label**, follow similar steps from above to project labels. Navigate to **Issues > Labels** in the group and create it from there. This page only shows group labels in this group. +Alternatively, you can create group labels also from Epic sidebar. Please note that the created label will belong to the immediate group to which epic belongs. + + Group labels appear in every label list page of the group's child projects. @@ -42,14 +92,14 @@ Group labels appear in every label list page of the group's child projects. ### New project label from sidebar -From the sidebar of an issue or a merge request, you can create a create a new **project label** inline immediately, instead of navigating to the project label list page. +From the sidebar of an issue or a merge request, you can create a new **project label** inline immediately, instead of navigating to the project label list page.  ## Editing labels NOTE: **Note:** -A permission level of `Developer` or higher is required to edit labels. +A permission level of Reporter or higher is required to edit labels. You can update a label by navigating to **Issues > Labels** in the project or group and clicking the pencil icon. @@ -81,7 +131,7 @@ top-right: GitLab will consider the label title and description for the search. -## Filtering issues and merge requests by label +## Filtering issues, merge requests and epics by label ### Filtering in list pages @@ -89,11 +139,16 @@ From the project issue list page and the project merge request list page, you ca From the group issue list page and the group merge request list page, you can [filter](../search/index.md#issues-and-merge-requests) by both group labels (including subgroup ancestors and subgroup descendants) and project labels. +From the group epic list page, you can [filter](../search/index.md#issues-and-merge-requests) by both current group labels as well as descendant group labels. +  ### Filtering in issue boards - From [project boards](issue_board.md), you can filter by both group labels and project labels in the [search and filter bar](../search/index.md#issue-boards). +- From [group issue boards](issue_board.md#group-issue-boards-premium), you can filter by only group labels in the [search and filter bar](../search/index.md#issue-boards). **[PREMIUM]** +- From [project boards](issue_board.md), you can filter by both group labels and project labels in the [issue board configuration](issue_board.md#configurable-issue-boards-starter). **[STARTER]** +- From [group issue boards](issue_board.md#group-issue-boards-premium), you can filter by only group labels in the [issue board configuration](issue_board.md#configurable-issue-boards-starter). **[STARTER]** ## Subscribing to labels diff --git a/doc/user/project/maven_packages.md b/doc/user/project/maven_packages.md new file mode 100644 index 00000000000..48835a2dac7 --- /dev/null +++ b/doc/user/project/maven_packages.md @@ -0,0 +1,5 @@ +--- +redirect_to: 'packages/maven_repository.md' +--- + +This document was moved to [another location](packages/maven_repository.md). diff --git a/doc/user/project/members/img/add_new_user_to_project_settings.png b/doc/user/project/members/img/add_new_user_to_project_settings.png Binary files differdeleted file mode 100644 index e49ea1a3e3d..00000000000 --- a/doc/user/project/members/img/add_new_user_to_project_settings.png +++ /dev/null diff --git a/doc/user/project/members/img/add_user_members_menu.png b/doc/user/project/members/img/add_user_members_menu.png Binary files differdeleted file mode 100644 index 6f08088b52f..00000000000 --- a/doc/user/project/members/img/add_user_members_menu.png +++ /dev/null diff --git a/doc/user/project/members/img/max_access_level.png b/doc/user/project/members/img/max_access_level.png Binary files differdeleted file mode 100644 index 42a0416ffbb..00000000000 --- a/doc/user/project/members/img/max_access_level.png +++ /dev/null diff --git a/doc/user/project/merge_requests.md b/doc/user/project/merge_requests.md index 84a79f04094..1d7ebc856c3 100644 --- a/doc/user/project/merge_requests.md +++ b/doc/user/project/merge_requests.md @@ -1 +1,5 @@ +--- +redirect_to: 'merge_requests/index.md' +--- + This document was moved to [merge_requests/index.md](merge_requests/index.md). diff --git a/doc/user/project/merge_requests/browser_performance_testing.md b/doc/user/project/merge_requests/browser_performance_testing.md new file mode 100644 index 00000000000..65ee2e128ae --- /dev/null +++ b/doc/user/project/merge_requests/browser_performance_testing.md @@ -0,0 +1,54 @@ +# Browser Performance Testing **[PREMIUM]** + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/3507) +in [GitLab Premium](https://about.gitlab.com/pricing/) 10.3. + +## Overview + +If your application offers a web interface and you are using +[GitLab CI/CD](../../../ci/README.md), you can quickly determine the performance +impact of pending code changes. + +GitLab uses [Sitespeed.io](https://www.sitespeed.io), a free and open source +tool for measuring the performance of web sites, and has built a simple +[Sitespeed plugin](https://gitlab.com/gitlab-org/gl-performance) +which outputs the results in a file called `performance.json`. This plugin +outputs the performance score for each page that is analyzed. + +The [Sitespeed.io performance score](http://examples.sitespeed.io/6.0/2017-11-23-23-43-35/help.html) +is a composite value based on best practices, and we will be expanding support +for [additional metrics](https://gitlab.com/gitlab-org/gitlab-ee/issues/4370) +in a future release. + +Going a step further, GitLab can show the Performance report right +in the merge request widget area: + +## Use cases + +For instance, consider the following workflow: + +1. A member of the marketing team is attempting to track engagement by adding a new tool +1. With browser performance metrics, they see how their changes are impacting the usability of the page for end users +1. The metrics show that after their changes the performance score of the page has gone down +1. When looking at the detailed report, they see that the new Javascript library was included in `<head>` which affects loading page speed +1. They ask a front end developer to help them, who sets the library to load asynchronously +1. The frontend developer approves the merge request and authorizes its deployment to production + +## How it works + +First of all, you need to define a job in your `.gitlab-ci.yml` file that generates the +[Performance report artifact](../../../ci/yaml/README.md#artifactsreportsperformance-premium). +For more information on how the Performance job should look like, check the +example on [Testing Browser Performance](../../../ci/examples/browser_performance.md). + +GitLab then checks this report, compares key performance metrics for each page +between the source and target branches, and shows the information right on the merge request. + +>**Note:** +If the Performance report doesn't have anything to compare to, no information +will be displayed in the merge request area. That is the case when you add the +Performance job in your `.gitlab-ci.yml` for the very first time. +Consecutive merge requests will have something to compare to and the Performance +report will be shown properly. + + diff --git a/doc/user/project/merge_requests/code_quality.md b/doc/user/project/merge_requests/code_quality.md new file mode 100644 index 00000000000..705ff333579 --- /dev/null +++ b/doc/user/project/merge_requests/code_quality.md @@ -0,0 +1,82 @@ +# Code Quality **[STARTER]** + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/1984) +in [GitLab Starter](https://about.gitlab.com/pricing/) 9.3. + +## Overview + +If you are using [GitLab CI/CD](../../../ci/README.md), you can analyze your +source code quality using GitLab Code Quality. +Code Quality uses [Code Climate Engines](https://codeclimate.com), which are +free and open source. Code Quality doesn’t require a Code Climate subscription. + +Going a step further, GitLab can show the Code Quality report right +in the merge request widget area: + + + +## Use cases + +For instance, consider the following workflow: + +1. Your backend team member starts a new implementation for making a certain feature in your app faster +1. With Code Quality reports, they analyze how their implementation is impacting the code quality +1. The metrics show that their code degrade the quality in 10 points +1. You ask a co-worker to help them with this modification +1. They both work on the changes until Code Quality report displays no degradations, only improvements +1. You approve the merge request and authorize its deployment to staging +1. Once verified, their changes are deployed to production + +## How it works + +First of all, you need to define a job in your `.gitlab-ci.yml` file that generates the +[Code Quality report artifact](../../../ci/yaml/README.md#artifactsreportscodequality-starter). + +The Code Quality report artifact is a subset of the +[Code Climate spec](https://github.com/codeclimate/spec/blob/master/SPEC.md#data-types). +It must be a JSON file containing an array of objects with the following properties: + +| Name | Description | +| ---------------------- | -------------------------------------------------------------------------------------- | +| `description` | A description of the code quality violation. | +| `fingerprint` | A unique fingerprint to identify the code quality violation. For example, an MD5 hash. | +| `location.path` | The relative path to the file containing the code quality violation. | +| `location.lines.begin` | The line on which the code quality violation occurred. | + +Example: + +```json +[ + { + "description": "'unused' is assigned a value but never used.", + "fingerprint": "7815696ecbf1c96e6894b779456d330e", + "location": { + "path": "lib/index.js", + "lines": { + "begin": 42 + } + } + } +] +``` + +NOTE: **Note:** +Although the Code Climate spec supports more properties, those are ignored by GitLab. + +For more information on what the Code Quality job should look like, check the +example on [analyzing a project's code quality](../../../ci/examples/code_quality.md). + +GitLab then checks this report, compares the metrics between the source and target +branches, and shows the information right on the merge request. + +CAUTION: **Caution:** +If multiple jobs in a pipeline generate a code quality artifact, only the artifact from +the last created job (the job with the largest job ID) is used. To avoid confusion, +configure only one job to generate a code quality artifact. + +NOTE: **Note:** +If the Code Quality report doesn't have anything to compare to, no information +will be displayed in the merge request area. That is the case when you add the +Code Quality job in your `.gitlab-ci.yml` for the very first time. +Consecutive merge requests will have something to compare to and the Code Quality +report will be shown properly. diff --git a/doc/user/project/merge_requests/code_quality_diff.md b/doc/user/project/merge_requests/code_quality_diff.md new file mode 100644 index 00000000000..ccc694672a6 --- /dev/null +++ b/doc/user/project/merge_requests/code_quality_diff.md @@ -0,0 +1,6 @@ +--- +redirect_from: 'code_quality_diff.md' +redirect_to: 'code_quality.md' +--- + +This document was moved to [another location](code_quality.md). diff --git a/doc/user/project/merge_requests/container_scanning.md b/doc/user/project/merge_requests/container_scanning.md new file mode 100644 index 00000000000..a062731ea35 --- /dev/null +++ b/doc/user/project/merge_requests/container_scanning.md @@ -0,0 +1,5 @@ +--- +redirect_to: '../../application_security/container_scanning/index.md' +--- + +This document was moved to [another location](../../application_security/container_scanning/index.md). diff --git a/doc/user/project/merge_requests/dast.md b/doc/user/project/merge_requests/dast.md new file mode 100644 index 00000000000..98a2906e560 --- /dev/null +++ b/doc/user/project/merge_requests/dast.md @@ -0,0 +1,5 @@ +--- +redirect_to: '../../application_security/dast/index.md' +--- + +This document was moved to [another location](../../application_security/dast/index.md). diff --git a/doc/user/project/merge_requests/dependency_scanning.md b/doc/user/project/merge_requests/dependency_scanning.md new file mode 100644 index 00000000000..bdc1c355016 --- /dev/null +++ b/doc/user/project/merge_requests/dependency_scanning.md @@ -0,0 +1,5 @@ +--- +redirect_to: '../../application_security/dependency_scanning/index.md' +--- + +This document was moved to [another location](../../application_security/dependency_scanning/index.md). diff --git a/doc/user/project/merge_requests/img/approvals_can_override.png b/doc/user/project/merge_requests/img/approvals_can_override.png Binary files differnew file mode 100644 index 00000000000..8d207d018e0 --- /dev/null +++ b/doc/user/project/merge_requests/img/approvals_can_override.png diff --git a/doc/user/project/merge_requests/img/approvals_premium_mr_widget.png b/doc/user/project/merge_requests/img/approvals_premium_mr_widget.png Binary files differnew file mode 100644 index 00000000000..b6dc86f312e --- /dev/null +++ b/doc/user/project/merge_requests/img/approvals_premium_mr_widget.png diff --git a/doc/user/project/merge_requests/img/approvals_premium_project_edit.png b/doc/user/project/merge_requests/img/approvals_premium_project_edit.png Binary files differnew file mode 100644 index 00000000000..b6f6188b9cd --- /dev/null +++ b/doc/user/project/merge_requests/img/approvals_premium_project_edit.png diff --git a/doc/user/project/merge_requests/img/approvals_remove_on_push.png b/doc/user/project/merge_requests/img/approvals_remove_on_push.png Binary files differnew file mode 100644 index 00000000000..73964827587 --- /dev/null +++ b/doc/user/project/merge_requests/img/approvals_remove_on_push.png diff --git a/doc/user/project/merge_requests/img/approvals_starter_project_edit.png b/doc/user/project/merge_requests/img/approvals_starter_project_edit.png Binary files differnew file mode 100644 index 00000000000..868b9d58740 --- /dev/null +++ b/doc/user/project/merge_requests/img/approvals_starter_project_edit.png diff --git a/doc/user/project/merge_requests/img/approvals_starter_project_empty.png b/doc/user/project/merge_requests/img/approvals_starter_project_empty.png Binary files differnew file mode 100644 index 00000000000..7375820224c --- /dev/null +++ b/doc/user/project/merge_requests/img/approvals_starter_project_empty.png diff --git a/doc/user/project/merge_requests/img/approve.png b/doc/user/project/merge_requests/img/approve.png Binary files differnew file mode 100644 index 00000000000..e68259ac5c2 --- /dev/null +++ b/doc/user/project/merge_requests/img/approve.png diff --git a/doc/user/project/merge_requests/img/approve_additionally.png b/doc/user/project/merge_requests/img/approve_additionally.png Binary files differnew file mode 100644 index 00000000000..3db5a9159e5 --- /dev/null +++ b/doc/user/project/merge_requests/img/approve_additionally.png diff --git a/doc/user/project/merge_requests/img/browser_performance_testing.png b/doc/user/project/merge_requests/img/browser_performance_testing.png Binary files differnew file mode 100644 index 00000000000..eea77fb8b93 --- /dev/null +++ b/doc/user/project/merge_requests/img/browser_performance_testing.png diff --git a/doc/user/project/merge_requests/img/code_quality.gif b/doc/user/project/merge_requests/img/code_quality.gif Binary files differnew file mode 100644 index 00000000000..bab921cf38b --- /dev/null +++ b/doc/user/project/merge_requests/img/code_quality.gif diff --git a/doc/user/project/merge_requests/img/comment-on-any-diff-line.png b/doc/user/project/merge_requests/img/comment-on-any-diff-line.png Binary files differindex c2455c2d1e5..5b9844bf02f 100644 --- a/doc/user/project/merge_requests/img/comment-on-any-diff-line.png +++ b/doc/user/project/merge_requests/img/comment-on-any-diff-line.png diff --git a/doc/user/project/merge_requests/img/filter_approver_merge_requests.png b/doc/user/project/merge_requests/img/filter_approver_merge_requests.png Binary files differnew file mode 100644 index 00000000000..9c386391a4f --- /dev/null +++ b/doc/user/project/merge_requests/img/filter_approver_merge_requests.png diff --git a/doc/user/project/merge_requests/img/filter_wip_merge_requests.png b/doc/user/project/merge_requests/img/filter_wip_merge_requests.png Binary files differindex 81878709487..8df6a3c9a29 100644 --- a/doc/user/project/merge_requests/img/filter_wip_merge_requests.png +++ b/doc/user/project/merge_requests/img/filter_wip_merge_requests.png diff --git a/doc/user/project/merge_requests/img/merge_when_pipeline_succeeds_only_if_succeeds_settings.png b/doc/user/project/merge_requests/img/merge_when_pipeline_succeeds_only_if_succeeds_settings.png Binary files differindex 2a2101719ba..ea3aff59aa1 100644 --- a/doc/user/project/merge_requests/img/merge_when_pipeline_succeeds_only_if_succeeds_settings.png +++ b/doc/user/project/merge_requests/img/merge_when_pipeline_succeeds_only_if_succeeds_settings.png diff --git a/doc/user/project/merge_requests/img/multiple_assignees_for_merge_requests_sidebar.png b/doc/user/project/merge_requests/img/multiple_assignees_for_merge_requests_sidebar.png Binary files differnew file mode 100644 index 00000000000..9ae6e350798 --- /dev/null +++ b/doc/user/project/merge_requests/img/multiple_assignees_for_merge_requests_sidebar.png diff --git a/doc/user/project/merge_requests/img/remove_approval.png b/doc/user/project/merge_requests/img/remove_approval.png Binary files differnew file mode 100644 index 00000000000..6083e1745ef --- /dev/null +++ b/doc/user/project/merge_requests/img/remove_approval.png diff --git a/doc/user/project/merge_requests/img/revert_changes_commit.png b/doc/user/project/merge_requests/img/revert_changes_commit.png Binary files differdeleted file mode 100644 index c9dd0019024..00000000000 --- a/doc/user/project/merge_requests/img/revert_changes_commit.png +++ /dev/null diff --git a/doc/user/project/merge_requests/img/revert_changes_mr.png b/doc/user/project/merge_requests/img/revert_changes_mr.png Binary files differdeleted file mode 100644 index 06b841b3002..00000000000 --- a/doc/user/project/merge_requests/img/revert_changes_mr.png +++ /dev/null diff --git a/doc/user/project/merge_requests/img/wip_blocked_accept_button.png b/doc/user/project/merge_requests/img/wip_blocked_accept_button.png Binary files differindex 31f23be4d3d..b6d38d85165 100644 --- a/doc/user/project/merge_requests/img/wip_blocked_accept_button.png +++ b/doc/user/project/merge_requests/img/wip_blocked_accept_button.png diff --git a/doc/user/project/merge_requests/img/wip_mark_as_wip.png b/doc/user/project/merge_requests/img/wip_mark_as_wip.png Binary files differdeleted file mode 100644 index 2c2a263b316..00000000000 --- a/doc/user/project/merge_requests/img/wip_mark_as_wip.png +++ /dev/null diff --git a/doc/user/project/merge_requests/img/wip_unmark_as_wip.png b/doc/user/project/merge_requests/img/wip_unmark_as_wip.png Binary files differdeleted file mode 100644 index 327ad9a8448..00000000000 --- a/doc/user/project/merge_requests/img/wip_unmark_as_wip.png +++ /dev/null diff --git a/doc/user/project/merge_requests/index.md b/doc/user/project/merge_requests/index.md index 593eb80e044..86e06c2ea55 100644 --- a/doc/user/project/merge_requests/index.md +++ b/doc/user/project/merge_requests/index.md @@ -18,12 +18,12 @@ With GitLab merge requests, you can: - Live preview the changes when [Review Apps](../../../ci/review_apps/index.md) is configured for your project - Build, test, and deploy your code in a per-branch basis with built-in [GitLab CI/CD](../../../ci/README.md) - Prevent the merge request from being merged before it's ready with [WIP MRs](#work-in-progress-merge-requests) -- View the deployment process through [Pipeline Graphs](../../../ci/pipelines.md#pipeline-graphs) +- View the deployment process through [Pipeline Graphs](../../../ci/pipelines.md#visualizing-pipelines) - [Automatically close the issue(s)](../../project/issues/closing_issues.md#via-merge-request) that originated the implementation proposed in the merge request - Assign it to any registered user, and change the assignee how many times you need - Assign a [milestone](../../project/milestones/index.md) and track the development of a broader implementation - Organize your issues and merge requests consistently throughout the project with [labels](../../project/labels.md) -- Add a time estimation and the time spent with that merge request with [Time Tracking](../../../workflow/time_tracking.html#time-tracking) +- Add a time estimation and the time spent with that merge request with [Time Tracking](../../../workflow/time_tracking.md#time-tracking) - [Resolve merge conflicts from the UI](#resolve-conflicts) - Enable [fast-forward merge requests](#fast-forward-merge-requests) - Enable [semi-linear history merge requests](#semi-linear-history-merge-requests) as another security layer to guarantee the pipeline is passing in the target branch @@ -33,9 +33,16 @@ With GitLab merge requests, you can: With **[GitLab Enterprise Edition][ee]**, you can also: -- View the deployment process across projects with [Multi-Project Pipeline Graphs](https://docs.gitlab.com/ee/ci/multi_project_pipeline_graphs.html#multi-project-pipeline-graphs) **[PREMIUM]** -- Request [approvals](https://docs.gitlab.com/ee/user/project/merge_requests/merge_request_approvals.html) from your managers **[STARTER]** -- Analyze the impact of your changes with [Code Quality reports](https://docs.gitlab.com/ee/user/project/merge_requests/code_quality.html) **[STARTER]** +- Prepare a full review and submit it once it's ready with [Merge Request Reviews](../../discussions/index.md#merge-request-reviews-premium) **[PREMIUM]** +- View the deployment process across projects with [Multi-Project Pipelines](../../../ci/multi_project_pipelines.md) **[PREMIUM]** +- Request [approvals](merge_request_approvals.md) from your managers **[STARTER]** +- Analyze the impact of your changes with [Code Quality reports](code_quality.md) **[STARTER]** +- Manage the licenses of your dependencies with [License Management](../../application_security/license_management/index.md) **[ULTIMATE]** +- Analyze your source code for vulnerabilities with [Static Application Security Testing](../../application_security/sast/index.md) **[ULTIMATE]** +- Analyze your running web applications for vulnerabilities with [Dynamic Application Security Testing](../../application_security/dast/index.md) **[ULTIMATE]** +- Analyze your dependencies for vulnerabilities with [Dependency Scanning](../../application_security/dependency_scanning/index.md) **[ULTIMATE]** +- Analyze your Docker images for vulnerabilities with [Container Scanning](../../application_security/container_scanning/index.md) **[ULTIMATE]** +- Determine the performance impact of changes with [Browser Performance Testing](#browser-performance-testing-premium) **[PREMIUM]** ## Use cases @@ -43,19 +50,21 @@ A. Consider you are a software developer working in a team: 1. You checkout a new branch, and submit your changes through a merge request 1. You gather feedback from your team +1. You work on the implementation optimizing code with [Code Quality reports](code_quality.md) **[STARTER]** 1. You verify your changes with [JUnit test reports](../../../ci/junit_test_reports.md) in GitLab CI/CD -1. You request the approval from your manager -1. Your manager pushes a commit with his final review, [approves the merge request](https://docs.gitlab.com/ee/user/project/merge_requests/merge_request_approvals.html), and set it to [merge when pipeline succeeds](#merge-when-pipeline-succeeds) (Merge Request Approvals are available in GitLab Starter) +1. You avoid using dependencies whose license is not compatible with your project with [License Management reports](license_management.md) **[ULTIMATE]** +1. You request the [approval](#merge-request-approvals-starter) from your manager +1. Your manager pushes a commit with their final review, [approves the merge request](merge_request_approvals.md), and set it to [merge when pipeline succeeds](#merge-when-pipeline-succeeds) (Merge Request Approvals are available in GitLab Starter) 1. Your changes get deployed to production with [manual actions](../../../ci/yaml/README.md#whenmanual) for GitLab CI/CD 1. Your implementations were successfully shipped to your customer -B. Consider you're a web developer writing a webpage for your company's: +B. Consider you're a web developer writing a webpage for your company's website: 1. You checkout a new branch, and submit a new page through a merge request 1. You gather feedback from your reviewers 1. Your changes are previewed with [Review Apps](../../../ci/review_apps/index.md) 1. You request your web designers for their implementation -1. You request the [approval](https://docs.gitlab.com/ee/user/project/merge_requests/merge_request_approvals.html) from your manager **[STARTER]** +1. You request the [approval](merge_request_approvals.md) from your manager **[STARTER]** 1. Once approved, your merge request is [squashed and merged](squash_and_merge.md), and [deployed to staging with GitLab Pages](https://about.gitlab.com/2016/08/26/ci-deployment-and-environments/) 1. Your production team [cherry picks](#cherry-pick-changes) the merge commit into production @@ -159,6 +168,21 @@ in a Merge Request. To do so, click the **...** button in the gutter of the Merg  +## Perform a Review **[PREMIUM]** + +Start a review in order to create multiple comments on a diff and publish them once you're ready. +Starting a review allows you to get all your thoughts in order and ensure you haven't missed anything +before submitting all your comments. + +[Learn more about Merge Request Reviews](../../discussions/index.md#merge-request-reviews-premium) + +## Squash and merge + +GitLab allows you to squash all changes present in a merge request into a single +commit when merging, to allow for a neater commit history. + +[Learn more about squash and merge.](squash_and_merge.md) + ## Suggest changes > [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/issues/18008) in GitLab 11.6. @@ -169,6 +193,28 @@ can easily apply them to the codebase directly from the UI. Read through the documentation on [Suggest changes](../../discussions/index.md#suggest-changes) to learn more. +## Multiple assignees **[STARTER]** + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/issues/2004) +in [GitLab Starter 11.11](https://about.gitlab.com/pricing). + +Multiple people often review merge requests at the same time. GitLab allows you to have multiple assignees for merge requests to indicate everyone that is reviewing or accountable for it. + + + +To assign multiple assignees to a merge request: + +1. From a merge request, expand the right sidebar and locate the **Assignees** section. +1. Click on **Edit** and from the dropdown menu, select as many users as you want + to assign the merge request to. + +Similarly, assignees are removed by deselecting them from the same dropdown menu. + +It's also possible to manage multiple assignees: + +- When creating a merge request. +- Using [quick actions](../quick_actions.md#quick-actions-for-issues-and-merge-requests). + ## Resolve conflicts When a merge request has conflicts, GitLab may provide the option to resolve @@ -219,6 +265,64 @@ apply the patches. The target branch can be specified using the [`/target_branch` quick action](../quick_actions.md). If the source branch already exists, the patches will be applied on top of it. +## Git push options + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/26752) in GitLab 11.10. + +NOTE: **Note:** +Git push options are only available with Git 2.10 or newer. + +GitLab supports using +[Git push options](https://git-scm.com/docs/git-push#Documentation/git-push.txt--oltoptiongt) +to perform the following actions against merge requests at the same time +as pushing changes: + +- Create a new merge request for the pushed branch. +- Set the target of the merge request to a particular branch. +- Set the merge request to merge when its pipeline succeeds. + +### Create a new merge request using git push options + +To create a new merge request for a branch, use the +`merge_request.create` push option: + +```sh +git push -o merge_request.create +``` + +### Set the target branch of a merge request using git push options + +To update an existing merge request's target branch, use the +`merge_request.target=<branch_name>` push option: + +```sh +git push -o merge_request.target=branch_name +``` + +You can also create a merge request and set its target branch at the +same time using a `-o` flag per push option: + +```sh +git push -o merge_request.create -o merge_request.target=branch_name +``` + +### Set merge when pipeline succeeds using git push options + +To set an existing merge request to +[merge when its pipeline succeeds](merge_when_pipeline_succeeds.md), use +the `merge_request.merge_when_pipeline_succeeds` push option: + +```sh +git push -o merge_request.merge_when_pipeline_succeeds +``` + +You can also create a merge request and set it to merge when its +pipeline succeeds at the same time using a `-o` flag per push option: + +```sh +git push -o merge_request.create -o merge_request.merge_when_pipeline_succeeds +``` + ## Find the merge request that introduced a change > [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/issues/2383) in GitLab 10.5. @@ -254,7 +358,61 @@ To prevent merge requests from accidentally being accepted before they're completely ready, GitLab blocks the "Accept" button for merge requests that have been marked as a **Work In Progress**. -[Learn more about settings a merge request as "Work In Progress".](work_in_progress_merge_requests.md) +[Learn more about setting a merge request as "Work In Progress".](work_in_progress_merge_requests.md) + +## Merge request approvals **[STARTER]** + +> Included in [GitLab Starter][products]. + +If you want to make sure every merge request is approved by one or more people, +you can enforce this workflow by using merge request approvals. Merge request +approvals allow you to set the number of necessary approvals and predefine a +list of approvers that will need to approve every merge request in a project. + +[Read more about merge request approvals.](merge_request_approvals.md) + +## Code Quality **[STARTER]** + +> Introduced in [GitLab Starter][products] 9.3. + +If you are using [GitLab CI][ci], you can analyze your source code quality using +the [Code Climate][cc] analyzer [Docker image][cd]. Going a step further, GitLab +can show the Code Climate report right in the merge request widget area. + +[Read more about Code Quality reports.](code_quality.md) + +## Browser Performance Testing **[PREMIUM]** + +> Introduced in [GitLab Premium][products] 10.3. + +If your application offers a web interface and you are using [GitLab CI/CD][ci], you can quickly determine the performance impact of pending code changes. GitLab uses [Sitespeed.io][sitespeed], a free and open source tool for measuring the performance of web sites, to analyze the performance of specific pages. + +GitLab runs the [Sitespeed.io container][sitespeed-container] and displays the difference in overall performance scores between the source and target branches. + +[Read more about Browser Performance Testing.](browser_performance_testing.md) + +## Security reports **[ULTIMATE]** + +GitLab can scan and report any vulnerabilities found in your project. + +[Read more about security reports.](../../application_security/index.md) + +## JUnit test reports + +Configure your CI jobs to use JUnit test reports, and let GitLab display a report +on the merge request so that it’s easier and faster to identify the failure +without having to check the entire job log. + +[Read more about JUnit test reports](../../../ci/junit_test_reports.md). + +## Live preview with Review Apps + +If you configured [Review Apps](https://about.gitlab.com/features/review-apps/) for your project, +you can preview the changes submitted to a feature-branch through a merge request +in a per-branch basis. No need to checkout the branch, install and preview locally; +all your changes will be available to preview by anyone with the Review Apps link. + +[Read more about Review Apps.](../../../ci/review_apps/index.md) ## Merge request diff file navigation @@ -353,7 +511,7 @@ seconds and the status will update automatically. Merge Request pipeline statuses can't be retrieved when the following occurs: -1. A Merge Requst is created +1. A Merge Request is created 1. The Merge Request is closed 1. Changes are made in the project 1. The Merge Request is reopened @@ -389,13 +547,13 @@ Add the following alias to your `~/.gitconfig`: Now you can check out a particular merge request from any repository and any remote. For example, to check out the merge request with ID 5 as shown in GitLab -from the `upstream` remote, do: +from the `origin` remote, do: ``` -git mr upstream 5 +git mr origin 5 ``` -This will fetch the merge request into a local `mr-upstream-5` branch and check +This will fetch the merge request into a local `mr-origin-5` branch and check it out. #### Checkout locally by modifying `.git/config` for a given repository @@ -448,5 +606,14 @@ And to check out a particular merge request: git checkout origin/merge-requests/1 ``` +all the above can be done with [git-mr] script. + +[git-mr]: https://gitlab.com/glensc/git-mr +[products]: https://about.gitlab.com/products/ "GitLab products page" [protected branches]: ../protected_branches.md +[ci]: ../../../ci/README.md +[cc]: https://codeclimate.com/ +[cd]: https://hub.docker.com/r/codeclimate/codeclimate/ +[sitespeed]: https://www.sitespeed.io +[sitespeed-container]: https://hub.docker.com/r/sitespeedio/sitespeed.io/ [ee]: https://about.gitlab.com/pricing/ "GitLab Enterprise Edition" diff --git a/doc/user/project/merge_requests/license_management.md b/doc/user/project/merge_requests/license_management.md new file mode 100644 index 00000000000..93116ebd7c6 --- /dev/null +++ b/doc/user/project/merge_requests/license_management.md @@ -0,0 +1,5 @@ +--- +redirect_to: '../../application_security/license_management/index.md' +--- + +This document was moved to [another location](../../application_security/license_management/index.md). diff --git a/doc/user/project/merge_requests/maintainer_access.md b/doc/user/project/merge_requests/maintainer_access.md index d59afecd375..fe7e1f558c7 100644 --- a/doc/user/project/merge_requests/maintainer_access.md +++ b/doc/user/project/merge_requests/maintainer_access.md @@ -1 +1,5 @@ +--- +redirect_to: 'allow_collaboration.md' +--- + This document was moved to [another location](allow_collaboration.md). diff --git a/doc/user/project/merge_requests/merge_request_approvals.md b/doc/user/project/merge_requests/merge_request_approvals.md new file mode 100644 index 00000000000..2e9db949890 --- /dev/null +++ b/doc/user/project/merge_requests/merge_request_approvals.md @@ -0,0 +1,330 @@ +# Merge request approvals **[STARTER]** + +> Introduced in [GitLab Enterprise Edition 7.12](https://about.gitlab.com/2015/06/22/gitlab-7-12-released/#merge-request-approvers-ee-only). + +NOTE: **Note:** +Prior to 12.0, if you are running a self-managed instance, the new interface shown on +this page will not be available unless the feature flag +`approval_rules` is enabled, which can be done from the Rails console by +instance administrators. + +Use these commands to start the Rails console: + +```sh +# Omnibus GitLab +gitlab-rails console + +# Installation from source +cd /home/git/gitlab +sudo -u git -H bin/rails console RAILS_ENV=production +``` + +Then run `Feature.enable(:approval_rules)` to enable the feature flag. + +The documentation for the older interface can be accessed +[here](/11.7/ee/user/project/merge_requests/merge_request_approvals.html). + +## Overview + +Merge request approvals enable enforced code review by requiring specified people to approve a merge request before it can be unblocked for merging. + +## Use cases + +1. Enforcing review of all code that gets merged into a repository. +2. Specifying code maintainers for an entire repository. +3. Specifying reviewers for a given proposed code change. +4. Specifying categories of reviewers, such as BE, FE, QA, DB, etc., for all proposed code changes. + +## Editing approvals + +To edit the merge request approvals: + +1. Navigate to your project's **Settings > General** and expand + **Merge request approvals**. + +  + +1. Click **Edit**. +1. Search for users or groups that will be [eligible to approve](#eligible-approvers) + merge requests and click the **Add** button to add them as approvers. Note: selecting + approvers is optional. +1. Set the minimum number of required approvals under the **No. approvals required** + box. Note: the minimum can be 0. +1. Click **Update approvers**. + +  + +The steps above are the minimum required to get approvals working in your +merge requests, but there are a couple more options available that might be +suitable to your workflow: + +- Choose whether the default settings can be + [overridden per merge request](#overriding-the-merge-request-approvals-default-settings) +- Choose whether [approvals will be reset with new pushed commits](#resetting-approvals-on-push) + +## Editing approvals **[PREMIUM]** + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/issues/1979) in [GitLab Premium](https://about.gitlab.com/pricing/) 11.8. + +CAUTION: **Caution:** +There was a [regression affecting this feature in 11.8](https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/9648). We recommend upgrading _at least_ to version 11.8.2. to avoid any issues. + +NOTE: **Note:** +In 11.8 this feature does not work in [private groups](https://gitlab.com/gitlab-org/gitlab-ee/issues/10356). + +For GitLab Premium, [multiple approver rules](#multiple-approval-rules-premium) can be configured. To configure the merge +request approval rules: + +1. Navigate to your project's **Settings > General** and expand **Merge request approvals**. +1. Click **Add approvers** to create a new approval rule. +1. Just like in [GitLab Starter](#editing-approvals), select the approval members and approvals required. +1. Give the approval rule a name that describes the set of approvers selected. +1. Click **Add approvers** to submit the new rule. + +  + +## Multiple approval rules **[PREMIUM]** + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/issues/1979) in [GitLab Premium](https://about.gitlab.com/pricing/) 11.8. + +For GitLab Premium, a merge request's overall approval status is determined by a set of rules. Each rule contains: + +- A set of [eligible approvers](#eligible-approvers). +- A minimum number of approvals required. + +When an [eligible approver](#eligible-approvers) approves a merge request, it will reduce the number of approvals left for +all rules that the approver belongs to. + + + +If no approval rules are set, then the overall minimum number of approvals required can be configured. With no approval rules, +any [eligible approver](#eligible-approvers) may approve. + +## Eligible approvers + +The following can approve merge requests: + +- Users being added as approvers at project or merge request level. +- [Code owners](../code_owners.md) related to the merge request ([introduced](https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/7933) in [GitLab Starter](https://about.gitlab.com/pricing/) 11.5). + +An individual user can be added as an approver for a project if they are a member of: + +- The project. +- The project's immediate parent group. +- A group that has access to the project via a [share](../members/share_project_with_groups.md). + +A group can also be added as an approver. [In the future](https://gitlab.com/gitlab-org/gitlab-ee/issues/2048), +group approvers will be restricted. + +If a user is added as an individual approver and is also part of a group approver, +then that user is just counted once. The merge request author, as well as users who have committed +to the merge request, do not count as eligible approvers, +if [**Prevent author approval**](#allowing-merge-request-authors-to-approve-their-own-merge-requests) (enabled by default) +and [**Prevent committers approval**](#prevent-approval-of-merge-requests-by-their-committers) (disabled by default) +are enabled on the project settings. + +### Implicit approvers + +If the number of required approvals is greater than the number of approvers, +other users will become implicit approvers to fill the gap. +Those implicit approvers include members of the given project with Developer role or higher. + +## Adding or removing an approval + +If approvals are activated for the given project, when a user visits an open +merge request, depending on their [eligibility](#eligible-approvers), one of +the following is possible: + +- **They are not an eligible approver**: They cannot do anything with respect + to approving this merge request. + +- **They have not approved this merge request**: + + - If the required number of approvals has _not_ been yet met, they can approve + it by clicking the displayed **Approve** button. +  + - If the required number of approvals has already been met, they can still + approve it by clicking the displayed **Approve additionally** button. +  + +- **They have already approved this merge request**: They can remove their approval. + +  + +NOTE: **Note:** +The merge request author is only allowed to approve their own merge request +if [**Prevent author approval**](#allowing-merge-request-authors-to-approve-their-own-merge-requests) is disabled on the project settings. + +For a given merge request, if the approval restrictions have been satisfied, +the merge request is unblocked and can be merged. +Note, that meeting the required number of approvals is a necessary, but not +sufficient condition for unblocking a merge request from being merged. There +are other conditions that may block it, such as merge conflicts, +[pending discussions](../../discussions/index.md#only-allow-merge-requests-to-be-merged-if-all-discussions-are-resolved) +or a [failed CI/CD pipeline](merge_when_pipeline_succeeds.md). + +## Code Owners approvals **[PREMIUM]** + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/issues/4418) in [GitLab Premium](https://about.gitlab.com/pricing/) 11.9. + +It is possible to require at least one approval for each entry in the +[`CODEOWNERS` file](../code_owners.md) that matches a file changed in +the merge request. To enable this feature: + +1. Navigate to your project's **Settings > General** and expand + **Merge request approvals**. +1. Tick the **Require approval from code owners** checkbox. +1. Click **Save changes**. + +When this feature is enabled, all merge requests will need approval +from one code owner per matched rule before it can be merged. + +## Overriding the merge request approvals default settings + +> Introduced in GitLab Enterprise Edition 9.4. + +NOTE: **Note:** +If you are using GitLab Premium, things are a little different with [multiple approval rules](#multiple-approval-rules-premium). +Read the differences [in GitLab Premium when overriding merge request approvals](#overriding-merge-request-approvals-default-settings-premium). + +If approvals are [set at the project level](#editing-approvals), the +default configuration (number of required approvals and approvers) can be +overridden for each merge request in that project. + +One possible scenario would be to to assign a group of approvers at the project +level and change them later when creating or editing the merge request. + +First, you have to enable this option in the project's settings: + +1. Navigate to your project's **Settings > General** and expand + **Merge request approvals** +1. Tick the "Can override approvers and approvals required per merge request" + checkbox + +  + +1. Click **Save changes** + +NOTE: **Note:** +If approver overriding is enabled +and the project level approvers are changed after a merge request is created, +the merge request retains the previous approvers. +However, the approvers can be changed by [editing the merge request](#overriding-the-merge-request-approvals-default-settings). + +--- + +The default approval settings can now be overridden when creating a +[merge request](index.md) or by editing it after it's been created: + +1. Click **Edit** under the **Approvers** section. +1. Search for users or groups that will be [eligible to approve](#eligible-approvers) + merge requests and click the **Add** button to add them as approvers or + remove existing approvers that were set in the project's settings. +1. If you want to change the number of required approvals, set a new number + in the **No. approvals required** box. +1. Click **Update approvers**. + +There are however some restrictions: + +- The amount of required approvals, if changed, must be greater than the default + set at the project level. This ensures that you're not forced to adjust settings + when someone is unavailable for approval, yet the process is still enforced. + +NOTE: **Note:** +If you are contributing to a forked project, things are a little different. +Read what happens when the +[source and target branches are not the same](#merge-requests-with-different-source-branch-and-target-branch-projects). + +## Overriding merge request approvals default settings **[PREMIUM]** + +In GitLab Premium, when the approval rules are [set at the project level](#editing-approvals-premium), and +**Can override approvers and approvals required per merge request** is checked, there are a few more +restrictions (compared to [GitLab Starter](#overriding-the-merge-request-approvals-default-settings)): + +- Approval rules can be added to an MR with no restriction. +- For project sourced approval rules, editing and removing approvers is not allowed. +- The approvals required of all approval rules is configurable, but if a rule is backed by a project rule, then it is restricted +to the minimum approvals required set in the project's corresponding rule. + +## Resetting approvals on push + +If approvals are [set at the project level](#editing-approvals), +you can choose whether all approvals on a merge request are removed when +new commits are pushed to the source branch of the merge request: + +1. Navigate to your project's **Settings > General** and expand + **Merge request approvals** +1. Tick the "Remove all approvals in a merge request when new commits are pushed to its source branch" + checkbox + +  + +1. Click **Save changes** + +NOTE: **Note:** +Approvals do not get reset when [rebasing a merge request](fast_forward_merge.md) +from the UI. +However, approvals will be reset if the target branch is changed. + +If you want approvals to persist, independent of changes to the merge request, +turn this setting to off by unchecking the box and saving the changes. + +## Allowing merge request authors to approve their own merge requests + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/issues/3349) in [GitLab Starter](https://about.gitlab.com/pricing/) 11.3. + +You can allow merge request authors to self-approve merge requests by +enabling it [at the project level](#editing-approvals). Authors +also need to be included in the approvers list in order to be able to +approve their merge request. + +1. Navigate to your project's **Settings > General** and expand **Merge request approvals**. +1. Uncheck the **Prevent approval of merge requests by merge request author** checkbox, which is enabled by default. +1. Click **Save changes**. + +## Prevent approval of merge requests by their committers + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/issues/10441) in [GitLab Starter](https://about.gitlab.com/pricing/) 11.10. + +You can prevent users that have committed to a merge request from approving it by +enabling [**Prevent approval of merge requests by their committers**](#prevent-approval-of-merge-requests-by-their-committers). + +1. Navigate to your project's **Settings > General** and expand **Merge request approvals**. +1. Tick the checkbox **Prevent approval of merge requests by their committers**. +1. Click **Save changes**. + +## Require authentication when approving a merge request **[STARTER]** + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/issues/5981) in [GitLab Starter](https://about.gitlab.com/pricing/) 12.0. + +You can force the approver to enter a password in order to authenticate who is approving the merge request by +enabling **Require user password to approve**. This enables an Electronic Signature +for approvals such as the one defined by [CFR Part 11](https://www.accessdata.fda.gov/scripts/cdrh/cfdocs/cfcfr/CFRSearch.cfm?CFRPart=11&showFR=1&subpartNode=21:1.0.1.1.8.3)): + +1. Navigate to your project's **Settings > General** and expand **Merge request approvals**. +1. Tick the checkbox **Require user password to approve**. +1. Click **Save changes**. + +## Merge requests with different source branch and target branch projects + +If the merge request source branch and target branch belong to different +projects (which happens in merge requests in forked projects), everything is +with respect to the target branch's project (typically the original project). +In particular, since the merge request in this case is part of the target +branch's project, the relevant settings are the target project's. The source +branch's project settings are not applicable. Even if you start the merge +request from the source branch's project UI, pay attention to the created merge +request itself. It belongs to the target branch's project. + +## Approver suggestions + +Approvers are suggested for merge requests based on the previous authors of the files affected by the merge request. + +## Filtering merge requests by approvers + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/9468) in [GitLab Starter](https://about.gitlab.com/pricing/) 11.9. + +To filter merge requests by an individual approver, you can type (or select from +the dropdown) `approver` and select the user. + + diff --git a/doc/user/project/merge_requests/merge_request_discussion_resolution.md b/doc/user/project/merge_requests/merge_request_discussion_resolution.md index 200965875a1..a554d727ca4 100644 --- a/doc/user/project/merge_requests/merge_request_discussion_resolution.md +++ b/doc/user/project/merge_requests/merge_request_discussion_resolution.md @@ -1 +1,5 @@ +--- +redirect_to: '../../discussions/index.md' +--- + This document was moved to [another location](../../discussions/index.md). diff --git a/doc/user/project/merge_requests/merge_when_build_succeeds.md b/doc/user/project/merge_requests/merge_when_build_succeeds.md index 2167fdfbf7e..cf5e3af16c9 100644 --- a/doc/user/project/merge_requests/merge_when_build_succeeds.md +++ b/doc/user/project/merge_requests/merge_when_build_succeeds.md @@ -1,5 +1,7 @@ -This document was moved to [merge_when_pipeline_succeeds](merge_when_pipeline_succeeds.md). +--- +redirect_to: 'merge_when_pipeline_succeeds.md' +--- ->[Introduced][ce-7135] by the "Rename MWBS service to Merge When Pipeline Succeeds" change. +This document was moved to [merge_when_pipeline_succeeds](merge_when_pipeline_succeeds.md). -[ce-7135]: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/7135 +>[Introduced](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/7135) by the "Rename MWBS service to Merge When Pipeline Succeeds" change. diff --git a/doc/user/project/merge_requests/merge_when_pipeline_succeeds.md b/doc/user/project/merge_requests/merge_when_pipeline_succeeds.md index bdd7d0022e6..1477e35dca8 100644 --- a/doc/user/project/merge_requests/merge_when_pipeline_succeeds.md +++ b/doc/user/project/merge_requests/merge_when_pipeline_succeeds.md @@ -1,4 +1,4 @@ -# Merge When Pipeline Succeeds +# Merge when pipeline succeeds When reviewing a merge request that looks ready to merge but still has one or more CI jobs running, you can set it to be merged automatically when the @@ -35,11 +35,11 @@ You need to have jobs configured to enable this feature. You can prevent merge requests from being merged if their pipeline did not succeed or if there are discussions to be resolved. -Navigate to your project's settings page, select the -**Only allow merge requests to be merged if the pipeline succeeds** check box and -hit **Save** for the changes to take effect. +Navigate to your project's settings page and expand the **Merge requests** section. +In the **Merge checks** subsection, select the **Pipelines must succeed** check +box and hit **Save** for the changes to take effect. - +  From now on, every time the pipeline fails you will not be able to merge the merge request from the UI, until you make all relevant jobs pass. diff --git a/doc/user/project/merge_requests/sast.md b/doc/user/project/merge_requests/sast.md new file mode 100644 index 00000000000..165290eb114 --- /dev/null +++ b/doc/user/project/merge_requests/sast.md @@ -0,0 +1,5 @@ +--- +redirect_to: '../../application_security/sast/index.md' +--- + +This document was moved to [another location](../../application_security/sast/index.md). diff --git a/doc/user/project/merge_requests/sast_docker.md b/doc/user/project/merge_requests/sast_docker.md new file mode 100644 index 00000000000..a062731ea35 --- /dev/null +++ b/doc/user/project/merge_requests/sast_docker.md @@ -0,0 +1,5 @@ +--- +redirect_to: '../../application_security/container_scanning/index.md' +--- + +This document was moved to [another location](../../application_security/container_scanning/index.md). diff --git a/doc/user/project/merge_requests/versions.md b/doc/user/project/merge_requests/versions.md index 90500fd9c21..70bd1e60594 100644 --- a/doc/user/project/merge_requests/versions.md +++ b/doc/user/project/merge_requests/versions.md @@ -1,6 +1,7 @@ # Merge requests versions > **Notes:** +> > - [Introduced][ce-5467] in GitLab 8.12. > - Comments are disabled while viewing outdated merge versions or comparing to > versions other than base. diff --git a/doc/user/project/merge_requests/work_in_progress_merge_requests.md b/doc/user/project/merge_requests/work_in_progress_merge_requests.md index 66ac7740157..6f33eb9a482 100644 --- a/doc/user/project/merge_requests/work_in_progress_merge_requests.md +++ b/doc/user/project/merge_requests/work_in_progress_merge_requests.md @@ -1,25 +1,46 @@ # "Work In Progress" Merge Requests -To prevent merge requests from accidentally being accepted before they're -completely ready, GitLab blocks the "Accept" button for merge requests that -have been marked a **Work In Progress**. +If a merge request is not yet ready to be merged, perhaps due to continued development +or open discussions, you can prevent it from being accepted before it's ready by flagging +it as a **Work In Progress**. This will disable the "Merge" button, preventing it from +being merged, and it will stay disabled until the "WIP" flag has been removed.  -To mark a merge request a Work In Progress, simply start its title with `[WIP]` -or `WIP:`. As an alternative, you're also able to do it by sending a commit -with its title starting with `wip` or `WIP` to the merge request's source branch. - - - -To allow a Work In Progress merge request to be accepted again when it's ready, -simply remove the `WIP` prefix. - - - -## Filtering merge requests with WIP Status - -To filter merge requests with the `WIP` status, you can type `wip` -and select the value for your filter from the merge request search input. +## Adding the "Work In Progress" flag to a Merge Request + +There are several ways to flag a merge request as a Work In Progress: + +- Add "[WIP]" or "WIP:" to the start of the merge request's title. Clicking on + **Start the title with WIP:**, under the title box, when editing the merge request's + description will have the same effect. +- Add the `/wip` [quick action](../quick_actions.md#quick-actions-for-issues-and-merge-requests) + in a discussion comment in the merge request. This is a toggle, and can be repeated + to change the status back. Note that any other text in the comment will be discarded. +- Add "wip" or "WIP" to the start of a commit message targeting the merge request's + source branch. This is not a toggle, and doing it again in another commit will have + no effect. + +## Removing the "Work In Progress" flag from a Merge Request + +Similar to above, when a Merge Request is ready to be merged, you can remove the +"Work in Progress" flag in several ways: + +- Remove "[WIP]" or "WIP:" from the start of the merge request's title. Clicking on + **Remove the WIP: prefix from the title**, under the title box, when editing the merge + request's description, will have the same effect. +- Add the `/wip` [quick action](../quick_actions.md#quick-actions-for-issues-and-merge-requests) + in a discussion comment in the merge request. This is a toggle, and can be repeated + to change the status back. Note that any other text in the comment will be discarded. +- Click on the **Resolve WIP status** button near the bottom of the merge request description, + next to the "Merge" button (see [image above](#work-in-progress-merge-requests)). + Must have at least Developer level permissions on the project for the button to + be visible. + +## Including/Excluding WIP Merge Requests when searching + +When viewing/searching the merge requests list, you can choose to include or exclude +WIP merge requests by adding a "WIP" filter in the search box, and choosing "Yes" +(to include) or "No" (to exclude).  diff --git a/doc/user/project/milestones/burndown_charts.md b/doc/user/project/milestones/burndown_charts.md new file mode 100644 index 00000000000..7ffeb032d7f --- /dev/null +++ b/doc/user/project/milestones/burndown_charts.md @@ -0,0 +1,70 @@ +# Burndown Charts **[STARTER]** + +> **Notes:** +> - [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/1540) in [GitLab Starter](https://about.gitlab.com/pricing/) 9.1 for project milestones. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/5354) in [GitLab Premium](https://about.gitlab.com/pricing/) 10.8 for group milestones. +> - [Added](https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/6495) to [GitLab Starter](https://about.gitlab.com/pricing/) 11.2 for group milestones. +> - Closed or reopened issues prior to GitLab 9.1 won't have a `closed_at` +> value, so the burndown chart considers them as closed on the milestone +> `start_date`. In that case, a warning will be displayed. + +## Overview + +Burndown Charts are visual representations of the progress of completing a milestone. + + + +At a glance, you see the current state for the completion a given milestone. +Without them, you would have to organize the data from the milestone and plot it +yourself to have the same sense of progress. + +GitLab Starter plots it for you and presents it in a clear and beautiful chart. + +For an overview, check the video demonstration on [Mapping Work Versus Time, With Burndown Charts](https://about.gitlab.com/2017/04/25/mapping-work-to-do-versus-time-with-burndown-charts/). + +## Use cases + +Burndown Charts, in general, are used for tracking and analyzing the completion of +a milestone. Therefore, their use cases are tied to the +[use you are assigning your milestone to](index.md). + +To exemplify, suppose you lead a team of developers in a large company, +and you follow this workflow: + +- Your company set the goal for the quarter to deliver 10 new features for your app + in the upcoming major release. +- You create a milestone, and remind your team to assign that milestone to every new issue + and merge request that's part of the launch of your app. +- Every week, you open the milestone, visualize the progress, identify the gaps, + and help your team to get their work done. +- Every month, you check in with your supervisor, and show the progress of that milestone + from the Burndown Chart. +- By the end of the quarter, your team successfully delivered 100% of that milestone, as + it was taken care of closely throughout the whole quarter. + +## How it works + +A Burndown Chart is available for every project or group milestone that has been attributed a **start +date** and a **due date**. + +Find your project's **Burndown Chart** under **Project > Issues > Milestones**, +and select a milestone from your current ones, while for group's, access the **Groups** dashboard, +select a group, and go through **Issues > Milestones** on the sidebar. + +NOTE: **Note:** +You're able to [promote project](index.md#promoting-project-milestones-to-group-milestones) to group milestones and still see the **Burndown Chart** for them, respecting license limitations. + +The chart indicates the project's progress throughout that milestone (for issues assigned to it). + +In particular, it shows how many issues were or are still open for a given day in the +milestone's corresponding period. + +The Burndown Chart tracks when issues were created and when they were last closed—not their full history. For each day, it takes the number of issues still open and issues created that day and subtracts the number of issues closed that day. +**Issues that were created and assigned a milestone before its start date—and remain open as of the start date—are considered as having been opened on the start date**. Therefore, when the milestone start date is changed the number of opened issues on each day may change. +Reopened issues are +considered as having been opened on the day after they were last closed. + +The Burndown Chart can also be toggled to display the cumulative open issue +weight for a given day. When using this feature, make sure issue weights have +been properly assigned, since an open issue with no weight adds zero to the +cumulative value. diff --git a/doc/user/project/milestones/img/burndown_chart.png b/doc/user/project/milestones/img/burndown_chart.png Binary files differnew file mode 100644 index 00000000000..e06b24f9907 --- /dev/null +++ b/doc/user/project/milestones/img/burndown_chart.png diff --git a/doc/user/project/milestones/index.md b/doc/user/project/milestones/index.md index e6033ca8655..6cd866b5c0d 100644 --- a/doc/user/project/milestones/index.md +++ b/doc/user/project/milestones/index.md @@ -83,7 +83,10 @@ From the project issue/merge request list pages and the group issue/merge reques ### Filtering in issue boards -From [project issue boards](../issue_board.md), you can filter by both group milestones and project milestones in the [search and filter bar](../../search/index.md#issue-boards). +- From [project issue boards](../issue_board.md), you can filter by both group milestones and project milestones in the [search and filter bar](../../search/index.md#issue-boards). +- From [group issue boards](../issue_board.md#group-issue-boards-premium), you can filter by only group milestones in the [search and filter bar](../../search/index.md#issue-boards). **[PREMIUM]** +- From [project issue boards](../issue_board.md), you can filter by both group milestones and project milestones in the [issue board configuration](../issue_board.md#configurable-issue-boards-starter). **[STARTER]** +- From [group issue boards](../issue_board.md#group-issue-boards-premium) you can filter by only group milestones in the [issue board configuration](../issue_board.md#configurable-issue-boards-starter). **[STARTER]** ### Special milestone filters @@ -92,22 +95,23 @@ When filtering by milestone, in addition to choosing a specific project mileston - **None**: Show issues or merge requests with no assigned milestone. - **Any**: Show issues or merge requests that have an assigned milestone. - **Upcoming**: Show issues or merge requests that have been assigned the open milestone that has the next upcoming due date (i.e. nearest due date in the future). -- **Started**: Show issues or merge requests that have an assigned milestone with a start date that is before today. +- **Started**: Show issues or merge requests that have an open assigned milestone with a start date that is before today. ## Milestone view Not all features in the project milestone view are available in the group milestone view. This table summarizes the differences: -| Feature | Project milestone view | Group milestone view | -|---|:---:|:---:| -| Title an description | ✓ | ✓ | -| Issues assigned to milestone | ✓ | | -| Merge requests assigned to milestone | ✓ | | -| Participants and labels used | ✓ | | -| Percentage complete | ✓ | ✓ | -| Start date and due date | ✓ | ✓ | -| Total issue time spent | ✓ | ✓ | -| Total issue weight | ✓ | | +| Feature | Project milestone view | Group milestone view | +|--------------------------------------|:----------------------:|:--------------------:| +| Title an description | ✓ | ✓ | +| Issues assigned to milestone | ✓ | | +| Merge requests assigned to milestone | ✓ | | +| Participants and labels used | ✓ | | +| Percentage complete | ✓ | ✓ | +| Start date and due date | ✓ | ✓ | +| Total issue time spent | ✓ | ✓ | +| Total issue weight | ✓ | | +| Burndown chart **[STARTER}** | ✓ | ✓ | The milestone view shows the title and description. @@ -118,12 +122,23 @@ These features are only available for project milestones and not group milestone - Issues assigned to the milestone are displayed in three columns: Unstarted issues, ongoing issues, and completed issues. - Merge requests assigned to the milestone are displayed in four columns: Work in progress merge requests, waiting for merge, rejected, and closed. - Participants and labels that are used in issues and merge requests that have the milestone assigned are displayed. +- [Burndown chart](#project-burndown-charts-starter). + +### Project Burndown Charts **[STARTER]** + +For project milestones in [GitLab Starter](https://about.gitlab.com/pricing), a [burndown chart](burndown_charts.md) is in the milestone view, showing the progress of completing a milestone. + + + +### Group Burndown Charts **[PREMIUM]** + +For group milestones in [GitLab Premium](https://about.gitlab.com/pricing), a [burndown chart](burndown_charts.md) is in the milestone view, showing the progress of completing a milestone. ### Milestone sidebar The milestone sidebar on the milestone view shows the following: -- Percentage complete, which is calculated as number of closed issues plus number of closed/merged merge requests divided by total number issues and merge requests. +- Percentage complete, which is calculated as number of closed issues divided by total number of issues. - The start date and due date. - The total time spent on all issues that have the milestone assigned. diff --git a/doc/user/project/new_ci_build_permissions_model.md b/doc/user/project/new_ci_build_permissions_model.md index d7a1a69f29d..d36312c9b8d 100644 --- a/doc/user/project/new_ci_build_permissions_model.md +++ b/doc/user/project/new_ci_build_permissions_model.md @@ -44,14 +44,14 @@ It is important to note that we have a few types of users: - **Administrators**: CI jobs created by Administrators will not have access to all GitLab projects, but only to projects and container images of projects - that the administrator is a member of.That means that if a project is either + that the administrator is a member of. That means that if a project is either public or internal users have access anyway, but if a project is private, the Administrator will have to be a member of it in order to have access to it via another project's job. -- **External users**: CI jobs created by [external users][ext] will have +- **External users**: CI jobs created by [external users](../permissions.md#external-users-permissions) will have access only to projects to which user has at least reporter access. This - rules out accessing all internal projects by default, + rules out accessing all internal projects by default. This allows us to make the CI and permission system more trustworthy. Let's consider the following scenario: @@ -60,7 +60,7 @@ Let's consider the following scenario: hosted in private repositories and you have multiple CI jobs that make use of these repositories. -1. You invite a new [external user][ext]. CI jobs created by that user do not +1. You invite a new [external user](../permissions.md#external-users-permissions). CI jobs created by that user do not have access to internal repositories, because the user also doesn't have the access from within GitLab. You as an employee have to grant explicit access for this user. This allows us to prevent from accidental data leakage. @@ -205,6 +205,7 @@ With the update permission model we also extended the support for accessing Container Registries for private projects. > **Notes:** +> > - GitLab Runner versions prior to 1.8 don't incorporate the introduced changes > for permissions. This makes the `image:` directive to not work with private > projects automatically and it needs to be configured manually on Runner's host @@ -232,7 +233,6 @@ test: [job permissions]: ../permissions.md#job-permissions [comment]: https://gitlab.com/gitlab-org/gitlab-ce/issues/22484#note_16648302 -[ext]: ../permissions.md#external-users [gitsub]: ../../ci/git_submodules.md [https]: ../admin_area/settings/visibility_and_access_controls.md#enabled-git-access-protocols [triggers]: ../../ci/triggers/README.md diff --git a/doc/user/project/operations/error_tracking.md b/doc/user/project/operations/error_tracking.md index 41aa5b91aa7..1b319c5641c 100644 --- a/doc/user/project/operations/error_tracking.md +++ b/doc/user/project/operations/error_tracking.md @@ -22,8 +22,12 @@ GitLab provides an easy way to connect Sentry to your project: 1. Sign up to Sentry.io or [deploy your own](#deploying-sentry) Sentry instance. 1. [Find or generate](https://docs.sentry.io/api/auth/) a Sentry auth token for your Sentry project. Make sure to give the token at least the following scopes: `event:read` and `project:read`. -1. Navigate to your project’s **Settings > Operations** and provide the Sentry API URL and auth token. -1. Ensure that the 'Active' checkbox is set. +1. Navigate to your project’s **Settings > Operations**. +1. Ensure that the **Active** checkbox is set. +1. In the **Sentry API URL** field, enter your Sentry hostname. For example, `https://sentry.example.com`. +1. In the **Auth Token** field, enter the token you previously generated. +1. Click the **Connect** button to test the connection to Sentry and populate the **Project** dropdown. +1. From the **Project** dropdown, choose a Sentry project to link to your GitLab project. 1. Click **Save changes** for the changes to take effect. 1. You can now visit **Operations > Error Tracking** in your project's sidebar to [view a list](#error-tracking-list) of Sentry errors. diff --git a/doc/user/project/operations/feature_flags.md b/doc/user/project/operations/feature_flags.md new file mode 100644 index 00000000000..a5db3d70bb9 --- /dev/null +++ b/doc/user/project/operations/feature_flags.md @@ -0,0 +1,177 @@ +# Feature Flags **[PREMIUM]** + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/11845) in GitLab 11.4. + +CAUTION: **Warning:** +This an _alpha_ feature and is subject to change at any time without +prior notice. + +Feature flags allow you to ship a project in different flavors by +dynamically toggling certain functionality. + +## Overview + +Feature Flags offer a feature toggle system for your application. They enable teams +to achieve Continuous Delivery by deploying new features to production at smaller +batches for controlled testing, separating feature delivery from customer launch. +This helps reducing risk and allows you to easily manage which features to enable. + +GitLab offers a Feature Flags interface that allows you to create, toggle and +remove feature flags. + +## How it works + +Underneath, GitLab uses [unleash](https://github.com/Unleash/unleash), a feature +toggle service. GitLab provides an API where your application can talk to and get the +list of feature flags you set in GitLab. + +The application must be configured to talk to GitLab, so that's up to the +developers to use a compatible [client library](#client-libraries) and +integrate it in their app. + +By setting a flag active or inactive via GitLab, your application will automatically +know which features to enable or disable respectively. + +## Adding a new feature flag + +To add a new feature flag: + +1. Navigate to your project's **Operations > Feature Flags**. +1. Click on the **New Feature Flag** button. +1. Give it a name. + + NOTE: **Note:** + A name can contain only lowercase letters, digits, underscores (`_`) + and dashes (`-`), must start with a letter, and cannot end with a dash (`-`) + or an underscore (`_`). + +1. Give it a description (optional, 255 characters max). +1. Define environment [specs](#define-environment-specs). If you want the flag on by default, enable the catch-all [wildcard spec (`*`)](#define-environment-specs) +1. Click **Create feature flag**. + +Once a feature flag is created, the list of existing feature flags will be presented +with ability to edit or remove them. + +To make a feature flag active or inactive, click the pencil icon to edit it, +and toggle the status for each [spec](#define-environment-specs). + + + +## Define environment specs + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/issues/8621) in GitLab 11.8. + +In general, an application is deployed to multiple environments, such as +production, staging and [review apps](../../../ci/review_apps/index.md). +For example, you may not want to enable a feature flag on production until your QA team has +first confirmed that the feature is working correctly on testing environments. + +To handle these situations, you can enable a feature flag on a particular environment +with [Environment specs](../../../ci/environments.md#scoping-environments-with-specs-premium). +You can define multiple specs per flag so that you can control your feature flag more granularly. + +To define specs for each environment: + +1. Navigate to your project's **Operations > Feature Flags**. +1. Click on the **New Feature Flag** button or edit an existing flag. +1. Set the status of the default [spec](../../../ci/environments.md#scoping-environments-with-specs-premium) (`*`). This status will be used for _all_ environments. +1. If you want to enable/disable the feature on a specific environment, create a new [spec](../../../ci/environments.md#scoping-environments-with-specs-premium) and type the environment name. +1. Set the status of the additional spec. This status takes precedence over the default spec's status since we always use the most specific match available. +1. Click **Create feature flag** or **Update feature flag**. + + + +NOTE: **NOTE** +We'd highly recommend you to use the [Environment](../../../ci/environments.md) +feature in order to quickly assess which flag is enabled per environment. + +## Integrating with your application + +In order to use Feature Flags, you need to first +[get the access credentials](#configuring-feature-flags) from GitLab and then +prepare your application and hook it with a [client library](#client-libraries). + +### Configuring Feature Flags + +To get the access credentials that your application will need to talk to GitLab: + +1. Navigate to your project's **Operations > Feature Flags**. +1. Click on the **Configure** button to see the values: + - **API URL**: URL where the client (application) connects to get a list of feature flags. + - **Instance ID**: Unique token that authorizes the retrieval of the feature flags. + - **Application name**: The name of the running environment. For instance, + if the application runs for production server, application name would be + `production` or similar. This value is used for + [the environment spec evaluation](#define-environment-specs). + +NOTE: **Note:** +The meaning of these fields might change over time. For example, we are not sure +if **Instance ID** will be single token or multiple tokens, assigned to the +**Environment**. Also, **Application name** could describe the version of +application instead of the running environment. + +### Client libraries + +GitLab currently implements a single backend that is compatible with +[Unleash](https://github.com/Unleash/unleash#client-implementations) clients. + +Unleash clients allow the developers to define in the app's code the default +values for flags. Each feature flag evaluation can express the desired +outcome in case the flag isn't present on the provided configuration file. + +Unleash currently offers a number of official SDKs for various frameworks and +a number of community contributed libraries. + +Official clients: + +- [unleash/unleash-client-java](https://github.com/unleash/unleash-client-java) +- [unleash/unleash-client-node](https://github.com/unleash/unleash-client-node) +- [unleash/unleash-client-go](https://github.com/unleash/unleash-client-go) +- [unleash/unleash-client-ruby](https://github.com/unleash/unleash-client-ruby) + +Community contributed clients: + +- [stiano/unleash-client-dotnet](https://github.com/stiano/unleash-client-dotnet) (.Net Core) +- [onybo/unleash-client-core](https://github.com/onybo/unleash-client-core) (.Net Core) +- [aes/unleash-client-python](https://github.com/aes/unleash-client-python) (Python 3) + +### Golang application example + +Here's an example of how to integrate the feature flags in a Golang application: + +```golang +package main + +import ( + "io" + "log" + "net/http" + + "github.com/Unleash/unleash-client-go" +) + +type metricsInterface struct { +} + +func init() { + unleash.Initialize( + unleash.WithUrl("https://gitlab.com/api/v4/feature_flags/unleash/42"), + unleash.WithInstanceId("29QmjsW6KngPR5JNPMWx"), + unleash.WithAppName("production"), + unleash.WithListener(&metricsInterface{}), + ) +} + +func helloServer(w http.ResponseWriter, req *http.Request) { + if unleash.IsEnabled("my_feature_name") { + io.WriteString(w, "Feature enabled\n") + } else { + io.WriteString(w, "hello, world!\n") + } +} + +func main() { + http.HandleFunc("/", helloServer) + log.Fatal(http.ListenAndServe(":8080", nil)) +} +``` diff --git a/doc/user/project/operations/img/feature_flags_list.png b/doc/user/project/operations/img/feature_flags_list.png Binary files differnew file mode 100644 index 00000000000..5313a163fec --- /dev/null +++ b/doc/user/project/operations/img/feature_flags_list.png diff --git a/doc/user/project/operations/img/specs_list.png b/doc/user/project/operations/img/specs_list.png Binary files differnew file mode 100644 index 00000000000..9630c907cfc --- /dev/null +++ b/doc/user/project/operations/img/specs_list.png diff --git a/doc/user/project/operations/index.md b/doc/user/project/operations/index.md index b0f9936be5c..0086c15c98a 100644 --- a/doc/user/project/operations/index.md +++ b/doc/user/project/operations/index.md @@ -7,5 +7,5 @@ your applications: - Deploy to different [environments](../../../ci/environments.md). - Connect your project to a [Kubernetes cluster](../clusters/index.md). - Discover and view errors generated by your applications with [Error Tracking](error_tracking.md). -- Create, toggle, and remove [Feature Flags](https://docs.gitlab.com/ee/user/project/operations/feature_flags.html). **[PREMIUM]** -- [Trace](https://docs.gitlab.com/ee/user/project/operations/tracing.html) the performance and health of a deployed application. **[ULTIMATE]** +- Create, toggle, and remove [Feature Flags](feature_flags.md). **[PREMIUM]** +- [Trace](tracing.md) the performance and health of a deployed application. **[ULTIMATE]** diff --git a/doc/user/project/operations/tracing.md b/doc/user/project/operations/tracing.md new file mode 100644 index 00000000000..0416e096cf2 --- /dev/null +++ b/doc/user/project/operations/tracing.md @@ -0,0 +1,34 @@ +# Tracing **[ULTIMATE]** + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/7903) in GitLab Ultimate 11.5. + +Tracing provides insight into the performance and health of a deployed application, +tracking each function or microservice which handles a given request. + +This makes it easy to +understand the end-to-end flow of a request, regardless of whether you are using a monolithic or distributed system. + +## Jaeger tracing + +[Jaeger](https://www.jaegertracing.io/) is an open source, end-to-end distributed +tracing system used for monitoring and troubleshooting microservices-based distributed +systems. + +### Deploying Jaeger + +To learn more about deploying Jaeger, read the official +[Getting Started documentation](https://www.jaegertracing.io/docs/latest/getting-started/). +There is an easy to use [all-in-one Docker image](https://www.jaegertracing.io/docs/latest/getting-started/#AllinoneDockerimage), +as well as deployment options for [Kubernetes](https://github.com/jaegertracing/jaeger-kubernetes) +and [OpenShift](https://github.com/jaegertracing/jaeger-openshift). + +### Enabling Jaeger + +GitLab provides an easy way to open the Jaeger UI from within your project: + +1. [Set up Jaeger](#deploying-jaeger) and configure your application using one of the + [client libraries](https://www.jaegertracing.io/docs/latest/client-libraries/). +1. Navigate to your project's **Settings > Operations** and provide the Jaeger URL. +1. Click **Save changes** for the changes to take effect. +1. You can now visit **Operations > Tracing** in your project's sidebar and + GitLab will redirect you to the configured Jaeger URL.
\ No newline at end of file diff --git a/doc/user/project/packages/img/maven_package_view.png b/doc/user/project/packages/img/maven_package_view.png Binary files differnew file mode 100644 index 00000000000..2eb4b6f76b4 --- /dev/null +++ b/doc/user/project/packages/img/maven_package_view.png diff --git a/doc/user/project/packages/img/npm_package_view.png b/doc/user/project/packages/img/npm_package_view.png Binary files differnew file mode 100644 index 00000000000..8baf7d0ef9f --- /dev/null +++ b/doc/user/project/packages/img/npm_package_view.png diff --git a/doc/user/project/packages/maven.md b/doc/user/project/packages/maven.md new file mode 100644 index 00000000000..266225fdb8d --- /dev/null +++ b/doc/user/project/packages/maven.md @@ -0,0 +1,5 @@ +--- +redirect_to: 'maven_repository.md' +--- + +This document was moved to [another location](maven_repository.md). diff --git a/doc/user/project/packages/maven_packages.md b/doc/user/project/packages/maven_packages.md new file mode 100644 index 00000000000..266225fdb8d --- /dev/null +++ b/doc/user/project/packages/maven_packages.md @@ -0,0 +1,5 @@ +--- +redirect_to: 'maven_repository.md' +--- + +This document was moved to [another location](maven_repository.md). diff --git a/doc/user/project/packages/maven_repository.md b/doc/user/project/packages/maven_repository.md new file mode 100644 index 00000000000..9b7af738696 --- /dev/null +++ b/doc/user/project/packages/maven_repository.md @@ -0,0 +1,341 @@ +# GitLab Maven Repository **[PREMIUM]** + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/issues/5811) in + [GitLab Premium](https://about.gitlab.com/pricing/) 11.3. + +With the GitLab [Maven](https://maven.apache.org) Repository, every +project can have its own space to store its Maven artifacts. + + + +## Enabling the Maven Repository + +NOTE: **Note:** +This option is available only if your GitLab administrator has +[enabled support for the Maven repository](../../../administration/packages.md).**[PREMIUM ONLY]** + +After the Packages feature is enabled, the Maven Repository will be available for +all new projects by default. To enable it for existing projects, or if you want +to disable it: + +1. Navigate to your project's **Settings > General > Permissions**. +1. Find the Packages feature and enable or disable it. +1. Click on **Save changes** for the changes to take effect. + +You should then be able to see the **Packages** section on the left sidebar. +Next, you must configure your project to authorize with the GitLab Maven +repository. + +## Authenticating to the GitLab Maven Repository + +If a project is private or you want to upload Maven artifacts to GitLab, +credentials will need to be provided for authorization. Support is available for +[personal access tokens](#authenticating-with-a-personal-access-token) and +[CI job tokens](#authenticating-with-a-ci-job-token) only. +[Deploy tokens](../deploy_tokens/index.md) and regular username/password +credentials do not work. + +### Authenticating with a personal access token + +To authenticate with a [personal access token](../../profile/personal_access_tokens.md), +add a corresponding section to your +[`settings.xml`](https://maven.apache.org/settings.html) file: + +```xml +<settings> + <servers> + <server> + <id>gitlab-maven</id> + <configuration> + <httpHeaders> + <property> + <name>Private-Token</name> + <value>REPLACE_WITH_YOUR_PERSONAL_ACCESS_TOKEN</value> + </property> + </httpHeaders> + </configuration> + </server> + </servers> +</settings> +``` + +You should now be able to upload Maven artifacts to your project. + +### Authenticating with a CI job token + +If you're using Maven with GitLab CI/CD, a CI job token can be used instead +of a personal access token. + +To authenticate with a CI job token, add a corresponding section to your +[`settings.xml`](https://maven.apache.org/settings.html) file: + +```xml +<settings> + <servers> + <server> + <id>gitlab-maven</id> + <configuration> + <httpHeaders> + <property> + <name>Job-Token</name> + <value>${env.CI_JOB_TOKEN}</value> + </property> + </httpHeaders> + </configuration> + </server> + </servers> +</settings> +``` + +You can read more on +[how to create Maven packages using GitLab CI/CD](#creating-maven-packages-with-gitlab-cicd). + +## Configuring your project to use the GitLab Maven repository URL + +To download and upload packages from GitLab, you need a `repository` and +`distributionManagement` section in your `pom.xml` file. + +Depending on your workflow and the amount of Maven packages you have, there are +3 ways you can configure your project to use the GitLab endpoint for Maven packages: + +- **Project level**: Useful when you have few Maven packages which are not under + the same GitLab group. +- **Group level**: Useful when you have many Maven packages under the same GitLab + group. +- **Instance level**: Useful when you have many Maven packages under different + GitLab groups or on their own namespace. + +NOTE: **Note:** +In all cases, you need a project specific URL for uploading a package in +the `distributionManagement` section. + +### Project level Maven endpoint + +The example below shows how the relevant `repository` section of your `pom.xml` +would look like: + +```xml +<repositories> + <repository> + <id>gitlab-maven</id> + <url>https://gitlab.com/api/v4/projects/PROJECT_ID/packages/maven</url> + </repository> +</repositories> +<distributionManagement> + <repository> + <id>gitlab-maven</id> + <url>https://gitlab.com/api/v4/projects/PROJECT_ID/packages/maven</url> + </repository> + <snapshotRepository> + <id>gitlab-maven</id> + <url>https://gitlab.com/api/v4/projects/PROJECT_ID/packages/maven</url> + </snapshotRepository> +</distributionManagement> +``` + +The `id` must be the same with what you +[defined in `settings.xml`](#authenticating-to-the-gitlab-maven-repository). + +Replace `PROJECT_ID` with your project ID which can be found on the home page +of your project. + +If you have a self-hosted GitLab installation, replace `gitlab.com` with your +domain name. + +NOTE: **Note:** +For retrieving artifacts, you can use either the +[URL encoded](../../../api/README.md#namespaced-path-encoding) path of the project +(e.g., `group%2Fproject`) or the project's ID (e.g., `42`). However, only the +project's ID can be used for uploading. + +### Group level Maven endpoint + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/8798) in GitLab Premium 11.7. + +If you rely on many packages, it might be inefficient to include the `repository` section +with a unique URL for each package. Instead, you can use the group level endpoint for +all your Maven packages stored within one GitLab group. Only packages you have access to +will be available for download. + +The group level endpoint works with any package names, which means the you +have the flexibility of naming compared to [instance level endpoint](#instance-level-maven-endpoint). +However, GitLab will not guarantee the uniqueness of the package names within +the group. You can have two projects with the same package name and package +version. As a result, GitLab will serve whichever one is more recent. + +The example below shows how the relevant `repository` section of your `pom.xml` +would look like. You still need a project specific URL for uploading a package in +the `distributionManagement` section: + +```xml +<repositories> + <repository> + <id>gitlab-maven</id> + <url>https://gitlab.com/api/v4/groups/my-group/-/packages/maven</url> + </repository> +</repositories> +<distributionManagement> + <repository> + <id>gitlab-maven</id> + <url>https://gitlab.com/api/v4/projects/PROJECT_ID/packages/maven</url> + </repository> + <snapshotRepository> + <id>gitlab-maven</id> + <url>https://gitlab.com/api/v4/projects/PROJECT_ID/packages/maven</url> + </snapshotRepository> +</distributionManagement> +``` + +The `id` must be the same with what you +[defined in `settings.xml`](#authenticating-to-the-gitlab-maven-repository). + +Replace `my-group` with your group name and `PROJECT_ID` with your project ID +which can be found on the home page of your project. + +If you have a self-hosted GitLab installation, replace `gitlab.com` with your +domain name. + +NOTE: **Note:** +For retrieving artifacts, you can use either the +[URL encoded](../../../api/README.md#namespaced-path-encoding) path of the group +(e.g., `group%2Fsubgroup`) or the group's ID (e.g., `12`). + +### Instance level Maven endpoint + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/8274) in GitLab Premium 11.7. + +If you rely on many packages, it might be inefficient to include the `repository` section +with a unique URL for each package. Instead, you can use the instance level endpoint for +all maven packages stored in GitLab and the packages you have access to will be available +for download. + +Note that **only packages that have the same path as the project** are exposed via +the instance level endpoint. + +| Project | Package | Instance level endpoint available | +| ------- | ------- | --------------------------------- | +| `foo/bar` | `foo/bar/1.0-SNAPSHOT` | Yes | +| `gitlab-org/gitlab-ce` | `foo/bar/1.0-SNAPSHOT` | No | +| `gitlab-org/gitlab-ce` | `gitlab-org/gitlab-ce/1.0-SNAPSHOT` | Yes | + +The example below shows how the relevant `repository` section of your `pom.xml` +would look like. You still need a project specific URL for uploading a package in +the `distributionManagement` section: + +```xml +<repositories> + <repository> + <id>gitlab-maven</id> + <url>https://gitlab.com/api/v4/packages/maven</url> + </repository> +</repositories> +<distributionManagement> + <repository> + <id>gitlab-maven</id> + <url>https://gitlab.com/api/v4/projects/PROJECT_ID/packages/maven</url> + </repository> + <snapshotRepository> + <id>gitlab-maven</id> + <url>https://gitlab.com/api/v4/projects/PROJECT_ID/packages/maven</url> + </snapshotRepository> +</distributionManagement> +``` + +The `id` must be the same with what you +[defined in `settings.xml`](#authenticating-to-the-gitlab-maven-repository). + +Replace `PROJECT_ID` with your project ID which can be found on the home page +of your project. + +If you have a self-hosted GitLab installation, replace `gitlab.com` with your +domain name. + +NOTE: **Note:** +For retrieving artifacts, you can use either the +[URL encoded](../../../api/README.md#namespaced-path-encoding) path of the project +(e.g., `group%2Fproject`) or the project's ID (e.g., `42`). However, only the +project's ID can be used for uploading. + +## Uploading packages + +Once you have set up the [authentication](#authenticating-to-the-gitlab-maven-repository) +and [configuration](#configuring-your-project-to-use-the-gitlab-maven-repository-url), +test to upload a Maven artifact from a project of yours: + +```sh +mvn deploy +``` + +You can then navigate to your project's **Packages** page and see the uploaded +artifacts or even delete them. + +## Creating Maven packages with GitLab CI/CD + +Once you have your repository configured to use the GitLab Maven Repository, +you can configure GitLab CI/CD to build new packages automatically. The example below +shows how to create a new package each time the `master` branch is updated: + +1. Create a `ci_settings.xml` file that will serve as Maven's `settings.xml` file. + Add the server section with the same id you defined in your `pom.xml` file. + For example, in our case it's `gitlab-maven`: + + ```xml + <settings xmlns="http://maven.apache.org/SETTINGS/1.1.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" + xsi:schemaLocation="http://maven.apache.org/SETTINGS/1.1.0 http://maven.apache.org/xsd/settings-1.1.0.xsd"> + <servers> + <server> + <id>gitlab-maven</id> + <configuration> + <httpHeaders> + <property> + <name>Job-Token</name> + <value>${env.CI_JOB_TOKEN}</value> + </property> + </httpHeaders> + </configuration> + </server> + </servers> + </settings> + ``` + +1. Make sure your `pom.xml` file includes the following: + + ```xml + <repositories> + <repository> + <id>gitlab-maven</id> + <url>https://gitlab.com/api/v4/projects/${env.CI_PROJECT_ID}/packages/maven</url> + </repository> + </repositories> + <distributionManagement> + <repository> + <id>gitlab-maven</id> + <url>https://gitlab.com/api/v4/projects/${env.CI_PROJECT_ID}/packages/maven</url> + </repository> + <snapshotRepository> + <id>gitlab-maven</id> + <url>https://gitlab.com/api/v4/projects/${env.CI_PROJECT_ID}/packages/maven</url> + </snapshotRepository> + </distributionManagement> + ``` + + TIP: **Tip:** + You can either let Maven utilize the CI environment variables or hardcode your project's ID. + +1. Add a `deploy` job to your `.gitlab-ci.yml` file: + + ```yaml + deploy: + image: maven:3.3.9-jdk-8 + script: + - 'mvn deploy -s ci_settings.xml' + only: + - master + ``` + +1. Push those files to your repository. + +The next time the `deploy` job runs, it will copy `ci_settings.xml` to the +user's home location (in this case the user is `root` since it runs in a +Docker container), and Maven will utilize the configured CI +[environment variables](../../../ci/variables/README.md#predefined-environment-variables). diff --git a/doc/user/project/packages/npm_registry.md b/doc/user/project/packages/npm_registry.md new file mode 100644 index 00000000000..2e274573434 --- /dev/null +++ b/doc/user/project/packages/npm_registry.md @@ -0,0 +1,120 @@ +# GitLab NPM Registry **[PREMIUM]** + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/issues/5934) + in [GitLab Premium](https://about.gitlab.com/pricing/) 11.7. + +With the GitLab NPM Registry, every +project can have its own space to store NPM packages. + + + +NOTE: **Note:** +Only [scoped](https://docs.npmjs.com/misc/scope) packages are supported. + + +NOTE: **Note:** +As `@group/subgroup/project` is not a valid NPM package name, publishing a package +within a subgroup is not supported yet. + +## Enabling the NPM Registry + +NOTE: **Note:** +This option is available only if your GitLab administrator has +[enabled support for the NPM registry](../../../administration/packages.md).**[PREMIUM ONLY]** + +After the NPM registry is enabled, it will be available for all new projects +by default. To enable it for existing projects, or if you want to disable it: + +1. Navigate to your project's **Settings > General > Permissions**. +1. Find the Packages feature and enable or disable it. +1. Click on **Save changes** for the changes to take effect. + +You should then be able to see the **Packages** section on the left sidebar. + +Before proceeding to authenticating with the GitLab NPM Registry, you should +get familiar with the package naming convention. + +## Package naming convention + +**Only packages that have the same path as the project** are supported. For + example: + +| Project | Package | Supported | +| ---------------------- | ----------------------- | --------- | +| `foo/bar` | `@foo/bar` | Yes | +| `gitlab-org/gitlab-ce` | `@gitlab-org/gitlab-ce` | Yes | +| `gitlab-org/gitlab-ce` | `@foo/bar` | No | + +Now, you can configure your project to authenticate with the GitLab NPM +Registry. + +## Authenticating to the GitLab NPM Registry + +If a project is private or you want to upload an NPM package to GitLab, +credentials will need to be provided for authentication. Support is available +only for [OAuth tokens](../../../api/oauth2.md#resource-owner-password-credentials-flow). + +CAUTION: **2FA not supported:** +Authentication for personal access tokens is not yet supported +([#9140](https://gitlab.com/gitlab-org/gitlab-ee/issues/9140)). If you have 2FA +enabled, you won't be able to authenticate to the GitLab NPM Registry. + +### Authenticating with an OAuth token + +To authenticate with an [OAuth token](../../../api/oauth2.md#resource-owner-password-credentials-flow), +add a corresponding section to your `.npmrc` file: + +```ini +; Set URL for your scoped packages. +; For example package with name `@foo/bar` will use this URL for download +@foo:registry=https://gitlab.com/api/v4/packages/npm/ + +; Add the OAuth token for the scoped packages URL. This will allow you to download +; `@foo/` packages from private projects. +//gitlab.com/api/v4/packages/npm/:_authToken=<your_oauth_token> + +; Add OAuth token for uploading to the registry. Replace <your_project_id> +; with the project you want your package to be uploaded to. +//gitlab.com/api/v4/projects/<your_project_id>/packages/npm/:_authToken=<your_oauth_token> +``` + +Replace `<your_project_id>` with your project ID which can be found on the home page +of your project and `<your_oauth_token>` with your OAuth token. + +If you have a self-hosted GitLab installation, replace `gitlab.com` with your +domain name. + +You should now be able to download and upload NPM packages to your project. + +## Uploading packages + +Before you will be able to upload a package, you need to specify the registry +for NPM. To do this, add the following section to the bottom of `package.json`: + +```json + "publishConfig": { + "@foo:registry":"https://gitlab.com/api/v4/projects/<your_project_id>/packages/npm/" + } +``` + +Replace `<your_project_id>` with your project ID, which can be found on the home +page of your project, and replace `@foo` with your own scope. + +If you have a self-hosted GitLab installation, replace `gitlab.com` with your +domain name. + +Once you have enabled it and set up [authentication](#authenticating-to-the-gitlab-npm-registry), +you can upload an NPM package to your project: + +```sh +npm publish +``` + +You can then navigate to your project's **Packages** page and see the uploaded +packages or even delete them. + +## Uploading a package with the same version twice + +If you upload a package with a same name and version twice, GitLab will show +both packages in the UI, but the GitLab NPM Registry will expose the most recent +one as it supports only one package per version for `npm install`. diff --git a/doc/user/project/pages/getting_started_part_four.md b/doc/user/project/pages/getting_started_part_four.md index 05d5a2fd99a..8baf41dba78 100644 --- a/doc/user/project/pages/getting_started_part_four.md +++ b/doc/user/project/pages/getting_started_part_four.md @@ -1,22 +1,30 @@ --- -last_updated: 2018-02-16 -author: Marcia Ramos -author_gitlab: marcia -level: intermediate -article_type: user guide -date: 2017-02-22 +last_updated: 2019-06-04 +type: reference, howto --- # Creating and Tweaking GitLab CI/CD for GitLab Pages -[GitLab CI](https://about.gitlab.com/gitlab-ci/) serves +To [get started with GitLab Pages](index.md#getting-started), you can +use one of the project templates, a `.gitlab-ci.yml` template, +or fork an existing example project. Therefore, you don't need to +understand _all_ the ins and odds of GitLab CI/CD to get your site +deployed. Still, there are cases where you want to write your own +script or tweak an existing one. This document guides you through +this process. + +This guide also provides a general overview and clear introduction +for **getting familiar with the `.gitlab-ci.yml` file and writing +one for the first time.** + +[GitLab CI/CD](../../../ci/README.md) serves numerous purposes, to build, test, and deploy your app from GitLab through -[Continuous Integration, Continuous Delivery, and Continuous Deployment](https://about.gitlab.com/2016/08/05/continuous-integration-delivery-and-deployment-with-gitlab/) +[Continuous Integration, Continuous Delivery, and Continuous Deployment](../../../ci/introduction/index.md#introduction-to-cicd-methodologies) methods. You will need it to build your website with GitLab Pages, and deploy it to the Pages server. -To implement GitLab CI/CD, the first thing we need is a configuration +To implement GitLab CI/CD, the first thing you need is a configuration file called `.gitlab-ci.yml` placed at your website's root directory. What this file actually does is telling the @@ -26,7 +34,7 @@ terminal. GitLab CI/CD tells the Runner which commands to run. Both are built-in in GitLab, and you don't need to set up anything for them to work. -Explaining [every detail of GitLab CI](https://docs.gitlab.com/ce/ci/yaml/README.html) +Explaining [every detail of GitLab CI/CD](../../../ci/yaml/README.md) and GitLab Runner is out of the scope of this guide, but we'll need to understand just a few things to be able to write our own `.gitlab-ci.yml` or tweak an existing one. It's an @@ -42,7 +50,7 @@ Of course, before building it, you had to install Jekyll in your computer. For that, you had to open your terminal and run `gem install jekyll`. Right? GitLab CI + GitLab Runner do the same thing. But you need to write in the `.gitlab-ci.yml` the script you want to run so -GitLab Runner will do it for you. It looks more complicated then it +GitLab Runner will do it for you. It looks more complicated than it is. What you need to tell the Runner: ``` diff --git a/doc/user/project/pages/getting_started_part_one.md b/doc/user/project/pages/getting_started_part_one.md index 9a95fb70964..6d538ca2455 100644 --- a/doc/user/project/pages/getting_started_part_one.md +++ b/doc/user/project/pages/getting_started_part_one.md @@ -1,42 +1,12 @@ --- -last_updated: 2018-02-16 -author: Marcia Ramos -author_gitlab: marcia -level: beginner -article_type: user guide -date: 2017-02-22 +last_updated: 2018-06-04 +type: concepts, reference --- # Static sites and GitLab Pages domains -This document is the beginning of a comprehensive guide, made for those who want to -publish a website with GitLab Pages but aren't familiar with -the entire process involved. - -This [first document](#what-you-need-to-know-before-getting-started) of this series will present you to the concepts of -static sites, and go over how the default Pages domains work. - -The [second document](getting_started_part_two.md) covers how to get started with GitLab Pages: deploy -a website from a forked project or create a new one from scratch. - -The [third document](getting_started_part_three.md) will show you how to set up a custom domain or subdomain -to your site already deployed. - -The [fourth document](getting_started_part_four.md) will show you how to create and tweak GitLab CI for -GitLab Pages. - -To **enable** GitLab Pages for GitLab CE (Community Edition) -and GitLab EE (Enterprise Edition), please read the -[admin documentation](https://docs.gitlab.com/ce/administration/pages/index.html), -and/or watch this [video tutorial](https://youtu.be/dD8c7WNcc6s). - ->**Note:** -For this guide, we assume you already have GitLab Pages -server up and running for your GitLab instance. - -## What you need to know before getting started - -Before we begin, let's understand a few concepts first. +On this docucument, learn how to name your project for GitLab Pages +according to your intended website's URL. ## Static sites @@ -48,20 +18,10 @@ CSS, and JS, or use a [Static Site Generator (SSG)](https://www.staticgen.com/) to simplify your code and build the static site for you, which is highly recommendable and much faster than hardcoding. -### Further reading - -- Read through this technical overview on [Static versus Dynamic Websites](https://about.gitlab.com/2016/06/03/ssg-overview-gitlab-pages-part-1-dynamic-x-static/) -- Understand [how modern Static Site Generators work](https://about.gitlab.com/2016/06/10/ssg-overview-gitlab-pages-part-2/) and what you can add to your static site -- You can use [any SSG with GitLab Pages](https://about.gitlab.com/2016/06/17/ssg-overview-gitlab-pages-part-3-examples-ci/) -- Fork an [example project](https://gitlab.com/pages) to build your website based upon - -## GitLab Pages domain +See the [further reading](#further-reading) section below for +references on static site concepts. -If you set up a GitLab Pages project on GitLab.com, -it will automatically be accessible under a -[subdomain of `namespace.gitlab.io`](introduction.md#gitlab-pages-on-gitlab-com). -The `namespace` is defined by your username on GitLab.com, -or the group name you created this project under. +## GitLab Pages domain names >**Note:** If you use your own GitLab instance to deploy your @@ -70,11 +30,32 @@ Pages wildcard domain. This guide is valid for any GitLab instance, you just need to replace Pages wildcard domain on GitLab.com (`*.gitlab.io`) with your own. -Learn more about [namespaces](../../group/index.md#namespaces). +If you set up a GitLab Pages project on GitLab, +it will automatically be accessible under a +subdomain of `namespace.example.io`. +The [`namespace`](../../group/index.md#namespaces) +is defined by your username on GitLab.com, +or the group name you created this project under. +For GitLab self-managed instances, replace `example.io` +with your instance's Pages domain. For GitLab.com, +Pages domains are `*.gitlab.io`. + +| Type of GitLab Pages | The name of the project created in GitLab | Website URL | +| -------------------- | ------------ | ----------- | +| User pages | `username.example.io` | `http(s)://username.example.io` | +| Group pages | `groupname.example.io` | `http(s)://groupname.example.io` | +| Project pages owned by a user | `projectname` | `http(s)://username.example.io/projectname` | +| Project pages owned by a group | `projectname` | `http(s)://groupname.example.io/projectname`| +| Project pages owned by a subgroup | `subgroup/projectname` | `http(s)://groupname.example.io/subgroup/projectname`| + +CAUTION: **Warning:** +There are some known [limitations](introduction.md#limitations) +regarding namespaces served under the general domain name and HTTPS. +Make sure to read that section. -### Practical examples +To understand Pages domains clearly, read the examples below. -#### Project Websites +### Project website examples - You created a project called `blog` under your username `john`, therefore your project URL is `https://gitlab.com/john/blog/`. @@ -92,7 +73,7 @@ Learn more about [namespaces](../../group/index.md#namespaces). GitLab Pages for this project, the site will live under `https://engineering.gitlab.io/docs/workflows`. -#### User and Group Websites +### User and Group website examples - Under your username, `john`, you created a project called `john.gitlab.io`. Your project URL will be `https://gitlab.com/john/john.gitlab.io`. @@ -103,8 +84,6 @@ Learn more about [namespaces](../../group/index.md#namespaces). Once you enable GitLab Pages for your project, your website will be published under `https://websites.gitlab.io`. -> Support for subgroup project's websites was [introduced](https://gitlab.com/gitlab-org/gitlab-ce/issues/30548) in GitLab 11.8. - **General example:** - On GitLab.com, a project site will always be available under @@ -115,3 +94,10 @@ Learn more about [namespaces](../../group/index.md#namespaces). Pages server domain. Ask your sysadmin for this information. _Read on about [Projects for GitLab Pages and URL structure](getting_started_part_two.md)._ + +### Further reading + +- Read through this technical overview on [Static versus Dynamic Websites](https://about.gitlab.com/2016/06/03/ssg-overview-gitlab-pages-part-1-dynamic-x-static/) +- Understand [how modern Static Site Generators work](https://about.gitlab.com/2016/06/10/ssg-overview-gitlab-pages-part-2/) and what you can add to your static site +- You can use [any SSG with GitLab Pages](https://about.gitlab.com/2016/06/17/ssg-overview-gitlab-pages-part-3-examples-ci/) +- Fork an [example project](https://gitlab.com/pages) to build your website based upon
\ No newline at end of file diff --git a/doc/user/project/pages/getting_started_part_three.md b/doc/user/project/pages/getting_started_part_three.md index daae2f4b5a3..d585c19fc5c 100644 --- a/doc/user/project/pages/getting_started_part_three.md +++ b/doc/user/project/pages/getting_started_part_three.md @@ -1,10 +1,6 @@ --- -last_updated: 2018-11-19 -author: Marcia Ramos -author_gitlab: marcia -level: beginner -article_type: user guide -date: 2017-02-22 +last_updated: 2019-06-04 +type: concepts, reference, howto --- # GitLab Pages custom domains and SSL/TLS Certificates @@ -20,13 +16,13 @@ you should consider first, which we'll cover in this guide: 1. Either if you're adding a **root domain** or a **subdomain**, for which you'll need to set up [DNS records](#dns-records) -1. Whether you want to add an [SSL/TLS certificate](#ssl-tls-certificates) or not +1. Whether you want to add an [SSL/TLS certificate](#ssltls-certificates) or not To finish the association, you need to [add your domain to your project's Pages settings](#add-your-custom-domain-to-gitlab-pages-settings). Let's start from the beginning with [DNS records](#dns-records). If you already know how they work and want to skip the introduction to DNS, -you may be interested in skipping it until the [TL;DR](#tl-dr) section below. +you may be interested in skipping it until the [TL;DR](#tldr) section below. ### DNS Records @@ -96,7 +92,7 @@ you need to log into your domain's admin control panel and add a DNS `CNAME` record pointing your subdomain to your website URL (`namespace.gitlab.io`) address. -Notice that, despite it's a user or project website, the `CNAME` +Note that, whether it's a user or a project website, the `CNAME` should point to your Pages domain (`namespace.gitlab.io`), without any `/project-name`. @@ -115,6 +111,8 @@ If using a [DNS A record](#dns-a-record), you can place the TXT record directly under the domain. If using a [DNS CNAME record](#dns-cname-record), the two record types won't co-exist, so you need to place the TXT record in a special subdomain of its own. +If the domain cannot be verified for 7 days, it will be removed from the GitLab project. + #### TL;DR For root domains (`domain.com`), set a DNS `A` record and verify your @@ -148,8 +146,8 @@ verify your domain's ownership with a TXT record: Once you've set the DNS record, you'll need navigate to your project's **Setting > Pages** and click **+ New domain** to add your custom domain to -GitLab Pages. You can choose whether to add an [SSL/TLS certificate](#ssl-tls-certificates) -to make your website accessible under HTTPS or leave it blank. If don't add a certificate, +GitLab Pages. You can choose whether to add an [SSL/TLS certificate](#ssltls-certificates) +to make your website accessible under HTTPS or leave it blank. If you don't add a certificate, your site will be accessible only via HTTP:  @@ -175,9 +173,6 @@ Note that [DNS propagation may take some time (up to 24h)](http://www.inmotionho although it's usually a matter of minutes to complete. Until it does, verification will fail and attempts to visit your domain will respond with a 404. -Read through the [general documentation on GitLab Pages](introduction.md#add-a-custom-domain-to-your-pages-website) to learn more about adding -custom domains to GitLab Pages sites. - ### Redirecting `www.domain.com` to `domain.com` with Cloudflare If you use Cloudflare, you can redirect `www` to `domain.com` without the need of adding both diff --git a/doc/user/project/pages/getting_started_part_two.md b/doc/user/project/pages/getting_started_part_two.md index 644a1c951d3..3e50cd4887c 100644 --- a/doc/user/project/pages/getting_started_part_two.md +++ b/doc/user/project/pages/getting_started_part_two.md @@ -1,10 +1,6 @@ --- -last_updated: 2018-02-16 -author: Marcia Ramos -author_gitlab: marcia -level: beginner -article_type: user guide -date: 2017-02-22 +last_updated: 2019-06-04 +type: reference, howto --- # Projects for GitLab Pages and URL structure @@ -13,16 +9,16 @@ date: 2017-02-22 To get started with GitLab Pages, you need: -1. A project -1. A configuration file (`.gitlab-ci.yml`) to deploy your site +1. A project, thus a repository to hold your website's codebase. +1. A configuration file (`.gitlab-ci.yml`) to deploy your site. 1. A specific `job` called `pages` in the configuration file - that will make GitLab aware that you are deploying a GitLab Pages website -1. A `public` directory with the content of the website + that will make GitLab aware that you are deploying a GitLab Pages website. +1. A `public` directory with the static content of the website. Optional Features: -1. A custom domain or subdomain -1. A DNS pointing your (sub)domain to your Pages site +1. A custom domain or subdomain. +1. A DNS pointing your (sub)domain to your Pages site. 1. **Optional**: an SSL/TLS certificate so your custom domain is accessible under HTTPS. @@ -33,60 +29,80 @@ The optional settings, custom domain, DNS records, and SSL/TLS certificates, are Your GitLab Pages project is a regular project created the same way you do for the other ones. To get started with GitLab Pages, you have three ways: -- Use one of the popular templates already in the app, -- Fork one of the templates from Page Examples, or -- Create a new project from scratch - -Let's go over each option. +- [Use one of the popular project templates bundled with GitLab](#use-one-of-the-popular-pages-templates-bundled-with-gitlab). +- [Fork one of the templates from Page Examples](#fork-a-project-to-get-started-from). +- [Create a new project from scratch](#create-a-project-from-scratch). ### Use one of the popular Pages templates bundled with GitLab > [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/issues/47857) in GitLab 11.8. -The simplest way to create a GitLab Pages site is to use one of the most -popular templates, which come already bundled and ready to go. To use one -of these templates: - -1. From the top navigation, click the **+** button and select **New project** -1. Select **Create from Template** -1. Choose one of the templates starting with **Pages** +The simplest way to create a GitLab Pages site is to +[use one of the most popular templates](index.md#getting-started), +which come already bundled with GitLab and are ready to go. ### Fork a project to get started from -To make things easy for you, we've created this -[group](https://gitlab.com/pages) of default projects -containing the most popular SSGs templates. - -Watch the [video tutorial](https://youtu.be/TWqh9MtT4Bg) we've -created for the steps below. - -1. [Fork a sample project](../../../gitlab-basics/fork-project.md) from the [Pages group](https://gitlab.com/pages) -1. Trigger a build (push a change to any file) -1. As soon as the build passes, your website will have been deployed with GitLab Pages. Your website URL will be available under your project's **Settings** > **Pages** -1. Optionally, remove the fork relationship by navigating to your project's **Settings** > expanding **Advanced settings** and scrolling down to **Remove fork relationship**: +If you don't find an existing project template that suits you, +we've created this [group](https://gitlab.com/pages) of default projects +containing the most popular SSGs templates to get you started. + +<table class="borderless-table center fixed-table middle width-80"> + <tr> + <td style="width: 30%"><img src="img/icons/fork.png" alt="Fork" class="image-noshadow half-width"></td> + <td style="width: 10%"> + <strong> + <i class="fa fa-angle-double-right" aria-hidden="true"></i> + </strong> + </td> + <td style="width: 30%"><img src="img/icons/terminal.png" alt="Deploy" class="image-noshadow half-width"></td> + <td style="width: 10%"> + <strong> + <i class="fa fa-angle-double-right" aria-hidden="true"></i> + </strong> + </td> + <td style="width: 30%"><img src="img/icons/click.png" alt="Visit" class="image-noshadow half-width"></td> + </tr> + <tr> + <td><em>Fork an example project</em></td> + <td></td> + <td><em>Deploy your website</em></td> + <td></td> + <td><em>Visit your website's URL</em></td> + </tr> +</table> + +**<i class="fa fa-youtube-play youtube" aria-hidden="true"></i> Watch a [video tutorial](https://www.youtube.com/watch?v=TWqh9MtT4Bg) with all the steps below.** + +1. [Fork](../../../gitlab-basics/fork-project.md) a sample project from the [GitLab Pages examples](https://gitlab.com/pages) group. +1. From the left sidebar, navigate to your project's **CI/CD > Pipelines** +and click **Run pipeline** to trigger GitLab CI/CD to build and deploy your +site to the server. +1. Once the pipeline has finished successfully, find the link to visit your +website from your project's **Settings > Pages**. + +You can also take some **optional** further steps: + +- _Remove the fork relationship._ The fork relationship is necessary to contribute back to the project you originally forked from. If you don't have any intentions to do so, you can remove it. To do so, navigate to your project's **Settings**, expand **Advanced settings**, and scroll down to **Remove fork relationship**:  -To turn a **project website** forked from the Pages group into a **user/group** website, you'll need to: - -- Rename it to `namespace.gitlab.io`: navigate to project's **Settings** > expand **Advanced settings** > and scroll down to **Rename repository** -- Adjust your SSG's [base URL](#urls-and-baseurls) from `"project-name"` to `""`. This setting will be at a different place for each SSG, as each of them have their own structure and file tree. Most likely, it will be in the SSG's config file. - -> **Notes:** -> -> Why do I need to remove the fork relationship? -> -> Unless you want to contribute to the original project, -you won't need it connected to the upstream. A -[fork](https://about.gitlab.com/2016/12/01/how-to-keep-your-fork-up-to-date-with-its-origin/#fork) -is useful for submitting merge requests to the upstream. +- _Make it a user or group website._ To turn a **project website** forked +from the Pages group into a **user/group** website, you'll need to: + - Rename it to `namespace.gitlab.io`: go to your project's + **Settings > General** and expand **Advanced**. Scroll down to + **Rename repository** and change the path to `namespace.gitlab.io`. + - Adjust your SSG's [base URL](#urls-and-baseurls) from `"project-name"` to + `""`. This setting will be at a different place for each SSG, as each of them + have their own structure and file tree. Most likely, it will be in the SSG's + config file. ### Create a project from scratch 1. From your **Project**'s **[Dashboard](https://gitlab.com/dashboard/projects)**, - click **New project**, and name it considering the - [practical examples](getting_started_part_one.md#practical-examples). + click **New project**, and name it according to the + [Pages domain names](getting_started_part_one.md#gitlab-pages-domain-names). 1. Clone it to your local computer, add your website files to your project, add, commit and push to GitLab. 1. From the your **Project**'s page, click **Set up CI/CD**: @@ -120,7 +136,7 @@ where you'll find its default URL. repository to you local computer and moving your site files into it, you can run `git init` in your local website directory, add the remote URL: `git remote add origin git@gitlab.com:namespace/project-name.git`, - then add, commit, and push. + then add, commit, and push to GitLab. ## URLs and Baseurls @@ -153,4 +169,4 @@ baseurl: "" ## Custom Domains GitLab Pages supports custom domains and subdomains, served under HTTP or HTTPS. -Please check the [next part](getting_started_part_three.md) of this series for an overview. +See [GitLab Pages custom domains and SSL/TLS Certificates](getting_started_part_three.md) for more information. diff --git a/doc/user/project/pages/img/icons/click.png b/doc/user/project/pages/img/icons/click.png Binary files differindex a534ae29e0f..62a997a7591 100644 --- a/doc/user/project/pages/img/icons/click.png +++ b/doc/user/project/pages/img/icons/click.png diff --git a/doc/user/project/pages/img/icons/fork.png b/doc/user/project/pages/img/icons/fork.png Binary files differindex 8a3aa46eb37..1ae8fa722b7 100644 --- a/doc/user/project/pages/img/icons/fork.png +++ b/doc/user/project/pages/img/icons/fork.png diff --git a/doc/user/project/pages/img/icons/free.png b/doc/user/project/pages/img/icons/free.png Binary files differindex ae455033e94..c74cd90fa1a 100644 --- a/doc/user/project/pages/img/icons/free.png +++ b/doc/user/project/pages/img/icons/free.png diff --git a/doc/user/project/pages/img/icons/lock.png b/doc/user/project/pages/img/icons/lock.png Binary files differindex f4c35c84112..f7f32fded45 100644 --- a/doc/user/project/pages/img/icons/lock.png +++ b/doc/user/project/pages/img/icons/lock.png diff --git a/doc/user/project/pages/img/icons/monitor.png b/doc/user/project/pages/img/icons/monitor.png Binary files differindex 8bad059a74c..ce27b7e177e 100644 --- a/doc/user/project/pages/img/icons/monitor.png +++ b/doc/user/project/pages/img/icons/monitor.png diff --git a/doc/user/project/pages/img/icons/terminal.png b/doc/user/project/pages/img/icons/terminal.png Binary files differindex 377eeb4edc6..5a711f05c4e 100644 --- a/doc/user/project/pages/img/icons/terminal.png +++ b/doc/user/project/pages/img/icons/terminal.png diff --git a/doc/user/project/pages/img/pages_create_project.png b/doc/user/project/pages/img/pages_create_project.png Binary files differdeleted file mode 100644 index 69e84b84984..00000000000 --- a/doc/user/project/pages/img/pages_create_project.png +++ /dev/null diff --git a/doc/user/project/pages/img/pages_create_user_page.png b/doc/user/project/pages/img/pages_create_user_page.png Binary files differdeleted file mode 100644 index 2f1a19ae424..00000000000 --- a/doc/user/project/pages/img/pages_create_user_page.png +++ /dev/null diff --git a/doc/user/project/pages/img/pages_dns_details.png b/doc/user/project/pages/img/pages_dns_details.png Binary files differdeleted file mode 100644 index 3e57f43f7ba..00000000000 --- a/doc/user/project/pages/img/pages_dns_details.png +++ /dev/null diff --git a/doc/user/project/pages/img/pages_multiple_domains.png b/doc/user/project/pages/img/pages_multiple_domains.png Binary files differdeleted file mode 100644 index 76c39101439..00000000000 --- a/doc/user/project/pages/img/pages_multiple_domains.png +++ /dev/null diff --git a/doc/user/project/pages/img/pages_new_domain_button.png b/doc/user/project/pages/img/pages_new_domain_button.png Binary files differdeleted file mode 100644 index cd59defa006..00000000000 --- a/doc/user/project/pages/img/pages_new_domain_button.png +++ /dev/null diff --git a/doc/user/project/pages/img/pages_project_templates_11-8.png b/doc/user/project/pages/img/pages_project_templates_11-8.png Binary files differnew file mode 100644 index 00000000000..a645d28260b --- /dev/null +++ b/doc/user/project/pages/img/pages_project_templates_11-8.png diff --git a/doc/user/project/pages/img/pages_remove.png b/doc/user/project/pages/img/pages_remove.png Binary files differdeleted file mode 100644 index 10299880247..00000000000 --- a/doc/user/project/pages/img/pages_remove.png +++ /dev/null diff --git a/doc/user/project/pages/img/pages_upload_cert.png b/doc/user/project/pages/img/pages_upload_cert.png Binary files differdeleted file mode 100644 index 64e5f8eced1..00000000000 --- a/doc/user/project/pages/img/pages_upload_cert.png +++ /dev/null diff --git a/doc/user/project/pages/img/remove_pages.png b/doc/user/project/pages/img/remove_pages.png Binary files differnew file mode 100644 index 00000000000..60f76f15f93 --- /dev/null +++ b/doc/user/project/pages/img/remove_pages.png diff --git a/doc/user/project/pages/index.md b/doc/user/project/pages/index.md index e0b78753e21..04bda212128 100644 --- a/doc/user/project/pages/index.md +++ b/doc/user/project/pages/index.md @@ -1,15 +1,24 @@ --- description: 'Learn how to use GitLab Pages to deploy a static website at no additional cost.' +last_updated: 2019-06-04 +type: index, reference --- # GitLab Pages +> - [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/80) in GitLab Enterprise Edition 8.3. +> - Custom CNAMEs with TLS support were [introduced](https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/173) in GitLab Enterprise Edition 8.5. +> - [Ported](https://gitlab.com/gitlab-org/gitlab-ce/issues/14605) to GitLab Community Edition in GitLab 8.17. +> - Support for subgroup project's websites was [introduced](https://gitlab.com/gitlab-org/gitlab-ce/issues/30548) in GitLab 11.8. +> - Bundled project templates were [introduced](https://gitlab.com/gitlab-org/gitlab-ce/issues/47857) in GitLab 11.8. + + **GitLab Pages is a feature that allows you to publish static websites directly from a repository in GitLab.** You can use it either for personal or business websites, such as -portfolios, documentation, manifestos, and business presentations, -and attribute any license to your content. +portfolios, documentation, manifestos, and business presentations. +You can also attribute any license to your content. <table class="borderless-table center fixed-table"> <tr> @@ -61,14 +70,6 @@ publish any website written directly in plain HTML, CSS, and JavaScript.</p> <div class="col-md-3"><img src="img/ssgs_pages.png" alt="Examples of SSGs supported by Pages" class="image-noshadow middle display-block"></div> </div> -### Availability - -If you're using GitLab.com, your website will be publicly available to the internet. -If you're using self-managed instances (Core, Starter, Premium, or Ultimate), -your websites will be published on your own server, according to the -[Pages admin settings](../../../administration/pages/index.md) chosen by your sysadmin, -who can opt for making them public or internal to your server. - ### How it works To use GitLab Pages, first you need to create a project in GitLab to upload your website's @@ -78,99 +79,77 @@ repository. Note that when you create a new project in GitLab, a [repository](.. becomes available automatically. To deploy your site, GitLab will use its built-in tool called [GitLab CI/CD](../../../ci/README.md), -that will build your site and publish it to the GitLab Pages server. The sequence of +to build your site and publish it to the GitLab Pages server. The sequence of scripts that GitLab CI/CD runs to accomplish this task is created from a file named `.gitlab-ci.yml`, which you can [create and modify](getting_started_part_four.md) at will. -You can either use GitLab's [default domain for GitLab Pages websites](getting_started_part_one.md#gitlab-pages-domain), +You can either use GitLab's [default domain for GitLab Pages websites](getting_started_part_one.md#gitlab-pages-domain-names), `*.gitlab.io`, or your own domain (`example.com`). In that case, you'll need admin access to your domain's registrar (or control panel) to set it up with Pages. Optionally, when adding your own domain, you can add an SSL/TLS certificate to secure your site under the HTTPS protocol. -## Getting started +### Getting started -To get started with GitLab Pages, you can either [create a project from scratch](getting_started_part_two.md#create-a-project-from-scratch), -use a [bundled template](getting_started_part_two.md#use-one-of-the-popular-pages-templates-bundled-with-gitlab), or copy any of our existing example projects: +To get started with GitLab Pages, you can either: -1. Choose an [example project](https://gitlab.com/pages) to [fork](../../../gitlab-basics/fork-project.md#how-to-fork-a-project): - by forking a project, you create a copy of the codebase you're forking from to start from a template instead of starting from scratch. -1. From the left sidebar, navigate to your project's **CI/CD > Pipelines** and click - **Run pipeline** so that GitLab CI/CD will build and deploy your site to the server. -1. Once the pipeline has finished successfully, find the link to visit your website from your - project's **Settings > Pages**. - -<table class="borderless-table center fixed-table middle width-80"> - <tr> - <td style="width: 30%"><img src="img/icons/fork.png" alt="Fork" class="image-noshadow half-width"></td> - <td style="width: 10%"> - <strong> - <i class="fa fa-angle-double-right" aria-hidden="true"></i> - </strong> - </td> - <td style="width: 30%"><img src="img/icons/terminal.png" alt="Deploy" class="image-noshadow half-width"></td> - <td style="width: 10%"> - <strong> - <i class="fa fa-angle-double-right" aria-hidden="true"></i> - </strong> - </td> - <td style="width: 30%"><img src="img/icons/click.png" alt="Visit" class="image-noshadow half-width"></td> - </tr> - <tr> - <td><em>Fork an example project</em></td> - <td></td> - <td><em>Deploy your website</em></td> - <td></td> - <td><em>Visit your website's URL</em></td> - </tr> -</table> +- [Create a project from scratch](getting_started_part_two.md#create-a-project-from-scratch). +- [Copy an existing example project](getting_started_part_two.md#fork-a-project-to-get-started-from). +- Use a bundled project template ready to go: -Your website is then visible on your domain, and you can modify your files -as you wish. For every modification pushed to your repository, GitLab CI/CD will run -a new pipeline to publish your changes to the server. +1. From the top navigation, click the **+** button and select **New project**. +1. Select **Create from Template**. +1. Choose one of the templates starting with **Pages**: -You can also take some optional further steps: +  -- Remove the [fork relationship](getting_started_part_two.md#fork-a-project-to-get-started-from) - (_You don't need the relationship unless you intent to contribute back to the example project you forked from_). -- Make it a [user/group website](getting_started_part_one.md#user-and-group-websites) +1. From the left sidebar, navigate to your project's **CI/CD > Pipelines** +and click **Run pipeline** to trigger GitLab CI/CD to build and deploy your +site to the server. +1. Once the pipeline has finished successfully, find the link to visit your +website from your project's **Settings > Pages**. -**<i class="fa fa-youtube-play youtube" aria-hidden="true"></i> Watch a [video tutorial](https://www.youtube.com/watch?v=TWqh9MtT4Bg) with all the steps above!** +Your website is then visible on your domain, and you can modify yourfiles +as you wish. For every modification pushed to your repository, GitLab CI/CD +will run a new pipeline to publish your changes to the server. _Advanced options:_ - [Use a custom domain](getting_started_part_three.md#adding-your-custom-domain-to-gitlab-pages) -- Apply [SSL/TLS certification](getting_started_part_three.md#ssl-tls-certificates) to your custom domain - -## Explore GitLab Pages +- Apply [SSL/TLS certification](getting_started_part_three.md#ssltls-certificates) to your custom domain -To learn more about GitLab Pages, read the following tutorials: +## Availability -- [Static websites and GitLab Pages domains](getting_started_part_one.md): Understand what is a static website, and how GitLab Pages default domains work -- [Projects for GitLab Pages and URL structure](getting_started_part_two.md): Forking projects and creating new ones from scratch, understanding URLs structure and baseurls -- [GitLab Pages custom domains and SSL/TLS Certificates](getting_started_part_three.md): How to add custom domains and subdomains to your website, configure DNS records and SSL/TLS certificates -- [Creating and Tweaking GitLab CI/CD for GitLab Pages](getting_started_part_four.md): Understand how to create your own `.gitlab-ci.yml` for your site -- [Technical aspects, custom 404 pages, limitations](introduction.md) - -### GitLab Pages with Static Site Generators (SSGs) - -To understand more about SSGs, their advantages, and how to get the most from them -with Pages, read through this series: - -- [SSGs part 1: Static vs dynamic websites](https://about.gitlab.com/2016/06/03/ssg-overview-gitlab-pages-part-1-dynamic-x-static/) -- [SSGs part 2: Modern static site generators](https://about.gitlab.com/2016/06/10/ssg-overview-gitlab-pages-part-2/) -- [SSGs part 3: Build any SSG site with GitLab Pages](https://about.gitlab.com/2016/06/17/ssg-overview-gitlab-pages-part-3-examples-ci/) +If you're using GitLab.com, your website will be publicly available to the internet. +If you're using self-managed instances (Core, Starter, Premium, or Ultimate), +your websites will be published on your own server, according to the +[Pages admin settings](../../../administration/pages/index.md) chosen by your sysadmin, +who can opt for making them public or internal to your server. -### GitLab Pages with SSL/TLS certificates +Note that, if you're using GitLab Pages default domain (`.gitlab.io`), +your website will be automatically secure and available under +HTTPS. If you're using your own custom domain, you can +optionally secure it with SSL/TLS certificates. -If you're using GitLab Pages default domain (`.gitlab.io`), your website will be -automatically secure and available under HTTPS. If you're using your own domain, you can -optionally secure it with SSL/TLS certificates. You can read the following -tutorials to learn how to use these third-party certificates with GitLab Pages: +## Explore GitLab Pages -- [CloudFlare](https://about.gitlab.com/2017/02/07/setting-up-gitlab-pages-with-cloudflare-certificates/) -- [Let's Encrypt](lets_encrypt_for_gitlab_pages.md) +To learn more about configuration options for GitLab Pages, read the following: + +| Document | Description | +| --- | --- | +| [Static websites and Pages domains](getting_started_part_one.md) | Understand what is a static website, and how GitLab Pages default domains work. | +| [Projects and URL structure](getting_started_part_two.md) | Forking projects and creating new ones from scratch, understanding URLs structure and baseurls. | +| [GitLab CI/CD for GitLab Pages](getting_started_part_four.md) | Understand how to create your own `.gitlab-ci.yml` for your site. | +| [Exploring GitLab Pages](introduction.md) | Requirements, technical aspects, specific GitLab CI's configuration options, custom 404 pages, limitations, FAQ. | +|---+---| +| [Custom domains and SSL/TLS Certificates](getting_started_part_three.md) | How to add custom domains and subdomains to your website, configure DNS records and SSL/TLS certificates. | +| [CloudFlare certificates](https://about.gitlab.com/2017/02/07/setting-up-gitlab-pages-with-cloudflare-certificates/) | Secure your Pages site with CloudFlare certificates. | +| [Let's Encrypt certificates](lets_encrypt_for_gitlab_pages.md) | Secure your Pages site with Let's Encrypt certificates. | +|---+---| +| [Static vs dynamic websites](https://about.gitlab.com/2016/06/03/ssg-overview-gitlab-pages-part-1-dynamic-x-static/) | A conceptual overview on static versus dynamic sites. | +| [Modern static site generators](https://about.gitlab.com/2016/06/10/ssg-overview-gitlab-pages-part-2/) | A conceptual overview on SSGs. | +| [Build any SSG site with GitLab Pages](https://about.gitlab.com/2016/06/17/ssg-overview-gitlab-pages-part-3-examples-ci/) | An overview on using SSGs for GitLab Pages. | ## Advanced use @@ -178,13 +157,13 @@ There are quite some great examples of GitLab Pages websites built for some specific reasons. These examples can teach you some advanced techniques to use and adapt to your own needs: -- [Posting to your GitLab Pages blog from iOS](https://about.gitlab.com/2016/08/19/posting-to-your-gitlab-pages-blog-from-ios/) -- [GitLab CI: Run jobs sequentially, in parallel, or build a custom pipeline](https://about.gitlab.com/2016/07/29/the-basics-of-gitlab-ci/) -- [GitLab CI: Deployment & environments](https://about.gitlab.com/2016/08/26/ci-deployment-and-environments/) -- [Building a new GitLab docs site with Nanoc, GitLab CI, and GitLab Pages](https://about.gitlab.com/2016/12/07/building-a-new-gitlab-docs-site-with-nanoc-gitlab-ci-and-gitlab-pages/) -- [Publish code coverage reports with GitLab Pages](https://about.gitlab.com/2016/11/03/publish-code-coverage-report-with-gitlab-pages/) +- [Posting to your GitLab Pages blog from iOS](https://about.gitlab.com/2016/08/19/posting-to-your-gitlab-pages-blog-from-ios/). +- [GitLab CI: Run jobs sequentially, in parallel, or build a custom pipeline](https://about.gitlab.com/2016/07/29/the-basics-of-gitlab-ci/). +- [GitLab CI: Deployment & environments](https://about.gitlab.com/2016/08/26/ci-deployment-and-environments/). +- [Building a new GitLab docs site with Nanoc, GitLab CI, and GitLab Pages](https://about.gitlab.com/2016/12/07/building-a-new-gitlab-docs-site-with-nanoc-gitlab-ci-and-gitlab-pages/). +- [Publish code coverage reports with GitLab Pages](https://about.gitlab.com/2016/11/03/publish-code-coverage-report-with-gitlab-pages/). -## Admin GitLab Pages for CE and EE +## Admin GitLab Pages for self-managed instances Enable and configure GitLab Pages on your own instance (GitLab Community Edition and Enterprise Editions) with the [admin guide](../../../administration/pages/index.md). diff --git a/doc/user/project/pages/introduction.md b/doc/user/project/pages/introduction.md index 23eb88fd305..4fab7f79e0c 100644 --- a/doc/user/project/pages/introduction.md +++ b/doc/user/project/pages/introduction.md @@ -1,177 +1,142 @@ -# Exploring GitLab Pages - -> **Notes:** -> - This feature was [introduced][ee-80] in GitLab EE 8.3. -> - Custom CNAMEs with TLS support were [introduced][ee-173] in GitLab EE 8.5. -> - GitLab Pages [was ported][ce-14605] to Community Edition in GitLab 8.17. -> - This document is about the user guide. To learn how to enable GitLab Pages -> across your GitLab instance, visit the [administrator documentation](../../../administration/pages/index.md). - -With GitLab Pages you can host for free your static websites on GitLab. -Combined with the power of [GitLab CI] and the help of [GitLab Runner] you can -deploy static pages for your individual projects, your user or your group. - -Read [GitLab Pages on GitLab.com](#gitlab-pages-on-gitlab-com) for specific -information, if you are using GitLab.com to host your website. +--- +type: reference +last_updated: 2018-06-04 +--- -## Getting started with GitLab Pages domains +# Exploring GitLab Pages -> **Note:** -> In the rest of this document we will assume that the general domain name that -> is used for GitLab Pages is `example.io`. +This document is a user guide to explore the options and settings +GitLab Pages offers. -In general there are two types of pages one might create: +To familiarize yourself with GitLab Pages first: -- Pages per user (`username.example.io`) or per group (`groupname.example.io`) -- Pages per project (`username.example.io/projectname` or `groupname.example.io/projectname`) +- Read an [introduction to GitLab Pages](index.md#overview). +- Learn [how to get started with Pages](index.md#getting-started). +- Learn how to enable GitLab Pages +across your GitLab instance on the [administrator documentation](../../../administration/pages/index.md). -In GitLab, usernames and groupnames are unique and we often refer to them -as [namespaces](../../group/index.md#namespaces). There can be only one namespace -in a GitLab instance. Below you -can see the connection between the type of GitLab Pages, what the project name -that is created on GitLab looks like and the website URL it will be ultimately -be served on. +## GitLab Pages requirements -| Type of GitLab Pages | The name of the project created in GitLab | Website URL | -| -------------------- | ------------ | ----------- | -| User pages | `username.example.io` | `http(s)://username.example.io` | -| Group pages | `groupname.example.io` | `http(s)://groupname.example.io` | -| Project pages owned by a user | `projectname` | `http(s)://username.example.io/projectname` | -| Project pages owned by a group | `projectname` | `http(s)://groupname.example.io/projectname`| -| Project pages owned by a subgroup | `subgroup/projectname` | `http(s)://groupname.example.io/subgroup/projectname`| +In brief, this is what you need to upload your website in GitLab Pages: -> **Warning:** -> There are some known [limitations](#limitations) regarding namespaces served -> under the general domain name and HTTPS. Make sure to read that section. +1. Domain of the instance: domain name that is used for GitLab Pages +(ask your administrator). +1. GitLab CI/CD: a `.gitlab-ci.yml` file with a specific job named [`pages`][pages] in the root directory of your repository. +1. A directory called `public` in your site's repo containing the content +to be published. +1. GitLab Runner enabled for the project. -### GitLab Pages requirements +## GitLab Pages on GitLab.com -In brief, this is what you need to upload your website in GitLab Pages: +If you are using [GitLab Pages on GitLab.com](#gitlab-pages-on-gitlabcom) to host your website, then: -1. Find out the general domain name that is used for GitLab Pages - (ask your administrator). This is very important, so you should first make - sure you get that right. -1. Create a project -1. Push a [`.gitlab-ci.yml` file][yaml] in the root directory - of your repository with a specific job named [`pages`][pages] -1. Set up a GitLab Runner to build your website +- The domain name for GitLab Pages on GitLab.com is `gitlab.io`. +- Custom domains and TLS support are enabled. +- Shared runners are enabled by default, provided for free and can be used to + build your website. If you want you can still bring your own Runner. -> **Note:** -If [shared runners](../../../ci/runners/README.md) are enabled by your GitLab -administrator, you should be able to use them instead of bringing your own. +## Example projects -### User or group Pages +Visit the [GitLab Pages group](https://gitlab.com/groups/pages) for a complete list of example projects. Contributions are very welcome. -For user and group pages, the name of the project should be specific to the -username or groupname and the general domain name that is used for GitLab Pages. -Head over your GitLab instance that supports GitLab Pages and create a -repository named `username.example.io`, where `username` is your username on -GitLab. If the first part of the project name doesn't match exactly your -username, it won’t work, so make sure to get it right. +## Custom error codes Pages -To create a group page, the steps are the same like when creating a website for -users. Just make sure that you are creating the project within the group's -namespace. +You can provide your own 403 and 404 error pages by creating the `403.html` and +`404.html` files respectively in the root directory of the `public/` directory +that will be included in the artifacts. Usually this is the root directory of +your project, but that may differ depending on your static generator +configuration. - +If the case of `404.html`, there are different scenarios. For example: ---- +- If you use project Pages (served under `/projectname/`) and try to access + `/projectname/non/existing_file`, GitLab Pages will try to serve first + `/projectname/404.html`, and then `/404.html`. +- If you use user/group Pages (served under `/`) and try to access + `/non/existing_file` GitLab Pages will try to serve `/404.html`. +- If you use a custom domain and try to access `/non/existing_file`, GitLab + Pages will try to serve only `/404.html`. -After you push some static content to your repository and GitLab Runner uploads -the artifacts to GitLab CI, you will be able to access your website under -`http(s)://username.example.io`. Keep reading to find out how. +## Redirects in GitLab Pages ->**Note:** -If your username/groupname contains a dot, for example `foo.bar`, you will not -be able to use the wildcard domain HTTPS, read more at [limitations](#limitations). +Since you cannot use any custom server configuration files, like `.htaccess` or +any `.conf` file, if you want to redirect a page to another +location, you can use the [HTTP meta refresh tag][metarefresh]. -### Project Pages +Some static site generators provide plugins for that functionality so that you +don't have to create and edit HTML files manually. For example, Jekyll has the +[redirect-from plugin](https://github.com/jekyll/jekyll-redirect-from). -GitLab Pages for projects can be created by both user and group accounts. -The steps to create a project page for a user or a group are identical: +## GitLab Pages Access Control **[CORE ONLY]** -1. Create a new project -1. Push a [`.gitlab-ci.yml` file][yaml] in the root directory - of your repository with a specific job named [`pages`][pages]. -1. Set up a GitLab Runner to build your website +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/issues/33422) in GitLab 11.5. -A user's project will be served under `http(s)://username.example.io/projectname` -whereas a group's project under `http(s)://groupname.example.io/projectname`. +NOTE: **Note:** +GitLab Pages access control is not activated on GitLab.com. You can check its +progress on the +[infrastructure issue tracker](https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/5576). -For practical examples for group and project Pages, read through the guide -[GitLab Pages from A to Z: Part 1 - Static sites and GitLab Pages domains](getting_started_part_one.md#practical-examples). +You can enable Pages access control on your project, so that only +[members of your project](../../permissions.md#project-members-permissions) +(at least Guest) can access your website: -## Quick Start +1. Navigate to your project's **Settings > General > Permissions**. +1. Toggle the **Pages** button to enable the access control. -Read through [GitLab Pages Quick Start Guide][pages-quick] or watch the video tutorial on -[how to publish a website with GitLab Pages on GitLab.com from a forked project][video-pages-fork]. + NOTE: **Note:** + If you don't see the toggle button, that means that it's not enabled. + Ask your administrator to [enable it](../../../administration/pages/index.md#access-control). -See also [All you Need to Know About GitLab Pages][pages-index-guide] for a list with all the resources we have for GitLab Pages. +1. The Pages access control dropdown allows you to set who can view pages hosted + with GitLab Pages, depending on your project's visibility: -### Explore the contents of `.gitlab-ci.yml` + - If your project is private: + - **Only project members**: Only project members will be able to browse the website. + - **Everyone**: Everyone, both logged into and logged out of GitLab, will be able to browse the website, no matter their project membership. + - If your project is internal: + - **Only project members**: Only project members will be able to browse the website. + - **Everyone with access**: Everyone logged into GitLab will be able to browse the website, no matter their project membership. + - **Everyone**: Everyone, both logged into and logged out of GitLab, will be able to browse the website, no matter their project membership. + - If your project is public: + - **Only project members**: Only project members will be able to browse the website. + - **Everyone with access**: Everyone, both logged into and logged out of GitLab, will be able to browse the website, no matter their project membership. -The key thing about GitLab Pages is the `.gitlab-ci.yml` file, something that -gives you absolute control over the build process. You can actually watch your -website being built live by following the CI job traces. +1. Click **Save changes**. -For a simplified user guide on setting up GitLab CI/CD for Pages, read through -the article [GitLab Pages from A to Z: Part 4 - Creating and Tweaking `.gitlab-ci.yml` for GitLab Pages](getting_started_part_four.md#creating-and-tweaking-gitlab-ci-yml-for-gitlab-pages) +--- -> **Note:** -> Before reading this section, make sure you familiarize yourself with GitLab CI -> and the specific syntax of[`.gitlab-ci.yml`][yaml] by -> following our [quick start guide]. +The next time someone tries to access your website and the access control is +enabled, they will be presented with a page to sign into GitLab and verify they +can access the website. -To make use of GitLab Pages, the contents of `.gitlab-ci.yml` must follow the -rules below: +## Unpublishing your Pages -1. A special job named [`pages`][pages] must be defined -1. Any static content which will be served by GitLab Pages must be placed under - a `public/` directory -1. `artifacts` with a path to the `public/` directory must be defined +If you ever feel the need to purge your Pages content, you can do so by going +to your project's settings through the gear icon in the top right, and then +navigating to **Pages**. Hit the **Remove pages** button and your Pages website +will be deleted. -In its simplest form, `.gitlab-ci.yml` looks like: + -```yaml -pages: - script: - - my_commands - artifacts: - paths: - - public -``` +## Limitations -When the Runner reaches to build the `pages` job, it executes whatever is -defined in the `script` parameter and if the job completes with a non-zero -exit status, it then uploads the `public/` directory to GitLab Pages. +When using Pages under the general domain of a GitLab instance (`*.example.io`), +you _cannot_ use HTTPS with sub-subdomains. That means that if your +username/groupname contains a dot, for example `foo.bar`, the domain +`https://foo.bar.example.io` will _not_ work. This is a limitation of the +[HTTP Over TLS protocol][rfc]. HTTP pages will continue to work provided you +don't redirect HTTP to HTTPS. -The `public/` directory should contain all the static content of your website. -Depending on how you plan to publish your website, the steps defined in the -[`script` parameter](../../../ci/yaml/README.md#script) may differ. +[rfc]: https://tools.ietf.org/html/rfc2818#section-3.1 "HTTP Over TLS RFC" -Be aware that Pages are by default branch/tag agnostic and their deployment -relies solely on what you specify in `.gitlab-ci.yml`. If you don't limit the -`pages` job with the [`only` parameter](../../../ci/yaml/README.md#only-and-except-simplified), -whenever a new commit is pushed to whatever branch or tag, the Pages will be -overwritten. In the example below, we limit the Pages to be deployed whenever -a commit is pushed only on the `master` branch: +GitLab Pages [does **not** support group websites for subgroups](../../group/subgroups/index.md#limitations). +You can only create the highest-level group website. -```yaml -pages: - script: - - my_commands - artifacts: - paths: - - public - only: - - master -``` +## Specific configuration options for Pages -We then tell the Runner to treat the `public/` directory as `artifacts` and -upload it to GitLab. And since all these parameters were all under a `pages` -job, the contents of the `public` directory will be served by GitLab Pages. +Learn how to set up GitLab CI/CD for specific use cases. -#### How `.gitlab-ci.yml` looks like when the static content is in your repository +### `.gitlab-ci.yml` for plain HTML websites Supposed your repository contained the following files: @@ -200,59 +165,15 @@ pages: - master ``` -#### How `.gitlab-ci.yml` looks like when using a static generator - -In general, GitLab Pages support any kind of [static site generator][staticgen], -since `.gitlab-ci.yml` can be configured to run any possible command. - -In the root directory of your Git repository, place the source files of your -favorite static generator. Then provide a `.gitlab-ci.yml` file which is -specific to your static generator. +### `.gitlab-ci.yml` for a static site generator -The example below, uses [Jekyll] to build the static site: +See this document for a [step-by-step guide](getting_started_part_four.md). -```yaml -image: ruby:2.1 # the script will run in Ruby 2.1 using the Docker image ruby:2.1 - -pages: # the build job must be named pages - script: - - gem install jekyll # we install jekyll - - jekyll build -d public/ # we tell jekyll to build the site for us - artifacts: - paths: - - public # this is where the site will live and the Runner uploads it in GitLab - only: - - master # this script is only affecting the master branch -``` - -Here, we used the Docker executor and in the first line we specified the base -image against which our jobs will run. - -You have to make sure that the generated static files are ultimately placed -under the `public` directory, that's why in the `script` section we run the -`jekyll` command that jobs the website and puts all content in the `public/` -directory. Depending on the static generator of your choice, this command will -differ. Search in the documentation of the static generator you will use if -there is an option to explicitly set the output directory. If there is not -such an option, you can always add one more line under `script` to rename the -resulting directory in `public/`. - -We then tell the Runner to treat the `public/` directory as `artifacts` and -upload it to GitLab. - ---- - -See the [jekyll example project][pages-jekyll] to better understand how this -works. - -For a list of Pages projects, see the [example projects](#example-projects) to -get you started. - -#### How to set up GitLab Pages in a repository where there's also actual code +### `.gitlab-ci.yml` for a repository where there's also actual code Remember that GitLab Pages are by default branch/tag agnostic and their deployment relies solely on what you specify in `.gitlab-ci.yml`. You can limit -the `pages` job with the [`only` parameter](../../../ci/yaml/README.md#only-and-except-simplified), +the `pages` job with the [`only` parameter](../../../ci/yaml/README.md#onlyexcept-basic), whenever a new commit is pushed to a branch that will be used specifically for your pages. @@ -293,28 +214,6 @@ also includes `.gitlab-ci.yml`. [jekyll-master]: https://gitlab.com/pages/jekyll-branched/tree/master [jekyll-pages]: https://gitlab.com/pages/jekyll-branched/tree/pages -## Next steps - -So you have successfully deployed your website, congratulations! Let's check -what more you can do with GitLab Pages. - -### Example projects - -Below is a list of example projects for GitLab Pages with a plain HTML website -or various static site generators. Contributions are very welcome. - -- [Plain HTML](https://gitlab.com/pages/plain-html) -- [Jekyll](https://gitlab.com/pages/jekyll) -- [Hugo](https://gitlab.com/pages/hugo) -- [Middleman](https://gitlab.com/pages/middleman) -- [Hexo](https://gitlab.com/pages/hexo) -- [Brunch](https://gitlab.com/pages/brunch) -- [Metalsmith](https://gitlab.com/pages/metalsmith) -- [Harp](https://gitlab.com/pages/harp) - -Visit the GitLab Pages group for a full list of example projects: -<https://gitlab.com/groups/pages>. - ### Serving compressed assets Most modern browsers support downloading files in a compressed format. This @@ -407,158 +306,6 @@ NOTE: **Note:** When `public/data/index.html` exists, it takes priority over the `public/data.html` file for both the `/data` and `/data/` URL paths. -### Add a custom domain to your Pages website - -For a complete guide on Pages domains, read through the article -[GitLab Pages from A to Z: Part 3 - Setting Up Custom Domains - DNS Records and SSL/TLS Certificates](getting_started_part_three.md#setting-up-custom-domains-dns-records-and-ssl-tls-certificates) - -If this setting is enabled by your GitLab administrator, you should be able to -see the **New Domain** button when visiting your project's settings through the -gear icon in the top right and then navigating to **Pages**. - - - ---- - -You can add multiple domains pointing to your website hosted under GitLab. -Once the domain is added, you can see it listed under the **Domains** section. - - - ---- - -As a last step, you need to configure your DNS and add a CNAME pointing to your -user/group page. Click on the **Details** button of a domain for further -instructions. - - - ---- - ->**Note:** -Currently there is support only for custom domains on per-project basis. That -means that if you add a custom domain (`example.com`) for your user website -(`username.example.io`), a project that is served under `username.example.io/foo`, -will not be accessible under `example.com/foo`. - -### Secure your custom domain website with TLS - -When you add a new custom domain, you also have the chance to add a TLS -certificate. If this setting is enabled by your GitLab administrator, you -should be able to see the option to upload the public certificate and the -private key when adding a new domain. - - - -For a complete guide on Pages domains, read through the article -[GitLab Pages from A to Z: Part 3 - Setting Up Custom Domains - DNS Records and SSL/TLS Certificates](getting_started_part_three.md#setting-up-custom-domains-dns-records-and-ssl-tls-certificates) - -### Custom error codes pages - -You can provide your own 403 and 404 error pages by creating the `403.html` and -`404.html` files respectively in the root directory of the `public/` directory -that will be included in the artifacts. Usually this is the root directory of -your project, but that may differ depending on your static generator -configuration. - -If the case of `404.html`, there are different scenarios. For example: - -- If you use project Pages (served under `/projectname/`) and try to access - `/projectname/non/existing_file`, GitLab Pages will try to serve first - `/projectname/404.html`, and then `/404.html`. -- If you use user/group Pages (served under `/`) and try to access - `/non/existing_file` GitLab Pages will try to serve `/404.html`. -- If you use a custom domain and try to access `/non/existing_file`, GitLab - Pages will try to serve only `/404.html`. - -### Remove the contents of your pages - -If you ever feel the need to purge your Pages content, you can do so by going -to your project's settings through the gear icon in the top right, and then -navigating to **Pages**. Hit the **Remove pages** button and your Pages website -will be deleted. Simple as that. - - - -## GitLab Pages on GitLab.com - -If you are using GitLab.com to host your website, then: - -- The general domain name for GitLab Pages on GitLab.com is `gitlab.io`. -- Custom domains and TLS support are enabled. -- Shared runners are enabled by default, provided for free and can be used to - build your website. If you want you can still bring your own Runner. - -The rest of the guide still applies. - -See also: [GitLab Pages from A to Z: Part 1 - Static sites and GitLab Pages domains](getting_started_part_one.md#gitlab-pages-domain). - -## GitLab Pages access control **[CORE ONLY]** - -> [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/issues/33422) in GitLab 11.5. - -NOTE: **Note:** -GitLab Pages access control is not activated on GitLab.com. You can check its -progress on the -[infrastructure issue tracker](https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/5576). - -You can enable Pages access control on your project, so that only -[members of your project](../../permissions.md#project-members-permissions) -(at least Guest) can access your website: - -1. Navigate to your project's **Settings > General > Permissions**. -1. Toggle the **Pages** button to enable the access control. - - NOTE: **Note:** - If you don't see the toggle button, that means that it's not enabled. - Ask your administrator to [enable it](../../../administration/pages/index.md#access-control). - -1. The Pages access control dropdown allows you to set who can view pages hosted - with GitLab Pages, depending on your project's visibility: - - - If your project is private: - - **Only project members**: Only project members will be able to browse the website. - - **Everyone**: Everyone, both logged into and logged out of GitLab, will be able to browse the website, no matter their project membership. - - If your project is internal: - - **Only project members**: Only project members will be able to browse the website. - - **Everyone with access**: Everyone logged into GitLab will be able to browse the website, no matter their project membership. - - **Everyone**: Everyone, both logged into and logged out of GitLab, will be able to browse the website, no matter their project membership. - - If your project is public: - - **Only project members**: Only project members will be able to browse the website. - - **Everyone with access**: Everyone, both logged into and logged out of GitLab, will be able to browse the website, no matter their project membership. - -1. Click **Save changes**. - ---- - -The next time someone tries to access your website and the access control is -enabled, they will be presented with a page to sign into GitLab and verify they -can access the website. - -## Limitations - -When using Pages under the general domain of a GitLab instance (`*.example.io`), -you _cannot_ use HTTPS with sub-subdomains. That means that if your -username/groupname contains a dot, for example `foo.bar`, the domain -`https://foo.bar.example.io` will _not_ work. This is a limitation of the -[HTTP Over TLS protocol][rfc]. HTTP pages will continue to work provided you -don't redirect HTTP to HTTPS. - -[rfc]: https://tools.ietf.org/html/rfc2818#section-3.1 "HTTP Over TLS RFC" - -GitLab Pages [does **not** support group websites for subgroups](../../group/subgroups/index.md#limitations). -You can only create the highest-level group website. - -## Redirects in GitLab Pages - -Since you cannot use any custom server configuration files, like `.htaccess` or -any `.conf` file, if you want to redirect a page to another -location, you can use the [HTTP meta refresh tag][metarefresh]. - -Some static site generators provide plugins for that functionality so that you -don't have to create and edit HTML files manually. For example, Jekyll has the -[redirect-from plugin](https://github.com/jekyll/jekyll-redirect-from). - ## Frequently Asked Questions ### Can I download my generated pages? @@ -580,8 +327,6 @@ No, you don't. You can create your project first and it will be accessed under For a list of known issues, visit GitLab's [public issue tracker]. [jekyll]: http://jekyllrb.com/ -[ee-80]: https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/80 -[ee-173]: https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/173 [pages-daemon]: https://gitlab.com/gitlab-org/gitlab-pages [gitlab ci]: https://about.gitlab.com/gitlab-ci [gitlab runner]: https://docs.gitlab.com/runner/ @@ -591,7 +336,6 @@ For a list of known issues, visit GitLab's [public issue tracker]. [pages-jekyll]: https://gitlab.com/pages/jekyll [metarefresh]: https://en.wikipedia.org/wiki/Meta_refresh [public issue tracker]: https://gitlab.com/gitlab-org/gitlab-ce/issues?label_name=pages -[ce-14605]: https://gitlab.com/gitlab-org/gitlab-ce/issues/14605 [quick start guide]: ../../../ci/quick_start/README.md [pages-index-guide]: index.md [pages-quick]: getting_started_part_one.md diff --git a/doc/user/project/pages/lets_encrypt_for_gitlab_pages.md b/doc/user/project/pages/lets_encrypt_for_gitlab_pages.md index f639188684b..91a660c0f7a 100644 --- a/doc/user/project/pages/lets_encrypt_for_gitlab_pages.md +++ b/doc/user/project/pages/lets_encrypt_for_gitlab_pages.md @@ -1,5 +1,7 @@ --- description: "How to secure GitLab Pages websites with Let's Encrypt." +type: howto +last_updated: 2019-06-04 --- # Let's Encrypt for GitLab Pages @@ -134,19 +136,19 @@ Now that your certificate has been issued, let's add it to your Pages site: sudo cat /etc/letsencrypt/live/example.com/fullchain.pem | pbcopy ``` -1. Copy and paste the public key into the second field **Key (PEM)**: +1. Copy and paste the private key into the second field **Key (PEM)**: ```bash sudo cat /etc/letsencrypt/live/example.com/privkey.pem | pbcopy ``` 1. Click **Save changes** to apply them to your website. -1. Wait a few minutes for DNS propagation. +1. Wait a few minutes for the configuration changes to take effect. 1. Visit your website at `https://example.com`. To force `https` connections on your site, navigate to your -project's **Settings > Pages** and check **Force domains with SSL -certificates to use HTTPS**. +project's **Settings > Pages** and check **Force HTTPS (requires +valid certificates)**. ## Renewal diff --git a/doc/user/project/pipelines/img/pipeline_schedule_variables.png b/doc/user/project/pipelines/img/pipeline_schedule_variables.png Binary files differindex 74692add93a..29846206491 100644 --- a/doc/user/project/pipelines/img/pipeline_schedule_variables.png +++ b/doc/user/project/pipelines/img/pipeline_schedule_variables.png diff --git a/doc/user/project/pipelines/img/pipeline_schedules_new_form.png b/doc/user/project/pipelines/img/pipeline_schedules_new_form.png Binary files differindex 95203ec861b..e135dd51070 100644 --- a/doc/user/project/pipelines/img/pipeline_schedules_new_form.png +++ b/doc/user/project/pipelines/img/pipeline_schedules_new_form.png diff --git a/doc/user/project/pipelines/job_artifacts.md b/doc/user/project/pipelines/job_artifacts.md index bf939dbdaa3..002addfc043 100644 --- a/doc/user/project/pipelines/job_artifacts.md +++ b/doc/user/project/pipelines/job_artifacts.md @@ -1,6 +1,7 @@ # Introduction to job artifacts > **Notes:** +> > - Since GitLab 8.2 and GitLab Runner 0.7.0, job artifacts that are created by > GitLab Runner are uploaded to GitLab and are downloadable as a single archive > (`tar.gz`) using the GitLab UI. @@ -15,7 +16,7 @@ > [administration/job_artifacts](../../../administration/job_artifacts.md). Artifacts is a list of files and directories which are attached to a job -after it completes successfully. This feature is enabled by default in all +after it finishes. This feature is enabled by default in all GitLab installations. ## Defining artifacts in `.gitlab-ci.yml` @@ -35,12 +36,14 @@ pdf: A job named `pdf` calls the `xelatex` command in order to build a pdf file from the latex source file `mycv.tex`. We then define the `artifacts` paths which in turn are defined with the `paths` keyword. All paths to files and directories -are relative to the repository that was cloned during the build. These uploaded -artifacts will be kept in GitLab for 1 week as defined by the `expire_in` -definition. You have the option to keep the artifacts from expiring via the -[web interface](#browsing-job-artifacts). If the expiry time is not defined, -it defaults to the [instance wide -setting](../../admin_area/settings/continuous_integration.md#default-artifacts-expiration-core-only). +are relative to the repository that was cloned during the build. + +The artifacts will be uploaded when the job succeeds by default, but can be set to upload +when the job fails, or always, if the [`artifacts:when`](../../../ci/yaml/README.md#artifactswhen) +parameter is used. These uploaded artifacts will be kept in GitLab for 1 week as defined +by the `expire_in` definition. You have the option to keep the artifacts from expiring +via the [web interface](#browsing-artifacts). If the expiry time is not defined, it defaults +to the [instance wide setting](../../admin_area/settings/continuous_integration.md#default-artifacts-expiration-core-only). For more examples on artifacts, follow the [artifacts reference in `.gitlab-ci.yml`](../../../ci/yaml/README.md#artifacts). @@ -54,7 +57,8 @@ For more examples on artifacts, follow the [artifacts reference in > **Note:** > With [GitLab 10.1][ce-14399], HTML files in a public project can be previewed > directly in a new tab without the need to download them when -> [GitLab Pages](../../../administration/pages/index.md) is enabled +> [GitLab Pages](../../../administration/pages/index.md) is enabled. +> The same holds for textual formats (currently supported extensions: `.txt`, `.json`, and `.log`). After a job finishes, if you visit the job's specific page, there are three buttons. You can download the artifacts archive or browse its contents, whereas @@ -152,7 +156,7 @@ For example: https://gitlab.com/gitlab-org/gitlab-ce/-/jobs/artifacts/master/browse?job=coverage ``` -There is also a URL to specific files, including html files that +There is also a URL to specific files, including html files that are shown in [GitLab Pages](../../../administration/pages/index.md): ``` @@ -183,7 +187,7 @@ information in the UI. DANGER: **Warning:** This is a destructive action that leads to data loss. Use with caution. -If you have at least Developer [permissions](../../permissions.md#gitlab-ci-cd-permissions) +If you have at least Developer [permissions](../../permissions.md#gitlab-cicd-permissions) on the project, you can erase a single job via the UI which will also remove the artifacts and the job's trace. @@ -191,9 +195,9 @@ artifacts and the job's trace. 1. Click the trash icon at the top right of the job's trace. 1. Confirm the deletion. -## Retrieve artifacts of private projects when using GitLab CI +## Retrieve artifacts of private projects when using GitLab CI In order to retrieve a job artifact of a different project, you might need to use a private token in order to [authenticate and download](../../../api/jobs.md#get-job-artifacts) the artifacts. [expiry date]: ../../../ci/yaml/README.md#artifactsexpire_in -[ce-14399]: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/14399
\ No newline at end of file +[ce-14399]: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/14399 diff --git a/doc/user/project/pipelines/schedules.md b/doc/user/project/pipelines/schedules.md index 58a0fbc97cd..dd5427ab35a 100644 --- a/doc/user/project/pipelines/schedules.md +++ b/doc/user/project/pipelines/schedules.md @@ -1,29 +1,34 @@ # Pipeline schedules > **Notes**: -> - This feature was introduced in 9.1 as [Trigger Schedule][ce-10533]. -> - In 9.2, the feature was [renamed to Pipeline Schedule][ce-10853]. +> +> - Introduced in GitLab 9.1 as [Trigger Schedule](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/10533). +> - [Renamed to Pipeline Schedule](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/10853) in GitLab 9.2. > - Cron notation is parsed by [Fugit](https://github.com/floraison/fugit). -Pipeline schedules can be used to run a pipeline at specific intervals, for example every -month on the 22nd for a certain branch. +Pipelines are normally run based on certain conditions being met. For example, when a branch is pushed to repository. -## Using Pipeline schedules +Pipeline schedules can be used to also run [pipelines](../../../ci/pipelines.md) at specific intervals. For example: -In order to schedule a pipeline: +- Every month on the 22nd for a certain branch. +- Once every day. -1. Navigate to your project's **CI / CD ➔ Schedules** and click the - **New Schedule** button. -1. Fill in the form -1. Hit **Save pipeline schedule** for the changes to take effect. +In addition to using the GitLab UI, pipeline schedules can be maintained using the +[Pipeline schedules API](../../../api/pipeline_schedules.md). + +## Configuring pipeline schedules + +To schedule a pipeline for project: + +1. Navigate to the project's **CI / CD > Schedules** page. +1. Click the **New schedule** button. +1. Fill in the **Schedule a new pipeline** form. +1. Click the **Save pipeline schedule** button.  -> **Attention:** -The pipelines won't be executed precisely, because schedules are handled by -Sidekiq, which runs according to its interval. -See [advanced admin configuration](#advanced-admin-configuration) for more -information. +NOTE: **Note:** +Pipelines execution [timing is dependent](#advanced-configuration) on Sidekiq's own schedule. In the **Schedules** index page you can see a list of the pipelines that are scheduled to run. The next run is automatically calculated by the server GitLab @@ -31,36 +36,24 @@ is installed on.  -### Running a scheduled pipeline manually - -> [Introduced][ce-15700] in GitLab 10.4. - -To trigger a pipeline schedule manually, click the "Play" button: - - - -This will schedule a background job to run the pipeline schedule. A flash -message will provide a link to the CI/CD Pipeline index page. - -To help avoid abuse, users are rate limited to triggering a pipeline once per -minute. - -### Making use of scheduled pipeline variables +### Using variables -> [Introduced][ce-12328] in GitLab 9.4. +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/12328) in GitLab 9.4. You can pass any number of arbitrary variables and they will be available in -GitLab CI so that they can be used in your `.gitlab-ci.yml` file. +GitLab CI so that they can be used in your [`.gitlab-ci.yml` file](../../../ci/yaml/README.md).  -## Using only and except +### Using only and except To configure that a job can be executed only when the pipeline has been scheduled (or the opposite), you can use -[only and except](../../../ci/yaml/README.md#only-and-except-simplified) configuration keywords. +[only and except](../../../ci/yaml/README.md#onlyexcept-basic) configuration keywords. -``` +For example: + +```yaml job:on-schedule: only: - schedules @@ -74,11 +67,47 @@ job: - make build ``` -## Taking ownership +### Advanced configuration + +The pipelines won't be executed exactly on schedule because schedules are handled by +Sidekiq, which runs according to its interval. + +For example, only two pipelines will be created per day if: -Pipelines are executed as a user, who owns a schedule. This influences what -projects and other resources the pipeline has access to. If a user does not own -a pipeline, you can take ownership by clicking the **Take ownership** button. +- You set a schedule to create a pipeline every minute (`* * * * *`). +- The Sidekiq worker runs on 00:00 and 12:00 every day (`0 */12 * * *`). + +To change the Sidekiq worker's frequency: + +1. Edit the `gitlab_rails['pipeline_schedule_worker_cron']` value in your instance's `gitlab.rb` file. +1. [Restart GitLab](../../../administration/restart_gitlab.md). + +For GitLab.com, refer to the [dedicated settings page](../../gitlab_com/index.md#cron-jobs). + +## Working with scheduled pipelines + +Once configured, GitLab supports many functions for working with scheduled pipelines. + +### Running manually + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/15700) in GitLab 10.4. + +To trigger a pipeline schedule manually, click the "Play" button: + + + +This will schedule a background job to run the pipeline schedule. A flash +message will provide a link to the CI/CD Pipeline index page. + +NOTE: **Note:** +To help avoid abuse, users are rate limited to triggering a pipeline once per +minute. + +### Taking ownership + +Pipelines are executed as a user, who owns a schedule. This influences what projects and other resources the pipeline has access to. + +If a user does not own a pipeline, you can take ownership by clicking the **Take ownership** button. The next time a pipeline is scheduled, your credentials will be used.  @@ -89,20 +118,3 @@ on the target branch, the schedule will stop creating new pipelines. This can happen if, for example, the owner is blocked or removed from the project, or the target branch or tag is protected. In this case, someone with sufficient privileges must take ownership of the schedule. - -## Advanced admin configuration - -The pipelines won't be executed precisely, because schedules are handled by -Sidekiq, which runs according to its interval. For example, if you set a -schedule to create a pipeline every minute (`* * * * *`) and the Sidekiq worker -runs on 00:00 and 12:00 every day (`0 */12 * * *`), only 2 pipelines will be -created per day. To change the Sidekiq worker's frequency, you have to edit the -`pipeline_schedule_worker_cron` value in your `gitlab.rb` and restart GitLab. -For GitLab.com, you can check the [dedicated settings page][settings]. If you -don't have admin access to the server, ask your administrator. - -[ce-10533]: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/10533 -[ce-10853]: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/10853 -[ce-12328]: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/12328 -[ce-15700]: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/15700 -[settings]: https://about.gitlab.com/gitlab-com/settings/#cron-jobs diff --git a/doc/user/project/pipelines/settings.md b/doc/user/project/pipelines/settings.md index 4a989afad4d..16f48c462eb 100644 --- a/doc/user/project/pipelines/settings.md +++ b/doc/user/project/pipelines/settings.md @@ -20,6 +20,22 @@ There are two options. Using: The default Git strategy can be overridden by the [GIT_STRATEGY variable](../../../ci/yaml/README.md#git-strategy) in `.gitlab-ci.yml`. +## Git shallow clone + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/28919) in GitLab 12.0. + +NOTE: **Note**: As of GitLab 12.0, newly created projects will automaticallyl have a default +`git depth` value of `50`. + +It is possible to limit the number of changes that GitLab CI/CD will fetch when cloning +a repository. Setting a limit to `git depth` can speed up Pipelines execution. Maximum +allowed value is `1000`. + +To disable shallow clone and make GitLab CI/CD fetch all branches and tags each time, +keep the value empty or set to `0`. + +This value can also be [overridden by `GIT_DEPTH`](../../../ci/large_repositories/index.md#shallow-cloning) variable in `.gitlab-ci.yml` file. + ## Timeout Timeout defines the maximum amount of time in minutes that a job is able run. @@ -182,4 +198,4 @@ https://example.gitlab.com/<namespace>/<project>/badges/<branch>/coverage.svg?st ## Environment Variables -[Environment variables](../../../ci/variables/README.html#variables) can be set in an environment to be available to a runner. +[Environment variables](../../../ci/variables/README.html#gitlab-cicd-environment-variables) can be set in an environment to be available to a runner. diff --git a/doc/user/project/protected_branches.md b/doc/user/project/protected_branches.md index db706e5020e..56e8f1731ae 100644 --- a/doc/user/project/protected_branches.md +++ b/doc/user/project/protected_branches.md @@ -15,12 +15,10 @@ By default, a protected branch does four simple things: - it prevents **anyone** from force pushing to the branch - it prevents **anyone** from deleting the branch -See the [Changelog](#changelog) section for changes over time. +>**Note**: +A GitLab admin is allowed to push to the protected branches. -> ->Additional functionality for GitLab Enterprise Edition: -> ->- Restrict push and merge access to [certain users][ee-restrict] +See the [Changelog](#changelog) section for changes over time. ## Configuring protected branches @@ -68,6 +66,21 @@ dropdown list in the "Already protected" area. If you don't choose any of those options while creating a protected branch, they are set to "Maintainers" by default. +## Restricting push and merge access to certain users **[STARTER]** + +> This feature was [introduced][ce-5081] in [GitLab Starter][ee] 8.11. + +With GitLab Enterprise Edition you can restrict access to protected branches +by choosing a role (Maintainers, Developers) as well as certain users. From the +dropdown menu select the role and/or the users you want to have merge or push +access. + + + +Click **Protect** and the branch will appear in the "Protected branch" list. + + + ## Wildcard protected branches > [Introduced][ce-4665] in GitLab 8.10. @@ -94,6 +107,25 @@ all matching branches:  +## Creating a protected branch + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/issues/53361) in GitLab 11.9. + +When a protected branch or wildcard protected branches are set to +[**No one** is **Allowed to push**](#using-the-allowed-to-merge-and-allowed-to-push-settings), +Developers (and users with higher [permission levels](../permissions.md)) are allowed +to create a new protected branch, but only via the UI or through the API (to avoid +creating protected branches accidentally from the command line or from a Git +client application). + +To create a new branch through the user interface: + +1. Visit **Repository > Branches**. +1. Click on **New branch**. +1. Fill in the branch name and select an existing branch, tag, or commit that + the new branch will be based off. Only existing protected branches and commits + that are already in protected branches will be accepted. + ## Deleting a protected branch > [Introduced][ce-21393] in GitLab 9.3. @@ -125,6 +157,10 @@ for details about the pipelines security model. ## Changelog +**11.9** + +- [Allow protected branches to be created](https://gitlab.com/gitlab-org/gitlab-ce/issues/53361) by Developers (and users with higher permission levels) through the API and the user interface. + **9.2** - Allow deletion of protected branches via the web interface [gitlab-org/gitlab-ce#21393][ce-21393] @@ -146,3 +182,4 @@ for details about the pipelines security model. [ce-21393]: https://gitlab.com/gitlab-org/gitlab-ce/issues/21393 [ee-restrict]: http://docs.gitlab.com/ee/user/project/protected_branches.html#restricting-push-and-merge-access-to-certain-users [perm]: ../permissions.md +[ee]: https://about.gitlab.com/pricing/ diff --git a/doc/user/project/quick_actions.md b/doc/user/project/quick_actions.md index 392e72dcc5c..1d640966013 100644 --- a/doc/user/project/quick_actions.md +++ b/doc/user/project/quick_actions.md @@ -26,15 +26,16 @@ discussions, and descriptions: | `/award :emoji:` | Toggle emoji award | ✓ | ✓ | | `/assign me` | Assign yourself | ✓ | ✓ | | `/assign @user` | Assign one user | ✓ | ✓ | -| `/assign @user1 @user2` | Assign multiple users **[STARTER]** | ✓ | | -| `/unassign` | Remove assignee(s) | ✓ | ✓ | -| `/reassign @user1 @user2` | Change assignee | ✓ | ✓ | +| `/assign @user1 @user2` | Assign multiple users **[STARTER]** | ✓ | ✓ | +| `/unassign @user1 @user2` | Remove assignee(s) **[STARTER]** | ✓ | ✓ | +| `/reassign @user1 @user2` | Change assignee **[STARTER]** | ✓ | ✓ | +| `/unassign` | Remove current assignee | ✓ | ✓ | | `/milestone %milestone` | Set milestone | ✓ | ✓ | | `/remove_milestone` | Remove milestone | ✓ | ✓ | -| `/label ~label1 ~label2` | Add label(s) | ✓ | ✓ | +| `/label ~label1 ~label2` | Add label(s). Label names can also start without ~ but mixed syntax is not supported. | ✓ | ✓ | | `/unlabel ~label1 ~label2` | Remove all or specific label(s)| ✓ | ✓ | | `/relabel ~label1 ~label2` | Replace label | ✓ | ✓ | -| <code>/copy_metadata #issue | !merge_request</code> | Copy labels and milestone from other issue or merge request | ✓ | ✓ | +| <code>/copy_metadata #issue | !merge_request</code> | Copy labels and milestone from other issue or merge request in the project | ✓ | ✓ | | <code>/estimate <1w 3d 2h 14m></code> | Set time estimate | ✓ | ✓ | | `/remove_estimate` | Remove time estimate | ✓ | ✓ | | <code>/spend <time(1h 30m | -1h 5m)> <date(YYYY-MM-DD)></code> | Add or subtract spent time; optionally, specify the date that time was spent on | ✓ | ✓ | @@ -45,13 +46,15 @@ discussions, and descriptions: | `/remove_due_date` | Remove due date | ✓ | | | `/weight 0,1,2, ...` | Set weight **[STARTER]** | ✓ | | | `/clear_weight` | Clears weight **[STARTER]** | ✓ | | -| `/epic <group&epic | Epic URL>` | Add to epic **[ULTIMATE]** | ✓ | | +| `/epic <&epic | group&epic | Epic URL>` | Add to epic **[ULTIMATE]** | ✓ | | | `/remove_epic` | Removes from epic **[ULTIMATE]** | ✓ | | +| `/promote` | Promote issue to epic **[ULTIMATE]** | ✓ | | | `/confidential` | Make confidential | ✓ | | | `/duplicate #issue` | Mark this issue as a duplicate of another issue | ✓ | | `/move path/to/project` | Move this issue to another project | ✓ | | | `/target_branch <Local branch Name>` | Set target branch | | ✓ | | `/wip` | Toggle the Work In Progress status | | ✓ | +| `/approve` | Approve the merge request | | ✓ | | `/merge` | Merge (when pipeline succeeds) | | ✓ | | `/create_merge_request <branch name>` | Create a new merge request starting from the current issue | ✓ | | diff --git a/doc/user/project/releases.md b/doc/user/project/releases.md index 737842962a9..baa7a295273 100644 --- a/doc/user/project/releases.md +++ b/doc/user/project/releases.md @@ -1 +1,5 @@ +--- +redirect_to: 'releases/index.md' +--- + This document was moved to [another location](releases/index.md). diff --git a/doc/user/project/repository/branches/img/branch_filter_search_box.png b/doc/user/project/repository/branches/img/branch_filter_search_box.png Binary files differindex 5dc7eccf189..5dc300cf24e 100644 --- a/doc/user/project/repository/branches/img/branch_filter_search_box.png +++ b/doc/user/project/repository/branches/img/branch_filter_search_box.png diff --git a/doc/user/project/repository/branches/index.md b/doc/user/project/repository/branches/index.md index f05554ffc5b..13e4f2ce163 100644 --- a/doc/user/project/repository/branches/index.md +++ b/doc/user/project/repository/branches/index.md @@ -11,7 +11,7 @@ Read through GiLab's branching documentation: See also: - [Branches API](../../../../api/branches.md), for information on operating on repository branches using the GitLab API. -- [GitLab Flow](../../../../university/training/gitlab_flow.md#gitlab-flow). Use the best of GitLab for your branching strategies. +- [GitLab Flow](../../../../university/training/gitlab_flow.md). Use the best of GitLab for your branching strategies. - [Getting started with Git](../../../../topics/git/index.md) and GitLab. ## Default branch diff --git a/doc/user/project/repository/gpg_signed_commits/index.md b/doc/user/project/repository/gpg_signed_commits/index.md index 6495ede42e0..3260a355fdc 100644 --- a/doc/user/project/repository/gpg_signed_commits/index.md +++ b/doc/user/project/repository/gpg_signed_commits/index.md @@ -264,7 +264,7 @@ To remove a GPG key from your account: ## Rejecting commits that are not signed **[PREMIUM]** You can configure your project to reject commits that aren't GPG-signed -via [push rules](https://docs.gitlab.com/ee/push_rules/push_rules.html). +via [push rules](../../../../push_rules/push_rules.md). ## GPG signing API diff --git a/doc/user/project/repository/img/download_source_code.png b/doc/user/project/repository/img/download_source_code.png Binary files differnew file mode 100644 index 00000000000..17f2cb4b3e8 --- /dev/null +++ b/doc/user/project/repository/img/download_source_code.png diff --git a/doc/user/project/repository/img/repository_cleanup.png b/doc/user/project/repository/img/repository_cleanup.png Binary files differindex bda40d3e193..e343f23ac27 100644 --- a/doc/user/project/repository/img/repository_cleanup.png +++ b/doc/user/project/repository/img/repository_cleanup.png diff --git a/doc/user/project/repository/index.md b/doc/user/project/repository/index.md index 22d912cd9d1..6fccfd40987 100644 --- a/doc/user/project/repository/index.md +++ b/doc/user/project/repository/index.md @@ -97,7 +97,7 @@ Some things to note about precedence: > [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/issues/2508) in GitLab 9.1 -[Jupyter][jupyter] Notebook (previously IPython Notebook) files are used for +[Jupyter](https://jupyter.org) Notebook (previously IPython Notebook) files are used for interactive computing in many fields and contain a complete record of the user's sessions and include code, narrative text, equations and rich output. @@ -123,7 +123,7 @@ You can live preview changes submitted to a new branch with [Review Apps](../../../ci/review_apps/index.md). With [GitLab Starter](https://about.gitlab.com/pricing/), you can also request -[approval](https://docs.gitlab.com/ee/user/project/merge_requests/merge_request_approvals.html) from your managers. +[approval](../merge_requests/merge_request_approvals.md) from your managers. To create, delete, and view [branches](branches/index.md) via GitLab's UI: @@ -154,7 +154,7 @@ Via command line, you can commit multiple times before pushing. you will trigger a pipeline per push, not per commit. - **Skip pipelines:** You can add to you commit message the keyword - [`[ci skip]`](../../../ci/yaml/README.html#skipping-jobs) + [`[ci skip]`](../../../ci/yaml/README.md#skipping-jobs) and GitLab CI will skip that pipeline. - **Cross-link issues and merge requests:** [Cross-linking](../issues/crosslinking_issues.md#from-commit-messages) @@ -173,11 +173,15 @@ Via command line, you can commit multiple times before pushing. ## Repository size -On GitLab.com, your [repository size limit is 10GB](../../gitlab_com/index.md#repository-size-limit) -(including LFS). For other instances, the repository size is limited by your -system administrators. +A project's repository size is reported on the project's **Details** page. The reported size is +updated every 15 minutes at most, so may not reflect recent activity. -You can also [reduce a repository size using Git](reducing_the_repo_size_using_git.md). +The repository size for: + +- GitLab.com [is set by GitLab](../../gitlab_com/index.md#repository-size-limit). +- Self-managed instances is set by your GitLab administrators. + +You can [reduce a repository's size using Git](reducing_the_repo_size_using_git.md). ## Contributors @@ -220,14 +224,10 @@ Select branches to compare using the [branch filter search box](branches/index.m Find it under your project's **Repository > Compare**. -## Locked files - -> Available in [GitLab Premium](https://about.gitlab.com/pricing/). - -Lock your files to prevent any conflicting changes. +## Locked files **[PREMIUM]** -[File Locking](https://docs.gitlab.com/ee/user/project/file_lock.html) is available only in -[GitLab Premium](https://about.gitlab.com/pricing/). +Use [File Locking](../file_lock.md) to +lock your files to prevent any conflicting changes. ## Repository's API @@ -241,4 +241,21 @@ Projects that contain a `.xcodeproj` or `.xcworkspace` directory can now be clon in Xcode using the new **Open in Xcode** button, located next to the Git URL used for cloning your project. The button is only shown on macOS. -[jupyter]: https://jupyter.org +## Download Source Code + +> Support for directory download was [introduced](https://gitlab.com/gitlab-org/gitlab-ce/issues/24704) in GitLab 11.11. + +The source code stored in a repository can be downloaded from the UI. +By clicking the download icon, a dropdown will open with links to download the following: + + + +- **Source code:** + allows users to download the source code on branch they're currently + viewing. Available extensions: `zip`, `tar`, `tar.gz`, and `tar.bz2`. +- **Directory:** + only shows up when viewing a sub-directory. This allows users to download + the specific directory they're currently viewing. Also available in `zip`, + `tar`, `tar.gz`, and `tar.bz2`. +- **Artifacts:** + allows users to download the artifacts of the latest CI build. diff --git a/doc/user/project/repository/reducing_the_repo_size_using_git.md b/doc/user/project/repository/reducing_the_repo_size_using_git.md index 672567a8d7d..e3d771524ce 100644 --- a/doc/user/project/repository/reducing_the_repo_size_using_git.md +++ b/doc/user/project/repository/reducing_the_repo_size_using_git.md @@ -1,6 +1,6 @@ # Reducing the repository size using Git -A GitLab Enterprise Edition administrator can set a [repository size limit][admin-repo-size] +A GitLab Enterprise Edition administrator can set a [repository size limit](../../admin_area/settings/account_and_limit_settings.md) which will prevent you from exceeding it. When a project has reached its size limit, you will not be able to push to it, @@ -14,7 +14,8 @@ move some blobs to LFS, or remove some old dependency updates from history. Unfortunately, it's not so easy and that workflow won't work. Deleting files in a commit doesn't actually reduce the size of the repo since the earlier commits and blobs are still around. What you need to do is rewrite history with Git's -[`filter-branch` option][gitscm], or a tool like the [BFG Repo-Cleaner][bfg]. +[`filter-branch` option](https://git-scm.com/book/en/v2/Git-Tools-Rewriting-History#The-Nuclear-Option:-filter-branch), +or a tool like the [BFG Repo-Cleaner](https://rtyley.github.io/bfg-repo-cleaner/). Note that even with that method, until `git gc` runs on the GitLab side, the "removed" commits and blobs will still be around. You also need to be able to @@ -98,6 +99,12 @@ up its own internal state, maximizing the space saved. `git gc` against the repository. You will receive an email once it has completed. +This process will remove some copies of the rewritten commits from GitLab's +cache and database, but there are still numerous gaps in coverage - at present, +some of the copies may persist indefinitely. [Clearing the instance cache] +(../../../administration/raketasks/maintenance.md#clear-redis-cache) may help to +remove some of them, but it should not be depended on for security purposes! + ## Using `git filter-branch` 1. Navigate to your repository: @@ -131,7 +138,3 @@ up its own internal state, maximizing the space saved. ``` Your repository should now be below the size limit. - -[admin-repo-size]: https://docs.gitlab.com/ee/user/admin_area/settings/account_and_limit_settings.html#repository-size-limit -[bfg]: https://rtyley.github.io/bfg-repo-cleaner/ -[gitscm]: https://git-scm.com/book/en/v2/Git-Tools-Rewriting-History#The-Nuclear-Option:-filter-branch diff --git a/doc/user/project/security_dashboard.md b/doc/user/project/security_dashboard.md new file mode 100644 index 00000000000..a3da1ec97d3 --- /dev/null +++ b/doc/user/project/security_dashboard.md @@ -0,0 +1,5 @@ +--- +redirect_to: '../application_security/security_dashboard/index.md' +--- + +This document was moved to [another location](../application_security/security_dashboard/index.md). diff --git a/doc/user/project/service_desk.md b/doc/user/project/service_desk.md new file mode 100644 index 00000000000..1a582164d03 --- /dev/null +++ b/doc/user/project/service_desk.md @@ -0,0 +1,124 @@ +# Service Desk **[PREMIUM]** + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/issues/149) in [GitLab Premium 9.1](https://about.gitlab.com/2017/04/22/gitlab-9-1-released/#service-desk-eep). + +## Overview + +Service Desk is a module that allows your team to connect directly +with any external party through email right inside of GitLab; no external tools required. +An ongoing conversation right where your software is built ensures that user feedback ends +up directly where it's needed, helping you build the right features to solve your users' +real problems. + +With Service Desk, you can provide efficient email support to your customers, who can now +email you bug reports, feature requests, or general feedback that will all end up in your +GitLab project as new issues. In turn, your team can respond straight from the project. + +As Service Desk is built right into GitLab itself, the complexity and inefficiencies +of multiple tools and external integrations are eliminated, significantly shortening +the cycle time from feedback to software update. + +For an overview, check the video demonstration on [GitLab Service Desk](https://about.gitlab.com/2017/05/09/demo-service-desk/). + +## Use cases + +For instance, let's assume you develop a game for iOS or Android. +The codebase is hosted in your GitLab instance, built and deployed +with GitLab CI. + +Here's how Service Desk will work for you: + +1. You'll provide a project-specific email address to your paying customers, who can email you directly from within the app +1. Each email they send creates an issue in the appropriate project +1. Your team members navigate to the Service Desk issue tracker, where they can see new support requests and respond inside associated issues +1. Your team communicates back and forth with the customer to understand the request +1. Your team starts working on implementing code to solve your customer's problem +1. When your team finishes the implementation, whereupon the merge request is merged and the issue is closed automatically +1. The customer will have been attended successfully via email, without having real access to your GitLab instance +1. Your team saved time by not having to leave GitLab (or setup any integrations) to follow up with your customer + +## How it works + +GitLab Service Desk is a simple way to allow people to create issues in your +GitLab instance without needing their own user account. + +It provides a unique email address for end users to create issues in a project, +and replies can be sent either through the GitLab interface or by email. End +users will only see the thread through email. + +## Configuring Service Desk + +> **Note:** +Service Desk is enabled on GitLab.com. If you're a +[Silver subscriber](https://about.gitlab.com/gitlab-com/), +you can skip the step 1 below; you only need to enable it per project. + +1. [Set up incoming email](../../administration/incoming_email.md#set-it-up) for the GitLab instance. This must + support [email sub-addressing](../../administration/incoming_email.md#email-sub-addressing). +2. Navigate to your project's **Settings** and scroll down to the **Service Desk** + section. +3. If you have the correct access and an Premium license, + you will see an option to set up Service Desk: + +  + +4. Checking that box will enable Service Desk for the project, and show a + unique email address to email issues to the project. These issues will be + [confidential](issues/confidential_issues.md), so they will only be visible to project members. + + **Warning**: this email address can be used by anyone to create an issue on + this project, whether or not they have access to your GitLab instance. + We recommend **putting this behind an alias** so that it can be changed if + needed, and **[enabling Akismet](../../integration/akismet.md)** on your GitLab instance to add spam + checking to this service. Unblocked email spam would result in many spam + issues being created, and may disrupt your GitLab service. + +  + + _In GitLab 11.7, we updated the format of the generated email address. + However the older format is still supported, allowing existing aliases + or contacts to continue working._ + + +5. Service Desk is now enabled for this project! You should be able to access it from your project's navigation **Issue submenu**: + +  + +## Using Service Desk + +### As an end user (issue creator) + +To create a Service Desk issue, an end user doesn't need to know anything about +the GitLab instance. They just send an email to the address they are given, and +receive an email back confirming receipt: + + + +This also gives the end user an option to unsubscribe. + +If they don't choose to unsubscribe, then any new comments added to the issue +will be sent as emails: + + + +And any responses they send will be displayed in the issue itself. + +### As a responder to the issue + +For responders to the issue, everything works as usual. They'll see a familiar looking +issue tracker, where they can see issues created via customer support requests and +filter and interact with them just like other GitLab issues. + + + +Messages from the end user will show as coming from the special Support Bot user, but apart from that, +you can read and write comments as you normally do: + + + +> Note that the project's visibility (private, internal, public) does not affect Service Desk. + +### Support Bot user + +Behind the scenes, Service Desk works by the special Support Bot user creating issues. This user +does not count toward the license limit count. diff --git a/doc/user/project/settings/img/import_export_download_export.png b/doc/user/project/settings/img/import_export_download_export.png Binary files differindex 668254073e8..ab81c87bf5f 100644 --- a/doc/user/project/settings/img/import_export_download_export.png +++ b/doc/user/project/settings/img/import_export_download_export.png diff --git a/doc/user/project/settings/img/import_export_export_button.png b/doc/user/project/settings/img/import_export_export_button.png Binary files differindex 7f21bb2335b..9e368739695 100644 --- a/doc/user/project/settings/img/import_export_export_button.png +++ b/doc/user/project/settings/img/import_export_export_button.png diff --git a/doc/user/project/settings/img/import_export_mail_link.png b/doc/user/project/settings/img/import_export_mail_link.png Binary files differindex 48ef42855bc..985c37650d3 100644 --- a/doc/user/project/settings/img/import_export_mail_link.png +++ b/doc/user/project/settings/img/import_export_mail_link.png diff --git a/doc/user/project/settings/img/import_export_new_project.png b/doc/user/project/settings/img/import_export_new_project.png Binary files differindex b335700c5be..fc1f73c5d6e 100644 --- a/doc/user/project/settings/img/import_export_new_project.png +++ b/doc/user/project/settings/img/import_export_new_project.png diff --git a/doc/user/project/settings/img/import_export_select_file.png b/doc/user/project/settings/img/import_export_select_file.png Binary files differindex e1e5e031d81..e3e1a5ef980 100644 --- a/doc/user/project/settings/img/import_export_select_file.png +++ b/doc/user/project/settings/img/import_export_select_file.png diff --git a/doc/user/project/settings/img/settings_edit_button.png b/doc/user/project/settings/img/settings_edit_button.png Binary files differdeleted file mode 100644 index 32bcda03c7e..00000000000 --- a/doc/user/project/settings/img/settings_edit_button.png +++ /dev/null diff --git a/doc/user/project/settings/import_export.md b/doc/user/project/settings/import_export.md index 89008fd15b9..819515d7a4c 100644 --- a/doc/user/project/settings/import_export.md +++ b/doc/user/project/settings/import_export.md @@ -2,10 +2,11 @@ >**Notes:** > -> - [Introduced][ce-3050] in GitLab 8.9. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/issues/3050) in GitLab 8.9. > - Importing will not be possible if the import instance version differs from > that of the exporter. -> - For GitLab admins, please read through [Project import/export administration](../../../administration/raketasks/project_import_export.md). +> - For GitLab admins, please read through +> [Project import/export administration](../../../administration/raketasks/project_import_export.md). > - For existing installations, the project import option has to be enabled in > application settings (`/admin/application_settings`) under 'Import sources'. > Ask your administrator if you don't see the **GitLab export** button when @@ -14,15 +15,15 @@ > on the GitLab instance in application settings (`/admin/application_settings`) > under 'Visibility and Access Controls'. > - You can find some useful raketasks if you are an administrator in the -> [import_export](../../../administration/raketasks/project_import_export.md) -> raketask. -> - The exports are stored in a temporary [shared directory][tmp] and are deleted -> every 24 hours by a specific worker. +> [import_export](../../../administration/raketasks/project_import_export.md) raketask. +> - The exports are stored in a temporary [shared directory](../../../development/shared_files.md) +> and are deleted every 24 hours by a specific worker. > - Group members will get exported as project members, as long as the user has > maintainer or admin access to the group where the exported project lives. An admin > in the import side is required to map the users, based on email or username. > Otherwise, a supplementary comment is left to mention the original author and > the MRs, notes or issues will be owned by the importer. +> - Project members with owner access will get imported as maintainers. > - Control project Import/Export with the [API](../../../api/project_import_export.md). > - If an imported project contains merge requests originated from forks, > then new branches associated with such merge requests will be created @@ -76,9 +77,9 @@ The following items will NOT be exported: ## Exporting a project and its data -1. Go to the project settings page by clicking on **Edit Project**: +1. Go to your project's homepage. -  +1. Click **Settings** in the sidebar. 1. Scroll down to find the **Export project** button: @@ -97,19 +98,14 @@ The following items will NOT be exported: ## Importing the project -1. The new GitLab project import feature is at the far right of the import - options when creating a New Project. Make sure you are in the right namespace - and you have entered a project name. Click on **GitLab export**: +1. The GitLab project import feature is the first import option when creating a + new project. Click on **GitLab export**:  -1. You can see where the project will be imported to. You can now select file - exported previously: +1. Enter your project name and URL. Then select the file you exported previously:  1. Click on **Import project** to begin importing. Your newly imported project page will appear soon. - -[ce-3050]: https://gitlab.com/gitlab-org/gitlab-ce/issues/3050 -[tmp]: ../../../development/shared_files.md diff --git a/doc/user/project/settings/index.md b/doc/user/project/settings/index.md index d5f4a7fd4ac..e3502a632d9 100644 --- a/doc/user/project/settings/index.md +++ b/doc/user/project/settings/index.md @@ -36,15 +36,15 @@ Set up your project's merge request settings: - Set up the merge request method (merge commit, [fast-forward merge](../merge_requests/fast_forward_merge.html)). - Merge request [description templates](../description_templates.md#description-templates). -- Enable [merge request approvals](https://docs.gitlab.com/ee/user/project/merge_requests/merge_request_approvals.html#merge-request-approvals). **[STARTER]** +- Enable [merge request approvals](../merge_requests/merge_request_approvals.md). **[STARTER]** - Enable [merge only of pipeline succeeds](../merge_requests/merge_when_pipeline_succeeds.md). - Enable [merge only when all discussions are resolved](../../discussions/index.md#only-allow-merge-requests-to-be-merged-if-all-discussions-are-resolved).  -### Service Desk +### Service Desk **[PREMIUM]** -Enable [Service Desk](https://docs.gitlab.com/ee/user/project/service_desk.html) for your project to offer customer support. Service Desk is available in [GitLab Premium](https://about.gitlab.com/pricing/). +Enable [Service Desk](../service_desk.md) for your project to offer customer support. ### Export project @@ -100,9 +100,9 @@ Only project Owners and Admin users have the [permissions] to transfer a project You can transfer an existing project into a [group](../../group/index.md) if: -1. you have at least **Maintainer** [permissions] to that group -1. you are an **Owner** of the project. - +1. You have at least **Maintainer** [permissions] to that group. +1. The project is in a subgroup you own. +1. You are at least a **Maintainer** of the project under your personal namespace. Similarly, if you are an owner of a group, you can transfer any of its projects under your own user. @@ -128,3 +128,7 @@ namespace if needed. ### Error Tracking Configure Error Tracking to discover and view [Sentry errors within GitLab](../operations/error_tracking.md). + +### Jaeger tracing **[ULTIMATE]** + +Add the URL of a Jaeger server to allow your users to [easily access the Jaeger UI from within GitLab](../operations/tracing.md). diff --git a/doc/user/project/slash_commands.md b/doc/user/project/slash_commands.md index e9103a3f49c..1280524bc9b 100644 --- a/doc/user/project/slash_commands.md +++ b/doc/user/project/slash_commands.md @@ -1 +1,5 @@ +--- +redirect_to: 'quick_actions.md' +--- + This document was moved to [user/project/quick_actions.md](quick_actions.md). diff --git a/doc/user/project/web_ide/img/enable_web_ide.png b/doc/user/project/web_ide/img/enable_web_ide.png Binary files differdeleted file mode 100644 index 196baa82ad2..00000000000 --- a/doc/user/project/web_ide/img/enable_web_ide.png +++ /dev/null diff --git a/doc/user/project/web_ide/index.md b/doc/user/project/web_ide/index.md index 46a1b2bc3aa..a634a8b2f54 100644 --- a/doc/user/project/web_ide/index.md +++ b/doc/user/project/web_ide/index.md @@ -8,7 +8,7 @@ projects by providing an advanced editor with commit staging. ## Open the Web IDE -The Web IDE can be opened when viewing a file, from the repository file list, +You can open the Web IDE when viewing a file, from the repository file list, and from merge requests.  @@ -45,7 +45,7 @@ Single file editing is based on the [Ace Editor](https://ace.c9.io). ## Stage and commit changes -After making your changes, click the Commit button in the bottom left to +After making your changes, click the **Commit** button in the bottom left to review the list of changed files. Click on each file to review the changes and click the tick icon to stage the file. @@ -67,10 +67,11 @@ shows you a preview of the merge request diff if you commit your changes. > [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/19279) in [GitLab Core][ce] 11.0. -The Web IDE can be used to quickly fix failing tests by opening the branch or -merge request in the Web IDE and opening the logs of the failed job. The status -of all jobs for the most recent pipeline and job traces for the current commit -can be accessed by clicking the **Pipelines** button in the top right. +You can use the Web IDE to quickly fix failing tests by opening +the branch or merge request in the Web IDE and opening the logs of the failed +job. You can access the status of all jobs for the most recent pipeline and job +traces for the current commit by clicking the **Pipelines** button in the top +right. The pipeline status is also shown at all times in the status bar in the bottom left. @@ -79,31 +80,31 @@ left. > [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/19318) in [GitLab Core][ce] 11.0. -Switching between your authored and assigned merge requests can be done without -leaving the Web IDE. Click the dropdown in the top of the sidebar to open a list -of merge requests. You will need to commit or discard all your changes before -switching to a different merge request. +To switch between your authored and assigned merge requests, click the +dropdown in the top of the sidebar to open a list of merge requests. You will +need to commit or discard all your changes before switching to a different merge +request. ## Switching branches > [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/20850) in [GitLab Core][ce] 11.2. -Switching between branches of the current project repository can be done without -leaving the Web IDE. Click the dropdown in the top of the sidebar to open a list -of branches. You will need to commit or discard all your changes before -switching to a different branch. +To switch between branches of the current project repository, click the dropdown +in the top of the sidebar to open a list of branches. +You will need to commit or discard all your changes before switching to a +different branch. ## Client Side Evaluation > [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/19764) in [GitLab Core][ce] 11.2. -The Web IDE can be used to preview JavaScript projects right in the browser. +You can use the Web IDE to preview JavaScript projects right in the browser. This feature uses CodeSandbox to compile and bundle the JavaScript used to preview the web application.  -Additionally, for public projects an `Open in CodeSandbox` button is available +Additionally, for public projects an **Open in CodeSandbox** button is available to transfer the contents of the project into a public CodeSandbox project to quickly share your project with others. @@ -115,9 +116,9 @@ GitLab.com  -Once it has been enabled in application settings, projects with a -`package.json` file and a `main` entry point can be previewed inside of the Web -IDE. An example `package.json` is below. +Once you have done that, you can preview projects with a `package.json` file and +a `main` entry point inside the Web IDE. An example `package.json` is shown +below. ```json { @@ -128,5 +129,123 @@ IDE. An example `package.json` is below. } ``` +## Interactive Web Terminals for the Web IDE **[ULTIMATE ONLY]** + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/issues/5426) in [GitLab Ultimate](https://about.gitlab.com/pricing/) 11.6. + +CAUTION: **Warning:** +Interactive Web Terminals for the Web IDE is currently in **Beta**. + +[Interactive web terminals](../../../ci/interactive_web_terminal/index.md) +give the user access to a terminal to interact with the Runner directly from +GitLab, including through the Web IDE. + +Only project [**maintainers**](../../permissions.md#project-members-permissions) +can run Interactive Web Terminals through the Web IDE. + +CAUTION: **Warning:** +GitLab.com [does not support Interactive Web Terminals yet](https://gitlab.com/gitlab-org/gitlab-ce/issues/52611). +Shared Runners in private instances are not supported either. + +### Runner configuration + +Some things need to be configured in the runner for the interactive web terminal +to work: + +- The Runner needs to have + [`[session_server]` configured properly](https://docs.gitlab.com/runner/configuration/advanced-configuration.html#the-session_server-section). +- If you are using a reverse proxy with your GitLab instance, web terminals need to be + [enabled](../../../administration/integration/terminal.md#enabling-and-disabling-terminal-support). **[ULTIMATE ONLY]** + +If you have the terminal open and the job has finished with its tasks, the +terminal will block the job from finishing for the duration configured in +[`[session_server].terminal_max_retention_time`](https://docs.gitlab.com/runner/configuration/advanced-configuration.html#the-session_server-section) +until you close the terminal window. + +NOTE: **Note:** Not all executors are +[supported](https://docs.gitlab.com/runner/executors/#compatibility-chart) + +### Web IDE configuration file + +In order to enable the Web IDE terminals you need to create the file +`.gitlab/.gitlab-webide.yml` inside the repository's root. This +file is fairly similar to the [CI configuration file](../../../ci/yaml/README.md) +syntax but with some restrictions: + +- No global blocks can be defined (ie: `before_script` or `after_script`) +- Only one job named `terminal` can be added to this file. +- Only the keywords `image`, `services`, `tags`, `before_script`, `script`, and +`variables` are allowed to be used to configure the job. +- To connect to the interactive terminal, the `terminal` job must be still alive +and running, otherwise the terminal won't be able to connect to the job's session. +By default the `script` keyword has the value `sleep 60` to prevent +the job from ending and giving the Web IDE enough time to connect. This means +that, if you override the default `script` value, you'll have to add a command +which would keep the job running, like `sleep`. + +In the code below there is an example of this configuration file: + +```yaml +terminal: + before_script: + - apt-get update + script: sleep 60 + variables: + RAILS_ENV: "test" + NODE_ENV: "test" +``` + +Once the terminal has started, the console will be displayed and we could access +the project repository files. + +**Important**. The terminal job is branch dependant. This means that the +configuration file used to trigger and configure the terminal will be the one in +the selected branch of the Web IDE. + +If there is no configuration file in a branch, an error message will be shown. + +### Running Interactive Terminals in the Web IDE + +If Interactive Terminals are available for the current user, the **Terminal** button +will be visible in the right sidebar of the Web IDE. Click this button to open +or close the terminal tab. + +Once open, the tab will show the **Start Web Terminal** button. This button may +be disabled if the environment is not configured correctly. If so, a status +message will describe the issue. Here are some reasons why **Start Web Terminal** +may be disabled: + +- `.gitlab/.gitlab-webide.yml` does not exist or is set up incorrectly. +- No active private runners are available for the project. + +If active, clicking the **Start Web Terminal** button will load the terminal view +and start connecting to the runner's terminal. At any time, the **Terminal** tab +can be closed and reopened and the state of the terminal will not be affected. + +When the terminal is started and is successfully connected to the runner, then the +runner's shell prompt will appear in the terminal. From here, you can enter +commands that will be executed within the runner's environment. This is similar +to running commands in a local terminal or through SSH. + +While the terminal is running, it can be stopped by clicking **Stop Terminal**. +This will disconnect the terminal and stop the runner's terminal job. From here, +click **Restart Terminal** to start a new terminal session. + +### Limitations + +Interactive Terminals is in a beta phase and will continue to be improved upon in upcoming +releases. In the meantime, please note that the user is limited to having only one +active terminal at a time. + +### Troubleshooting + +- If the terminal's text is gray and unresponsive, then the terminal has stopped + and it can no longer be used. A stopped terminal can be restarted by clicking + **Restart Terminal**. +- If the terminal displays **Connection Failure**, then the terminal could not + connect to the runner. Please try to stop and restart the terminal. If the + problem persists, double check your runner configuration. + + [ce]: https://about.gitlab.com/pricing/ [ee]: https://about.gitlab.com/pricing/ diff --git a/doc/user/project/wiki/index.md b/doc/user/project/wiki/index.md index 127a30d6669..e3c6cd6d6ff 100644 --- a/doc/user/project/wiki/index.md +++ b/doc/user/project/wiki/index.md @@ -12,10 +12,6 @@ You can create Wiki pages in the web interface or [locally using Git](#adding-and-editing-wiki-pages-locally) since every Wiki is a separate Git repository. ->**Note:** -A [permission level][permissions] of **Guest** is needed to view a Wiki and -**Developer** is needed to create and edit Wiki pages. - ## First time creating the Home page The first time you visit a Wiki, you will be directed to create the Home page. @@ -28,6 +24,9 @@ message. ## Creating a new wiki page +NOTE: **Note:** +Requires Developer [permissions](../../permissions.md). + Create a new page by clicking the **New page** button that can be found in all wiki pages. You will be asked to fill in the page name from which GitLab will create the path to the page. You can specify a full path for the new file @@ -58,12 +57,18 @@ repository, you will have to upload them again. ## Editing a wiki page +NOTE: **Note:** +Requires Developer [permissions](../../permissions.md). + To edit a page, simply click on the **Edit** button. From there on, you can change its content. When done, click **Save changes** for the changes to take effect. ## Deleting a wiki page +NOTE: **Note:** +Requires Maintainer [permissions](../../permissions.md). + You can find the **Delete** button only when editing a page. Click on it and confirm you want the page to be deleted. @@ -109,11 +114,11 @@ them like you would do with every other Git repository. On the right sidebar, click on **Clone repository** and follow the on-screen instructions. -[permissions]: ../../permissions.md - ## Customizing sidebar -By default, the wiki would render a sidebar which lists all the pages for the -wiki. You could as well provide a `_sidebar` page to replace this default -sidebar. When this customized sidebar page is provided, the default sidebar -would not be rendered, but the customized one. +On the project's Wiki page, there is a right side navigation that renders the full Wiki pages list by default, with hierarchy. + +If the Wiki repository contains a `_sidebar` page, the file of this page replaces the default side navigation. +This custom file serves to render it's custom content, fully replacing the standard sidebar. + +Support for displaying a generated TOC with a custom side navigation is planned. diff --git a/doc/user/reserved_names.md b/doc/user/reserved_names.md index 9aa81e33fc0..9e8475d8294 100644 --- a/doc/user/reserved_names.md +++ b/doc/user/reserved_names.md @@ -83,6 +83,7 @@ Currently the following names are reserved as top level groups: - unsubscribes - uploads - users +- v2 These group names are unavailable as subgroup names: diff --git a/doc/user/search/advanced_global_search.md b/doc/user/search/advanced_global_search.md new file mode 100644 index 00000000000..f80f4183802 --- /dev/null +++ b/doc/user/search/advanced_global_search.md @@ -0,0 +1,75 @@ +# Advanced Global Search **[STARTER ONLY]** + +> - [Introduced][ee-109] in GitLab [Starter][ee] 8.4. +> - This is the user documentation. To install and configure Elasticsearch, +> visit the [administrator documentation](../../integration/elasticsearch.md). + +NOTE: **Note** +Advanced Global Search (powered by Elasticsearch) is not yet available on GitLab.com. We are working on adding it. [Follow this epic for the latest updates](https://gitlab.com/groups/gitlab-org/-/epics/153). + +Leverage Elasticsearch for faster, more advanced code search across your entire +GitLab instance. + +## Overview + +The Advanced Global Search in GitLab is a powerful search service that saves +you time. Instead of creating duplicate code and wasting time, you can +now search for code within other teams that can help your own project. + +GitLab leverages the search capabilities of [Elasticsearch] and enables it when +searching in: + +- GitLab application +- Projects +- Repositories +- Commits +- Issues +- Merge requests +- Milestones +- Notes (comments) +- Snippets +- Wiki + +## Use cases + +The Advanced Global Search can be useful in various scenarios. + +### Faster searches + +If you are dealing with huge amount of data and want to keep GitLab's search +fast, the Advanced Global Search will help you achieve that. + +### Promote innersourcing + +Your company may consist of many different developer teams each of which has +their own group where the various projects are hosted. Some of your applications +may be connected to each other, so your developers need to instantly search +throughout the GitLab instance and find the code they search for. + +## Searching globally + +Just use the search as before and GitLab will show you matching code from each +project you have access to. + + + +You can also use the [Advanced Syntax Search](advanced_search_syntax.md) which +provides some useful queries. + +>**Note:** +Elasticsearch has only data for the default branch. That means that if you go +to the repository tree and switch the branch from the default to something else, +then the "Code" tab in the search result page will be served by the regular +search even if Elasticsearch is enabled. + +[ee-1305]: https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/1305 +[aws-elastic]: http://docs.aws.amazon.com/elasticsearch-service/latest/developerguide/es-gsg.html +[aws-iam]: http://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_access-keys.html +[aws-instance-profile]: http://docs.aws.amazon.com/codedeploy/latest/userguide/getting-started-create-iam-instance-profile.html#getting-started-create-iam-instance-profile-cli +[ee-109]: https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/109 "Elasticsearch Merge Request" +[elasticsearch]: https://www.elastic.co/products/elasticsearch "Elasticsearch website" +[install]: https://www.elastic.co/guide/en/elasticsearch/reference/current/_installation.html "Elasticsearch installation documentation" +[pkg]: https://about.gitlab.com/downloads/ "Download Omnibus GitLab" +[elastic-settings]: https://www.elastic.co/guide/en/elasticsearch/reference/current/setup-configuration.html#settings "Elasticsearch configuration settings" +[ee]: https://about.gitlab.com/pricing/ +[es]: https://www.elastic.co/products/elasticsearch diff --git a/doc/user/search/advanced_search_syntax.md b/doc/user/search/advanced_search_syntax.md new file mode 100644 index 00000000000..d302cb7a809 --- /dev/null +++ b/doc/user/search/advanced_search_syntax.md @@ -0,0 +1,69 @@ +# Advanced Syntax Search **[STARTER ONLY]** + +> **Notes:** +> - Introduced in [GitLab Enterprise Starter][ee] 9.2 +> - This is the user documentation. To install and configure Elasticsearch, +> visit the [administrator documentation](../../integration/elasticsearch.md). + +NOTE: **Note** +Advanced Global Search (powered by Elasticsearch) is not yet available on GitLab.com. We are working on adding it. [Follow this epic for the latest updates](https://gitlab.com/groups/gitlab-org/-/epics/153). + +Use advanced queries for more targeted search results. + +## Overview + +The Advanced Syntax Search is a subset of the +[Advanced Global Search](advanced_global_search.md), which you can use if you +want to have more specific search results. + +## Use cases + +Let's say for example that the product you develop relies on the code of another +product that's hosted under some other group. + +Since under your GitLab instance there are hosted hundreds of different projects, +you need the search results to be as efficient as possible. You have a feeling +of what you want to find (e.g., a function name), but at the same you're also +not so sure. + +In that case, using the advanced search syntax in your query will yield much +better results. + +## Using the Advanced Syntax Search + +The Advanced Syntax Search supports fuzzy or exact search queries with prefixes, +boolean operators, and much more. + +Full details can be found in the [Elasticsearch documentation][elastic], but +here's a quick guide: + +- Searches look for all the words in a query, in any order - e.g.: searching + issues for `display bug` will return all issues matching both those words, in any order. +- To find the exact phrase (stemming still applies), use double quotes: `"display bug"` +- To find bugs not mentioning display, use `-`: `bug -display` +- To find a bug in display or sound, use `|`: `bug display | sound` +- To group terms together, use parentheses: `bug | (display +sound)` +- To match a partial word, use `*`: `bug find_by_*` +- To find a term containing one of these symbols, use `\`: `argument \-last` + +### Syntax search filters + +The Advanced Syntax Search also supports the use of filters. The available filters are: + + - filename: Filters by filename. You can use the glob (`*`) operator for fuzzy matching. + - path: Filters by path. You can use the glob (`*`) operator for fuzzy matching. + - extension: Filters by extension in the filename. Please write the extension without a leading dot. Exact match only. + +To use them, simply add them to your query in the format `<filter_name>:<value>` without + any spaces between the colon (`:`) and the value. + +Examples: + +- Finding a file with any content named `hello_world.rb`: `* filename:hello_world.rb` +- Finding a file named `hello_world` with the text `whatever` inside of it: `whatever filename:hello_world` +- Finding the text 'def create' inside files with the `.rb` extension: `def create extension:rb` +- Finding the text `sha` inside files in a folder called `encryption`: `sha path:encryption` +- Finding any file starting with `hello` containing `world` and with the `.js` extension: `world filename:hello* extension:js` + +[ee]: https://about.gitlab.com/pricing/ +[elastic]: https://www.elastic.co/guide/en/elasticsearch/reference/5.3/query-dsl-simple-query-string-query.html#_simple_query_string_syntax diff --git a/doc/user/search/img/advanced_global_search.png b/doc/user/search/img/advanced_global_search.png Binary files differnew file mode 100644 index 00000000000..4903bbb07e1 --- /dev/null +++ b/doc/user/search/img/advanced_global_search.png diff --git a/doc/user/search/img/issues_any_assignee.png b/doc/user/search/img/issues_any_assignee.png Binary files differdeleted file mode 100644 index 2f902bcc66c..00000000000 --- a/doc/user/search/img/issues_any_assignee.png +++ /dev/null diff --git a/doc/user/search/img/issues_author.png b/doc/user/search/img/issues_author.png Binary files differdeleted file mode 100644 index 792f9746db6..00000000000 --- a/doc/user/search/img/issues_author.png +++ /dev/null diff --git a/doc/user/search/img/multiple_assignees.png b/doc/user/search/img/multiple_assignees.png Binary files differnew file mode 100644 index 00000000000..5c46f3dda46 --- /dev/null +++ b/doc/user/search/img/multiple_assignees.png diff --git a/doc/user/search/index.md b/doc/user/search/index.md index 770cef42995..bb6c48471c7 100644 --- a/doc/user/search/index.md +++ b/doc/user/search/index.md @@ -84,6 +84,12 @@ You can view recent searches by clicking on the little arrow-clock icon, which i Individual filters can be removed by clicking on the filter's (x) button or backspacing. The entire search filter can be cleared by clicking on the search box's (x) button. +## Filtering with multiple filters of the same type + +Some filters can be added multiple times. These include but are not limited to assignees and labels. When you filter with these multiple filters of the same type, the AND logic is applied. For example, if you were filtering `assignee:@sam assignee:@sarah`, your results will only include entries whereby the assignees are assigned to both Sam and Sarah are returned. + + + ### Shortcut You'll also find a shortcut on the search field on the top-right of the project's dashboard to @@ -96,7 +102,7 @@ quickly access issues and merge requests created or assigned to you within that Your [todos](../../workflow/todos.md#gitlab-todos) can be searched by "to do" and "done". You can [filter](../../workflow/todos.md#filtering-your-todos) them per project, author, type, and action. Also, you can sort them by -[**Label priority**](../../user/project/labels.md#prioritize-labels), +[**Label priority**](../../user/project/labels.md#label-priority), **Last created** and **Oldest created**. ## Projects @@ -136,3 +142,18 @@ you'll be able to, besides filtering them by **Name**, **Author**, **Assignee**, and **Labels**, select multiple issues to add to a list of your choice:  + +## Advanced Global Search **[STARTER]** + +Leverage Elasticsearch for faster, more advanced code search across your entire +GitLab instance. + +[Learn how to use the Advanced Global Search.](advanced_global_search.md) + +## Advanced Syntax Search **[STARTER]** + +Use advanced queries for more targeted search results. + +[Learn how to use the Advanced Syntax Search.](advanced_search_syntax.md) + +[ee]: https://about.gitlab.com/pricing/ diff --git a/doc/user/snippets.md b/doc/user/snippets.md index 569bdc9e2d5..7b580a057f2 100644 --- a/doc/user/snippets.md +++ b/doc/user/snippets.md @@ -20,7 +20,7 @@ and private. See [Public access](../public_access/public_access.md) for more inf ## Project snippets Project snippets are always related to a specific project. -See [Project's features](project/index.md#projects-features) for more information. +See [Project features](project/index.md#project-features) for more information. ## Discover snippets diff --git a/doc/web_hooks/web_hooks.md b/doc/web_hooks/web_hooks.md index 0ebe5eea173..fffb6a5d86d 100644 --- a/doc/web_hooks/web_hooks.md +++ b/doc/web_hooks/web_hooks.md @@ -1 +1,5 @@ +--- +redirect_to: '../user/project/integrations/webhooks.md' +--- + This document was moved to [project/integrations/webhooks](../user/project/integrations/webhooks.md). diff --git a/doc/workflow/README.md b/doc/workflow/README.md index 272f7807ac0..40e2486ace5 100644 --- a/doc/workflow/README.md +++ b/doc/workflow/README.md @@ -13,12 +13,15 @@ comments: false - [Groups](../user/group/index.md) - Issues - The GitLab Issue Tracker is an advanced and complete tool for tracking the evolution of a new idea or the process of solving a problem. + - [Exporting Issues](../user/project/issues/csv_export.md) **[STARTER]** Export issues as a CSV, emailed as an attachment. - [Confidential issues](../user/project/issues/confidential_issues.md) - [Due date for issues](../user/project/issues/due_dates.md) - [Issue Board](../user/project/issue_board.md) - [Keyboard shortcuts](shortcuts.md) - [File finder](file_finder.md) +- [File lock](../user/project/file_lock.md) **[PREMIUM]** - [Labels](../user/project/labels.md) +- [Issue weight](issue_weight.md) **[STARTER]** - [Notification emails](notifications.md) - [Projects](../user/project/index.md) - [Project forking workflow](forking_workflow.md) @@ -41,6 +44,9 @@ comments: false - [Merge requests versions](../user/project/merge_requests/versions.md) - ["Work In Progress" merge requests](../user/project/merge_requests/work_in_progress_merge_requests.md) - [Fast-forward merge requests](../user/project/merge_requests/fast_forward_merge.md) + - [Merge request approvals](../user/project/merge_requests/merge_request_approvals.md) **[STARTER]** +- [Repository mirroring](repository_mirroring.md) **[STARTER]** +- [Service Desk](../user/project/service_desk.md) **[PREMIUM]** - [Manage large binaries with Git LFS](lfs/manage_large_binaries_with_git_lfs.md) - [Importing from SVN, GitHub, Bitbucket, etc](importing/README.md) - [Todos](todos.md) diff --git a/doc/workflow/add-user/add-user.md b/doc/workflow/add-user/add-user.md index 35cc080d2b7..f1ec771dd9a 100644 --- a/doc/workflow/add-user/add-user.md +++ b/doc/workflow/add-user/add-user.md @@ -1 +1,5 @@ +--- +redirect_to: '../../user/project/members/index.md' +--- + This document was moved to [../../user/project/members/index.md](../../user/project/members/index.md) diff --git a/doc/workflow/authorization_for_merge_requests.md b/doc/workflow/authorization_for_merge_requests.md index 7bf80a3ad0d..8e43d340613 100644 --- a/doc/workflow/authorization_for_merge_requests.md +++ b/doc/workflow/authorization_for_merge_requests.md @@ -1 +1,5 @@ +--- +redirect_to: '../user/project/merge_requests/authorization_for_merge_requests.md' +--- + This document was moved to [user/project/merge_requests/authorization_for_merge_requests](../user/project/merge_requests/authorization_for_merge_requests.md) diff --git a/doc/workflow/award_emoji.md b/doc/workflow/award_emoji.md index d74378cc564..02db97b8dd6 100644 --- a/doc/workflow/award_emoji.md +++ b/doc/workflow/award_emoji.md @@ -1 +1,5 @@ +--- +redirect_to: '../user/award_emojis.md' +--- + This document was moved to [another location](../user/award_emojis.md). diff --git a/doc/workflow/award_emoji.png b/doc/workflow/award_emoji.png Binary files differdeleted file mode 100644 index 1ad634a343e..00000000000 --- a/doc/workflow/award_emoji.png +++ /dev/null diff --git a/doc/workflow/cherry_pick_changes.md b/doc/workflow/cherry_pick_changes.md index 663ffd3f746..29c4f854416 100644 --- a/doc/workflow/cherry_pick_changes.md +++ b/doc/workflow/cherry_pick_changes.md @@ -1 +1,5 @@ -This document was moved to [user/project/merge_requests/cherry_pick_changes](../user/project/merge_requests/cherry_pick_changes.md). +--- +redirect_to: '../user/project/merge_requests/cherry_pick_changes.md' +--- + +This document was moved to [another location](../user/project/merge_requests/cherry_pick_changes.md). diff --git a/doc/workflow/ff_merge.md b/doc/workflow/ff_merge.md new file mode 100644 index 00000000000..11e9e1bbd6b --- /dev/null +++ b/doc/workflow/ff_merge.md @@ -0,0 +1,5 @@ +--- +redirect_to: '../user/project/merge_requests/fast_forward_merge.md' +--- + +This document was moved to [user/project/merge_requests/fast_forward_merge](../user/project/merge_requests/fast_forward_merge.md). diff --git a/doc/workflow/git_annex.md b/doc/workflow/git_annex.md new file mode 100644 index 00000000000..84d25951908 --- /dev/null +++ b/doc/workflow/git_annex.md @@ -0,0 +1,238 @@ +# Git annex + +> **Warning:** GitLab has [completely +removed][deprecate-annex-issue] in GitLab 9.0 (2017/03/22). +Read through the [migration guide from git-annex to git-lfs][guide]. + +The biggest limitation of Git, compared to some older centralized version +control systems, has been the maximum size of the repositories. + +The general recommendation is to not have Git repositories larger than 1GB to +preserve performance. Although GitLab has no limit (some repositories in GitLab +are over 50GB!), we subscribe to the advice to keep repositories as small as +you can. + +Not being able to version control large binaries is a big problem for many +larger organizations. +Videos, photos, audio, compiled binaries and many other types of files are too +large. As a workaround, people keep artwork-in-progress in a Dropbox folder and +only check in the final result. This results in using outdated files, not +having a complete history and increases the risk of losing work. + +This problem is solved in GitLab Enterprise Edition by integrating the +[git-annex] application. + +`git-annex` allows managing large binaries with Git without checking the +contents into Git. +You check-in only a symlink that contains the SHA-1 of the large binary. If you +need the large binary, you can sync it from the GitLab server over `rsync`, a +very fast file copying tool. + +## GitLab git-annex Configuration + +`git-annex` is disabled by default in GitLab. Below you will find the +configuration options required to enable it. + +### Requirements + +`git-annex` needs to be installed both on the server and the client side. + +For Debian-like systems (e.g., Debian, Ubuntu) this can be achieved by running: + +``` +sudo apt-get update && sudo apt-get install git-annex +``` + +For RedHat-like systems (e.g., CentOS, RHEL) this can be achieved by running: + +``` +sudo yum install epel-release && sudo yum install git-annex +``` + +### Configuration for Omnibus packages + +For omnibus-gitlab packages, only one configuration setting is needed. +The Omnibus package will internally set the correct options in all locations. + +1. In `/etc/gitlab/gitlab.rb` add the following line: + + ```ruby + gitlab_shell['git_annex_enabled'] = true + ``` + +1. Save the file and [reconfigure GitLab][] for the changes to take effect. + +### Configuration for installations from source + +There are 2 settings to enable git-annex on your GitLab server. + +One is located in `config/gitlab.yml` of the GitLab repository and the other +one is located in `config.yml` of gitlab-shell. + +1. In `config/gitlab.yml` add or edit the following lines: + + ```yaml + gitlab_shell: + git_annex_enabled: true + ``` + +1. In `config.yml` of gitlab-shell add or edit the following lines: + + ```yaml + git_annex_enabled: true + ``` + +1. Save the files and [restart GitLab][] for the changes to take effect. + +## Using GitLab git-annex + +> **Note:** +> Your Git remotes must be using the SSH protocol, not HTTP(S). + +Here is an example workflow of uploading a very large file and then checking it +into your Git repository: + +```bash +git clone git@example.com:group/project.git + +git annex init 'My Laptop' # initialize the annex project and give an optional description +cp ~/tmp/debian.iso ./ # copy a large file into the current directory +git annex add debian.iso # add the large file to git annex +git commit -am "Add Debian iso" # commit the file metadata +git annex sync --content # sync the Git repo and large file to the GitLab server +``` + +The output should look like this: + +``` +commit +On branch master +Your branch is ahead of 'origin/master' by 1 commit. + (use "git push" to publish your local commits) +nothing to commit, working tree clean +ok +pull origin +remote: Counting objects: 5, done. +remote: Compressing objects: 100% (4/4), done. +remote: Total 5 (delta 2), reused 0 (delta 0) +Unpacking objects: 100% (5/5), done. +From example.com:group/project + 497842b..5162f80 git-annex -> origin/git-annex +ok +(merging origin/git-annex into git-annex...) +(recording state in git...) +copy debian.iso (checking origin...) (to origin...) +SHA256E-s26214400--8092b3d482fb1b7a5cf28c43bc1425c8f2d380e86869c0686c49aa7b0f086ab2.iso + 26,214,400 100% 638.88kB/s 0:00:40 (xfr#1, to-chk=0/1) +ok +pull origin +ok +(recording state in git...) +push origin +Counting objects: 15, done. +Delta compression using up to 4 threads. +Compressing objects: 100% (13/13), done. +Writing objects: 100% (15/15), 1.64 KiB | 0 bytes/s, done. +Total 15 (delta 1), reused 0 (delta 0) +To example.com:group/project.git + * [new branch] git-annex -> synced/git-annex + * [new branch] master -> synced/master +ok +``` + +Your files can be found in the `master` branch, but you'll notice that there +are more branches created by the `annex sync` command. + +Git Annex will also create a new directory at `.git/annex/` and will record the +tracked files in the `.git/config` file. The files you assign to be tracked +with `git-annex` will not affect the existing `.git/config` records. The files +are turned into symbolic links that point to data in `.git/annex/objects/`. + +The `debian.iso` file in the example will contain the symbolic link: + +``` +.git/annex/objects/ZW/1k/SHA256E-s82701--6384039733b5035b559efd5a2e25a493ab6e09aabfd5162cc03f6f0ec238429d.png/SHA256E-s82701--6384039733b5035b559efd5a2e25a493ab6e09aabfd5162cc03f6f0ec238429d.iso +``` + +Use `git annex info` to retrieve the information about the local copy of your +repository. + +--- + +Downloading a single large file is also very simple: + +```bash +git clone git@gitlab.example.com:group/project.git + +git annex sync # sync Git branches but not the large file +git annex get debian.iso # download the large file +``` + +To download all files: + +```bash +git clone git@gitlab.example.com:group/project.git + +git annex sync --content # sync Git branches and download all the large files +``` + +By using `git-annex` without GitLab, anyone that can access the server can also +access the files of all projects, but GitLab Annex ensures that you can only +access files of projects you have access to (developer, maintainer, or owner role). + +## How it works + +Internally GitLab uses [GitLab Shell] to handle SSH access and this was a great +integration point for `git-annex`. +There is a setting in gitlab-shell so you can disable GitLab Annex support +if you want to. + +## Troubleshooting tips + +Differences in version of `git-annex` on the GitLab server and on local machines +can cause `git-annex` to raise unpredicted warnings and errors. + +Consult the [Annex upgrade page][annex-upgrade] for more information about +the differences between versions. You can find out which version is installed +on your server by navigating to <https://pkgs.org/download/git-annex> and +searching for your distribution. + +Although there is no general guide for `git-annex` errors, there are a few tips +on how to go around the warnings. + +### git-annex-shell: Not a git-annex or gcrypt repository. + +This warning can appear on the initial `git annex sync --content` and is caused +by differences in `git-annex-shell`. You can read more about it +[in this git-annex issue][issue]. + +One important thing to note is that despite the warning, the `sync` succeeds +and the files are pushed to the GitLab repository. + +If you get hit by this, you can run the following command inside the repository +that the warning was raised: + +``` +git config remote.origin.annex-ignore false +``` + +Consecutive runs of `git annex sync --content` **should not** produce this +warning and the output should look like this: + +``` +commit ok +pull origin +ok +pull origin +ok +push origin +``` + +[annex-upgrade]: https://git-annex.branchable.com/upgrades/ +[deprecate-annex-issue]: https://gitlab.com/gitlab-org/gitlab-ee/issues/1648 +[git-annex]: https://git-annex.branchable.com/ "git-annex website" +[gitlab shell]: https://gitlab.com/gitlab-org/gitlab-shell "GitLab Shell repository" +[guide]: lfs/migrate_from_git_annex_to_git_lfs.html +[issue]: https://git-annex.branchable.com/forum/Error_from_git-annex-shell_on_creation_of_gcrypt_special_remote/ "git-annex issue" +[reconfigure GitLab]: ../administration/restart_gitlab.md#omnibus-gitlab-reconfigure +[restart GitLab]: ../administration/restart_gitlab.md#installations-from-source diff --git a/doc/workflow/git_lfs.md b/doc/workflow/git_lfs.md new file mode 100644 index 00000000000..da217b0a5da --- /dev/null +++ b/doc/workflow/git_lfs.md @@ -0,0 +1,5 @@ +--- +redirect_to: 'lfs/manage_large_binaries_with_git_lfs.md' +--- + +This document was moved to [another location](lfs/manage_large_binaries_with_git_lfs.md). diff --git a/doc/workflow/gitlab_flow.md b/doc/workflow/gitlab_flow.md index 1b9fb504b15..7d0abb93262 100644 --- a/doc/workflow/gitlab_flow.md +++ b/doc/workflow/gitlab_flow.md @@ -166,7 +166,7 @@ This branch is the place for any work related to this change. NOTE: **Note:** The name of a branch might be dictated by organizational standards. -For example, in GitLab, any branches in GitLab EE that are equivalent to branches in GitLab CE [must end in `-ee`](https://docs.gitlab.com/ee/development/automatic_ce_ee_merge.html#cherry-picking-from-ce-to-ee). +For example, in GitLab, any branches in GitLab EE that are equivalent to branches in GitLab CE [must end in `-ee`](../development/automatic_ce_ee_merge.md#cherry-picking-from-ce-to-ee). When you are done or want to discuss the code, open a merge request. A merge request is an online place to discuss the change and review the code. diff --git a/doc/workflow/groups.md b/doc/workflow/groups.md index 06eec1ed928..c7f4647baa9 100644 --- a/doc/workflow/groups.md +++ b/doc/workflow/groups.md @@ -1,2 +1,5 @@ +--- +redirect_to: '../user/group/index.md' +--- This document was moved to [another location](../user/group/index.md). diff --git a/doc/workflow/img/copy_ssh_public_key_button.png b/doc/workflow/img/copy_ssh_public_key_button.png Binary files differnew file mode 100644 index 00000000000..e20dae09a4d --- /dev/null +++ b/doc/workflow/img/copy_ssh_public_key_button.png diff --git a/doc/workflow/img/new_branch_from_issue.png b/doc/workflow/img/new_branch_from_issue.png Binary files differdeleted file mode 100644 index 286d775bb9e..00000000000 --- a/doc/workflow/img/new_branch_from_issue.png +++ /dev/null diff --git a/doc/workflow/img/notification_global_settings.png b/doc/workflow/img/notification_global_settings.png Binary files differindex 8a5494d16a8..72f7418f1f8 100644 --- a/doc/workflow/img/notification_global_settings.png +++ b/doc/workflow/img/notification_global_settings.png diff --git a/doc/workflow/img/repository_mirroring_pull_settings_upper.png b/doc/workflow/img/repository_mirroring_pull_settings_upper.png Binary files differindex c60354fdca7..8e15b5a0784 100644 --- a/doc/workflow/img/repository_mirroring_pull_settings_upper.png +++ b/doc/workflow/img/repository_mirroring_pull_settings_upper.png diff --git a/doc/workflow/importing/README.md b/doc/workflow/importing/README.md index e0a445920b4..29da321ba46 100644 --- a/doc/workflow/importing/README.md +++ b/doc/workflow/importing/README.md @@ -1 +1,5 @@ +---
+redirect_to: '../../user/project/import/index.md'
+---
+
This document was moved to [another location](../../user/project/import/index.md).
diff --git a/doc/workflow/importing/import_projects_from_bitbucket.md b/doc/workflow/importing/import_projects_from_bitbucket.md index ec9a11f390e..a42ba7d4518 100644 --- a/doc/workflow/importing/import_projects_from_bitbucket.md +++ b/doc/workflow/importing/import_projects_from_bitbucket.md @@ -1 +1,5 @@ +---
+redirect_to: '../../user/project/import/bitbucket.md'
+---
+
This document was moved to [another location](../../user/project/import/bitbucket.md).
diff --git a/doc/workflow/importing/import_projects_from_fogbugz.md b/doc/workflow/importing/import_projects_from_fogbugz.md index 876eb0434f0..f5c791dc6de 100644 --- a/doc/workflow/importing/import_projects_from_fogbugz.md +++ b/doc/workflow/importing/import_projects_from_fogbugz.md @@ -1 +1,5 @@ +--- +redirect_to: '../../user/project/import/fogbugz.md' +--- + This document was moved to [another location](../../user/project/import/fogbugz.md). diff --git a/doc/workflow/importing/import_projects_from_gitea.md b/doc/workflow/importing/import_projects_from_gitea.md index 8b55b6c23eb..df053835b44 100644 --- a/doc/workflow/importing/import_projects_from_gitea.md +++ b/doc/workflow/importing/import_projects_from_gitea.md @@ -1 +1,5 @@ +--- +redirect_to: '../../user/project/import/gitea.md' +--- + This document was moved to [another location](../../user/project/import/gitea.md). diff --git a/doc/workflow/importing/import_projects_from_github.md b/doc/workflow/importing/import_projects_from_github.md index 72dfe5403c3..6397fcc74b8 100644 --- a/doc/workflow/importing/import_projects_from_github.md +++ b/doc/workflow/importing/import_projects_from_github.md @@ -1 +1,5 @@ +---
+redirect_to: '../../user/project/import/github.md'
+---
+
This document was moved to [another location](../../user/project/import/github.md).
diff --git a/doc/workflow/importing/import_projects_from_gitlab_com.md b/doc/workflow/importing/import_projects_from_gitlab_com.md index 3256088c014..135b9704df9 100644 --- a/doc/workflow/importing/import_projects_from_gitlab_com.md +++ b/doc/workflow/importing/import_projects_from_gitlab_com.md @@ -1 +1,5 @@ +--- +redirect_to: '../../user/project/import/gitlab_com.md' +--- + This document was moved to [another location](../../user/project/import/gitlab_com.md). diff --git a/doc/workflow/importing/migrating_from_svn.md b/doc/workflow/importing/migrating_from_svn.md index 32a75a6c6af..99f13d6354c 100644 --- a/doc/workflow/importing/migrating_from_svn.md +++ b/doc/workflow/importing/migrating_from_svn.md @@ -1 +1,5 @@ +--- +redirect_to: '../../user/project/import/svn.md' +--- + This document was moved to [another location](../../user/project/import/svn.md). diff --git a/doc/workflow/issue_weight.md b/doc/workflow/issue_weight.md new file mode 100644 index 00000000000..267160dae2a --- /dev/null +++ b/doc/workflow/issue_weight.md @@ -0,0 +1,22 @@ +# Issue Weight **[STARTER]** + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/76) +> in [GitLab Starter](https://about.gitlab.com/pricing/) 8.3. + +When you have a lot of issues, it can be hard to get an overview. +By adding a weight to each issue, you can get a better idea of how much time, +value or complexity a given issue has or will cost. + +You can set the weight of an issue during its creation, by simply changing the +value in the dropdown menu. You can set it to a non-negative integer +value from 0, 1, 2, and so on. (The database stores a 4-byte value, so the +upper bound is essentially limitless). +You can remove weight from an issue +as well. + +This value will appear on the right sidebar of an individual issue, as well as +in the issues page next to a distinctive balance scale icon. + +As an added bonus, you can see the total sum of all issues on the milestone page. + + diff --git a/doc/workflow/issue_weight/issue.png b/doc/workflow/issue_weight/issue.png Binary files differnew file mode 100644 index 00000000000..3800b5940b8 --- /dev/null +++ b/doc/workflow/issue_weight/issue.png diff --git a/doc/workflow/labels.md b/doc/workflow/labels.md index 5c09891dfdd..3d07d411dd4 100644 --- a/doc/workflow/labels.md +++ b/doc/workflow/labels.md @@ -1,3 +1,7 @@ +--- +redirect_to: '../user/project/labels.md' +--- + # Labels This document was moved to [user/project/labels.md](../user/project/labels.md). diff --git a/doc/workflow/lfs/images/git-annex-branches.png b/doc/workflow/lfs/images/git-annex-branches.png Binary files differnew file mode 100644 index 00000000000..3d614f68177 --- /dev/null +++ b/doc/workflow/lfs/images/git-annex-branches.png diff --git a/doc/workflow/lfs/lfs_administration.md b/doc/workflow/lfs/lfs_administration.md index 3fb553280d0..6d941135bf2 100644 --- a/doc/workflow/lfs/lfs_administration.md +++ b/doc/workflow/lfs/lfs_administration.md @@ -108,6 +108,18 @@ Here is a configuration example with GCS. _NOTE: The service account must have permission to access the bucket. [See more](https://cloud.google.com/storage/docs/authentication)_ +Here is a configuration example with Rackspace Cloud Files. + +| Setting | Description | example | +|---------|-------------|---------| +| `provider` | The provider name | `Rackspace` | +| `rackspace_username` | The username of the Rackspace account with access to the container | `joe.smith` | +| `rackspace_api_key` | The API key of the Rackspace account with access to the container | `ABC123DEF456ABC123DEF456ABC123DE` | +| `rackspace_region` | The Rackspace storage region to use, a three letter code from the [list of service access endpoints](https://developer.rackspace.com/docs/cloud-files/v1/general-api-info/service-access/) | `iad` | +| `rackspace_temp_url_key` | The private key you have set in the Rackspace API for temporary URLs. Read more [here](https://developer.rackspace.com/docs/cloud-files/v1/use-cases/public-access-to-your-cloud-files-account/#tempurl) | `ABC123DEF456ABC123DEF456ABC123DE` | + +_NOTES: Regardless of whether the container has public access enabled or disabled, Fog will use the TempURL method to grant access to LFS objects. If you see errors in logs referencing instantiating storage with a temp-url-key, ensure that you have set they key properly on the Rackspace API and in gitlab.rb. You can verify the value of the key Rackspace has set by sending a GET request with token header to the service access endpoint URL and comparing the output of the returned headers._ + ### Manual uploading to an object storage There are two ways to manually do the same thing as automatic uploading (described above). diff --git a/doc/workflow/lfs/manage_large_binaries_with_git_lfs.md b/doc/workflow/lfs/manage_large_binaries_with_git_lfs.md index da0243705aa..202f2e39975 100644 --- a/doc/workflow/lfs/manage_large_binaries_with_git_lfs.md +++ b/doc/workflow/lfs/manage_large_binaries_with_git_lfs.md @@ -250,6 +250,7 @@ If you are storing LFS files outside of GitLab you can disable LFS on the projec It is possible to host LFS objects externally by setting a custom LFS url with `git config -f .lfsconfig lfs.url https://example.com/<project>.git/info/lfs`. -Because GitLab verifies the existence of objects referenced by LFS pointers, push will fail when LFS is enabled for the project. +You might choose to do this if you are using an appliance like a Sonatype Nexus to store LFS data. If you choose to use an external LFS store, +GitLab will not be able to verify LFS objects which means that pushes will fail if you have GitLab LFS support enabled. -LFS can be disabled from the [Project settings](../../user/project/settings/index.md). +To stop push failure, LFS support can be disabled in the [Project settings](../../user/project/settings/index.md). This means you will lose GitLab LFS value-adds (Verifying LFS objects, UI integration for LFS). diff --git a/doc/workflow/lfs/migrate_from_git_annex_to_git_lfs.md b/doc/workflow/lfs/migrate_from_git_annex_to_git_lfs.md new file mode 100644 index 00000000000..71c73e3dffe --- /dev/null +++ b/doc/workflow/lfs/migrate_from_git_annex_to_git_lfs.md @@ -0,0 +1,255 @@ +# Migration guide from Git Annex to Git LFS + +>**Note:** +Git Annex support [has been removed][issue-remove-annex] in GitLab Enterprise +Edition 9.0 (2017/03/22). + +Both [Git Annex][] and [Git LFS][] are tools to manage large files in Git. + +## History + +Git Annex [was introduced in GitLab Enterprise Edition 7.8][post-3], at a time +where Git LFS didn't yet exist. A few months later, GitLab brought support for +Git LFS in [GitLab 8.2][post-2] and is available for both Community and +Enterprise editions. + +## Differences between Git Annex and Git LFS + +Some items below are general differences between the two protocols and some are +ones that GitLab developed. + +- Git Annex works only through SSH, whereas Git LFS works both with SSH and HTTPS + (SSH support was added in GitLab 8.12). +- Annex files are stored in a sub-directory of the normal repositories, whereas + LFS files are stored outside of the repositories in a place you can define. +- Git Annex requires a more complex setup, but has much more options than Git + LFS. You can compare the commands each one offers by running `man git-annex` + and `man git-lfs`. +- Annex files cannot be browsed directly in GitLab's interface, whereas LFS + files can. + +## Migration steps + +>**Note:** +Since Git Annex files are stored in a sub-directory of the normal repositories +(`.git/annex/objects`) and LFS files are stored outside of the repositories, +they are not compatible as they are using a different scheme. Therefore, the +migration has to be done manually per repository. + +There are basically two steps you need to take in order to migrate from Git +Annex to Git LFS. + +### TL; DR + +If you know what you are doing and want to skip the reading, this is what you +need to do (we assume you have [git-annex enabled](../git_annex.md#using-gitlab-git-annex) in your +repository and that you have made backups in case something goes wrong). +Fire up a terminal, navigate to your Git repository and: + + +1. Disable `git-annex`: + + ```bash + git annex sync --content + git annex direct + git annex uninit + git annex indirect + ``` + +1. Enable `git-lfs`: + + ``` + git lfs install + git lfs track <files> + git add . + git commit -m "commit message" + git push + ``` + +### Disabling Git Annex in your repo + +Before changing anything, make sure you have a backup of your repository first. +There are a couple of ways to do that, but you can simply clone it to another +local path and maybe push it to GitLab if you want a remote backup as well. +Here you'll find a guide on +[how to back up a **git-annex** repository to an external hard drive][bkp-ext-drive]. + +Since Annex files are stored as objects with symlinks and cannot be directly +modified, we need to first remove those symlinks. + +>**Note:** +Make sure the you read about the [`direct` mode][annex-direct] as it contains +useful information that may fit in your use case. Note that `annex direct` is +deprecated in Git Annex version 6, so you may need to upgrade your repository +if the server also has Git Annex 6 installed. Read more in the +[Git Annex troubleshooting tips](../git_annex.md#troubleshooting-tips) section. + +1. Backup your repository + + ```bash + cd repository + git annex sync --content + cd .. + git clone repository repository-backup + cd repository-backup + git annex get + cd .. + ``` + +1. Use `annex direct`: + + ```bash + cd repository + git annex direct + ``` + + The output should be similar to this: + + ```bash + commit + On branch master + Your branch is up-to-date with 'origin/master'. + nothing to commit, working tree clean + ok + direct debian.iso ok + direct ok + ``` + +1. Disable Git Annex with [`annex uninit`][uninit]: + + ```bash + git annex uninit + ``` + + The output should be similar to this: + + ```bash + unannex debian.iso ok + Deleted branch git-annex (was 2534d2c). + ``` + + This will `unannex` every file in the repository, leaving the original files. + +1. Switch back to `indirect` mode: + + ```bash + git annex indirect + ``` + + The output should be similar to this: + + ```bash + (merging origin/git-annex into git-annex...) + (recording state in git...) + commit (recording state in git...) + + ok + (recording state in git...) + [master fac3194] commit before switching to indirect mode + 1 file changed, 1 deletion(-) + delete mode 120000 alpine-virt-3.4.4-x86_64.iso + ok + indirect ok + ok + ``` + +--- + +At this point, you have two options. Either add, commit and push the files +directly back to GitLab or switch to Git LFS. We will tackle the LFS switch in +the next section. + +### Enabling Git LFS in your repo + +Git LFS is enabled by default on all GitLab products (GitLab CE, GitLab EE, +GitLab.com), therefore, you don't need to do anything server-side. + +1. First, make sure you have `git-lfs` installed locally: + + ```bash + git lfs help + ``` + + If the terminal doesn't prompt you with a full response on `git-lfs` commands, + [install the Git LFS client][install-lfs] first. + +1. Inside the repo, run the following command to initiate LFS: + + ```bash + git lfs install + ``` + +1. Enable `git-lfs` for the group of files you want to track. You + can track specific files, all files containing the same extension, or an + entire directory: + + ```bash + git lfs track images/01.png # per file + git lfs track **/*.png # per extension + git lfs track images/ # per directory + ``` + + Once you do that, run `git status` and you'll see `.gitattributes` added + to your repo. It collects all file patterns that you chose to track via + `git-lfs`. + +1. Add the files, commit and push them to GitLab: + + ```bash + git add . + git commit -m "commit message" + git push + ``` + + If your remote is set up with HTTP, you will be asked to enter your login + credentials. If you have [2FA enabled](../../user/profile/account/two_factor_authentication.md), make sure to use a + [personal access token](../../user/profile/account/two_factor_authentication.md#personal-access-tokens) + instead of your password. + +## Removing the Git Annex branches + +After the migration finishes successfully, you can remove all `git-annex` +related branches from your repository. + +On GitLab, navigate to your project's **Repository ➔ Branches** and delete all +branches created by Git Annex: `git-annex`, and all under `synced/`. + + + +You can also do this on the commandline with: + + ```bash + git branch -d synced/master + git branch -d synced/git-annex + git push origin :synced/master + git push origin :synced/git-annex + git push origin :git-annex + git remote prune origin + ``` + +If there are still some Annex objects inside your repository (`.git/annex/`) +or references inside `.git/config`, run `annex uninit` again: + +```bash +git annex uninit +``` + +## Further Reading + +- (Blog Post) [Getting Started with Git FLS][post-1] +- (Blog Post) [Announcing LFS Support in GitLab][post-2] +- (Blog Post) [GitLab Annex Solves the Problem of Versioning Large Binaries with Git][post-3] +- (GitLab Docs) [Git Annex](../git_annex.md) +- (GitLab Docs) [Git LFS](manage_large_binaries_with_git_lfs.md) + +[annex-direct]: https://git-annex.branchable.com/direct_mode/ +[bkp-ext-drive]: https://www.thomas-krenn.com/en/wiki/Git-annex_Repository_on_an_External_Hard_Drive +[Git Annex]: http://git-annex.branchable.com/ +[Git LFS]: https://git-lfs.github.com/ +[install-lfs]: https://git-lfs.github.com/ +[issue-remove-annex]: https://gitlab.com/gitlab-org/gitlab-ee/issues/1648 +[lfs-track]: https://about.gitlab.com/2017/01/30/getting-started-with-git-lfs-tutorial/#tracking-files-with-lfs +[post-1]: https://about.gitlab.com/2017/01/30/getting-started-with-git-lfs-tutorial/ +[post-2]: https://about.gitlab.com/2015/11/23/announcing-git-lfs-support-in-gitlab/ +[post-3]: https://about.gitlab.com/2015/02/17/gitlab-annex-solves-the-problem-of-versioning-large-binaries-with-git/ +[uninit]: https://git-annex.branchable.com/git-annex-uninit/ diff --git a/doc/workflow/merge_request_approvals.md b/doc/workflow/merge_request_approvals.md new file mode 100644 index 00000000000..bfcd8faf236 --- /dev/null +++ b/doc/workflow/merge_request_approvals.md @@ -0,0 +1,5 @@ +--- +redirect_to: '../user/project/merge_requests/merge_request_approvals.md' +--- + +This document was moved to [another location](../user/project/merge_requests/merge_request_approvals.md). diff --git a/doc/workflow/merge_requests.md b/doc/workflow/merge_requests.md index dc6da1938f3..fd9f9b81bc9 100644 --- a/doc/workflow/merge_requests.md +++ b/doc/workflow/merge_requests.md @@ -1 +1,5 @@ +--- +redirect_to: '../user/project/merge_requests/index.md' +--- + This document was moved to [user/project/merge_requests/index.md](../user/project/merge_requests/index.md). diff --git a/doc/workflow/merge_when_build_succeeds.md b/doc/workflow/merge_when_build_succeeds.md index b4f6d6117de..41e6ff0cdd6 100644 --- a/doc/workflow/merge_when_build_succeeds.md +++ b/doc/workflow/merge_when_build_succeeds.md @@ -1 +1,5 @@ +--- +redirect_to: '../user/project/merge_requests/merge_when_pipeline_succeeds.md' +--- + This document was moved to [merge_when_pipeline_succeeds](../user/project/merge_requests/merge_when_pipeline_succeeds.md). diff --git a/doc/workflow/milestones.md b/doc/workflow/milestones.md index 69eb6b286b0..18dc15f7327 100644 --- a/doc/workflow/milestones.md +++ b/doc/workflow/milestones.md @@ -1 +1,5 @@ +--- +redirect_to: '../user/project/milestones/index.md' +--- + This document was moved to [another location](../user/project/milestones/index.md). diff --git a/doc/workflow/notifications.md b/doc/workflow/notifications.md index 6ce789998a4..5d560f2e000 100644 --- a/doc/workflow/notifications.md +++ b/doc/workflow/notifications.md @@ -16,12 +16,16 @@ Notification settings are divided into three groups: Each of these settings have levels of notification: +- Global: For groups and projects, notifications as per global settings. - Watch: Receive notifications for any activity. -- On Mention: Receive notifications when `@mentioned` in comments. - Participate: Receive notifications for threads you have participated in. +- On Mention: Receive notifications when `@mentioned` in comments. - Disabled: Turns off notifications. - Custom: Receive notifications for custom selected events. -- Global: For groups and projects, notifications as per global settings. + +> Introduced in GitLab 12.0 + +You can also select an email address to receive notifications for each group you belong to. ### Global Settings @@ -73,40 +77,43 @@ Below is the table of events users can be notified of: | Group access level changed | User | Sent when user group access level is changed | | Project moved | Project members [1] | [1] not disabled | -### Issue / Merge request events +### Issue / Epics / Merge request events In most of the below cases, the notification will be sent to: - Participants: - the author and assignee of the issue/merge request - authors of comments on the issue/merge request - - anyone mentioned by `@username` in the issue/merge request title or description - - anyone mentioned by `@username` in any of the comments on the issue/merge request - ...with notification level "Participating" or higher + - anyone mentioned by `@username` in the title or description of the issue, merge request or epic **[ULTIMATE]** + - anyone with notification level "Participating" or higher that is mentioned by `@username` + in any of the comments on the issue, merge request, or epic **[ULTIMATE]** - Watchers: users with notification level "Watch" -- Subscribers: anyone who manually subscribed to the issue/merge request +- Subscribers: anyone who manually subscribed to the issue, merge request, or epic **[ULTIMATE]** - Custom: Users with notification level "custom" who turned on notifications for any of the events present in the table below | Event | Sent to | |------------------------|---------| -| New issue | | -| Close issue | | +| New issue | | +| Close issue | | | Reassign issue | The above, plus the old assignee | -| Reopen issue | | +| Reopen issue | | | Due issue | Participants and Custom notification level with this event selected | | Change milestone issue | Subscribers, participants mentioned, and Custom notification level with this event selected | | Remove milestone issue | Subscribers, participants mentioned, and Custom notification level with this event selected | -| New merge request | | +| New merge request | | | Push to merge request | Participants and Custom notification level with this event selected | | Reassign merge request | The above, plus the old assignee | -| Close merge request | | -| Reopen merge request | | -| Merge merge request | | +| Close merge request | | +| Reopen merge request | | +| Merge merge request | | | Change milestone merge request | Subscribers, participants mentioned, and Custom notification level with this event selected | | Remove milestone merge request | Subscribers, participants mentioned, and Custom notification level with this event selected | | New comment | The above, plus anyone mentioned by `@username` in the comment, with notification level "Mention" or higher | | Failed pipeline | The author of the pipeline | | Successful pipeline | The author of the pipeline, if they have the custom notification setting for successful pipelines set | +| New epic **[ULTIMATE]** | | +| Close epic **[ULTIMATE]** | | +| Reopen epic **[ULTIMATE]** | | In addition, if the title or description of an Issue or Merge Request is changed, notifications will be sent to any **new** mentions by `@username` as diff --git a/doc/workflow/project_features.md b/doc/workflow/project_features.md index feb88712f5a..f54afb768a1 100644 --- a/doc/workflow/project_features.md +++ b/doc/workflow/project_features.md @@ -1 +1,5 @@ +--- +redirect_to: '../user/project/index.md' +--- + This document was moved to [../user/project/index.md](../user/project/index.md) diff --git a/doc/workflow/protected_branches.md b/doc/workflow/protected_branches.md index ced7d391ace..1bcac4a2de5 100644 --- a/doc/workflow/protected_branches.md +++ b/doc/workflow/protected_branches.md @@ -1 +1,5 @@ +--- +redirect_to: '../user/project/protected_branches.md' +--- + This document was moved to [another location](../user/project/protected_branches.md). diff --git a/doc/workflow/rebase_before_merge.md b/doc/workflow/rebase_before_merge.md new file mode 100644 index 00000000000..10e768d3371 --- /dev/null +++ b/doc/workflow/rebase_before_merge.md @@ -0,0 +1,5 @@ +--- +redirect_to: '../user/project/merge_requests/fast_forward_merge.md' +--- + +This document was moved to [another location](../user/project/merge_requests/fast_forward_merge.md). diff --git a/doc/workflow/repository_mirroring.md b/doc/workflow/repository_mirroring.md index 8a2f4e1b40e..9772bd385ba 100644 --- a/doc/workflow/repository_mirroring.md +++ b/doc/workflow/repository_mirroring.md @@ -84,7 +84,7 @@ To set up a mirror from GitLab to GitHub, you need to follow these steps: 1. Fill in **Password** field with your GitHub personal access token. 1. Click the **Mirror repository** button. -The mirrored repository will be listed. For example, `https://*****:*****@github.com/<your_github_group>/<your_github_project>.git`. +The mirrored repository will be listed. For example, `https://*****:*****@github.com/<your_github_group>/<your_github_project>.git`. The repository will push soon. To force a push, click the appropriate button. @@ -99,6 +99,7 @@ The repository will push soon. To force a push, click the appropriate button. ## Pulling from a remote repository **[STARTER]** > [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/51) in GitLab Enterprise Edition 8.2. +> [Added Git LFS support](https://gitlab.com/gitlab-org/gitlab-ee/issues/10871) in [GitLab Starter](https://about.gitlab.com/pricing/) 11.11. You can set up a repository to automatically have its branches, tags, and commits updated from an upstream repository. @@ -138,13 +139,18 @@ upstream and GitLab will no longer automatically update this branch to prevent a ### How it works -Once you activate the pull mirroring feature, the mirror will be inserted into a queue. A scheduler -will start every minute and schedule a fixed number of mirrors for update, based on the configured maximum capacity. +Once the pull mirroring feature has been enabled for a repository, the repository is added to a queue. -If the mirror updates successfully, it will be enqueued once again with a small backoff period. +Once per minute, a Sidekiq cron job schedules repository mirrors to update, based on: -If the mirror fails (for example, a branch diverged from upstream), the project's backoff period is -increased each time it fails, up to a maximum amount of time. +- The capacity available. This is determined by Sidekiq settings. For GitLab.com, see [GitLab.com Sidekiq settings](../user/gitlab_com/index.md#sidekiq). +- The number of repository mirrors already in the queue that are due to be updated. Being due depends on when the repository mirror was last updated and how many times it's been retried. + +Repository mirrors are updated as Sidekiq becomes available to process them. If the process of updating the repository mirror: + +- Succeeds, an update will be enqueued again with at least a 30 minute wait. +- Fails (for example, a branch diverged from upstream), it will be attempted again later. Mirrors can fail + up to 14 times before they will not be enqueued for update again. ### SSH authentication @@ -217,8 +223,10 @@ being injected into your mirror, or your password being stolen. ### SSH public key authentication To use SSH public key authentication, you'll also need to choose that option -from the **Authentication method** dropdown. GitLab will generate a 4096-bit RSA -key and display the public component of that key to you. +from the **Authentication method** dropdown. When the mirror is created, +GitLab generates a 4096-bit RSA key that can be copied by clicking the **Copy SSH public key** button. + + You then need to add the public SSH key to the other repository's configuration: @@ -275,10 +283,10 @@ project mirroring again by [Forcing an update](#forcing-an-update-core). [GitLab Starter](https://about.gitlab.com/pricing/) 10.3. Pull mirroring uses polling to detect new branches and commits added upstream, often minutes -afterwards. If you notify GitLab by [API](https://docs.gitlab.com/ee/api/projects.html#start-the-pull-mirroring-process-for-a-project), +afterwards. If you notify GitLab by [API](https://docs.gitlab.com/ee/api/projects.html#start-the-pull-mirroring-process-for-a-project-starter), updates will be pulled immediately. -For more information, see [Start the pull mirroring process for a Project](https://docs.gitlab.com/ee/api/projects.html#start-the-pull-mirroring-process-for-a-project). +For more information, see [Start the pull mirroring process for a Project](https://docs.gitlab.com/ee/api/projects.html#start-the-pull-mirroring-process-for-a-project-starter). ## Forcing an update **[CORE]** @@ -408,3 +416,11 @@ settings are recommended: Perforce Helix. Read about [Git Fusion settings on Perforce.com](https://www.perforce.com/perforce/doc.current/manuals/git-fusion/Content/Git-Fusion/section_vss_bdw_w3.html#section_zdp_zz1_3l). + +## Troubleshooting + +Should an error occur during a push, GitLab will display an "Error" highlight for that repository. Details on the error can then be seen by hovering over the highlight text. + +### 13:Received RST_STREAM with error code 2 with GitHub + +If you receive an "13:Received RST_STREAM with error code 2" while mirroring to a GitHub repository, your GitHub settings might be set to block pushes that expose your email address used in commits. Either set your email address on GitHub to be public, or disable the [Block command line pushes that expose my email](http://github.com/settings/emails) setting. diff --git a/doc/workflow/revert_changes.md b/doc/workflow/revert_changes.md index cf1292253fc..15f199af703 100644 --- a/doc/workflow/revert_changes.md +++ b/doc/workflow/revert_changes.md @@ -1 +1,5 @@ +--- +redirect_to: '../user/project/merge_requests/revert_changes.md' +--- + This document was moved to [user/project/merge_requests/revert_changes](../user/project/merge_requests/revert_changes.md). diff --git a/doc/workflow/share_projects_with_other_groups.md b/doc/workflow/share_projects_with_other_groups.md index 2eb4d24958a..c39cd78f32d 100644 --- a/doc/workflow/share_projects_with_other_groups.md +++ b/doc/workflow/share_projects_with_other_groups.md @@ -1 +1,5 @@ +--- +redirect_to: '../user/project/members/share_project_with_groups.md' +--- + This document was moved to [../user/project/members/share_project_with_groups.md](../user/project/members/share_project_with_groups.md) diff --git a/doc/workflow/share_with_group.md b/doc/workflow/share_with_group.md index 2eb4d24958a..c39cd78f32d 100644 --- a/doc/workflow/share_with_group.md +++ b/doc/workflow/share_with_group.md @@ -1 +1,5 @@ +--- +redirect_to: '../user/project/members/share_project_with_groups.md' +--- + This document was moved to [../user/project/members/share_project_with_groups.md](../user/project/members/share_project_with_groups.md) diff --git a/doc/workflow/share_with_group.png b/doc/workflow/share_with_group.png Binary files differdeleted file mode 100644 index 2c47625e29a..00000000000 --- a/doc/workflow/share_with_group.png +++ /dev/null diff --git a/doc/workflow/shortcuts.md b/doc/workflow/shortcuts.md index 7359e1c6119..5068b5d4d20 100644 --- a/doc/workflow/shortcuts.md +++ b/doc/workflow/shortcuts.md @@ -1,6 +1,6 @@ # GitLab keyboard shortcuts -You can see GitLab's keyboard shortcuts by using 'shift + ?' +You can see GitLab's keyboard shortcuts by using <kbd>shift</kbd> + <kbd>?</kbd> ## Global Shortcuts @@ -82,6 +82,16 @@ You can see GitLab's keyboard shortcuts by using 'shift + ?' | <kbd>r</kbd> | Reply (quoting selected text) | | <kbd>e</kbd> | Edit issue/merge request | | <kbd>l</kbd> | Change label | +| <kbd>]</kbd> or <kbd>j</kbd> | Move to next file | +| <kbd>[</kbd> or <kbd>k</kbd> | Move to previous file | + +## Epics **[ULTIMATE]** + +| Keyboard Shortcut | Description | +| ----------------- | ----------- | +| <kbd>r</kbd> | Reply (quoting selected text) | +| <kbd>e</kbd> | Edit description | +| <kbd>l</kbd> | Change label | ## Wiki pages diff --git a/doc/workflow/time_tracking.md b/doc/workflow/time_tracking.md index e60b6819bf1..c03dffa967d 100644 --- a/doc/workflow/time_tracking.md +++ b/doc/workflow/time_tracking.md @@ -75,6 +75,6 @@ Default conversion rates are 1mo = 4w, 1w = 5d and 1d = 8h. Other interesting links: -- [Time Tracking landing page on about.gitlab.com](https://about.gitlab.com/features/time-tracking) +- [Time Tracking landing page on about.gitlab.com](https://about.gitlab.com/solutions/time-tracking/) [quick actions]: ../user/project/quick_actions.md diff --git a/doc/workflow/timezone.md b/doc/workflow/timezone.md index 338b3a32265..da51c0f2c93 100644 --- a/doc/workflow/timezone.md +++ b/doc/workflow/timezone.md @@ -1,31 +1,39 @@ # Changing your time zone The global time zone configuration parameter can be changed in `config/gitlab.yml`: + ``` - # time_zone: 'UTC' +# time_zone: 'UTC' ``` -Uncomment and customize if you want to change the default time zone of GitLab application. +Uncomment and customize if you want to change the default time zone of the GitLab application. + + +## Viewing available timezones To see all available time zones, run `bundle exec rake time:zones:all`. -With Omnibus installations, run `gitlab-rake time:zones:all`. +For Omnibus installations, run `gitlab-rake time:zones:all`. + +NOTE: **Note:** +Currently, this rake task does not list timezones in TZInfo format required by GitLab Omnibus during a reconfigure: [#58672](https://gitlab.com/gitlab-org/gitlab-ce/issues/58672). ## Changing time zone in omnibus installations GitLab defaults its time zone to UTC. It has a global timezone configuration parameter in `/etc/gitlab/gitlab.rb`. -To update, add the time zone that best applies to your location. Here are two examples: +To obtain a list of timezones, log in to your GitLab application server and run a command that generates a list of timezones in TZInfo format for the server. For example, install `timedatectl` and run `timedatectl list-timezones`. + +To update, add the timezone that best applies to your location. For example: + ``` gitlab_rails['time_zone'] = 'America/New_York' ``` -or -``` -gitlab_rails['time_zone'] = 'Europe/Brussels' -``` -After you added this field, reconfigure and restart: +After adding the configuration parameter, reconfigure and restart your GitLab instance: + ``` gitlab-ctl reconfigure gitlab-ctl restart ``` + diff --git a/doc/workflow/todos.md b/doc/workflow/todos.md index 830f17aa7f2..32907db4f46 100644 --- a/doc/workflow/todos.md +++ b/doc/workflow/todos.md @@ -25,9 +25,8 @@ will still be shown in the body of the _To do_ tab. A Todo appears in your Todos dashboard when: -- an issue or merge request is assigned to you, -- you are `@mentioned` in an issue or merge request, be it the description of - the issue/merge request or in a comment, +- an issue or merge request is assigned to you +- you are `@mentioned` in the description or in a comment of an issue, merge request, or epic **[ULTIMATE]** - you are `@mentioned` in a comment on a commit, - a job in the CI pipeline running for your merge request failed, but this job is not allowed to fail. @@ -63,14 +62,14 @@ for filtering; otherwise, they appear as normal. ### Manually creating a Todo -You can also add an issue or merge request to your Todos dashboard by clicking -the "Add todo" button in the issue or merge request sidebar. +You can also add an issue, merge request or epic to your Todos dashboard by clicking +the "Add todo" button in the sidebar of the issue, merge request, or epic **[ULTIMATE]**.  ## Marking a Todo as done -Any action to the corresponding issue or merge request will mark your Todo as +Any action to the corresponding issue, merge request or epic **[ULTIMATE]** will mark your Todo as **Done**. Actions that dismiss Todos include: - changing the assignee @@ -84,10 +83,10 @@ Todos are personal, and they're only marked as done if the action is coming from you. If you close the issue or merge request, your Todo will automatically be marked as done. -If someone else closes, merges, or takes action on the issue or merge +If someone else closes, merges, or takes action on the issue, epic or merge request, your Todo will remain pending. This prevents other users from closing issues without you being notified. -There is just one Todo per issue or merge request, so mentioning a user a +There is just one Todo per issue, epic or merge request, so mentioning a user a hundred times in an issue will only trigger one Todo. --- @@ -97,7 +96,7 @@ corresponding **Done** button, and it will disappear from your Todo list.  -A Todo can also be marked as done from the issue or merge request sidebar using +A Todo can also be marked as done from the issue, merge request or epic sidebar using the "Mark todo as done" button.  @@ -114,7 +113,7 @@ There are four kinds of filters you can use on your Todos dashboard. | Project | Filter by project | | Group | Filter by group | | Author | Filter by the author that triggered the Todo | -| Type | Filter by issue or merge request | +| Type | Filter by issue, merge request, or epic **[ULTIMATE]** | | Action | Filter by the action that triggered the Todo | You can also filter by more than one of these at the same time. The possible Actions are `Any Action`, `Assigned`, `Mentioned`, `Added`, `Pipelines`, and `Directly Addressed`, [as described above](#what-triggers-a-todo). diff --git a/doc/workflow/web_editor.md b/doc/workflow/web_editor.md index 595c7da155b..2366372d984 100644 --- a/doc/workflow/web_editor.md +++ b/doc/workflow/web_editor.md @@ -1 +1,5 @@ +--- +redirect_to: '../user/project/repository/web_editor.md' +--- + This document was moved to [user/project/repository/web_editor](../user/project/repository/web_editor.md). diff --git a/doc/workflow/wip_merge_requests.md b/doc/workflow/wip_merge_requests.md index abb8002f442..020455dcbdc 100644 --- a/doc/workflow/wip_merge_requests.md +++ b/doc/workflow/wip_merge_requests.md @@ -1 +1,5 @@ +--- +redirect_to: '../user/project/merge_requests/work_in_progress_merge_requests.md' +--- + This document was moved to [user/project/merge_requests/work_in_progress_merge_requests](../user/project/merge_requests/work_in_progress_merge_requests.md). |
