diff options
Diffstat (limited to 'doc')
65 files changed, 1687 insertions, 536 deletions
diff --git a/doc/README.md b/doc/README.md index bf93c73843f..b15c3a63d92 100644 --- a/doc/README.md +++ b/doc/README.md @@ -3,12 +3,18 @@ comments: false description: 'Learn how to use and administer GitLab, the most scalable Git-based fully integrated platform for software development.' --- +<div class="display-none"> + <em>Visit <a href="https://docs.gitlab.com/ce/">docs.gitlab.com</a> for optimized + navigation, discoverability, and readability.</em> +</div> +<!-- the div above will not display on the docs site but will display on /help --> + # GitLab Documentation Welcome to [GitLab](https://about.gitlab.com/) Documentation. Here you can access the complete documentation for GitLab, the single application for the -[entire DevOps lifecycle](#complete-devops-with-gitlab). +[entire DevOps lifecycle](#the-entire-devops-lifecycle). ## Overview @@ -72,11 +78,11 @@ GitLab provides statistics and insight into ways you can maximize the value of G The following documentation relates to the DevOps **Manage** stage: -| Manage Topics | Description | -|:----------------------------------------------------------------------------------|:-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| [Authentication and Authorization](administration/auth/README.md) **[CORE ONLY]** | Supported authentication and authorization providers. | -| [GitLab Cycle Analytics](user/project/cycle_analytics.md) | Measure the time it takes to go from an [idea to production](https://about.gitlab.com/2016/08/05/continuous-integration-delivery-and-deployment-with-gitlab/#from-idea-to-production-with-gitlab) for each project you have. | -| [Instance Statistics](user/instance_statistics/index.md) | Discover statistics on how many GitLab features you use and user activity. | +| Manage Topics | Description | +|:--------------------------------------------------------------------------------------|:-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| [Authentication and<br/>Authorization](administration/auth/README.md) **[CORE ONLY]** | Supported authentication and authorization providers. | +| [GitLab Cycle Analytics](user/project/cycle_analytics.md) | Measure the time it takes to go from an [idea to production](https://about.gitlab.com/2016/08/05/continuous-integration-delivery-and-deployment-with-gitlab/#from-idea-to-production-with-gitlab) for each project you have. | +| [Instance Statistics](user/instance_statistics/index.md) | Discover statistics on how many GitLab features you use and user activity. | <div align="right"> <a type="button" class="btn btn-default" href="#overview"> @@ -93,17 +99,17 @@ management tools. The following documentation relates to the DevOps **Plan** stage: -| Plan Topics | Description | -|:-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:-------------------------------------------------------------------------------------------------------------------------------------------------| -| [Discussions](user/discussions/index.md) | Threads, comments, and resolvable discussions in issues, commits, and merge requests. | -| [Due Dates](user/project/issues/due_dates.md) | Keep track of issue deadlines. | -| [Quick Actions](user/project/quick_actions.md) | Shortcuts for common actions on issues or merge requests, replacing the need to click buttons or use dropdowns in GitLab's UI. | -| [Issues](user/project/issues/index.md), including [confidential issues](user/project/issues/confidential_issues.md), [issue and merge request templates](user/project/description_templates.md), and [moving issues](user/project/issues/moving_issues.md) | Project issues, restricting access to issues, create templates for submitting new issues and merge requests, and moving issues between projects. | -| [Labels](user/project/labels.md) | Categorize issues or merge requests with descriptive labels. | -| [Milestones](user/project/milestones/index.md) | Set milestones for delivery of issues and merge requests, with optional due date. | -| [Project Issue Board](user/project/issue_board.md) | Display issues on a Scrum or Kanban board. | -| [Time Tracking](workflow/time_tracking.md) | Track time spent on issues and merge requests. | -| [Todos](workflow/todos.md) | Keep track of work requiring attention with a chronological list displayed on a simple dashboard. | +| Plan Topics | Description | +|:-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:-------------------------------------------------------------------------------------------------------------------------------------------------| +| [Discussions](user/discussions/index.md) | Threads, comments, and resolvable discussions in issues, commits, and merge requests. | +| [Due Dates](user/project/issues/due_dates.md) | Keep track of issue deadlines. | +| [Quick Actions](user/project/quick_actions.md) | Shortcuts for common actions on issues or merge requests, replacing the need to click buttons or use dropdowns in GitLab's UI. | +| [Issues](user/project/issues/index.md), including [confidential issues](user/project/issues/confidential_issues.md),<br/>[issue and merge request templates](user/project/description_templates.md),<br/>and [moving issues](user/project/issues/moving_issues.md) | Project issues, restricting access to issues, create templates for submitting new issues and merge requests, and moving issues between projects. | +| [Labels](user/project/labels.md) | Categorize issues or merge requests with descriptive labels. | +| [Milestones](user/project/milestones/index.md) | Set milestones for delivery of issues and merge requests, with optional due date. | +| [Project Issue Board](user/project/issue_board.md) | Display issues on a Scrum or Kanban board. | +| [Time Tracking](workflow/time_tracking.md) | Track time spent on issues and merge requests. | +| [Todos](workflow/todos.md) | Keep track of work requiring attention with a chronological list displayed on a simple dashboard. | <div align="right"> <a type="button" class="btn btn-default" href="#overview"> @@ -124,16 +130,16 @@ The following documentation relates to the DevOps **Create** stage: #### Projects and Groups -| Create Topics - Projects and Groups | Description | -|:-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:-------------------------------------------------------------------------------| -| [Create](gitlab-basics/create-project.md) and [fork](gitlab-basics/fork-project.md) projects, and [import and export projects between instances](user/project/settings/import_export.md) | Create, duplicate, and move projects. | -| [GitLab Pages](user/project/pages/index.md) | Build, test, and deploy your static website with GitLab Pages. | -| [Groups](user/group/index.md) and [Subgroups](user/group/subgroups/index.md) | Organize your projects in groups. | -| [Projects](user/project/index.md), including [project access](public_access/public_access.md) and [settings](user/project/settings/index.md) | Host source code, and control your project's visibility and set configuration. | -| [Search through GitLab](user/search/index.md) | Search for issues, merge requests, projects, groups, and todos. | -| [Snippets](user/snippets.md) | Snippets allow you to create little bits of code. | -| [Web IDE](user/project/web_ide/index.md) | Edit files within GitLab's user interface. | -| [Wikis](user/project/wiki/index.md) | Enhance your repository documentation with built-in wikis. | +| Create Topics - Projects and Groups | Description | +|:-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:-------------------------------------------------------------------------------| +| [Create](gitlab-basics/create-project.md) and [fork](gitlab-basics/fork-project.md) projects, and<br/>[import and export<br/>projects between instances](user/project/settings/import_export.md) | Create, duplicate, and move projects. | +| [GitLab Pages](user/project/pages/index.md) | Build, test, and deploy your static website with GitLab Pages. | +| [Groups](user/group/index.md) and [Subgroups](user/group/subgroups/index.md) | Organize your projects in groups. | +| [Projects](user/project/index.md), including [project access](public_access/public_access.md)<br/>and [settings](user/project/settings/index.md) | Host source code, and control your project's visibility and set configuration. | +| [Search through GitLab](user/search/index.md) | Search for issues, merge requests, projects, groups, and todos. | +| [Snippets](user/snippets.md) | Snippets allow you to create little bits of code. | +| [Web IDE](user/project/web_ide/index.md) | Edit files within GitLab's user interface. | +| [Wikis](user/project/wiki/index.md) | Enhance your repository documentation with built-in wikis. | <div align="right"> <a type="button" class="btn btn-default" href="#overview"> @@ -143,18 +149,18 @@ The following documentation relates to the DevOps **Create** stage: #### Repositories -| Create Topics - Repositories | Description | -|:--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:--------------------------------------------------------------------------------| -| [Branches](user/project/repository/branches/index.md) and the [default branch](user/project/repository/branches/index.md#default-branch) | How to use branches in GitLab. | -| [Commits](user/project/repository/index.md#commits) and [signing commits](user/project/repository/gpg_signed_commits/index.md) | Work with commits, and use GPG to sign your commits. | -| [Create branches](user/project/repository/web_editor.md#create-a-new-branch), [create](user/project/repository/web_editor.md#create-a-file) and [upload](user/project/repository/web_editor.md#upload-a-file) files, and [create directories](user/project/repository/web_editor.md#create-a-directory) | Create branches, create and upload files, and create directories within GitLab. | -| [Delete merged branches](user/project/repository/branches/index.md#delete-merged-branches) | Bulk delete branches after their changes are merged. | -| [File templates](user/project/repository/web_editor.md#template-dropdowns) | File templates for common files. | -| [Files](user/project/repository/index.md#files) | Files management. | -| [Jupyter Notebook files](user/project/repository/index.md#jupyter-notebook-files) | GitLab's support for `.ipynb` files. | -| [Protected branches](user/project/protected_branches.md) | Use protected branches. | -| [Repositories](user/project/repository/index.md) | Manage source code repositories in GitLab's user interface. | -| [Start a merge request](user/project/repository/web_editor.md#tips) | Start merge request when committing via GitLab's user interface. | +| Create Topics - Repositories | Description | +|:------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:--------------------------------------------------------------------------------| +| [Branches](user/project/repository/branches/index.md) and the [default branch](user/project/repository/branches/index.md#default-branch) | How to use branches in GitLab. | +| [Commits](user/project/repository/index.md#commits) and [signing commits](user/project/repository/gpg_signed_commits/index.md) | Work with commits, and use GPG to sign your commits. | +| [Create branches](user/project/repository/web_editor.md#create-a-new-branch), [create](user/project/repository/web_editor.md#create-a-file)<br/>and [upload](user/project/repository/web_editor.md#upload-a-file) files, and [create directories](user/project/repository/web_editor.md#create-a-directory) | Create branches, create and upload files, and create directories within GitLab. | +| [Delete merged branches](user/project/repository/branches/index.md#delete-merged-branches) | Bulk delete branches after their changes are merged. | +| [File templates](user/project/repository/web_editor.md#template-dropdowns) | File templates for common files. | +| [Files](user/project/repository/index.md#files) | Files management. | +| [Jupyter Notebook files](user/project/repository/index.md#jupyter-notebook-files) | GitLab's support for `.ipynb` files. | +| [Protected branches](user/project/protected_branches.md) | Use protected branches. | +| [Repositories](user/project/repository/index.md) | Manage source code repositories in GitLab's user interface. | +| [Start a merge request](user/project/repository/web_editor.md#tips) | Start merge request when committing via GitLab's user interface. | <div align="right"> <a type="button" class="btn btn-default" href="#overview"> @@ -268,15 +274,15 @@ configuration. Then customize everything from buildpacks to CI/CD. The following documentation relates to the DevOps **Configure** stage: -| Configure Topics | Description | -|:-------------------------------------------------------------------------------------------------------------------------------|:--------------------------------------------------------------------------| -| [Auto DevOps](topics/autodevops/index.md) | Automatically employ a complete DevOps lifecycle. | -| [Easy creation of Kubernetes clusters on GKE](user/project/clusters/index.md#adding-and-creating-a-new-gke-cluster-via-gitlab) | Use Google Kubernetes Engine and GitLab. | -| [Executable Runbooks](user/project/clusters/runbooks/index.md) | Documented procedures that explain how to carry out particular processes. | -| [Installing Applications](user/project/clusters/index.md#installing-applications) | Deploy Helm, Ingress, and Prometheus on Kubernetes. | -| [Mattermost slash commands](user/project/integrations/mattermost_slash_commands.md) | Enable and use slash commands from within Mattermost. | -| [Protected variables](ci/variables/README.md#protected-variables) | Restrict variables to protected branches and tags. | -| [Slack slash commands](user/project/integrations/slack_slash_commands.md) | Enable and use slash commands from within Slack. | +| Configure Topics | Description | +|:-----------------------------------------------------------------------------------------------------------------------------------|:--------------------------------------------------------------------------| +| [Auto DevOps](topics/autodevops/index.md) | Automatically employ a complete DevOps lifecycle. | +| [Easy creation of Kubernetes<br/>clusters on GKE](user/project/clusters/index.md#adding-and-creating-a-new-gke-cluster-via-gitlab) | Use Google Kubernetes Engine and GitLab. | +| [Executable Runbooks](user/project/clusters/runbooks/index.md) | Documented procedures that explain how to carry out particular processes. | +| [Installing Applications](user/project/clusters/index.md#installing-applications) | Deploy Helm, Ingress, and Prometheus on Kubernetes. | +| [Mattermost slash commands](user/project/integrations/mattermost_slash_commands.md) | Enable and use slash commands from within Mattermost. | +| [Protected variables](ci/variables/README.md#protected-variables) | Restrict variables to protected branches and tags. | +| [Slack slash commands](user/project/integrations/slack_slash_commands.md) | Enable and use slash commands from within Slack. | <div align="right"> <a type="button" class="btn btn-default" href="#overview"> diff --git a/doc/administration/auth/README.md b/doc/administration/auth/README.md index 373d4239f71..54be7b616cc 100644 --- a/doc/administration/auth/README.md +++ b/doc/administration/auth/README.md @@ -10,7 +10,7 @@ providers. - [LDAP](ldap.md) Includes Active Directory, Apple Open Directory, Open LDAP, and 389 Server - [OmniAuth](../../integration/omniauth.md) Sign in via Twitter, GitHub, GitLab.com, Google, - Bitbucket, Facebook, Shibboleth, Crowd, Azure and Authentiq ID + Bitbucket, Facebook, Shibboleth, Crowd, Azure, Authentiq ID, and JWT - [CAS](../../integration/cas.md) Configure GitLab to sign in using CAS - [SAML](../../integration/saml.md) Configure GitLab as a SAML 2.0 Service Provider - [Okta](okta.md) Configure GitLab to sign in using Okta diff --git a/doc/administration/auth/jwt.md b/doc/administration/auth/jwt.md index 8b00f52ffc1..497298503ad 100644 --- a/doc/administration/auth/jwt.md +++ b/doc/administration/auth/jwt.md @@ -26,15 +26,15 @@ JWT will provide you with a secret key for you to use. ```ruby gitlab_rails['omniauth_providers'] = [ { name: 'jwt', - app_secret: 'YOUR_APP_SECRET', args: { - algorithm: 'HS256', - uid_claim: 'email', - required_claims: ["name", "email"], - info_maps: { name: "name", email: "email" }, - auth_url: 'https://example.com/', - valid_within: nil, - } + secret: 'YOUR_APP_SECRET', + algorithm: 'HS256', # Supported algorithms: 'RS256', 'RS384', 'RS512', 'ES256', 'ES384', 'ES512', 'HS256', 'HS384', 'HS512' + uid_claim: 'email', + required_claims: ['name', 'email'], + info_maps: { name: 'name', email: 'email' }, + auth_url: 'https://example.com/', + valid_within: 3600 # 1 hour + } } ] ``` @@ -43,15 +43,15 @@ JWT will provide you with a secret key for you to use. ``` - { name: 'jwt', - app_secret: 'YOUR_APP_SECRET', args: { - algorithm: 'HS256', - uid_claim: 'email', - required_claims: ["name", "email"], - info_map: { name: "name", email: "email" }, - auth_url: 'https://example.com/', - valid_within: null, - } + secret: 'YOUR_APP_SECRET', + algorithm: 'HS256', # Supported algorithms: 'RS256', 'RS384', 'RS512', 'ES256', 'ES384', 'ES512', 'HS256', 'HS384', 'HS512' + uid_claim: 'email', + required_claims: ['name', 'email'], + info_map: { name: 'name', email: 'email' }, + auth_url: 'https://example.com/', + valid_within: 3600 # 1 hour + } } ``` @@ -60,7 +60,7 @@ JWT will provide you with a secret key for you to use. 1. Change `YOUR_APP_SECRET` to the client secret and set `auth_url` to your redirect URL. 1. Save the configuration file. -1. [Reconfigure GitLab][] or [restart GitLab][] for the changes to take effect if you +1. [Reconfigure][] or [restart GitLab][] for the changes to take effect if you installed GitLab via Omnibus or from source respectively. On the sign in page there should now be a JWT icon below the regular sign in form. @@ -68,5 +68,5 @@ Click the icon to begin the authentication process. JWT will ask the user to sign in and authorize the GitLab application. If everything goes well, the user will be redirected to GitLab and will be signed in. -[reconfigure GitLab]: ../restart_gitlab.md#omnibus-gitlab-reconfigure +[reconfigure]: ../restart_gitlab.md#omnibus-gitlab-reconfigure [restart GitLab]: ../restart_gitlab.md#installations-from-source diff --git a/doc/administration/gitaly/index.md b/doc/administration/gitaly/index.md index 3395c503ced..dc6a71e2ebd 100644 --- a/doc/administration/gitaly/index.md +++ b/doc/administration/gitaly/index.md @@ -237,3 +237,14 @@ gitaly_enabled=false When you run `service gitlab restart` Gitaly will be disabled on this particular machine. + +## Troubleshooting Gitaly in production + +Since GitLab 11.6, Gitaly comes with a command-line tool called +`gitaly-debug` that can be run on a Gitaly server to aid in +troubleshooting. In GitLab 11.6 its only sub-command is +`simulate-http-clone` which allows you to measure the maximum possible +Git clone speed for a specific repository on the server. + +For an up to date list of sub-commands see [the gitaly-debug +README](https://gitlab.com/gitlab-org/gitaly/blob/master/cmd/gitaly-debug/README.md). diff --git a/doc/administration/monitoring/performance/request_profiling.md b/doc/administration/monitoring/performance/request_profiling.md index c358dfbead2..dfd9be3d04c 100644 --- a/doc/administration/monitoring/performance/request_profiling.md +++ b/doc/administration/monitoring/performance/request_profiling.md @@ -1,16 +1,17 @@ # Request Profiling ## Procedure + 1. Grab the profiling token from `Monitoring > Requests Profiles` admin page -(highlighted in a blue in the image below). - -1. Pass the header `X-Profile-Token: <token>` to the request you want to profile. You can use any of these tools - * [ModHeader](https://chrome.google.com/webstore/detail/modheader/idgpnmonknjnojddfkpgkljpfnnfcklj) Chrome extension - * [Modify Headers](https://addons.mozilla.org/en-US/firefox/addon/modify-headers/) Firefox extension - * `curl --header 'X-Profile-Token: <token>' https://gitlab.example.com/group/project` + (highlighted in a blue in the image below). +  +1. Pass the header `X-Profile-Token: <token>` to the request you want to profile. You can use: + - Browser extensions. For example, [ModHeader](https://chrome.google.com/webstore/detail/modheader/idgpnmonknjnojddfkpgkljpfnnfcklj) Chrome extension. + - `curl`. For example, `curl --header 'X-Profile-Token: <token>' https://gitlab.example.com/group/project`. 1. Once request is finished (which will take a little longer than usual), you can -view the profiling output from `Monitoring > Requests Profiles` admin page. - + view the profiling output from `Monitoring > Requests Profiles` admin page. +  ## Cleaning up + Profiling output will be cleared out every day via a Sidekiq worker. diff --git a/doc/api/README.md b/doc/api/README.md index 0e8d1aaf86e..b49c3a198f1 100644 --- a/doc/api/README.md +++ b/doc/api/README.md @@ -225,6 +225,42 @@ For more information, refer to the Impersonation tokens are used exactly like regular personal access tokens, and can be passed in either the `private_token` parameter or the `Private-Token` header. +#### Disable impersonation + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/issues/40385) in GitLab +11.6. + +By default, impersonation is enabled. To disable impersonation: + +**For Omnibus installations** + +1. Edit `/etc/gitlab/gitlab.rb`: + + ```ruby + gitlab_rails['impersonation_enabled'] = false + ``` + +1. Save the file and [reconfigure](../administration/restart_gitlab.md#omnibus-gitlab-reconfigure) + GitLab for the changes to take effect. + +To re-enable impersonation, remove this configuration and reconfigure GitLab. + +--- + +**For installations from source** + +1. Edit `config/gitlab.yml`: + + ```yaml + gitlab: + impersonation_enabled: false + ``` + +1. Save the file and [restart](../administration/restart_gitlab.md#installations-from-source) + GitLab for the changes to take effect. + +To re-enable impersonation, remove this configuration and restart GitLab. + ### Sudo NOTE: **Note:** diff --git a/doc/api/commits.md b/doc/api/commits.md index 7d9b52ec24f..6c16216429d 100644 --- a/doc/api/commits.md +++ b/doc/api/commits.md @@ -87,7 +87,7 @@ POST /projects/:id/repository/commits | `action` | string | yes | The action to perform, `create`, `delete`, `move`, `update`, `chmod`| | `file_path` | string | yes | Full path to the file. Ex. `lib/class.rb` | | `previous_path` | string | no | Original full path to the file being moved. Ex. `lib/class1.rb`. Only considered for `move` action. | -| `content` | string | no | File content, required for all except `delete` and `chmod`. Optional for `move` | +| `content` | string | no | File content, required for all except `delete`, `chmod`, and `move`. Move actions that do not specify `content` will preserve the existing file content, and any other value of `content` will overwrite the file content. | | `encoding` | string | no | `text` or `base64`. `text` is default. | | `last_commit_id` | string | no | Last known file commit id. Will be only considered in update, move and delete actions. | | `execute_filemode` | boolean | no | When `true/false` enables/disables the execute flag on the file. Only considered for `chmod` action. | diff --git a/doc/api/merge_requests.md b/doc/api/merge_requests.md index da70c74c4ce..9ff6c73b1b6 100644 --- a/doc/api/merge_requests.md +++ b/doc/api/merge_requests.md @@ -408,6 +408,7 @@ Parameters: - `merge_request_iid` (required) - The internal ID of the merge request - `render_html` (optional) - If `true` response includes rendered HTML for title and description - `include_diverged_commits_count` (optional) - If `true` response includes the commits behind the target branch +- `include_rebase_in_progress` (optional) - If `true` response includes whether a rebase operation is in progress ```json { @@ -461,6 +462,7 @@ Parameters: }, "merge_when_pipeline_succeeds": true, "merge_status": "can_be_merged", + "merge_error": null, "sha": "8888888888888888888888888888888888888888", "merge_commit_sha": null, "user_notes_count": 1, @@ -505,7 +507,8 @@ Parameters: "head_sha": "2be7ddb704c7b6b83732fdd5b9f09d5a397b5f8f", "start_sha": "c380d3acebd181f13629a25d2e2acca46ffe1e00" }, - "diverged_commits_count": 2 + "diverged_commits_count": 2, + "rebase_in_progress": false } ``` @@ -773,6 +776,7 @@ POST /projects/:id/merge_requests }, "merge_when_pipeline_succeeds": true, "merge_status": "can_be_merged", + "merge_error": null, "sha": "8888888888888888888888888888888888888888", "merge_commit_sha": null, "user_notes_count": 1, @@ -900,6 +904,7 @@ Must include at least one non-required attribute from above. }, "merge_when_pipeline_succeeds": true, "merge_status": "can_be_merged", + "merge_error": null, "sha": "8888888888888888888888888888888888888888", "merge_commit_sha": null, "user_notes_count": 1, @@ -969,10 +974,9 @@ curl --request DELETE --header "PRIVATE-TOKEN: 9koXpg98eAheJpvBs5tK" https://git Merge changes submitted with MR using this API. +If merge request is unable to be accepted (ie: Work in Progress, Closed, Pipeline Pending Completion, or Failed while requiring Success) - you'll get a `405` and the error message 'Method Not Allowed' -If it has some conflicts and can not be merged - you'll get a `405` and the error message 'Branch cannot be merged' - -If merge request is already merged or closed - you'll get a `406` and the error message 'Method Not Allowed' +If it has some conflicts and can not be merged - you'll get a `406` and the error message 'Branch cannot be merged' If the `sha` parameter is passed and does not match the HEAD of the source - you'll get a `409` and the error message 'SHA does not match HEAD of source branch' @@ -1043,6 +1047,7 @@ Parameters: }, "merge_when_pipeline_succeeds": true, "merge_status": "can_be_merged", + "merge_error": null, "sha": "8888888888888888888888888888888888888888", "merge_commit_sha": null, "user_notes_count": 1, @@ -1158,6 +1163,7 @@ Parameters: }, "merge_when_pipeline_succeeds": false, "merge_status": "can_be_merged", + "merge_error": null, "sha": "8888888888888888888888888888888888888888", "merge_commit_sha": null, "user_notes_count": 1, @@ -1206,6 +1212,62 @@ Parameters: } ``` +## Rebase a merge request + +Automatically rebase the `source_branch` of the merge request against its +`target_branch`. + +If you don't have permissions to push to the merge request's source branch - +you'll get a `403 Forbidden` response. + +``` +PUT /projects/:id/merge_requests/:merge_request_iid/rebase +``` + +| Attribute | Type | Required | Description | +| --------- | ---- | -------- | ----------- | +| `id` | integer/string | yes | The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding) owned by the authenticated user | +| `merge_request_iid` | integer | yes | The internal ID of the merge request | + +```bash +curl --header "PRIVATE-TOKEN: 9koXpg98eAheJpvBs5tK" https://gitlab.example.com/api/v4/projects/76/merge_requests/1/rebase +``` + +This is an asynchronous request. The API will return an empty `202 Accepted` +response if the request is enqueued successfully. + +You can poll the [Get single MR](#get-single-mr) endpoint with the +`include_rebase_in_progress` parameter to check the status of the +asynchronous request. + +If the rebase operation is ongoing, the response will include the following: + +```json +{ + "rebase_in_progress": true + "merge_error": null +} +``` + +Once the rebase operation has completed successfully, the response will include +the following: + +```json +{ + "rebase_in_progress": false, + "merge_error": null, +} +``` + +If the rebase operation fails, the response will include the following: + +```json +{ + "rebase_in_progress": false, + "merge_error": "Rebase failed. Please rebase locally", +} +``` + ## Comments on merge requests Comments are done via the [notes](notes.md) resource. diff --git a/doc/api/repositories.md b/doc/api/repositories.md index c8de7f2191d..55f5a4cc3b2 100644 --- a/doc/api/repositories.md +++ b/doc/api/repositories.md @@ -108,14 +108,18 @@ Get an archive of the repository. This endpoint can be accessed without authentication if the repository is publicly accessible. ``` -GET /projects/:id/repository/archive +GET /projects/:id/repository/archive[.format] ``` +`format` is an optional suffix for the archive format. Default is +`tar.gz`. Options are `tar.gz`, `tar.bz2`, `tbz`, 'tbz2`, `tb2`, +`bz2`, `tar`, and `zip`. For example, specifying `archive.zip` +would send an archive in ZIP format. + Parameters: - `id` (required) - The ID or [URL-encoded path of the project](README.md#namespaced-path-encoding) owned by the authenticated user - `sha` (optional) - The commit SHA to download. A tag, branch reference or sha can be used. This defaults to the tip of the default branch if not specified -- `format` (optional) - The archive format. Default is `tar.gz`. Options are `tar.gz`, `tar.bz2`, `tbz`, `tbz2`, `tb2`, `bz2`, `tar`, `zip` ## Compare branches, tags or commits diff --git a/doc/api/repository_files.md b/doc/api/repository_files.md index c3624f1a535..5f587f480b6 100644 --- a/doc/api/repository_files.md +++ b/doc/api/repository_files.md @@ -4,6 +4,16 @@ **Create, read, update and delete repository files using this API** +The different scopes available using [personal access tokens](../user/profile/personal_access_tokens.md) are depicted +in the following table. + +| Scope | Description | +| ----- | ----------- | +| `read_repository` | Allows read-access to the repository files. | +| `api` | Allows read-write access to the repository files. | + +> `read_repository` scope was [introduced](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/23534) in GitLab 11.6. + ## Get file from repository Allows you to receive information about file in repository like name, size, diff --git a/doc/api/repository_submodules.md b/doc/api/repository_submodules.md index 2e6797f18f4..11b04c81172 100644 --- a/doc/api/repository_submodules.md +++ b/doc/api/repository_submodules.md @@ -22,7 +22,7 @@ PUT /projects/:id/repository/submodules/:submodule | `commit_message` | string | no | Commit message. If no message is provided, a default one will be set | ```sh -curl --request PUT --header "PRIVATE-TOKEN: 9koXpg98eAheJpvBs5tK" "https://gitlab.example.com/api/v4/projects/5/repositories/submodules/lib%2Fmodules%2Fexample" +curl --request PUT --header "PRIVATE-TOKEN: 9koXpg98eAheJpvBs5tK" "https://gitlab.example.com/api/v4/projects/5/repository/submodules/lib%2Fmodules%2Fexample" --data "branch=master&commit_sha=3ddec28ea23acc5caa5d8331a6ecb2a65fc03e88&commit_message=Update submodule reference" ``` diff --git a/doc/api/search.md b/doc/api/search.md index 9716f682ace..7e3ae7404a3 100644 --- a/doc/api/search.md +++ b/doc/api/search.md @@ -722,6 +722,23 @@ Example response: ### Scope: wiki_blobs +Filters are available for this scope: + +- filename +- path +- extension + +To use a filter simply include it in your query like: `a query filename:some_name*`. +You may use wildcards (`*`) to use glob matching. + +Wiki blobs searches are performed on both filenames and contents. Search +results: + +- Found in filenames are displayed before results found in contents. +- May contain multiple matches for the same blob because the search string + might be found in both the filename and content, or might appear multiple + times in the content. + ```bash curl --request GET --header "PRIVATE-TOKEN: 9koXpg98eAheJpvBs5tK" https://gitlab.example.com/api/v4/projects/6/search?scope=wiki_blobs&search=bye ``` @@ -777,14 +794,21 @@ Example response: ### Scope: blobs Filters are available for this scope: + - filename - path - extension -to use a filter simply include it in your query like so: `a query filename:some_name*`. - +To use a filter simply include it in your query like: `a query filename:some_name*`. You may use wildcards (`*`) to use glob matching. +Blobs searches are performed on both filenames and contents. Search results: + +- Found in filenames are displayed before results found in contents. +- May contain multiple matches for the same blob because the search string + might be found in both the filename and content, or might appear multiple + times in the content. + ```bash curl --request GET --header "PRIVATE-TOKEN: 9koXpg98eAheJpvBs5tK" https://gitlab.example.com/api/v4/projects/6/search?scope=blobs&search=installation ``` diff --git a/doc/api/snippets.md b/doc/api/snippets.md index 7892866cd8e..e840e640377 100644 --- a/doc/api/snippets.md +++ b/doc/api/snippets.md @@ -37,13 +37,13 @@ Parameters: | --------- | ---- | -------- | ----------- | | `id` | Integer | yes | The ID of a snippet | -``` bash +```bash curl --header "PRIVATE-TOKEN: 9koXpg98eAheJpvBs5tK" https://gitlab.example.com/api/v4/snippets/1 ``` Example response: -``` json +```json { "id": 1, "title": "test", @@ -65,6 +65,30 @@ Example response: } ``` +## Single snippet contents + +Get a single snippet's raw contents. + +``` +GET /snippets/:id/raw +``` + +Parameters: + +| Attribute | Type | Required | Description | +| --------- | ---- | -------- | ----------- | +| `id` | Integer | yes | The ID of a snippet | + +```bash +curl --header "PRIVATE-TOKEN: 9koXpg98eAheJpvBs5tK" https://gitlab.example.com/api/v4/snippets/1/raw +``` + +Example response: + +``` +Hello World snippet +``` + ## Create new snippet Creates a new snippet. The user must have permission to create new snippets. @@ -84,7 +108,7 @@ Parameters: | `visibility` | String | no | The snippet's visibility | -``` bash +```bash curl --request POST \ --data '{"title": "This is a snippet", "content": "Hello world", "description": "Hello World snippet", "file_name": "test.txt", "visibility": "internal" }' \ --header 'Content-Type: application/json' \ @@ -94,7 +118,7 @@ curl --request POST \ Example response: -``` json +```json { "id": 1, "title": "This is a snippet", @@ -136,7 +160,7 @@ Parameters: | `visibility` | String | no | The snippet's visibility | -``` bash +```bash curl --request PUT \ --data '{"title": "foo", "content": "bar"}' \ --header 'Content-Type: application/json' \ @@ -146,7 +170,7 @@ curl --request PUT \ Example response: -``` json +```json { "id": 1, "title": "test", @@ -201,13 +225,13 @@ GET /snippets/public | `per_page` | Integer | no | number of snippets to return per page | | `page` | Integer | no | the page to retrieve | -``` bash +```bash curl --header "PRIVATE-TOKEN: 9koXpg98eAheJpvBs5tK" https://gitlab.example.com/api/v4/snippets/public?per_page=2&page=1 ``` Example response: -``` json +```json [ { "author": { diff --git a/doc/ci/README.md b/doc/ci/README.md index dba1f38abe2..4e066a0df97 100644 --- a/doc/ci/README.md +++ b/doc/ci/README.md @@ -71,6 +71,7 @@ learn how to leverage its potential even more. - [Caching dependencies](caching/index.md) - [Git submodules](git_submodules.md) - How to run your CI jobs when Git submodules are involved +- [Pipelines for merge requests](merge_request_pipelines/index.md) - [Use SSH keys in your build environment](ssh_keys/README.md) - [Trigger pipelines through the GitLab API](triggers/README.md) - [Trigger pipelines on a schedule](../user/project/pipelines/schedules.md) diff --git a/doc/ci/docker/using_docker_images.md b/doc/ci/docker/using_docker_images.md index 31649ee2792..959271d8abc 100644 --- a/doc/ci/docker/using_docker_images.md +++ b/doc/ci/docker/using_docker_images.md @@ -67,6 +67,9 @@ services you need to `.gitlab-ci.yml` or manually modify `config.toml`. Any image found at [Docker Hub][hub] or your private Container Registry can be used as a service. +Services inherit the same DNS servers, search domains, and additional hosts as +the CI container itself. + You can see some widely used services examples in the relevant documentation of [CI services examples](../services/README.md). diff --git a/doc/ci/examples/laravel_with_gitlab_and_envoy/index.md b/doc/ci/examples/laravel_with_gitlab_and_envoy/index.md index b090ea014dc..b1ccce744d8 100644 --- a/doc/ci/examples/laravel_with_gitlab_and_envoy/index.md +++ b/doc/ci/examples/laravel_with_gitlab_and_envoy/index.md @@ -273,6 +273,8 @@ The `releases` directory will hold all our deployments: echo 'Cloning repository' [ -d {{ $releases_dir }} ] || mkdir {{ $releases_dir }} git clone --depth 1 {{ $repository }} {{ $new_release_dir }} + cd {{ $releases_dir }} + git reset --hard {{ $commit }} @endtask ... @@ -349,6 +351,8 @@ At the end, our `Envoy.blade.php` file will look like this: echo 'Cloning repository' [ -d {{ $releases_dir }} ] || mkdir {{ $releases_dir }} git clone --depth 1 {{ $repository }} {{ $new_release_dir }} + cd {{ $releases_dir }} + git reset --hard {{ $commit }} @endtask @task('run_composer') @@ -519,7 +523,7 @@ deploy_production: - mkdir -p ~/.ssh - '[[ -f /.dockerenv ]] && echo -e "Host *\n\tStrictHostKeyChecking no\n\n" > ~/.ssh/config' - - ~/.composer/vendor/bin/envoy run deploy + - ~/.composer/vendor/bin/envoy run deploy --commit="$CI_COMMIT_SHA" environment: name: production url: http://192.168.1.1 diff --git a/doc/ci/interactive_web_terminal/index.md b/doc/ci/interactive_web_terminal/index.md index 1ddc1bf4d7e..2c799e83a5f 100644 --- a/doc/ci/interactive_web_terminal/index.md +++ b/doc/ci/interactive_web_terminal/index.md @@ -1,4 +1,4 @@ -# Getting started with interactive web terminals +# Getting started with interactive web terminals **[CORE ONLY]** > [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/issues/50144) in GitLab 11.3. @@ -6,10 +6,9 @@ Interactive web terminals give the user access to a terminal in GitLab for running one-off commands for their CI pipeline. NOTE: **Note:** -This is not available for the shared Runners on GitLab.com. -To make use of this feature, you need to provide your -[own Runner](https://docs.gitlab.com/runner/install/) and properly -[configure it](#configuration). +GitLab.com does not support interactive web terminal at the moment. Please +follow [this issue](https://gitlab.com/gitlab-org/gitlab-ce/issues/52611) for +progress. ## Configuration diff --git a/doc/ci/merge_request_pipelines/img/merge_request.png b/doc/ci/merge_request_pipelines/img/merge_request.png Binary files differnew file mode 100644 index 00000000000..1fe2eec2008 --- /dev/null +++ b/doc/ci/merge_request_pipelines/img/merge_request.png diff --git a/doc/ci/merge_request_pipelines/img/pipeline_detail.png b/doc/ci/merge_request_pipelines/img/pipeline_detail.png Binary files differnew file mode 100644 index 00000000000..def1781dd75 --- /dev/null +++ b/doc/ci/merge_request_pipelines/img/pipeline_detail.png diff --git a/doc/ci/merge_request_pipelines/index.md b/doc/ci/merge_request_pipelines/index.md new file mode 100644 index 00000000000..706e83abf44 --- /dev/null +++ b/doc/ci/merge_request_pipelines/index.md @@ -0,0 +1,84 @@ +# Pipelines for merge requests + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/issues/15310) in GitLab 11.6 + +Usually, when a developer creates a new merge request, a pipeline runs on the +new change and checks if it's qualified to be merged into a target branch. This +pipeline should contain only necessary jobs for checking the new changes. +For example, unit tests, lint checks, and Review Apps are often used in this cycle. + +With pipelines for merge requests, you can design a specific pipeline structure +for merge requests. All you need to do is just adding `only: [merge_requests]` to +the jobs that you want it to run for only merge requests. +Every time, when developers create or update merge requests, a pipeline runs on +their new commits at every push to GitLab. + +NOTE: **Note**: +If you use both this feature and the [Merge When Pipeline Succeeds](../../user/project/merge_requests/merge_when_pipeline_succeeds.md) +feature, pipelines for merge requests take precendence over the other regular pipelines. + +For example, consider a GitLab CI/CD configuration in .gitlab-ci.yml as follows: + +```yaml +build: + stage: build + script: ./build + only: + - branches + - tags + - merge_requests + +test: + stage: test + script: ./test + only: + - merge_requests + +deploy: + stage: deploy + script: ./deploy +``` + +After a developer updated code in a merge request with whatever methods (e.g. `git push`), +GitLab detects that the code is updated and create a new pipeline for the merge request. +The pipeline fetches the latest code from the source branch and run tests against it. +In this example, the pipeline contains only `build` and `test` jobs. +Since `deploy` job does not have the `only: [merge_requests]` rule, +deployment jobs will not happen in the merge request. + +Consider this pipeline list viewed from the **Pipelines** tab in a merge request: + + + +Note that pipelines tagged as **merge request** indicate that they were triggered +when a merge request was created or updated. + +The same tag is shown on the pipeline's details: + + + +## Important notes about merge requests from forked projects + +Note that the current behavior is subject to change. In the usual contribution +flow, external contributors follow the following steps: + +1. Fork a parent project. +1. Create a merge request from the forked project that targets the `master` branch +in the parent project. +1. A pipeline runs on the merge request. +1. A mainatiner from the parent project checks the pipeline result, and merge +into a target branch if the latest pipeline has passed. + +Currently, those pipelines are created in a **forked** project, not in the +parent project. This means you cannot completely trust the pipeline result, +because, technically, external contributors can disguise their pipeline results +by tweaking their GitLab Runner in the forked project. + +There are multiple reasons about why GitLab doesn't allow those pipelines to be +created in the parent project, but one of the biggest reasons is security concern. +External users could steal secret variables from the parent project by modifying +.gitlab-ci.yml, which could be some sort of credentials. This should not happen. + +We're discussing a secure solution of running pipelines for merge requests +that submitted from forked projects, +see [the issue about the permission extension](https://gitlab.com/gitlab-org/gitlab-ce/issues/23902). diff --git a/doc/ci/variables/README.md b/doc/ci/variables/README.md index bdbcf8c9435..87799be8ab4 100644 --- a/doc/ci/variables/README.md +++ b/doc/ci/variables/README.md @@ -40,75 +40,86 @@ Starting with GitLab 9.0, we have deprecated some variables. Read the strongly advised to use the new variables as we will remove the old ones in future GitLab releases.** -| Variable | GitLab | Runner | Description | -|-------------------------------- |--------|--------|-------------| -| **ARTIFACT_DOWNLOAD_ATTEMPTS** | 8.15 | 1.9 | Number of attempts to download artifacts running a job | -| **CI** | all | 0.4 | Mark that job is executed in CI environment | -| **CI_COMMIT_REF_NAME** | 9.0 | all | The branch or tag name for which project is built | -| **CI_COMMIT_REF_SLUG** | 9.0 | all | `$CI_COMMIT_REF_NAME` lowercased, shortened to 63 bytes, and with everything except `0-9` and `a-z` replaced with `-`. No leading / trailing `-`. Use in URLs, host names and domain names. | -| **CI_COMMIT_SHA** | 9.0 | all | The commit revision for which project is built | -| **CI_COMMIT_BEFORE_SHA** | 11.2 | all | The previous latest commit present on a branch before a push request. | -| **CI_COMMIT_TAG** | 9.0 | 0.5 | The commit tag name. Present only when building tags. | -| **CI_COMMIT_MESSAGE** | 10.8 | all | The full commit message. | -| **CI_COMMIT_TITLE** | 10.8 | all | The title of the commit - the full first line of the message | -| **CI_COMMIT_DESCRIPTION** | 10.8 | all | The description of the commit: the message without first line, if the title is shorter than 100 characters; full message in other case. | -| **CI_CONFIG_PATH** | 9.4 | 0.5 | The path to CI config file. Defaults to `.gitlab-ci.yml` | -| **CI_DEBUG_TRACE** | all | 1.7 | Whether [debug tracing](#debug-tracing) is enabled | -| **CI_DEPLOY_USER** | 10.8 | all | Authentication username of the [GitLab Deploy Token][gitlab-deploy-token], only present if the Project has one related.| -| **CI_DEPLOY_PASSWORD** |Â 10.8 |Â all | Authentication password of the [GitLab Deploy Token][gitlab-deploy-token], only present if the Project has one related.| -| **CI_DISPOSABLE_ENVIRONMENT** | all | 10.1 | Marks that the job is executed in a disposable environment (something that is created only for this job and disposed of/destroyed after the execution - all executors except `shell` and `ssh`). If the environment is disposable, it is set to true, otherwise it is not defined at all. | -| **CI_ENVIRONMENT_NAME** | 8.15 | all | The name of the environment for this job | -| **CI_ENVIRONMENT_SLUG** | 8.15 | all | A simplified version of the environment name, suitable for inclusion in DNS, URLs, Kubernetes labels, etc. | -| **CI_ENVIRONMENT_URL** | 9.3 | all | The URL of the environment for this job | -| **CI_JOB_ID** | 9.0 | all | The unique id of the current job that GitLab CI uses internally | -| **CI_JOB_MANUAL** | 8.12 | all | The flag to indicate that job was manually started | -| **CI_JOB_NAME** | 9.0 | 0.5 | The name of the job as defined in `.gitlab-ci.yml` | -| **CI_JOB_STAGE** | 9.0 | 0.5 | The name of the stage as defined in `.gitlab-ci.yml` | -| **CI_JOB_TOKEN** | 9.0 | 1.2 | Token used for authenticating with the [GitLab Container Registry][registry] and downloading [dependent repositories][dependent-repositories] | -| **CI_NODE_INDEX** | 11.5 | all | Index of the job in the job set. If the job is not parallelized, this variable is not set. | -| **CI_NODE_TOTAL** | 11.5 | all | Total number of instances of this job running in parallel. If the job is not parallelized, this variable is set to `1`. | -| **CI_JOB_URL** | 11.1 | 0.5 | Job details URL | -| **CI_REPOSITORY_URL** | 9.0 | all | The URL to clone the Git repository | -| **CI_RUNNER_DESCRIPTION** | 8.10 | 0.5 | The description of the runner as saved in GitLab | -| **CI_RUNNER_ID** | 8.10 | 0.5 | The unique id of runner being used | -| **CI_RUNNER_TAGS** | 8.10 | 0.5 | The defined runner tags | -| **CI_RUNNER_VERSION** | all | 10.6 | GitLab Runner version that is executing the current job | -| **CI_RUNNER_REVISION** | all | 10.6 | GitLab Runner revision that is executing the current job | -| **CI_RUNNER_EXECUTABLE_ARCH** | all | 10.6 | The OS/architecture of the GitLab Runner executable (note that this is not necessarily the same as the environment of the executor) | -| **CI_PIPELINE_ID** | 8.10 | 0.5 | The unique id of the current pipeline that GitLab CI uses internally | -| **CI_PIPELINE_IID** | 11.0 | all | The unique id of the current pipeline scoped to project | -| **CI_PIPELINE_TRIGGERED** | all | all | The flag to indicate that job was [triggered] | -| **CI_PIPELINE_SOURCE** | 10.0 | all | Indicates how the pipeline was triggered. Possible options are: `push`, `web`, `trigger`, `schedule`, `api`, and `pipeline`. For pipelines created before GitLab 9.5, this will show as `unknown` | -| **CI_PROJECT_DIR** | all | all | The full path where the repository is cloned and where the job is run | -| **CI_PROJECT_ID** | all | all | The unique id of the current project that GitLab CI uses internally | -| **CI_PROJECT_NAME** | 8.10 | 0.5 | The project name that is currently being built (actually it is project folder name) | -| **CI_PROJECT_NAMESPACE** | 8.10 | 0.5 | The project namespace (username or groupname) that is currently being built | -| **CI_PROJECT_PATH** | 8.10 | 0.5 | The namespace with project name | -| **CI_PROJECT_PATH_SLUG** | 9.3 | all | `$CI_PROJECT_PATH` lowercased and with everything except `0-9` and `a-z` replaced with `-`. Use in URLs and domain names. | -| **CI_PIPELINE_URL** | 11.1 | 0.5 | Pipeline details URL | -| **CI_PROJECT_URL** | 8.10 | 0.5 | The HTTP address to access project | -| **CI_PROJECT_VISIBILITY** | 10.3 | all | The project visibility (internal, private, public) | -| **CI_REGISTRY** | 8.10 | 0.5 | If the Container Registry is enabled it returns the address of GitLab's Container Registry | -| **CI_REGISTRY_IMAGE** | 8.10 | 0.5 | If the Container Registry is enabled for the project it returns the address of the registry tied to the specific project | -| **CI_REGISTRY_PASSWORD** | 9.0 | all | The password to use to push containers to the GitLab Container Registry | -| **CI_REGISTRY_USER** | 9.0 | all | The username to use to push containers to the GitLab Container Registry | -| **CI_SERVER** | all | all | Mark that job is executed in CI environment | -| **CI_SERVER_NAME** | all | all | The name of CI server that is used to coordinate jobs | -| **CI_SERVER_REVISION** | all | all | GitLab revision that is used to schedule jobs | -| **CI_SERVER_VERSION** | all | all | GitLab version that is used to schedule jobs | -| **CI_SERVER_VERSION_MAJOR** | 11.4 | all | GitLab version major component | -| **CI_SERVER_VERSION_MINOR** | 11.4 | all | GitLab version minor component | -| **CI_SERVER_VERSION_PATCH** | 11.4 | all | GitLab version patch component | -| **CI_SHARED_ENVIRONMENT** | all | 10.1 | Marks that the job is executed in a shared environment (something that is persisted across CI invocations like `shell` or `ssh` executor). If the environment is shared, it is set to true, otherwise it is not defined at all. | -| **GET_SOURCES_ATTEMPTS** | 8.15 | 1.9 | Number of attempts to fetch sources running a job | -| **GITLAB_CI** | all | all | Mark that job is executed in GitLab CI environment | -| **GITLAB_USER_EMAIL** | 8.12 | all | The email of the user who started the job | -| **GITLAB_USER_ID** | 8.12 | all | The id of the user who started the job | -| **GITLAB_USER_LOGIN** | 10.0 | all | The login username of the user who started the job | -| **GITLAB_USER_NAME** | 10.0 | all | The real name of the user who started the job | -| **RESTORE_CACHE_ATTEMPTS** | 8.15 | 1.9 | Number of attempts to restore the cache running a job | - -## 9.0 Renaming +| Variable | GitLab | Runner | Description | +|-------------------------------------------|--------|--------|-------------| +| **ARTIFACT_DOWNLOAD_ATTEMPTS** | 8.15 | 1.9 | Number of attempts to download artifacts running a job | +| **CI** | all | 0.4 | Mark that job is executed in CI environment | +| **CI_COMMIT_REF_NAME** | 9.0 | all | The branch or tag name for which project is built | +| **CI_COMMIT_REF_SLUG** | 9.0 | all | `$CI_COMMIT_REF_NAME` lowercased, shortened to 63 bytes, and with everything except `0-9` and `a-z` replaced with `-`. No leading / trailing `-`. Use in URLs, host names and domain names. | +| **CI_COMMIT_SHA** | 9.0 | all | The commit revision for which project is built | +| **CI_COMMIT_BEFORE_SHA** | 11.2 | all | The previous latest commit present on a branch before a push request. | +| **CI_COMMIT_TAG** | 9.0 | 0.5 | The commit tag name. Present only when building tags. | +| **CI_COMMIT_MESSAGE** | 10.8 | all | The full commit message. | +| **CI_COMMIT_TITLE** | 10.8 | all | The title of the commit - the full first line of the message | +| **CI_COMMIT_DESCRIPTION** | 10.8 | all | The description of the commit: the message without first line, if the title is shorter than 100 characters; full message in other case. | +| **CI_CONFIG_PATH** | 9.4 | 0.5 | The path to CI config file. Defaults to `.gitlab-ci.yml` | +| **CI_DEBUG_TRACE** | all | 1.7 | Whether [debug tracing](#debug-tracing) is enabled | +| **CI_DEPLOY_USER** | 10.8 | all | Authentication username of the [GitLab Deploy Token][gitlab-deploy-token], only present if the Project has one related.| +| **CI_DEPLOY_PASSWORD** |Â 10.8 |Â all | Authentication password of the [GitLab Deploy Token][gitlab-deploy-token], only present if the Project has one related.| +| **CI_DISPOSABLE_ENVIRONMENT** | all | 10.1 | Marks that the job is executed in a disposable environment (something that is created only for this job and disposed of/destroyed after the execution - all executors except `shell` and `ssh`). If the environment is disposable, it is set to true, otherwise it is not defined at all. | +| **CI_ENVIRONMENT_NAME** | 8.15 | all | The name of the environment for this job | +| **CI_ENVIRONMENT_SLUG** | 8.15 | all | A simplified version of the environment name, suitable for inclusion in DNS, URLs, Kubernetes labels, etc. | +| **CI_ENVIRONMENT_URL** | 9.3 | all | The URL of the environment for this job | +| **CI_JOB_ID** | 9.0 | all | The unique id of the current job that GitLab CI uses internally | +| **CI_JOB_MANUAL** | 8.12 | all | The flag to indicate that job was manually started | +| **CI_JOB_NAME** | 9.0 | 0.5 | The name of the job as defined in `.gitlab-ci.yml` | +| **CI_JOB_STAGE** | 9.0 | 0.5 | The name of the stage as defined in `.gitlab-ci.yml` | +| **CI_JOB_TOKEN** | 9.0 | 1.2 | Token used for authenticating with the [GitLab Container Registry][registry] and downloading [dependent repositories][dependent-repositories] | +| **CI_MERGE_REQUEST_ID** | 11.6 | all | The ID of the merge request if it's [pipelines for merge requests](../merge_request_pipelines/index.md) | +| **CI_MERGE_REQUEST_IID** | 11.6 | all | The IID of the merge request if it's [pipelines for merge requests](../merge_request_pipelines/index.md) | +| **CI_MERGE_REQUEST_REF_PATH** | 11.6 | all | The ref path of the merge request if it's [pipelines for merge requests](../merge_request_pipelines/index.md). (e.g. `refs/merge-requests/1/head`) | +| **CI_MERGE_REQUEST_PROJECT_ID** | 11.6 | all | The ID of the project of the merge request if it's [pipelines for merge requests](../merge_request_pipelines/index.md) | +| **CI_MERGE_REQUEST_PROJECT_PATH** | 11.6 | all | The path of the project of the merge request if it's [pipelines for merge requests](../merge_request_pipelines/index.md) (e.g. `namespace/awesome-project`) | +| **CI_MERGE_REQUEST_PROJECT_URL** | 11.6 | all | The URL of the project of the merge request if it's [pipelines for merge requests](../merge_request_pipelines/index.md) (e.g. `http://192.168.10.15:3000/namespace/awesome-project`) | +| **CI_MERGE_REQUEST_TARGET_BRANCH_NAME** | 11.6 | all | The target branch name of the merge request if it's [pipelines for merge requests](../merge_request_pipelines/index.md) | +| **CI_MERGE_REQUEST_SOURCE_PROJECT_ID** | 11.6 | all | The ID of the source project of the merge request if it's [pipelines for merge requests](../merge_request_pipelines/index.md) | +| **CI_MERGE_REQUEST_SOURCE_PROJECT_PATH** | 11.6 | all | The path of the source project of the merge request if it's [pipelines for merge requests](../merge_request_pipelines/index.md) | +| **CI_MERGE_REQUEST_SOURCE_PROJECT_URL** | 11.6 | all | The URL of the source project of the merge request if it's [pipelines for merge requests](../merge_request_pipelines/index.md) | +| **CI_MERGE_REQUEST_SOURCE_BRANCH_NAME** | 11.6 | all | The source branch name of the merge request if it's [pipelines for merge requests](../merge_request_pipelines/index.md) | +| **CI_NODE_INDEX** | 11.5 | all | Index of the job in the job set. If the job is not parallelized, this variable is not set. | +| **CI_NODE_TOTAL** | 11.5 | all | Total number of instances of this job running in parallel. If the job is not parallelized, this variable is set to `1`. | +| **CI_JOB_URL** | 11.1 | 0.5 | Job details URL | +| **CI_REPOSITORY_URL** | 9.0 | all | The URL to clone the Git repository | +| **CI_RUNNER_DESCRIPTION** | 8.10 | 0.5 | The description of the runner as saved in GitLab | +| **CI_RUNNER_ID** | 8.10 | 0.5 | The unique id of runner being used | +| **CI_RUNNER_TAGS** | 8.10 | 0.5 | The defined runner tags | +| **CI_RUNNER_VERSION** | all | 10.6 | GitLab Runner version that is executing the current job | +| **CI_RUNNER_REVISION** | all | 10.6 | GitLab Runner revision that is executing the current job | +| **CI_RUNNER_EXECUTABLE_ARCH** | all | 10.6 | The OS/architecture of the GitLab Runner executable (note that this is not necessarily the same as the environment of the executor) | +| **CI_PIPELINE_ID** | 8.10 | 0.5 | The unique id of the current pipeline that GitLab CI uses internally | +| **CI_PIPELINE_IID** | 11.0 | all | The unique id of the current pipeline scoped to project | +| **CI_PIPELINE_TRIGGERED** | all | all | The flag to indicate that job was [triggered] | +| **CI_PIPELINE_SOURCE** | 10.0 | all | Indicates how the pipeline was triggered. Possible options are: `push`, `web`, `trigger`, `schedule`, `api`, and `pipeline`. For pipelines created before GitLab 9.5, this will show as `unknown` | +| **CI_PROJECT_DIR** | all | all | The full path where the repository is cloned and where the job is run | +| **CI_PROJECT_ID** | all | all | The unique id of the current project that GitLab CI uses internally | +| **CI_PROJECT_NAME** | 8.10 | 0.5 | The project name that is currently being built (actually it is project folder name) | +| **CI_PROJECT_NAMESPACE** | 8.10 | 0.5 | The project namespace (username or groupname) that is currently being built | +| **CI_PROJECT_PATH** | 8.10 | 0.5 | The namespace with project name | +| **CI_PROJECT_PATH_SLUG** | 9.3 | all | `$CI_PROJECT_PATH` lowercased and with everything except `0-9` and `a-z` replaced with `-`. Use in URLs and domain names. | +| **CI_PIPELINE_URL** | 11.1 | 0.5 | Pipeline details URL | +| **CI_PROJECT_URL** | 8.10 | 0.5 | The HTTP address to access project | +| **CI_PROJECT_VISIBILITY** | 10.3 | all | The project visibility (internal, private, public) | +| **CI_REGISTRY** | 8.10 | 0.5 | If the Container Registry is enabled it returns the address of GitLab's Container Registry | +| **CI_REGISTRY_IMAGE** | 8.10 | 0.5 | If the Container Registry is enabled for the project it returns the address of the registry tied to the specific project | +| **CI_REGISTRY_PASSWORD** | 9.0 | all | The password to use to push containers to the GitLab Container Registry | +| **CI_REGISTRY_USER** | 9.0 | all | The username to use to push containers to the GitLab Container Registry | +| **CI_SERVER** | all | all | Mark that job is executed in CI environment | +| **CI_SERVER_NAME** | all | all | The name of CI server that is used to coordinate jobs | +| **CI_SERVER_REVISION** | all | all | GitLab revision that is used to schedule jobs | +| **CI_SERVER_VERSION** | all | all | GitLab version that is used to schedule jobs | +| **CI_SERVER_VERSION_MAJOR** | 11.4 | all | GitLab version major component | +| **CI_SERVER_VERSION_MINOR** | 11.4 | all | GitLab version minor component | +| **CI_SERVER_VERSION_PATCH** | 11.4 | all | GitLab version patch component | +| **CI_SHARED_ENVIRONMENT** | all | 10.1 | Marks that the job is executed in a shared environment (something that is persisted across CI invocations like `shell` or `ssh` executor). If the environment is shared, it is set to true, otherwise it is not defined at all. | +| **GET_SOURCES_ATTEMPTS** | 8.15 | 1.9 | Number of attempts to fetch sources running a job | +| **GITLAB_CI** | all | all | Mark that job is executed in GitLab CI environment | +| **GITLAB_USER_EMAIL** | 8.12 | all | The email of the user who started the job | +| **GITLAB_USER_ID** | 8.12 | all | The id of the user who started the job | +| **GITLAB_USER_LOGIN** | 10.0 | all | The login username of the user who started the job | +| **GITLAB_USER_NAME** | 10.0 | all | The real name of the user who started the job | +| **RESTORE_CACHE_ATTEMPTS** | 8.15 | 1.9 | Number of attempts to restore the cache running a job | + +## GitLab 9.0 renaming To follow conventions of naming across GitLab, and to further move away from the `build` term and toward `job` CI variables have been renamed for the 9.0 @@ -137,7 +148,7 @@ future GitLab releases.** ## `.gitlab-ci.yml` defined variables NOTE **Note:** -This feature requires GitLab Runner 0.5.0 or higher and GitLab CI 7.14 or higher. +This feature requires GitLab Runner 0.5.0 or higher and GitLab 7.14 or higher. GitLab CI allows you to add to `.gitlab-ci.yml` variables that are set in the build environment. The variables are hence saved in the repository, and they @@ -176,8 +187,7 @@ script: ## Variables -NOTE: **Note:** -Group-level variables were added in GitLab 9.4. +> Group-level variables were introduced in GitLab 9.4. CAUTION: **Important:** Be aware that variables are not masked, and their values can be shown @@ -206,8 +216,7 @@ Once you set them, they will be available for all subsequent pipelines. You can ### Protected variables ->**Notes:** -This feature requires GitLab 9.3 or higher. +> Introduced in GitLab 9.3. Variables could be protected. Whenever a variable is protected, it would only be securely passed to pipelines running on the @@ -228,8 +237,7 @@ Variables can be specified for a single pipeline run when a [manual pipeline](.. ## Deployment variables -NOTE: **Note:** -This feature requires GitLab CI 8.15 or higher. +> Introduced in GitLab 8.15. [Project services](../../user/project/integrations/project_services.md) that are responsible for deployment configuration may define their own variables that @@ -490,7 +498,7 @@ export CI_REGISTRY_PASSWORD="longalfanumstring" ## Variables expressions -> Variables expressions were added in GitLab 10.7. +> Introduced in GitLab 10.7. It is possible to use variables expressions with only / except policies in `.gitlab-ci.yml`. By using this approach you can limit what jobs are going to diff --git a/doc/ci/yaml/README.md b/doc/ci/yaml/README.md index 77406493578..1277d1fdf8b 100644 --- a/doc/ci/yaml/README.md +++ b/doc/ci/yaml/README.md @@ -342,15 +342,16 @@ In addition, `only` and `except` allow the use of special keywords: | **Value** | **Description** | | --------- | ---------------- | -| `branches` | When a branch is pushed. | -| `tags` | When a tag is pushed. | -| `api` | When pipeline has been triggered by a second pipelines API (not triggers API). | -| `external` | When using CI services other than GitLab. | -| `pipelines` | For multi-project triggers, created using the API with `CI_JOB_TOKEN`. | -| `pushes` | Pipeline is triggered by a `git push` by the user. | -| `schedules` | For [scheduled pipelines][schedules]. | -| `triggers` | For pipelines created using a trigger token. | -| `web` | For pipelines created using **Run pipeline** button in GitLab UI (under your project's **Pipelines**). | +| `branches` | When a git reference of a pipeline is a branch. | +| `tags` | When a git reference of a pipeline is a tag. | +| `api` | When pipeline has been triggered by a second pipelines API (not triggers API). | +| `external` | When using CI services other than GitLab. | +| `pipelines` | For multi-project triggers, created using the API with `CI_JOB_TOKEN`. | +| `pushes` | Pipeline is triggered by a `git push` by the user. | +| `schedules` | For [scheduled pipelines][schedules]. | +| `triggers` | For pipelines created using a trigger token. | +| `web` | For pipelines created using **Run pipeline** button in GitLab UI (under your project's **Pipelines**). | +| `merge_requests` | When a merge request is created or updated (See [pipelines for merge requests](../merge_request_pipelines/index.md)). | In the example below, `job` will run only for refs that start with `issue-`, whereas all branches will be skipped: @@ -391,6 +392,24 @@ job: The above example will run `job` for all branches on `gitlab-org/gitlab-ce`, except master. +If a job does not have neither `only` nor `except` rule, +`only: ['branches', 'tags']` is set by default. + +For example, + +```yaml +job: + script: echo 'test' +``` + +is translated to + +```yaml +job: + script: echo 'test' + only: ['branches', 'tags'] +``` + ## `only` and `except` (complex) > `refs` and `kubernetes` policies introduced in GitLab 10.0 @@ -1590,7 +1609,7 @@ Possible values for `when` are: > [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/22631) in GitLab 11.5. `parallel` allows you to configure how many instances of a job to run in -parallel. This value has to be greater than or equal to two (2) and less or equal than 50. +parallel. This value has to be greater than or equal to two (2) and less than or equal to 50. This creates N instances of the same job that run in parallel. They're named sequentially from `job_name 1/N` to `job_name N/N`. @@ -1821,13 +1840,6 @@ These variables can be later used in all executed commands and scripts. The YAML-defined variables are also set to all created service containers, thus allowing to fine tune them. -To turn off global defined variables in a specific job, define an empty hash: - -```yaml -job_name: - variables: {} -``` - Except for the user defined variables, there are also the ones [set up by the Runner itself](../variables/README.md#predefined-variables-environment-variables). One example would be `CI_COMMIT_REF_NAME` which has the value of diff --git a/doc/development/architecture.md b/doc/development/architecture.md index 931a7a8e6d5..e65c5f05505 100644 --- a/doc/development/architecture.md +++ b/doc/development/architecture.md @@ -55,7 +55,7 @@ GitLab can be considered to have two layers from a process perspective: ### gitaly -- [Omnibus confiugration options](https://gitlab.com/gitlab-org/gitaly/tree/master/doc/configuration) +- [Omnibus configuration options](https://gitlab.com/gitlab-org/gitaly/tree/master/doc/configuration) - Layer: Core Service (Data) Gitaly is a service designed by GitLab to remove our need for NFS for Git storage in distributed deployments of GitLab. (Think GitLab.com or High Availablity Deployments) As of 11.3.0, This service handles all Git level access in GitLab. You can read more about the project [in the project's readme](https://gitlab.com/gitlab-org/gitaly). diff --git a/doc/development/automatic_ce_ee_merge.md b/doc/development/automatic_ce_ee_merge.md index 58e08d432cc..e4eb26b3aca 100644 --- a/doc/development/automatic_ce_ee_merge.md +++ b/doc/development/automatic_ce_ee_merge.md @@ -1,57 +1,33 @@ # Automatic CE->EE merge -GitLab Community Edition is merged automatically every 3 hours into the -Enterprise Edition (look for the [`CE Upstream` merge requests]). - -This merge is done automatically in a -[scheduled pipeline](https://gitlab.com/gitlab-org/release-tools/-/jobs/43201679). - -## What to do if you are pinged in a `CE Upstream` merge request to resolve a conflict? - -1. Please resolve the conflict as soon as possible or ask someone else to do it - - It's ok to resolve more conflicts than the one that you are asked to resolve. - In that case, it's a good habit to ask for a double-check on your resolution - by someone who is familiar with the code you touched. -1. Once you have resolved your conflicts, push to the branch (no force-push) -1. Assign the merge request to the next person that has to resolve a conflict -1. If all conflicts are resolved after your resolution is pushed, keep the merge - request assigned to you: **you are now responsible for the merge request to be - green** -1. If you are the last person to resolve the conflicts, the pipeline is green, - and you have merge rights, merge the MR, but **do not** choose to squash. - Otherwise, assign the MR to someone that can merge. -1. If you need any help, you can ping the current [release managers], or ask in - the `#ce-to-ee` Slack channel - -A few notes about the automatic CE->EE merge job: - -- If a merge is already in progress, the job - [doesn't create a new one](https://gitlab.com/gitlab-org/release-tools/-/jobs/43157687). -- If there is nothing to merge (i.e. EE is up-to-date with CE), the job doesn't - create a new one -- The job posts messages to the `#ce-to-ee` Slack channel to inform what's the - current CE->EE merge status (e.g. "A new MR has been created", "A MR is still pending") - -[`CE Upstream` merge requests]: https://gitlab.com/gitlab-org/gitlab-ee/merge_requests?label_name%5B%5D=CE+upstream -[release managers]: https://about.gitlab.com/release-managers/ +Whenever a commit is pushed to the CE `master` branch, it is automatically +merged into the EE `master` branch. If the commit produces any conflicts, it is +instead reverted from CE `master`. When this happens, a merge request will be +set up automatically that can be used to reinstate the changes. This merge +request will be assigned to the author of the conflicting commit, or the merge +request author if the commit author could not be associated with a GitLab user. +If no author could be found, the merge request is assigned to a random member of +the Delivery team. It is then up to this team member to figure out who to assign +the merge request to. + +Because some commits can not be reverted if new commits depend on them, we also +run a job periodically that processes a range of commits and tries to merge or +revert them. This should ensure that all commits are either merged into EE +`master`, or reverted, instead of just being left behind in CE. ## Always merge EE merge requests before their CE counterparts **In order to avoid conflicts in the CE->EE merge, you should always merge the EE version of your CE merge request first, if present.** -The rationale for this is that as CE->EE merges are done automatically every few -hours, it can happen that: +The rationale for this is that as CE->EE merges are done automatically, it can +happen that: -1. A CE merge request that needs EE-specific changes is merged -1. The automatic CE->EE merge happens +1. A CE merge request that needs EE-specific changes is merged. +1. The automatic CE->EE merge happens. 1. Conflicts due to the CE merge request occur since its EE merge request isn't - merged yet -1. The automatic merge bot will ping someone to resolve the conflict **that are - already resolved in the EE merge request that isn't merged yet** - -That's a waste of time, and that's why you should merge EE merge request before -their CE counterpart. + merged yet. +1. The CE changes are reverted. ## Avoiding CE->EE merge conflicts beforehand @@ -69,136 +45,89 @@ detect if the current branch's changes will conflict during the CE->EE merge. The job reports what files are conflicting and how to set up a merge request against EE. -#### How the job works - -1. Generates the diff between your branch and current CE `master` -1. Tries to apply it to current EE `master` -1. If it applies cleanly, the job succeeds, otherwise... -1. Detects a branch with the `ee-` prefix or `-ee` suffix in EE -1. If it exists, generate the diff between this branch and current EE `master` -1. Tries to apply it to current EE `master` -1. If it applies cleanly, the job succeeds - -In the case where the job fails, it means you should create an `ee-<ce_branch>` -or `<ce_branch>-ee` branch, push it to EE and open a merge request against EE -`master`. -At this point if you retry the failing job in your CE merge request, it should -now pass. - -Notes: - -- This task is not a silver-bullet, its current goal is to bring awareness to - developers that their work needs to be ported to EE. -- Community contributors shouldn't be required to submit merge requests against - EE, but reviewers should take actions by either creating such EE merge request - or asking a GitLab developer to do it **before the merge request is merged**. -- If you branch is too far behind `master`, the job will fail. In that case you - should rebase your branch upon latest `master`. -- Code reviews for merge requests often consist of multiple iterations of - feedback and fixes. There is no need to update your EE MR after each - iteration. Instead, create an EE MR as soon as you see the - `ee_compat_check` job failing. After you receive the final approval - from a Maintainer (but **before the CE MR is merged**) update the EE MR. - This helps to identify significant conflicts sooner, but also reduces the - number of times you have to resolve conflicts. -- Please remember to - [always have your EE merge request merged before the CE version](#always-merge-ee-merge-requests-before-their-ce-counterparts). -- You can use [`git rerere`](https://git-scm.com/docs/git-rerere) - to avoid resolving the same conflicts multiple times. - -### Cherry-picking from CE to EE - -For avoiding merge conflicts, we use a method of creating equivalent branches -for CE and EE. If the `ee-compat-check` job fails, this process is required. - -This method only requires that you have cloned both CE and EE into your computer. -If you don't have them yet, please go ahead and clone them: - -- Clone CE repo: `git clone git@gitlab.com:gitlab-org/gitlab-ce.git` -- Clone EE repo: `git clone git@gitlab.com:gitlab-org/gitlab-ee.git` - -And the only additional setup we need is to add CE as remote of EE and vice-versa: - -- Open two terminal windows, one in CE, and another one in EE: - - In EE: `git remote add ce git@gitlab.com:gitlab-org/gitlab-ce.git` - - In CE: `git remote add ee git@gitlab.com:gitlab-org/gitlab-ee.git` - -That's all setup we need, so that we can cherry-pick a commit from CE to EE, and -from EE to CE. - -Now, every time you create an MR for CE and EE: - -1. Open two terminal windows, one in CE, and another one in EE -1. In the CE terminal: - 1. Create the CE branch, e.g., `branch-example` - 1. Make your changes and push a commit (commit A) - 1. Create the CE merge request in GitLab -1. In the EE terminal: - 1. Create the EE-equivalent branch ending with `-ee`, e.g., - `git checkout -b branch-example-ee` - 1. Fetch the CE branch: `git fetch ce branch-example` - 1. Cherry-pick the commit A: `git cherry-pick commit-A-SHA` - 1. If Git prompts you to fix the conflicts, do a `git status` - to check which files contain conflicts, fix them, save the files - 1. Add the changes with `git add .` but **DO NOT commit** them - 1. Continue cherry-picking: `git cherry-pick --continue` - 1. Push to EE: `git push origin branch-example-ee` -1. Create the EE-equivalent MR and link to the CE MR from the -description "Ports [CE-MR-LINK] to EE" -1. Once all the jobs are passing in both CE and EE, you've addressed the -feedback from your own team, and got them approved, the merge requests can be merged. -1. When both MRs are ready, the EE merge request will be merged first, and the -CE-equivalent will be merged next. - -**Important notes:** - -- The commit SHA can be easily found from the GitLab UI. From a merge request, -open the tab **Commits** and click the copy icon to copy the commit SHA. -- To cherry-pick a **commit range**, such as [A > B > C > D] use: - - ```shell - git cherry-pick "oldest-commit-SHA^..newest-commit-SHA" - ``` - - For example, suppose the commit A is the oldest, and its SHA is `4f5e4018c09ed797fdf446b3752f82e46f5af502`, - and the commit D is the newest, and its SHA is `80e1c9e56783bd57bd7129828ec20b252ebc0538`. - The cherry-pick command will be: - - ```shell - git cherry-pick "4f5e4018c09ed797fdf446b3752f82e46f5af502^..80e1c9e56783bd57bd7129828ec20b252ebc0538" - ``` - -- To cherry-pick a **merge commit**, use the flag `-m 1`. For example, suppose that the -merge commit SHA is `138f5e2f20289bb376caffa0303adb0cac859ce1`: - - ```shell - git cherry-pick -m 1 138f5e2f20289bb376caffa0303adb0cac859ce1 - ``` -- To cherry-pick multiple commits, such as B and D in a range [A > B > C > D], use: - - ```shell - git cherry-pick commmit-B-SHA commit-D-SHA - ``` - - For example, suppose commit B SHA = `4f5e4018c09ed797fdf446b3752f82e46f5af502`, - and the commit D SHA = `80e1c9e56783bd57bd7129828ec20b252ebc0538`. - The cherry-pick command will be: - - ```shell - git cherry-pick 4f5e4018c09ed797fdf446b3752f82e46f5af502 80e1c9e56783bd57bd7129828ec20b252ebc0538 - ``` - - This case is particularly useful when you have a merge commit in a sequence of - commits and you want to cherry-pick all but the merge commit. - -- If you push more commits to the CE branch, you can safely repeat the procedure -to cherry-pick them to the EE-equivalent branch. You can do that as many times as -necessary, using the same CE and EE branches. -- If you submitted the merge request to the CE repo and the `ee-compat-check` job passed, -you are not required to submit the EE-equivalent MR, but it's still recommended. If the -job failed, you are required to submit the EE MR so that you can fix the conflicts in EE -before merging your changes into CE. - ---- - -[Return to Development documentation](README.md) +## How to reinstate changes + +When a commit is reverted, the corresponding merge request to reinstate the +changes will include all the details necessary to ensure the changes make it +back into CE and EE. However, you still need to manually set up an EE merge +request that resolves the conflicts. + +Each merge request used to reinstate changes will have the "reverted" label +applied. Please do not remove this label, as it will be used to determine how +many times commits are reverted and how long it takes to reinstate the changes. + +An example merge request can be found in [CE merge request +23280](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/23280). + +## How it works + +The automatic merging is performed using a project called [Merge +Train](https://gitlab.com/gitlab-org/merge-train/). For every commit to merge or +revert, we generate patches using `git format-patch` which we then try to apply +using `git am --3way`. If this succeeds we push the changes to EE, if this fails +we decide what to do based on the failure reason: + +1. If the patch could not be applied because it was already applied, we just + skip it. +1. If the patch caused conflicts, we revert the source commits. + +Commits are reverted in reverse order, ensuring that if commit B depends on A, +and both conflict, we first revert B followed by reverting A. + +## FAQ + +### Why? + +We want to work towards being able to deploy continuously, but this requires +that `master` is always stable and has all the changes we need. If CE `master` +can not be merged into EE `master` due to merge conflicts, this prevents _any_ +change from CE making its way into EE. Since GitLab.com runs on EE, this +effectively prevents us from deploying changes. + +Past experiences and data have shown that periodic CE to EE merge requests do +not scale, and often take a very long time to complete. For example, [in this +comment](https://gitlab.com/gitlab-org/release/framework/issues/49#note_114614619) +we determined that the average time to close an upstream merge request is around +5 hours, with peaks up to several days. Periodic merge requests are also +frustrating to work with, because they often include many changes unrelated to +your own changes. + +Automatically merging or reverting commits allows us to keep merging changes +from CE into EE, as we never have to wait hours for somebody to resolve a set of +merge conflicts. + +### Does the CE to EE merge take into account merge commits? + +No. When merging CE changes into EE, merge commits are ignored. + +### My changes are reverted, but I set up an EE MR to resolve conflicts + +Most likely the automatic merge job ran before the EE merge request was merged. +If this keeps happening, consider reporting a bug in the [Merge Train issue +tracker](https://gitlab.com/gitlab-org/merge-train/issues). + +### My changes keep getting reverted, and this is really annoying! + +This is understandable, but the solution to this is fairly straightforward: +simply set up an EE merge request for every CE merge request, and resolve your +conflicts before the changes are reverted. + +### Will we allow certain people to still merge changes, even if they conflict? + +No. + +### Some files I work with often conflict, how can I best deal with this? + +If you find you keep running into merge conflicts, consider refactoring the file +so that the EE specific changes are not intertwined with CE code. For Ruby code +you can do this by moving the EE code to a separate module, which can then be +injected into the appropriate classes or modules. See [Guidelines for +implementing Enterprise Edition features](ee_features.md) for more information. + +### Will changelog entries be reverted automatically? + +Only if the changelog was added in the commit that was reverted. If a changelog +entry was added in a separate commit, it is possible for it to be left behind. +Since changelog entries are related to the changes in question, there is no real +reason to commit the changelog separately, and as such this should not be a big +problem. diff --git a/doc/development/code_review.md b/doc/development/code_review.md index 52710e54e86..7788d155154 100644 --- a/doc/development/code_review.md +++ b/doc/development/code_review.md @@ -5,7 +5,7 @@ having your code reviewed. All merge requests for GitLab CE and EE, whether written by a GitLab team member or a volunteer contributor, must go through a code review process to ensure the -code is effective, understandable, and maintainable. +code is effective, understandable, maintainable, and secure. ## Getting your merge request reviewed, approved, and merged @@ -20,12 +20,24 @@ importance of involving reviewer(s) in the section on the responsibility of the If you need some guidance (e.g. it's your first merge request), feel free to ask one of the [Merge request coaches][team]. +If you need assistance with security scans or comments, feel free to include the +Security Team (`@gitlab-com/gl-security`) in the review. + Depending on the areas your merge request touches, it must be **approved** by one or more [maintainers](https://about.gitlab.com/handbook/engineering/#maintainer): For approvals, we use the approval functionality found in the merge request widget. Reviewers can add their approval by [approving additionally](https://docs.gitlab.com/ee/user/project/merge_requests/merge_request_approvals.html#adding-or-removing-an-approval). +Getting your merge request **merged** also requires a maintainer. If it requires +more than one approval, the last maintainer to review and approve it will also merge it. + +### Approval guidelines + +As described in the section on the responsibility of the maintainer below, you +are recommended to get your merge request approved and merged by maintainer(s) +from teams other than your own. + 1. If your merge request includes backend changes [^1], it must be **approved by a [backend maintainer](https://about.gitlab.com/handbook/engineering/projects/#gitlab-ce_maintainers_backend)**. 1. If your merge request includes frontend changes [^1], it must be @@ -39,12 +51,14 @@ widget. Reviewers can add their approval by [approving additionally](https://doc 1. If your merge request includes a new dependency or a filesystem change, it must be **approved by a [Distribution team member][team]**. See how to work with the [Distribution team](https://about.gitlab.com/handbook/engineering/dev-backend/distribution/) for more details. -Getting your merge request **merged** also requires a maintainer. If it requires -more than one approval, the last maintainer to review and approve it will also merge it. +#### Security requirements -As described in the section on the responsibility of the maintainer below, you -are recommended to get your merge request approved and merged by maintainer(s) -from other teams than your own. + 1. If your merge request is processing, storing, or transferring any kind of [RED or ORANGE data](https://docs.google.com/document/d/15eNKGA3zyZazsJMldqTBFbYMnVUSQSpU14lo22JMZQY/edit) (this is a confidential document), it must be + **approved by a [Security Engineer][team]**. + 1. If your merge request involves implementing, utilizing, or is otherwise related to any type of authentication, authorization, or session handling mechanism, it must be + **approved by a [Security Engineer][team]**. + 1. If your merge request has a goal which requires a cryptographic function such as: confidentiality, integrity, authentication, or non-repudiation, it must be + **approved by a [Security Engineer][team]**. ### The responsibility of the merge request author @@ -54,9 +68,10 @@ merge request author. Before assigning a merge request to a maintainer for approval and merge, they should be confident that it actually solves the problem it was meant to solve, that it does so in the most appropriate way, that it satisfies all requirements, -and that there are no remaining bugs, logical problems, or uncovered edge cases. -The merge request should also have a completed task list in its description and -a passing CI pipeline to avoid unnecessary back and forth. +and that there are no remaining bugs, logical problems, uncovered edge cases, +or known vulnerabilities. The merge request should also have a completed task +list in its description and a passing CI pipeline to avoid unnecessary back and +forth. To reach the required level of confidence in their solution, an author is expected to involve other people in the investigation and implementation processes as @@ -72,6 +87,23 @@ If an author is unsure if a merge request needs a domain expert's opinion, that' usually a pretty good sign that it does, since without it the required level of confidence in their solution will not have been reached. +Before the review, the author is requested to submit comments on the merge +request diff alerting the reviewer to anything important as well as for anything +that demands further explanation or attention. Examples of content that may +warrant a comment could be: + +- The addition of a linting rule (Rubocop, JS etc) +- The addition of a library (Ruby gem, JS lib etc) +- Where not obvious, a link to the parent class or method +- Any benchmarking performed to complement the change +- Potentially insecure code + +Do not add these comments directly to the source code, unless the +reviewer requires you to do so. + +This +[saves reviewers time and helps authors catch mistakes earlier](https://www.ibm.com/developerworks/rational/library/11-proven-practices-for-peer-review/index.html#__RefHeading__97_174136755). + ### The responsibility of the maintainer Maintainers are responsible for the overall health, quality, and consistency of @@ -85,8 +117,8 @@ Since a maintainer's job only depends on their knowledge of the overall GitLab codebase, and not that of any specific domain, they can review, approve and merge merge requests from any team and in any product area. -In fact, authors are recommended to get their merge requests merged by maintainers -from other teams than their own, to ensure that all code across GitLab is consistent +In fact, authors are encouraged to get their merge requests merged by maintainers +from teams other than their own, to ensure that all code across GitLab is consistent and can be easily understood by all contributors, from both inside and outside the company, without requiring team-specific expertise. @@ -103,6 +135,18 @@ as the maintainer to ultimately approve and merge it. Maintainers should check before merging if the merge request is approved by the required approvers. +Maintainers must check before merging if the merge request is introducing new +vulnerabilities, by inspecting the list in the Merge Request [Security +Widget](https://docs.gitlab.com/ee/user/project/merge_requests/#security-reports-ultimate). +When in doubt, a [Security Engineer][team] can be involved. The list of detected +vulnerabilities must be either empty or containing: + +- dismissed vulnerabilities in case of false positives +- vulnerabilities converted to issues + +Maintainers should **never** dismiss vulnerabilities to "empty" the list, +without duly verifying them. + ## Best practices ### Everyone @@ -153,7 +197,7 @@ first time. ### Assigning a merge request for a review -If you want to have your merge request reviewed you can assign it to any reviewer. The list of reviewers can be found on [Engineering projects](https://about.gitlab.com/handbook/engineering/projects/) page. +If you want to have your merge request reviewed, you can assign it to any reviewer. The list of reviewers can be found on [Engineering projects](https://about.gitlab.com/handbook/engineering/projects/) page. You can also use `ready for review` label. That means that your merge request is ready to be reviewed and any reviewer can pick it. It is recommended to use that label only if there isn't time pressure and make sure the merge request is assigned to a reviewer. @@ -189,6 +233,9 @@ experience, refactors the existing code). Then: subsequent revisions for anything that would be spotted after that. - Consider using the [Squash and merge][squash-and-merge] feature when the merge request has a lot of commits. + When merging code a maintainer should only use the squash feature if the + author has already set this option or if the merge request clearly contains a + messy commit history that is intended to be squashed. [squash-and-merge]: https://docs.gitlab.com/ee/user/project/merge_requests/squash_and_merge.html#squash-and-merge diff --git a/doc/development/documentation/index.md b/doc/development/documentation/index.md index b7990e1b558..55aed023325 100644 --- a/doc/development/documentation/index.md +++ b/doc/development/documentation/index.md @@ -368,6 +368,16 @@ You can combine one or more of the following: = link_to 'Help page', help_page_path('user/permissions') ``` +### GitLab `/help` tests + +Several [rspec tests](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/spec/features/help_pages_spec.rb) +are run to ensure GitLab documentation renders and works correctly. In particular, that [main docs landing page](../../README.md) will work correctly from `/help`. +For example, [GitLab.com's `/help`](https://gitlab.com/help). + +CAUTION: **Caution:** +Because the rspec tests only run in a full pipeline, and not a special [docs-only pipeline](#branch-naming), it is possible +to merge changes that will break `master` from a merge request with a successful docs-only pipeline run. + ## General Documentation vs Technical Articles ### General documentation @@ -552,6 +562,7 @@ Currently, the following tests are in place: As CE is merged into EE once a day, it's important to avoid merge conflicts. Submitting an EE-equivalent merge request cherry-picking all commits from CE to EE is essential to avoid them. +1. In a full pipeline, tests for [`/help`](#gitlab-help-tests). ### Linting diff --git a/doc/development/ee_features.md b/doc/development/ee_features.md index a227e2f6640..790b1bf951b 100644 --- a/doc/development/ee_features.md +++ b/doc/development/ee_features.md @@ -511,7 +511,7 @@ module EE params do requires :id, type: String, desc: 'The ID of a project' end - resource :projects, requirements: ::API::API::PROJECT_ENDPOINT_REQUIREMENTS do + resource :projects, requirements: ::API::API::NAMESPACE_OR_PROJECT_REQUIREMENTS do # ... end end diff --git a/doc/development/fe_guide/graphql.md b/doc/development/fe_guide/graphql.md new file mode 100644 index 00000000000..f55f01720fd --- /dev/null +++ b/doc/development/fe_guide/graphql.md @@ -0,0 +1,83 @@ +# GraphQL + +We use [Apollo] and [Vue Apollo][vue-apollo] for working with GraphQL +on the frontend. + +In order to use GraphQL, you need to enable the `graphql` feature flag, +read more about [Feature Flags][feature-flags]. + +## Apollo Client + +To save duplicated clients getting created in different apps, we have a +[default client][defualt-client] that should be used. This setups the +Apollo client with the correct URL and also sets the CSRF headers. + +## GraphQL Queries + +To save query compilation at runtime, webpack can directly import `.graphql` +files. This allows webpack to preprocess the query at compile time instead +of the client doing compilation of queries. + +## Usage in Vue + +To use Vue Apollo, import the [Vue Apollo][vue-apollo] plugin as well +as the default client. This should be created at the same point +the Vue application is mounted. + +```javascript +import Vue from 'vue'; +import VueApollo from 'vue-apollo'; +import defaultClient from '~/lib/graphql'; +Vue.use(VueApollo); + +const apolloProvider = new VueApollo({ + defaultClient, +}); + +new Vue({ + ..., + apolloProvider, + ... +}); +``` + +Read more about [Vue Apollo][vue-apollo] in the [Vue Apollo documentation][vue-apollo-docs]. + +### Testing + +With [Vue test utils][vue-test-utils] it is easy to quickly test components that +fetch GraphQL queries. The simplest way is to use `shallowMount` and then set +the data on the component + +```javascript +it('tests apollo component', () => { + const vm = shallowMount(App); + + vm.setData({ + ...mock data + }); +}); +``` + +## Usage outside of Vue + +It is also possible to use GraphQL outside of Vue by directly importing +and using the default client with queries. + +```javascript +import defaultClient from '~/lib/graphql'; +import query from './query.graphql'; + +defaultClient.query(query) + .then(result => console.log(result)); +``` + +Read more about the [Apollo] client in the [Apollo documentation][apollo-client-docs]. + +[Apollo]: https://www.apollographql.com/ +[vue-apollo]: https://github.com/Akryum/vue-apollo/ +[vue-apollo-docs]: https://akryum.github.io/vue-apollo/ +[feature-flags]: ../feature_flags.md +[default-client]: https://gitlab.com/gitlab-org/gitlab-ce/blob/master/app/assets/javascripts/lib/graphql.js +[apollo-client-docs]: https://www.apollographql.com/docs/tutorial/client.html +[vue-test-utils]: https://vue-test-utils.vuejs.org/ diff --git a/doc/development/fe_guide/index.md b/doc/development/fe_guide/index.md index 11b9a2e6a64..cca3ad6fae6 100644 --- a/doc/development/fe_guide/index.md +++ b/doc/development/fe_guide/index.md @@ -54,6 +54,9 @@ Vuex specific design patterns and practices. ## [Axios](axios.md) Axios specific practices and gotchas. +## [GraphQL](graphql.md) +How to use GraphQL + ## [Icons and Illustrations](icons.md) How we use SVG for our Icons and Illustrations. diff --git a/doc/development/feature_flags.md b/doc/development/feature_flags.md index 1019a1fd0e2..b6161cd6163 100644 --- a/doc/development/feature_flags.md +++ b/doc/development/feature_flags.md @@ -113,7 +113,15 @@ feature flag. You can stub a feature flag as follows: stub_feature_flags(my_feature_flag: false) ``` -## Enabling a feature flag +## Enabling a feature flag (in development) + +In the rails console (`rails c`), enter the following command to enable your feature flag + +```ruby +Feature.enable(:feature_flag_name) +``` + +## Enabling a feature flag (in production) Check how to [roll out changes using feature flags](rolling_out_changes_using_feature_flags.md). diff --git a/doc/development/github_importer.md b/doc/development/github_importer.md index e860bde48dc..863ac049db6 100644 --- a/doc/development/github_importer.md +++ b/doc/development/github_importer.md @@ -131,7 +131,7 @@ our import as failed because of this. To prevent this from happening we periodically refresh the expiration time of the import process. This works by storing the JID of the import job in the database, then refreshing this JID's TTL at various stages throughout the import -process. This is done by calling `Project#refresh_import_jid_expiration`. By +process. This is done by calling `ProjectImportState#refresh_jid_expiration`. By refreshing this TTL we can ensure our import does not get marked as failed so long we're still performing work. diff --git a/doc/development/gotchas.md b/doc/development/gotchas.md index 84dea7ce9aa..a5a34d82bec 100644 --- a/doc/development/gotchas.md +++ b/doc/development/gotchas.md @@ -92,7 +92,7 @@ describe API::Labels do end ``` -## Avoid using `allow_any_instance_of` in RSpec +## Avoid using `expect_any_instance_of` or `allow_any_instance_of` in RSpec ### Why @@ -102,7 +102,7 @@ end error like this: ``` - 1.1) Failure/Error: allow_any_instance_of(ApplicationSetting).to receive_messages(messages) + 1.1) Failure/Error: expect_any_instance_of(ApplicationSetting).to receive_messages(messages) Using `any_instance` to stub a method (elasticsearch_indexing) that has been defined on a prepended module (EE::ApplicationSetting) is not supported. ``` @@ -112,7 +112,7 @@ Instead of writing: ```ruby # Don't do this: -allow_any_instance_of(Project).to receive(:add_import_job) +expect_any_instance_of(Project).to receive(:add_import_job) ``` We could write: diff --git a/doc/development/i18n/proofreader.md b/doc/development/i18n/proofreader.md index f75f5882b56..8f1317e235d 100644 --- a/doc/development/i18n/proofreader.md +++ b/doc/development/i18n/proofreader.md @@ -43,7 +43,7 @@ are very appreciative of the work done by translators and proofreaders! - Greek - Proofreaders needed. - Hebrew - - Proofreaders needed. + - Yaron Shahrabani - [GitLab](https://gitlab.com/yarons), [Crowdin](https://crowdin.com/profile/YaronSh) - Hungarian - Proofreaders needed. - Indonesian @@ -57,12 +57,14 @@ are very appreciative of the work done by translators and proofreaders! - Chang-Ho Cha - [GitLab](https://gitlab.com/changho-cha), [Crowdin](https://crowdin.com/profile/zzazang) - Huang Tao - [GitLab](https://gitlab.com/htve), [Crowdin](https://crowdin.com/profile/htve) - Ji Hun Oh - [GitLab](https://gitlab.com/Baw-Appie), [Crowdin](https://crowdin.com/profile/BawAppie) + - Jeongwhan Choi - [GitLab](https://gitlab.com/jeongwhanchoi), [Crowdin](https://crowdin.com/profile/jeongwhanchoi) - Mongolian - Proofreaders needed. - Norwegian Bokmal - Proofreaders needed. - Polish - Filip Mech - [GitLab](https://gitlab.com/mehenz), [Crowdin](https://crowdin.com/profile/mehenz) + - Maksymilian Roman - [GitLab](https://gitlab.com/villaincandle), [Crowdin](https://crowdin.com/profile/villaincandle) - Portuguese - Proofreaders needed. - Portuguese, Brazilian diff --git a/doc/development/new_fe_guide/development/testing.md b/doc/development/new_fe_guide/development/testing.md index 082acbedcd2..0d98180add0 100644 --- a/doc/development/new_fe_guide/development/testing.md +++ b/doc/development/new_fe_guide/development/testing.md @@ -6,9 +6,15 @@ Tests relevant for frontend development can be found at two places: - [frontend unit tests](#frontend-unit-tests) - [frontend component tests](#frontend-component-tests) - [frontend integration tests](#frontend-integration-tests) +- `spec/frontend/` which are run by Jest and contain + - [frontend unit tests](#frontend-unit-tests) + - [frontend component tests](#frontend-component-tests) + - [frontend integration tests](#frontend-integration-tests) - `spec/features/` which are run by RSpec and contain - [feature tests](#feature-tests) +All tests in `spec/javascripts/` will eventually be migrated to `spec/frontend/` (see also [#53757]). + In addition there were feature tests in `features/` run by Spinach in the past. These have been removed from our codebase in May 2018 ([#23036](https://gitlab.com/gitlab-org/gitlab-ce/issues/23036)). @@ -17,6 +23,8 @@ See also: - [old testing guide](../../testing_guide/frontend_testing.html) - [notes on testing Vue components](../../fe_guide/vue.html#testing-vue-components) +[#53757]: https://gitlab.com/gitlab-org/gitlab-ce/issues/53757 + ## Frontend unit tests Unit tests are on the lowest abstraction level and typically test functionality that is not directly perceivable by a user. diff --git a/doc/development/profiling.md b/doc/development/profiling.md index 0ca8bb67a77..0b0c6dfc8cf 100644 --- a/doc/development/profiling.md +++ b/doc/development/profiling.md @@ -77,8 +77,11 @@ that builds on this to add some additional niceties, such as allowing configuration with a single Yaml file for multiple URLs, and uploading of the profile and log output to S3. -For GitLab.com, you can find the latest results here: -<http://redash.gitlab.com/dashboard/gitlab-profiler-statistics> +For GitLab.com, currently the latest profiling data has been [moved from +Redash to Looker](https://gitlab.com/gitlab-com/Product/issues/5#note_121194467). +We are [currently investigating how to make this data +public](https://gitlab.com/meltano/looker/issues/294). + ## Sherlock diff --git a/doc/development/testing_guide/ci.md b/doc/development/testing_guide/ci.md index 8d9706a9501..d685cacf9ea 100644 --- a/doc/development/testing_guide/ci.md +++ b/doc/development/testing_guide/ci.md @@ -31,11 +31,7 @@ After that, the next pipeline will use the up-to-date The GitLab test suite is [monitored] for the `master` branch, and any branch that includes `rspec-profile` in their name. -A [public dashboard] is available for everyone to see. Feel free to look at the -slowest test files and try to improve them. - [monitored]: ../performance.md#rspec-profiling -[public dashboard]: https://redash.gitlab.com/public/dashboards/l1WhHXaxrCWM5Ai9D7YDqHKehq6OU3bx5gssaiWe?org_slug=default ## CI setup diff --git a/doc/development/testing_guide/review_apps.md b/doc/development/testing_guide/review_apps.md index 1830641431e..309babb5f94 100644 --- a/doc/development/testing_guide/review_apps.md +++ b/doc/development/testing_guide/review_apps.md @@ -7,31 +7,36 @@ Review Apps are automatically deployed by each pipeline, both in ## How does it work? 1. On every [pipeline][gitlab-pipeline] during the `test` stage, the - [`review` job][review-job] is automatically started. -1. The `review` job [triggers a pipeline][cng-pipeline] in the - [`CNG-mirror`][cng-mirror] project. - - We use the `CNG-mirror` project so that the `CNG`, (**C**loud **N**ative - **G**itLab), project's registry is not overloaded with a lot of transient - Docker images. -1. The `CNG-mirror` pipeline creates the Docker images of each component (e.g. - `gitlab-rails-ee`, `gitlab-shell`, `gitaly` etc.) based on the commit from the - [GitLab pipeline][gitlab-pipeline] and store them in its - [registry][cng-mirror-registry]. -1. Once all images are built, the Review App is deployed using - [the official GitLab Helm chart][helm-chart] to the - [`review-apps-ee` Kubernetes cluster on GCP][review-apps-ee] - - The actual scripts used to deploy the Review App can be found at - [`scripts/review_apps/review-apps.sh`][review-apps.sh] - - These scripts are basically - [our official Auto DevOps scripts][Auto-DevOps.gitlab-ci.yml] where the - default CNG images are overridden with the images built and stored in the - [`CNG-mirror` project's registry][cng-mirror-registry]. - - Since we're using [the official GitLab Helm chart][helm-chart], this means - you get a dedicated environment for your branch that's very close to what it - would look in production. -1. Once the `review` job succeeds, you should be able to use your Review App - thanks to the direct link to it from the MR widget. The default username is - `root` and its password can be found in the 1Password secure note named + [`review-deploy`][review-deploy-job] job is automatically started. +1. The `review-deploy` job: + 1. Waits for the `gitlab:assets:compile` job to finish since the + [`CNG-mirror`][cng-mirror] pipeline triggerred in the following step + depends on it. + 1. [Triggers a pipeline][cng-pipeline] in the [`CNG-mirror`][cng-mirror] + project. + - We use the `CNG-mirror` project so that the `CNG`, (**C**loud + **N**ative **G**itLab), project's registry is not overloaded with a + lot of transient Docker images. + - The `CNG-mirror` pipeline creates the Docker images of each component + (e.g. `gitlab-rails-ee`, `gitlab-shell`, `gitaly` etc.) based on the + commit from the [GitLab pipeline][gitlab-pipeline] and store them in + its [registry][cng-mirror-registry]. + 1. Once all images are built by [`CNG-mirror`][cng-mirror], the Review App + is deployed using [the official GitLab Helm chart][helm-chart] to the + [`review-apps-ce`][review-apps-ce] / [`review-apps-ee`][review-apps-ee] + Kubernetes cluster on GCP. + - The actual scripts used to deploy the Review App can be found at + [`scripts/review_apps/review-apps.sh`][review-apps.sh]. + - These scripts are basically + [our official Auto DevOps scripts][Auto-DevOps.gitlab-ci.yml] where the + default CNG images are overridden with the images built and stored in the + [`CNG-mirror` project's registry][cng-mirror-registry]. + - Since we're using [the official GitLab Helm chart][helm-chart], this means + you get a dedicated environment for your branch that's very close to what + it would look in production. +1. Once the `review-deploy` job succeeds, you should be able to use your Review + App thanks to the direct link to it from the MR widget. The default username + is `root` and its password can be found in the 1Password secure note named **gitlab-{ce,ee} Review App's root password** (note that there's currently [a bug where the default password seems to be overridden][password-bug]). @@ -39,16 +44,58 @@ Review Apps are automatically deployed by each pipeline, both in - The Kubernetes cluster is connected to the `gitlab-{ce,ee}` projects using [GitLab's Kubernetes integration][gitlab-k8s-integration]. This basically - allows to have a link to the Review App directly from the merge request widget. -- The manual `stop_review` in the `test` stage can be used to stop a Review App - manually, and is also started by GitLab once a branch is deleted. -- Review Apps are cleaned up regularly using a pipeline schedule that runs - the [`scripts/review_apps/automated_cleanup.rb`][automated_cleanup.rb] script. + allows to have a link to the Review App directly from the merge request + widget. - If the Review App deployment fails, you can simply retry it (there's no need - to run the `stop_review` job first). -- If you're unable to log in using the `root` username and password, you may - encounter [this bug][password-bug]. Stop the Review App via the `stop_review` - manual job and then retry the `review` job to redeploy the Review App. + to run the [`review-stop`][gitlab-ci-yml] job first). +- The manual [`review-stop`][gitlab-ci-yml] in the `test` stage can be used to + stop a Review App manually, and is also started by GitLab once a branch is + deleted. +- Review Apps are cleaned up regularly using a pipeline schedule that runs + the [`schedule:review-cleanup`][gitlab-ci-yml] job. + +## QA runs + +On every [pipeline][gitlab-pipeline] during the `test` stage, the +`review-qa-smoke` job is automatically started: it runs the smoke QA suite. +You can also manually start the `review-qa-all`: it runs the full QA suite. + +Note that both jobs first wait for the `review-deploy` job to be finished. + +## How to? + +### Find my Review App slug? + +1. Open the `review-deploy` job. +1. Look for `Checking for previous deployment of review-*`. +1. For instance for `Checking for previous deployment of review-qa-raise-e-12chm0`, + your Review App slug would be `review-qa-raise-e-12chm0` in this case. + +### Run a Rails console? + +1. [Filter Workloads by your Review App slug](https://console.cloud.google.com/kubernetes/workload?project=gitlab-review-apps) + , e.g. `review-29951-issu-id2qax`. +1. Find and open the `task-runner` Deployment, e.g. `review-29951-issu-id2qax-task-runner`. +1. Click on the Pod in the "Managed pods" section, e.g. `review-29951-issu-id2qax-task-runner-d5455cc8-2lsvz`. +1. Click on the `KUBECTL` dropdown, then `Exec` -> `task-runner`. +1. Replace `-c task-runner -- ls` with `-- /srv/gitlab/bin/rails c` from the + default command or + - Run `kubectl exec --namespace review-apps-ce -it review-29951-issu-id2qax-task-runner-d5455cc8-2lsvz -- /srv/gitlab/bin/rails c` + and + - Replace `review-apps-ce` with `review-apps-ee` if the Review App + is running EE, and + - Replace `review-29951-issu-id2qax-task-runner-d5455cc8-2lsvz` + with your Pod's name. + +### Dig into a Pod's logs? + +1. [Filter Workloads by your Review App slug](https://console.cloud.google.com/kubernetes/workload?project=gitlab-review-apps) + , e.g. `review-1979-1-mul-dnvlhv`. +1. Find and open the `migrations` Deployment, e.g. + `review-1979-1-mul-dnvlhv-migrations.1`. +1. Click on the Pod in the "Managed pods" section, e.g. + `review-1979-1-mul-dnvlhv-migrations.1-nqwtx`. +1. Click on the `Container logs` link. ## Frequently Asked Questions @@ -74,15 +121,17 @@ find a way to limit it to only us.** > This isn't enabled for forks. [gitlab-pipeline]: https://gitlab.com/gitlab-org/gitlab-ce/pipelines/35850709 -[review-job]: https://gitlab.com/gitlab-org/gitlab-ce/-/jobs/118076368 +[review-deploy-job]: https://gitlab.com/gitlab-org/gitlab-ce/-/jobs/118076368 [cng-mirror]: https://gitlab.com/gitlab-org/build/CNG-mirror [cng-pipeline]: https://gitlab.com/gitlab-org/build/CNG-mirror/pipelines/35883435 [cng-mirror-registry]: https://gitlab.com/gitlab-org/build/CNG-mirror/container_registry [helm-chart]: https://gitlab.com/charts/gitlab/ +[review-apps-ce]: https://console.cloud.google.com/kubernetes/clusters/details/us-central1-a/review-apps-ce?project=gitlab-review-apps [review-apps-ee]: https://console.cloud.google.com/kubernetes/clusters/details/us-central1-b/review-apps-ee?project=gitlab-review-apps [review-apps.sh]: https://gitlab.com/gitlab-org/gitlab-ee/blob/master/scripts/review_apps/review-apps.sh [automated_cleanup.rb]: https://gitlab.com/gitlab-org/gitlab-ee/blob/master/scripts/review_apps/automated_cleanup.rb [Auto-DevOps.gitlab-ci.yml]: https://gitlab.com/gitlab-org/gitlab-ce/blob/master/lib/gitlab/ci/templates/Auto-DevOps.gitlab-ci.yml +[gitlab-ci-yml]: https://gitlab.com/gitlab-org/gitlab-ce/blob/master/.gitlab-ci.yml [gitlab-k8s-integration]: https://docs.gitlab.com/ee/user/project/clusters/index.html [password-bug]: https://gitlab.com/gitlab-org/gitlab-ce/issues/53621 diff --git a/doc/install/docker.md b/doc/install/docker.md index c7dc9db70c5..e90f6645b0c 100644 --- a/doc/install/docker.md +++ b/doc/install/docker.md @@ -7,9 +7,10 @@ GitLab provides official Docker images to allowing you to easily take advantage ## Omnibus GitLab based images GitLab maintains a set of [official Docker images](https://hub.docker.com/r/gitlab) based on our [Omnibus GitLab package](https://docs.gitlab.com/omnibus/README.html). These images include: -* [GitLab Community Edition](https://hub.docker.com/r/gitlab/gitlab-ce/) -* [GitLab Enterprise Edition](https://hub.docker.com/r/gitlab/gitlab-ee/) -* [GitLab Runner](https://hub.docker.com/r/gitlab/gitlab-runner/) + +- [GitLab Community Edition](https://hub.docker.com/r/gitlab/gitlab-ce/). +- [GitLab Enterprise Edition](https://hub.docker.com/r/gitlab/gitlab-ee/). +- [GitLab Runner](https://hub.docker.com/r/gitlab/gitlab-runner/). A [complete usage guide](https://docs.gitlab.com/omnibus/docker/) to these images is available, as well as the [Dockerfile used for building the images](https://gitlab.com/gitlab-org/omnibus-gitlab/tree/master/docker). diff --git a/doc/install/installation.md b/doc/install/installation.md index 06293b8caf5..d041bfa863a 100644 --- a/doc/install/installation.md +++ b/doc/install/installation.md @@ -12,7 +12,7 @@ Since installations from source don't have Runit, Sidekiq can't be terminated an ## Select Version to Install -Make sure you view [this installation guide](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/doc/install/installation.md) from the branch (version) of GitLab you would like to install (e.g., `11-5-stable`). +Make sure you view [this installation guide](https://gitlab.com/gitlab-org/gitlab-ce/blob/master/doc/install/installation.md) from the branch (version) of GitLab you would like to install (e.g., `11-6-stable`). You can select the branch in the version dropdown in the top left corner of GitLab (below the menu bar). If the highest number stable branch is unclear please check the [GitLab Blog](https://about.gitlab.com/blog/) for installation guide links by version. @@ -300,9 +300,9 @@ sudo usermod -aG redis git ### Clone the Source # Clone GitLab repository - sudo -u git -H git clone https://gitlab.com/gitlab-org/gitlab-ce.git -b 11-5-stable gitlab + sudo -u git -H git clone https://gitlab.com/gitlab-org/gitlab-ce.git -b 11-6-stable gitlab -**Note:** You can change `11-5-stable` to `master` if you want the *bleeding edge* version, but never install master on a production server! +**Note:** You can change `11-6-stable` to `master` if you want the *bleeding edge* version, but never install master on a production server! ### Configure It diff --git a/doc/raketasks/backup_restore.md b/doc/raketasks/backup_restore.md index 18c9cd116f1..a63656fafef 100644 --- a/doc/raketasks/backup_restore.md +++ b/doc/raketasks/backup_restore.md @@ -604,6 +604,8 @@ If you fail to restore this encryption key file along with the application data backup, users with two-factor authentication enabled and GitLab Runners will lose access to your GitLab server. +You may also want to restore any TLS keys, certificates, or [SSH host keys](https://superuser.com/questions/532040/copy-ssh-keys-from-one-server-to-another-server/532079#532079). + Depending on your case, you might want to run the restore command with one or more of the following options: diff --git a/doc/raketasks/web_hooks.md b/doc/raketasks/web_hooks.md index 5f3143f76cd..df3dab118b2 100644 --- a/doc/raketasks/web_hooks.md +++ b/doc/raketasks/web_hooks.md @@ -38,8 +38,6 @@ ## List the webhooks from projects in a given **NAMESPACE**: # omnibus-gitlab - sudo gitlab-rake gitlab:web_hook:list NAMESPACE=/ + sudo gitlab-rake gitlab:web_hook:list NAMESPACE=acme # source installations - bundle exec rake gitlab:web_hook:list NAMESPACE=/ RAILS_ENV=production - -> Note: `/` is the global namespace. + bundle exec rake gitlab:web_hook:list NAMESPACE=acme RAILS_ENV=production diff --git a/doc/security/two_factor_authentication.md b/doc/security/two_factor_authentication.md index b770f2544d2..4b65b901487 100644 --- a/doc/security/two_factor_authentication.md +++ b/doc/security/two_factor_authentication.md @@ -6,7 +6,7 @@ password to login, they'll be prompted for a code generated by an application on their phone. You can read more about it here: -[Two-factor Authentication (2FA)](../profile/two_factor_authentication.md) +[Two-factor Authentication (2FA)](../user/profile/account/two_factor_authentication.md) ## Enforcing 2FA for all users diff --git a/doc/topics/authentication/index.md b/doc/topics/authentication/index.md index 394f3ea60b7..766a23c419d 100644 --- a/doc/topics/authentication/index.md +++ b/doc/topics/authentication/index.md @@ -44,6 +44,5 @@ This page gathers all the resources for the topic **Authentication** within GitL - [Kanboard Plugin GitLab Authentication](https://github.com/kanboard/plugin-gitlab-auth) - [Jenkins GitLab OAuth Plugin](https://wiki.jenkins.io/display/JENKINS/GitLab+OAuth+Plugin) -- [Set up Gitlab CE with Active Directory authentication](https://www.caseylabs.com/setup-gitlab-ce-with-active-directory-authentication/) - [How to customize GitLab to support OpenID authentication](http://eric.van-der-vlist.com/blog/2013/11/23/how-to-customize-gitlab-to-support-openid-authentication/) - [Openshift - Configuring Authentication and User Agent](https://docs.openshift.org/latest/install_config/configuring_authentication.html#GitLab) diff --git a/doc/topics/autodevops/index.md b/doc/topics/autodevops/index.md index 6bb2e236dc1..7885cffd107 100644 --- a/doc/topics/autodevops/index.md +++ b/doc/topics/autodevops/index.md @@ -132,7 +132,8 @@ in three places: - either under the project's CI/CD settings while [enabling Auto DevOps](#enabling-auto-devops) - or in instance-wide settings in the **admin area > Settings** under the "Continuous Integration and Delivery" section -- or at the project or group level as a variable: `AUTO_DEVOPS_DOMAIN` (required if you want to use [multiple clusters](#using-multiple-kubernetes-clusters)) +- or at the project as a variable: `AUTO_DEVOPS_DOMAIN` (required if you want to use [multiple clusters](#using-multiple-kubernetes-clusters)) +- or at the group level as a variable: `AUTO_DEVOPS_DOMAIN` A wildcard DNS A record matching the base domain(s) is required, for example, given a base domain of `example.com`, you'd need a DNS entry like: @@ -203,6 +204,12 @@ and verifying that your app is deployed as a review app in the Kubernetes cluster with the `review/*` environment scope. Similarly, you can check the other environments. +NOTE: **Note:** +Auto DevOps is not supported for a group with multiple clusters, as it +is not possible to set `AUTO_DEVOPS_DOMAIN` per environment on the group +level. This will be resolved in the future with the [following issue]( +https://gitlab.com/gitlab-org/gitlab-ce/issues/52363). + ## Enabling/Disabling Auto DevOps When first using Auto Devops, review the [requirements](#requirements) to ensure all necessary components to make @@ -479,14 +486,23 @@ no longer be valid as soon as the deployment job finishes. This means that Kubernetes can run the application, but in case it should be restarted or executed somewhere else, it cannot be accessed again. +#### Migrations + > [Introduced][ce-21955] in GitLab 11.4 Database initialization and migrations for PostgreSQL can be configured to run within the application pod by setting the project variables `DB_INITIALIZE` and `DB_MIGRATE` respectively. -If present, `DB_INITIALIZE` will be run as a shell command within an application pod as a helm -post-install hook. Note that this means that if any deploy succeeds, +If present, `DB_INITIALIZE` will be run as a shell command within an +application pod as a helm post-install hook. As some applications will +not run without a successful database initialization step, GitLab will +deploy the first release without the application deployment and only the +database initialization step. After the database initialization completes, +GitLab will deploy a second release with the application deployment as +normal. + +Note that a post-install hook means that if any deploy succeeds, `DB_INITIALIZE` will not be processed thereafter. If present, `DB_MIGRATE` will be run as a shell command within an application pod as @@ -657,8 +673,6 @@ also be customized, and you can easily use a [custom buildpack](#custom-buildpac | `REVIEW_DISABLED` | From GitLab 11.0, this variable can be used to disable the `review` and the manual `review:stop` job. If the variable is present, these jobs will not be created. | | `DAST_DISABLED` | From GitLab 11.0, this variable can be used to disable the `dast` job. If the variable is present, the job will not be created. | | `PERFORMANCE_DISABLED` | From GitLab 11.0, this variable can be used to disable the `performance` job. If the variable is present, the job will not be created. | -| `OLD_REPORTS_DISABLED` | From GitLab 11.5, this variable can be used to disable the `sast` job. If the variable is present, the job will not be created. | -| `NEW_REPORTS_DISABLED` | From GitLab 11.5, this variable can be used to disable the `sast_dashboard` job. If the variable is present, the job will not be created. | TIP: **Tip:** Set up the replica variables using a diff --git a/doc/update/11.5-to-11.6.md b/doc/update/11.5-to-11.6.md new file mode 100644 index 00000000000..031abc434ca --- /dev/null +++ b/doc/update/11.5-to-11.6.md @@ -0,0 +1,390 @@ +--- +comments: false +--- + +# From 11.5 to 11.6 + +Make sure you view this update guide from the branch (version) of GitLab you would +like to install (e.g., `11-6-stable`. You can select the branch in the version +dropdown at the top left corner of GitLab (below the menu bar). + +If the highest number stable branch is unclear please check the +[GitLab Blog](https://about.gitlab.com/blog/archives.html) for installation +guide links by version. + +### 1. Stop server + +```bash +sudo service gitlab stop +``` + +### 2. Backup + +NOTE: If you installed GitLab from source, make sure `rsync` is installed. + +```bash +cd /home/git/gitlab + +sudo -u git -H bundle exec rake gitlab:backup:create RAILS_ENV=production +``` + +### 3. Update Ruby + +NOTE: GitLab 11.0 and higher only support Ruby 2.4.x and dropped support for Ruby 2.3.x. Be +sure to upgrade your interpreter if necessary. + +You can check which version you are running with `ruby -v`. + +Download Ruby and compile it: + +```bash +mkdir /tmp/ruby && cd /tmp/ruby +curl --remote-name --progress https://cache.ruby-lang.org/pub/ruby/2.5/ruby-2.5.3.tar.gz +echo 'f919a9fbcdb7abecd887157b49833663c5c15fda ruby-2.5.3.tar.gz' | shasum -c - && tar xzf ruby-2.5.3.tar.gz +cd ruby-2.5.3 + +./configure --disable-install-rdoc +make +sudo make install +``` + +Install Bundler: + +```bash +sudo gem install bundler --no-ri --no-rdoc +``` + +### 4. Update Node + +GitLab utilizes [webpack](http://webpack.js.org) to compile frontend assets. +This requires a minimum version of node v6.0.0. + +You can check which version you are running with `node -v`. If you are running +a version older than `v6.0.0` you will need to update to a newer version. You +can find instructions to install from community maintained packages or compile +from source at the nodejs.org website. + +<https://nodejs.org/en/download/> + +GitLab also requires the use of yarn `>= v1.2.0` to manage JavaScript +dependencies. + +```bash +curl --silent --show-error https://dl.yarnpkg.com/debian/pubkey.gpg | sudo apt-key add - +echo "deb https://dl.yarnpkg.com/debian/ stable main" | sudo tee /etc/apt/sources.list.d/yarn.list +sudo apt-get update +sudo apt-get install yarn +``` + +More information can be found on the [yarn website](https://yarnpkg.com/en/docs/install). + +### 5. Update Go + +NOTE: GitLab 11.4 and higher only supports Go 1.10.x and newer, and dropped support for Go +1.9.x. Be sure to upgrade your installation if necessary. + +You can check which version you are running with `go version`. + +Download and install Go: + +```bash +# Remove former Go installation folder +sudo rm -rf /usr/local/go + +curl --remote-name --progress https://dl.google.com/go/go1.10.5.linux-amd64.tar.gz +echo 'a035d9beda8341b645d3f45a1b620cf2d8fb0c5eb409be36b389c0fd384ecc3a go1.10.5.linux-amd64.tar.gz' | shasum -a256 -c - && \ + sudo tar -C /usr/local -xzf go1.10.5.linux-amd64.tar.gz +sudo ln -sf /usr/local/go/bin/{go,godoc,gofmt} /usr/local/bin/ +rm go1.10.5.linux-amd64.tar.gz +``` + +### 6. Get latest code + +```bash +cd /home/git/gitlab + +sudo -u git -H git fetch --all --prune +sudo -u git -H git checkout -- db/schema.rb # local changes will be restored automatically +sudo -u git -H git checkout -- locale +``` + +For GitLab Community Edition: + +```bash +cd /home/git/gitlab + +sudo -u git -H git checkout 11-6-stable +``` + +OR + +For GitLab Enterprise Edition: + +```bash +cd /home/git/gitlab + +sudo -u git -H git checkout 11-6-stable-ee +``` + +### 7. Update gitlab-shell + +```bash +cd /home/git/gitlab-shell + +sudo -u git -H git fetch --all --tags --prune +sudo -u git -H git checkout v$(</home/git/gitlab/GITLAB_SHELL_VERSION) +sudo -u git -H bin/compile +``` + +### 8. Update gitlab-workhorse + +Install and compile gitlab-workhorse. GitLab-Workhorse uses +[GNU Make](https://www.gnu.org/software/make/). +If you are not using Linux you may have to run `gmake` instead of +`make` below. + +```bash +cd /home/git/gitlab-workhorse + +sudo -u git -H git fetch --all --tags --prune +sudo -u git -H git checkout v$(</home/git/gitlab/GITLAB_WORKHORSE_VERSION) +sudo -u git -H make +``` + +### 9. Update Gitaly + +#### Check Gitaly configuration + +Due to a bug in the `rake gitlab:gitaly:install` script your Gitaly +configuration file may contain syntax errors. The block name +`[[storages]]`, which may occur more than once in your `config.toml` +file, should be `[[storage]]` instead. + +```shell +sudo -u git -H sed -i.pre-10.1 's/\[\[storages\]\]/[[storage]]/' /home/git/gitaly/config.toml +``` + +#### Compile Gitaly + +```shell +cd /home/git/gitaly +sudo -u git -H git fetch --all --tags --prune +sudo -u git -H git checkout v$(</home/git/gitlab/GITALY_SERVER_VERSION) +sudo -u git -H make +``` + +### 10. Update gitlab-pages + +#### Only needed if you use GitLab Pages + +Install and compile gitlab-pages. GitLab-Pages uses +[GNU Make](https://www.gnu.org/software/make/). +If you are not using Linux you may have to run `gmake` instead of +`make` below. + +```bash +cd /home/git/gitlab-pages + +sudo -u git -H git fetch --all --tags --prune +sudo -u git -H git checkout v$(</home/git/gitlab/GITLAB_PAGES_VERSION) +sudo -u git -H make +``` + +### 11. Update MySQL permissions + +If you are using MySQL you need to grant the GitLab user the necessary +permissions on the database: + +```bash +mysql -u root -p -e "GRANT TRIGGER ON \`gitlabhq_production\`.* TO 'git'@'localhost';" +``` + +If you use MySQL with replication, or just have MySQL configured with binary logging, +you will need to also run the following on all of your MySQL servers: + +```bash +mysql -u root -p -e "SET GLOBAL log_bin_trust_function_creators = 1;" +``` + +You can make this setting permanent by adding it to your `my.cnf`: + +``` +log_bin_trust_function_creators=1 +``` + +### 12. Update configuration files + +#### New `unicorn.rb` configuration + +We have made [changes](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/22372) to `unicorn.rb` to allow GitLab run with both Unicorn and Puma in future. + +Make `/home/git/gitlab/config/unicorn.rb` the same as https://gitlab.com/gitlab-org/gitlab-ce/blob/11-6-stable/config/unicorn.rb.example but with your settings. +In particular, make sure that `require_relative "/home/git/gitlab/lib/gitlab/cluster/lifecycle_events"` line exists and the `before_exec`, `before_fork`, and `after_fork` handlers are configured as shown below: + +```ruby +require_relative "/home/git/gitlab/lib/gitlab/cluster/lifecycle_events" + +before_exec do |server| + # Signal application hooks that we're about to restart + Gitlab::Cluster::LifecycleEvents.do_master_restart +end + +before_fork do |server, worker| + # Signal application hooks that we're about to fork + Gitlab::Cluster::LifecycleEvents.do_before_fork +end + +after_fork do |server, worker| + # Signal application hooks of worker start + Gitlab::Cluster::LifecycleEvents.do_worker_start +end +``` + +#### New configuration options for `gitlab.yml` + +There might be configuration options available for [`gitlab.yml`][yaml]. View them with the command below and apply them manually to your current `gitlab.yml`: + +```sh +cd /home/git/gitlab + +git diff origin/11-5-stable:config/gitlab.yml.example origin/11-6-stable:config/gitlab.yml.example +``` + +#### Nginx configuration + +Ensure you're still up-to-date with the latest NGINX configuration changes: + +```sh +cd /home/git/gitlab + +# For HTTPS configurations +git diff origin/11-5-stable:lib/support/nginx/gitlab-ssl origin/11-6-stable:lib/support/nginx/gitlab-ssl + +# For HTTP configurations +git diff origin/11-5-stable:lib/support/nginx/gitlab origin/11-6-stable:lib/support/nginx/gitlab +``` + +If you are using Strict-Transport-Security in your installation to continue using it you must enable it in your Nginx +configuration as GitLab application no longer handles setting it. + +If you are using Apache instead of NGINX please see the updated [Apache templates]. +Also note that because Apache does not support upstreams behind Unix sockets you +will need to let gitlab-workhorse listen on a TCP port. You can do this +via [/etc/default/gitlab]. + +[Apache templates]: https://gitlab.com/gitlab-org/gitlab-recipes/tree/master/web-server/apache +[/etc/default/gitlab]: https://gitlab.com/gitlab-org/gitlab-ce/blob/11-6-stable/lib/support/init.d/gitlab.default.example#L38 + +#### SMTP configuration + +If you're installing from source and use SMTP to deliver mail, you will need to add the following line +to config/initializers/smtp_settings.rb: + +```ruby +ActionMailer::Base.delivery_method = :smtp +``` + +See [smtp_settings.rb.sample] as an example. + +[smtp_settings.rb.sample]: https://gitlab.com/gitlab-org/gitlab-ce/blob/11-6-stable/config/initializers/smtp_settings.rb.sample#L13 + +#### Init script + +There might be new configuration options available for [`gitlab.default.example`][gl-example]. View them with the command below and apply them manually to your current `/etc/default/gitlab`: + +```sh +cd /home/git/gitlab + +git diff origin/11-5-stable:lib/support/init.d/gitlab.default.example origin/11-6-stable:lib/support/init.d/gitlab.default.example +``` + +Ensure you're still up-to-date with the latest init script changes: + +```bash +cd /home/git/gitlab + +sudo cp lib/support/init.d/gitlab /etc/init.d/gitlab +``` + +For Ubuntu 16.04.1 LTS: + +```bash +sudo systemctl daemon-reload +``` + +### 13. Install libs, migrations, etc. + +```bash +cd /home/git/gitlab + +# MySQL installations (note: the line below states '--without postgres') +sudo -u git -H bundle install --without postgres development test --deployment + +# PostgreSQL installations (note: the line below states '--without mysql') +sudo -u git -H bundle install --without mysql development test --deployment + +# Optional: clean up old gems +sudo -u git -H bundle clean + +# Run database migrations +sudo -u git -H bundle exec rake db:migrate RAILS_ENV=production + +# Compile GetText PO files + +sudo -u git -H bundle exec rake gettext:compile RAILS_ENV=production + +# Update node dependencies and recompile assets +sudo -u git -H bundle exec rake yarn:install gitlab:assets:clean gitlab:assets:compile RAILS_ENV=production NODE_ENV=production + +# Clean up cache +sudo -u git -H bundle exec rake cache:clear RAILS_ENV=production +``` + +**MySQL installations**: Run through the `MySQL strings limits` and `Tables and data conversion to utf8mb4` [tasks](../install/database_mysql.md). + +### 14. Start application + +```bash +sudo service gitlab start +sudo service nginx restart +``` + +### 15. Check application status + +Check if GitLab and its environment are configured correctly: + +```bash +cd /home/git/gitlab + +sudo -u git -H bundle exec rake gitlab:env:info RAILS_ENV=production +``` + +To make sure you didn't miss anything run a more thorough check: + +```bash +cd /home/git/gitlab + +sudo -u git -H bundle exec rake gitlab:check RAILS_ENV=production +``` + +If all items are green, then congratulations, the upgrade is complete! + +## Things went south? Revert to previous version (11.5) + +### 1. Revert the code to the previous version + +Follow the [upgrade guide from 11.4 to 11.5](11.4-to-11.5.md), except for the +database migration (the backup is already migrated to the previous version). + +### 2. Restore from the backup + +```bash +cd /home/git/gitlab + +sudo -u git -H bundle exec rake gitlab:backup:restore RAILS_ENV=production +``` + +If you have more than one backup `*.tar` file(s) please add `BACKUP=timestamp_of_backup` to the command above. + +[yaml]: https://gitlab.com/gitlab-org/gitlab-ce/blob/11-6-stable/config/gitlab.yml.example +[gl-example]: https://gitlab.com/gitlab-org/gitlab-ce/blob/11-6-stable/lib/support/init.d/gitlab.default.example diff --git a/doc/user/admin_area/custom_project_templates.md b/doc/user/admin_area/custom_project_templates.md new file mode 100644 index 00000000000..5afbf9f2934 --- /dev/null +++ b/doc/user/admin_area/custom_project_templates.md @@ -0,0 +1,25 @@ +# Custom instance-level project templates **[PREMIUM ONLY]** + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/issues/6860) in [GitLab Premium](https://about.gitlab.com/pricing) 11.2. + +When you create a new project, creating it based on custom project templates is +a convenient option to bootstrap from an existing project boilerplate. +The administration setting to configure a GitLab group that serves as template +source can be found under **Admin > Settings > Custom project templates**. + +Within this section, you can configure the group where all the custom project +templates are sourced. Every project directly under the group namespace will be +available to the user if they have access to them. For example, every public +project in the group will be available to every logged user. However, +private projects will be available only if the user has view [permissions](../permissions.md) +in the project: + +- Project Owner, Maintainer, Developer, Reporter or Guest +- Is a member of the Group: Owner, Maintainer, Developer, Reporter or Guest + +Projects below subgroups of the template group are **not** supported. + +Repository and database information that are copied over to each new project are +identical to the data exported with [GitLab's Project Import/Export](../project/settings/import_export.md). + +If you would like to set project templates at a group level, please see [Custom group-level project templates](../group/custom_project_templates.md).
\ No newline at end of file diff --git a/doc/user/group/clusters/index.md b/doc/user/group/clusters/index.md new file mode 100644 index 00000000000..adc43921d47 --- /dev/null +++ b/doc/user/group/clusters/index.md @@ -0,0 +1,126 @@ +# Group-level Kubernetes clusters + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/issues/34758) in GitLab 11.6. + +CAUTION: **Warning:** +Group Cluster integration is currently in **Beta**. + +## Overview + +Similar to [project Kubernetes +clusters](../../project/clusters/index.md), Group-level Kubernetes +clusters allow you to connect a Kubernetes cluster to your group, +enabling you to use the same cluster across multiple projects. + +## Installing applications + +GitLab provides a one-click install for various applications that can be +added directly to your cluster. + +NOTE: **Note:** +Applications will be installed in a dedicated namespace called +`gitlab-managed-apps`. If you have added an existing Kubernetes cluster +with Tiller already installed, you should be careful as GitLab cannot +detect it. In this event, installing Tiller via the applications will +result in the cluster having it twice. This can lead to confusion during +deployments. + +| Application | GitLab version | Description | Helm Chart | +| ----------- | -------------- | ----------- | ---------- | +| [Helm Tiller](https://docs.helm.sh) | 10.2+ | Helm is a package manager for Kubernetes and is required to install all the other applications. It is installed in its own pod inside the cluster which can run the `helm` CLI in a safe environment. | n/a | +| [Ingress](https://kubernetes.io/docs/concepts/services-networking/ingress) | 10.2+ | Ingress can provide load balancing, SSL termination, and name-based virtual hosting. It acts as a web proxy for your applications and is useful if you want to use [Auto DevOps](../../../topics/autodevops/index.md) or deploy your own web apps. | [stable/nginx-ingress](https://github.com/helm/charts/tree/master/stable/nginx-ingress) | + +## RBAC compatibility + +For each project under a group with a Kubernetes cluster, GitLab will +create a restricted service account with [`edit` +privileges](https://kubernetes.io/docs/reference/access-authn-authz/rbac/#user-facing-roles) +in the project namespace. + +NOTE: **Note:** +RBAC support was introduced in +[GitLab 11.4](https://gitlab.com/gitlab-org/gitlab-ce/issues/29398), and +Project namespace restriction was introduced in +[GitLab 11.5](https://gitlab.com/gitlab-org/gitlab-ce/issues/51716). + +## Cluster precedence + +GitLab will use the project's cluster before using any cluster belonging +to the group containing the project if the project's cluster is available and not disabled. + +In the case of sub-groups, GitLab will use the cluster of the closest ancestor group +to the project, provided the cluster is not disabled. + +## Multiple Kubernetes clusters **[PREMIUM]** + +With GitLab Premium, you can associate more than one Kubernetes clusters to your +group. That way you can have different clusters for different environments, +like dev, staging, production, etc. + +Add another cluster similar to the first one and make sure to +[set an environment scope](#environment-scopes) that will +differentiate the new cluster from the rest. + +NOTE: **Note:** +Auto DevOps is not supported for a group with multiple clusters, as it +is not possible to set `AUTO_DEVOPS_DOMAIN` per environment on the group +level. This will be resolved in the future with the [following issue]( +https://gitlab.com/gitlab-org/gitlab-ce/issues/52363). + +## Environment scopes **[PREMIUM]** + +When adding more than one Kubernetes cluster to your project, you need +to differentiate them with an environment scope. The environment scope +associates clusters with [environments](../../../ci/environments.md) +similar to how the [environment-specific +variables](../../../ci/variables/README.md#limiting-environment-scopes-of-variables) +work. + +While evaluating which environment matches the environment scope of a +cluster, [cluster precedence](#cluster-precedence) will take +effect. The cluster at the project level will take precedence, followed +by the closest ancestor group, followed by that groups' parent and so +on. + +For example, let's say we have the following Kubernetes clusters: + +| Cluster | Environment scope | Where | +| ---------- | ------------------- | ----------| +| Project | `*` | Project | +| Staging | `staging/*` | Project | +| Production | `production/*` | Project | +| Test | `test` | Group | +| Development| `*` | Group | + + +And the following environments are set in [`.gitlab-ci.yml`](../../../ci/yaml/README.md): + +```yaml +stages: +- test +- deploy + +test: + stage: test + script: sh test + +deploy to staging: + stage: deploy + script: make deploy + environment: + name: staging/$CI_COMMIT_REF_NAME + url: https://staging.example.com/ + +deploy to production: + stage: deploy + script: make deploy + environment: + name: production/$CI_COMMIT_REF_NAME + url: https://example.com/ +``` + +The result will then be: + +- The Project cluster will be used for the `test` job. +- The Staging cluster will be used for the `deploy to staging` job. +- The Production cluster will be used for the `deploy to production` job. diff --git a/doc/user/group/custom_project_templates.md b/doc/user/group/custom_project_templates.md new file mode 100644 index 00000000000..eaf0273050b --- /dev/null +++ b/doc/user/group/custom_project_templates.md @@ -0,0 +1,23 @@ +# Custom group-level project templates **[PREMIUM ONLY]** + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ee/issues/6861) in [GitLab Premium](https://about.gitlab.com/pricing) 11.6. + +When you create a new project, creating it based on custom project templates is +a convenient option to bootstrap from an existing project boilerplate. +The group-level setting to configure a GitLab group that serves as template +source can be found under **Group > Settings > General > Custom project templates**. + +Within this section, you can configure the group where all the custom project +templates are sourced. Every project directly under the group namespace will be +available to the user if they have access to them. For example, every public +project in the group will be available to every logged in user. However, +private projects will be available only if the user has view [permissions](../permissions.md) +in the project. That is, users with Owner, Maintainer, Developer, Reporter or Guest roles for projects, +or for groups to which the project belongs. + +Projects of nested subgroups of a selected template source cannot be used. + +Repository and database information that are copied over to each new project are +identical to the data exported with [GitLab's Project Import/Export](../project/settings/import_export.md). + +If you would like to set project templates at an instance level, please see [Custom instance-level project templates](../admin_area/custom_project_templates.md).
\ No newline at end of file diff --git a/doc/user/group/index.md b/doc/user/group/index.md index d673fa4d21a..5fea683a7fd 100644 --- a/doc/user/group/index.md +++ b/doc/user/group/index.md @@ -259,11 +259,17 @@ types with every project in a group. Learn more about [Group-level file templates](https://docs.gitlab.com/ee/user/group/index.html#group-level-file-templates-premium). +#### Group-level project templates **[PREMIUM]** + +Define project templates at a group-level by setting a group as a template source. +[Learn more about group-level project templates](custom_project_templates.md). + ### Advanced settings - **Projects**: view all projects within that group, add members to each project, access each project's settings, and remove any project from the same screen. - **Webhooks**: configure [webhooks](../project/integrations/webhooks.md) to your group. +- **Kubernetes cluster integration**: connect your GitLab group with [Kubernetes clusters](clusters/index.md). - **Audit Events**: view [Audit Events](https://docs.gitlab.com/ee/administration/audit_events.html#audit-events) for the group. **[STARTER ONLY]** -- **Pipelines quota**: keep track of the [pipeline quota](../admin_area/settings/continuous_integration.md) for the group +- **Pipelines quota**: keep track of the [pipeline quota](../admin_area/settings/continuous_integration.md) for the group. diff --git a/doc/user/group/subgroups/index.md b/doc/user/group/subgroups/index.md index 8db36c4a0e8..943b0c693c0 100644 --- a/doc/user/group/subgroups/index.md +++ b/doc/user/group/subgroups/index.md @@ -167,7 +167,6 @@ Here's a list of what you can't do with subgroups: - [GitLab Pages](../../project/pages/index.md) are not currently working for projects hosted under a subgroup. That means that only projects hosted under the first parent group will work. -- Group level labels don't work in subgroups / sub projects - It is not possible to share a project with a group that's an ancestor of the group the project is in. That means you can only share as you walk down the hierarchy. For example, `group/subgroup01/project` **cannot** be shared diff --git a/doc/user/markdown.md b/doc/user/markdown.md index 6c6119a2691..debebd4c081 100644 --- a/doc/user/markdown.md +++ b/doc/user/markdown.md @@ -1022,7 +1022,7 @@ A link starting with a `/` is relative to the wiki root. [rouge]: http://rouge.jneen.net/ "Rouge website" [redcarpet]: https://github.com/vmg/redcarpet "Redcarpet website" [katex]: https://github.com/Khan/KaTeX "KaTeX website" -[katex-subset]: https://github.com/Khan/KaTeX/wiki/Function-Support-in-KaTeX "Macros supported by KaTeX" +[katex-subset]: https://katex.org/docs/supported.html "Macros supported by KaTeX" [asciidoctor-manual]: http://asciidoctor.org/docs/user-manual/#activating-stem-support "Asciidoctor user manual" [commonmarker]: https://github.com/gjtorikian/commonmarker [commonmark-spec]: https://spec.commonmark.org/current/ diff --git a/doc/user/project/clusters/index.md b/doc/user/project/clusters/index.md index 6735710e2bb..6d05e2feeec 100644 --- a/doc/user/project/clusters/index.md +++ b/doc/user/project/clusters/index.md @@ -17,6 +17,11 @@ your account with Google Kubernetes Engine (GKE) so that you can [create new clusters](#adding-and-creating-a-new-gke-cluster-via-gitlab) from within GitLab, or provide the credentials to an [existing Kubernetes cluster](#adding-an-existing-kubernetes-cluster). +NOTE: **Note:** +From [GitLab 11.6](https://gitlab.com/gitlab-org/gitlab-ce/issues/34758) you +can also associate a Kubernetes cluster to your groups. Learn more about +[group Kubernetes clusters](../../group/clusters/index.md). + ## Adding and creating a new GKE cluster via GitLab TIP: **Tip:** @@ -92,13 +97,47 @@ To add an existing Kubernetes cluster to your project: the `ca.crt` contents here. - **Token** - GitLab authenticates against Kubernetes using service tokens, which are - scoped to a particular `namespace`. If you don't have a service token yet, - you can follow the - [Kubernetes documentation](https://kubernetes.io/docs/tasks/configure-pod-container/configure-service-account/) - to create one. You can also view or create service tokens in the - [Kubernetes dashboard](https://kubernetes.io/docs/tasks/access-application-cluster/web-ui-dashboard/) - (under **Config > Secrets**). **The account that will issue the service token - must have admin privileges on the cluster.** + scoped to a particular `namespace`. + **The token used should belong to a service account with + [`cluster-admin`](https://kubernetes.io/docs/reference/access-authn-authz/rbac/#user-facing-roles) + privileges.** To create this service account: + + 1. Create a `gitlab` service account in the `default` namespace: + + ```bash + kubectl create -f - <<EOF + apiVersion: v1 + kind: ServiceAccount + metadata: + name: gitlab + namespace: default + EOF + ``` + 1. Create a cluster role binding to give the `gitlab` service account + `cluster-admin` privileges: + + ```bash + kubectl create -f - <<EOF + kind: ClusterRoleBinding + apiVersion: rbac.authorization.k8s.io/v1 + metadata: + name: gitlab-cluster-admin + subjects: + - kind: ServiceAccount + name: gitlab + namespace: default + roleRef: + kind: ClusterRole + name: cluster-admin + apiGroup: rbac.authorization.k8s.io + EOF + ``` + NOTE: **Note:** + For GKE clusters, you will need the + `container.clusterRoleBindings.create` permission to create a cluster + role binding. You can follow the [Google Cloud + documentation](https://cloud.google.com/iam/docs/granting-changing-revoking-access) + to grant access. - **Project namespace** (optional) - You don't have to fill it in; by leaving it blank, GitLab will create one for you. Also: - Each project should have a unique namespace. @@ -142,8 +181,9 @@ Whether ABAC or RBAC is enabled, GitLab will create the necessary service accounts and privileges in order to install and run [GitLab managed applications](#installing-applications): -- A `gitlab` service account with `cluster-admin` privileges will be created in the - `default` namespace, which will be used by GitLab to manage the newly created cluster. +- If GitLab is creating the cluster, a `gitlab` service account with + `cluster-admin` privileges will be created in the `default` namespace, + which will be used by GitLab to manage the newly created cluster. - A project service account with [`edit` privileges](https://kubernetes.io/docs/reference/access-authn-authz/rbac/#user-facing-roles) @@ -210,16 +250,18 @@ install it manually. ## Installing applications -GitLab provides a one-click install for various applications which will be -added directly to your configured cluster. Those applications are needed for -[Review Apps](../../../ci/review_apps/index.md) and [deployments](../../../ci/environments.md). +GitLab provides a one-click install for various applications which can +be added directly to your configured cluster. Those applications are +needed for [Review Apps](../../../ci/review_apps/index.md) and +[deployments](../../../ci/environments.md). NOTE: **Note:** With the exception of Knative, the applications will be installed in a dedicated namespace called `gitlab-managed-apps`. In case you have added an existing Kubernetes cluster with Tiller already installed, you should be careful as GitLab cannot -detect it. By installing it via the applications will result into having it -twice, which can lead to confusion during deployments. +detect it. In this event, installing Tiller via the applications will +result in the cluster having it twice. This can lead to confusion during +deployments. | Application | GitLab version | Description | Helm Chart | | ----------- | :------------: | ----------- | --------------- | @@ -238,13 +280,10 @@ by GitLab before installing any of the above applications. ## Getting the external IP address NOTE: **Note:** -You need a load balancer installed in your cluster in order to obtain the -external IP address with the following procedure. It can be deployed using the -[**Ingress** application](#installing-applications). - -NOTE: **Note:** -Knative will include its own load balancer in the form of [Istio](https://istio.io). -At this time, to determine the external IP address, you will need to follow the manual approach. +With the following procedure, a load balancer must be installed in your cluster +to obtain the external IP address. You can use either +[Ingress](#installing-applications), or Knative's own load balancer +([Istio](https://istio.io)) if using [Knative](#installing-applications). In order to publish your web application, you first need to find the external IP address associated to your load balancer. @@ -253,7 +292,7 @@ address associated to your load balancer. > [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/17052) in GitLab 10.6. -If you installed the Ingress [via the **Applications**](#installing-applications), +If you [installed Ingress or Knative](#installing-applications), you should see the Ingress IP address on this same page within a few minutes. If you don't see this, GitLab might not be able to determine the IP address of your ingress application in which case you should manually determine it. @@ -315,17 +354,13 @@ to reach your apps. This heavily depends on your domain provider, but in case you aren't sure, just create an A record with a wildcard host like `*.example.com.`. -## Setting the environment scope +## Setting the environment scope **[PREMIUM]** -NOTE: **Note:** -This is only available for [GitLab Premium][ee] where you can add more than -one Kubernetes cluster. - -When adding more than one Kubernetes clusters to your project, you need to -differentiate them with an environment scope. The environment scope associates -clusters and [environments](../../../ci/environments.md) in an 1:1 relationship -similar to how the -[environment-specific variables](../../../ci/variables/README.md#limiting-environment-scopes-of-variables) +When adding more than one Kubernetes clusters to your project, you need +to differentiate them with an environment scope. The environment scope +associates clusters with [environments](../../../ci/environments.md) +similar to how the [environment-specific +variables](../../../ci/variables/README.md#limiting-environment-scopes-of-variables) work. The default environment scope is `*`, which means all jobs, regardless of their @@ -400,12 +435,34 @@ GitLab CI/CD build environment. | `KUBE_NAMESPACE` | The Kubernetes namespace is auto-generated if not specified. The default value is `<project_name>-<project_id>`. You can overwrite it to use different one if needed, otherwise the `KUBE_NAMESPACE` variable will receive the default value. | | `KUBE_CA_PEM_FILE` | Path to a file containing PEM data. Only present if a custom CA bundle was specified. | | `KUBE_CA_PEM` | (**deprecated**) Raw PEM data. Only if a custom CA bundle was specified. | -| `KUBECONFIG` | Path to a file containing `kubeconfig` for this deployment. CA bundle would be embedded if specified. | +| `KUBECONFIG` | Path to a file containing `kubeconfig` for this deployment. CA bundle would be embedded if specified. This config also embeds the same token defined in `KUBE_TOKEN` so you likely will only need this variable. This variable name is also automatically picked up by `kubectl` so you won't actually need to reference it explicitly if using `kubectl`. | NOTE: **NOTE:** Prior to GitLab 11.5, `KUBE_TOKEN` was the Kubernetes token of the main service account of the cluster integration. +### Troubleshooting missing `KUBECONFIG` or `KUBE_TOKEN` + +GitLab will create a new service account specifically for your CI builds. The +new service account is created when the cluster is added to the project. +Sometimes there may be errors that cause the service account creation to fail. + +In such instances, your build will not be passed the `KUBECONFIG` or +`KUBE_TOKEN` variables and, if you are using Auto DevOps, your Auto DevOps +pipelines will no longer trigger a `production` deploy build. You will need to +check the [logs](../../../administration/logs.md) to debug why the service +account creation failed. + +A common reason for failure is that the token you gave GitLab did not have +[`cluster-admin`](https://kubernetes.io/docs/reference/access-authn-authz/rbac/#user-facing-roles) +privileges as GitLab expects. + +Another common problem for why these variables are not being passed to your +builds is that they must have a matching +[`environment:name`](../../../ci/environments.md#defining-environments). If +your build has no `environment:name` set, it will not be passed the Kubernetes +credentials. + ## Enabling or disabling the Kubernetes cluster integration After you have successfully added your cluster information, you can enable the diff --git a/doc/user/project/clusters/serverless/img/install-knative.png b/doc/user/project/clusters/serverless/img/install-knative.png Binary files differindex dd576a9df35..a9fcc127240 100644 --- a/doc/user/project/clusters/serverless/img/install-knative.png +++ b/doc/user/project/clusters/serverless/img/install-knative.png diff --git a/doc/user/project/clusters/serverless/img/serverless-page.png b/doc/user/project/clusters/serverless/img/serverless-page.png Binary files differnew file mode 100644 index 00000000000..473ee801f10 --- /dev/null +++ b/doc/user/project/clusters/serverless/img/serverless-page.png diff --git a/doc/user/project/import/bitbucket_server.md b/doc/user/project/import/bitbucket_server.md index dc985e87a96..ebf87890cfd 100644 --- a/doc/user/project/import/bitbucket_server.md +++ b/doc/user/project/import/bitbucket_server.md @@ -27,7 +27,7 @@ request. 1. Bitbucket Server allows multiple levels of threading. GitLab import will collapse this into one discussion and quote part of the original comment. -1. Declined pull requests have unrecahable commits, which prevents the GitLab +1. Declined pull requests have unreachable commits, which prevents the GitLab importer from generating a proper diff. These pull requests will show up as empty changes. 1. Attachments in Markdown are currently not imported. diff --git a/doc/user/project/integrations/emails_on_push.md b/doc/user/project/integrations/emails_on_push.md index b050c83fb72..df320de7e16 100644 --- a/doc/user/project/integrations/emails_on_push.md +++ b/doc/user/project/integrations/emails_on_push.md @@ -1,20 +1,20 @@ # Enabling emails on push -By enabling this service, you will be able to receive email notifications for -every change that is pushed to your project. +By enabling this service, you will receive email notifications for every change +that is pushed to your project. -Navigate to the [Integrations page](project_services.md#accessing-the-project-services) -and select the **Emails on push** service to configure it. +From the [Integrations page](project_services.md#accessing-the-project-services) +select **Emails on push** service to activate and configure it. In the _Recipients_ area, provide a list of emails separated by spaces or newlines. -You can configure any of the following settings depending on your preference. +The following options are available: + **Push events** - Email will be triggered when a push event is received + **Tag push events** - Email will be triggered when a tag is created and pushed + **Send from committer** - Send notifications from the committer's email address if the domain is part of the domain GitLab is running on (e.g. `user@gitlab.com`). + **Disable code diffs** - Don't include possibly sensitive code diffs in notification body. ---- - - +| Settings | Notification | +| --- | --- | +|  |  | diff --git a/doc/user/project/integrations/img/emails_on_push_email.png b/doc/user/project/integrations/img/emails_on_push_email.png Binary files differnew file mode 100644 index 00000000000..406eeb18410 --- /dev/null +++ b/doc/user/project/integrations/img/emails_on_push_email.png diff --git a/doc/user/project/integrations/img/emails_on_push_service.png b/doc/user/project/integrations/img/emails_on_push_service.png Binary files differindex df301aa1eeb..84e42eca9a2 100644 --- a/doc/user/project/integrations/img/emails_on_push_service.png +++ b/doc/user/project/integrations/img/emails_on_push_service.png diff --git a/doc/user/project/merge_requests/img/merge_request_widget.png b/doc/user/project/merge_requests/img/merge_request_widget.png Binary files differindex 6c2317b29b5..58562fcb034 100644 --- a/doc/user/project/merge_requests/img/merge_request_widget.png +++ b/doc/user/project/merge_requests/img/merge_request_widget.png diff --git a/doc/user/project/merge_requests/index.md b/doc/user/project/merge_requests/index.md index 5b54b6ecdd5..85d8d804133 100644 --- a/doc/user/project/merge_requests/index.md +++ b/doc/user/project/merge_requests/index.md @@ -259,6 +259,16 @@ all your changes will be available to preview by anyone with the Review Apps lin [Read more about Review Apps.](../../../ci/review_apps/index.md) +## Pipelines for merge requests + +When a developer updates a merge request, a pipeline should quickly report back +its result to the developer, but often pipelines take long time to complete +because general branch pipelines contain unnecessary jobs from the merge request standpoint. +You can customize a specific pipeline structure for merge requests in order to +speed the cycle up by running only important jobs. + +Learn more about [pipelines for merge requests](../../../ci/merge_request_pipelines/index.md). + ## Pipeline status in merge requests If you've set up [GitLab CI/CD](../../../ci/README.md) in your project, diff --git a/doc/user/project/merge_requests/resolve_conflicts.md b/doc/user/project/merge_requests/resolve_conflicts.md index ecbc8534eea..ccef2853e3f 100644 --- a/doc/user/project/merge_requests/resolve_conflicts.md +++ b/doc/user/project/merge_requests/resolve_conflicts.md @@ -1,15 +1,31 @@ -# Merge conflict resolution +# Merge request conflict resolution -> [Introduced][ce-5479] in GitLab 8.11. +Merge conflicts occur when two branches have different changes that cannot be +merged automatically. -When a merge request has conflicts, GitLab may provide the option to resolve -those conflicts in the GitLab UI. (See -[conflicts available for resolution](#conflicts-available-for-resolution) for -more information on when this is available.) If this is an option, you will see -a **resolve these conflicts** link in the merge request widget: +Git is able to automatically merge changes between branches in most cases, but +there are situations where Git will require your assistance to resolve the +conflicts manually. Typically, this is necessary when people change the same +parts of the same files. + +GitLab will prevent merge requests from being merged until all conflicts are +resolved. Conflicts can be resolved locally, or in many cases within GitLab +(see [conflicts available for resolution](#conflicts-available-for-resolution) +for information on when this is available).  +NOTE: **Note:** +GitLab resolves conflicts by creating a merge commit in the source branch that +is not automatically merged into the target branch. This allows the merge +commit to be reviewed and tested before the changes are merged, preventing +unintended changes entering the target branch without review or breaking the +build. + +## Resolve conflicts: interactive mode + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/5479) in GitLab 8.11. + Clicking this will show a list of files with conflicts, with conflict sections highlighted: @@ -21,9 +37,9 @@ request into the source branch, resolving the conflicts using the options chosen. If the source branch is `feature` and the target branch is `master`, this is similar to performing `git checkout feature; git merge master` locally. -## Merge conflict editor +## Resolve conflicts: inline editor -> Introduced in GitLab 8.13. +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/6374) in GitLab 8.13. The merge conflict resolution editor allows for more complex merge conflicts, which require the user to manually modify a file in order to resolve a conflict, @@ -50,5 +66,3 @@ Additionally, GitLab does not detect conflicts in renames away from a path. For example, this will not create a conflict: on branch `a`, doing `git mv file1 file2`; on branch `b`, doing `git mv file1 file3`. Instead, both files will be present in the branch after the merge request is merged. - -[ce-5479]: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/5479 diff --git a/doc/user/project/pipelines/schedules.md b/doc/user/project/pipelines/schedules.md index 9daacc37994..051277dfe02 100644 --- a/doc/user/project/pipelines/schedules.md +++ b/doc/user/project/pipelines/schedules.md @@ -83,12 +83,12 @@ The next time a pipeline is scheduled, your credentials will be used.  -> **Note:** -When the owner of the schedule doesn't have the ability to create pipelines -anymore, due to e.g., being blocked or removed from the project, or lacking -the permission to run on protected branches or tags. When this happened, the -schedule is deactivated. Another user can take ownership and activate it, so -the schedule can be run again. +NOTE: **Note:** +If the owner of a pipeline schedule doesn't have the ability to create pipelines +on the target branch, the schedule will stop creating new pipelines. This can +happen if, for example, the owner is blocked or removed from the project, or +the target branch or tag is protected. In this case, someone with sufficient +privileges must take ownership of the schedule. ## Advanced admin configuration diff --git a/doc/user/project/repository/img/repository_cleanup.png b/doc/user/project/repository/img/repository_cleanup.png Binary files differnew file mode 100644 index 00000000000..2749392ffa4 --- /dev/null +++ b/doc/user/project/repository/img/repository_cleanup.png diff --git a/doc/user/project/repository/reducing_the_repo_size_using_git.md b/doc/user/project/repository/reducing_the_repo_size_using_git.md index d534c8cbe4b..672567a8d7d 100644 --- a/doc/user/project/repository/reducing_the_repo_size_using_git.md +++ b/doc/user/project/repository/reducing_the_repo_size_using_git.md @@ -1,43 +1,105 @@ # Reducing the repository size using Git A GitLab Enterprise Edition administrator can set a [repository size limit][admin-repo-size] -which will prevent you to exceed it. +which will prevent you from exceeding it. When a project has reached its size limit, you will not be able to push to it, create a new merge request, or merge existing ones. You will still be able to create new issues, and clone the project though. Uploading LFS objects will also be denied. -In order to lift these restrictions, the administrator of the GitLab instance -needs to increase the limit on the particular project that exceeded it or you -need to instruct Git to rewrite changes. - If you exceed the repository size limit, your first thought might be to remove -some data, make a new commit and push back to the repository. Unfortunately, -it's not so easy and that workflow won't work. Deleting files in a commit doesn't -actually reduce the size of the repo since the earlier commits and blobs are -still around. What you need to do is rewrite history with Git's -[`filter-branch` option][gitscm]. +some data, make a new commit and push back to the repository. Perhaps you can +move some blobs to LFS, or remove some old dependency updates from history. +Unfortunately, it's not so easy and that workflow won't work. Deleting files in +a commit doesn't actually reduce the size of the repo since the earlier commits +and blobs are still around. What you need to do is rewrite history with Git's +[`filter-branch` option][gitscm], or a tool like the [BFG Repo-Cleaner][bfg]. Note that even with that method, until `git gc` runs on the GitLab side, the -"removed" commits and blobs will still be around. And if a commit was ever -included in an MR, or if a build was run for a commit, or if a user commented -on it, it will be kept around too. So, in these cases the size will not decrease. - -The only fool proof way to actually decrease the repository size is to prune all -the unneeded stuff locally, and then create a new project on GitLab and start -using that instead. +"removed" commits and blobs will still be around. You also need to be able to +push the rewritten history to GitLab, which may be impossible if you've already +exceeded the maximum size limit. -With that being said, you can try reducing your repository size with the -following method. - -## Using `git filter-branch` to purge files +In order to lift these restrictions, the administrator of the GitLab instance +needs to increase the limit on the particular project that exceeded it, so it's +always better to spot that you're approaching the limit and act proactively to +stay underneath it. If you hit the limit, and your admin can't - or won't - +temporarily increase it for you, your only option is to prune all the unneeded +stuff locally, and then create a new project on GitLab and start using that +instead. + +If you can continue to use the original project, we recommend [using the +BFG Repo-Cleaner](#using-the-bfg-repo-cleaner). It's faster and simpler than +`git filter-branch`, and GitLab can use its account of what has changed to clean +up its own internal state, maximizing the space saved. > **Warning:** > Make sure to first make a copy of your repository since rewriting history will > purge the files and information you are about to delete. Also make sure to > inform any collaborators to not use `pull` after your changes, but use `rebase`. +> **Warning:** +> This process is not suitable for removing sensitive data like password or keys +> from your repository. Information about commits, including file content, is +> cached in the database, and will remain visible even after they have been +> removed from the repository. + +## Using the BFG Repo-Cleaner + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/issues/19376) in GitLab 11.6. + +1. [Install BFG](https://rtyley.github.io/bfg-repo-cleaner/). + +1. Navigate to your repository: + + ``` + cd my_repository/ + ``` + +1. Change to the branch you want to remove the big file from: + + ``` + git checkout master + ``` + +1. Create a commit removing the large file from the branch, if it still exists: + + ``` + git rm path/to/big_file.mpg + git commit -m 'Remove unneeded large file' + ``` + +1. Rewrite history: + + ``` + bfg --delete-files path/to/big_file.mpg + ``` + + An object map file will be written to `object-id-map.old-new.txt`. Keep it + around - you'll need it for the final step! + +1. Force-push the changes to GitLab: + + ``` + git push --force-with-lease origin master + ``` + + If this step fails, someone has changed the `master` branch while you were + rewriting history. You could restore the branch and re-run BFG to preserve + their changes, or use `git push --force` to overwrite their changes. + +1. Navigate to **Project > Settings > Repository > Repository Cleanup**: + +  + + Upload the `object-id-map.old-new.txt` file and press **Start cleanup**. + This will remove any internal git references to the old commits, and run + `git gc` against the repository. You will receive an email once it has + completed. + +## Using `git filter-branch` + 1. Navigate to your repository: ``` @@ -70,11 +132,6 @@ following method. Your repository should now be below the size limit. -> **Note:** -> As an alternative to `filter-branch`, you can use the `bfg` tool with a -> command like: `bfg --delete-files path/to/big_file.mpg`. Read the -> [BFG Repo-Cleaner][bfg] documentation for more information. - [admin-repo-size]: https://docs.gitlab.com/ee/user/admin_area/settings/account_and_limit_settings.html#repository-size-limit [bfg]: https://rtyley.github.io/bfg-repo-cleaner/ [gitscm]: https://git-scm.com/book/en/v2/Git-Tools-Rewriting-History#The-Nuclear-Option:-filter-branch diff --git a/doc/workflow/notifications.md b/doc/workflow/notifications.md index c590ac4b0ba..020aa73f809 100644 --- a/doc/workflow/notifications.md +++ b/doc/workflow/notifications.md @@ -64,6 +64,8 @@ Below is the table of events users can be notified of: |------------------------------|-------------------------------------------------------------------|------------------------------| | New SSH key added | User | Security email, always sent. | | New email added | User | Security email, always sent. | +| Email changed | User | Security email, always sent. | +| Password changed | User | Security email, always sent. | | New user created | User | Sent on user creation, except for omniauth (LDAP)| | User added to project | User | Sent when user is added to project | | Project access level changed | User | Sent when user project access level is changed | |
