diff options
Diffstat (limited to 'doc/ci/pipelines')
28 files changed, 1341 insertions, 205 deletions
diff --git a/doc/ci/pipelines/img/coverage_check_approval_rule_14_1.png b/doc/ci/pipelines/img/coverage_check_approval_rule_14_1.png Binary files differnew file mode 100644 index 00000000000..00eb5c84ca9 --- /dev/null +++ b/doc/ci/pipelines/img/coverage_check_approval_rule_14_1.png diff --git a/doc/ci/pipelines/img/merge_train_cancel_v12_0.png b/doc/ci/pipelines/img/merge_train_cancel_v12_0.png Binary files differnew file mode 100644 index 00000000000..d7720ac1143 --- /dev/null +++ b/doc/ci/pipelines/img/merge_train_cancel_v12_0.png diff --git a/doc/ci/pipelines/img/merge_train_failure.png b/doc/ci/pipelines/img/merge_train_failure.png Binary files differnew file mode 100644 index 00000000000..8a795fff432 --- /dev/null +++ b/doc/ci/pipelines/img/merge_train_failure.png diff --git a/doc/ci/pipelines/img/merge_train_immediate_merge_v12_6.png b/doc/ci/pipelines/img/merge_train_immediate_merge_v12_6.png Binary files differnew file mode 100644 index 00000000000..7b903716a3d --- /dev/null +++ b/doc/ci/pipelines/img/merge_train_immediate_merge_v12_6.png diff --git a/doc/ci/pipelines/img/merge_train_position_v12_0.png b/doc/ci/pipelines/img/merge_train_position_v12_0.png Binary files differnew file mode 100644 index 00000000000..ec4b157d428 --- /dev/null +++ b/doc/ci/pipelines/img/merge_train_position_v12_0.png diff --git a/doc/ci/pipelines/img/merge_train_start_v12_0.png b/doc/ci/pipelines/img/merge_train_start_v12_0.png Binary files differnew file mode 100644 index 00000000000..a4d0c8cf0e6 --- /dev/null +++ b/doc/ci/pipelines/img/merge_train_start_v12_0.png diff --git a/doc/ci/pipelines/img/merge_train_start_when_pipeline_succeeds_v12_0.png b/doc/ci/pipelines/img/merge_train_start_when_pipeline_succeeds_v12_0.png Binary files differnew file mode 100644 index 00000000000..45762b8e85e --- /dev/null +++ b/doc/ci/pipelines/img/merge_train_start_when_pipeline_succeeds_v12_0.png diff --git a/doc/ci/pipelines/img/merged_result_pipeline.png b/doc/ci/pipelines/img/merged_result_pipeline.png Binary files differnew file mode 100644 index 00000000000..2584cd4d38d --- /dev/null +++ b/doc/ci/pipelines/img/merged_result_pipeline.png diff --git a/doc/ci/pipelines/img/multi_pipeline_mini_graph.gif b/doc/ci/pipelines/img/multi_pipeline_mini_graph.gif Binary files differnew file mode 100644 index 00000000000..de49ba5aa12 --- /dev/null +++ b/doc/ci/pipelines/img/multi_pipeline_mini_graph.gif diff --git a/doc/ci/pipelines/img/multi_project_pipeline_graph.png b/doc/ci/pipelines/img/multi_project_pipeline_graph.png Binary files differnew file mode 100644 index 00000000000..723a455cb4a --- /dev/null +++ b/doc/ci/pipelines/img/multi_project_pipeline_graph.png diff --git a/doc/ci/pipelines/img/parent_pipeline_graph_expanded_v12_6.png b/doc/ci/pipelines/img/parent_pipeline_graph_expanded_v12_6.png Binary files differnew file mode 100644 index 00000000000..db18cc201fc --- /dev/null +++ b/doc/ci/pipelines/img/parent_pipeline_graph_expanded_v12_6.png diff --git a/doc/ci/pipelines/img/pipeline-fork_v13_7.png b/doc/ci/pipelines/img/pipeline-fork_v13_7.png Binary files differnew file mode 100644 index 00000000000..eb44290aa66 --- /dev/null +++ b/doc/ci/pipelines/img/pipeline-fork_v13_7.png diff --git a/doc/ci/pipelines/img/pipelines_graph_dependency_view_hover_v13_12.png b/doc/ci/pipelines/img/pipelines_graph_dependency_view_hover_v13_12.png Binary files differindex e73845b39b0..ff6b3af0a28 100644 --- a/doc/ci/pipelines/img/pipelines_graph_dependency_view_hover_v13_12.png +++ b/doc/ci/pipelines/img/pipelines_graph_dependency_view_hover_v13_12.png diff --git a/doc/ci/pipelines/img/pipelines_graph_dependency_view_links_v13_12.png b/doc/ci/pipelines/img/pipelines_graph_dependency_view_links_v13_12.png Binary files differindex 1ce77881bc4..b0923ab96d9 100644 --- a/doc/ci/pipelines/img/pipelines_graph_dependency_view_links_v13_12.png +++ b/doc/ci/pipelines/img/pipelines_graph_dependency_view_links_v13_12.png diff --git a/doc/ci/pipelines/img/pipelines_graph_dependency_view_v13_12.png b/doc/ci/pipelines/img/pipelines_graph_dependency_view_v13_12.png Binary files differindex 41840108441..ae7cdc5b43e 100644 --- a/doc/ci/pipelines/img/pipelines_graph_dependency_view_v13_12.png +++ b/doc/ci/pipelines/img/pipelines_graph_dependency_view_v13_12.png diff --git a/doc/ci/pipelines/img/pipelines_graph_stage_view_v13_12.png b/doc/ci/pipelines/img/pipelines_graph_stage_view_v13_12.png Binary files differindex d7d8f3c63d2..b3b98313350 100644 --- a/doc/ci/pipelines/img/pipelines_graph_stage_view_v13_12.png +++ b/doc/ci/pipelines/img/pipelines_graph_stage_view_v13_12.png diff --git a/doc/ci/pipelines/index.md b/doc/ci/pipelines/index.md index af6b9e5b6b3..4f818d658b7 100644 --- a/doc/ci/pipelines/index.md +++ b/doc/ci/pipelines/index.md @@ -20,7 +20,7 @@ Pipelines comprise: - Jobs, which define *what* to do. For example, jobs that compile or test code. - Stages, which define *when* to run the jobs. For example, stages that run tests after stages that compile the code. -Jobs are executed by [runners](../runners/README.md). Multiple jobs in the same stage are executed in parallel, +Jobs are executed by [runners](../runners/index.md). Multiple jobs in the same stage are executed in parallel, if there are enough concurrent runners. If *all* jobs in a stage succeed, the pipeline moves on to the next stage. @@ -38,7 +38,7 @@ A typical pipeline might consist of four stages, executed in the following order - A `production` stage, with a job called `deploy-to-prod`. NOTE: -If you have a [mirrored repository that GitLab pulls from](../../user/project/repository/repository_mirroring.md#pulling-from-a-remote-repository), +If you have a [mirrored repository that GitLab pulls from](../../user/project/repository/repository_mirroring.md#pull-from-a-remote-repository), you may need to enable pipeline triggering in your project's **Settings > Repository > Pull from a remote repository > Trigger pipelines for mirror updates**. @@ -50,16 +50,16 @@ Pipelines can be configured in many different ways: followed by the next stage. - [Directed Acyclic Graph Pipeline (DAG) pipelines](../directed_acyclic_graph/index.md) are based on relationships between jobs and can run more quickly than basic pipelines. -- [Multi-project pipelines](../multi_project_pipelines.md) combine pipelines for different projects together. -- [Parent-Child pipelines](../parent_child_pipelines.md) break down complex pipelines +- [Multi-project pipelines](multi_project_pipelines.md) combine pipelines for different projects together. +- [Parent-Child pipelines](parent_child_pipelines.md) break down complex pipelines into one parent pipeline that can trigger multiple child sub-pipelines, which all run in the same project and with the same SHA. -- [Pipelines for Merge Requests](../merge_request_pipelines/index.md) run for merge +- [Pipelines for Merge Requests](../pipelines/merge_request_pipelines.md) run for merge requests only (rather than for every commit). -- [Pipelines for Merged Results](../merge_request_pipelines/pipelines_for_merged_results/index.md) +- [Pipelines for Merged Results](../pipelines/pipelines_for_merged_results.md) are merge request pipelines that act as though the changes from the source branch have already been merged into the target branch. -- [Merge Trains](../merge_request_pipelines/pipelines_for_merged_results/merge_trains/index.md) +- [Merge Trains](../pipelines/merge_trains.md) use pipelines for merged results to queue merges one after the other. ## Configure a pipeline @@ -67,15 +67,15 @@ Pipelines can be configured in many different ways: Pipelines and their component jobs and stages are defined in the CI/CD pipeline configuration file for each project. - [Jobs](../jobs/index.md) are the basic configuration component. -- Stages are defined by using the [`stages`](../yaml/README.md#stages) keyword. +- Stages are defined by using the [`stages`](../yaml/index.md#stages) keyword. -For a list of configuration options in the CI pipeline file, see the [GitLab CI/CD Pipeline Configuration Reference](../yaml/README.md). +For a list of configuration options in the CI pipeline file, see the [GitLab CI/CD Pipeline Configuration Reference](../yaml/index.md). You can also configure specific aspects of your pipelines through the GitLab UI. For example: - [Pipeline settings](settings.md) for each project. - [Pipeline schedules](schedules.md). -- [Custom CI/CD variables](../variables/README.md#custom-cicd-variables). +- [Custom CI/CD variables](../variables/index.md#custom-cicd-variables). ### Ref Specs for Runners @@ -89,13 +89,13 @@ This table lists the refspecs injected for each pipeline type: |--------------- |---------------------------------------- | | Pipeline for Branches | `+<sha>:refs/pipelines/<id>` and `+refs/heads/<name>:refs/remotes/origin/<name>` | | pipeline for Tags | `+<sha>:refs/pipelines/<id>` and `+refs/tags/<name>:refs/tags/<name>` | -| [Pipeline for Merge Requests](../merge_request_pipelines/index.md) | `+<sha>:refs/pipelines/<id>` | +| [Pipeline for Merge Requests](../pipelines/merge_request_pipelines.md) | `+<sha>:refs/pipelines/<id>` | The refs `refs/heads/<name>` and `refs/tags/<name>` exist in your project repository. GitLab generates the special ref `refs/pipelines/<id>` during a running pipeline job. This ref can be created even after the associated branch or tag has been deleted. It's therefore useful in some features such as [automatically stopping an environment](../environments/index.md#stopping-an-environment), -and [merge trains](../merge_request_pipelines/pipelines_for_merged_results/merge_trains/index.md) +and [merge trains](../pipelines/merge_trains.md) that might run pipelines after branch deletion. ### View pipelines @@ -125,7 +125,7 @@ you can filter the pipeline list by: ### Run a pipeline manually -Pipelines can be manually executed, with predefined or manually-specified [variables](../variables/README.md). +Pipelines can be manually executed, with predefined or manually-specified [variables](../variables/index.md). You might do this if the results of a pipeline (for example, a code build) are required outside the normal operation of the pipeline. @@ -136,7 +136,7 @@ To execute a pipeline manually: 1. Select the **Run pipeline** button. 1. On the **Run pipeline** page: 1. Select the branch or tag to run the pipeline for in the **Run for branch name or tag** field. - 1. Enter any [environment variables](../variables/README.md) required for the pipeline run. + 1. Enter any [environment variables](../variables/index.md) required for the pipeline run. You can set specific variables to have their [values prefilled in the form](#prefill-variables-in-manual-pipelines). 1. Click the **Run pipeline** button. @@ -146,9 +146,9 @@ The pipeline now executes the jobs as configured. > [Introduced in](https://gitlab.com/gitlab-org/gitlab/-/issues/30101) GitLab 13.7. -You can use the [`value` and `description`](../yaml/README.md#prefill-variables-in-manual-pipelines) +You can use the [`value` and `description`](../yaml/index.md#prefill-variables-in-manual-pipelines) keywords to define -[pipeline-level (global) variables](../variables/README.md#create-a-custom-cicd-variable-in-the-gitlab-ciyml-file) +[pipeline-level (global) variables](../variables/index.md#create-a-custom-cicd-variable-in-the-gitlab-ciyml-file) that are prefilled when running a pipeline manually. In pipelines triggered manually, the **Run pipelines** page displays all top-level variables @@ -200,7 +200,7 @@ For each `var` or `file_var`, a key and value are required. ### Add manual interaction to your pipeline -Manual actions, configured using the [`when:manual`](../yaml/README.md#whenmanual) keyword, +Manual actions, configured using the [`when:manual`](../yaml/index.md#whenmanual) keyword, allow you to require manual interaction before moving forward in the pipeline. You can do this straight from the pipeline graph. Just click the play button @@ -222,7 +222,7 @@ to an updated status. This functionality is only available: -- For users with at least Developer access. +- For users with at least the Developer role. - If the stage contains [manual actions](#add-manual-interaction-to-your-pipeline). ### Delete a pipeline @@ -301,7 +301,7 @@ A strict security model is enforced when pipelines are executed on [protected branches](../../user/project/protected_branches.md). The following actions are allowed on protected branches only if the user is -[allowed to merge or push](../../user/project/protected_branches.md#using-the-allowed-to-merge-and-allowed-to-push-settings) +[allowed to merge or push](../../user/project/protected_branches.md) on that specific branch: - Run manual pipelines (using the [Web UI](#run-a-pipeline-manually) or [pipelines API](#pipelines-api)). @@ -347,9 +347,9 @@ You can group the jobs by:  - [Job dependencies](#view-job-dependencies-in-the-pipeline-graph), which arranges - jobs based on their [`needs`](../yaml/README.md#needs) dependencies. + jobs based on their [`needs`](../yaml/index.md#needs) dependencies. -[Multi-project pipeline graphs](../multi_project_pipelines.md#multi-project-pipeline-visualization) help +[Multi-project pipeline graphs](multi_project_pipelines.md#multi-project-pipeline-visualization) help you visualize the entire pipeline, including all cross-project inter-dependencies. **(PREMIUM)** ### View job dependencies in the pipeline graph @@ -365,7 +365,7 @@ This in-development feature might not be available for your use. There can be [risks when enabling features still in development](../../user/feature_flags.md#risks-when-enabling-features-still-in-development). Refer to this feature's version history for more details. -You can arrange jobs in the pipeline graph based on their [`needs`](../yaml/README.md#needs) +You can arrange jobs in the pipeline graph based on their [`needs`](../yaml/index.md#needs) dependencies. Jobs in the leftmost column run first, and jobs that depend on them are grouped in the next columns. @@ -441,5 +441,5 @@ GitLab provides API endpoints to: - Perform basic functions. For more information, see [Pipelines API](../../api/pipelines.md). - Maintain pipeline schedules. For more information, see [Pipeline schedules API](../../api/pipeline_schedules.md). - Trigger pipeline runs. For more information, see: - - [Triggering pipelines through the API](../triggers/README.md). + - [Triggering pipelines through the API](../triggers/index.md). - [Pipeline triggers API](../../api/pipeline_triggers.md). diff --git a/doc/ci/pipelines/job_artifacts.md b/doc/ci/pipelines/job_artifacts.md index 0bb7007e7a9..b9a42c76293 100644 --- a/doc/ci/pipelines/job_artifacts.md +++ b/doc/ci/pipelines/job_artifacts.md @@ -48,7 +48,7 @@ is used. If you run two types of pipelines (like branch and scheduled) for the same ref, the pipeline that finishes later creates the job artifact. -For more examples, view the [keyword reference for the `.gitlab-ci.yml` file](../yaml/README.md#artifacts). +For more examples, view the [keyword reference for the `.gitlab-ci.yml` file](../yaml/index.md#artifacts). ## Download job artifacts @@ -112,7 +112,7 @@ the artifact. ## How searching for job artifacts works In [GitLab 13.5](https://gitlab.com/gitlab-org/gitlab/-/issues/201784) and later, artifacts -for [parent and child pipelines](../parent_child_pipelines.md) are searched in hierarchical +for [parent and child pipelines](parent_child_pipelines.md) are searched in hierarchical order from parent to child. For example, if both parent and child pipelines have a job with the same name, the job artifact from the parent pipeline is returned. @@ -173,7 +173,7 @@ https://gitlab.com/gitlab-org/gitlab/-/jobs/artifacts/main/file/htmlcov/index.ht ## When job artifacts are deleted -See the [`expire_in`](../yaml/README.md#artifactsexpire_in) documentation for information on when +See the [`expire_in`](../yaml/index.md#artifactsexpire_in) documentation for information on when job artifacts are deleted. ### Keep artifacts from most recent successful jobs @@ -203,5 +203,5 @@ This message is often preceded by other errors or warnings that specify the file generated. Check the job log for these messages. If you find no helpful messages, retry the failed job after activating -[CI/CD debug logging](../variables/README.md#debug-logging). +[CI/CD debug logging](../variables/index.md#debug-logging). This logging should provide information to help you investigate further. diff --git a/doc/ci/pipelines/merge_request_pipelines.md b/doc/ci/pipelines/merge_request_pipelines.md new file mode 100644 index 00000000000..29c12551f12 --- /dev/null +++ b/doc/ci/pipelines/merge_request_pipelines.md @@ -0,0 +1,215 @@ +--- +stage: Verify +group: Pipeline Execution +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +type: reference, index +last_update: 2019-07-03 +--- + +# Pipelines for merge requests **(FREE)** + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/15310) in GitLab 11.6. + +In a [basic configuration](pipeline_architectures.md#basic-pipelines), GitLab runs a pipeline each time +changes are pushed to a branch. + +If you want the pipeline to run jobs **only** on commits associated with a merge request, +you can use *pipelines for merge requests*. + +In the UI, these pipelines are labeled as `detached`. Otherwise, these pipelines are the same +as other pipelines. + +Pipelines for merge requests can run when you: + +- Create a new merge request. +- Commit changes to the source branch for the merge request. +- Select the **Run pipeline** button from the **Pipelines** tab in the merge request. + +If you use this feature with [merge when pipeline succeeds](../../user/project/merge_requests/merge_when_pipeline_succeeds.md), +pipelines for merge requests take precedence over other pipelines. + +## Prerequisites + +To enable pipelines for merge requests: + +- Your repository must be a GitLab repository, not an + [external repository](../ci_cd_for_external_repos/index.md). +- You must have the Developer [role](../../user/permissions.md) + to run a pipeline for merge requests. + +## Configure pipelines for merge requests + +To configure pipelines for merge requests, you must configure your [CI/CD configuration file](../yaml/index.md). +To do this, you can use [`rules`](#use-rules-to-run-pipelines-for-merge-requests) or [`only/except`](#use-only-or-except-to-run-pipelines-for-merge-requests). + +### Use `rules` to run pipelines for merge requests + +GitLab recommends that you use the `rules` keyword, which is available in +[`workflow:rules` templates](../yaml/index.md#workflowrules-templates). + +### Use `only` or `except` to run pipelines for merge requests + +You can use the `only/except` keywords. However, with this method, you must specify `only: - merge_requests` for each job. + +In the following example, the pipeline contains a `test` job that is configured to run on merge requests. +The `build` and `deploy` jobs don't have the `only: - merge_requests` keyword, +so they don't run on merge requests. + +```yaml +build: + stage: build + script: ./build + only: + - main + +test: + stage: test + script: ./test + only: + - merge_requests + +deploy: + stage: deploy + script: ./deploy + only: + - main +``` + +#### Exclude specific jobs + +When you use `only: [merge_requests]`, only jobs with +that keyword are run in the context of a merge request. No other jobs run. + +However, you can invert this behavior and have all of your jobs run except +for one or two. For example, you might have a pipeline with jobs `A`, `B`, and `C`, and you want: + +- All pipelines to always run `A` and `B`. +- `C` to run only for merge requests. + +To achieve this outcome, configure your `.gitlab-ci.yml` file as follows: + +```yaml +.only-default: &only-default + only: + - main + - merge_requests + - tags + +A: + <<: *only-default + script: + - ... + +B: + <<: *only-default + script: + - ... + +C: + script: + - ... + only: + - merge_requests +``` + +- `A` and `B` always run, because they get the `only:` rule to execute in all cases. +- `C` only runs for merge requests. It doesn't run for any pipeline + except a merge request pipeline. + +In this example, you don't have to add the `only:` rule to all of your jobs to make +them always run. You can use this format to set up a Review App, which helps to +save resources. + +#### Exclude specific branches + +Branch refs use this format: `refs/heads/my-feature-branch`. +Merge request refs use this format: `refs/merge-requests/:iid/head`. + +Because of this difference, the following configuration does not work as expected: + +```yaml +# Does not exclude a branch named "docs-my-fix"! +test: + only: [merge_requests] + except: [/^docs-/] +``` + +Instead, use the +[`$CI_COMMIT_REF_NAME` predefined environment +variable](../variables/predefined_variables.md) in +combination with +[`only:variables`](../yaml/index.md#onlyvariables--exceptvariables) to +accomplish this behavior: + +```yaml +test: + only: [merge_requests] + except: + variables: + - $CI_COMMIT_REF_NAME =~ /^docs-/ +``` + +## Run pipelines in the parent project for merge requests from a forked project **(PREMIUM)** + +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/217451) in GitLab 13.3. +> - [Moved](https://about.gitlab.com/blog/2021/01/26/new-gitlab-product-subscription-model/) to GitLab Premium in 13.9. + +By default, external contributors who work in forks can't create pipelines in the +parent project. When a merge request that comes from a fork triggers a pipeline: + +- The pipeline is created and runs in the fork (source) project, not the parent (target) project. +- The pipeline uses the fork project's CI/CD configuration and resources. + +If a pipeline runs in a fork, a **fork** badge appears for the pipeline in the merge request. + + + +Sometimes parent project members want the pipeline to run in the parent +project. They may want to ensure that the post-merge pipeline passes in the parent project. +For example, a fork project could try to use a corrupted runner that doesn't execute +test scripts properly, but reports a passed pipeline. Reviewers in the parent project +could mistakenly trust the merge request because it passed a faked pipeline. + +Parent project members with at least the [Developer role](../../user/permissions.md) +can create pipelines in the parent project for merge requests +from a forked project. In the merge request, go to the **Pipelines** tab and select +**Run pipeline**. + +WARNING: +Fork merge requests can contain malicious code that tries to steal secrets in the +parent project when the pipeline runs, even before merge. As a reviewer, you must carefully +check the changes in the merge request before triggering the pipeline. GitLab shows +a warning that you must accept before you can trigger the pipeline. + +## Predefined variables available for pipelines for merge requests + +When you use pipelines for merge requests, [additional predefined variables](../variables/predefined_variables.md#predefined-variables-for-merge-request-pipelines) are available to the CI/CD jobs. +These variables contain information from the associated merge request, so that you can +integrate your job with the [GitLab Merge Request API](../../api/merge_requests.md). + +## Troubleshooting + +### Two pipelines created when pushing to a merge request + +If you are experiencing duplicated pipelines when using `rules`, take a look at +the [important differences between `rules` and `only`/`except`](../jobs/job_control.md#avoid-duplicate-pipelines), +which helps you get your starting configuration correct. + +If you are seeing two pipelines when using `only/except`, please see the caveats +related to using `only/except` above (or, consider moving to `rules`). + +In [GitLab 13.7](https://gitlab.com/gitlab-org/gitlab/-/issues/201845) and later, +you can add `workflow:rules` to [switch from branch pipelines to merge request pipelines](../yaml/index.md#switch-between-branch-pipelines-and-merge-request-pipelines). +After a merge request is open on the branch, the pipeline switches to a merge request pipeline. + +### Two pipelines created when pushing an invalid CI configuration file + +Pushing to a branch with an invalid CI configuration file can trigger +the creation of two types of failed pipelines. One pipeline is a failed merge request +pipeline, and the other is a failed branch pipeline, but both are caused by the same +invalid configuration. + +## Related topics + +- [Pipelines for merged results](pipelines_for_merged_results.md). +- [Merge trains](merge_trains.md). diff --git a/doc/ci/pipelines/merge_trains.md b/doc/ci/pipelines/merge_trains.md new file mode 100644 index 00000000000..3e6ad071d7e --- /dev/null +++ b/doc/ci/pipelines/merge_trains.md @@ -0,0 +1,236 @@ +--- +stage: Verify +group: Pipeline Execution +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +type: reference +last_update: 2019-07-03 +--- + +# Merge Trains **(PREMIUM)** + +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/9186) in [GitLab Premium](https://about.gitlab.com/pricing/) 12.0. +> - [Squash and merge](../../user/project/merge_requests/squash_and_merge.md) support [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/13001) in [GitLab Premium](https://about.gitlab.com/pricing/) 12.6. + +For more information about why you might want to use Merge Trains, read [How merge trains keep your master green](https://about.gitlab.com/blog/2020/01/30/all-aboard-merge-trains/). + +When [pipelines for merged results](pipelines_for_merged_results.md) are +enabled, the pipeline jobs run as if the changes from your source branch have already +been merged into the target branch. + +However, the target branch may be changing rapidly. When you're ready to merge, +if you haven't run the pipeline in a while, the target branch may have already changed. +Merging now could introduce breaking changes. + +*Merge trains* can prevent this from happening. A merge train is a queued list of merge +requests, each waiting to be merged into the target branch. + +Many merge requests can be added to the train. Each merge request runs its own merged results pipeline, +which includes the changes from all of the other merge requests in *front* of it on the train. +All the pipelines run in parallel, to save time. + +If the pipeline for a merge request fails, the breaking changes are not merged, and the target +branch is unaffected. The merge request is removed from the train, and all pipelines behind it restart. + +If the pipeline for the merge request at the front of the train completes successfully, +the changes are merged into the target branch, and the other pipelines continue to +run. + +To add a merge request to a merge train, you need [permissions](../../user/permissions.md) to push to the target branch. + +Each merge train can run a maximum of **twenty** pipelines in parallel. +If more than twenty merge requests are added to the merge train, the merge requests +are queued until a slot in the merge train is free. There is no limit to the +number of merge requests that can be queued. + +## Merge train example + +Three merge requests (`A`, `B` and `C`) are added to a merge train in order, which +creates three merged results pipelines that run in parallel: + +1. The first pipeline runs on the changes from `A` combined with the target branch. +1. The second pipeline runs on the changes from `A` and `B` combined with the target branch. +1. The third pipeline runs on the changes from `A`, `B`, and `C` combined with the target branch. + +If the pipeline for `B` fails, it is removed from the train. The pipeline for +`C` restarts with the `A` and `C` changes, but without the `B` changes. + +If `A` then completes successfully, it merges into the target branch, and `C` continues +to run. If more merge requests are added to the train, they now include the `A` +changes that are included in the target branch, and the `C` changes that are from +the merge request already in the train. + +Read more about [how merge trains keep your master green](https://about.gitlab.com/blog/2020/01/30/all-aboard-merge-trains/). + +<i class="fa fa-youtube-play youtube" aria-hidden="true"></i> +Watch this video for a demonstration on [how parallel execution +of Merge Trains can prevent commits from breaking the default +branch](https://www.youtube.com/watch?v=D4qCqXgZkHQ). + +## Prerequisites + +To enable merge trains: + +- You must have the [Maintainer role](../../user/permissions.md). +- You must be using [GitLab Runner](https://gitlab.com/gitlab-org/gitlab-runner) 11.9 or later. +- In GitLab 13.0 and later, you need [Redis](https://redis.io/) 5.0 or later. +- Your repository must be a GitLab repository, not an + [external repository](../ci_cd_for_external_repos/index.md). + +## Enable merge trains + +To enable merge trains for your project: + +1. If you are on a self-managed GitLab instance, ensure the [feature flag](#merge-trains-feature-flag) is set correctly. +1. [Configure your CI/CD configuration file](merge_request_pipelines.md#configure-pipelines-for-merge-requests) + so that the pipeline or individual jobs run for merge requests. +1. Visit your project's **Settings > General** and expand **Merge requests**. +1. In the **Merge method** section, verify that **Merge commit** is selected. + You cannot use **Merge commit with semi-linear history** or **Fast-forward merge** with merge trains. +1. In the **Merge options** section, select **Enable merged results pipelines.** (if not already selected) and **Enable merge trains.** +1. Click **Save changes** + +In GitLab 13.5 and earlier, there is only one checkbox, named +**Enable merge trains and pipelines for merged results**. + +WARNING: +If you select the check box but don't configure your CI/CD to use +pipelines for merge requests, your merge requests may become stuck in an +unresolved state or your pipelines may be dropped. + +## Start a merge train + +To start a merge train: + +1. Visit a merge request. +1. Click the **Start merge train** button. + + + +Other merge requests can now be added to the train. + +## Add a merge request to a merge train + +To add a merge request to a merge train: + +1. Visit a merge request. +1. Click the **Add to merge train** button. + +If pipelines are already running for the merge request, you cannot add the merge request +to the train. Instead, you can schedule to add the merge request to a merge train **when the latest +pipeline succeeds**. + + + +## Remove a merge request from a merge train + +1. Visit a merge request. +1. Click the **Remove from merge train** button. + + + +If you want to add the merge request to a merge train again later, you can. + +## View a merge request's current position on the merge train + +After a merge request has been added to the merge train, the merge request's +current position is displayed under the pipeline widget: + + + +## Immediately merge a merge request with a merge train + +If you have a high-priority merge request (for example, a critical patch) that must +be merged urgently, you can bypass the merge train by using the **Merge Immediately** option. +This is the fastest option to get the change merged into the target branch. + + + +WARNING: +Each time you merge a merge request immediately, the current merge train +is recreated and all pipelines restart. + +## Troubleshooting + +### Merge request dropped from the merge train immediately + +If a merge request is not mergeable (for example, it's a draft merge request, there is a merge +conflict, etc.), your merge request is dropped from the merge train automatically. + +In these cases, the reason for dropping the merge request is in the **system notes**. + +To check the reason: + +1. Open the merge request that was dropped from the merge train. +1. Open the **Discussion** tab. +1. Find a system note that includes either: + - The text **... removed this merge request from the merge train because ...** + - **... aborted this merge request from the merge train because ...** + The reason is given in the text after the **because ...** phrase. + + + +### Merge When Pipeline Succeeds cannot be chosen + +[Merge When Pipeline Succeeds](../../user/project/merge_requests/merge_when_pipeline_succeeds.md) +is currently unavailable when Merge Trains are enabled. + +See [the related issue](https://gitlab.com/gitlab-org/gitlab/-/issues/12267) +for more information. + +### Merge Train Pipeline cannot be retried + +When a pipeline for merge trains fails the merge request is dropped from the train and the pipeline can't be retried. +Pipelines for merge trains run on the merged result of the changes in the merge request and +the changes from other merge requests already on the train. If the merge request is dropped from the train, +the merged result is out of date and the pipeline can't be retried. + +Instead, you should [add the merge request to the train](#add-a-merge-request-to-a-merge-train) +again, which triggers a new pipeline. + +### Unable to add to merge train with message "The pipeline for this merge request failed." + +Sometimes the **Start/Add to Merge Train** button is not available and the merge request says, +"The pipeline for this merge request failed. Please retry the job or push a new commit to fix the failure." + +This issue occurs when [**Pipelines must succeed**](../../user/project/merge_requests/merge_when_pipeline_succeeds.md#only-allow-merge-requests-to-be-merged-if-the-pipeline-succeeds) +is enabled in **Settings > General > Merge requests**. This option requires that you +run a new successful pipeline before you can re-add a merge request to a merge train. + +Merge trains ensure that each pipeline has succeeded before a merge happens, so +you can clear the **Pipelines must succeed** check box and keep +**Enable merge trains and pipelines for merged results** (merge trains) enabled. + +If you want to keep the **Pipelines must succeed** option enabled along with Merge +Trains, create a new pipeline for merged results when this error occurs: + +1. Go to the **Pipelines** tab and click **Run pipeline**. +1. Click **Start/Add to merge train when pipeline succeeds**. + +See [the related issue](https://gitlab.com/gitlab-org/gitlab/-/issues/35135) +for more information. + +### Merge Trains feature flag **(PREMIUM SELF)** + +In [GitLab 13.6 and later](https://gitlab.com/gitlab-org/gitlab/-/issues/244831), +you can [enable or disable merge trains in the project settings](#enable-merge-trains). + +In GitLab 13.5 and earlier, merge trains are automatically enabled when +[pipelines for merged results](pipelines_for_merged_results.md) are enabled. +To use pipelines for merged results without using merge trains, you can enable a +[feature flag](../../user/feature_flags.md) that blocks the merge trains feature. + +[GitLab administrators with access to the GitLab Rails console](../../administration/feature_flags.md) +can enable the feature flag to disable merge trains: + +```ruby +Feature.enable(:disable_merge_trains) +``` + +After you enable this feature flag, all existing merge trains are cancelled and +the **Start/Add to Merge Train** button no longer appears in merge requests. + +To disable the feature flag, and enable merge trains again: + +```ruby +Feature.disable(:disable_merge_trains) +``` diff --git a/doc/ci/pipelines/multi_project_pipelines.md b/doc/ci/pipelines/multi_project_pipelines.md new file mode 100644 index 00000000000..e3fe0fd20f5 --- /dev/null +++ b/doc/ci/pipelines/multi_project_pipelines.md @@ -0,0 +1,329 @@ +--- +stage: Verify +group: Pipeline Authoring +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +type: reference +--- + +# Multi-project pipelines **(FREE)** + +> [Moved](https://gitlab.com/gitlab-org/gitlab/-/issues/199224) to GitLab Free in 12.8. + +You can set up [GitLab CI/CD](../index.md) across multiple projects, so that a pipeline +in one project can trigger a pipeline in another project. You can visualize the entire pipeline +in one place, including all cross-project interdependencies. + +For example, you might deploy your web application from three different projects in GitLab. +Each project has its own build, test, and deploy process. With multi-project pipelines you can +visualize the entire pipeline, including all build and test stages for all three projects. + +<i class="fa fa-youtube-play youtube" aria-hidden="true"></i> +For an overview, see the [Multi-project pipelines demo](https://www.youtube.com/watch?v=g_PIwBM1J84). + +Multi-project pipelines are also useful for larger products that require cross-project interdependencies, like those +with a [microservices architecture](https://about.gitlab.com/blog/2016/08/16/trends-in-version-control-land-microservices/). +Learn more in the [Cross-project Pipeline Triggering and Visualization demo](https://about.gitlab.com/learn/) +at GitLab@learn, in the Continuous Integration section. + +If you trigger a pipeline in a downstream private project, on the upstream project's pipelines page, +you can view: + +- The name of the project. +- The status of the pipeline. + +If you have a public project that can trigger downstream pipelines in a private project, +make sure there are no confidentiality problems. + +## Create multi-project pipelines + +To create multi-project pipelines, you can: + +- [Define them in your `.gitlab-ci.yml` file](#define-multi-project-pipelines-in-your-gitlab-ciyml-file). +- [Use the API](#create-multi-project-pipelines-by-using-the-api). + +### Define multi-project pipelines in your `.gitlab-ci.yml` file + +> [Moved](https://gitlab.com/gitlab-org/gitlab/-/issues/199224) to GitLab Free in 12.8. + +When you create a multi-project pipeline in your `.gitlab-ci.yml` file, +you create what is called a *trigger job*. For example: + +```yaml +rspec: + stage: test + script: bundle exec rspec + +staging: + variables: + ENVIRONMENT: staging + stage: deploy + trigger: my/deployment +``` + +In this example, after the `rspec` job succeeds in the `test` stage, +the `staging` trigger job starts. The initial status of this +job is `pending`. + +GitLab then creates a downstream pipeline in the +`my/deployment` project and, as soon as the pipeline is created, the +`staging` job succeeds. The full path to the project is `my/deployment`. + +You can view the status for the pipeline, or you can display +[the downstream pipeline's status instead](#mirror-status-of-a-triggered-pipeline-in-the-trigger-job). + +The user that creates the upstream pipeline must be able to create pipelines in the +downstream project (`my/deployment`) too. If the downstream project is not found, +or the user does not have [permission](../../user/permissions.md) to create a pipeline there, +the `staging` job is marked as _failed_. + +#### Trigger job configuration keywords + +Trigger jobs can use only a limited set of the GitLab CI/CD [configuration keywords](../yaml/index.md). +The keywords available for use in trigger jobs are: + +- [`trigger`](../yaml/index.md#trigger) +- [`stage`](../yaml/index.md#stage) +- [`allow_failure`](../yaml/index.md#allow_failure) +- [`rules`](../yaml/index.md#rules) +- [`only` and `except`](../yaml/index.md#only--except) +- [`when`](../yaml/index.md#when) (only with a value of `on_success`, `on_failure`, or `always`) +- [`extends`](../yaml/index.md#extends) +- [`needs`](../yaml/index.md#needs) + +#### Specify a downstream pipeline branch + +You can specify a branch name for the downstream pipeline to use. +GitLab uses the commit on the head of the branch to +create the downstream pipeline. + +```yaml +rspec: + stage: test + script: bundle exec rspec + +staging: + stage: deploy + trigger: + project: my/deployment + branch: stable-11-2 +``` + +Use: + +- The `project` keyword to specify the full path to a downstream project. +- The `branch` keyword to specify the name of a branch in the project specified by `project`. + [In GitLab 12.4](https://gitlab.com/gitlab-org/gitlab/-/issues/10126) and later, variable expansion is + supported. + +Pipelines triggered on a protected branch in a downstream project use the [role](../../user/permissions.md) +of the user that ran the trigger job in the upstream project. If the user does not +have permission to run CI/CD pipelines against the protected branch, the pipeline fails. See +[pipeline security for protected branches](index.md#pipeline-security-on-protected-branches). + +#### Pass CI/CD variables to a downstream pipeline by using the `variables` keyword + +Sometimes you might want to pass CI/CD variables to a downstream pipeline. +You can do that by using the `variables` keyword, just like you would for any other job. + +```yaml +rspec: + stage: test + script: bundle exec rspec + +staging: + variables: + ENVIRONMENT: staging + stage: deploy + trigger: my/deployment +``` + +The `ENVIRONMENT` variable is passed to every job defined in a downstream +pipeline. It is available as a variable when GitLab Runner picks a job. + +In the following configuration, the `MY_VARIABLE` variable is passed to the downstream pipeline +that is created when the `trigger-downstream` job is queued. This is because `trigger-downstream` +job inherits variables declared in global variables blocks, and then we pass these variables to a downstream pipeline. + +```yaml +variables: + MY_VARIABLE: my-value + +trigger-downstream: + variables: + ENVIRONMENT: something + trigger: my/project +``` + +You can stop global variables from reaching the downstream pipeline by using the [`inherit` keyword](../yaml/index.md#inherit). +In this example, the `MY_GLOBAL_VAR` variable is not available in the triggered pipeline: + +```yaml +variables: + MY_GLOBAL_VAR: value + +trigger-downstream: + inherit: + variables: false + variables: + MY_LOCAL_VAR: value + trigger: my/project +``` + +You might want to pass some information about the upstream pipeline using, for +example, predefined variables. In order to do that, you can use interpolation +to pass any variable. For example: + +```yaml +downstream-job: + variables: + UPSTREAM_BRANCH: $CI_COMMIT_REF_NAME + trigger: my/project +``` + +In this scenario, the `UPSTREAM_BRANCH` variable with a value related to the +upstream pipeline is passed to the `downstream-job` job. It is available +in the context of all downstream builds. + +Upstream pipelines take precedence over downstream ones. If there are two +variables with the same name defined in both upstream and downstream projects, +the ones defined in the upstream project take precedence. + +#### Pass CI/CD variables to a downstream pipeline by using variable inheritance + +You can pass variables to a downstream pipeline with [`dotenv` variable inheritance](../variables/index.md#pass-an-environment-variable-to-another-job) and [cross project artifact downloads](../yaml/index.md#cross-project-artifact-downloads-with-needs). + +In the upstream pipeline: + +1. Save the variables in a `.env` file. +1. Save the `.env` file as a `dotenv` report. +1. Trigger the downstream pipeline. + + ```yaml + build_vars: + stage: build + script: + - echo "BUILD_VERSION=hello" >> build.env + artifacts: + reports: + dotenv: build.env + + deploy: + stage: deploy + trigger: my/downstream_project + ``` + +1. Set the `test` job in the downstream pipeline to inherit the variables from the `build_vars` + job in the upstream project with `needs:`. The `test` job inherits the variables in the + `dotenv` report and it can access `BUILD_VERSION` in the script: + + ```yaml + test: + stage: test + script: + - echo $BUILD_VERSION + needs: + - project: my/upstream_project + job: build_vars + ref: master + artifacts: true + ``` + +#### Use `rules` or `only`/`except` with multi-project pipelines + +You can use CI/CD variables or the [`rules`](../yaml/index.md#rulesif) keyword to +[control job behavior](../jobs/job_control.md) for multi-project pipelines. When a +downstream pipeline is triggered with the [`trigger`](../yaml/index.md#trigger) keyword, +the value of the [`$CI_PIPELINE_SOURCE` predefined variable](../variables/predefined_variables.md) +is `pipeline` for all its jobs. + +If you use [`only/except`](../yaml/index.md#only--except) to control job behavior, use the +[`pipelines`](../yaml/index.md#onlyrefs--exceptrefs) keyword. + +#### Mirror status of a triggered pipeline in the trigger job + +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/11238) in GitLab Premium 12.3. +> - [Moved](https://gitlab.com/gitlab-org/gitlab/-/issues/199224) to GitLab Free in 12.8. + +You can mirror the pipeline status from the triggered pipeline to the source +trigger job by using `strategy: depend`. For example: + +```yaml +trigger_job: + trigger: + project: my/project + strategy: depend +``` + +#### Mirror status from upstream pipeline + +You can mirror the pipeline status from an upstream pipeline to a bridge job by +using the `needs:pipeline` keyword. The latest pipeline status from the default branch is +replicated to the bridge job. + +For example: + +```yaml +upstream_bridge: + stage: test + needs: + pipeline: other/project +``` + +### Create multi-project pipelines by using the API + +> [Moved](https://gitlab.com/gitlab-org/gitlab/-/issues/31573) to GitLab Free in 12.4. + +When you use the [`CI_JOB_TOKEN` to trigger pipelines](../triggers/index.md#ci-job-token), +GitLab recognizes the source of the job token. The pipelines become related, +so you can visualize their relationships on pipeline graphs. + +These relationships are displayed in the pipeline graph by showing inbound and +outbound connections for upstream and downstream pipeline dependencies. + +When using: + +- CI/CD variables or [`rules`](../yaml/index.md#rulesif) to control job behavior, the value of + the [`$CI_PIPELINE_SOURCE` predefined variable](../variables/predefined_variables.md) is + `pipeline` for multi-project pipeline triggered through the API with `CI_JOB_TOKEN`. +- [`only/except`](../yaml/index.md#only--except) to control job behavior, use the + `pipelines` keyword. + +## Trigger a pipeline when an upstream project is rebuilt **(PREMIUM)** + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/9045) in GitLab Premium 12.8. + +You can trigger a pipeline in your project whenever a pipeline finishes for a new +tag in a different project. + +Prerequisites: + +- The upstream project must be [public](../../public_access/public_access.md). +- The user must have the [Developer role](../../user/permissions.md#project-members-permissions) + in the upstream project. + +To trigger the pipeline when the upstream project is rebuilt: + +1. On the top bar, select **Menu > Projects** and find your project. +1. On the left sidebar, select **Settings > CI/CD** page. +1. Expand the **Pipeline subscriptions** section. +1. Enter the project you want to subscribe to, in the format `<namespace>/<project>`. + For example, if the project is `https://gitlab.com/gitlab-org/gitlab`, use `gitlab-org/gitlab`. +1. Select **Subscribe**. + +Any pipelines that complete successfully for new tags in the subscribed project +now trigger a pipeline on the current project's default branch. The maximum +number of upstream pipeline subscriptions is 2 by default, for both the upstream and +downstream projects. On self-managed instances, an administrator can change this +[limit](../../administration/instance_limits.md#number-of-cicd-subscriptions-to-a-project). + +## Multi-project pipeline visualization **(PREMIUM)** + +When you configure GitLab CI/CD for your project, you can visualize the stages of your +[jobs](index.md#configure-a-pipeline) on a [pipeline graph](index.md#visualize-pipelines). + + + +In the merge request, on the **Pipelines** tab, multi-project pipeline mini-graphs are displayed. +They expand and are shown adjacent to each other when hovering (or tapping on touchscreen devices). + + diff --git a/doc/ci/pipelines/parent_child_pipelines.md b/doc/ci/pipelines/parent_child_pipelines.md new file mode 100644 index 00000000000..2e29f4fe812 --- /dev/null +++ b/doc/ci/pipelines/parent_child_pipelines.md @@ -0,0 +1,192 @@ +--- +stage: Verify +group: Pipeline Authoring +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +type: reference +--- + +# Parent-child pipelines **(FREE)** + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/16094) in GitLab 12.7. + +As pipelines grow more complex, a few related problems start to emerge: + +- The staged structure, where all steps in a stage must be completed before the first + job in next stage begins, causes arbitrary waits, slowing things down. +- Configuration for the single global pipeline becomes very long and complicated, + making it hard to manage. +- Imports with [`include`](../yaml/index.md#include) increase the complexity of the configuration, and create the potential + for namespace collisions where jobs are unintentionally duplicated. +- Pipeline UX can become unwieldy with so many jobs and stages to work with. + +Additionally, sometimes the behavior of a pipeline needs to be more dynamic. The ability +to choose to start sub-pipelines (or not) is a powerful ability, especially if the +YAML is dynamically generated. + + + +Similarly to [multi-project pipelines](multi_project_pipelines.md), a pipeline can trigger a +set of concurrently running child pipelines, but within the same project: + +- Child pipelines still execute each of their jobs according to a stage sequence, but + would be free to continue forward through their stages without waiting for unrelated + jobs in the parent pipeline to finish. +- The configuration is split up into smaller child pipeline configurations, which are + easier to understand. This reduces the cognitive load to understand the overall configuration. +- Imports are done at the child pipeline level, reducing the likelihood of collisions. +- Each pipeline has only relevant steps, making it easier to understand what's going on. + +Child pipelines work well with other GitLab CI/CD features: + +- Use [`rules: changes`](../yaml/index.md#ruleschanges) to trigger pipelines only when + certain files change. This is useful for monorepos, for example. +- Since the parent pipeline in `.gitlab-ci.yml` and the child pipeline run as normal + pipelines, they can have their own behaviors and sequencing in relation to triggers. + +See the [`trigger:`](../yaml/index.md#trigger) keyword documentation for full details on how to +include the child pipeline configuration. + +<i class="fa fa-youtube-play youtube" aria-hidden="true"></i> +For an overview, see [Parent-Child Pipelines feature demo](https://youtu.be/n8KpBSqZNbk). + +## Examples + +The simplest case is [triggering a child pipeline](../yaml/index.md#trigger) using a +local YAML file to define the pipeline configuration. In this case, the parent pipeline +triggers the child pipeline, and continues without waiting: + +```yaml +microservice_a: + trigger: + include: path/to/microservice_a.yml +``` + +You can include multiple files when composing a child pipeline: + +```yaml +microservice_a: + trigger: + include: + - local: path/to/microservice_a.yml + - template: Security/SAST.gitlab-ci.yml +``` + +In [GitLab 13.5](https://gitlab.com/gitlab-org/gitlab/-/issues/205157) and later, +you can use [`include:file`](../yaml/index.md#includefile) to trigger child pipelines +with a configuration file in a different project: + +```yaml +microservice_a: + trigger: + include: + - project: 'my-group/my-pipeline-library' + file: 'path/to/ci-config.yml' +``` + +The maximum number of entries that are accepted for `trigger:include:` is three. + +Similar to [multi-project pipelines](multi_project_pipelines.md#mirror-status-of-a-triggered-pipeline-in-the-trigger-job), +we can set the parent pipeline to depend on the status of the child pipeline upon completion: + +```yaml +microservice_a: + trigger: + include: + - local: path/to/microservice_a.yml + - template: Security/SAST.gitlab-ci.yml + strategy: depend +``` + +## Merge Request child pipelines + +To trigger a child pipeline as a [Merge Request Pipeline](merge_request_pipelines.md) we need to: + +- Set the trigger job to run on merge requests: + +```yaml +# parent .gitlab-ci.yml +microservice_a: + trigger: + include: path/to/microservice_a.yml + rules: + - if: $CI_MERGE_REQUEST_ID +``` + +- Configure the child pipeline by either: + + - Setting all jobs in the child pipeline to evaluate in the context of a merge request: + + ```yaml + # child path/to/microservice_a.yml + workflow: + rules: + - if: $CI_MERGE_REQUEST_ID + + job1: + script: ... + + job2: + script: ... + ``` + + - Alternatively, setting the rule per job. For example, to create only `job1` in + the context of merge request pipelines: + + ```yaml + # child path/to/microservice_a.yml + job1: + script: ... + rules: + - if: $CI_MERGE_REQUEST_ID + + job2: + script: ... + ``` + +## Dynamic child pipelines + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/35632) in GitLab 12.9. + +Instead of running a child pipeline from a static YAML file, you can define a job that runs +your own script to generate a YAML file, which is then [used to trigger a child pipeline](../yaml/index.md#trigger-child-pipeline-with-generated-configuration-file). + +This technique can be very powerful in generating pipelines targeting content that changed or to +build a matrix of targets and architectures. + +<i class="fa fa-youtube-play youtube" aria-hidden="true"></i> +For an overview, see [Create child pipelines using dynamically generated configurations](https://youtu.be/nMdfus2JWHM). + +<!-- vale gitlab.Spelling = NO --> +We also have an example project using +[Dynamic Child Pipelines with Jsonnet](https://gitlab.com/gitlab-org/project-templates/jsonnet) +which shows how to use a data templating language to generate your `.gitlab-ci.yml` at runtime. You could use a similar process for other templating languages like [Dhall](https://dhall-lang.org/) or [`ytt`](https://get-ytt.io/). +<!-- vale gitlab.Spelling = NO --> + +The artifact path is parsed by GitLab, not the runner, so the path must match the +syntax for the OS running GitLab. If GitLab is running on Linux but using a Windows +runner for testing, the path separator for the trigger job would be `/`. Other CI/CD +configuration for jobs, like scripts, that use the Windows runner would use `\`. + +In GitLab 12.9, the child pipeline could fail to be created in certain cases, causing the parent pipeline to fail. +This is [resolved in GitLab 12.10](https://gitlab.com/gitlab-org/gitlab/-/issues/209070). + +## Nested child pipelines + +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/29651) in GitLab 13.4. +> - [Feature flag removed](https://gitlab.com/gitlab-org/gitlab/-/issues/243747) in GitLab 13.5. + +Parent and child pipelines were introduced with a maximum depth of one level of child +pipelines, which was later increased to two. A parent pipeline can trigger many child +pipelines, and these child pipelines can trigger their own child pipelines. It's not +possible to trigger another level of child pipelines. + +<i class="fa fa-youtube-play youtube" aria-hidden="true"></i> +For an overview, see [Nested Dynamic Pipelines](https://youtu.be/C5j3ju9je2M). + +## Pass CI/CD variables to a child pipeline + +You can pass CI/CD variables to a downstream pipeline using the same methods as +multi-project pipelines: + +- [By using the `variable` keyword](multi_project_pipelines.md#pass-cicd-variables-to-a-downstream-pipeline-by-using-the-variables-keyword). +- [By using variable inheritance](multi_project_pipelines.md#pass-cicd-variables-to-a-downstream-pipeline-by-using-variable-inheritance). diff --git a/doc/ci/pipelines/pipeline_architectures.md b/doc/ci/pipelines/pipeline_architectures.md index 78031ec1d97..1b23727b142 100644 --- a/doc/ci/pipelines/pipeline_architectures.md +++ b/doc/ci/pipelines/pipeline_architectures.md @@ -18,7 +18,7 @@ own advantages. These methods can be mixed and matched if needed: - [Child/Parent Pipelines](#child--parent-pipelines): Good for monorepos and projects with lots of independently defined components. For more details about -any of the keywords used below, check out our [CI YAML reference](../yaml/README.md) for details. +any of the keywords used below, check out our [CI YAML reference](../yaml/index.md) for details. ## Basic Pipelines @@ -96,7 +96,7 @@ deploy_b: If efficiency is important to you and you want everything to run as quickly as possible, you can use [Directed Acyclic Graphs (DAG)](../directed_acyclic_graph/index.md). Use the -[`needs` keyword](../yaml/README.md#needs) to define dependency relationships between +[`needs` keyword](../yaml/index.md#needs) to define dependency relationships between your jobs. When GitLab knows the relationships between your jobs, it can run everything as fast as possible, and even skips into subsequent stages when possible. @@ -162,13 +162,13 @@ deploy_b: ## Child / Parent Pipelines In the examples above, it's clear we've got two types of things that could be built independently. -This is an ideal case for using [Child / Parent Pipelines](../parent_child_pipelines.md)) via -the [`trigger` keyword](../yaml/README.md#trigger). It separates out the configuration +This is an ideal case for using [Child / Parent Pipelines](parent_child_pipelines.md)) via +the [`trigger` keyword](../yaml/index.md#trigger). It separates out the configuration into multiple files, keeping things very simple. You can also combine this with: -- The [`rules` keyword](../yaml/README.md#rules): For example, have the child pipelines triggered only +- The [`rules` keyword](../yaml/index.md#rules): For example, have the child pipelines triggered only when there are changes to that area. -- The [`include` keyword](../yaml/README.md#include): Bring in common behaviors, ensuring +- The [`include` keyword](../yaml/index.md#include): Bring in common behaviors, ensuring you are not repeating yourself. - [DAG pipelines](#directed-acyclic-graph-pipelines) inside of child pipelines, achieving the benefits of both. diff --git a/doc/ci/pipelines/pipeline_artifacts.md b/doc/ci/pipelines/pipeline_artifacts.md index b80a056bbca..55555571f97 100644 --- a/doc/ci/pipelines/pipeline_artifacts.md +++ b/doc/ci/pipelines/pipeline_artifacts.md @@ -9,7 +9,7 @@ type: reference, howto Pipeline artifacts are files created by GitLab after a pipeline finishes. These are different than [job artifacts](job_artifacts.md) because they are not explicitly managed by the `.gitlab-ci.yml` definitions. -Pipeline artifacts are used by the [test coverage visualization feature](../../user/project/merge_requests/test_coverage_visualization.md) to collect coverage information. It uses the [`artifacts: reports`](../yaml/README.md#artifactsreports) CI/CD keyword. +Pipeline artifacts are used by the [test coverage visualization feature](../../user/project/merge_requests/test_coverage_visualization.md) to collect coverage information. It uses the [`artifacts: reports`](../yaml/index.md#artifactsreports) CI/CD keyword. ## Storage diff --git a/doc/ci/pipelines/pipeline_efficiency.md b/doc/ci/pipelines/pipeline_efficiency.md index 5bb435dddf6..91560a0420f 100644 --- a/doc/ci/pipelines/pipeline_efficiency.md +++ b/doc/ci/pipelines/pipeline_efficiency.md @@ -33,7 +33,7 @@ heavily influenced by the: - The ["critical path"](#directed-acyclic-graphs-dag-visualization), which represents the minimum and maximum pipeline duration. -Additional points to pay attention relate to [GitLab Runners](../runners/README.md): +Additional points to pay attention relate to [GitLab Runners](../runners/index.md): - Availability of the runners and the resources they are provisioned with. - Build dependencies and their installation time. @@ -62,7 +62,7 @@ It's important to understand and document the pipeline workflows, and discuss po actions and changes. Refactoring pipelines may need careful interaction between teams in the DevSecOps lifecycle. -Pipeline analysis can help identify issues with cost efficiency. For example, [runners](../runners/README.md) +Pipeline analysis can help identify issues with cost efficiency. For example, [runners](../runners/index.md) hosted with a paid cloud service may be provisioned with: - More resources than needed for CI/CD pipelines, wasting money. @@ -100,7 +100,7 @@ representation of pipeline health. Instance administrators have access to additional [performance metrics and self-monitoring](../../administration/monitoring/index.md). -You can fetch specific pipeline health metrics from the [API](../../api/README.md). +You can fetch specific pipeline health metrics from the [API](../../api/index.md). External monitoring tools can poll the API and verify pipeline health or collect metrics for long term SLA analytics. @@ -146,7 +146,7 @@ with embedded metric charts and all valuable details to analyze the problem. Review the storage use of the following to help analyze costs and efficiency: -- [Job artifacts](job_artifacts.md) and their [`expire_in`](../yaml/README.md#artifactsexpire_in) +- [Job artifacts](job_artifacts.md) and their [`expire_in`](../yaml/index.md#artifactsexpire_in) configuration. If kept for too long, storage usage grows and could slow pipelines down. - [Container registry](../../user/packages/container_registry/index.md) usage. - [Package registry](../../user/packages/package_registry/index.md) usage. @@ -162,9 +162,9 @@ make pipelines run faster and more efficiently. Try to find which jobs don't need to run in all situations, and use pipeline configuration to stop them from running: -- Use the [`interruptible`](../yaml/README.md#interruptible) keyword to stop old pipelines +- Use the [`interruptible`](../yaml/index.md#interruptible) keyword to stop old pipelines when they are superseded by a newer pipeline. -- Use [`rules`](../yaml/README.md#rules) to skip tests that aren't needed. For example, +- Use [`rules`](../yaml/index.md#rules) to skip tests that aren't needed. For example, skip backend tests when only the frontend code is changed. - Run non-essential [scheduled pipelines](schedules.md) less frequently. @@ -186,7 +186,7 @@ shouldn't run, saving pipeline resources. In a basic configuration, jobs always wait for all other jobs in earlier stages to complete before running. This is the simplest configuration, but it's also the slowest in most cases. [Directed Acyclic Graphs](../directed_acyclic_graph/index.md) and -[parent/child pipelines](../parent_child_pipelines.md) are more flexible and can +[parent/child pipelines](parent_child_pipelines.md) are more flexible and can be more efficient, but can also make pipelines harder to understand and analyze. ### Caching @@ -195,7 +195,7 @@ Another optimization method is to [cache](../caching/index.md) dependencies. If dependencies change rarely, like [NodeJS `/node_modules`](../caching/index.md#cache-nodejs-dependencies), caching can make pipeline execution much faster. -You can use [`cache:when`](../yaml/README.md#cachewhen) to cache downloaded dependencies +You can use [`cache:when`](../yaml/index.md#cachewhen) to cache downloaded dependencies even when a job fails. ### Docker Images diff --git a/doc/ci/pipelines/pipelines_for_merged_results.md b/doc/ci/pipelines/pipelines_for_merged_results.md new file mode 100644 index 00000000000..efa6a373ef3 --- /dev/null +++ b/doc/ci/pipelines/pipelines_for_merged_results.md @@ -0,0 +1,135 @@ +--- +stage: Verify +group: Pipeline Execution +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +type: reference +last_update: 2019-07-03 +--- + +# Pipelines for merged results **(PREMIUM)** + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/7380) in [GitLab Premium](https://about.gitlab.com/pricing/) 11.10. + +When you submit a merge request, you are requesting to merge changes from a +source branch into a target branch. By default, the CI pipeline runs jobs +against the source branch. + +With *pipelines for merged results*, the pipeline runs as if the changes from +the source branch have already been merged into the target branch. The commit shown for the pipeline does not exist on the source or target branches but represents the combined target and source branches. + + + +If the pipeline fails due to a problem in the target branch, you can wait until the +target is fixed and re-run the pipeline. +This new pipeline runs as if the source is merged with the updated target, and you +don't need to rebase. + +The pipeline does not automatically run when the target branch changes. Only changes +to the source branch trigger a new pipeline. If a long time has passed since the last successful +pipeline, you may want to re-run it before merge, to ensure that the source changes +can still be successfully merged into the target. + +When the merge request can't be merged, the pipeline runs against the source branch only. For example, when: + +- The target branch has changes that conflict with the changes in the source branch. +- The merge request is a [**Draft** merge request](../../user/project/merge_requests/drafts.md). + +In these cases, the pipeline runs as a [pipeline for merge requests](merge_request_pipelines.md) +and is labeled as `detached`. If these cases no longer exist, new pipelines +again run against the merged results. + +Any user who has developer [permissions](../../user/permissions.md) can run a +pipeline for merged results. + +## Prerequisites + +To enable pipelines for merge results: + +- You must have the [Maintainer role](../../user/permissions.md). +- You must be using [GitLab Runner](https://gitlab.com/gitlab-org/gitlab-runner) 11.9 or later. +- You must not be using + [fast forward merges](../../user/project/merge_requests/fast_forward_merge.md) yet. + To follow progress, see [#58226](https://gitlab.com/gitlab-org/gitlab/-/issues/26996). +- Your repository must be a GitLab repository, not an + [external repository](../ci_cd_for_external_repos/index.md). + +## Enable pipelines for merged results + +To enable pipelines for merged results for your project: + +1. [Configure your CI/CD configuration file](merge_request_pipelines.md#configure-pipelines-for-merge-requests) + so that the pipeline or individual jobs run for merge requests. +1. Visit your project's **Settings > General** and expand **Merge requests**. +1. Check **Enable merged results pipelines**. +1. Click **Save changes**. + +WARNING: +If you select the check box but don't configure your CI/CD to use +pipelines for merge requests, your merge requests may become stuck in an +unresolved state or your pipelines may be dropped. + +## Using Merge Trains + +When you enable [Pipelines for merged results](#pipelines-for-merged-results), +GitLab [automatically displays](merge_trains.md#add-a-merge-request-to-a-merge-train) +a **Start/Add Merge Train button**. + +Generally, this is a safer option than merging merge requests immediately, because your +merge request is evaluated with an expected post-merge result before the actual +merge happens. + +For more information, read the [documentation on Merge Trains](merge_trains.md). + +## Automatic pipeline cancellation + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/12996) in [GitLab Premium](https://about.gitlab.com/pricing/) 12.3. + +GitLab CI/CD can detect the presence of redundant pipelines, and cancels them +to conserve CI resources. + +When a user merges a merge request immediately in an ongoing merge +train, the train is reconstructed, because it recreates the expected +post-merge commit and pipeline. In this case, the merge train may already +have pipelines running against the previous expected post-merge commit. +These pipelines are considered redundant and are automatically +canceled. + +## Troubleshooting + +### Pipelines for merged results not created even with new change pushed to merge request + +Can be caused by some disabled feature flags. Please make sure that +the following feature flags are enabled on your GitLab instance: + +- `:merge_ref_auto_sync` + +To check and set these feature flag values, please ask an administrator to: + +1. Log into the Rails console of the GitLab instance: + + ```shell + sudo gitlab-rails console + ``` + +1. Check if the flags are enabled or not: + + ```ruby + Feature.enabled?(:merge_ref_auto_sync) + ``` + +1. If needed, enable the feature flags: + + ```ruby + Feature.enable(:merge_ref_auto_sync) + ``` + +### Intermittently pipelines fail by `fatal: reference is not a tree:` error + +Since pipelines for merged results are a run on a merge ref of a merge request +(`refs/merge-requests/<iid>/merge`), the Git reference could be overwritten at an +unexpected timing. For example, when a source or target branch is advanced. +In this case, the pipeline fails because of `fatal: reference is not a tree:` error, +which indicates that the checkout-SHA is not found in the merge ref. + +This behavior was improved at GitLab 12.4 by introducing [Persistent pipeline refs](../troubleshooting.md#fatal-reference-is-not-a-tree-error). +You should be able to create pipelines at any timings without concerning the error. diff --git a/doc/ci/pipelines/schedules.md b/doc/ci/pipelines/schedules.md index c6a40039816..9cb600ae551 100644 --- a/doc/ci/pipelines/schedules.md +++ b/doc/ci/pipelines/schedules.md @@ -52,14 +52,14 @@ is installed on. ### Using variables You can pass any number of arbitrary variables. They are available in -GitLab CI/CD so that they can be used in your [`.gitlab-ci.yml` file](../../ci/yaml/README.md). +GitLab CI/CD so that they can be used in your [`.gitlab-ci.yml` file](../../ci/yaml/index.md).  ### Using `rules` To configure a job to be executed only when the pipeline has been -scheduled, use the [`rules`](../yaml/README.md#rules) keyword. +scheduled, use the [`rules`](../yaml/index.md#rules) keyword. In this example, `make world` runs in scheduled pipelines, and `make build` runs in branch and tag pipelines: diff --git a/doc/ci/pipelines/settings.md b/doc/ci/pipelines/settings.md index 2e842856e55..db6fa7f4d23 100644 --- a/doc/ci/pipelines/settings.md +++ b/doc/ci/pipelines/settings.md @@ -6,125 +6,199 @@ disqus_identifier: 'https://docs.gitlab.com/ee/user/project/pipelines/settings.h type: reference, howto --- -# Pipeline settings **(FREE)** +# Customize pipeline configuration **(FREE)** -To reach the pipelines settings navigate to your project's -**Settings > CI/CD**. - -The following settings can be configured per project. +You can customize how pipelines run for your project. <i class="fa fa-youtube-play youtube" aria-hidden="true"></i> -For an overview, watch the video [GitLab CI Pipeline, Artifacts, and Environments](https://www.youtube.com/watch?v=PCKDICEe10s). +For an overview of pipelines, watch the video [GitLab CI Pipeline, Artifacts, and Environments](https://www.youtube.com/watch?v=PCKDICEe10s). Watch also [GitLab CI pipeline tutorial for beginners](https://www.youtube.com/watch?v=Jav4vbUrqII). -You can use the pipeline status to determine if a merge request can be merged: +## Change which users can view your pipelines -- [Merge when pipeline succeeds](../../user/project/merge_requests/merge_when_pipeline_succeeds.md). -- [Only allow merge requests to be merged if the pipeline succeeds](../../user/project/merge_requests/merge_when_pipeline_succeeds.md#only-allow-merge-requests-to-be-merged-if-the-pipeline-succeeds). +For public and internal projects, you can change who can see your: -## Git strategy +- Pipelines +- Job output logs +- Job artifacts +- [Pipeline security dashboard](../../user/application_security/security_dashboard/index.md#pipeline-security) -With Git strategy, you can choose the default way your repository is fetched -from GitLab in a job. +However: -There are two options. Using: +- Job output logs and artifacts are [never visible for Guest users and non-project members](https://gitlab.com/gitlab-org/gitlab/-/issues/25649). -- `git clone`, which is slower because it clones the repository from scratch - for every job, ensuring that the local working copy is always pristine. -- `git fetch`, which is default in GitLab and faster as it re-uses the local working copy (falling - back to clone if it doesn't exist). - This is recommended, especially for [large repositories](../large_repositories/index.md#git-strategy). +To change the visibility of your pipelines and related features: -The configured Git strategy can be overridden by the [`GIT_STRATEGY` variable](../runners/configure_runners.md#git-strategy) -in `.gitlab-ci.yml`. +1. On the top bar, select **Menu > Projects** and find your project. +1. On the left sidebar, select **Settings > CI/CD**. +1. Expand **General pipelines**. +1. Select or clear the **Public pipelines** checkbox. + When it is selected, pipelines and related features are visible: -## Git shallow clone + - For **public** projects, to everyone. + - For **internal** projects, to all logged-in users except [external users](../../user/permissions.md#external-users). + - For **private** projects, to all project members (Guest or higher). -> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/28919) in GitLab 12.0. + When it is cleared: -It is possible to limit the number of changes that GitLab CI/CD fetches when cloning -a repository. Setting a limit to `git depth` can speed up Pipelines execution. + - For **public** projects, pipelines are visible to everyone. Related features are visible + only to project members (Reporter or higher). + - For **internal** projects, pipelines are visible to all logged in users except [external users](../../user/permissions.md#external-users). + Related features are visible only to project members (Reporter or higher). + - For **private** projects, pipelines and related features are visible to project members (Reporter or higher) only. -In GitLab 12.0 and later, newly created projects automatically have a default -`git depth` value of `50`. The maximum allowed value is `1000`. +## Auto-cancel redundant pipelines + +You can set pending or running pipelines to cancel automatically when a new pipeline runs on the same branch. You can enable this in the project settings: + +1. On the top bar, select **Menu > Projects** and find your project. +1. On the left sidebar, select **Settings > CI/CD**. +1. Expand **General Pipelines**. +1. Select the **Auto-cancel redundant pipelines** checkbox. +1. Select **Save changes**. + +Use the [`interruptible`](../yaml/index.md#interruptible) keyword to indicate if a +running job can be cancelled before it completes. -To disable shallow clone and make GitLab CI/CD fetch all branches and tags each time, -keep the value empty or set to `0`. +## Skip outdated deployment jobs + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/25276) in GitLab 12.9. + +Your project may have multiple concurrent deployment jobs that are +scheduled to run in the same time frame. + +This can lead to a situation where an older deployment job runs after a +newer one, which may not be what you want. -This value can also be [overridden by `GIT_DEPTH`](../large_repositories/index.md#shallow-cloning) variable in `.gitlab-ci.yml` file. +To avoid this scenario: -## Timeout +1. On the top bar, select **Menu > Projects** and find your project. +1. On the left sidebar, select **Settings > CI/CD**. +1. Expand **General pipelines**. +1. Select the **Skip outdated deployment jobs** checkbox. +1. Select **Save changes**. -Timeout defines the maximum amount of time in minutes that a job is able run. -This is configurable under your project's **Settings > CI/CD > General pipelines settings**. -The default value is 60 minutes. Decrease the time limit if you want to impose -a hard limit on your jobs' running time or increase it otherwise. In any case, -if the job surpasses the threshold, it is marked as failed. +Older deployment job are skipped when a new deployment starts. -### Timeout overriding for runners +For more information, see [Deployment safety](../environments/deployment_safety.md). -> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/17221) in GitLab 10.7. +## Retry outdated jobs -Project defined timeout (either specific timeout set by user or the default -60 minutes timeout) may be [overridden for runners](../runners/configure_runners.md#set-maximum-job-timeout-for-a-runner). +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/211339) in GitLab 13.6. -## Maximum artifacts size **(FREE SELF)** +A deployment job can fail because a newer one has run. If you retry the failed deployment job, the +environment could be overwritten with older source code. If you click **Retry**, a modal warns you +about this and asks for confirmation. -For information about setting a maximum artifact size for a project, see -[Maximum artifacts size](../../user/admin_area/settings/continuous_integration.md#maximum-artifacts-size). +For more information, see [Deployment safety](../environments/deployment_safety.md). -## Custom CI/CD configuration file +## Specify a custom CI/CD configuration file > [Support for external `.gitlab-ci.yml` locations](https://gitlab.com/gitlab-org/gitlab/-/issues/14376) introduced in GitLab 12.6. -By default we look for the `.gitlab-ci.yml` file in the project's root -directory. If needed, you can specify an alternate path and filename, including locations outside the project. +GitLab expects to find the CI/CD configuration file (`.gitlab-ci.yml`) in the project's root +directory. However, you can specify an alternate filename path, including locations outside the project. To customize the path: -1. Go to the project's **Settings > CI/CD**. -1. Expand the **General pipelines** section. -1. Provide a value in the **CI/CD configuration file** field. -1. Click **Save changes**. +1. On the top bar, select **Menu > Projects** and find your project. +1. On the left sidebar, select **Settings > CI/CD**. +1. Expand **General pipelines**. +1. In the **CI/CD configuration file** field, enter the filename. If the file: + - Is not in the root directory, include the path. + - Is in a different project, include the group and project name. + - Is on an external site, enter the full URL. +1. Select **Save changes**. + +### Custom CI/CD configuration file examples -If the CI/CD configuration file is stored in the repository in a non-default -location, the path must be relative to the root directory. Examples of valid -paths and file names include: +If the CI/CD configuration file is not in the root directory, the path must be relative to it. +For example: -- `.gitlab-ci.yml` (default) -- `.my-custom-file.yml` - `my/path/.gitlab-ci.yml` - `my/path/.my-custom-file.yml` -If hosting the CI/CD configuration file on an external site, the URL link must end with `.yml`: +If the CI/CD configuration file is on an external site, the URL must end with `.yml`: - `http://example.com/generate/ci/config.yml` -If hosting the CI/CD configuration file in a different project in GitLab, the path must be relative +If the CI/CD configuration file is in a different project, the path must be relative to the root directory in the other project. Include the group and project name at the end: - `.gitlab-ci.yml@mygroup/another-project` - `my/path/.my-custom-file.yml@mygroup/another-project` -Hosting the configuration file in a separate project allows stricter control of the -configuration file. For example: +If the configuration file is in a separate project, you can more set more granular permissions. For example: - Create a public project to host the configuration file. - Give write permissions on the project only to users who are allowed to edit the file. -Other users and projects can access the configuration file without being +Then other users and projects can access the configuration file without being able to edit it. -## Test coverage parsing +## Choose the default Git strategy + +You can choose how your repository is fetched from GitLab when a job runs. + +1. On the top bar, select **Menu > Projects** and find your project. +1. On the left sidebar, select **Settings > CI/CD**. +1. Expand **General pipelines**. +1. Under **Git strategy**, select an option: + - `git clone` is slower because it clones the repository from scratch + for every job. However, the local working copy is always pristine. + - `git fetch` is faster because it re-uses the local working copy (and falls + back to clone if it doesn't exist). This is recommended, especially for + [large repositories](../large_repositories/index.md#git-strategy). + +The configured Git strategy can be overridden by the [`GIT_STRATEGY` variable](../runners/configure_runners.md#git-strategy) +in the `.gitlab-ci.yml` file. + +## Limit the number of changes fetched during clone + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/28919) in GitLab 12.0. + +You can limit the number of changes that GitLab CI/CD fetches when it clones +a repository. + +1. On the top bar, select **Menu > Projects** and find your project. +1. On the left sidebar, select **Settings > CI/CD**. +1. Expand **General pipelines**. +1. Under **Git strategy**, under **Git shallow clone**, enter a value. + The maximum value is `1000`. To disable shallow clone and make GitLab CI/CD + fetch all branches and tags each time, keep the value empty or set to `0`. + +In GitLab 12.0 and later, newly created projects automatically have a default +`git depth` value of `50`. + +This value can be overridden by the [`GIT_DEPTH` variable](../large_repositories/index.md#shallow-cloning) +in the `.gitlab-ci.yml` file. + +## Set a limit for how long jobs can run + +You can define how long a job can run before it times out. + +1. On the top bar, select **Menu > Projects** and find your project. +1. On the left sidebar, select **Settings > CI/CD**. +1. Expand **General pipelines**. +1. In the **Timeout** field, enter the number of minutes, or a human-readable value like `2 hours`. + +Jobs that exceed the timeout are marked as failed. + +You can override this value [for individual runners](../runners/configure_runners.md#set-maximum-job-timeout-for-a-runner). + +## Add test coverage results to a merge request -If you use test coverage in your code, GitLab can capture its output in the -job log using a regular expression. +If you use test coverage in your code, you can use a regular expression to +find coverage results in the job log. You can then include these results +in the merge request in GitLab. -In your project, go to **Settings > CI/CD** and expand the **General pipelines** -section. Enter the regular expression in the **Test coverage parsing** field. +1. On the top bar, select **Menu > Projects** and find your project. +1. On the left sidebar, select **Settings > CI/CD**. +1. Expand **General pipelines**. +1. In the **Test coverage parsing** field, enter a regular expression. + Leave blank to disable this feature. -Leave blank if you want to disable it or enter a Ruby regular expression. You -can use <https://rubular.com> to test your regex. The regex returns the **last** +You can use <https://rubular.com> to test your regex. The regex returns the **last** match found in the output. If the pipeline succeeds, the coverage is shown in the merge request widget and @@ -135,6 +209,10 @@ averaged.  +### Test coverage examples + +Use this regex for commonly used test tools. + <!-- vale gitlab.Spelling = NO --> - Simplecov (Ruby). Example: `\(\d+.\d+\%\) covered`. @@ -148,27 +226,51 @@ averaged. - `mix test --cover` (Elixir). Example: `\d+.\d+\%\s+\|\s+Total`. - JaCoCo (Java/Kotlin). Example: `Total.*?([0-9]{1,3})%`. - `go test -cover` (Go). Example: `coverage: \d+.\d+% of statements`. -- .Net (OpenCover). Example: `(Visited Points).*\((.*)\)`. -- .Net (`dotnet test` line coverage). Example: `Total\s*\|\s*(\d+(?:\.\d+)?)`. +- .NET (OpenCover). Example: `(Visited Points).*\((.*)\)`. +- .NET (`dotnet test` line coverage). Example: `Total\s*\|\s*(\d+(?:\.\d+)?)`. <!-- vale gitlab.Spelling = YES --> -### Code Coverage history +### View code coverage history > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/209121) the ability to download a `.csv` in GitLab 12.10. > - [Graph introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/33743) in GitLab 13.1. To see the evolution of your project code coverage over time, -you can view a graph or download a CSV file with this data. From your project: +you can view a graph or download a CSV file with this data. + +1. On the top bar, select **Menu > Projects** and find your project. +1. On the left sidebar, select **Analytics > Repository**. -1. Go to **Project Analytics > Repository** to see the historic data for each job listed in the dropdown above the graph. -1. If you want a CSV file of that data, click **Download raw data (`.csv`)** +The historic data for each job is listed in the dropdown above the graph. + +To view a CSV file of the data, select **Download raw data (`.csv`)**.  Code coverage data is also [available at the group level](../../user/group/repositories_analytics/index.md). -### Removing color codes +### Coverage check approval rule **(PREMIUM)** + +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/15765) in GitLab 14.0. +> - [Made configurable in Project Settings](https://gitlab.com/gitlab-org/gitlab/-/issues/331001) in GitLab 14.1. + +You can implement merge request approvals to require approval by selected users or a group +when merging a merge request would cause the project's test coverage to decline. + +Follow these steps to enable the `Coverage-Check` MR approval rule: + +1. Go to your project and select **Settings > General**. +1. Expand **Merge request approvals**. +1. Select **Enable** next to the `Coverage-Check` approval rule. +1. Select the **Target branch**. +1. Set the number of **Approvals required** to greater than zero. +1. Select the users or groups to provide approval. +1. Select **Add approval rule**. + + + +### Remove color codes from code coverage Some test coverage tools output with ANSI color codes that aren't parsed correctly by the regular expression. This causes coverage @@ -184,93 +286,20 @@ For example: lein cloverage | perl -pe 's/\e\[?.*?[\@-~]//g' ``` -## Visibility of pipelines - -Pipeline visibility is determined by: - -- Your current [user access level](../../user/permissions.md). -- The **Public pipelines** project setting under your project's **Settings > CI/CD > General pipelines**. - -NOTE: -If the project visibility is set to **Private**, the [**Public pipelines** setting has no effect](../enable_or_disable_ci.md#per-project-user-setting). - -This also determines the visibility of these related features: - -- Job output logs -- Job artifacts -- The [pipeline security dashboard](../../user/application_security/security_dashboard/index.md#pipeline-security) **(ULTIMATE)** - -Job logs and artifacts are [not visible for guest users and non-project members](https://gitlab.com/gitlab-org/gitlab/-/issues/25649). - -If **Public pipelines** is enabled (default): - -- For **public** projects, anyone can view the pipelines and related features. -- For **internal** projects, any logged in user except [external users](../../user/permissions.md#external-users) can view the pipelines - and related features. -- For **private** projects, any project member (guest or higher) can view the pipelines - and related features. - -If **Public pipelines** is disabled: - -- For **public** projects, anyone can view the pipelines, but only members - (reporter or higher) can access the related features. -- For **internal** projects, any logged in user except [external users](../../user/permissions.md#external-users) can view the pipelines. - However, only members (reporter or higher) can access the job related features. -- For **private** projects, only project members (reporter or higher) - can view the pipelines or access the related features. - -## Auto-cancel redundant pipelines - -You can set pending or running pipelines to cancel automatically when a new pipeline runs on the same branch. You can enable this in the project settings: - -1. Go to **Settings > CI/CD**. -1. Expand **General Pipelines**. -1. Check the **Auto-cancel redundant pipelines** checkbox. -1. Click **Save changes**. - -Use the [`interruptible`](../yaml/README.md#interruptible) keyword to indicate if a -running job can be cancelled before it completes. - -## Skip outdated deployment jobs - -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/25276) in GitLab 12.9. +## Pipeline badges -Your project may have multiple concurrent deployment jobs that are -scheduled to run in the same time frame. +Pipeline badges indicate the pipeline status and a test coverage value +for your project. These badges are determined by the latest successful pipeline. -This can lead to a situation where an older deployment job runs after a -newer one, which may not be what you want. +### View the code for the pipeline status and coverage reports badges -To avoid this scenario: +You can view the exact link for your badges. Then you can embed the badge in your HTML +or Markdown pages. -1. Go to **Settings > CI/CD**. +1. On the top bar, select **Menu > Projects** and find your project. +1. On the left sidebar, select **Settings > CI/CD**. 1. Expand **General pipelines**. -1. Check the **Skip outdated deployment jobs** checkbox. -1. Click **Save changes**. - -When enabled, any older deployments job are skipped when a new deployment starts. - -For more information, see [Deployment safety](../environments/deployment_safety.md). - -## Retry outdated jobs - -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/211339) in GitLab 13.6. - -A deployment job can fail because a newer one has run. If you retry the failed deployment job, the -environment could be overwritten with older source code. If you click **Retry**, a modal warns you -about this and asks for confirmation. - -For more information, see [Deployment safety](../environments/deployment_safety.md). - -## Pipeline Badges - -In the pipelines settings page you can find pipeline status and test coverage -badges for your project. The latest successful pipeline is used to read -the pipeline status and test coverage values. - -Visit the pipelines settings page in your project to see the exact link to -your badges. You can also see ways to embed the badge image in your HTML or Markdown -pages. +1. In the **Pipeline status** or **Coverage report** sections, view the URLs for the images.  @@ -286,7 +315,7 @@ Depending on the status of your pipeline, a badge can have the following values: - `canceled` - `unknown` -You can access a pipeline status badge image using the following link: +You can access a pipeline status badge image by using the following link: ```plaintext https://gitlab.example.com/<namespace>/<project>/badges/<branch>/pipeline.svg @@ -294,7 +323,7 @@ https://gitlab.example.com/<namespace>/<project>/badges/<branch>/pipeline.svg #### Display only non-skipped status -If you want the pipeline status badge to only display the last non-skipped status, you can use the `?ignore_skipped=true` query parameter: +To make the pipeline status badge display only the last non-skipped status, use the `?ignore_skipped=true` query parameter: ```plaintext https://gitlab.example.com/<namespace>/<project>/badges/<branch>/pipeline.svg?ignore_skipped=true @@ -302,20 +331,20 @@ https://gitlab.example.com/<namespace>/<project>/badges/<branch>/pipeline.svg?ig ### Test coverage report badge -GitLab makes it possible to define the regular expression for the [coverage report](#test-coverage-parsing), +You can define the regular expression for the [coverage report](#add-test-coverage-results-to-a-merge-request) that each job log is matched against. This means that each job in the pipeline can have the test coverage percentage value defined. -The test coverage badge can be accessed using following link: +To access the test coverage badge, use the following link: ```plaintext https://gitlab.example.com/<namespace>/<project>/badges/<branch>/coverage.svg ``` -If you would like to get the coverage report from a specific job, you can add +To get the coverage report from a specific job, add the `job=coverage_job_name` parameter to the URL. For example, the following Markdown code embeds the test coverage report badge of the `coverage` job -into your `README.md`: +in your `README.md`: ```markdown  |
