diff options
Diffstat (limited to 'doc/ci/pipelines.md')
| -rw-r--r-- | doc/ci/pipelines.md | 431 |
1 files changed, 246 insertions, 185 deletions
diff --git a/doc/ci/pipelines.md b/doc/ci/pipelines.md index 4f3106c6dc6..1ad3d516df9 100644 --- a/doc/ci/pipelines.md +++ b/doc/ci/pipelines.md @@ -1,271 +1,348 @@ -# Introduction to pipelines and jobs +--- +type: reference +--- + +# Creating and using CI/CD pipelines > Introduced in GitLab 8.8. +## Introduction + +Pipelines are the top-level component of continuous integration, delivery, and deployment. + +Pipelines comprise: + +- Jobs that define what to run. For example, code compilation or test runs. +- Stages that define when and how to run. For example, that tests run only after code compilation. + +Multiple jobs in the same stage are executed by [Runners](runners/README.md) in parallel, if there are enough concurrent [Runners](runners/README.md). + +If all the jobs in a stage: + +- Succeed, the pipeline moves on to the next stage. +- Fail, the next stage is not (usually) executed and the pipeline ends early. + NOTE: **Note:** -If you have a [mirrored repository where GitLab pulls from](https://docs.gitlab.com/ee/workflow/repository_mirroring.html#pulling-from-a-remote-repository-starter), +If you have a [mirrored repository that GitLab pulls from](../workflow/repository_mirroring.md#pulling-from-a-remote-repository-starter), you may need to enable pipeline triggering in your project's **Settings > Repository > Pull from a remote repository > Trigger pipelines for mirror updates**. -## Pipelines +### Simple pipeline example -A pipeline is a group of [jobs] that get executed in [stages]. -All of the jobs in a stage are executed in parallel (if there are enough -concurrent [Runners]), and if they all succeed, the pipeline moves on to the -next stage. If one of the jobs fails, the next stage is not (usually) -executed. You can access the pipelines page in your project's **Pipelines** tab. +As an example, imagine a pipeline consisting of four stages, executed in the following order: -In the following image you can see that the pipeline consists of four stages -(`build`, `test`, `staging`, `production`) each one having one or more jobs. +- `build`, with a job called `compile`. +- `test`, with two jobs called `test` and `test2`. +- `staging`, with a job called `deploy-to-stage`. +- `production`, with a job called `deploy-to-prod`. ->**Note:** -GitLab capitalizes the stages' names when shown in the [pipeline graphs](#pipeline-graphs). +## Visualizing pipelines - +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/5742) in GitLab 8.11. -## Types of pipelines +Pipelines can be complex structures with many sequential and parallel jobs. -There are three types of pipelines that often use the single shorthand of "pipeline". People often talk about them as if each one is "the" pipeline, but really, they're just pieces of a single, comprehensive pipeline. +To make it easier to understand the flow of a pipeline, GitLab has pipeline graphs for viewing pipelines +and their statuses. - +Pipeline graphs can be displayed in two different ways, depending on the page you +access the graph from. -1. **CI Pipeline**: Build and test stages defined in `.gitlab-ci.yml`. -1. **Deploy Pipeline**: Deploy stage(s) defined in `.gitlab-ci.yml` The flow of deploying code to servers through various stages: e.g. development to staging to production. -1. **Project Pipeline**: Cross-project CI dependencies [triggered via API][triggers], particularly for micro-services, but also for complicated build dependencies: e.g. api -> front-end, ce/ee -> omnibus. +NOTE: **Note:** +GitLab capitalizes the stages' names when shown in the pipeline graphs (below). -## Development workflows +### Regular pipeline graphs -Pipelines accommodate several development workflows: +Regular pipeline graphs show the names of the jobs of each stage. Regular pipeline graphs can +be found when you are on a [single pipeline page](#accessing-pipelines). For example: -1. **Branch Flow** (e.g. different branch for dev, qa, staging, production). -1. **Trunk-based Flow** (e.g. feature branches and single master branch, possibly with tags for releases). -1. **Fork-based Flow** (e.g. merge requests come from forks). + -Example continuous delivery flow: +### Pipeline mini graphs - +Pipeline mini graphs take less space and can tell you at a +quick glance if all jobs passed or something failed. The pipeline mini graph can +be found when you navigate to: -## Jobs +- The pipelines index page. +- A single commit page. +- A merge request page. -Jobs can be defined in the [`.gitlab-ci.yml`][jobs-yaml] file. Not to be -confused with a `build` job or `build` stage. +Pipeline mini graphs allow you to see all related jobs for a single commit and the net result +of each stage of your pipeline. This allows you to quickly see what failed and +fix it. -## Defining pipelines +Stages in pipeline mini graphs are collapsible. Hover your mouse over them and click to expand their jobs. -Pipelines are defined in `.gitlab-ci.yml` by specifying [jobs] that run in -[stages]. +| Mini graph | Mini graph expanded | +|:-------------------------------------------------------------|:---------------------------------------------------------------| +|  |  | -See the reference [documentation for jobs](yaml/README.md#jobs). +### Job ordering in pipeline graphs -## Manually executing pipelines +Job ordering depends on the type of pipeline graph. For [regular pipeline graphs](#regular-pipeline-graphs), jobs are sorted by name. -Pipelines can be manually executed, with predefined or manually-specified [variables](variables/README.md). +For [pipeline mini graphs](#pipeline-mini-graphs) ([introduced](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/9760) +in GitLab 9.0), jobs are sorted by severity and then by name. -To execute a pipeline manually: +The order of severity is: -1. Navigate to your project's **CI/CD > Pipelines**. -1. Click on the **Run Pipeline** button. -1. Select the branch to run the pipeline for and enter any environment variables required for the pipeline run. +- failed +- warning +- pending +- running +- manual +- scheduled +- canceled +- success +- skipped +- created -## Seeing pipeline status +For example: -You can find the current and historical pipeline runs under your project's -**Pipelines** tab. Clicking on a pipeline will show the jobs that were run for -that pipeline. - - + -## Seeing job status +### How pipeline duration is calculated -When you visit a single pipeline you can see the related jobs for that pipeline. -Clicking on an individual job will show you its job trace, and allow you to -cancel the job, retry it, or erase the job trace. +Total running time for a given pipeline excludes retries and pending +(queued) time. - +Each job is represented as a `Period`, which consists of: -## Seeing the failure reason for jobs +- `Period#first` (when the job started). +- `Period#last` (when the job finished). -> [Introduced][ce-17782] in GitLab 10.7. +A simple example is: -When a pipeline fails or is allowed to fail, there are several places where you -can quickly check the reason it failed: +- A (1, 3) +- B (2, 4) +- C (6, 7) -- **In the pipeline graph** present on the pipeline detail view. -- **In the pipeline widgets** present in the merge requests and commit pages. -- **In the job views** present in the global and detailed views of a job. +In the example: -In any case, if you hover over the failed job you can see the reason it failed. +- A begins at 1 and ends at 3. +- B begins at 2 and ends at 4. +- C begins at 6 and ends at 7. - +Visually, it can be viewed as: -From [GitLab 10.8][ce-17814] you can also see the reason it failed on the Job detail page. +```text +0 1 2 3 4 5 6 7 + AAAAAAA + BBBBBBB + CCCC +``` -## Pipeline graphs +The union of A, B, and C is (1, 4) and (6, 7). Therefore, the total running time is: -> [Introduced][ce-5742] in GitLab 8.11. +```text +(4 - 1) + (7 - 6) => 4 +``` -Pipelines can be complex structures with many sequential and parallel jobs. -To make it a little easier to see what is going on, you can view a graph -of a single pipeline and its status. +## Configuring pipelines -A pipeline graph can be shown in two different ways depending on what page you -are on. +Pipelines, and their component jobs and stages, are defined in the [`.gitlab-ci.yml`](yaml/README.md) file for each project. ---- +In particular: -The regular pipeline graph that shows the names of the jobs of each stage can -be found when you are on a [single pipeline page](#seeing-pipeline-status). +- Jobs are the [basic configuration](yaml/README.html#introduction) component. +- Stages are defined using the [`stages`](yaml/README.html#stages) keyword. - +For all available configuration options, see the [GitLab CI/CD Pipeline Configuration Reference](yaml/README.md). -Then, there is the pipeline mini graph which takes less space and can give you a -quick glance if all jobs pass or something failed. The pipeline mini graph can -be found when you visit: +### Settings and schedules -- The pipelines index page. -- A single commit page. -- A merge request page. +In addition to configuring jobs through `.gitlab-ci.yml`, additional configuration options are available +through the GitLab UI: -That way, you can see all related jobs for a single commit and the net result -of each stage of your pipeline. This allows you to quickly see what failed and -fix it. Stages in pipeline mini graphs are collapsible. Hover your mouse over -them and click to expand their jobs. +- Pipeline settings for each project. For more information, see [Pipeline settings](../user/project/pipelines/settings.md). +- Schedules for pipelines. For more information, see [Pipeline schedules](../user/project/pipelines/schedules.md). -| **Mini graph** | **Mini graph expanded** | -| :------------: | :---------------------: | -|  |  | +### Grouping jobs -### Grouping similar jobs in the pipeline graph +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/6242) in GitLab 8.12. -> [Introduced][ce-6242] in GitLab 8.12. +If you have many similar jobs, your [pipeline graph](#visualizing-pipelines) becomes long and hard +to read. -If you have many similar jobs, your pipeline graph becomes very long and hard -to read. For that reason, similar jobs can automatically be grouped together. +For that reason, similar jobs can automatically be grouped together. If the job names are formatted in certain ways, they will be collapsed into a single group in regular pipeline graphs (not the mini graphs). + You'll know when a pipeline has grouped jobs if you don't see the retry or cancel button inside them. Hovering over them will show the number of grouped jobs. Click to expand them.  -The basic requirements is that there are two numbers separated with one of +#### Configuring grouping + +In the pipeline [configuration file](yaml/README.md), job names must include two numbers separated with one of the following (you can even use them interchangeably): -- A space (` `) -- A slash (`/`) -- A colon (`:`) +- A space. +- A slash (`/`). +- A colon (`:`). + +NOTE: **Note:** +More specifically, it uses [this](https://gitlab.com/gitlab-org/gitlab-ce/blob/2f3dc314f42dbd79813e6251792853bc231e69dd/app/models/commit_status.rb#L99) regular expression: `\d+[\s:\/\\]+\d+\s*`. ->**Note:** -More specifically, [it uses][regexp] this regular expression: `\d+[\s:\/\\]+\d+\s*`. +#### How grouping works The jobs will be ordered by comparing those two numbers from left to right. You usually want the first to be the index and the second the total. For example, the following jobs will be grouped under a job named `test`: -- `test 0 3` => `test` -- `test 1 3` => `test` -- `test 2 3` => `test` +- `test 0 3` +- `test 1 3` +- `test 2 3` The following jobs will be grouped under a job named `test ruby`: -- `test 1:2 ruby` => `test ruby` -- `test 2:2 ruby` => `test ruby` +- `test 1:2 ruby` +- `test 2:2 ruby` The following jobs will be grouped under a job named `test ruby` as well: -- `1/3 test ruby` => `test ruby` -- `2/3 test ruby` => `test ruby` -- `3/3 test ruby` => `test ruby` +- `1/3 test ruby` +- `2/3 test ruby` +- `3/3 test ruby` -### Manual actions from the pipeline graph +### Pipelines for merge requests -> [Introduced][ce-7931] in GitLab 8.15. +GitLab supports configuring pipelines that run only for merge requests. For more information, see +[Pipelines for merge requests](merge_request_pipelines/index.md). -[Manual actions][manual] allow you to require manual interaction before moving -forward with a particular job in CI. Your entire pipeline can run automatically, -but the actual [deploy to production][env-manual] will require a click. +### Badges -You can do this straight from the pipeline graph. Just click on the play button -to execute that particular job. For example, in the image below, the `production` -stage has a job with a manual action. +Pipeline status and test coverage report badges are available and configurable for each project. - +For information on adding pipeline badges to projects, see [Pipeline badges](../user/project/pipelines/settings.md#pipeline-badges). -### Delay a particular job in the pipeline graph +## Multi-project pipelines **[PREMIUM]** -> [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/21767) in GitLab 11.4. +Pipelines for different projects can be combined and visualized together. -When you do not want to run a job immediately, you can [delay the job to run after a certain period](yaml/README.md#whendelayed). -This is especially useful for timed incremental rollout that new code is rolled out gradually. -For example, if you start rolling out new code and users do not experience trouble, GitLab automatically completes the deployment from 0% to 100%. -Alternatively, if you start rolling out and you noticed that a few users experience trouble with the version, -you can stop the timed incremental rollout by canceling the pipeline, and [rolling](environments.md#rolling-back-changes) it back to the stable version. +For more information, see [Multi-project pipelines](multi_project_pipelines.md). - +## Working with pipelines -### Ordering of jobs in pipeline graphs +In general, pipelines are executed automatically and require no intervention once created. -**Regular pipeline graph** +However, there are instances where you'll need to interact with pipelines. These are documented below. -In the single pipeline page, jobs are sorted by name. +### Manually executing pipelines -**Mini pipeline graph** +Pipelines can be manually executed, with predefined or manually-specified [variables](variables/README.md). -> [Introduced][ce-9760] in GitLab 9.0. +You might do this if the results of a pipeline (for example, a code build) is required outside the normal +operation of the pipeline. -In the pipeline mini graphs, the jobs are sorted first by severity and then -by name. The order of severity is: +To execute a pipeline manually: -- failed -- warning -- pending -- running -- manual -- scheduled -- canceled -- success -- skipped -- created +1. Navigate to your project's **CI/CD > Pipelines**. +1. Click on the **Run Pipeline** button. +1. On the **Run Pipeline** page: + 1. Select the branch to run the pipeline for in the **Create for** field. + 1. Enter any [environment variables](variables/README.md) required for the pipeline run. + 1. Click the **Create pipeline** button. - +The pipeline will execute the jobs as configured. -## How the pipeline duration is calculated +### Accessing pipelines -Total running time for a given pipeline would exclude retries and pending -(queue) time. We could reduce this problem down to finding the union of -periods. +You can find the current and historical pipeline runs under your project's +**CI/CD > Pipelines** page. Clicking on a pipeline will show the jobs that were run for +that pipeline. -So each job would be represented as a `Period`, which consists of -`Period#first` as when the job started and `Period#last` as when the -job was finished. A simple example here would be: + -- A (1, 3) -- B (2, 4) -- C (6, 7) +You can also access pipelines for a merge request by navigating to its **Pipelines** tab. -Here A begins from 1, and ends to 3. B begins from 2, and ends to 4. -C begins from 6, and ends to 7. Visually it could be viewed as: +### Accessing individual jobs -``` -0 1 2 3 4 5 6 7 - AAAAAAA - BBBBBBB - CCCC -``` +When you access a pipeline, you can see the related jobs for that pipeline. -The union of A, B, and C would be (1, 4) and (6, 7), therefore the -total running time should be: +Clicking on an individual job will show you its job trace, and allow you to: -``` -(4 - 1) + (7 - 6) => 4 -``` +- Cancel the job. +- Retry the job. +- Erase the job trace. -## Badges +### Seeing the failure reason for jobs -Pipeline status and test coverage report badges are available. You can find their -respective link in the [Pipelines settings] page. +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/17782) in GitLab 10.7. + +When a pipeline fails or is allowed to fail, there are several places where you +can quickly check the reason it failed: + +- In the pipeline graph, on the pipeline detail view. +- In the pipeline widgets, in the merge requests and commit pages. +- In the job views, in the global and detailed views of a job. + +In each place, if you hover over the failed job you can see the reason it failed. + + + +From [GitLab 10.8](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/17814), +you can also see the reason it failed on the Job detail page. + +### Manual actions from pipeline graphs + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/7931) in GitLab 8.15. + +Manual actions, configured using the [`when:manual`](yaml/README.md#whenmanual) parameter, +allow you to require manual interaction before moving forward in the pipeline. + +You can do this straight from the pipeline graph. Just click on the play button +to execute that particular job. + +For example, your pipeline start automatically, but require manual action to +[deploy to production](environments.md#configuring-manual-deployments). In the example below, the `production` +stage has a job with a manual action. + + + +### Delay a job in a pipeline graph + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/21767) in GitLab 11.4. + +When you do not want to run a job immediately, you can use the [`when:delayed`](yaml/README.md#whendelayed) parameter to +delay a job's execution for a certain period. + +This is especially useful for timed incremental rollout where new code is rolled out gradually. + +For example, if you start rolling out new code and: + +- Users do not experience trouble, GitLab can automatically complete the deployment from 0% to 100%. +- Users experience trouble with the new code, you can stop the timed incremental rollout by canceling the pipeline + and [rolling](environments.md#retrying-and-rolling-back) back to the last stable version. + + + +### Using the API + +GitLab provides API endpoints to: + +- Perform basic functions. For more information, see [Pipelines API](../api/pipelines.md). +- Maintain pipeline schedules. For more information, see [Pipeline schedules API](../api/pipeline_schedules.md). +- Trigger pipeline runs. For more information, see: + - [Triggering pipelines through the API](triggers/README.md). + - [Pipeline triggers API](../api/pipeline_triggers.md). + +### Start multiple manual actions in a stage + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/27188) in GitLab 11.11. + +Multiple manual actions in a single stage can be started at the same time using the "Play all manual" button. +Once the user clicks this button, each individual manual action will be triggered and refreshed +to an updated status. + +This functionality is only available: + +- For users with at least Developer access. +- If the the stage contains [manual actions](#manual-actions-from-pipeline-graphs). ## Security on protected branches @@ -276,14 +353,14 @@ The following actions are allowed on protected branches only if the user is [allowed to merge or push](../user/project/protected_branches.md#using-the-allowed-to-merge-and-allowed-to-push-settings) on that specific branch: -- Run **manual pipelines** (using [Web UI](#manually-executing-pipelines) or Pipelines API). -- Run **scheduled pipelines**. -- Run pipelines using **triggers**. -- Trigger **manual actions** on existing pipelines. -- **Retry/cancel** existing jobs (using Web UI or Pipelines API). +- Run manual pipelines (using the [Web UI](#manually-executing-pipelines) or pipelines API). +- Run scheduled pipelines. +- Run pipelines using triggers. +- Trigger manual actions on existing pipelines. +- Retry or cancel existing jobs (using the Web UI or pipelines API). **Variables** marked as **protected** are accessible only to jobs that -run on protected branches, avoiding untrusted users to get unintended access to +run on protected branches, preventing untrusted users getting unintended access to sensitive information like deployment credentials and tokens. **Runners** marked as **protected** can run jobs only on protected @@ -291,19 +368,3 @@ branches, avoiding untrusted code to be executed on the protected runner and preserving deployment keys and other credentials from being unintentionally accessed. In order to ensure that jobs intended to be executed on protected runners will not use regular runners, they must be tagged accordingly. - -[jobs]: #jobs -[jobs-yaml]: yaml/README.md#jobs -[manual]: yaml/README.md#whenmanual -[env-manual]: environments.md#manually-deploying-to-environments -[stages]: yaml/README.md#stages -[runners]: runners/README.html -[pipelines settings]: ../user/project/pipelines/settings.md -[triggers]: triggers/README.md -[ce-5742]: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/5742 -[ce-6242]: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/6242 -[ce-7931]: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/7931 -[ce-9760]: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/9760 -[ce-17782]: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/17782 -[ce-17814]: https://gitlab.com/gitlab-org/gitlab-ce/merge_requests/17814 -[regexp]: https://gitlab.com/gitlab-org/gitlab-ce/blob/2f3dc314f42dbd79813e6251792853bc231e69dd/app/models/commit_status.rb#L99 |