diff options
Diffstat (limited to 'doc/ci/pipelines')
| -rw-r--r-- | doc/ci/pipelines/img/collapsible_log_v12_6.png | bin | 96471 -> 0 bytes | |||
| -rw-r--r-- | doc/ci/pipelines/img/job_failure_reason.png | bin | 5288 -> 0 bytes | |||
| -rw-r--r-- | doc/ci/pipelines/img/job_group_v12_10.png | bin | 5436 -> 0 bytes | |||
| -rw-r--r-- | doc/ci/pipelines/img/manual_job_variables.png | bin | 111239 -> 0 bytes | |||
| -rw-r--r-- | doc/ci/pipelines/img/pipeline_incremental_rollout.png | bin | 4794 -> 0 bytes | |||
| -rw-r--r-- | doc/ci/pipelines/img/pipelines_grouped.png | bin | 12888 -> 0 bytes | |||
| -rw-r--r-- | doc/ci/pipelines/img/pipelines_mini_graph_sorting.png | bin | 10742 -> 0 bytes | |||
| -rw-r--r-- | doc/ci/pipelines/index.md | 244 | ||||
| -rw-r--r-- | doc/ci/pipelines/job_artifacts.md | 12 | ||||
| -rw-r--r-- | doc/ci/pipelines/pipeline_efficiency.md | 2 | ||||
| -rw-r--r-- | doc/ci/pipelines/schedules.md | 14 | ||||
| -rw-r--r-- | doc/ci/pipelines/settings.md | 19 |
12 files changed, 55 insertions, 236 deletions
diff --git a/doc/ci/pipelines/img/collapsible_log_v12_6.png b/doc/ci/pipelines/img/collapsible_log_v12_6.png Binary files differdeleted file mode 100644 index a1e9aeb244a..00000000000 --- a/doc/ci/pipelines/img/collapsible_log_v12_6.png +++ /dev/null diff --git a/doc/ci/pipelines/img/job_failure_reason.png b/doc/ci/pipelines/img/job_failure_reason.png Binary files differdeleted file mode 100644 index d44b8e6d1be..00000000000 --- a/doc/ci/pipelines/img/job_failure_reason.png +++ /dev/null diff --git a/doc/ci/pipelines/img/job_group_v12_10.png b/doc/ci/pipelines/img/job_group_v12_10.png Binary files differdeleted file mode 100644 index 27e6bfbfc0f..00000000000 --- a/doc/ci/pipelines/img/job_group_v12_10.png +++ /dev/null diff --git a/doc/ci/pipelines/img/manual_job_variables.png b/doc/ci/pipelines/img/manual_job_variables.png Binary files differdeleted file mode 100644 index a5ed351fdcd..00000000000 --- a/doc/ci/pipelines/img/manual_job_variables.png +++ /dev/null diff --git a/doc/ci/pipelines/img/pipeline_incremental_rollout.png b/doc/ci/pipelines/img/pipeline_incremental_rollout.png Binary files differdeleted file mode 100644 index b3498e9a5a5..00000000000 --- a/doc/ci/pipelines/img/pipeline_incremental_rollout.png +++ /dev/null diff --git a/doc/ci/pipelines/img/pipelines_grouped.png b/doc/ci/pipelines/img/pipelines_grouped.png Binary files differdeleted file mode 100644 index 82814754747..00000000000 --- a/doc/ci/pipelines/img/pipelines_grouped.png +++ /dev/null diff --git a/doc/ci/pipelines/img/pipelines_mini_graph_sorting.png b/doc/ci/pipelines/img/pipelines_mini_graph_sorting.png Binary files differdeleted file mode 100644 index 3a4e5453360..00000000000 --- a/doc/ci/pipelines/img/pipelines_mini_graph_sorting.png +++ /dev/null diff --git a/doc/ci/pipelines/index.md b/doc/ci/pipelines/index.md index f7e3698b6d4..f22d2373e5f 100644 --- a/doc/ci/pipelines/index.md +++ b/doc/ci/pipelines/index.md @@ -68,7 +68,7 @@ Pipelines can be configured in many different ways: Pipelines and their component jobs and stages are defined in the CI/CD pipeline configuration file for each project. -- Jobs are the [basic configuration](../yaml/README.md#introduction) component. +- [Jobs](../jobs/index.md) are the basic configuration component. - Stages are defined by using the [`stages`](../yaml/README.md#stages) keyword. For a list of configuration options in the CI pipeline file, see the [GitLab CI/CD Pipeline Configuration Reference](../yaml/README.md). @@ -79,6 +79,27 @@ You can also configure specific aspects of your pipelines through the GitLab UI. - [Pipeline schedules](schedules.md). - [Custom CI/CD variables](../variables/README.md#custom-environment-variables). +### Ref Specs for Runners + +When a runner picks a pipeline job, GitLab provides that job's metadata. This includes the [Git refspecs](https://git-scm.com/book/en/v2/Git-Internals-The-Refspec), +which indicate which ref (branch, tag, and so on) and commit (SHA1) are checked out from your +project repository. + +This table lists the refspecs injected for each pipeline type: + +| Pipeline type | Refspecs | +|--------------- |---------------------------------------- | +| Pipeline for Branches | `+refs/pipelines/<id>:refs/pipelines/<id>` and `+refs/heads/<name>:refs/remotes/origin/<name>` | +| pipeline for Tags | `+refs/pipelines/<id>:refs/pipelines/<id>` and `+refs/tags/<name>:refs/tags/<name>` | +| [Pipeline for Merge Requests](../merge_request_pipelines/index.md) | `+refs/pipelines/<id>:refs/pipelines/<id>` | + +The refs `refs/heads/<name>` and `refs/tags/<name>` exist in your +project repository. GitLab generates the special ref `refs/pipelines/<id>` during a +running pipeline job. This ref can be created even after the associated branch or tag has been +deleted. It's therefore useful in some features such as [automatically stopping an environment](../environments/index.md#automatically-stopping-an-environment), +and [merge trains](../merge_request_pipelines/pipelines_for_merged_results/merge_trains/index.md) +that might run pipelines after branch deletion. + ### View pipelines You can find the current and historical pipeline runs under your project's @@ -157,7 +178,7 @@ For each `var` or `file_var`, a key and value are required. > [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/7931) in GitLab 8.15. -Manual actions, configured using the [`when:manual`](../yaml/README.md#whenmanual) parameter, +Manual actions, configured using the [`when:manual`](../yaml/README.md#whenmanual) keyword, allow you to require manual interaction before moving forward in the pipeline. You can do this straight from the pipeline graph. Just click the play button @@ -174,7 +195,7 @@ stage has a job with a manual action. > [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/27188) in GitLab 11.11. Multiple manual actions in a single stage can be started at the same time using the "Play all manual" button. -Once you click this button, each individual manual action is triggered and refreshed +After you click this button, each individual manual action is triggered and refreshed to an updated status. This functionality is only available: @@ -266,223 +287,6 @@ preserving deployment keys and other credentials from being unintentionally accessed. In order to ensure that jobs intended to be executed on protected runners do not use regular runners, they must be tagged accordingly. -## View jobs in a pipeline - -When you access a pipeline, you can see the related jobs for that pipeline. - -Clicking an individual job shows you its job log, and allows you to: - -- Cancel the job. -- Retry the job. -- Erase the job log. - -### See why a job failed - -> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/17782) in GitLab 10.7. - -When a pipeline fails or is allowed to fail, there are several places where you -can find the reason: - -- In the [pipeline graph](#visualize-pipelines), on the pipeline detail view. -- In the pipeline widgets, in the merge requests and commit pages. -- In the job views, in the global and detailed views of a job. - -In each place, if you hover over the failed job you can see the reason it failed. - - - -In [GitLab 10.8](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/17814) and later, -you can also see the reason it failed on the Job detail page. - -### The order of jobs in a pipeline - -The order of jobs in a pipeline depends on the type of pipeline graph. - -- For [regular pipeline graphs](#regular-pipeline-graphs), jobs are sorted by name. -- For [pipeline mini graphs](#pipeline-mini-graphs), jobs are sorted by severity and then by name. - -The order of severity is: - -- failed -- warning -- pending -- running -- manual -- scheduled -- canceled -- success -- skipped -- created - -For example: - - - -### Group jobs in a pipeline - -> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/6242) in GitLab 8.12. - -If you have many similar jobs, your [pipeline graph](#visualize-pipelines) becomes long and hard -to read. - -You can automatically group similar jobs together. If the job names are formatted in a certain way, -they are collapsed into a single group in regular pipeline graphs (not the mini graphs). - -You can recognize when a pipeline has grouped jobs if you don't see the retry or -cancel button inside them. Hovering over them shows the number of grouped -jobs. Click to expand them. - - - -To create a group of jobs, in the [CI/CD pipeline configuration file](../yaml/README.md), -separate each job name with a number and one of the following: - -- A slash (`/`), for example, `test 1/3`, `test 2/3`, `test 3/3`. -- A colon (`:`), for example, `test 1:3`, `test 2:3`, `test 3:3`. -- A space, for example `test 0 3`, `test 1 3`, `test 2 3`. - -You can use these symbols interchangeably. - -In the example below, these three jobs are in a group named `build ruby`: - -```yaml -build ruby 1/3: - stage: build - script: - - echo "ruby1" - -build ruby 2/3: - stage: build - script: - - echo "ruby2" - -build ruby 3/3: - stage: build - script: - - echo "ruby3" -``` - -In the pipeline, the result is a group named `build ruby` with three jobs: - - - -The jobs are be ordered by comparing the numbers from left to right. You -usually want the first number to be the index and the second number to be the total. - -[This regular expression](https://gitlab.com/gitlab-org/gitlab/blob/2f3dc314f42dbd79813e6251792853bc231e69dd/app/models/commit_status.rb#L99) -evaluates the job names: `\d+[\s:\/\\]+\d+\s*`. - -### Specifying variables when running manual jobs - -> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/30485) in GitLab 12.2. - -When running manual jobs you can supply additional job specific variables. - -You can do this from the job page of the manual job you want to run with -additional variables. To access this page, click on the **name** of the manual job in -the pipeline view, *not* the play (**{play}**) button. - -This is useful when you want to alter the execution of a job that uses -[custom environment variables](../variables/README.md#custom-environment-variables). -Add a variable name (key) and value here to override the value defined in -[the UI or `.gitlab-ci.yml`](../variables/README.md#custom-environment-variables), -for a single run of the manual job. - - - -### Delay a job - -> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/21767) in GitLab 11.4. - -When you do not want to run a job immediately, you can use the [`when:delayed`](../yaml/README.md#whendelayed) parameter to -delay a job's execution for a certain period. - -This is especially useful for timed incremental rollout where new code is rolled out gradually. - -For example, if you start rolling out new code and: - -- Users do not experience trouble, GitLab can automatically complete the deployment from 0% to 100%. -- Users experience trouble with the new code, you can stop the timed incremental rollout by canceling the pipeline - and [rolling](../environments/index.md#retrying-and-rolling-back) back to the last stable version. - - - -### Expand and collapse job log sections - -> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/14664) in GitLab 12.0. - -Job logs are divided into sections that can be collapsed or expanded. Each section displays -the duration. - -In the following example: - -- Two sections are collapsed and can be expanded. -- Three sections are expanded and can be collapsed. - - - -#### Custom collapsible sections - -> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/14664) in GitLab 12.0. - -You can create [collapsible sections in job logs](../pipelines/index.md#expand-and-collapse-job-log-sections) -by manually outputting special codes -that GitLab uses to determine what sections to collapse: - -- Section start marker: `section_start:UNIX_TIMESTAMP:SECTION_NAME\r\e[0K` + `TEXT_OF_SECTION_HEADER` -- Section end marker: `section_end:UNIX_TIMESTAMP:SECTION_NAME\r\e[0K` - -You must add these codes to the script section of the CI configuration. For example, -using `echo`: - -```yaml -job1: - script: - - echo -e "section_start:`date +%s`:my_first_section\r\e[0KHeader of the 1st collapsible section" - - echo 'this line should be hidden when collapsed' - - echo -e "section_end:`date +%s`:my_first_section\r\e[0K" -``` - -In the example above: - -- `date +%s`: The Unix timestamp (for example `1560896352`). -- `my_first_section`: The name given to the section. -- `\r\e[0K`: Prevents the section markers from displaying in the rendered (colored) - job log, but they are displayed in the raw job log. To see them, in the top right - of the job log, click **{doc-text}** (**Show complete raw**). - - `\r`: carriage return. - - `\e[0K`: clear line ANSI escape code. - -Sample raw job log: - -```plaintext -section_start:1560896352:my_first_section\r\e[0KHeader of the 1st collapsible section -this line should be hidden when collapsed -section_end:1560896353:my_first_section\r\e[0K -``` - -#### Pre-collapse sections - -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/198413) in GitLab 13.5. - -You can make the job log automatically collapse collapsible sections by adding the `collapsed` option to the section start. -Add `[collapsed=true]` after the section name and before the `\r`. The section end marker -remains unchanged: - -- Section start marker with `[collapsed=true]`: `section_start:UNIX_TIMESTAMP:SECTION_NAME[collapsed=true]\r\e[0K` + `TEXT_OF_SECTION_HEADER` -- Section end marker: `section_end:UNIX_TIMESTAMP:SECTION_NAME\r\e[0K` - -Add the updated section start text to the CI configuration. For example, -using `echo`: - -```yaml -job1: - script: - - echo -e "section_start:`date +%s`:my_first_section[collapsed=true]\r\e[0KHeader of the 1st collapsible section" - - echo 'this line should be hidden automatically after loading the job log' - - echo -e "section_end:`date +%s`:my_first_section\r\e[0K" -``` - ## Visualize pipelines > [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/5742) in GitLab 8.11. diff --git a/doc/ci/pipelines/job_artifacts.md b/doc/ci/pipelines/job_artifacts.md index d904452a011..160399e4bc4 100644 --- a/doc/ci/pipelines/job_artifacts.md +++ b/doc/ci/pipelines/job_artifacts.md @@ -44,7 +44,7 @@ are relative to the repository that was cloned during the build. By default, the artifacts upload when the job succeeds. You can also set artifacts to upload when the job fails, or always, by using [`artifacts:when`](../yaml/README.md#artifactswhen) -parameter. GitLab keeps these uploaded artifacts for 1 week, as defined +keyword. GitLab keeps these uploaded artifacts for 1 week, as defined by the `expire_in` definition. You can keep the artifacts from expiring via the [web interface](#browsing-artifacts). If the expiry time is not defined, it defaults to the [instance wide setting](../../user/admin_area/settings/continuous_integration.md#default-artifacts-expiration). @@ -114,7 +114,9 @@ There are a couple of exceptions to the [original dotenv rules](https://github.c - The variable key can contain only letters, digits, and underscores (`_`). - The maximum size of the `.env` file is 5 KB. -- The maximum number of variables is 10. +- In GitLab 13.5 and older, the maximum number of inherited variables is 10. +- In [GitLab 13.6 and later](https://gitlab.com/gitlab-org/gitlab/-/issues/247913), + the maximum number of inherited variables is 20. - Variable substitution in the `.env` file is not supported. - The `.env` file can't have empty lines or comments (starting with `#`). - Key values in the `env` file cannot have spaces or newline characters (`\n`), including when using single or double quotes. @@ -137,10 +139,10 @@ third party ports for other languages like JavaScript, Python, Ruby, and so on. > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/207528) in GitLab 13.0. > - Requires [GitLab Runner](https://docs.gitlab.com/runner/) 11.5 and above. -The `terraform` report obtains a Terraform `tfplan.json` file. [JQ processing required to remove credentials](../../user/infrastructure/index.md#output-terraform-plan-information-into-a-merge-request). The collected Terraform +The `terraform` report obtains a Terraform `tfplan.json` file. [JQ processing required to remove credentials](../../user/infrastructure/mr_integration.md#setup). The collected Terraform plan report uploads to GitLab as an artifact and displays in merge requests. For more information, see -[Output `terraform plan` information into a merge request](../../user/infrastructure/index.md#output-terraform-plan-information-into-a-merge-request). +[Output `terraform plan` information into a merge request](../../user/infrastructure/mr_integration.md). #### `artifacts:reports:codequality` @@ -414,7 +416,7 @@ information in the UI. ## Erasing artifacts -DANGER: **Danger:** +DANGER: **Warning:** This is a destructive action that leads to data loss. Use with caution. You can erase a single job via the UI, which also removes the job's diff --git a/doc/ci/pipelines/pipeline_efficiency.md b/doc/ci/pipelines/pipeline_efficiency.md index c4febba8f44..149c596430c 100644 --- a/doc/ci/pipelines/pipeline_efficiency.md +++ b/doc/ci/pipelines/pipeline_efficiency.md @@ -222,6 +222,8 @@ Methods to reduce Docker image size: - Create a dedicated development image. - Disable man pages and docs installed by packages to save space. - Reduce the `RUN` layers and combine software installation steps. +- Use [multi-stage builds](https://blog.alexellis.io/mutli-stage-docker-builds/) + to merge multiple Dockerfiles that use the builder pattern into one Dockerfile, which can reduce image size. - If using `apt`, add `--no-install-recommends` to avoid unnecessary packages. - Clean up caches and files that are no longer needed at the end. For example `rm -rf /var/lib/apt/lists/*` for Debian and Ubuntu, or `yum clean all` for RHEL and CentOS. diff --git a/doc/ci/pipelines/schedules.md b/doc/ci/pipelines/schedules.md index bcdb7c4c8b6..9df73957293 100644 --- a/doc/ci/pipelines/schedules.md +++ b/doc/ci/pipelines/schedules.md @@ -85,10 +85,10 @@ job: ### Advanced configuration -The pipelines won't be executed exactly on schedule because schedules are handled by +The pipelines are not executed exactly on schedule because schedules are handled by Sidekiq, which runs according to its interval. -For example, only two pipelines will be created per day if: +For example, only two pipelines are created per day if: - You set a schedule to create a pipeline every minute (`* * * * *`). - The Sidekiq worker runs on 00:00 and 12:00 every day (`0 */12 * * *`). @@ -112,8 +112,8 @@ To trigger a pipeline schedule manually, click the "Play" button:  -This will schedule a background job to run the pipeline schedule. A flash -message will provide a link to the CI/CD Pipeline index page. +This schedules a background job to run the pipeline schedule. A flash +message provides a link to the CI/CD Pipeline index page. NOTE: **Note:** To help avoid abuse, users are rate limited to triggering a pipeline once per @@ -124,12 +124,12 @@ minute. Pipelines are executed as a user, who owns a schedule. This influences what projects and other resources the pipeline has access to. If a user does not own a pipeline, you can take ownership by clicking the **Take ownership** button. -The next time a pipeline is scheduled, your credentials will be used. +The next time a pipeline is scheduled, your credentials are used.  -If the owner of a pipeline schedule doesn't have the ability to create -pipelines on the target branch, the schedule will stop creating new +If the owner of a pipeline schedule does not have the ability to create +pipelines on the target branch, the schedule stops creating new pipelines. This can happen if, for example: diff --git a/doc/ci/pipelines/settings.md b/doc/ci/pipelines/settings.md index 143a5346e88..d2d2cb26256 100644 --- a/doc/ci/pipelines/settings.md +++ b/doc/ci/pipelines/settings.md @@ -26,10 +26,11 @@ There are two options. Using: - `git clone`, which is slower since it clones the repository from scratch for every job, ensuring that the local working copy is always pristine. -- `git fetch`, which is faster as it re-uses the local working copy (falling +- `git fetch`, which is GitLab's default and faster as it re-uses the local working copy (falling back to clone if it doesn't exist). + This is recommended, especially for [large repositories](../large_repositories/index.md#git-strategy). -The default Git strategy can be overridden by the [GIT_STRATEGY variable](../yaml/README.md#git-strategy) +The configured Git strategy can be overridden by the [`GIT_STRATEGY` variable](../runners/README.md#git-strategy) in `.gitlab-ci.yml`. ## Git shallow clone @@ -183,7 +184,7 @@ Job logs and artifacts are [not visible for guest users and non-project members] If **Public pipelines** is enabled (default): - For **public** projects, anyone can view the pipelines and related features. -- For **internal** projects, any logged in user can view the pipelines +- For **internal** projects, any logged in user except [external users](../../user/permissions.md#external-users) can view the pipelines and related features. - For **private** projects, any project member (guest or higher) can view the pipelines and related features. @@ -192,7 +193,7 @@ If **Public pipelines** is disabled: - For **public** projects, anyone can view the pipelines, but only members (reporter or higher) can access the related features. -- For **internal** projects, any logged in user can view the pipelines. +- For **internal** projects, any logged in user except [external users](../../user/permissions.md#external-users) can view the pipelines. However, only members (reporter or higher) can access the job related features. - For **private** projects, only project members (reporter or higher) can view the pipelines or access the related features. @@ -231,6 +232,16 @@ When enabled, any older deployments job are skipped when a new deployment starts For more information, see [Deployment safety](../environments/deployment_safety.md). +## Retry outdated jobs + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/211339) in GitLab 13.6. + +A deployment job can fail because a newer one has run. If you retry the failed deployment job, the +environment could be overwritten with older source code. If you click **Retry**, a modal warns you +about this and asks for confirmation. + +For more information, see [Deployment safety](../environments/deployment_safety.md). + ## Pipeline Badges In the pipelines settings page you can find pipeline status and test coverage |
